WO2022037548A1

WO2022037548A1 - Mri spinal image keypoint detection method based on deep learning

Info

Publication number: WO2022037548A1
Application number: PCT/CN2021/112874
Authority: WO
Inventors: 刘刚; 郑友怡; 方向前; 马成龙; 赵兴
Original assignee: 浙江大学
Priority date: 2020-08-17
Filing date: 2021-08-16
Publication date: 2022-02-24
Also published as: JP2023530023A; CN112184617A; JP7489732B2; CN112184617B

Abstract

The title of the invention is an MRI spinal image keypoint detection method based on deep learning. Disclosed is an MRI spinal image keypoint detection method based on deep learning. The method comprises: firstly, detecting and positioning a vertebra in an MRI spinal image by using a deep target detection network, and identifying a sacrum 1 (S1) as a positioned vertebra; then, in conjunction with structural information of a spine, filtering false positive detection results and determining a fine-grained label of each vertebra; next, respectively detecting six keypoints in total, namely, UA, UM, UP, LA, LM and LP, on upper and lower borders of each vertebra by using a keypoint detection network; afterwards, in conjunction with edge information, determining and correcting a keypoint position of each vertebra; and finally, developing interactive visual MRI spinal image keypoint automatic annotation software. By means of the present invention, an MRI spinal image keypoint can be automatically extracted, which has immense application value in terms of medical image analysis, medical treatment assistance, etc.

Description

A deep learning-based method for detecting key points in spine MRI images

technical field

The invention belongs to the fields of computer vision and artificial intelligence, and particularly relates to a method for detecting key points of spine MRI images based on deep learning.

Background technique

Artificial intelligence technology has been widely used in the medical field in recent years, among which computer vision has great application potential in medical image analysis. The present invention aims to detect the key points of the spine MRI image, and the previous key point detection work of the spine MRI image mostly relies on the manual annotation of experts. Manual annotation is inefficient and subject to the subjective influence of experts, especially not suitable for large-scale data processing and analysis. At present, most of the methods that try to use artificial intelligence technology to detect are using the underlying features of the image, such as the paper (Ebrahimi S, Angelini E, Gajny L, et al.Lumbar spine posterior corner detection in X-rays using Haar-based features[C]// In 2016 IEEE 13th international symposium on biomedical imaging (ISBI). IEEE, 2016: 180-183.), the Harr feature is used to detect the corners of the vertebrae, but this method only uses the underlying information of the image, and the robustness is poor, only suitable for In some specific scenarios, it is not suitable for complex and changeable medical scenarios. The present invention realizes a more robust and accurate spinal MRI image key point detection method by establishing a high-quality data set and utilizing the excellent learning ability of deep learning.

SUMMARY OF THE INVENTION

The present invention is achieved through the following technical solutions: a deep learning-based spine MRI image key point detection method, comprising the following steps:

Step 1: Input the spine MRI image into the trained target detection network to obtain the position information of each vertebra and whether it is the coarse-grained label of S1;

Step 2: Use all the vertebrae obtained in step 1 and the located S1 position, and combine with the physiological structure information of the spine itself to filter the false positive detection results and identify the fine-grained labels to which each vertebra belongs.

Step 3: Cut out the vertebrae detected in step 2 and some surrounding areas, and input them into the trained key point detection network to detect the upper and lower boundaries of each vertebra, UA, UM, UP, LA, LM, LP, a total of six key points location information.

Step 4: Use the trained segmentation network to segment the vertebrae obtained in step 2 to obtain edge information, and modify the position information of the key points obtained in step 3 according to the edge information obtained in step 4 to obtain the final key point prediction result.

Further, in the first step, the coarse-grained labels of the vertebrae are S1 and NS1 (wherein S1 refers to sacral 1, and NS1 refers to all other vertebrae except sacral 1), and the target detection network is YOLOv3.

Further, the third step is realized by the following sub-steps.

1) Using the detected S1 as the positioned vertebra, calculate the height of each vertebra center on the image and sort the detected vertebrae according to the centroid height.

2) According to the physiological structure information of the human spine, from bottom to top, assign corresponding fine-grained labels such as S1, L5, L4, L3, L2, L1, T12, T11 to each detected vertebra.

3) Filter this false positive target by calculating the aspect ratio of the vertebrae and the height of the upper side edge and according to whether the threshold requirements are met.

Further, the aspect ratio threshold is 1.6, and the upper edge height threshold is 5.

Further, the step 4 is specifically:

(4.1) Constructing the vertebral edge segmentation network: The vertebral edge segmentation network consists of a down-sampling part and an up-sampling part. The structure of the down-sampling part is to remove the resnet50 of the fully connected layer. The up-sampling part is composed of the corresponding four-stage up-sampling convolution blocks. The up-sampling convolution block structure is upsampling->conv->bn->relu.

(4.2) Training the vertebral edge segmentation network: First, use the key point annotation information to establish a coarse-grained segmentation data set to pre-train the vertebral edge segmentation network, and then construct an accurate fine-grained segmentation data set to further train the segmentation network.

(4.3) After the segmentation results are obtained, the segmentation results are further modified by using the characteristics of the CRF and the large gradient of the image at the edge to obtain more accurate edge segmentation information.

Further, the step (4.3) is specifically:

(4.3.1) Make an extension line connecting the two key points, use the vertebral edge segmentation network to obtain the vertebral edge information, and take the farthest intersection point between the extension line and the vertebral edge as the corrected key point coordinates.

(4.3.2) Obtain the label in combination with step 2 to obtain the final key point prediction result;

The beneficial effect of the present invention is that the present invention utilizes the deep learning technology to realize the key point detection of the spine MRI image, avoids the tedious manual labeling, and reduces the burden on the doctor. Compared with manual annotation by experts, the present invention can avoid the influence of subjective factors of doctors, and can also process large-scale data in batches, providing a data basis for further spinal MRI image analysis, and the method of the present invention can be developed into an interactive visual MRI spinal image. Key point automatic labeling software, the obtained key point detection results can calculate the intervertebral disc height index and lumbar anterior process angle.

Description of drawings

FIG. 1 is a schematic diagram of detecting key points in the present invention.

Fig. 2 is the overall flow chart of the present invention.

FIG. 3 is a schematic diagram of the key point correction of the present invention.

detailed description

The present invention will be described in detail below with reference to the accompanying drawings.

Referring to Fig. 1, the data sets of the training network in the present invention are all self-built data sets, and the key points of the spine are marked by doctors and experts, and the key points of detection are the upper and lower boundaries of each vertebra. UA, UM, UP, LA, LM, LP A total of six key points. The basic process of model training is as follows:

1. Collect spine MRI images and randomly select a part of the images as the initial data set.

2. Label or correct the dataset, and use the labeled dataset to train the model.

3. Use the trained model to predict the newly acquired spinal MRI images and add them to the dataset.

4. Repeat steps 2 and 3 until the model accuracy meets the usage requirements.

The target detection network described in Figure 2 is preferably YOLOv3. In the training process, the vertebra is divided into two categories: S1 and NS1 (sacral 1 and non-sacral 1) according to whether the vertebra is S1 or not, as coarse-grained label information. The data set for training the detection network is constructed from the original data set, and the YOLOv3 network is trained according to the key point labels of each vertebra to calculate the bounding boxes of the vertebrae and the coarse-grained category labels.

After obtaining the position of the vertebrae in the spine MRI image and whether it is the coarse-grained category information of S1, the method uses the structural information of the spine to filter the false positive prediction results and determines the category to which each vertebra belongs. The specific method is: use S1 as the To locate the vertebrae, calculate the height of each vertebra center on the image and sort the detected vertebrae according to the centroid height. Then, according to the physiological structure information of the human spine, from bottom to top, corresponding labels such as S1, L5, L4, L3, L2, L1, T12, T11 are assigned to each detected vertebra. The vertebrae may not be completely captured at the top of the picture. This method filters such objects by calculating the aspect ratio of the vertebrae and whether the center height meets the threshold requirements. The aspect ratio of the vertebrae can be calculated according to the target detection results.

The key point detection network described in Figure 2 is preferably a U-shaped network or a Stacked Hourglass Network. The training data is the vertebrae image cropped from the original image and the heat map corresponding to the surrounding image and key points. The online hard sample mining method is used in the training process of the point detection network.

There is a certain error in the original data labeling process, which will lead to a certain error at the edge of the spine at the labelled key point position. The method uses the edge information of the spine to further correct the detected key point results. Firstly, this method trains a U-shaped deep convolutional neural network as a vertebral edge segmentation network for segmenting vertebral edges. The vertebral edge segmentation network consists of a down-sampling part and an up-sampling part. The structure of the downsampling part is to remove the Resnet50 of the fully connected layer, and the upsampling part is composed of the corresponding four-stage upsampling convolution blocks. The upsampling convolution block structure is upsampling->conv->bn->relu. In order to improve the accuracy of the vertebral edge segmentation network, this method first uses the key point annotation information to establish a coarse-grained segmentation data set to pre-train the vertebral edge segmentation network, and then constructs an accurate fine-grained segmentation data set to further train the segmentation network. The method also uses the conditional random field (CRF) and the characteristics of large image gradient at the edge to further modify the segmentation result to obtain more accurate edge segmentation information. After the vertebral edge information is obtained by using the vertebral edge segmentation network, this method uses the edge information to correct the detected key points. The edge is an extension line, and the farthest intersection of the extension line and the edge of the vertebra is used as the corrected UA, LA, UM, LM, UP, and LP coordinates. In order to improve the efficiency of the data set production process and facilitate the use of medical personnel, the above method was developed into an interactive visual automatic detection and labeling software.

After the network training is completed, it can be applied to the entire spine MRI image key point detection process. According to the technical solution set forth in the present invention, the present invention needs to complete the following steps in the spine MRI image key point detection process:

Step 1: Input the spine MRI image into the trained target detection network YOLOv3 (Redmon J, Farhadi A.Yolov3:An incremental improvement[J].arXiv preprint arXiv:1804.02767,2018.), obtain the position information of each vertebra and whether it is Coarse-grained labels for S1;

3) Filter this false positive target by calculating the aspect ratio of the vertebrae and the height of the upper side edge and according to whether the threshold requirements are met. The aspect ratio threshold is 1.6 and the top edge height threshold is 5.

Step 3: Cut out the vertebrae detected in step 2 and some surrounding areas, and input them into the trained key point detection network to detect the upper and lower boundaries of each vertebra, UA, UM, UP, LA, LM, LP, a total of six key points heatmap. The coordinates of the key points in the heatmap can be extracted from but not limited to papers (Zhang F, Zhu X, Dai H, et al. Distribution-aware coordinate representation for human pose estimation [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020:7093-7102.) method.

Step 4: Use the trained segmentation network to segment the vertebrae obtained in step 2 to obtain edge information, and modify the position information of key points obtained in step 3 according to the edge information obtained in step 4.

1) Use the vertebral edge segmentation network to obtain the vertebral edge information;

2) This method makes an extension line along the line connecting UA-LA, UM-LM and UP-LP to the edge of the vertebra, and takes the farthest intersection of the extension line and the edge of the vertebra as the corrected UA, LA, UM, LM, UP , LP coordinates.

3) Combine the labels obtained in step 2 to output the final key point prediction result.

Based on the above process, an interactive visual MRI spine image key point automatic labeling software was developed, and based on the key point detection results, the intervertebral disc height index and lumbar protrusion angle were calculated.

The above is the main content of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and the accompanying drawings of the present invention, or directly or indirectly applied in other related technical fields, are similarly included in the present invention. within the scope of patent protection.

Claims

A method for detecting key points in spine MRI images based on deep learning, characterized in that it includes the following steps:

Step 1: Input the spine MRI image into the trained target detection network to obtain the position information of each vertebra and whether it is the coarse-grained label of S1;

Step 2: Use all the vertebrae obtained in step 1 and the located S1 position, and combine the physiological structure information of the spine itself to filter the false positive detection results and identify the fine-grained label to which each vertebra belongs;

Step 3: Cut out the vertebrae detected in step 2 and some surrounding areas, and input them into the trained key point detection network to detect the upper and lower boundaries of each vertebra, UA, UM, UP, LA, LM, LP, a total of six key points location information;

Step 4: Use the trained segmentation network to segment the vertebrae obtained in step 2 to obtain edge information, and modify the position information of the key points obtained in step 3 according to the edge information obtained in step 4 to obtain the final key point prediction The result, specifically:

(4.1) Constructing the vertebral edge segmentation network: The vertebral edge segmentation network consists of a down-sampling part and an up-sampling part; the structure of the down-sampling part is resnet50 that removes the fully connected layer, and the up-sampling part consists of corresponding four-stage up-sampling convolution blocks The structure of the upsampling convolution block is upsampling->conv->bn->relu;

(4.2) Training the vertebral edge segmentation network: first use the key point annotation information to establish a coarse-grained segmentation data set to pre-train the vertebral edge segmentation network, and then construct an accurate fine-grained segmentation data set to further train the segmentation network;

(4.3) After the segmentation result is obtained, the conditional random field and the large gradient of the image at the edge are used to further modify the segmentation result to obtain more accurate edge segmentation information, specifically:

(4.3.1) Make an extension line connecting the two key points, use the vertebral edge segmentation network to obtain the vertebral edge information, and take the farthest intersection of the extension line and the vertebral edge as the corrected key point coordinates;

(4.3.2) Combine the labels obtained in step 2 to output the final key point prediction result.
The deep learning-based method for detecting key points in spine MRI images according to claim 1, wherein in step 1, the coarse-grained labels of vertebrae are S1 and NS1, wherein S1 refers to sacral 1, and NS1 refers to sacral 1 except for sacral 1. For all other vertebrae, the object detection network is YOLOv3.
The deep learning-based spine MRI image key point detection method according to claim 1, wherein the step 2 is realized by the following sub-steps;

(2.1) Using the detected S1 as the positioned vertebra, calculate the height of the center of each vertebra on the image and sort the detected vertebrae according to the centroid height;

(2.2) According to the physiological structure information of the human spine, assign the corresponding S1, L5, L4, L3, L2, L1, T12, T11 fine-grained labels to each detected vertebra from bottom to top;

(2.3) By calculating the aspect ratio of the vertebrae and the height of the upper side edge and filtering the false positive targets according to whether the threshold requirements are met.
The deep learning-based method for detecting key points in spine MRI images according to claim 3, wherein the aspect ratio threshold is 1.6, and the upper edge height threshold is 5.