CN113763340A

CN113763340A - Automatic grading method based on multitask deep learning ankylosing spondylitis

Info

Publication number: CN113763340A
Application number: CN202110984170.8A
Authority: CN
Inventors: 韩青; 韩洁; 黄陆光; 朱平; 郑朝晖; 张葵; 丁进; 马晨超
Original assignee: Air Force Medical University of PLA
Current assignee: Air Force Medical University of PLA
Priority date: 2021-08-25
Filing date: 2021-08-25
Publication date: 2021-12-07

Abstract

The invention provides an automatic grading method for deep learning ankylosing spondylitis based on multitask, and relates to the technical field of image processing, wherein the method comprises the steps of obtaining a target sample data set; constructing a hip joint gap target identification network and a hip joint gap segmentation network based on global attention according to a target sample data set; carrying out target identification detection and segmentation on the plurality of preprocessed 2D hip orthotopic X-ray film images according to the hip joint gap target identification network and the hip joint gap segmentation network to obtain a hip joint gap segmentation result; performing edge extraction processing on the hip joint clearance segmentation result to obtain a hip joint clearance contour curve; measuring the distance of the hip joint clearance contour curve to obtain a measurement result; and grading the disease symptoms of the measurement result according to a preset grading system. The method realizes the automatic grading of the ankylosing spondylitis based on the X-ray picture of the hip joint of the multitask deep learning, and improves the automation degree and the accuracy of the hip joint clearance measurement.

Description

Automatic grading method based on multitask deep learning ankylosing spondylitis

Technical Field

The disclosure relates to the technical field of image processing, in particular to an automatic grading method for ankylosing spondylitis based on multitask deep learning.

Background

Ankylosing Spondylitis (AS) comprises a group of interrelated diseases characterized by inflammation of the sacroiliac joint and the spine, peripheral joints and tendon attachment sites, and is mainly manifested in narrowing of the bone space. The complex AS etiology, unclear pathogenesis, atypical early clinical manifestations of patients, lack of specific laboratory indexes and other characteristics all cause great obstacles for clinicians to accurately judge AS early, and serious consequences are often caused when diagnosis and treatment are not timely, irreversible bone destruction is brought to patients, and even the patients are disabled for life. Hip joint involvement is the most common extraspinal arthritis manifestation of ankylosing spondylitis and also a common cause of disability.

Currently, the imaging examination methods commonly used in clinic include X-ray, Computed Tomography (CT), Magnetic Resonance Imaging (MRI), ultrasound, and radio nuclide bone imaging examination. Although MRI is an excellent technique in showing small areas of degenerative changes, conventional X-ray films are still the main examination tool for detecting degenerative diseases of hip and knee joints. The most common clinical examination method is still X-ray, and the imaging examination result can show the structural morphological changes of bones, such as bone erosion, sclerosis, ankylosis, joint gap widening or narrowing. The severity of each ankylosing spondylitis patient may vary, and therefore there are different grading systems to score their severity, and these are based on these characteristics.

In the related technology, the hip joint bone clearance measurement is usually carried out by a professional doctor by using a graduated scale, so that the subjectivity is strong, the time is consumed, and the efficiency is not high; there are also some computer measurement methods, but manual preprocessing such as interactive operation of cutting and centering is needed, and the automation degree is low.

Disclosure of Invention

The embodiment of the disclosure provides an automatic grading method for deep learning ankylosing spondylitis based on multitask, which can solve the problem of low automation degree of hip joint bone gap measurement in the prior art. The technical scheme is as follows:

according to a first aspect of embodiments of the present disclosure, there is provided an automatic grading method for deep learning ankylosing spondylitis based on multitasking, the method comprising:

acquiring a target sample data set; the target sample dataset comprises a plurality of pre-processed 2D hip orthostatic X-ray film images;

constructing a hip joint gap target identification network according to the target sample data set;

constructing a hip joint gap segmentation network based on global attention according to the target sample data set;

carrying out target identification detection and segmentation on the plurality of preprocessed 2D hip orthotopic X-ray film images according to the hip joint gap target identification network and the hip joint gap segmentation network to obtain a hip joint gap segmentation result;

performing edge extraction processing on the hip joint gap segmentation result to obtain a hip joint gap contour curve;

measuring the distance of the hip joint clearance profile curve to obtain a measurement result;

and grading the disease symptoms of the measurement result according to a preset grading system.

The embodiment of the disclosure provides an automatic grading method for ankylosing spondylitis based on multitask deep learning, which comprises the steps of constructing a hip joint gap target identification network and a hip joint gap segmentation network according to a target sample data set when the target sample data set is obtained, carrying out target identification detection and segmentation on a plurality of preprocessed 2D hip orthostatic X-ray film images according to the hip joint gap target identification network and the hip joint gap segmentation network to obtain a hip joint gap segmentation result, carrying out edge extraction and distance measurement on the segmentation result, and finally carrying out disease grading on the measurement result according to a preset grading system.

In one embodiment, said constructing a hip-gap target recognition network from said target sample data set comprises:

marking a region to be segmented, position information of the region to be segmented, a region where the ankylosing spondylitis reaches a fourth-order disease and position information of the region where the ankylosing spondylitis reaches the fourth-order disease in all the 2D hip orthotopic X-ray image images, and making a data set in a first preset format;

storing the preprocessed 2D hip positive X-ray image as an image in a second preset format;

and inputting the image only containing the hip joint bone clearance area and the position information thereof into a preset detection network for training to obtain the hip joint clearance target identification network.

In one embodiment, the inputting the image only containing the hip joint bone gap region and the position information thereof into the first network for training, and obtaining the hip joint gap target identification network comprises:

acquiring a characteristic diagram of an X-ray image only containing a hip joint bone gap area through a convolutional neural network;

generating an anchor point on the feature map, wherein each pixel point is mapped to the original 2D hip orthotopic X-ray image, and setting nine candidate frames by taking each anchor point as the center;

determining whether the interior of each candidate frame contains a target object through a binary classification network, and outputting a probability value containing the target object;

calculating the deviation of the target object determined by the two classification networks through frame regression network regression branches to obtain the translation amount and the transformation scale size required by each candidate frame;

calculating cross entropy loss function L of two-class network_clsSmoothing L1 loss function L of bounding box regression network_reg；

According to the cross entropy loss function L_clsAnd a smoothing L1 loss function L of the bounding box regression network_regObtaining a loss function L_F；

Using a preset optimizer to the loss function L_FMinimization;

minimizing the loss function L by the preset optimizer_FPerforming back propagation to optimize the binary network and the frame regression network until the loss function L_FConverging to obtain the probability value of the target object contained in the candidate box, and the translation amount and the transformation scale size required by the candidate box;

receiving a probability value of a target object contained in a candidate frame, a translation amount and a transformation scale size required by the candidate frame through an extraction layer of a preset detection network to obtain the candidate frame after translation and scale transformation;

and according to a strategy of non-maximum value inhibition, removing the overlapped candidate frames, and reserving the candidate frame with the highest probability value to obtain the hip joint space target identification network.

In one embodiment, the cross-entropy loss function L is based on_clsAnd a smoothing L1 loss function L of the bounding box regression network_regObtaining a loss function L_FThe method comprises the following steps:

according to the formula

Obtaining a loss function L_F；

Where λ is the weighting parameter and σ is the loss function L of the control smoothing L1_regParameter of degree of smoothing, N_clsIs the number of candidate frames, N_regIs the size of the feature map, p_iRepresenting the probability that the ith candidate box is predicted by the classification network to contain the target object,

indicating that the ith candidate box only contains a real label with the target object being 1, t_iThe offset of the ith candidate box representing the bounding box regression network prediction,

representing the true offset of the ith candidate box with respect to the annotated region.

In one embodiment, the first preset percentage of the target sample data set is a first training set, and the second preset percentage of the target sample data set is a first test set;

the constructing a hip joint space segmentation network based on global attention according to the target sample data set comprises:

marking a real mask of a hip joint gap area in all the 2D hip orthostatic X-ray images, and storing the real mask as a mask image;

introducing a global attention up-sampling module GAU in a preset segmentation network;

and inputting the images of the first training set and the mask images into a segmentation network of the global attention up-sampling module for training, and updating weight parameters of each layer in the segmentation network by using a preset algorithm to obtain the hip joint gap segmentation network.

In one embodiment, the introducing a global attention upsampling module GAU in the pre-set partition network comprises:

adding multiple stages of GAU modules in the jump connection of the preset segmentation network, wherein each stage of GAU module receives the output of the upper stage of GAU module and the low-level feature map of the corresponding resolution of the coding stage as input;

each stage of GAU module respectively performs global pooling on the high-level feature map output by the last stage of GAU module, performs convolution on the low-level feature map with the corresponding resolution in the encoding stage, and outputs the two feature maps after the convolution after fusion;

and splicing the output result of each stage of GAU module with the input low-level feature diagram and the up-sampling result of the feature diagram output by the last stage of GAU module, and performing convolution operation and up-sampling step by step on the spliced feature diagram.

In one embodiment, the performing target recognition detection and segmentation on the plurality of 2D hip orthotopic X-ray film images according to the hip joint space target recognition network and the hip joint space segmentation network to obtain a hip joint space segmentation result comprises:

taking the 2D hip positive X-ray image and the data set in the first preset format as input, and performing model training through the hip joint gap target recognition network to obtain a target model;

testing the images of the first test set in the target model to obtain the interested region to be segmented, the position information of the region to be segmented, the region where the ankylosing spondylitis reaches the fourth-level disease and the position information of the region where the ankylosing spondylitis reaches the fourth-level disease;

taking the image of the interested region to be segmented as a segmentation data set; a third preset percentage of the segmented data set is used as a second training set, and a fourth preset percentage of the segmented data set is used as a second test set;

taking the second training set as input, adjusting network parameters of the hip joint gap segmentation network based on global attention, and performing iterative training to obtain a hip joint gap segmentation model;

and inputting the hip joint gap image to be segmented in the second test set into the segmentation model of the hip joint gap for testing to obtain a hip joint gap segmentation result of the second test set.

In one embodiment, the performing an edge extraction process on the hip joint gap segmentation result comprises:

performing opening operation and closing operation on the segmented hip joint gap image;

carrying out threshold processing on the hip joint gap image after the opening operation and the closing operation;

and (5) extracting the contour of the hip joint clearance image after threshold processing by adopting an edge operator to obtain a hip joint clearance contour curve.

In one embodiment, the performing an on operation on the segmented hip joint gap image comprises:

according to the expression

Carrying out opening operation, wherein the opening operation is to carry out corrosion operation firstly and then carry out expansion operation;

wherein the erosion operation is according to an expression

Performing operation;

said dilation operation being in accordance with an expression

Performing operation;

the closed operation of the segmented hip joint gap image comprises the following steps:

according to the expression

Performing a closing operation, wherein the closing operation is to perform an expansion operation and then perform a corrosion operation;

the method for extracting the contour of the hip joint clearance image after threshold processing by adopting the edge operator to obtain the hip joint clearance contour curve comprises the following steps:

according to the formula

Carrying out contour extraction;

wherein the content of the first and second substances,

in order to etch the operator, the etching process,

in order to do the operation of the dilation,

for the open operator,. for the closed operator, D_fAnd D_sF (x, y) is a gray level image of the segmented hip joint gap to be processed, and (x, y) is a pixel coordinate point of the gray level image of the segmented hip joint gap to be processed; s (x, y) is a structural element; n is 0,1, N-1; the value of M is 0, 1.., M-1; m and N are integers, and M<N; f (x, y) is a central pixel value, and three pixel values adjacent to f (x, y) are respectively f (x, y +1), f (x +1, y) and f (x +1, y + 1); t is t₁And t₂Respectively representing the difference values of two pixels in the diagonal direction; gradf (x.y) is (x, y) inGradient values of the heart pixels; h is₁And h₂Respectively, representing the Robert gradient operator, corresponding to h₁And h₂And performing convolution operation on the image.

In an embodiment, the performing distance measurement on the edge after the edge extraction processing to obtain a measurement result includes:

discretizing the first hip joint clearance contour curve, calculating the minimum distance from each discrete point to the second hip joint clearance contour curve, and selecting a preset number of point pairs meeting preset conditions as approximate solutions;

and establishing a corresponding optimization model according to the characteristics of the curves at each point pair, carrying out local optimization, and selecting the maximum distance value in the optimization result as the one-way Hausdorff distance between the first hip joint clearance profile curve and the second hip joint clearance profile curve to obtain a measurement result.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

Fig. 1 is a flowchart of an automatic grading method for deep learning ankylosing spondylitis based on multitasking according to an embodiment of the present disclosure;

fig. 2 is a flowchart of an automatic grading method for deep learning ankylosing spondylitis based on multitasking according to an embodiment of the present disclosure;

fig. 3 is a flowchart of an automatic grading method for deep learning ankylosing spondylitis based on multitasking according to an embodiment of the present disclosure;

fig. 4 is a flowchart of an automatic grading method for deep learning ankylosing spondylitis based on multitasking according to an embodiment of the present disclosure;

fig. 5 is a schematic diagram of a global attention upsampling module provided by an embodiment of the present disclosure;

fig. 6 is a flowchart of an automatic grading method for deep learning ankylosing spondylitis based on multitasking according to an embodiment of the present disclosure;

FIG. 7 is a schematic diagram of a hip joint space segmentation network provided by an embodiment of the present disclosure;

fig. 8 is a flowchart of an automatic grading method for deep learning ankylosing spondylitis based on multitasking according to an embodiment of the present disclosure;

fig. 9 is a flowchart of an automatic grading method for deep learning ankylosing spondylitis based on multitasking according to an embodiment of the present disclosure;

FIG. 10a is a graph of a hip joint gap identification result provided by an embodiment of the present disclosure;

FIG. 10b is a graph of the results of a hip joint gap segmentation provided by an embodiment of the present disclosure;

FIG. 11 is a block diagram of an automatically ranked confusion matrix provided by embodiments of the present disclosure.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

The embodiment of the disclosure provides an automatic grading method for ankylosing spondylitis based on multitask deep learning, which adopts X-ray image data of hip joints, provides clinical characteristics aiming at ankylosing spondylitis grading based on multitask deep learning, and is an automatic measuring method for hip joint gap distance. The multitask deep learning of the hip joint X-ray film comprises a target detection process of positioning the hip joint gap position from the whole orthostatic hip image, accurate medical image segmentation operation on the gap, medical image processing and distance measurement on the basis of the first two tasks, and finally grading of diseases according to the measurement result and the BASRI-h scoring standard of ankylosing spondylitis, wherein the method comprises the following steps of:

step 101, obtaining a target sample data set.

Wherein the target sample dataset comprises a plurality of pre-processed 2D hip orthonormal X-ray film images.

For example, 2D hip orthotopic X-ray images of a plurality of high quality 2D hip orthotopic X-ray images, such as 420, are obtained from 2D hip X-ray images of ankylosing spondylitis patients and normal persons; and aiming at the problem that the model identification and segmentation effects are poor due to small medical image data volume, data generation is carried out on the obtained sample. Taking data extracted from a 2D hip X-ray image as a real sample, randomly enhancing the sample through image processing operations such as rotation, histogram equalization and the like under the condition of not changing the anatomical structure and the imaging proportion of a hip joint, adding a generated new hip joint sample into a total data set to obtain a target sample data set, randomly selecting a first preset percentage of the data set as a first training set, and randomly selecting a second preset percentage of the data set as a first testing set; for example, if the first preset percentage is 70% and the second preset percentage is 30%, the first training set is 70% of the randomly selected target sample data set; the first test set is a randomly selected 30% target sample data set.

And 102, constructing a hip joint gap target identification network according to the target sample data set.

Optionally, as shown in fig. 2, constructing a hip joint space target identification network according to the target sample data set may be implemented by:

step 1021, marking the region to be segmented, the position information of the region to be segmented, the region where the ankylosing spondylitis reaches the fourth-order symptoms and the position information of the region where the ankylosing spondylitis reaches the fourth-order symptoms in all the 2D hip orthotopic X-ray images, and making a data set with a first preset format.

Wherein, the first preset format may be a VOC2007 format.

In an example, the position information of the region to be segmented, the region where ankylosing spondylitis reaches the fourth-order disease, and the position information of the region where ankylosing spondylitis reaches the fourth-order disease are marked in all 2D hip orthotopic X-ray image images, and the position information of the region to be segmented, the region where ankylosing spondylitis reaches the fourth-order disease, and the position information of the region where ankylosing spondylitis reaches the fourth-order disease are made into a data set in VOC2007 format.

And step 1022, storing the preprocessed 2D hip orthostatic X-ray image as an image in a second preset format.

The second preset format is a JPG format.

For example, the pre-processed 2D hip orthostatic X-ray images are stored as pictures in JPG format.

And 1023, inputting the image only containing the hip joint bone clearance area and the position information thereof into a preset detection network for training to obtain the hip joint clearance target identification network.

The preset detection network can be a Faster R-CNN network, and the Faster R-CNN network is a two-stage target detection algorithm.

Further, as shown in fig. 3, inputting the image only including the hip bone gap region and the position information thereof into the preset detection network for training can be implemented by the following steps:

and step 10231, acquiring a characteristic map of the X-ray image only containing the hip joint bone gap area through a convolutional neural network.

And 10232, generating anchor points on the feature map, wherein each pixel point is mapped to the original 2D hip orthotopic X-ray image, and setting nine candidate frames by taking each anchor point as the center.

Step 10233, determining whether each candidate box contains a target object through a binary classification network, and outputting a probability value containing the target object.

And 10234, calculating the deviation of the target object determined by the two classification networks through frame regression network regression branches to obtain the translation amount and the transformation scale size required by each candidate frame.

Step 10235, calculating cross entropy loss function L of the two-class network_clsSmoothing L1 loss function L of bounding box regression network_reg。

10236, according to the cross entropy loss function L_clsAnd a smoothing L1 loss function L of the bounding box regression network_regObtaining a loss function L_F。

Illustratively, according to a formula

Obtaining a loss function L_F。

representing the true deviation of the ith candidate box with respect to the labeled regionAnd (5) moving amount.

10237, using a preset optimizer to said loss function L_FAnd (4) minimizing.

Wherein the preset optimizer may be an Adam (adaptive moment estimation) optimizer.

10238, minimizing the loss function L by the preset optimizer_FPerforming back propagation to optimize the binary network and the frame regression network until the loss function L_FAnd converging to obtain the probability value of the target object contained in the candidate box, and the translation amount and the transformation scale size required by the candidate box.

And step 10239, receiving the probability value of the target object contained in the candidate box, the translation amount and the transformation scale size required by the candidate box through an extraction layer of the preset detection network to obtain the candidate box after translation and scale transformation.

And step 102310, according to a non-maximum value inhibition strategy, eliminating the overlapped candidate frames, and reserving the candidate frame with the highest probability value to obtain the hip joint space target identification network.

Illustratively, according to the strategy of non-maximum suppression, candidate frames overlapped at a local position are removed, only the candidate frame with the highest probability is given by the two preceding classification networks at the position, and therefore the hip joint space target identification network is obtained.

And 103, constructing a hip joint space segmentation network based on global attention according to the target sample data set.

The first preset percentage of the target sample data set is a first training set, and the second preset percentage of the target sample data set is a first test set.

Optionally, as shown in fig. 4, constructing a hip joint space segmentation network based on global attention according to the target sample data set may be implemented by:

step 1031, marking a real mask of the hip joint gap area in all the 2D hip orthotopic X-ray images, and storing the mask as a mask image.

Marking an image of a region to be segmented of the identified bone gap, marking a real mask of the hip joint gap region in all the obtained 2D hip orthotopic X-ray images, and storing the real mask into a mask image to provide conditions for the next segmentation; the acquired X-ray images of all bone spaces were divided into data sets, 70% of which were randomly selected as the first training set, and the remaining 30% were the test sets.

Step 1032, introduce a Global Attention upsampling module GAU (GAU) in the preset segmentation network.

The preset segmentation network can be a UNet segmentation network, the UNet network is an image semantic segmentation network, and the global attention upsampling module is a GAU module.

For example, as shown in fig. 5, the global attention upsampling module includes 3 × 3 convolutional layers, 1 × 1 convolutional layers, a global pooling layer, an input low-level feature map, and a high-level feature map. The 3 x 3 convolutional layers are used for channel processing of the lower level feature maps, and the global pooling layers and the 1 x1 convolutional layers are used for processing of the upper level feature maps. And selecting and fusing the features of the low-level feature map by using the processed high-level feature map, and outputting a result after feature fusion.

Optionally, as shown in fig. 6, introducing a global attention upsampling module GAU in the preset segmentation network may be implemented by the following steps:

step 10321, add multiple stages of GAU modules in the jump connection of the preset segmentation network, where the GAU module of each stage receives as input the output of the GAU module of the previous stage and the low-level feature map of the resolution corresponding to the encoding stage.

And 10322, each stage of GAU module performs global pooling on the high-level feature map output by the previous stage of GAU module, performs convolution (3 × 3 convolution) on the low-level feature map with the resolution corresponding to the encoding stage, and fuses and outputs the two feature maps after the convolution.

And 10323, splicing the output result of each stage of the GAU module with the input low-level feature map and the upsampling result of the feature map output by the last stage of the GAU module, and performing convolution operation on the spliced feature map and upsampling step by step.

And 1033, inputting the image of the first training set and the mask image into a segmentation network of the global attention upsampling module for training, and updating the weight parameters of each layer in the segmentation network by using a preset algorithm (Adam algorithm) to obtain the hip joint gap segmentation network.

An exemplary resulting hip-gap segmentation network is shown in fig. 7, where white boxes represent global pooling pools, slashed boxes represent convolution, and black boxes represent upsampling.

And step 104, carrying out target identification detection and segmentation on the plurality of preprocessed 2D hip orthotopic X-ray film images according to the hip joint gap target identification network and the hip joint gap segmentation network to obtain a hip joint gap segmentation result.

Optionally, since the segmentation result is not fine enough, the mathematical morphology theory is used to improve the problems of roughness, protrusion, indentation, hole, etc. existing in the segmentation result, as shown in fig. 8, the target identification detection and segmentation of the plurality of preprocessed 2D hip orthotopic X-ray images can be implemented by the following steps:

step 1041, taking the 2D hip orthostatic X-ray image and the data set in the first preset format as input, and performing model training through the hip joint gap target identification network to obtain a target model.

Illustratively, model training is performed by a hip joint space target recognition network using as input a 2D hip orthostatic X-ray image and a data set in VOC2007 format labeled with a number.

Step 1042, testing the images of the first test set in the target model to obtain the interested region to be segmented, the position information of the region to be segmented, the region where the ankylosing spondylitis reaches the fourth-order disease, and the position information of the region where the ankylosing spondylitis reaches the fourth-order disease.

Illustratively, the first test set is tested in the model obtained by training in step 1041, and two types of samples of the interested region to be segmented and ankylosing spondylitis up to level four and their corresponding position information are obtained.

And step 1043, taking the image of the interested region to be segmented as a segmentation data set.

And taking a third preset percentage of the segmented data set as a second training set, and taking a fourth preset percentage of the segmented data set as a second test set.

Illustratively, the third preset percentage is 70%, the fourth preset percentage is 30%, and the image of the region to be segmented obtained in step 1042 is used as the segmentation data set, where 70% is used as the second training set, and the other 30% is used as the second testing set.

And step 1044, taking the second training set as input, adjusting network parameters of the hip joint gap segmentation network based on global attention, and performing iterative training to obtain a hip joint gap segmentation model.

Step 1045, inputting the hip joint gap image to be segmented in the second test set into the segmentation model of the hip joint gap for testing, and obtaining the hip joint gap segmentation result of the second test set.

And 105, performing edge extraction processing on the hip joint gap segmentation result to obtain a hip joint gap contour curve.

Optionally, performing opening operation and closing operation on the segmented hip joint gap image; performing threshold processing (e.g., Otsu threshold processing) on the hip joint gap image after the opening operation and the closing operation; and (5) extracting the contour of the hip joint clearance image after threshold processing by adopting an edge operator to obtain a hip joint clearance contour curve.

For example, the opening operation on the segmented hip joint gap image can be implemented by the following steps:

according to the expression

And performing an opening operation.

Wherein, the opening operation is to perform the corrosion operation first and then perform the expansion operation.

Wherein the erosion operation is according to an expression

And (6) performing operation.

Said dilation operation being in accordance with an expression

And (6) performing operation.

For example, the closing operation on the segmented hip joint gap image can be implemented by the following steps:

according to the expression

And performing a closing operation.

In the formula (I), the compound is shown in the specification,

in order to etch the operator, the etching process,

in order to do the operation of the dilation,

for the open operator,. for the closed operator, D_fAnd D_sF (x, y) is a gray scale image of the segmented hip joint gap to be processed, and (x, y) is a pixel coordinate point of the gray scale image of the segmented hip joint gap to be processed; s (x, y) is a structural element; n is 0,1, N-1; the value of M is 0, 1.., M-1; m and N are integers, and M<N。

Wherein, the closed operation is to perform expansion operation first and then perform corrosion operation.

Specifically, the opening operation can smooth the contour of the image, remove noise existing in the image, and the closing operation can smooth the contour and close the holes.

Furthermore, after the opening and closing operation is performed, the hip joint bone gap image after the opening and closing operation is subjected to threshold processing, and Otsu is a method for performing binarization by performing global adaptive threshold on the image. The method divides the image into a foreground part and a background part according to the gray characteristic of the image. Since variance is a measure of the uniformity of the gray scale distribution, the larger the between-class variance of the foreground and background, the larger the difference between the two parts making up the image, so we can consider that the best result is when the threshold is chosen to maximize the between-class variance, which is the result of Otsu. For an image I, assuming that the size of the image is M multiplied by N, the threshold value is T, the pixel points with the gray value lower than T in the image are N0 types, the pixel points with the gray value higher than the threshold value T are N1 types, and if N pixel points are higher than T as a foreground, L-N pixel points are lower than T as a background. Let P0 and P1 denote the probability of foreground and background correspondence, respectively, and U0 and U1 correspond to the average gray levels of N0 and N1, respectively. X_iIs the number of pixels with gray level i, C_iThat is, the probability of the occurrence of the pixel point with the gray level i is as follows:

it is derived that the maximum inter-class variance g (n) is the optimal threshold T for image segmentation, where:

g(n)＝P₀(n)×U₀(n)²+P₁(n)×U₁(n)²

further, contour extraction is carried out on the hip joint gap image after threshold processing by adopting an edge operator, and a hip joint gap contour curve can be obtained by the following method:

according to the formula

And (5) extracting the contour.

The edge operator can be a Robert operator, and the Robert operator is an operator for searching for an edge by using a local difference operator; after contour extraction is carried out by using an edge operator, a continuous, smooth and noiseless hip joint clearance contour curve can be obtained.

When the template is used for representation, the following steps are provided:

the calculation principle of the Robert gradient is that a template h is used₁Calculated absolute value and template h₂The sum of the calculated absolute values replaces the gradient value of the central pixel (x, y). According to the method, the images are traversed in sequence to obtain a gradient map, and therefore edge extraction is achieved.

Wherein the ratio of f (x,y) is a central pixel value, and three pixel values adjacent to f (x, y) are respectively f (x, y +1), f (x +1, y) and f (x +1, y + 1); t is t₁And t₂Respectively representing the difference values of two pixels in the diagonal direction; gradf (x.y) is a gradient value of a pixel centered at (x, y); h is₁And h₂Respectively, representing the Robert gradient operator, corresponding to h₁And h₂And performing convolution operation on the image.

And step 106, performing distance measurement on the hip joint clearance profile curve to obtain a measurement result.

In an example, the obtained hip joint clearance contour curve is defined as a first hip joint clearance contour curve and a second hip joint clearance contour curve, a distance matrix between the two curves is obtained through an algorithm for calculating the Housdov distance between the plane curves, and a numerical value of the shortest distance is obtained; the obtained pixel distance is converted into an actual distance based on information such as the resolution of the input image, and an error index of the distance measured by the contour obtained by different segmentation methods is quantitatively evaluated.

Optionally, discretizing the first hip joint gap contour curve, calculating the minimum distance between each discrete point and the second hip joint gap contour curve, and selecting a preset number of point pairs meeting preset conditions as an approximate solution; and establishing a corresponding optimization model according to the characteristics of the curves at each point pair, carrying out local optimization, and selecting the maximum distance value in the optimization result as the one-way Hausdorff distance between the first hip joint clearance profile curve and the second hip joint clearance profile curve to obtain a measurement result. The method converts the calculation of the Hausdorff distance between the plane curves into the calculation of the minimum distance from the point to the curve, and the calculation process is simple and effective.

The preset condition is that a point pair which has a larger distance and satisfies that the distance from the adjacent point on the first hip joint clearance contour curve to the second hip joint clearance contour curve is small is selected.

And 107, grading the disease symptoms of the measurement result according to a preset grading system.

In an example, evaluation and grading are performed on the measurement results according to a BASRI-h scoring system, the BASRI-h scoring system can evaluate the severity of the imaging lesion of the hip joint in the ankylosing spondylitis, the hip joint clearance distance can be divided into 5 grades, and after the distance result of the hip joint clearance is automatically measured according to the disclosure, the result of grading the ankylosing spondylitis is obtained according to the BASRI-h scoring system, and error analysis is performed.

FIG. 9 is a main flowchart of the automatic grading method for ankylosing spondylitis based on multitask deep learning according to the present disclosure, which includes three parts, a first part (I) is a data set preparation and preprocessing part, which mainly includes a network extraction target area, an original image labeling mask, and four-stage patient identification; the second part (II) is a hip joint gap segmentation part and mainly comprises various segmentation network training and testing, and contour information is obtained by edge extraction; the third part (III) is a gap distance measurement and evaluation grading part, a distance matrix is measured after the edge is extracted, and the patients are graded into several grades according to a BASRI-h scoring system through the minimum distance.

The effects of the present disclosure can be further illustrated by the following simulations:

1. simulation conditions

The experiment is carried out on a desktop computer with a Windows10 system by using a python3.5 programming language, a network model is realized by using Tensoflow1.6.0 and Pythroch 1.1.0, and operation acceleration is carried out by using GeForce RTX1080Ti, Cuda10.0 and CuDNN7.4. The simulation data contained 420 2D hip X-ray images, with front and back projection (AP) and image resolution 2021X 2021, from 420 individuals.

2. Emulated content

Simulation 1, inputting the image of the hip joint bone gap region obtained by simulation 1 and the position information (position label) thereof into a preset detection network in pairs for training, and iterating for 4000 times; in order to give consideration to the accuracy and the recall rate of the model, whether a certain prediction box contains a threshold value of the confidence score of a target or not is judged to be 0.4; considering that the probability of overlapping of different types is low, a non-maximum suppression strategy is adopted, and the threshold value of the overlapping area of the candidate frames when the redundant frames are removed is set to be 0.1, so that the overlapped redundant frames are reduced to the maximum extent. And after each experiment is finished, calculating indexes such as accuracy of the identification model and the like.

In recent years, the evaluation notation of target detection that has been widely used was "Average Precision (AP)" which was originally introduced in VOC 2007. AP is defined as the average detection accuracy at different recall rates and is evaluated under a specific category. In order to compare the performance of all target classes, mean AP (mAP) is often used as the final metric of performance, averaged over all target classes. The results of the identification are shown in fig. 10a, and the bone gap to be segmented and the quaternary disorder mAP are 0.86 and 0.79, respectively.

And 2, simulating, namely using the data set for testing the trained model and calculating the related performance index under the conditions.

For hip joint bone gap segmentation, a mask image and an original image of a 2D hip X-ray film are respectively input into the improved hip joint gap segmentation network in pairs for training, the iteration number is set to be 50 times, and the results are compared with the existing segmentation network full convolution network, UNet + +, UNet + residual error network and UNet + attention mechanism, and are shown in FIG. 10 b.

The average Dice coefficient index of the segmentation results of the improved hip joint space segmentation network and the existing segmentation networks UNet, UNet + +, AttentionUNet in the invention are calculated by the following formula respectively:

wherein, X is the segmentation result of the segmentation network, Y is the real result of the label, and the numeric area of the Dice coefficient is [0,1 ]. The larger the Dice coefficient is, the larger the overlapping degree between the segmentation result and the real result of the network is, and the better the segmentation effect is. The smaller the value of the Hausdorff distance is, the smaller the error between the segmentation result and the real result is.

Another evaluation index is the mean cross-over ratio (MIoU):

this is a standard evaluation index for the segmentation problem, which computes the coincidence ratio of the intersection of the two sets to its union, which computes the intersection ratio between the true segmentation and the systematic predicted segmentation. This ratio may be redefined as the number of true positive examples divided by the total number (including true positive examples, false negative examples, and false positive examples). Evaluation index results are shown in table 1, which is an index comparison table of hip joint bone gap segmentation networks and other existing network segmentation results.

TABLE 1

The MIOU index is a standard scale for evaluating semantic segmentation, the average Hausdorff distance is sensitive to a boundary, and the Dice index is a measurement function of set similarity.

As can be seen from the table 1, compared with the existing segmented network full convolution network, UNet + +, UNet + residual error network and UNet + attention mechanism network, the GAU module is introduced into the UNet network, and the obtained hip joint bone gap segmented network has better segmentation performance. Compared with the method that the low-level feature map and the high-level feature map are directly spliced in the UNet network, the GAU module guides the mapping of the low-level features with different scales by utilizing the abundant semantic information of the high-level feature map, more accurate category pixel space positioning information can be provided, and the segmentation performance is improved.

And 3, simulating, namely performing opening and closing operation and threshold processing on the segmented image in sequence, and performing contour extraction by using an edge operator, so as to obtain a continuous, smooth and noiseless hip joint bone gap contour curve. Obtaining a distance matrix between two curves by using the obtained hip joint bone gap contour curve and calculating the distance between plane curves to obtain a numerical value of the shortest distance; the obtained pixel distance is converted into an actual distance based on information such as the resolution of the input image, and an error index of the distance measured by the contour obtained by different segmentation methods is quantitatively evaluated. The evaluation index results are shown in table 2.

TABLE 2

Segmentation method \| evaluation index	SD index	MAD index	RMSE index	F-test
					UNet++	0.6586	0.4549	0.7279	0.9809
UNet + residual network	0.6652	0.3497	0.5326	0.1143
					Full convolution network	0.5977	0.2531	0.4216	0.2301
UNet + attention mechanism	0.6999	0.1984	0.3535	0.4943
					UNet+GAU	0.6661	0.1573	0.2235	0.5066

Where SD is the standard deviation, MAD is the mean absolute deviation, RMSE is the root mean square error, F-test is the F test, joint hypothesis testing.

Simulation 4, evaluation and grading are performed on the results obtained by the measurement according to a basic ri-h scoring system, the ASRI-h scoring system can evaluate the severity of the imaging lesion of the hip joint in ankylosing spondylitis, and the severity can be divided into 5 grades according to the hip joint clearance distance, as shown in table 3, the severity of the imaging lesion of the hip joint in ankylosing spondylitis is graded, including the severity of 5 grades. As shown in table 4, this is a classification result evaluation table for different segmentation methods. With 120 individuals of 2D orthostatic hip X-ray images as the test samples, the ranked confusion matrix is shown in fig. 11 for 0-class 3, 1-class 5, 2-class 83, 3-class 21, and most severe 4-class 8 persons.

TABLE 3

TABLE 4

Where the actual relative error is a percentage of the absolute error versus the fact, expressed as a percentage. The 0 and 1 stages have larger relative errors due to less data, while the 2 and 3 stages have smaller errors.

Compared with the prior art, the method has the following advantages:

first, there is a more intelligent hip joint gap measurement method

According to the method, the existing joint bone clearance measurement method mostly needs manual pretreatment, such as interactive operation of cutting, centering and the like, and the automation degree is low. In addition, most of the current artificial intelligence methods focus on automatic segmentation of bones in X-ray pictures, and key points are mainly extracted from single or multiple bone structures or based on a landmark map method. Therefore, the method based on the multi-task deep learning is provided, and multiple tasks such as hip joint gap identification, bone gap segmentation, edge extraction, distance measurement and disease condition grading are fused together to form a complete automatic measurement and diagnosis process.

Second, better segmentation performance and measurement accuracy

According to the method, the global attention up-sampling module is introduced into the existing UNet network, the segmentation network for hip joint bone gap measurement is constructed, an accurate segmentation result can be obtained, edge extraction and bone gap distance measurement are carried out on the segmentation result, interference of other bone structures is avoided, and the measurement precision is greatly improved.

Thirdly, the method has the effect of directly grading the disease condition and provides a more direct and efficient basis for the diagnosis of doctors

The present disclosure uses the BASRI-h scoring system, a reliable, disease-specific, change-sensitive method for grading hip images of ankylosing spondylitis into five severity levels.

Based on the automatic grading method based on the multitask deep learning ankylosing spondylitis described in the above embodiments, the following is an embodiment of the apparatus of the present disclosure, which can be used to execute the embodiments of the method of the present disclosure.

The embodiment of the present disclosure provides an automatic grading device based on multitask deep learning ankylosing spondylitis, which includes: the device comprises an acquisition module, a first construction module, a second construction module, a segmentation module, an extraction module, a measurement module and a grading module.

The acquisition module is used for acquiring a target sample data set; the target sample dataset comprises a plurality of pre-processed 2D hip orthonormal X-ray film images.

And the first construction module is used for constructing the hip joint gap target identification network according to the target sample data set.

And the second construction module is used for constructing the hip joint gap segmentation network based on the global attention according to the target sample data set.

And the segmentation module is used for carrying out target identification detection and segmentation on the plurality of preprocessed 2D hip orthotopic X-ray film images according to the hip joint gap target identification network and the hip joint gap segmentation network to obtain a hip joint gap segmentation result.

And the extraction module is used for carrying out edge extraction processing on the hip joint gap segmentation result to obtain a hip joint gap contour curve.

And the measuring module is used for measuring the distance of the hip joint clearance profile curve to obtain a measuring result.

And the grading module is used for grading the disease symptoms of the measuring result according to a preset grading system.

In one embodiment, the first construction module includes a first labeling submodule, a storage submodule, and a first training submodule.

The first labeling submodule is used for labeling a region to be segmented, position information of the region to be segmented, a region where ankylosing spondylitis reaches a fourth-order disease and position information of the region where ankylosing spondylitis reaches the fourth-order disease in all the 2D hip orthotopic X-ray image images, and making a data set in a first preset format.

And the storage sub-module is used for storing the preprocessed 2D hip positive position X-ray image as an image in a second preset format.

And the first training submodule is used for inputting the image only containing the hip joint bone gap area and the position information thereof into a preset detection network for training to obtain the hip joint gap target identification network.

In one embodiment, the first training submodule includes an acquisition unit, a generation unit, a first determination unit, a second determination unit, a calculation unit, a reception unit, and a processing unit.

The acquisition unit is used for acquiring a characteristic diagram of the X-ray image only containing the hip joint bone gap area through a convolution neural network.

And the generating unit is used for generating anchor points on the characteristic diagram, wherein each pixel point is mapped to the original 2D hip orthotopic X-ray image, and nine candidate frames are set by taking each anchor point as the center.

The first determining unit is used for determining whether each candidate box internally contains the target object through a binary network and outputting a probability value containing the target object.

And the second determining unit is used for calculating the deviation of the target object determined by the two classification networks through frame regression network regression branches to obtain the translation amount and the transformation scale size required by each candidate frame.

A calculation unit for calculating a cross entropy loss function L of the two-class network_clsSmoothing L1 loss function L of bounding box regression network_regAccording to said cross entropy loss function L_clsAnd a smoothing L1 loss function L of the bounding box regression network_regObtaining a loss function L_F(ii) a Using a preset optimizer to the loss function L_FMinimization; minimizing the loss function L by the preset optimizer_FPerforming back propagation to realize the two-class network and the frameOptimization of the regression network until the loss function L_FAnd converging to obtain the probability value of the target object contained in the candidate box, and the translation amount and the transformation scale size required by the candidate box.

And the receiving unit is used for receiving the probability value of the target object contained in the candidate frame, the translation amount and the transformation scale size required by the candidate frame through an extraction layer of a preset detection network to obtain the candidate frame after translation and scale transformation.

And the processing unit is used for eliminating the overlapped candidate frames according to a non-maximum value inhibition strategy, reserving the candidate frame with the highest probability value and obtaining the hip joint space target identification network.

In one embodiment, the calculation unit comprises a calculation subunit.

Wherein the calculating subunit is used for calculating according to a formula

Obtaining a loss function L_F；

In one embodiment, the first preset percentage of the target sample data set is a first training set, and the second preset percentage of the target sample data set is a first test set; the second construction module comprises a second labeling submodule, a leading-in submodule and a second training submodule.

And the second labeling submodule is used for labeling a real mask of the hip joint gap area in all the 2D hip orthostatic X-ray images and storing the real mask as a mask image.

And the introduction submodule is used for introducing a global attention upsampling module GAU into the preset segmentation network.

And the second training submodule is used for inputting the images of the first training set and the mask images into the segmentation network of the global attention up-sampling module for training, and updating the weight parameters of each layer in the segmentation network by using a preset algorithm to obtain the hip joint gap segmentation network.

In one embodiment, the lead-in submodule comprises a joining unit, an output unit and a splicing unit.

The adding unit is used for adding a plurality of stages of GAU modules in the jump connection of the preset segmentation network, and each stage of GAU module receives the output of the upper stage of GAU module and the low-level feature map of the resolution corresponding to the coding stage as input.

And the output unit is used for performing global pooling on the high-level feature map output by the upper-level GAU module by each level of GAU module, performing convolution on the low-level feature map with the corresponding resolution in the encoding stage, and fusing the two feature maps after the convolution and outputting the feature maps.

And the splicing unit is used for splicing the output result of each stage of GAU module with the input low-level feature diagram and the up-sampling result of the feature diagram output by the last stage of GAU module, and performing convolution operation and up-sampling step by step on the spliced feature diagram.

In one embodiment, the segmentation module includes a third training submodule, a first testing submodule, a partitioning submodule, a fourth training submodule, and a second testing submodule.

And the third training submodule is used for inputting the 2D hip positive X-ray image and the data set in the first preset format and carrying out model training through the hip joint gap target identification network to obtain a target model.

The first testing submodule is used for testing the images of the first testing set in the target model to obtain the interested region to be segmented, the position information of the region to be segmented, the region where the ankylosing spondylitis reaches the fourth-level disease and the position information of the region where the ankylosing spondylitis reaches the fourth-level disease.

The dividing submodule is used for taking the image of the interested region to be divided as a division data set; and taking a third preset percentage of the segmented data set as a second training set, and taking a fourth preset percentage of the segmented data set as a second test set.

And the fourth training submodule is used for adjusting network parameters of the hip joint gap segmentation network based on the global attention by taking the second training set as input, and performing iterative training to obtain a hip joint gap segmentation model.

And the second testing submodule is used for inputting the hip joint gap image to be segmented in the second testing set into the segmentation model of the hip joint gap for testing to obtain a hip joint gap segmentation result of the second testing set.

In one embodiment, the extraction module includes an operation sub-module, a thresholding sub-module, and an extraction sub-module.

And the operation submodule is used for performing opening operation and closing operation on the segmented hip joint gap image.

And the threshold processing submodule is used for carrying out threshold processing on the hip joint gap image after the opening operation and the closing operation.

And the extraction submodule is used for extracting the contour of the hip joint clearance image subjected to threshold processing by adopting an edge operator to obtain a hip joint clearance contour curve.

In one embodiment, the operation submodule comprises a first operation unit and a second operation unit, and the extraction submodule comprises an extraction unit.

Wherein the first operation unit is used for calculating the expression

wherein the erosion operation is according to an expression

Performing operation;

said dilation operation being in accordance with an expression

And (6) performing operation.

A second operation unit for calculating a second operation value according to the expression

And performing a closing operation, wherein the closing operation is to perform expansion operation and then perform corrosion operation.

An extraction unit for extracting the content according to a formula

And (5) extracting the contour.

In one embodiment, the measurement module includes a discretization processing sub-module and a measurement sub-module.

The discretization processing submodule is used for discretizing the first hip joint gap contour curve, calculating the minimum distance from each discrete point to the second hip joint gap contour curve, and selecting a preset number of point pairs which have larger distances and meet the condition that the distances from adjacent points on the first hip joint gap contour curve to the second hip joint gap contour curve are small as approximate solutions.

And the measurement submodule is used for establishing a corresponding optimization model according to the characteristics of curves at each point pair, carrying out local optimization, and selecting the maximum distance value in the optimization result as the one-way Hausdorff distance between the first hip joint clearance contour curve and the second hip joint clearance contour curve to obtain the measurement result.

Based on the above-described automatic classification method for ankylosing spondylitis based on multitask deep learning in the embodiment corresponding to fig. 1, an embodiment of the present disclosure further provides a computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be a Read Only Memory (ROM), a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. The storage medium stores computer instructions for executing the automatic grading method based on multitask deep learning ankylosing spondylitis described in the embodiment corresponding to fig. 1, and details are not repeated here.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. An automatic grading method for deep learning ankylosing spondylitis based on multitasking, characterized in that the method comprises:

2. The method of claim 1, wherein said constructing a hip-gap target recognition network from said target sample dataset comprises:

3. The method of claim 2, wherein the inputting the image only containing the hip bone gap region and the position information thereof into a preset detection network for training to obtain the hip bone gap target identification network comprises:

Using a preset optimizer to the loss function L_FMinimization;

4. The method of claim 3, wherein said cross-entropy loss function L is based on said cross-entropy loss function_clsAnd a smoothing L1 loss function L of the bounding box regression network_regObtaining a loss function L_FThe method comprises the following steps:

according to the formula

Obtaining a loss function L_F；

indicates the ith candidateThe box only contains the true tag with target object 1, t_iThe offset of the ith candidate box representing the bounding box regression network prediction,

5. The method of claim 1, wherein the first predetermined percentage of the target sample data set is a first training set and the second predetermined percentage of the target sample data set is a first test set;

6. The method according to claim 5, wherein said introducing a global attention upsampling module GAU in the pre-defined split network comprises:

7. The method of claim 6, wherein the performing target recognition detection and segmentation on the plurality of 2D hip orthotopic X-ray film images according to the hip joint space target recognition network and the hip joint space segmentation network to obtain a hip joint space segmentation result comprises:

8. The method of claim 7, wherein the performing an edge extraction process on the hip gap segmentation result comprises:

9. The method of claim 8, wherein the opening the segmented hip-joint-gap image comprises:

according to the expression

wherein the erosion operation is according to an expression

Performing operation;

said dilation operation being in accordance with an expression

Performing operation;

according to the expression

Performing a closed-loop operation, the closed-loop operation being advancedPerforming expansion operation, and then performing corrosion operation;

according to the formula

Carrying out contour extraction;

wherein the content of the first and second substances,

in order to etch the operator, the etching process,

for the expand operator, ° for the open operator,. for the close operator, D_fAnd D_sF (x, y) is a gray level image of the segmented hip joint gap to be processed, and (x, y) is a pixel coordinate point of the gray level image of the segmented hip joint gap to be processed; s (x, y) is a structural element; n is 0,1, N-1; the value of M is 0, 1.., M-1; m and N are integers, and M is less than N; f (x, y) is a central pixel value, and three pixel values adjacent to f (x, y) are respectively f (x, y +1), f (x +1, y) and f (x +1, y + 1); t is t₁And t₂Respectively representing the difference values of two pixels in the diagonal direction; gradf (x.y) is a gradient value of a pixel centered at (x, y); h is₁And h₂Respectively, representing the Robert gradient operator, corresponding to h₁And h₂And performing convolution operation on the image.

10. The method of claim 1, wherein performing distance measurement on the edge after the edge extraction process to obtain a measurement result comprises: