CN113763309A

CN113763309A - Liver blood vessel ultrasonic image target identification and tracking method based on improved U-net network and LSTM network

Info

Publication number: CN113763309A
Application number: CN202011155220.3A
Authority: CN
Inventors: 赵悦; 姜宇; 金晶; 李丹丹; 沈毅
Original assignee: Harbin Institute of Technology
Current assignee: Harbin Institute of Technology
Priority date: 2020-10-26
Filing date: 2020-10-26
Publication date: 2021-12-07

Abstract

The invention discloses a liver blood vessel ultrasonic image target identification and tracking method based on an improved U-net network and an LSTM network. Step 1: preprocessing an ultrasonic image sequence; step 2: training an ROI extraction model, and segmenting a region from the ultrasonic image in the step 1; and step 3: realizing accurate segmentation of the target in the area in the step 2 based on the improved U-net network; and 4, step 4: classifying the ultrasonic image sequence in the step 1 by using a CNN-LSTM network; and 5: and (4) based on the segmentation result in the step (3) and the classification result in the step (4), utilizing an LSTM network to realize accurate prediction of the target position in the ultrasonic image sequence. The invention aims to solve the problem that the existing method can not simultaneously carry out target identification and real-time tracking on a dynamic ultrasonic image sequence.

Description

Liver blood vessel ultrasonic image target identification and tracking method based on improved U-net network and LSTM network

Technical Field

The invention relates to the technical field of image processing and the field of medical auxiliary diagnosis, in particular to a liver blood vessel ultrasonic image target identification and tracking method based on an improved U-net network and an LSTM network.

Background

Because medical ultrasonic imaging has the characteristic of real-time property, the technology is widely applied to lesion identification and tracking in ablation and minimally invasive treatment of liver tumors. However, respiratory motion can cause changes in the location of lesions and obstructions, thereby increasing the difficulty of accurate lesion and lesion localization.

The target identification and tracking based on the ultrasonic image mainly comprises the following steps: 1) identifying a target in an ultrasonic image; 2) and tracking the target in the ultrasonic image. The object recognition of the ultrasonic image is to accurately draw the contour of valuable objects such as blood vessels, tumors and the like in the image, and because the automatic object recognition method uses a computer to automatically extract the contour of the object, the segmentation speed is high, and the traditional semi-automatic and manual methods are gradually replaced. However, the ultrasound image has the characteristics of large speckle noise, unclear target, low brightness and the like, which can affect the accuracy of target identification. In addition, the position of a focus and a blood vessel in the living body treatment can be changed in consideration of the influence of the natural respiration of the human body, so that the positioning of an ultrasonic image target can be influenced, and the rapidity and the accuracy of target tracking are influenced. Therefore, it is very important to realize accurate and real-time ultrasound image target identification and tracking.

In recent years, with the development of deep learning theory, deep neural networks have been widely used in the technical fields of image segmentation, target detection and the like with excellent performance.

The full convolutional neural network (FCN) is extremely effective in medical image target recognition. The U-net is an FCN-based deep neural network with a symmetric structure, and accuracy of target identification in a medical image is improved through combination of low-level features and high-level features.

The Recurrent Neural Network (RNN) has been successfully applied to the fields of speech recognition, image annotation, etc., but it is difficult to deal with the long-term dependence problem. The long-short term memory network (LSTM) is based on an RNN framework, the problem that the gradient disappears in the process of back propagation during training is solved by adding a gate unit, and the purpose of predicting the output at the next moment according to the output in a long time before can be achieved. LSTM networks have found wide application due to their accuracy in spatio-temporal sequence prediction.

Disclosure of Invention

The invention provides a liver blood vessel ultrasonic image target identification and tracking method based on an improved U-net network and an LSTM network, and aims to solve the problem that the existing method cannot simultaneously perform target identification and real-time tracking on a dynamic ultrasonic image sequence.

The invention is realized by the following technical scheme:

a liver blood vessel ultrasonic image target identification and tracking method based on an improved U-net network and an LSTM network comprises the following steps:

step 1: preprocessing an ultrasonic image sequence;

step 2: training an ROI extraction model, and segmenting a region from the ultrasonic image in the step 1;

and step 3: realizing accurate segmentation of the target in the area in the step 2 based on the improved U-net network;

and 4, step 4: classifying the ultrasonic image sequence in the step 1 by using a CNN-LSTM network;

and 5: and (4) based on the segmentation result in the step (3) and the classification result in the step (4), utilizing an LSTM network to realize accurate prediction of the target position in the ultrasonic image sequence.

Further, the preprocessing method in step 1 is specifically to process the liver blood vessel ultrasonic image data by using a histogram equalization method, and map the gray value of the original image into the range of [0, 255 ].

Further, the specific steps of step 2 are as follows:

step 2.1: marking the position (rectangular frame) of a liver blood vessel in each image by taking the preprocessed ultrasonic image as a target detection network data set, and storing the position as an xml format file for training as a label;

step 2.2: improving an original target detection network, selecting Focal loss as a loss function of a classification network, adding non-maximum suppression (NMS) to screen a prediction frame, and constructing a retinet network;

step 2.3: training the retinet network on the training set to obtain an accurate target detection model, and testing the test set by using the trained target detection model to obtain an automatically extracted target area ultrasonic image.

Further, the specific steps of step 3 are:

step 3.1: taking the target area ultrasonic image obtained in the step 2 as a segmentation network data set, outlining a liver blood vessel edge appearing in each image, processing the edge into a binary image serving as a label of a segmentation network, and then adjusting the image and the label image to be in the same size;

step 3.2: improving the original U-net network, adding edge completion, introducing a Batch Normalization algorithm, and training the improved U-net network on a training set to obtain an accurate segmentation model;

step 3.3: and testing the test set by using the trained segmentation model to obtain the segmented ultrasonic image data.

Further, the specific steps of step 4 are as follows:

step 4.1: taking the preprocessed ultrasonic image as a data set of a picture classification network, taking a sequence to which pictures belong as a label, and carrying out classification model training on a training set based on the established CNN network;

step 4.2: and selecting adjacent M frames of pictures to generate a new image sequence as a data set of the sequence classification network by taking the random frames as intervals for the preprocessed ultrasonic image sequence. Taking the sequence name as a label, and carrying out one-hot coding on the label;

step 4.3: constructing an LSTM prediction network, extracting the characteristics of an image sequence by applying the trained CNN classification model, and training and adjusting the network by taking the extracted characteristics as the input of the LSTM network so as to obtain a sequence classification model;

step 4.4: and testing by adopting the test set data based on the obtained sequence classification model so as to verify the accuracy of the model.

Further, the specific steps of step 5 are:

step 5.1: based on the segmentation result and the classification result, respectively training the LSTM network for different ultrasonic image sequences;

step 5.2: taking the segmented ultrasonic image sequence obtained in the step 3 as a data set of an LSTM prediction network, considering the correlation among the ultrasonic image sequences, selecting images of the first frames as input and the next frame as output when predicting the position of the blood vessel in the next frame of ultrasonic image by adopting the LSTM network, and generating a training set and a test set;

step 5.3: and training and adjusting the LSTM network based on the processed data set to obtain a liver blood vessel target position prediction model, and finally realizing accurate tracking of the liver blood vessel target position in the ultrasonic image.

The invention has the beneficial effects that:

1) the method uses the trained target detection model to automatically extract the region of interest in the ultrasonic image, is beneficial to the identification and tracking of subsequent targets, and can improve the precision and speed of the targets to a certain extent;

2) the invention uses the improved U-net network to carry out target segmentation on the liver blood vessel ultrasonic image, thereby realizing the improvement of the segmentation precision;

3) the invention uses the CNN-LSTM network to realize the accurate classification of the ultrasonic image sequence, which is beneficial to improving the accuracy of target position prediction;

4) the invention uses the LSTM network to predict the position of the liver blood vessel in the ultrasonic image, which is beneficial to improving the accuracy and speed of segmentation;

5) the invention relates to the technical field of image processing and the field of medical auxiliary diagnosis, and has positive promoting effect on intelligent auxiliary diagnosis.

Drawings

FIG. 1 is a flow chart of the method of the present invention.

Fig. 2 is a schematic diagram of a network structure of a target detection network for automatically extracting a target area according to the present invention.

Fig. 3 is a schematic diagram of the invention for automatically extracting a target region of a liver blood vessel by using a RetinaNet network.

Fig. 4 is a network structure diagram of the improved U-net network for automatically segmenting liver blood vessels according to the present invention.

FIG. 5 is a schematic diagram of an original image of an ultrasound image of a hepatic blood vessel according to the present invention.

Fig. 6 is an example of the detection results of 3 objects in the same ultrasound image after the suppression of non-maxima is added.

FIG. 7 is the segmentation result of the improved U-net network of the present invention for 3 objects in the same ultrasound image.

Detailed Description

The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example 1

step 1: preprocessing an ultrasonic image sequence;

the liver blood vessel ultrasonic images adopted in the experiment are all clinical data collected by ultrasonic machines such as Ultrasonix MDP, Siemens Antares and the like, and the total number of the clinical data is 81738. For different ultrasonic image sequences in the data set, respectively selecting 1000 pictures as the data set required by the experiment, renaming according to the format of the sequence name _ image _ picture sequence number bmp (such as CIL-01_ image _0.bmp), and adjusting the same size to 224 x 224; and for the ultrasonic image data set, processing the ultrasonic image data of the liver blood vessels by adopting a histogram equalization method, and mapping the gray value of the original image into a range of [0, 255 ].

step 2.1: taking an ultrasonic image subjected to image preprocessing as a target detection network data set, randomly selecting 70% as a training set and 30% as a test set, marking the area (rectangular frame) where the hepatic vessels appear in each image by using labelImg software, and storing the area as an xml format file as a picture label;

step 2.2: improving an original target detection network, selecting a focusing loss function (Focal loss) as a loss function of a classification network, adding non-maximum suppression (NMS) for screening a prediction frame, and constructing a retinet network, as shown in FIG. 2;

step 2.3: setting various network parameters, such as: the basic network architecture is "resnet 50", the batch size is 8, the maximum iteration number is 10000, the model storage mode is 1 time per 10 times of training, and the like, the retinet network is trained on the training set to obtain an accurate target detection model, the trained target detection model is applied to test the data of the test set to obtain the automatically extracted target area ultrasonic image, as shown in fig. 5. At the time of testing, each picture took about 0.66s, and the average detection accuracy (mAP) of the model was about 99.78%.

step 3.1: taking the target area ultrasonic image obtained in the step 2 as a segmentation network data set, outlining a liver blood vessel edge appearing in each image, processing the edge into a binary image serving as a label of a segmentation network, and then adjusting the image and the label image to be 224 multiplied by 224 with the same size;

data enhancement is carried out by rotating the data set picture by 90 degrees, 180 degrees and 270 degrees respectively and translating the picture by 5 pixels from left to right and 10 pixels from top to bottom respectively. Similarly, the label image is processed accordingly. For the expanded data set, randomly selecting 70% as a training set and 30% as a test set;

step 3.2: improving the original U-net network, adding edge completion, introducing a Batch Normalization algorithm, and building an improved U-net network, as shown in FIG. 3, training and adjusting the improved U-net network on a training set to obtain an accurate improved U-net liver vessel segmentation model;

step 3.3: testing the test set data by using the trained segmentation model to obtain segmented ultrasonic image data; and comparing the test set label with the model segmentation result, and selecting a region evaluation criterion and a boundary evaluation criterion as evaluation indexes. Wherein:

the regional assessment criteria include True Positive Rate (TPR) and similarity rate (SIR), defined as follows:

wherein A is_hFor manually marking target areas, A_aThe larger the TPR and SIR, the better the target area divided by the algorithm.

The boundary evaluation criteria include a Hausdorff Distance (HD) and an average distance (MD), defined as follows:

wherein Q ═ { Q ═ Q₁,q₂,…,q_nIs the manually calibrated target boundary, P ═ P₁,p₂,…,p_mD (p) is the boundary calibrated by the algorithm_jQ) is any point p on the boundary_jTwo-dimensional Euclidean distance to Q, i.e.

Therefore, HD,The smaller the MD, the better.

Meanwhile, liver vessel segmentation is performed on the test set by using a typical FCN (fuzzy C-means network) as a control, and specific comparison results are shown in Table 1.

TABLE 1 evaluation index when testing with different algorithms

	TPR	SIR	HD	MD
					FCN (comparison algorithm)	0.47	0.42	60.49	11.43
U-net	0.85	0.67	69.70	3.94

As can be seen from the data in the table, the segmentation result of the segmentation algorithm provided by the invention is superior to that of a comparison algorithm, and the accurate segmentation of the liver blood vessel can be realized.

step 4.1: taking the preprocessed ultrasonic image as a data set of a picture classification network, taking a sequence of pictures as a label, randomly selecting 70% of the sequence as a training set and 30% of the sequence as a test set, training a classification model of the training set based on the established CNN network, and testing the test set;

step 4.2: for the preprocessed ultrasonic image sequence, selecting adjacent M frames of pictures to generate a new image sequence at intervals of random frames, and using the new image sequence as a data set of the sequence classification network, for example, selecting adjacent 50 frames of pictures to generate a new image sequence at intervals of 20 frames from a first frame of image, and using the new image sequence as a data set of the sequence classification network. Taking the sequence name as a label, carrying out one-hot coding on the label, and randomly selecting 70% of the sequence name as a training set and 30% of the sequence name as a test set;

And 5: based on the segmentation result in the step 3 and the classification result in the step 4, the target position in the ultrasonic image sequence is accurately predicted by using an LSTM network;

step 5.1: considering that the motion of the target in different image sequences is different, respectively training the LSTM network for different ultrasonic image sequences based on the segmentation result and the classification result;

step 5.2: taking the segmented ultrasonic image sequence obtained in the step 3 as a data set of an LSTM prediction network, respectively training the LSTM prediction network for each sequence, and taking the correlation among the ultrasonic image sequences into consideration, when predicting the position of a blood vessel in the next frame of ultrasonic image by adopting the LSTM network, selecting images of the previous frames as input, such as the previous 10 frames, and the next frame of image as output, and generating a training set and a testing set;

step 5.3: based on the processed data set, setting each network parameter as: the LSTM network is trained and adjusted to obtain a liver blood vessel position prediction model, wherein the number of layers of the LSTM network is 1, the number of hidden layer units is 50, the size of a data batch is 50, the training period is 10000 and the like;

step 5.4: based on the obtained training model, testing is carried out by adopting test set data, and the test set labels are compared with the model prediction result, so that the blood vessel target tracking precision of the test position prediction model can reach the millimeter level, and the accurate prediction of the position of the liver blood vessel in the ultrasonic image can be realized.

Claims

1. A liver blood vessel ultrasonic image target identification and tracking method based on an improved U-net network and an LSTM network is characterized in that the liver blood vessel ultrasonic image target identification and tracking method comprises the following steps:

step 1: preprocessing an ultrasonic image sequence;

2. The method for identifying and tracking the target of the ultrasound image of the liver vessel based on the improved U-net network and the LSTM network as claimed in claim 1, wherein the preprocessing method in step 1 is to process the ultrasound image data of the liver vessel by using a histogram equalization method, and map the gray value of the original image into the range of [0, 255 ].

3. The method for recognizing and tracking the target of the hepatic vascular ultrasound image based on the improved U-net network and the LSTM network as claimed in claim 1, wherein the specific steps of step 2 are as follows:

step 2.1: marking the position of a liver blood vessel in each image by taking the preprocessed ultrasonic image as a target detection network data set, and storing the position as an xml format file for label training;

step 2.2: improving the original target detection network, selecting a focusing loss function as a loss function of a classification network, increasing non-maximum value inhibition to screen a prediction frame, and constructing a retinet network;

4. The method for recognizing and tracking the target of the hepatic vascular ultrasound image based on the improved U-net network and the LSTM network as claimed in claim 1, wherein the specific steps of step 3 are as follows:

step 3.2: improving the original U-net network, adding edge completion, introducing a BatchNormalization algorithm, and training the improved U-net network on a training set to obtain an accurate segmentation model;

5. The method for recognizing and tracking the target of the hepatic vascular ultrasound image based on the improved U-net network and the LSTM network as claimed in claim 1, wherein the specific steps of step 4 are as follows:

step 4.2: selecting adjacent M frames of pictures to generate a new image sequence as a data set of a sequence classification network by taking random frames as intervals for the preprocessed ultrasonic image sequence, taking the sequence name as a label, and carrying out one-hot coding on the label;

step 4.3: and constructing an LSTM prediction network, extracting the characteristics of the image sequence by applying the trained CNN classification model, and training and adjusting the network by taking the extracted characteristics as the input of the LSTM network so as to obtain a sequence classification model.

6. The method for recognizing and tracking the target of the hepatic vascular ultrasound image based on the improved U-net network and the LSTM network as claimed in claim 1, wherein the specific steps of step 5 are as follows:

step 5.2: taking the segmented ultrasonic image sequence obtained in the step 3 as a data set of an LSTM prediction network, and selecting images of the previous frames as input and the next frame as output when the LSTM network is adopted to predict the position of the blood vessel in the next frame of ultrasonic image to generate a training set and a test set;