WO2023134097A1

WO2023134097A1 - Hysteromyoma target image acquisition method based on residual network structure

Info

Publication number: WO2023134097A1
Application number: PCT/CN2022/093810
Authority: WO
Inventors: 霍彤彤; 邓凯贤; 李丽欣; 叶哲伟; 吴蔚; 王子毅
Original assignee: 南方医科大学顺德医院(佛山市顺德区第一人民医院); 华中科技大学同济医学院附属协和医院
Priority date: 2022-01-11
Filing date: 2022-05-19
Publication date: 2023-07-20
Also published as: CN114399485A; ZA202211365B

Abstract

A hysteromyoma target image acquisition method based on a residual network structure. The method comprises two stages, i.e. model training and model application. The model training comprises: S1, on an original sample ultrasonic image, annotating, in the form of a rectangular box, a lesion target image area in an area containing a hysteromyoma image; S2, detecting a standard annotation image by means of an improved YOLOv3 target detection model; and S3, unifying a model detection result and a standard marking result, so as to obtain a trained improved YOLOv3 target detection model. During application, an ultrasonic image to be subjected to detection is input into the trained improved YOLOv3 target detection model for detection, so as to obtain a hysteromyoma target image area. The method can improve the acquisition precision of a hysteromyoma image area, the detection speed is high, and the adaptability to small-target and multi-target tasks is high.

Description

Target Image Acquisition Method of Uterine Fibroids Based on Residual Network Structure

technical field

The invention relates to the technical field of artificial intelligence, in particular to a method for acquiring a target image of uterine fibroids based on a residual network structure.

Background technique

Uterine fibroids often appear round or oval in ultrasound image data, with clear borders. Ultrasound manifestations of uterine fibroids are diverse. The reasons for the diversity include the following three aspects: first, the size and shape of uterine fibroids are complex; on the other hand, Due to the special imaging method of ultrasound imaging, the target of uterine fibroids in the image is similar to the gray scale of the background, and it is difficult to accurately delineate the boundary of the target area; Moreover, the contrast and hue of the image itself may vary due to differences in image acquisition equipment and environments. The diversity of uterine ultrasound images and targets makes it difficult to extract target features of fibroids, which increases the difficulty of target image acquisition.

In the selection of detection algorithm, it is necessary to select a more "flexible" algorithm to adapt to the task of acquiring the target area of uterine fibroids in the case of multiple targets and small targets. At the same time, the algorithm has a low false detection rate against the background and has strong versatility. It is usually difficult for target detection algorithms to have both of the above characteristics.

Therefore, in view of the deficiencies of the prior art, it is necessary to provide a method for acquiring target images of uterine fibroids based on the residual network structure to overcome the deficiencies of the prior art.

Contents of the invention

Aiming at the problem in the prior art that the accuracy and speed of ultrasonic image detection cannot be taken into account at the same time, the present invention provides a method for acquiring a target image of uterine fibroids based on a residual network structure.

The present invention is realized through the following technical solutions:

A method for acquiring a target image of uterine fibroids based on a residual network structure is provided, and the method includes the following two stages:

Phase 1, model training

S1. On the original sample ultrasound image, mark the area containing the uterine fibroid image in the form of a rectangular frame to obtain a standard marking result, which includes a standard marking image and a standard marking file;

S2. The standard labeled image is detected by the improved YOLOv3 target detection model to obtain a model detection result, the model detection result including the position, size and quantity of the uterine fibroid target image in the image;

S3. Unify the model detection result in step S2 with the standard marking result in step S1 to obtain the trained improved YOLOv3 target detection model;

Phase 2, model application

The ultrasonic image to be detected is input to the trained improved YOLOv3 target detection model for detection, and the result of the target image area of uterine fibroids is obtained.

Preferably, the above-mentioned improved YOLOv3 target detection model is provided with a ResNet residual learning structure, and each convolutional layer is sequentially provided with the ResNet residual learning structure.

Preferably, the above-mentioned improved YOLOv3 target detection model specifically replaces the backbone network in YOLO v3 with Resnet50.

Preferably, the above-mentioned improved YOLOv3 target detection model, the specifically designed anchors size is: [[10,13], [16,30], [33,23], [30,61], [62,45] , [59,119], [116,90], [156,198], and [373,326].

Preferably, in S3, the model detection result in step S2 is unified with the standard marking result in step S1, specifically including:

Calculate the loss function, and make a loss on the predicted center coordinates, using the loss function of formula (1):

Equation (1) calculates the loss value relative to the predicted bounding box position (x, y); where λ is a given constant, indicating the weight of the loss; (x, y) is obtained from the training data get the actual location,

is the position of the predicted bounding box; this function calculates the sum of each bounding box predicted value (j=0,...,B) of each grid cell (i=0,...,S ² );

Defined as follows: If an object exists in grid cell i, the jth bounding box prediction is valid for that prediction,

If no target exists in grid cell i,

For each grid unit YOLO predicts the corresponding bounding box, during training, according to which prediction has the highest real-time IOU and GT, it is confirmed that it is effective for predicting a target;

Make a loss on the width and height of the predicted bounding box, specifically using the loss function of formula (2):

Make a loss on the predicted category, specifically using the loss function of formula (3):

use

is that when there is no target in the grid cell, the classification error will not be penalized;

Make a loss on the predicted confidence, specifically using the loss function of formula (4):

C is the confidence score,

is the intersection of the predicted bounding box and the GT box, when there is an object in a grid cell,

otherwise

Finally, add the four parts of the loss function together to get the total loss function:

According to the total loss function, the improved YOLOv3 target detection model is continuously modified, and when the final total loss function is no longer decreased after the correction, the trained improved YOLOv3 target detection model is obtained.

Preferably, in the above-mentioned method for acquiring target image of uterine fibroids based on the residual network structure, senior doctors mark the region containing the image of uterine fibroids in the form of a rectangular frame to mark the lesion target image area.

Preferably, the above-mentioned method for acquiring a target image of uterine fibroids based on a residual network structure, in S2, before the standard tagged image is detected by the improved YOLOv3 target detection model, data arrangement of the tagged image is also included;

Carry out regional segmentation processing on standard labeled images, and only retain valid images with fibroid lesions;

After masking the effective images, they are divided into training set and test set, and complete the data arrangement of standard labeled images;

The training set and test set are used for training and testing of the improved YOLOv3 target detection model.

Preferably, the above-mentioned method for acquiring a target image of uterine fibroids based on a residual network structure, before detecting the standard tagged image through the improved YOLOv3 target detection model, also includes data enhancement of the tagged image, including image random flip, twist, expand and crop.

Preferably, the above-mentioned method for acquiring a target image of uterine fibroids based on a residual network structure randomly flips, distorts, expands and cuts the image, specifically including:

1) Random scaling, the image size is normalized to be between -0.5 and 0.5;

2) With a probability of 0.5, the hue of the image is randomly increased by -18 to 18, and the saturation, brightness and contrast are randomly increased by 0.5 to 1.5; and the image is randomly flipped left and right, and randomly distorted;

3) Then randomly expand the image, the execution probability is 0.5, the maximum expansion ratio is 4, and the filling color value for expansion is R: 123.675, G: 116.28, B: 103.53;

4) Randomly crop the image, the aspect ratio of the cropped area is 0.5-2, the effective IOU cropping thresholds are 0, 0.1, 0.3, 0.5, 0.7, 0.9, and the ratio of the cropped area to the original image is 0.3-1.

Preferably, the above-mentioned a kind of uterine fibroid target image acquisition method based on the residual network structure, the ultrasound image adopts the format and is jpg format, and the acquisition instrument includes Toshiba 300, 400, 500, Siemens, GE S8S9 color Doppler ultrasound Instrument, wherein the data includes images of abdominal ultrasound and vaginal ultrasound, wherein the frequency of the abdominal ultrasound probe is set to 2-7MHz, and the frequency of the vaginal ultrasound probe is set to 5-7MHz.

The method for acquiring uterine fibroid target images based on the residual network structure of the present invention obtains the trained improved YOLOv3 target detection model through training, which improves the accuracy of uterine fibroid target image acquisition, and at the same time, the method is simple, generalizable and applicable powerful.

Description of drawings

Fig. 1 is a schematic diagram of the ResNet residual learning structure described in the embodiment of the present invention.

Fig. 2 is a schematic diagram of the improved YOLOv3 target detection model described in the embodiment of the present invention.

Fig. 3 is an image of the detection result of uterine fibroids described in the embodiment of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the embodiments and the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

A uterine fibroid target image acquisition method based on residual network structure, the method includes the following two stages:

Phase 1, model training

S1. On the ultrasound image of the original sample, mark the lesion target image area in the form of a rectangular frame for the area containing the uterine fibroid image to obtain a standard marking result, which includes a standard marking image and a standard marking file;

Phase 2, model application

The method process of the above two stages will be described in detail below.

First of all, the images of abdominal ultrasound and vaginal ultrasound can be collected as sample images through Toshiba 300, 400, 500, Siemens, GE S8 S9 color Doppler ultrasound instruments. The frequency of abdominal ultrasound examination probe is set to 2-7MHz. The frequency of the probe is set to 5 ~ 7MHz. Ultrasound images are converted to jpg format.

Further, perform region segmentation processing on the standard labeled images at different resolutions, retain only effective images of the fibroid lesion area, and segment them into images of size (W, H);

Further, for the input uterine ultrasound image with a size of (W, H), mark the circumscribed rectangle of uterine fibroids as the Ground Truth for uterine fibroids detection, mask the effective images and divide them into training set and test set, and complete the Data curation for annotated images.

Then, before the standard labeled image is detected by the improved YOLOv3 target detection model, it also includes data enhancement of the labeled image, including random flipping, twisting, expanding and cropping of the image, including:

1) Random scaling, the image size is normalized to be between -0.5 and 0.5;

As shown in FIG. 1 , the improved YOLOv3 target detection model of the present invention is provided with a ResNet residual learning structure, and each convolutional layer is sequentially provided with the ResNet residual learning structure. Specifically, replace the backbone network in YOLO v3 with Resnet50.

The improved YOLOv3 target detection model, through a large number of analysis of the characteristics of ultrasound image data of uterine fibroids, combined with the feature extraction ability of the unique residual structure of the backbone network Resnet50, shows the advantages in the direction of target detection and positioning accuracy, and the YOLO v3 framework in the detection speed And the advantages of versatility, replace the original backbone network in YOLO v3 with Resnet50 to realize the detection task of uterine fibroids. The specific network structure is shown in Figure 1.

The present invention applies a 50-layer ResNet50 network, and uses a unique residual module to learn more complex feature representations from ultrasound images of uterine fibroids. Compared with previous models, this model has better detection accuracy.

After that, as shown in Figure 2, the improved YOLOv3 target detection model, according to the size characteristics of uterine fibroids in the ultrasound image, designs the anchors size suitable for the sub-task, and the specifically designed anchors size is: [[10, 13], [16,30], [33,23], [30,61], [62,45], [59,119], [116,90], [156,198], [373,326]].

Next, unify the model detection results with the senior doctor's marking results GT, calculate the loss function, and perform a loss on the predicted center coordinates, using the loss function of formula (1):

This formula calculates the loss value relative to the predicted bounding box position (x, y). Where λ is a given constant, indicating the weight of the loss. (x,y) is the actual position obtained from the training data,

is the location of the predicted bounding box. This function computes the sum of each bounding box prediction (j=0,...,B) for each grid cell (i=0,...,S2).

If no target exists in grid cell i,

For each grid cell YOLO predicts pairs of bounding boxes. At training time, we only want one bounding box predictor per object. We confirm that it is effective for predicting an object based on which prediction has the highest real-time IOU and GT.

Further, the loss function of formula (2) is specifically used for loss of the width and height of the predicted bounding box:

This is the loss related to the width and height of the predicted bounding box. Because we found that the deviation of the large box is smaller than that of the small box. So instead of predicting the width and height directly, we adopt the method of predicting the square root of the width and height of the bounding box.

Further, make a loss on the predicted category, specifically using the loss function of formula (3):

use

is that we do not penalize the classification error when no object exists in the grid cell.

Further, the loss function of formula (4) is specifically used for loss of the predicted confidence:

The loss function is associated with a confidence score for each bounding box prediction. C is the confidence score,

is the intersection of the predicted bounding box and the GT box. When there is a target in a grid cell,

otherwise

The lambda parameter that appears here and in the first section is used for different weighted parts of the loss function. This is very critical to improve the stability of the model. The highest penalty is for coordinate predictions (λ _coord =5), with the lowest confidence prediction penalty (λ _noobj =0.5) when no object is detected.

Further, finally, the four parts of the loss function are added together to obtain the total loss function:

After obtaining the trained improved YOLOv3 target detection model, in the subsequent use, you only need to input the ultrasound image to be detected into the trained improved YOLOv3 target detection model for detection, and obtain the result of the target image area of uterine fibroids. Fig. 3 is a schematic diagram of the result of the target image area of uterine fibroids obtained by processing part of the ultrasonic images by the method of the present invention.

The present invention designs an improved YOLOv3 target detection model, and obtains the trained improved YOLOv3 target detection model through training. Through this model, the target area of uterine fibroids in the ultrasound image can be accurately acquired, and the method is simple, generalized and Features of strong applicability.

Claims

A uterine fibroid target image acquisition method based on a residual network structure, characterized in that the method includes the following two stages:

Phase 1, model training

S1. On the ultrasound image of the original sample, mark the lesion target image area in the form of a rectangular frame for the area containing the uterine fibroid image to obtain a standard marking result, which includes a standard marking image and a standard marking file;

S2. The standard labeled image is detected by the improved YOLOv3 target detection model to obtain a model detection result, the model detection result including the position, size and quantity of the uterine fibroid target image in the image;

S3. Unify the model detection result in step S2 with the standard marking result in step S1 to obtain the trained improved YOLOv3 target detection model;

Phase 2, model application

The ultrasonic image to be detected is input to the trained improved YOLOv3 target detection model for detection, and the result of the target image area of uterine fibroids is obtained.
A method for acquiring uterine fibroid target images based on a residual network structure according to claim 1, wherein the improved YOLOv3 target detection model is provided with a ResNet residual learning structure, and in each convolutional layer The ResNet residual learning structure is arranged in turn.
A method for acquiring uterine fibroid target images based on a residual network structure according to claim 2, wherein the improved YOLOv3 target detection model specifically replaces the backbone network in YOLO v3 with Resnet50.
A kind of uterine fibroid target image acquisition method based on residual network structure according to claim 3, it is characterized in that, described improved YOLOv3 target detection model, the anchors size of specific design is: [[10,13 ], [16,30], [33,23], [30,61], [62,45], [59,119], [116,90], [156,198], and [373,326].
A method for acquiring a target image of uterine fibroids based on a residual network structure according to any one of claims 1 to 4, wherein in S3, the model detection result of step S2 is compared with the standard marking result in step S1 unification, including:

Calculate the loss function, and make a loss on the predicted center coordinates, using the loss function of formula (1):

Equation (1) calculates the loss value relative to the predicted bounding box position (x, y); where λ is a given constant, indicating the weight of the loss; (x, y) is obtained from the training data get the actual location,
is the position of the predicted bounding box; this function calculates the sum of each bounding box predicted value (j=0,...,B) of each grid cell (i=0,...,S 2 );

Defined as follows: If an object exists in grid cell i, the jth bounding box prediction is valid for that prediction,
If no target exists in grid cell i,

For each grid unit YOLO predicts the corresponding bounding box, during training, according to which prediction has the highest real-time IOU and GT, it is confirmed that it is effective for predicting a target;

Make a loss on the width and height of the predicted bounding box, specifically using the loss function of formula (2):

Make a loss on the predicted category, specifically using the loss function of formula (3):

use
is that when there is no target in the grid cell, the classification error will not be penalized;

Make a loss on the predicted confidence, specifically using the loss function of formula (4):

C is the confidence score,
is the intersection of the predicted bounding box and the GT box, when there is an object in a grid cell,
otherwise

Finally, add the four parts of the loss function together to get the total loss function:

According to the total loss function, the improved YOLOv3 target detection model is continuously modified, and when the final total loss function is no longer decreased after the correction, the trained improved YOLOv3 target detection model is obtained.
A method for acquiring a target image of uterine fibroids based on a residual network structure according to claim 5, characterized in that, the region containing the image of uterine fibroids is processed by a senior doctor in the form of a rectangular frame to determine the lesion target image area label.
A method for acquiring a target image of uterine fibroids based on a residual network structure according to claim 5, wherein in S2, before the standard labeled image is detected by the improved YOLOv3 target detection model, it also includes labeling Image data collation;

Carry out regional segmentation processing on standard labeled images, and only retain valid images with fibroid lesions;

After masking the effective images, they are divided into training set and test set, and complete the data arrangement of standard labeled images;

The training set and test set are used for training and testing of the improved YOLOv3 target detection model.
A method for acquiring a target image of uterine fibroids based on a residual network structure according to claim 5, wherein, before the standard tagged image is detected by the improved YOLOv3 target detection model, the tagged image is also included Data augmentation, including random flipping, warping, expanding, and cropping of images.
A method for acquiring a target image of uterine fibroids based on a residual network structure according to claim 8, wherein the random flipping, twisting, expanding and cutting of the image specifically includes:

1) Random scaling, the image size is normalized to be between -0.5 and 0.5;

2) With a probability of 0.5, the hue of the image is randomly increased by -18 to 18, and the saturation, brightness and contrast are randomly increased by 0.5 to 1.5; and the image is randomly flipped left and right, and randomly distorted;

3) Then randomly expand the image, the execution probability is 0.5, the maximum expansion ratio is 4, and the filling color value for expansion is R: 123.675, G: 116.28, B: 103.53;

4) Randomly crop the image, the aspect ratio of the cropped area is 0.5-2, the effective IOU cropping thresholds are 0, 0.1, 0.3, 0.5, 0.7, 0.9, and the ratio of the cropped area to the original image is 0.3-1.
A kind of uterine fibroid target image acquisition method based on residual network structure according to claim 5, it is characterized in that, ultrasonic image adopts format and is jpg format, and acquisition instrument comprises Toshiba 300,400,500, Siemens, GE S8 S9 color Doppler ultrasonography, the data includes images of abdominal ultrasound and vaginal ultrasound, wherein the frequency of the abdominal ultrasound probe is set to 2-7MHz, and the frequency of the vaginal ultrasound probe is set to 5-7MHz.