CN115578400A

CN115578400A - Image processing method, and training method and device of image segmentation network

Info

Publication number: CN115578400A
Application number: CN202110672460.9A
Authority: CN
Inventors: 任厚桦; 唐明轩; 柳岸; 陈诚; 袁磊; 黄承基
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Chengdu ICT Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Chengdu ICT Co Ltd
Priority date: 2021-06-17
Filing date: 2021-06-17
Publication date: 2023-01-06

Abstract

The invention provides an image processing method, and an image segmentation network training method and device. The method comprises the following steps: aiming at an MRI image, carrying out inclusion convolution by using an inclusion convolution module of an image segmentation network to obtain a first image characteristic; performing hole convolution on the first image characteristic by using a hole convolution module of the image segmentation network to obtain a second image characteristic; and segmenting a tumor lesion region from the MRI image based on the second image characteristic, wherein the tumor lesion region is a region where pixels imaged by tumor lesion tissues are located. Therefore, more contour information and detail information of the MRI image can be obtained, and the image segmentation precision is improved; the depth of the network can be increased, the adaptability of the network to the scale and the nonlinear capability of the network are enhanced, and the segmentation precision of the MRI image is effectively improved.

Description

Image processing method, and training method and device of image segmentation network

Technical Field

The invention relates to the technical field of intelligent medical treatment, in particular to an image processing method, an image segmentation network training device, electronic equipment and a storage medium.

Background

Brain tumor refers to various tumors located in the brain, the growth rate and location of which determine its effect on the nervous system. According to the investigation, the number of patients with brain tumor increases year by year, and the situation is very severe. The development of the intelligent medical industry is promoted by the progress of computer science and Imaging technology, and an image and deep learning technology based on Magnetic Resonance Imaging (MRI) is an effective means for realizing automatic segmentation of brain tumors; the automatic and accurate segmentation of tumor regions from MRI images is one of the key steps in establishing an automatic brain tumor segmentation computer-aided diagnosis system in clinical practice.

In the related art, the method for segmenting the brain tumor MRI image includes: a semi-automatic image segmentation method and a full-automatic image segmentation method.

For the semi-automatic image segmentation method, an image doctor is required to participate a little in the segmentation process of an MRI image, but fine labeling of pixel level is not required, the doctor is usually required to manually select an initial seed point or mark a Region of Interest (ROI), and then an image threshold segmentation algorithm based on a gray histogram is combined with a Gaussian equation to calculate the gray range of a tumor in the ROI, so that the tumor Region is obtained through threshold segmentation; or taking the boundary of the soil as the initial contour of the target region, and adopting a Snake algorithm to jointly control contour change by an external energy item and an internal energy item so as to continuously approach the real tumor contour. However, this method requires manual initial seed point or ROI selection, which is tedious for the imaging physician and depends heavily on clinical experience, and is time-consuming and highly susceptible to manual error when faced with automatic diagnosis of large-scale MRI images.

For the full-automatic image segmentation method, manual participation is not needed, and the image features can be directly converted and extracted through the deeply stacked nonlinear module, so that high-dimensional feature expression is output, and tumor region segmentation is automatically realized. However, the full-automatic method is trained by fully supervised learning, large-scale labeled training data is needed to effectively avoid overfitting of the model, and the robustness of the neural network model is ensured. However, it is difficult to obtain a large number of labeled medical image images, the data labeling process is also difficult, and the labeling of the medical image images is often completed by doctors with industry experience over ten years. The accuracy of the neural network module is influenced by a small number of labels, and huge expenses of manpower and financial resources are brought by large-scale image data labeling.

However, since the size conversion range of the lesion area of the tumor is relatively large, the above two schemes cannot simultaneously consider the segmentation of lesion areas of different sizes, and the segmentation is prone to be inaccurate.

Disclosure of Invention

The embodiment of the invention provides an image processing method, an image segmentation network training device, electronic equipment and a storage medium.

The technical scheme of the embodiment of the invention is realized as follows:

in a first aspect, an embodiment of the present invention provides an image processing method, including:

aiming at an MRI image, carrying out inclusion convolution by using an inclusion convolution module of an image segmentation network to obtain a first image characteristic;

performing hole convolution on the first image characteristic by using a hole convolution module of the image segmentation network to obtain a second image characteristic;

and segmenting a tumor lesion region from the MRI image based on the second image characteristic, wherein the tumor lesion region is a region where pixels imaged by tumor lesion tissues are located.

In a second aspect, an embodiment of the present invention provides a method for training an image segmentation network, including:

acquiring a training sample set of an MRI image; wherein the training sample set comprises: labeled training samples and unlabeled training samples;

inputting the MRI image of the training sample set into a first network which completes pre-training by using the labeled training data to obtain the predicted segmentation information output by the first network; wherein the prediction partitioning information includes: prediction segmentation information of the labeled training samples;

obtaining a first segmentation quality score of the prediction segmentation information of the first network based on the prediction segmentation information of the labeled training sample through a second network;

determining a first loss value of the first network according to a difference between the predicted segmentation information of the labeled training samples and the label;

determining a second loss value for the second network based on the first segmentation quality score and labels of the labeled training samples;

determining whether to continue training the first network based on the first loss value and the second loss value.

In a third aspect, an embodiment of the present invention provides an image processing apparatus, including:

the first convolution module is used for carrying out inclusion convolution on the MRI image by utilizing an inclusion convolution module of the image segmentation network to obtain a first image characteristic;

the second convolution module is used for performing hole convolution on the first image characteristic by using a hole convolution module of the image segmentation network to obtain a second image characteristic;

and the segmentation module is used for segmenting a tumor lesion area from the MRI image based on the second image characteristic, wherein the tumor lesion area is an area where pixels of tumor lesion tissue imaging are located.

In a fourth aspect, an embodiment of the present invention provides a training apparatus for an image segmentation network, where the training apparatus includes:

the acquisition module is used for acquiring a training sample set of MRI images; wherein the training sample set comprises: labeled training samples and unlabeled training samples;

the first network module is used for inputting the MRI image of the training sample set into a first network which completes pre-training by using the labeled training data to obtain the predicted segmentation information output by the first network; wherein the prediction partitioning information includes: prediction segmentation information of the labeled training samples;

the second network module is used for obtaining a first segmentation quality score of the prediction segmentation information of the first network based on the prediction segmentation information of the labeled training sample through a second network;

a loss determination module configured to determine a first loss value of the first network according to a difference between the predicted segmentation information of the labeled training samples and the label; determining a second loss value for the second network based on the first segmentation quality score and labels of the labeled training samples; determining whether to continue training the first network based on the first loss value and the second loss value.

In a fifth aspect, an embodiment of the present invention provides an electronic device, including:

a memory for storing executable instructions;

and the processor is used for implementing the image processing method provided by one or more of the technical schemes when executing the executable instructions stored in the memory.

In a sixth aspect, an embodiment of the present invention provides an electronic device, including:

a memory for storing executable instructions;

and the processor is used for implementing the training method of the image segmentation network provided by one or more of the technical solutions when executing the executable instructions stored in the memory.

In a seventh aspect, an embodiment of the present invention provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions; after being executed by a processor, the computer-executable instructions can implement the image processing method provided by one or more of the technical solutions.

In an eighth aspect, an embodiment of the present invention provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions; the computer-executable instructions, when executed by a processor, can implement the method for training the image segmentation network provided by one or more of the foregoing technical solutions.

The embodiment of the invention provides an image processing method, an image segmentation network training device, electronic equipment and a storage medium. Inputting an MRI image into a trained image segmentation network, and carrying out Incep convolution on the MRI image by an Incep convolution module in the image segmentation network to obtain a first image characteristic; performing hole convolution on the first image characteristics through a hole convolution module to obtain second image characteristics; segmenting a tumor lesion region from the MRI image based on the second image feature.

The inclusion convolution module and the void convolution module are used for performing inclusion convolution and void convolution on the MRI image, on one hand, convolution kernels with different sizes in the inclusion convolution module can be used for extracting multi-scale image features in the MRI image so as to obtain more contour information and detail information of the MRI image and improve the segmentation precision of the image; and the depth of the network can be increased, and the adaptability of the network to the scale and the nonlinear capability of the network are enhanced.

On the other hand, the cavity convolution module can be utilized to increase the receptive field under the condition of ensuring that the image resolution is not changed so as to adapt to different sizes of tumor lesion areas in the MRI image. Therefore, the inclusion convolution and the void convolution are directly carried out on the MRI image, so that the segmentation precision of the MRI image can be effectively improved.

Drawings

Fig. 1 is a schematic flowchart of an image processing method according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating a training method of an image segmentation network according to an embodiment of the present invention;

fig. 3 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of a training apparatus for an image segmentation network according to an embodiment of the present invention;

fig. 5 is a schematic flowchart of an image processing method provided by this example;

fig. 6 is a schematic network architecture diagram of an image segmentation network provided by the present example;

fig. 7 is a schematic diagram of a network architecture of a discriminant network provided in this example;

fig. 8 is a schematic diagram of a network architecture for generating a countermeasure network according to the present example.

Detailed Description

The technical solution of the present invention is further described in detail with reference to the drawings and specific embodiments.

An embodiment of the present invention provides an image processing method, and fig. 1 is a schematic flowchart of the image processing method provided in the embodiment of the present invention, as shown in fig. 1, including the following steps:

step 101, aiming at an MRI image, carrying out inclusion convolution by using an inclusion convolution module of an image segmentation network to obtain a first image characteristic;

102, performing hole convolution on the first image characteristic by using a hole convolution module of the image segmentation network to obtain a second image characteristic;

and 103, segmenting a tumor lesion area from the MRI image based on the second image characteristic, wherein the tumor lesion area is an area where pixels of tumor lesion tissue imaging are located.

The image processing method related in the embodiment of the present invention can be applied to an electronic device; here, the electronic device includes a terminal or a server, and the terminal may be a mobile phone, a tablet computer, a notebook computer, or the like; the server may be an independent physical server, or may be a server cluster or a distributed system formed by a plurality of physical servers.

In step 101, the MRI image is input into a pre-trained image segmentation network, and an inclusion convolution module of the image segmentation network performs an inclusion convolution to obtain a first image feature.

It should be noted that the inclusion convolution module may execute a plurality of convolution operations or pooling operations in parallel, and concatenate all output results into one feature map, thereby improving the utilization rate of network resources, and improving the width and depth of the network under the condition that the calculation amount is not changed.

In some embodiments, the step 101 may include:

determining a plurality of images to be segmented based on sliding operation of a sliding window with a preset size on the MRI image;

utilizing an inclusion convolution module of an image segmentation network to carry out inclusion convolution on the plurality of images to be segmented to obtain a first image characteristic;

it should be noted that, since the size of the MRI image acquired by the medical imaging device is not necessarily the same as the size adapted to the image segmentation network in the training process, a plurality of images to be segmented with preset sizes can be extracted from the MRI image by sliding a sliding window with preset sizes on the MRI image, so that the size of the images to be segmented is adapted to the network model.

In the embodiment of the invention, the adjacent images to be segmented can have overlapping parts, and all the images to be segmented can comprise information in the complete MRI image, so that the accuracy of the obtained prediction result after the MRI image is input into the image segmentation network can be ensured, and the completeness of the finally obtained result can be ensured.

In step 102, the hole convolution module performs hole convolution on the first image feature to obtain a second image feature after the receptive field is expanded.

The hole convolution module can extract multi-scale features, has the functions of a convolution layer and a pooling layer, ensures that the size of an output image is the same as that of an input hole convolution module, can expand the receptive field of a convolution kernel, extracts deeper information of the image and retains shallow information. That is to say, by performing the hole convolution on the image features, the utilization rate of the pixels in the receptive field range can be increased, so that all the pixels participate in the calculation, and the input image features are fully utilized, so that the prediction result is more accurate.

Here, in the convolution operation of the neural network, the receptive field is the size of the area on the input feature map to which the pixel points on the output feature map of each layer of the neural network are mapped, that is, the receptive field is the size on the input image to which the pixel points on the output result of a certain layer of the neural network are mapped. Generally, the size of the field of the input features of the first layer of convolutional layer is equal to the size of the convolutional kernel, and the size of the field of the higher layer of convolutional layer is related to the size and the step length of the convolutional kernel of all the previous layers, so that different levels of information can be captured based on different fields, and the purpose of extracting different scales of feature information is achieved.

It should be noted that, in the conventional method for enlarging the receptive field, the image features are pooled using the maximum pooling layer, but this approach may reduce the resolution of the image. And the hole convolution is based on the conventional convolution operation, and inserts holes with the value of 0 around each parameter in a convolution kernel, so that the resolution of a feature map is not lost while the receptive field is increased.

In step 103, the second image feature may be a multi-scale image feature indicating a tumor lesion region in the MRI image.

In the embodiment of the invention, the second image characteristics output by the hole convolution module under different scales can be fused to obtain the multi-channel image characteristics, and the multi-channel image characteristics are subjected to channel transformation processing to obtain the prediction segmentation information.

Illustratively, second image features of different scales can be spliced to obtain multi-channel image features; the channels are transformed by using 1 × 1 convolutional layers, so that the predicted segmentation information output by the 1 × 1 convolutional layers is consistent with the number of channels of the input image of the image segmentation network.

The prediction segmentation information can be subjected to up-sampling processing to obtain a segmentation probability map with the same size as the input MRI image; and segmenting the MRI image according to the segmentation probability map to obtain a tumor lesion area in the MRI image.

Optionally, the inclusion convolution module has N inclusion convolution layers, and convolution kernels of different inclusion convolution layers are different; the different inclusion convolution layers are used for extracting the first image features with different scales.

Here, N is a positive integer greater than or equal to 1;

in the embodiment of the invention, convolution kernels of different inclusion convolution layers in the inclusion convolution modules are different, and the convolution kernels of different scales are adopted for the MRI image to carry out the inclusion convolution, so that the image features of the MRI image in multiple scales can be extracted simultaneously.

The inclusion convolution module divides the input into a plurality of branches by applying different convolution kernels, and extracts image features of different scales of the plurality of branches respectively by using the inclusion convolution layers of different convolution kernels.

Optionally, the hole convolution module includes: n said hole convolution dense sub-modules;

the inclusion convolution layer and the hole convolution dense sub-modules are alternately distributed in the image segmentation network, and the hole convolution dense sub-modules are used for performing hole convolution processing on the first image features of different scales.

Here, N is a positive integer greater than or equal to 1; it is understood that the number of hole convolution dense sub-modules in the hole convolution module is the same as the number of inclusion convolution layers in the inclusion convolution module.

In the embodiment of the invention, the inclusion convolutional layers and the cavity convolutional dense sub-modules are alternately distributed in the image segmentation network, and the inclusion convolutional layers are used for carrying out inclusion convolution on the input MRI image to obtain first image features with different scales; and performing hole convolution processing on the first image features of different scales output by the inclusion convolution layer through a hole convolution dense submodule to obtain second image features of multiple scales.

It should be noted that the dense sub-modules are composed of a plurality of convolutional layers, each dense sub-module uses the same number of output channels, but in forward propagation, the output of each dense sub-module is combined with its output in the channel dimension for the next dense sub-module.

Optionally, the performing, by using a hole convolution module of the image segmentation network, a hole convolution on the first image feature to obtain a second image feature includes:

performing hole convolution on the received first image characteristics to obtain a hole convolution result;

performing Batch Normalization (BN) processing on the cavity convolution result to obtain a Normalization processing result;

and carrying out rectification processing on the normalization processing result based on a leakage rectification function to obtain the second image characteristic.

In the embodiment of the present invention, the performing batch normalization processing on the cavity convolution result to obtain a normalization processing result includes:

acquiring the mean value and the standard deviation of the void convolution result;

based on the mean value and the standard deviation, carrying out normalization processing on the cavity convolution result;

and reconstructing the normalized void convolution result based on preset learning parameters to obtain the normalized result.

Here, the preset learning parameters may be set according to actual requirements;

according to the embodiment of the invention, the normalization processing is firstly carried out on the cavity convolution result through the mean value and the standard deviation, so that the output result of each cavity convolution dense sub-module has the same data distribution, the generalization capability of the network can be improved, and the convergence speed is accelerated; and then reconstructing the linear cavity convolution result after the normalization processing by using a preset learning parameter, so that the normalization processing result can be more suitable for the real distribution of the cavity convolution result, and the nonlinear expression capability of the model is ensured.

Illustratively, the normalizing the result of the hole convolution based on the mean and the variance may include:

subtracting the average value from the void convolution result to obtain a first void convolution result;

and dividing the first cavity convolution result by the variance to obtain a cavity convolution result after batch normalization processing.

It should be noted that, the leakage-accompanied rectification function (leakage-Linear Unit, leakage-ReLU) is used as an activation function, and can maintain some negative axis values while correcting data distribution, so that information of the negative axis is not completely lost, the problem of gradient disappearance is avoided, and occurrence of silent neurons can be reduced.

Optionally, the image processing method further comprises:

performing standard convolution processing on the MRI image to obtain a standard convolution result;

the method for obtaining the first image features of different scales by utilizing the inclusion convolution module of the image segmentation network to perform the inclusion convolution on the MRI image comprises the following steps:

and carrying out the inclusion convolution on the standard convolution result based on different convolution kernels by using the inclusion convolution module to obtain a plurality of first image characteristics with different scales.

In the embodiment of the invention, a standard convolution result for indicating the low-dimensional characteristics of the MRI image is obtained by performing standard convolution processing on the MRI image; and inputting the standard convolution result into an inclusion convolution module, and carrying out inclusion convolution processing on the standard convolution result by utilizing different convolution kernels in the inclusion convolution module to obtain first image features with different scales.

Here, the low-dimensional feature is used to indicate global contour information of the MRI picture image, for example, edge, line information, and the like; the first image feature may be a high-dimensional feature for indicating detail information of an MRI picture image.

Next, an embodiment of the present invention provides a training method for an image segmentation network, as shown in fig. 2, fig. 2 is a schematic flow chart of the training method for an image segmentation network provided in the embodiment of the present invention. The method comprises the following steps:

step 201, acquiring a training sample set of an MRI image; wherein the training sample set comprises: labeled training samples and unlabeled training samples;

step 202, inputting the MRI image of the training sample set to a first network which completes pre-training by using the labeled training data, and obtaining the predicted segmentation information output by the first network; wherein the prediction partitioning information includes: prediction segmentation information of the labeled training samples;

step 203, obtaining a first segmentation quality score of the prediction segmentation information of the first network through a second network based on the prediction segmentation information of the labeled training sample;

step 204, determining a first loss value of the first network according to the difference between the predicted segmentation information of the labeled training sample and the label;

step 205, determining a second loss value of the second network based on the first segmentation quality score and the label of the labeled training sample;

step 206, determining whether to continue training the first network based on the first loss value and the second loss value.

In step 201, a large number of MRI image images are acquired, and the MRI image images are divided into labeled training samples and unlabeled training samples according to whether the MRI image images carry labels. Wherein the label is used to indicate a tumor lesion area in an MRI image of the training sample.

It should be noted that the training sample set is an MRI image used for adjusting parameters to be trained in the first network and the second network; in an embodiment of the present invention, the training sample set includes: labeled training samples and unlabeled training samples.

In the related art, since the image segmentation network is trained in a complete supervised learning manner, a large number of labeled MRI image images are required to participate in the training, so that the overfitting of the network model can be effectively avoided, and the robustness of the network model is ensured. However, in the medical field, the process of labeling the MRI image is difficult to obtain the label of the MRI image, and the number of the MRI image with the label is small, which affects the accuracy of the image segmentation network. In contrast, in the embodiment of the invention, the image segmentation network is pre-trained through the labeled MRI image, and the pre-trained image segmentation network is trained in a semi-supervised learning mode by using the unlabeled MRI image and the labeled MRI image through the discrimination network, so that a large number of unlabeled MRI images are fully utilized, and the accuracy of the image segmentation network is improved.

In step 202, the predicted segmentation information is used to indicate a tumor lesion region in the MRI image.

The first network is a network model obtained by training based on a complete supervision mode by utilizing the labeled training data; inputting MRI image images in a training sample set into a first network to obtain prediction segmentation information output by the first network;

in step 203, the MRI image and the predictive segmentation information corresponding to the MRI image are input to a second network, and a first segmentation quality score corresponding to the predictive segmentation information is obtained.

In an embodiment of the present invention, the first segmentation quality score is used to indicate an accuracy of a predicted segmentation result of the labeled training samples output by the first network. Exemplarily, if the first segmentation quality score is 1, the segmentation quality of the predicted segmentation information output by the first network is characterized to be good; and if the first segmentation quality score is 0, representing that the segmentation quality of the predicted segmentation information output by the first network is poor.

In the training process of the image segmentation network, the second network may be pre-trained by using the predicted segmentation information and the labels of the labeled training samples.

The MRI image of the training sample with the label, the label corresponding to the MRI image and the prediction segmentation information can be input into a second network, and a first segmentation quality score of the prediction segmentation information corresponding to the MRI image with the label output by the second network is obtained; and adjusting the parameters to be trained in the second network by using the first segmentation quality score and the prediction segmentation information.

In step 204, a first loss value of the first network may be determined using a difference between the predicted segmentation information of the labeled training samples and the labels, and a first segmentation quality score of the labeled training samples;

in the embodiment of the invention, the first network and the second network are respectively a generating sub-network and a judging sub-network in the generation confrontation network, the output result of the first network is utilized to guide the second network to output more accurate segmentation quality scores through the confrontation learning between the first network and the second network, and the segmentation quality scores output by the second network are utilized to promote the first network to improve the accuracy of image segmentation.

Thus, in determining the first loss function value for the first network, both the difference between the predicted segmentation information for the labeled training samples and the labels and the first segmentation quality score for the labeled training samples need to be considered.

In step 205, for the labeled training sample, the MRI picture image of the labeled training sample, the predictive segmentation information of the MRI picture image, and the label are input to a second network, and a second loss value of the second network is determined according to a first segmentation quality score for the predictive segmentation information and a segmentation quality score for the label output by the second network.

In the embodiment of the invention, in the training process of the image segmentation network, the predicted segmentation information and the label of a training sample with a label can be used for determining a second loss value of the second network; pre-training the second network using the second loss value; and performing segmentation quality evaluation on the labeled training samples and the unlabeled training samples again by using the pre-trained second network, so as to train the first network according to the first segmentation quality scores of the labeled training samples and the second segmentation quality scores of the unlabeled training samples.

In step 206, it is determined whether the first loss value and the second loss value satisfy the condition for stopping training, and if the first loss value or the second loss value does not satisfy the condition for stopping training, parameters to be trained in the first network and the second network are continuously optimized until the first loss value and the second loss value both satisfy the condition for stopping training, so as to obtain the image segmentation network.

Here, the stop training condition may be that the training reaches the number of iterations and/or that the first loss function and the second loss function converge. Here, the number of iterations may be determined according to the number of MRI image in the training sample set and the size of the initial network, and the present invention is not particularly limited.

Optionally, the training method of the image segmentation network further includes:

obtaining a second segmentation quality score of the prediction segmentation information of the first network based on the prediction segmentation information of the unlabeled training sample through a second network;

determining, by the computing device, a first loss value of the first network based on a difference between the predicted segmentation information of the labeled training samples and the label, including:

and obtaining the first loss value according to the difference between the predicted segmentation information of the labeled training sample and the label, the first segmentation quality score and the second segmentation quality score.

In an embodiment of the present invention, the second segmentation quality score is used to indicate an accuracy of a predicted segmentation result of the unlabeled training samples output by the first network.

Inputting an MRI image in the unlabeled training sample into a first network to obtain the predictive segmentation information of the unlabeled training sample output by the first network; and inputting the predicted segmentation information and the MRI image into a second network to obtain a second segmentation quality score output by the second network.

It should be noted that, in practical applications, the number of MRI images with labels is limited, and training of the first network in a fully supervised manner requires a large amount of labeled training data, so as to effectively improve the accuracy of the first network.

In contrast, in the embodiment of the present invention, a semi-supervised mode is adopted, and the prediction segmentation information corresponding to the unlabeled training sample and the prediction segmentation information corresponding to the labeled training sample, which are output by the first network, are obtained by inputting the unlabeled training sample and the labeled training sample to the first network; respectively obtaining a first segmentation quality score corresponding to the labeled prediction segmentation information and a second segmentation quality score corresponding to the unlabeled prediction segmentation information by using a second network; and determining a loss function value of the first network by using the difference between the predicted segmentation information and the label of the labeled training data, the first segmentation quality score and the second segmentation quality score, so as to adjust the parameter to be trained in the first network.

Optionally, the obtaining the first loss value according to the difference between the predicted segmentation information of the labeled training sample and the label, the first segmentation quality score and the second segmentation quality score includes:

determining a loss value for the second network based on the first segmentation quality score;

adjusting a parameter to be trained in the second network based on the loss value of the second network;

estimating the prediction segmentation information of the labeled training samples and the prediction segmentation information of the unlabeled training samples by using the adjusted second network to obtain a first segmentation quality score of the prediction segmentation information of the labeled training samples and a second segmentation quality score of the prediction segmentation information of the unlabeled training samples;

determining a first loss value for the first network based on a difference between the predicted segmentation information and the label for the labeled training samples, the first segmentation quality score, and the second segmentation quality score.

In the embodiment of the invention, the first network and the second network are in a mode of confrontation training, and the output result of the first network can be used for guiding the second network to output a more accurate segmentation quality score; the segmentation quality score output by the second network can be used to prompt the first network to improve the accuracy of image segmentation.

Determining a loss value for the second network using the first segmentation quality score and a difference between the predicted segmentation information for the labeled training samples and the label; pre-training the second network according to the loss value of the second network; respectively inputting the labeled predicted segmentation information and the unlabeled predicted segmentation information output by the first network into a pre-trained second network, and determining the loss value of the first network by using a first segmentation quality score and a second segmentation quality score output by the second network, the predicted segmentation information and the labels; therefore, the first network can be trained by using the labeled training samples and the unlabeled training samples to obtain the image segmentation network.

Optionally, the determining a first loss value of the first network according to the difference between the predicted segmentation information of the labeled training samples and the label, a first segmentation quality score and a second segmentation quality score includes:

determining a segmentation loss function value corresponding to the label according to the predicted segmentation information of the labeled training sample and the label;

determining a binary cross entropy corresponding to the first segmentation quality score according to the first segmentation quality score;

determining a binary cross entropy corresponding to the second segmentation quality score according to the second segmentation quality score;

and performing weighted fusion on the segmentation loss function value, the binary cross entropy corresponding to the first segmentation quality score and the binary cross entropy corresponding to the second segmentation quality score to obtain a first loss value of the first network.

In the embodiment of the present invention, the first loss function of the first network is composed of a segmentation loss function and a countermeasure loss function; the segmentation loss function value corresponding to the label is used to indicate a difference between the labeled training samples and the label. The countermeasure loss function can include: the binary cross entropy corresponding to the first segmentation quality score and the binary cross entropy corresponding to the second segmentation quality score;

the binary cross entropy corresponding to the first segmentation quality score is used for determining a loss value of the first segmentation quality score corresponding to the labeled training sample; and the binary cross entropy corresponding to the second segmentation quality score is used for determining the loss value of the second segmentation quality score corresponding to the unlabeled training sample.

It should be noted that, by using the segmentation loss function and the labeled training sample, the basic segmentation effect of the first network is improved; representing the confrontation relation between the predicted segmentation information and the real label output by the first network through the binary cross entropy corresponding to the first segmentation quality score and the binary cross entropy corresponding to the second segmentation quality score; thereby bringing the labeled training samples and unlabeled training samples closer to the labels.

The division loss function value, the binary cross entropy of the first division quality score and the weight corresponding to the binary cross entropy of the second division quality score can be set according to actual requirements.

In some embodiments, the first loss function of the first network may be represented by:

wherein, the _G () is a first loss function of the first network; said I _l The MRI image with the label is obtained; said I _ul The image is an MRI image without a label; said y _l ' is the prediction segmentation information corresponding to the labeled MRI image; said y _ul ' is prediction segmentation information corresponding to an unlabeled MRI image; theta is described _G A weight parameter to be trained for the first network; the above-mentioned _seg () a segmentation loss function for the first network; the above-mentioned

A weight of a binary cross entropy scored for the first segmentation quality; the above-mentioned

The weight of the binary cross entropy of the second segmentation quality score is obtained; l is _DCE () is the Dice cross entropy loss function; said D (I) _l ,y _l ') is a first segmentation quality score output by the second network; said D (I) _ul ,y _ul ') is a second segmentation quality score output by the second network.

Optionally, the determining a second loss value for the second network based on the first segmentation quality score and the label of the labeled training sample comprises:

determining a second loss value for the second network based on the first segmentation quality score, the labels of the labeled training samples, and the second segmentation quality score.

In the embodiment of the invention, the prediction segmentation information of the labeled MRI image and the prediction segmentation information of the unlabeled MRI image are output through the first network; inputting the labeled MRI image, and the label and the prediction segmentation information corresponding to the labeled MRI image into a second network to obtain a first segmentation quality score corresponding to the labeled prediction segmentation information and a segmentation quality score corresponding to the label output by the second network; inputting the unlabeled MRI image and the prediction segmentation information corresponding to the unlabeled MRI image into a second network to obtain a second segmentation quality score corresponding to the unlabeled prediction segmentation information output by the second network;

and determining the difference between the labeled predicted segmentation result, the unlabeled predicted segmentation result and the MRI image and the difference between the label and the MRI image according to the segmentation quality score corresponding to the label, the first segmentation quality score and the second segmentation quality score, thereby determining a second loss value of the second network.

Optionally, the determining a second loss value for the second network based on the first segmentation quality score, the labels of the labeled training samples, and the second segmentation quality score comprises:

determining a binary cross entropy corresponding to the label according to the segmentation quality score corresponding to the label;

and performing weighted fusion on the binary cross entropy corresponding to the label, the binary cross entropy corresponding to the first segmentation quality score and the binary cross entropy corresponding to the second segmentation quality score to obtain a second loss value of the second network.

In the embodiment of the invention, a second network is utilized, based on the MRI image with the label and the label, the segmentation quality score and the first segmentation quality score corresponding to the label are determined, and the binary cross entropy of the segmentation quality score corresponding to the label, the binary cross entropy of the first segmentation quality score and the binary cross entropy of the second segmentation quality score are utilized

In some embodiments, the second loss function of the second network may be represented by:

wherein, the l _D () a second loss function for the second network; theta is described _D A weight parameter to be trained for the second network; the above-mentioned

And the weight of the binary cross entropy of the second segmentation quality score is obtained.

inputting the labeled training data into an initial network to obtain the prediction segmentation information output by the initial network;

determining a segmentation loss function value of the initial network according to a difference between the predicted segmentation information and the label;

and adjusting the parameters to be trained of the initial network according to the segmentation loss function values of the initial network to obtain a first network.

In the embodiment of the invention, the initial network is trained based on a complete supervision mode by utilizing the labeled training data to obtain the first network.

The parameters to be trained of the initial network may comprise one or more of the following parameters: iteration number, batch processing size, image size, learning rate, and learning rate attenuation value.

Here, the number of iterations refers to the number of training times of all training images in the training sample set, and is generally determined according to the number of training images and the size of an initial network; illustratively, the number of iterations may be set to 1500. The batch size represents the number of images loaded by one forward propagation of the network; illustratively, the batch size may be set to 16. The image size refers to that the image needs to be adjusted to a proper size before training, and the size of the image is changed according to a predefined length and width, and can be set to 224 × 224 by way of example. The learning rate represents the learning rate of the control model, and if the set learning rate is too low, the trained network model is too slow in convergence; if the set learning rate is too large, it will cause the loss function to oscillate. Therefore, in order to avoid the unreasonable learning rate setting in the training process, a learning rate is dynamically adjusted, and a learning rate attenuation value is set, so that the learning rate in the training process is decreased exponentially along with the number of training rounds, and the convergence gradient is decreased.

Inputting training data of the MRI image with the label into an initial network to obtain predicted segmentation information which is output by the initial network and used for indicating a tumor lesion area in the MRI image; and calculating a segmentation loss function value of the initial network according to the difference between the prediction segmentation information and the label, calculating a back propagation gradient according to the segmentation loss function value, updating a parameter to be trained in the initial network, and repeating the steps until the iteration times or the loss function convergence is reached to obtain a first network.

Optionally, the determining a segmentation loss function value of the initial network according to a difference between the predicted segmentation information and the labeling information includes:

respectively determining binary cross entropy and Dice cross entropy of the prediction division information and the labeling information based on the prediction division information and the labeling information;

and carrying out weighted fusion on the binary cross entropy and the Dice cross entropy to obtain a segmentation loss function value of the initial network.

In the embodiment of the invention, the segmentation loss function of the initial network consists of a Dice cross entropy loss function and a binary cross entropy loss function, wherein the Dice cross entropy loss function is used for determining the similarity degree between two tumor lesion areas in the MR I image; the binary cross entropy loss function is used for determining the loss value of each pixel point in the MRI image, and has a good effect on the segmentation details of the MRI image with undefined boundary and uneven gray level.

By performing weighted fusion on the binary cross entropy and the Dice cross entropy of the prediction segmentation information and the labeling information, not only can different scale shapes of the lesion region in the MRI image be considered, but also the influence of the imaging quality (such as uneven gray scale and noise) of the MRI image on the boundary segmentation of the lesion region can be considered.

Here, the binary cross entropy and the weight of the Dice cross entropy may be set according to actual requirements, and for example, the binary cross entropy and the weight of the Dice cross entropy may be set to 0.5.

In some embodiments, the segmentation loss function of the initial network may be represented by:

l _seg (y′,y；θ _G )＝αl _DCE (y′,y；θ _G )+βl _BCE (y′,y；θ _G )；

wherein, the _seg () is a segmentation loss function for the initial network; y is a label of the MRI image; the y' is the prediction segmentation information of the MRI image; theta is described _G A weight parameter to be trained of the initial network; the alpha is the weight of the Dice cross entropy; the above-mentioned _DCE (. Is the Dice cross entropy loss function; the beta is the weight of the binary cross entropy; the above-mentioned _BCE (. H) is the binary cross entropy loss function.

Optionally, the obtaining, by the second network, a segmentation quality score of the predictive segmentation information of the first network based on the predictive segmentation information of the labeled training sample includes:

inputting the prediction segmentation information and the labeled training sample into a second network, and processing the prediction segmentation information through an image intensity attention module and a geometric boundary attention module of the second network to obtain an intensity information difference characteristic between a tumor lesion area and a background area of an MRI image of the labeled training sample, a contour characteristic of the tumor lesion area and a contour characteristic of the background area;

and determining a segmentation quality score corresponding to the prediction segmentation information according to the intensity signal difference characteristic, the contour characteristic of the tumor lesion region and the contour characteristic of the background region.

In an embodiment of the present invention, the image intensity attention module is configured to extract an intensity information difference feature between a tumor lesion area and a background area in the MRI image based on the prediction segmentation information and the training sample; the geometric boundary attention module is used for extracting contour features of a tumor lesion region and contour features of a background region in the MRI image.

It should be noted that, by using the image intensity attention module and the geometric boundary attention module, the information contained in the lesion region of the tumor and the background region in the MRI image is fully utilized, and the contrast between the lesion region of the tumor and the background region is expanded, so that the second network can more effectively identify the lesion region of the tumor by using the difference characteristic of the intensity signal, the contour characteristic of the lesion region of the tumor and the contour characteristic of the background region, and thus the accuracy of the prediction segmentation information is determined.

Extracting image features of the MRI image based on the MRI image; extracting intensity signal difference characteristics, contour characteristics of a tumor lesion area and contour characteristics of the background area based on the prediction segmentation information of the MRI image; and determining the difference between the MRI image and the predicted segmentation information by using the intensity signal difference characteristic and the contour characteristic of the MRI image and the predicted segmentation information, thereby determining the segmentation quality score corresponding to the predicted segmentation information.

Optionally, the processing the predictive segmentation information by the image intensity attention module and the geometric boundary attention module of the second network to obtain an intensity information difference feature between a tumor lesion region and a background region, a contour feature of the tumor lesion region, and a contour feature of the background region of the MRI image of the labeled training sample includes:

performing standard convolution processing on the prediction segmentation information to respectively obtain tumor lesion area characteristics and the background area characteristics of the MRI image;

classifying the tumor lesion region characteristics and the background region characteristics to obtain attention information corresponding to the tumor lesion region of the MRI image and attention information of the background region;

and fusing the MRI image, the attention information corresponding to the tumor lesion area and the attention information of the background area to obtain the intensity information difference characteristic between the tumor lesion area and the background area of the MRI image, the contour characteristic of the tumor lesion area and the contour characteristic of the background area.

In the embodiment of the invention, the characteristics of a tumor lesion area and the characteristics of a background area in the MRI image are respectively extracted by carrying out parallel standard convolution processing on the prediction segmentation information; respectively processing the tumor lesion region characteristics and the background region characteristics by using an activation function to obtain attention information corresponding to the tumor lesion region and the background region;

the fusing the MRI image, the attention information corresponding to the tumor lesion region, and the attention information of the background region may include:

performing convolution processing on the MRI image to obtain image characteristics of the MRI image;

multiplying the image characteristics of the MRI image with the attention information corresponding to the tumor lesion area to obtain a first attention characteristic;

and multiplying the image characteristics of the MRI image with the attention information corresponding to the background area to obtain a second attention characteristic.

In an embodiment of the present disclosure, the first attention feature is an image feature fused with attention information corresponding to a tumor lesion region; the second attention feature is an image feature fused with attention information corresponding to the background area.

The first attention feature may be used to indicate a contour feature of the tumor lesion region, and the second attention feature may be used to indicate a contour feature of the background region; and determining an intensity information difference characteristic between the tumor lesion region and a background region by using the difference between the first attention characteristic and the second attention characteristic.

Optionally, the determining, according to the intensity signal difference feature, the contour feature of the tumor lesion region, and the contour feature of the background region, a segmentation quality score corresponding to the predicted segmentation information includes:

carrying out cascade processing on the intensity signal difference characteristic, the contour characteristic of a tumor lesion area and the contour characteristic of the background area to obtain dual attention characteristics;

and determining a segmentation quality score corresponding to the prediction segmentation information based on the dual attention features.

In the embodiment of the invention, the intensity signal difference feature, the contour feature of the tumor lesion region and the contour feature of the background region can be cascaded through feature fusion of the first attention feature and the second attention feature, so that the dual attention feature is obtained.

Illustratively, the feature fusion of the first attention feature and the second attention feature may be achieved by adding elements of the first attention feature and the second attention feature.

It should be noted that, based on the dual attention feature, the information contained in the background region and the information contained in the tumor lesion region can be fully utilized to expand the contrast between the background region and the tumor lesion region, thereby facilitating the identification of the tumor lesion region.

Performing convolution processing on the double attention features to obtain third image features; the third image feature may be used to indicate a probability value that the predictive segmentation information is an annotated sample. And obtaining a segmentation quality score corresponding to the prediction segmentation information according to the third image probability.

Optionally, the determining, based on the dual attention feature, a segmentation quality score corresponding to the predicted segmentation information includes:

aiming at the dual attention features, performing inclusion convolution by using an inclusion convolution module of the second network to obtain third image features with different scales;

performing hole convolution on the third image characteristic by using a hole convolution module of a second network to obtain a fourth image characteristic;

and determining a segmentation quality score corresponding to the predicted segmentation information based on the fourth image feature.

In the embodiment of the invention, the inclusion convolution module is provided with N inclusion convolution layers, and convolution kernels of different inclusion convolution layers are different; here, N is a positive integer greater than or equal to 1.

Extracting third image features of different scales from the dual attention features respectively through different inclusion convolution layers in the inclusion convolution module;

the hole convolution module is provided with N hole convolution dense sub-modules, the inclusion convolution layer and the hole convolution dense sub-modules are arranged in the second network in an interpenetration mode, and hole convolution processing is carried out on third image features of different scales output by the inclusion convolution layer through the hole convolution dense sub-modules to obtain fourth image features.

cutting an MRI image in a training sample set to obtain a tumor lesion area image in the MRI image;

carrying out gray scale normalization processing on the tumor lesion area image;

carrying out geometric transformation processing on the tumor lesion area image subjected to gray level normalization processing to obtain an enhanced MRI image;

the inputting of the MRI image of the training sample set to a first network that performs pre-training using labeled training data includes:

inputting the enhanced MRI image into the first network.

In an embodiment of the present invention, the clipping process may include: cutting the area which does not contain tumor information in the MRI image, and the boundary area and the background area in the MRI image;

cutting MRI image images in a training sample set to obtain cut MRI image images, and sliding the cut MRI image images through a sliding window with a preset size to obtain a plurality of images to be cut with the preset size;

here, the preset size of the sliding window is the same as the image size adapted by the first network. For example, the preset size is 224 × 224.

The gray level normalization processing refers to that the gray level value of each pixel point in the image to be processed is set in a specific range. In an embodiment of the present invention, the gray-scale normalization process may be performed on the images of the tumor lesion areas to make the gray-scale value distributions of the images of the tumor lesion areas similar.

The performing gray scale normalization processing on the tumor lesion region image may include:

determining a maximum pixel value and a minimum pixel value in the tumor lesion area image and difference value information between the maximum pixel value and the minimum pixel value according to the tumor lesion area image;

determining a pixel difference value corresponding to each pixel point according to the pixel value of each pixel point in the tumor lesion area image and the minimum pixel value;

and determining a tumor lesion area image after gray level normalization according to the pixel difference value corresponding to each pixel point and the ratio of the difference value information.

The geometric transformation processing of the tumor lesion area image can be realized by operating the tumor lesion area image after the gray level normalization processing and a geometric transformation function. Here, the specific form of the geometric transformation function is not limited, and may be a geometric transformation function representing horizontal inversion, a geometric transformation function representing vertical inversion, or the like.

It should be noted that, although the training image segmentation network needs a large number of labeled training samples, in practice, the number of labeled training samples is limited, and in order to diversify the training sample set as much as possible and improve the generalization capability of the image segmentation network, it is necessary to perform geometric transformation processing on the MRI image in the training sample set to enhance the data of the training sample set.

Next, an image processing apparatus 30 according to an embodiment of the present invention is provided, as shown in fig. 3, and fig. 3 is a schematic structural diagram of the image processing apparatus according to the embodiment of the present invention. The image processing apparatus includes:

the first convolution module 31 is configured to perform inclusion convolution on an MRI image by using an inclusion n convolution module of an image segmentation network to obtain a first image feature;

the second convolution module 32 is configured to perform a hole convolution on the first image feature by using a hole convolution module of the image segmentation network to obtain a second image feature;

a segmentation module 33, configured to segment a tumor lesion region from the MRI image based on the second image feature, where the tumor lesion region is a region where pixels of an image of a tumor lesion tissue are located.

Optionally, the inclusion convolution module has N inclusion convolution layers, where convolution kernels of different inclusion convolution layers are different; the different inclusion convolution layers are used for extracting the first image features with different scales.

the inclusion convolution layer and the hole convolution dense sub-modules are arranged in the image segmentation network in an interspersed mode, and the hole convolution dense sub-modules are used for performing hole convolution processing on the first image features of different scales.

Optionally, the second convolution module is specifically configured to:

carrying out batch normalization processing on the void convolution results to obtain normalization processing results;

and carrying out rectification processing on the normalization processing result based on the rectification function with leakage to obtain the second image characteristic.

Optionally, the image processing apparatus further comprises: the third convolution module is used for performing standard convolution processing on the MRI image to obtain a standard convolution result;

the first convolution module is used for carrying out increment convolution on the standard convolution result based on different convolution kernels by utilizing the increment convolution module to obtain a plurality of first image features with different scales.

Next, an image segmentation network training device 40 according to an embodiment of the present invention is provided, as shown in fig. 4, and fig. 4 is a schematic structural diagram of the image segmentation network training device according to an embodiment of the present invention. The training device of the image segmentation network comprises:

an obtaining module 41, configured to obtain a training sample set of MRI images; wherein the training sample set comprises: labeled training samples and unlabeled training samples;

a first network module 42, configured to input the MRI image of the training sample set to a first network that completes pre-training by using labeled training data, so as to obtain predicted segmentation information output by the first network; wherein the prediction partitioning information includes: prediction segmentation information of the labeled training samples;

a second network module 43, configured to obtain, through a second network, a first segmentation quality score of the prediction segmentation information of the first network based on the prediction segmentation information of the labeled training sample;

a loss determining module 44, configured to determine a first loss value of the first network according to a difference between the predicted segmentation information of the labeled training samples and the label; determining a second loss value for the second network based on the first segmentation quality score and the labels of the labeled training samples; determining whether to continue training the first network based on the first loss value and the second loss value.

Optionally, the second network module is further configured to obtain, by a second network, a second segmentation quality score for the predicted segmentation information of the first network based on the predicted segmentation information of the unlabeled training sample;

the loss determining module is configured to obtain the first loss value according to a difference between the prediction segmentation information of the labeled training sample and the label, and the first segmentation quality score and the second segmentation quality score.

Optionally, the loss determining module is specifically configured to:

Optionally, the loss determining module is configured to:

Optionally, the loss determining module is specifically configured to:

Optionally, the training apparatus for the image segmentation network further includes: a pre-training module to:

Optionally, the pre-training module is further configured to:

and performing weighted fusion on the binary cross entropy and the Dice cross entropy to obtain a segmentation loss function value of the initial network.

Optionally, the second network module is configured to:

and determining a segmentation quality score corresponding to the prediction segmentation information according to the intensity signal difference characteristic, the contour characteristic of the tumor lesion area and the contour characteristic of the background area.

Optionally, the second network module is specifically configured to:

performing standard convolution processing on the prediction segmentation information to respectively obtain the tumor lesion area characteristics and the background area characteristics of the MRI image;

classifying the tumor lesion region characteristics and the background region characteristics to obtain attention information corresponding to the tumor lesion region of the MRI image and the attention information of the background region;

Optionally, the second network module is specifically configured to:

and determining a segmentation quality score corresponding to the predicted segmentation information based on the dual attention features.

Optionally, the second network module is further specifically configured to:

Optionally, the apparatus further comprises: a pre-processing module to:

carrying out gray level normalization processing on the tumor lesion area image;

the first network module is configured to input the enhanced MRI image into the first network.

With reference to the above embodiments of the present invention, an exemplary application of the embodiments of the present invention in a practical application scenario will be described below.

The present example provides an image processing method, as shown in fig. 5, and fig. 5 is a flowchart illustrating an image processing method provided by the present example. The method comprises the following specific steps:

step 501, acquiring a training sample set of an MRI image, and preprocessing the MRI image;

acquiring a large number of MRI images to form a training sample set; wherein the training sample set comprises: labeled training samples and unlabeled training samples; and cutting the MRI image in the training sample set, and removing the area which does not contain any tumor information in the MRI image, and the boundary information and the background information of which the pixel values are 0 in the MRI image. And the MRI image obtained after cutting is uniformly zoomed into 224 multiplied by 224; carrying out image gray normalization processing on the zoomed MRI image, and compressing the value range of the gray value of the MRI image to [ -1,1] to enable the gray value distribution of all the MRI image images to be similar; and performing geometric transformation processing on the MRI image subjected to the gray level normalization processing.

It should be noted that, because a large number of labeled training samples are required for training the neural network in a fully supervised manner, in practical applications, the number of labeled MRI image sets is limited, and therefore, in order to diversify the training sample set as much as possible and improve the generalization capability of the neural network, data enhancement of the training sample set can be realized by performing geometric transformation processing on the MRI image sets.

Here, the geometric transformation process may include: random angular rotation, horizontal flipping, vertical flipping, and the like.

Step 502, constructing a semi-supervised generated confrontation network;

in an embodiment of the present invention, semi-supervised generation of a countermeasure network includes: an image segmentation network and a discrimination network; the image segmentation network is used for mapping random noise to real image data distribution; the discrimination network is used to discriminate whether the input image belongs to a real sample or generates an image sample. The confrontation network is generated through semi-supervision, a large number of unlabelled training samples can be effectively utilized, the accuracy of the image segmentation network is improved, and the expenditure of manpower and material resources brought by data labeling is greatly reduced.

In the embodiment of the invention, the semi-supervised generation countermeasure network can be constructed by respectively constructing the first network and the second network.

Here, the first network is a basic segmentation network obtained by training an initial network based on a labeled training sample. The initial network may employ an encoder-decoder architecture.

In the encoder part, as shown in fig. 6, fig. 6 is a schematic diagram of a network architecture of an image segmentation network provided by the present example. And setting a standard convolution layer at the initial part of the initial network, and performing standard convolution on the input MRI image through the standard convolution layer to extract the low-dimensional features of the MRI image. Specifically, the convolution kernel size of the standard convolution layer is 5 × 5, and the step size is 2.

Performing inclusion convolution on the basis of the low-dimensional features of the MRI image through an inclusion convolution module located at the output end of the standard convolution layer in the initial network to obtain first image features; and performing hole convolution on the first image characteristic through a hole convolution module to obtain a second image characteristic.

Since the boundary of a tumor lesion in the brain tumor MRI image is unclear, the image is blurred, and the gradation distribution is uneven, the feature information of higher layers is generated in order to more effectively integrate the feature information of lower layers of the MRI image.

Here, the inclusion convolution module is composed of a plurality of different inclusion convolution layers, and convolution kernels of the different inclusion convolution layers are different. For example, the inclusion convolution module may be composed of inclusion convolution layers corresponding to convolution kernels of 1 × 1, 3 × 3, and 5 × 5, etc. The hole convolution layer comprises a plurality of hole convolution dense sub-modules; wherein each hole convolution dense submodule comprises: void convolution layer, BN layer, and Leaky-ReLU function.

The inclusion convolution module can perform inclusion convolution on the low-dimensional features of the MRI image through a plurality of different inclusion convolution layers to obtain first image features of different scales;

the cavity convolution module can perform cavity convolution on the received first image characteristics through the cavity convolution layer to obtain a cavity convolution result; carrying out batch normalization on the cavity convolution result through the BN layer to obtain a normalization processing result; and rectifying the normalization processing result through a Leaky-ReLU function to obtain a second image characteristic.

In this example, the hole convolution layer in each hole convolution dense sub-module may adopt a hole convolution layer with a convolution kernel size of 3 × 3, an expansion rate parameter of 2, and a step size of 1; the receptive field is increased by performing a hole convolution on the first image feature to accommodate different sized tumor lesion areas.

It should be noted that each hole convolution dense sub-module in the hole convolution module is formed by connecting output iterations of all layers in a feedforward network in series according to channels, and a larger field of view can be sensed by the hole convolution layer with fewer down-sampling times. The image features of the lower dimensional layers may be called up by the hole convolution dense sub-module to help generate image features of the higher dimensional layers and ensure that the gradient propagates into deeper layers. According to the embodiment of the invention, the size of the tumor lesion area of different MRI images is considered to be possibly different, and the Incep convolution module and the cavity convolution module can capture multi-scale image characteristics, more accurately represent the tumor lesion area and adapt to the appearance shapes of the tumor lesion area in the MRI images with various changes.

For a decoder part, a standard convolutional layer, a BN layer and a Leaky-ReLU can be arranged at the output end of the cavity convolution module, second image features output by the cavity convolution module are combined through the standard convolutional layer, the BN layer and the Leaky-ReLU to obtain a segmentation probability graph, and the convolutional layer with the convolution kernel size of 1 multiplied by 1 is used for carrying out channel conversion processing on the segmentation probability graph to enable the segmentation probability graph to be consistent with the number of channels of an input MRI image; and the segmentation probability map is up-sampled by utilizing an up-sampling layer, and the size of the segmentation probability map is restored to be the same as that of the input MRI image.

The second network is a discriminant network for determining whether or not a distribution of the segmentation probability map output by the first network satisfies a genuine label. The second network requires two inputs, namely a brain tumor MRI image and a tumor segmentation result corresponding to the brain tumor MRI image. In practical application, for a training sample with a label, a brain tumor MRI image, the label and a segmentation probability map corresponding to the brain tumor MRI image output by an image segmentation network are required; and for the unlabeled training sample, only the brain tumor MRI image and the segmentation probability map corresponding to the brain tumor MRI image output by the image segmentation network are needed.

As shown in fig. 7, fig. 7 is a schematic diagram of a network architecture of a discriminant network according to this example. A dual attention module, namely an image intensity attention module and a geometric boundary attention module, is arranged at the initial part of the second network; wherein the image intensity attention module is used for acquiring the intensity signal difference between a tumor lesion area and a background area in the MRI image; the geometric boundary attention module is used for acquiring boundary characteristics and geometric deformation characteristics of a tumor lesion area in the MRI image. Important features in a tumor lesion region and a background region in the MRI image can be respectively extracted through a dual attention module, so that the quality of a prediction segmentation result output by an image segmentation network can be effectively evaluated according to the important features.

Firstly, extracting image characteristics from the brain tumor MRI image by using a dual attention module; the features of the tumor lesion area and the background area in the segmentation probability map are respectively extracted through two parallel convolution layers; and respectively carrying out activation processing on the characteristics of the tumor lesion area and the background area output by the two parallel convolution layers by adopting an S-shaped function to obtain an attention map of the tumor lesion area and an attention map of the background area.

And combining the image characteristics of the brain tumor MRI image with the attention diagram of the tumor lesion area and the attention diagram of the background area respectively in an element-by-element corresponding multiplication manner to obtain the attention characteristics used for indicating the tumor lesion area information and the reverse attention characteristics used for indicating the background area information.

And extracting deep image features used for indicating detail information in the segmentation probability map from the attention features and the reverse attention features output by the dual attention module through an inclusion convolution module and a hole convolution module.

Converting the extracted deep image features into feature vectors by a global average pooling layer, inputting the feature vectors into a full-link layer, and outputting a final evaluation score through the full-link layer based on the feature vector classification.

Step 503, pre-training the initial network by using the training sample with the label to obtain a first network;

training an initial network by using a training sample with a label, inputting preprocessed image data into the initial network in each iteration of a training stage, and determining a segmentation loss function value of the initial network according to a difference between a predicted segmentation probability graph output by the initial network and the label.

Setting the batch processing size of the initial network to be 16, and setting the iteration number of the initial network to be 1500; in order to train the initial network and minimize the segmentation loss function value of the initial network, an Adam optimizer can be adopted to optimize the initial network, and the parameters of the Adam optimizer are set to be beta ₁ ＝0.9，β ₂ =0.999, the initial learning rate is set to 0.01.

The segmentation loss function of the initial network can be composed of a Dice cross entropy loss function and a binary cross entropy loss function, and the Dice cross entropy loss function identifies the similarity degree between the outlines of two segmented tumor target areas in a brain tumor MRI image; but when both tumor target areas are small, the gradient changes dramatically and the training is difficult to converge. The binary cross entropy loss function calculates the logarithmic loss of each pixel in the image, and has a good effect on the segmentation details of the brain tumor MRI image with undefined boundary and uneven gray level. Through the weighted combination of the Dice cross entropy loss function and the binary cross entropy loss function, the different scale shapes of the tumor lesion areas in the MRI image are considered, the influence of the imaging quality (uneven gray scale, noise and the like) of the MRI image on the boundary segmentation of the tumor lesion areas is also considered, and the complete supervision training of the initial network for segmenting the labeled MRI image can be better realized. And in the continuous iterative training, the segmentation probability graph generated by the initial network is enabled to be closer to the label by minimizing the segmentation loss function value, so that the first network is obtained.

Step 504, training the generated countermeasure network by using the labeled training samples and the unlabeled training samples;

the first network and the second network are trained by using labeled training samples and unlabeled training samples, as shown in fig. 8, and fig. 8 is a schematic diagram of a network architecture for generating an anti-network provided by this example. In each iteration of the training stage, inputting the preprocessed image data into a first network, and inputting a segmentation probability graph output by the first network and the image data into a second network to obtain a quality score output by the second network; and determining a first loss value of the first network and a second loss value of the second network according to the difference between the segmentation probability map and the label and the quality score.

Wherein, the batch processing size is set to 16, and the iteration number is set to 1500; in order to train the first network and the second network and minimize the first loss value of the first network and the second loss value of the second network, an Adam optimizer can be adopted to optimize the initial network, and the parameter of the Adam optimizer is set to be beta ₁ ＝0.9，β ₂ =0.999, and the initial learning rates of the first network and the second network are both set to 0.01.

The method comprises the steps of utilizing a first network obtained by pre-training based on a labeled training sample as a generator in the generation of the countermeasure network, inputting the labeled training sample and an unlabeled training sample into the first network, and obtaining a segmentation probability map corresponding to the labeled training sample and a prediction segmentation probability map corresponding to the unlabeled training sample output by the first network. Then, using the second network as a discriminator in generating the countermeasure network; inputting the segmentation probability map, the labeled training samples and the unlabeled training samples into a second network to obtain a quality score output by the second network; the segmentation result object of the second network evaluation includes: the labels of the labeled training samples, and the predictive segmentation probability map output by the first network. And performing countermeasure learning between the first network and the second network by using the quality score output by the second network, wherein the second network can improve the evaluation capability of the segmentation quality through countermeasure training and in turn guides the first network to output a more accurate segmentation result.

For training of the second network, the second network requires two inputs, including: brain tumor MRI image and corresponding tumor segmentation result. For the training sample with the label, the path of the tumor segmentation result has two sources, one is the label of the tumor lesion area marked by the imaging doctor; the other is a segmentation probability map output by the first network. For unlabeled training samples, the path of the tumor segmentation result only contains the segmentation probability map output from the first network.

Based on the input image of the second network, a dual attention module (i.e., an image intensity attention module and a geometric boundary attention module) extracts dual attention features of the input image. And extracting deep features from the mixed features proposed by the dual attention module by an inclusion convolution module and a hole convolution module. Then, converting the deep-layer features into feature vectors through a global average pooling layer; and outputting a quality score according to the feature vector by utilizing the full-connection layer.

In this example, the loss function of the first network is composed of a segmentation loss function and an antagonistic loss function under complete supervision, the main function of the segmentation loss function is to learn the training samples of the labeled MRI image and determine the basic segmentation effect of the generated antagonistic network, and the antagonistic loss function establishes the antagonistic relationship between the segmentation probability map predicted by the first network and the real label, so that the distribution of the predicted segmentation probability maps of the labeled training samples and the unlabeled training samples is closer to the label. And the loss function of the second network can make the second network effectively identify the prediction segmentation probability map and the real label output by the first network.

And 505, inputting the MRI image to be segmented into the trained image segmentation network to obtain a segmentation probability map output by the image segmentation network, and determining a tumor lesion area in the MRI image based on the segmentation probability map.

An embodiment of the present invention further provides an electronic device, where the electronic device includes:

a memory for storing executable instructions;

and the processor is used for realizing the image processing method provided by one or more of the technical schemes when executing the executable instructions stored in the memory.

Wherein the memory may include: various types of storage media may be used for data storage. In this embodiment, the storage medium included in the memory is at least partially a non-volatile storage medium, and may be used to store the computer program.

The processor may include: a central processing unit, a microprocessor, a digital signal processor, an application specific integrated circuit or a programmable array, etc., may be used for the image processing method according to one or more of the preceding claims by a computer program.

In this embodiment, the processor may be connected to the memory via an intra-device bus, such as an integrated circuit bus.

a memory for storing executable instructions;

and the processor is used for implementing the training method of the image segmentation network provided by one or more of the technical solutions when the processor executes the executable instructions stored in the memory.

An embodiment of the present invention further provides a computer storage medium, where a computer program is stored, and after the computer program is executed by a processor, the computer program executes an image processing method provided by one or more of the foregoing technical solutions, for example, the method shown in fig. 1 may be executed.

An embodiment of the present invention further provides a computer storage medium, where a computer program is stored, and after the computer program is executed by a processor, the computer program executes a training method for an image segmentation network provided in one or more of the foregoing technical solutions, for example, the method shown in fig. 2 may be executed.

The computer storage medium provided by the embodiment of the invention comprises: a mobile storage device, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes. Alternatively, the computer storage medium may be a non-transitory storage medium. The non-transitory storage medium herein may also be referred to as a non-volatile storage medium.

The above description is only an example of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present invention shall fall within the protection scope of the present invention.

Claims

1. An image processing method, characterized by comprising:

and segmenting a tumor lesion area from the MRI image based on the second image characteristic, wherein the tumor lesion area is an area where pixels of tumor lesion tissue are located.

2. The method of claim 1, wherein the inclusion convolution module has N inclusion convolution layers, the convolution kernels of different ones of the inclusion convolution layers being different; different inclusion convolution layers are used for extracting the first image features with different scales.

3. The method of claim 2, wherein the hole convolution module comprises: n said hole convolution dense sub-modules;

4. The method according to any one of claims 1 to 3, wherein the performing the hole convolution on the first image feature by using a hole convolution module of the image segmentation network to obtain the second image feature comprises:

carrying out batch normalization processing on the cavity convolution result to obtain a normalization processing result;

5. The method of claim 1, further comprising:

and carrying out increment convolution on the standard convolution result based on different convolution kernels by utilizing the increment convolution module to obtain a plurality of first image characteristics with different scales.

6. A training method for an image segmentation network is characterized by comprising the following steps:

obtaining a first segmentation quality score of the prediction segmentation information of the first network through a second network based on the prediction segmentation information of the labeled training samples;

determining a second loss value for the second network based on the first segmentation quality score and the labels of the labeled training samples;

7. The method of claim 6, further comprising:

determining, by the computing device, a first loss value of the first network according to a difference between the predicted segmentation information of the labeled training samples and the label, including:

8. The method of claim 7, wherein the deriving the first loss value according to the difference between the label and the predictive segmentation information of the labeled training sample, the first segmentation quality score, and the second segmentation quality score comprises:

9. The method of claim 8, wherein determining a first loss value for the first network based on a difference between the predicted segmentation information and the label for the labeled training samples, a first segmentation quality score, and a second segmentation quality score comprises:

10. The method of claim 6, wherein determining a second loss value for the second network based on the first segmentation quality score and the label of the labeled training sample comprises:

11. The method of claim 10, wherein determining a second loss value for the second network based on the first segmentation quality score, the labels of the labeled training samples, and the second segmentation quality score comprises:

12. The method of claim 6, further comprising:

13. The method of claim 12, wherein determining the segmentation loss function value for the initial network based on the difference between the predicted segmentation information and the annotation information comprises:

respectively determining binary cross entropy and Dice cross entropy of the prediction partition information and the labeling information based on the prediction partition information and the labeling information;

14. The method of claim 6, wherein obtaining, by the second network, a segmentation quality score for the predicted segmentation information of the first network based on the predicted segmentation information of the labeled training samples comprises:

inputting the predicted segmentation information and the labeled training sample into a second network, and processing the predicted segmentation information through an image intensity attention module and a geometric boundary attention module of the second network to obtain an intensity information difference characteristic between a tumor lesion region and a background region of an MRI image of the labeled training sample, a contour characteristic of the tumor lesion region and a contour characteristic of the background region;

15. The method of claim 14, wherein the processing the predictive segmentation information by the image intensity attention module and the geometric boundary attention module of the second network to obtain an intensity information difference feature between a tumor lesion region and a background region, a contour feature of the tumor lesion region and a contour feature of the background region of the MRI image of the labeled training sample comprises:

16. The method of claim 14, wherein determining a segmentation quality score corresponding to the predicted segmentation information according to the intensity signal difference feature, the contour feature of the tumor lesion region, and the contour feature of the background region comprises:

17. The method of claim 16, wherein determining a segmentation quality score corresponding to the predicted segmentation information based on the dual attention feature comprises:

18. The method of claim 6, further comprising:

inputting the enhanced MRI image into the first network.

19. An image processing apparatus characterized by comprising:

the first convolution module is used for carrying out increment convolution on the MRI image by utilizing an increment convolution module of the image segmentation network to obtain a first image characteristic;

and the segmentation module is used for segmenting a tumor lesion region from the MRI image based on the second image characteristic, wherein the tumor lesion region is a region where pixels of tumor lesion tissue imaging are located.

20. An apparatus for training an image segmentation network, comprising:

the acquisition module is used for acquiring a training sample set of the MRI image; wherein the training sample set comprises: labeled training samples and unlabeled training samples;

the second network module is used for obtaining a first segmentation quality score of the prediction segmentation information of the first network through a second network based on the prediction segmentation information of the labeled training sample;

a loss determining module, configured to determine a first loss value of the first network according to a difference between the predicted segmentation information of the labeled training sample and the label; determining a second loss value for the second network based on the first segmentation quality score and labels of the labeled training samples; determining whether to continue training the first network based on the first loss value and the second loss value.

21. An electronic device, comprising:

a memory for storing executable instructions;

a processor for implementing the image processing method of any one of claims 1 to 5 when executing executable instructions stored in the memory.

22. An electronic device, comprising:

a memory for storing executable instructions;

a processor for implementing the method of training an image segmentation network according to any one of claims 6 to 18 when executing executable instructions stored in the memory.

23. A computer-readable storage medium, characterized in that it stores executable instructions which, when executed by a processor, implement the image processing method according to any one of claims 1 to 5.

24. A computer-readable storage medium storing executable instructions that, when executed by a processor, implement a method of training an image segmentation network according to any one of claims 6 to 18.