CN114445423A

CN114445423A - Medical image segmentation method based on weak supervised learning and training method of model thereof

Info

Publication number: CN114445423A
Application number: CN202210041649.2A
Authority: CN
Inventors: 何良华; 王方鑫; 李聪
Original assignee: AVCON INFORMATION TECHNOLOGY CO LTD; Tongji University
Current assignee: AVCON INFORMATION TECHNOLOGY CO LTD; Tongji University
Priority date: 2022-01-14
Filing date: 2022-01-14
Publication date: 2022-05-06

Abstract

The invention provides a medical image segmentation method based on weak supervised learning and a training method of a model thereof. The training method comprises the following steps: acquiring a plurality of medical images to be segmented, and labeling focus areas in the medical images; preprocessing each medical image to be segmented, and dividing the preprocessed image data into a training set and a test set according to a preset proportion; training a weak supervision seed clue extraction model by using the training set, and training a weak supervision image segmentation model by using pseudo labels generated by the training set and the weak supervision seed clue extraction model; and after the training is finished, testing the trained weak supervision image segmentation model by using the test set. The model obtained by training can carry out segmentation on the medical image according to the lesion risk degree, so that the requirement of an actual medical segmentation scene is met.

Description

Medical image segmentation method based on weak supervised learning and training method of model thereof

Technical Field

The invention relates to the field of medical image segmentation, in particular to a medical image segmentation method based on weak supervised learning and a training method of a model thereof.

Background

In recent years, with the continuous development of medical imaging technology, artificial intelligence is increasingly applied to the field of medical image analysis, and doctors can be helped to perform some auxiliary medical functions, so that the doctors can better perform surgical treatment on patients.

At present, although image segmentation based on full supervision obtains a better segmentation result, because the task of labeling medical image data is complex, the task of labeling pixels one by one is large, and the medical image data also needs to have knowledge in the field of professional medicine, the full supervision cannot play a good role, and a new solution still needs to be continuously explored.

Disclosure of Invention

In view of the above-mentioned shortcomings of the prior art, the present invention aims to provide a medical image segmentation method based on weak supervised learning and a training method of a model thereof, which can obtain a more ideal segmentation result under the condition of using weak labels at the image level by performing image segmentation through weak supervision. In addition, the problem of segmentation precision is considered, the risk degree of segmentation is also considered, and segmentation results under different conditions can be given, so that the defects in the prior art are overcome.

In order to achieve the above objects and other related objects, the present invention provides a method for training a medical image segmentation model based on weak supervised learning, comprising the following steps:

acquiring a plurality of medical images to be segmented, and labeling focus areas in the medical images;

preprocessing each medical image to be segmented, and dividing the preprocessed image data into a training set and a test set according to a preset proportion;

training a weak supervision seed clue extraction model by using the training set, and training a weak supervision image segmentation model by using pseudo labels generated by the training set and the weak supervision seed clue extraction model;

and after the training is finished, testing the trained weak supervision image segmentation model by using the test set.

In order to achieve the above objects and other related objects, the present invention provides a medical image segmentation method based on weak supervised learning, which utilizes a medical image segmentation model based on weak supervised learning obtained by the training method; the medical image segmentation method comprises the following steps:

acquiring the focus risk degree of a medical image to be segmented;

and segmenting the medical image to be segmented by utilizing the medical image segmentation model based on weak supervised learning so as to obtain a focus region image corresponding to the focus risk degree from the medical image to be segmented.

To achieve the above and other related objects, the present invention provides a computer-readable storage medium, in which a computer program is stored, which, when being loaded and executed by a processor, implements any one of the above-mentioned methods.

To achieve the above and other related objects, the present invention provides an electronic device, comprising: a processor and a memory; wherein the memory is for storing a computer program; the processor is configured to load and execute the computer program, so as to cause the electronic device to perform any one of the methods described above.

As described above, the medical image segmentation method based on weak supervised learning and the training method of the model thereof of the present invention have the following beneficial effects: based on the simple and easily-obtained weak label, the medical image is segmented by using a weak supervision method, and a certain segmentation effect is achieved. In addition, for some special lesions, such as extremity melanoma, it is necessary to give a corresponding segmentation result according to the risk degree of the melanoma, because only the segmentation accuracy cannot be considered. Therefore, the invention introduces the risk degree in the weak supervision medical image segmentation, and realizes accurate segmentation by comprehensively considering the influence of various factors.

Drawings

Fig. 1 is a flowchart of a training method of a medical image segmentation model based on weak supervised learning in an embodiment of the present invention.

Fig. 2 is a schematic structural diagram of a weakly supervised seed clue extraction model in an embodiment of the present invention.

Fig. 3 is a schematic flow chart of segmenting a medical image according to an embodiment of the present invention.

Fig. 4 is a schematic structural diagram of a medical image segmentation model based on weak supervised learning in an embodiment of the present invention.

Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the invention.

Detailed Description

The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.

It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention, and the drawings only show the components related to the present invention rather than the number, shape and size of the components in actual implementation, and the type, quantity and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.

As shown in fig. 1, the present embodiment provides a method for training a medical image segmentation model based on weak supervised learning, including the following steps:

s1, acquiring a plurality of medical images to be segmented, and labeling the focus areas;

marking a focus region of the medical image to be segmented, wherein a non-focus pixel point in the medical image to be segmented can be marked by 0; and marking focus pixel points in the medical image to be segmented by using 1.

Taking melanoma segmentation as an example, the melanoma image data used for the experiment was selected from ISIC skin pathology image segmentation and classification games of MICCAI tissue at the medical image top conference, the ISIC archive containing more than 23,000 skin lesion images, labeled "benign" or "malignant". Each example contains an image of a lesion, metadata about the lesion (including classification and segmentation), and metadata about the patient. Each picture is a 600x450 size three channel RGB image. The mask image and the corresponding lesion image have the same size, and the mask image is a single-channel (grayscale) 8-bit lossless image, where 0 represents the background of the image, or the region outside the primary lesion, and 255 represents the foreground of the image, or the region inside the primary lesion.

S2, preprocessing each medical image to be segmented, and dividing the preprocessed image data into a training set and a test set according to a preset proportion;

wherein, the medical image to be segmented is preprocessed, which comprises the following steps: and resampling each medical image to be segmented so as to keep the size and the resolution of each medical image to be segmented the same. For example, in the data preprocessing process, to ensure that the images have the same size and resolution, the spatial resolution of the images of the training data set may be uniformly resampled to 512 × 512, the preset ratio of the training set to the test set is set to 8:2, and so on.

S3, training a weak supervision seed clue extraction model by using the training set, and training a weak supervision image segmentation model by using pseudo labels generated by the training set and the weak supervision seed clue extraction model;

referring to fig. 2, in the present embodiment, the structure of the weakly supervised seed clue extraction model includes the following main parts:

1) a thermodynamic diagram generation module; a backbone network of the thermodynamic diagram generation module adopts a classical classification network Vgg network and a CAM module, wherein a feature diagram output by the last layer of convolution of the Vgg network is input to a full connection layer after being subjected to global average pooling; the CAM module is used for generating a thermodynamic diagram of the medical image, as shown in FIG. 3;

2) a pseudo tag generation module; the pseudo label generating module generates a corresponding pseudo label (seed) according to the focus risk degree of the medical image; wherein, the lower the focus risk degree, the more concentrated the pixel point of the core region of the thermodynamic diagram will be marked as 1, the rest regions are marked as 0, thus, the focus core part is segmented as much as possible, and the rest parts are kept as much as possible.

In the case of melanoma, from an operative point of view, acro-melanoma surgery is not only considered to be clean of the tumor, but also to be fully considered to preserve the function as much as possible, especially the finger function. The pseudo tag generation module of this embodiment gives a segmentation result of the corresponding melanoma according to the melanoma risk degree (such as mild, moderate, severe, etc.) determined by the doctor, and if the determination result given by the doctor is mild, the segmentation range should be narrowed to a core region of the thermodynamic diagram, such as a red region; if the judgment result given by the doctor is serious, the segmentation range of the thermodynamic diagram is enlarged. Each pixel in the segmentation region is marked as 1, and each pixel outside the segmentation region is marked as 0, so that a pseudo label of the medical image is generated, namely the pseudo label is formed by the marks of all pixel points of the medical image.

In the embodiment, different risk degrees of the lesion are divided by a self-defined threshold in advance, and the higher the risk degree is, the lower the set threshold should be, so as to expand the division range and completely divide the lesion area; similarly, the lower the risk degree, the higher the threshold should be set, so as to narrow the segmentation range, to segment out a smaller region of the lesion, and to reserve other functional regions as much as possible.

For example, the threshold expressions of various risk degrees corresponding to the pixel point marked with 1 in the pseudo label are as follows:

wherein, delta represents a threshold value for dividing focus and non-focus areas according to a thermal value, and the threshold value delta_l<δ_m<δ_h，M_A,(i,j)Indicating the thermal value, label, of the (i, j) location in image A_A,(i,j)A label of 1 indicates that the pixel point located at the (i, j) position of image a is 1, and the label is labeled 1 to indicate that the point (i, j) position is a lesion.

Optionally, in the process of training the weakly supervised seed clue extraction model, the Adam algorithm is adopted to optimize the loss function, the size of the batch size can be set to 8, the epoch can be set to 200, the learning rate can be set to 0.0001, and the like, so that a better training effect can be achieved.

And S4, after the training is finished, testing the trained weak supervision image segmentation model by using the test set.

In the training process, if the value of the loss function loss reaches convergence or the training frequency reaches a preset value, the training is finished. Preferably, the loss function can adopt a loss function optimized by an Adam algorithm.

Referring to fig. 4, in the present embodiment, the structure of the weakly supervised image segmentation model includes the following parts:

an encoding module; the coding module comprises the following components which are connected in sequence: a DCNN network with hole convolution, a ResNet classification network, a Spatial Pyramid module with hole convolution (atmospheric Spatial Pyramid Powing), and a convolution dimensionality reduction module (e.g., 1 × 1 convolution);

a decoding module; and the decoding module performs upsampling (such as 4 times upsampling) on the result of the coding module, connects the upsampling with the result of the downsampling of the ResNet classification network, performs convolution processing (such as 3-by-3 convolution) and then performs upsampling (such as 4 times upsampling) so as to reduce the difference between the segmentation result of the weakly supervised seed clue extraction model and the real segmentation result.

In addition, the present application also provides a medical image segmentation method based on weak supervised learning, which performs segmentation of medical images by using a medical image segmentation model based on weak supervised learning obtained by training with the above training method, and mainly includes the following steps:

firstly, acquiring the focus risk degree of a medical image to be segmented;

then, the medical image to be segmented is segmented by using the medical image segmentation model based on weak supervised learning, so as to obtain a lesion region image corresponding to the lesion risk degree from the medical image to be segmented.

The medical image segmentation model based on the weak supervised learning can perform corresponding segmentation according to the focus risk degree of the medical image, and if the focus risk degree is higher, the segmentation range is larger, so that a focus area is segmented as much as possible; if the lesion risk degree is low, the segmentation range is small, so that a small region of the lesion is segmented, and the region is a core region of the lesion, so that other functional regions are reserved as much as possible.

In summary, the present invention is based on the requirements of the actual medical segmentation, and by using the image-level weak labels and considering the preservation of the limb functions, segmentation is performed at different degrees according to the lesion risk degree, thereby realizing accurate and appropriate lesion image region segmentation.

To verify the validity of the segmentation model proposed by the present invention, the segmentation results are evaluated by mean intersection ratio (mIOU) and Dice coefficient values as follows.

As shown in table 1, the average cross-over ratio and the value of the Dice coefficient were calculated for 260 melanoma images, respectively. It can be seen that the average intersection ratio value of the current model is 0.6262, and the value of the Dice coefficient is 0.7701. The invention obtains more accurate segmentation result under the condition of using weak labels.

TABLE 1 weakly supervised medical image segmentation results

All or part of the steps for implementing the above method embodiments may be performed by hardware associated with a computer program. Based upon such an understanding, the present invention also provides a computer program product comprising one or more computer instructions. The computer instructions may be stored in a computer readable storage medium. The computer-readable storage medium can be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

As shown in fig. 5, the present application further provides an electronic device, which is a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. for executing the method described in the foregoing embodiment.

The electronic device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention. As shown in FIG. 5, the electronic device is in the form of a general purpose computing device, the components of which may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.

Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

The electronic device typically includes a variety of computer system readable media. Such media may be any available media that is accessible by the electronic device and includes both volatile and nonvolatile media, removable and non-removable media.

The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. The electronic device may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, and commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.

The electronic device may also communicate with one or more external devices 14 (e.g., keyboard, speakers, display 24, etc.), with one or more devices that enable a user to interact with the electronic device, and/or with any devices (e.g., network card, modem, etc.) that enable the electronic device to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, the electronic device may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 20. As shown in FIG. 5, the network adapter 20 communicates with other modules of the electronic device via the bus 18. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system. Furthermore, in some embodiments, aspects of the invention may also be embodied in the form of a computer program product in one or more computer-readable media having computer-readable program code embodied therein.

Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

It should be understood that the methods of the present invention can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

In conclusion, the medical image segmentation method based on the weak supervised learning and the training method of the model thereof effectively overcome various defects in the prior art and have high industrial utilization value.

The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.

Claims

1. A training method of a medical image segmentation model based on weak supervised learning is characterized by comprising the following steps:

2. The method according to claim 1, wherein labeling a lesion region of the medical image to be segmented comprises: marking non-focus pixel points in the medical image to be segmented by 0; and marking focus pixel points in the medical image to be segmented by using 1.

3. The method according to claim 1, wherein preprocessing the medical image to be segmented comprises: and resampling each medical image to be segmented so as to keep the size and the resolution of each medical image to be segmented the same.

4. The method of claim 1, wherein the structure of the weakly supervised seed cue extraction model comprises:

a thermodynamic diagram generation module; a Vgg network and a CAM module are adopted by a main network of the thermodynamic diagram generation module; the characteristic diagram output by the last layer of convolution of the Vgg network is input to a full connection layer after being subjected to global average pooling; the CAM module is used for generating a thermodynamic diagram of the medical image;

a pseudo tag generation module; the pseudo label generating module generates a corresponding pseudo label according to the lesion risk degree of the medical image; the lower the lesion risk degree is, the more concentrated the pixel points in the core region of the thermodynamic diagram are marked as 1, the rest regions are marked as 0, and the pseudo label is a marked set of all the pixel points of the medical image.

5. The method of claim 4, wherein the structure of the weakly supervised image segmentation model comprises:

an encoding module; the coding module comprises the following components which are connected in sequence: the system comprises a DCNN (distributed computing network) with cavity convolution, a ResNet classification network, a space pyramid module with cavity convolution and a convolution dimension reduction module;

a decoding module; and the decoding module performs up-sampling on the result of the coding module, connects the result with the result of the ResNet classification network down-sampling, performs convolution processing and then performs up-sampling so as to reduce the difference between the segmentation result of the weakly supervised seed clue extraction model and the real segmentation result.

6. The method of claim 1, wherein the training is terminated when the value of the loss function converges or when the number of training passes reaches a predetermined value.

7. The method of claim 6, wherein the loss function is optimized by Adam algorithm.

8. A medical image segmentation method based on weak supervised learning is characterized in that a medical image segmentation model based on weak supervised learning is obtained by training through the training method of any one of claims 1 to 7; the medical image segmentation method comprises the following steps:

acquiring the focus risk degree of a medical image to be segmented;

9. A computer-readable storage medium, in which a computer program is stored which, when loaded and executed by a processor, carries out the method according to any one of claims 1 to 8.

10. An electronic device, comprising: a processor and a memory; wherein the content of the first and second substances,

the memory is used for storing a computer program;

the processor is configured to load and execute the computer program to cause the electronic device to perform the method according to any one of claims 1 to 8.