CN114612451A - Multi-organ segmentation method and system based on multi-source integrated distillation - Google Patents

Multi-organ segmentation method and system based on multi-source integrated distillation Download PDF

Info

Publication number
CN114612451A
CN114612451A CN202210268374.6A CN202210268374A CN114612451A CN 114612451 A CN114612451 A CN 114612451A CN 202210268374 A CN202210268374 A CN 202210268374A CN 114612451 A CN114612451 A CN 114612451A
Authority
CN
China
Prior art keywords
organ
teacher
model
segmentation
prediction
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210268374.6A
Other languages
Chinese (zh)
Inventor
王延峰
张乐飞
冯世祥
王钰
张娅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Jiaotong University
Original Assignee
Shanghai Jiaotong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Jiaotong University filed Critical Shanghai Jiaotong University
Priority to CN202210268374.6A priority Critical patent/CN114612451A/en
Publication of CN114612451A publication Critical patent/CN114612451A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Epidemiology (AREA)
  • Primary Health Care (AREA)
  • Public Health (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides a multi-organ segmentation method and a multi-organ segmentation system based on multi-source integrated distillation, which comprise the following steps: inputting pictures, and predicting by the teacher model and the student model to obtain a multi-component segmentation graph; performing output conversion on the teacher picture segmentation picture, and expanding the teacher picture segmentation picture to multi-class output; performing organ and background region conversion on the output according to the region-based mask; respectively performing supervised learning on different organ areas and background areas to align the converted prediction results of the teacher model and the student models so as to obtain an efficient student multi-organ segmentation model; and inputting the prediction picture, and obtaining an organ segmentation prediction result through a student multi-organ segmentation model. According to the invention, a multi-organ segmentation model with better effect is guided and trained by a multi-source integrated distillation method, so that more accurate unsupervised multi-organ segmentation under the condition of privacy friendliness is realized.

Description

Multi-organ segmentation method and system based on multi-source integrated distillation
Technical Field
The invention relates to the technical field of computer vision, image processing and computer-aided medical diagnosis, in particular to a multi-organ segmentation method and system based on multi-source integrated distillation.
Background
Organ segmentation is one of the basic tasks of computer-aided medical diagnosis, and plays a crucial role in disease discovery and subsequent treatment. The purpose of medical image segmentation is to extract an anatomical region of interest from a medical image in an automatic or semi-automatic manner. Depending on the clinical application, the specific region of interest may range from a tumor to bone to blood vessels. In recent years, the vigorous development of deep learning also brings new ideas and breakthroughs to medical image processing. Although deep learning methods have been widely used for this task, their success depends largely on high quality labeling, however, medical labeling is costly and difficult to obtain in practice. Furthermore, physicians often label only certain organs of interest and that are good at themselves. Therefore, the existing main research focuses on single-organ segmentation models, and the multi-organ segmentation models are short of exploration. The multi-organ segmentation model can reduce the calculation and storage overhead and can also utilize a priori knowledge between organs. In order to create a multi-organ data set, multiple organs in a single image need to be annotated, which requires the cooperation of multiple experts in different fields, greatly increases the difficulty and cost of annotation, causes the lack of the multi-organ annotated data set, and limits the realization of a multi-organ segmentation model.
The success of depth learning based semantic segmentation of medical images also depends on the availability of medical image data. However, organizations are reluctant to share image data for various reasons, most notably for privacy protection. Strict health information privacy regulations require that protected health information needs to be deleted before data is shared outside the home establishment, a process that is both expensive and time consuming in the context of medical image data. Organizations that issue health information, either intentionally or unintentionally, may suffer serious consequences, including a large penalty and criminal prosecution. Furthermore, in many cases, when imaging data is publicly accessible (e.g., cancer imaging data), the corresponding tags are rarely available due to the lack of expert annotation of the target area and the lack of information on the equipment and criteria of the annotation. In practice, there are many reasons why such data is not available, including data storage and transmission, limited number of cases, and difficulty in data cleaning and labeling. Therefore, it is very difficult to train an efficient medical model in such a situation. The problems of multi-organ labeling loss and data privacy provide new requirements for the modeling method.
Aiming at the defects in the prior art, a new technical scheme needs to be provided.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a multi-organ segmentation method and system based on multi-source integrated distillation.
According to the invention, the multi-organ segmentation method based on multi-source integrated distillation comprises the following steps:
step S1: inputting pictures, and predicting by a teacher model and a student model to obtain a multi-component segmentation picture;
step S2: performing output conversion on the teacher picture segmentation picture, and expanding the teacher picture segmentation picture to multi-class output;
step S3: performing organ and background region conversion on the output according to the region-based mask;
step S4: respectively performing supervised learning on different organ areas and background areas, and aligning converted prediction results of a teacher model and a student model to obtain a student multi-organ segmentation model;
step S5: and inputting the prediction picture to obtain an organ segmentation prediction result through a student multi-organ segmentation model.
Preferably, in step S1, the teacher single-organ neural network segmentation model composed of the convolutional layer and the full-link layer is used to predict the input picture, and the formula is as follows:
Figure BDA0003553361570000021
where x is the input picture, the superscript j indicates the jth dimension, fsmIs the mth teacher neural network model, δkRepresenting the softmax prediction output in the k-th dimension,
Figure BDA0003553361570000022
is the prediction probability of the kth dimension obtained through the mth teacher neural network model;
the input picture is predicted by utilizing a student multi-organ neural network segmentation model formed by the convolutional layer and the full connecting layer, and the formula is as follows:
Figure BDA0003553361570000023
wherein f istIs a neural network model of a student,
Figure BDA0003553361570000024
is the predicted probability of the kth dimension obtained by the student neural network model.
Preferably, the step S2 includes:
preserving background prediction of teacher segmentation model for organ regions
Figure BDA0003553361570000025
Without change, the prediction of the organ numbered m
Figure BDA0003553361570000026
Move to the m-th dimension and then set the values of all other dimensions to 0, the formula is as follows:
Figure BDA0003553361570000027
wherein the content of the first and second substances,
Figure BDA0003553361570000031
is a teacher model inputOutputting the organ with the number m after the conversion mapping;
for the background region, the predicted values of all M teacher single organ segmentation models are averaged, and the formula is as follows:
Figure BDA0003553361570000032
wherein the content of the first and second substances,
Figure BDA0003553361570000033
the teacher model outputs the transformed and mapped output to the background.
Preferably, the step S3 includes:
a prediction mask for organ m is obtained, the formula is as follows:
Figure BDA0003553361570000034
and converting the prediction of the teacher single-organ segmentation model on the organ region, wherein the formula is as follows:
Figure BDA0003553361570000035
transforming the predictions of the student multi-organ segmentation model for the organ regions, the formula is as follows:
Figure BDA0003553361570000036
obtaining the mth teacher single organ segmentation model
Figure BDA0003553361570000037
For the mask of the background, the formula is as follows:
Figure BDA0003553361570000038
aggregating the masks of all M teacher single organ segmentation models for the background, the formula is as follows:
Figure BDA0003553361570000039
converting the prediction of the teacher single organ segmentation model for the background area, wherein the formula is as follows:
Figure BDA00035533615700000310
converting the prediction of the student multi-organ segmentation model for the background area, wherein the formula is as follows:
Figure BDA00035533615700000311
preferably, the step S4 sets different weights for different organ areas and background areas, and aligns the converted prediction results of the teacher and student models, as follows:
Figure BDA00035533615700000312
where λ is a hyper-parameter used to balance the weights of the organ region and the background region.
Preferably, the step S5 inputs the prediction picture to be predicted by the student multi-organ segmentation model, and the formula is as follows:
p＝δ(ft(x)) (13)
where p is the resulting multi-organ segmentation result.
The invention also provides a multi-organ segmentation system based on multi-source integrated distillation, which comprises the following modules:
module M1: inputting pictures, and predicting by a teacher model and a student model to obtain a multi-component segmentation picture;
module M2: performing output conversion on the teacher picture segmentation picture, and expanding the teacher picture segmentation picture to multi-class output;
module M3: performing organ and background region conversion on the output according to the region-based mask;
module M4: respectively performing supervised learning on different organ areas and background areas, and aligning converted prediction results of a teacher model and a student model to obtain a student multi-organ segmentation model;
module M5: and inputting the prediction picture to obtain an organ segmentation prediction result through a student multi-organ segmentation model.
Preferably, the module M1 predicts the input picture by using a teacher single-organ neural network segmentation model composed of convolutional layers and full-link layers, and the formula is as follows:
Figure BDA0003553361570000041
where x is the input picture, the superscript j indicates the jth dimension,
Figure BDA0003553361570000042
is the mth teacher neural network model, δkRepresenting the softmax prediction output in the k-th dimension,
Figure BDA0003553361570000043
is the prediction probability of the kth dimension obtained through the mth teacher neural network model;
the input picture is predicted by utilizing a student multi-organ neural network segmentation model formed by the convolutional layer and the full connecting layer, and the formula is as follows:
Figure BDA0003553361570000044
wherein f istIs a neural network model for a student,
Figure BDA0003553361570000045
is the predicted probability of the kth dimension obtained by the student neural network model;
the module M2 includes:
for organ regions, keeping a background prediction of teacher segmentation model
Figure BDA0003553361570000046
Without change, the prediction of the organ numbered m
Figure BDA0003553361570000047
Move to the m-th dimension and then set the values of all other dimensions to 0, the formula is as follows:
Figure BDA0003553361570000048
wherein the content of the first and second substances,
Figure BDA0003553361570000049
outputting the organ with the number of m after the teacher model outputs the conversion mapping;
for the background region, the predicted values of all M teacher single organ segmentation models are averaged, and the formula is as follows:
Figure BDA00035533615700000410
wherein the content of the first and second substances,
Figure BDA00035533615700000411
the teacher model outputs the transformed and mapped output to the background.
Preferably, said module M3 comprises:
a prediction mask for organ m is obtained, the formula is as follows:
Figure BDA00035533615700000412
and converting the prediction of the teacher single-organ segmentation model on the organ region, wherein the formula is as follows:
Figure BDA00035533615700000413
transforming the predictions of the student multi-organ segmentation model for the organ regions, the formula is as follows:
Figure BDA0003553361570000051
obtaining the mth teacher single organ segmentation model
Figure BDA0003553361570000052
For the mask of the background, the formula is as follows:
Figure BDA0003553361570000053
aggregating the masks of all M teacher single organ segmentation models for the background, the formula is as follows:
Figure BDA0003553361570000054
converting the prediction of the teacher single organ segmentation model on the background area, wherein the formula is as follows:
Figure BDA0003553361570000055
converting the prediction of the student multi-organ segmentation model for the background area, wherein the formula is as follows:
Figure BDA0003553361570000056
preferably, the module M4 sets different weights for different organ and background regions, aligning the transformed prediction results of the teacher and student models, as follows:
Figure BDA0003553361570000057
wherein λ is a hyper-parameter for balancing weights of the organ region and the background region;
the module M5 inputs a prediction picture to be predicted by a student multi-organ segmentation model, and the formula is as follows:
p＝δ(ft(x)) (13)
where p is the resulting multi-organ segmentation result.
Compared with the prior art, the invention has the following beneficial effects:
1. the invention provides a privacy-friendly multi-organ segmentation method and system based on multi-source integrated distillation, which can relieve the problems of multi-organ labeling loss and data privacy to a certain extent. Under the condition of lacking original training data, a plurality of pre-training single organ segmentation models aiming at different organs and a target domain data set without a label are utilized to obtain an efficient multi-organ segmentation model through training, the labeling requirement on the training data is reduced, and meanwhile, the original training data is not needed;
2. the multi-organ segmentation model obtained by training of the invention aggregates knowledge of a plurality of pre-trained single-organ segmentation models, and compared with a single-organ segmentation model and other multi-organ segmentation models, the multi-organ segmentation model further improves the segmentation accuracy of each organ, and simultaneously saves a large amount of calculation time consumption compared with the single-organ segmentation model;
3. the multi-organ segmentation model is respectively studied and trained aiming at different organs and background areas, and the proportion of the organs and the background areas is controlled through hyper-parameters, so that the multi-organ segmentation model can simultaneously attach importance to the organs and the background areas, the study on the organs and the background areas is mutually assisted and mutually constrained, and the segmentation performance of the organs is jointly improved;
4. the multi-organ segmentation model obtained by training utilizes the input of the target domain for training, so that the model can adapt to the prediction distribution and the environment, and the performance is better.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a flow chart of a method in an embodiment of the present invention;
fig. 2 is a schematic diagram of a system in an embodiment of the invention.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications can be made by persons skilled in the art without departing from the concept of the invention. All falling within the scope of the invention.
The invention aims to provide a multi-organ segmentation method and system based on multi-source integrated distillation, which can train a model capable of accurately segmenting multiple organs according to a scene only having a single-organ pre-training model without original training data, and help data modeling in a practical medical scene.
The invention provides an embodiment, a multi-organ segmentation method based on multi-source integrated distillation, which comprises the following steps:
step S1: and inputting the pictures, and predicting by the teacher model and the student model to obtain a multi-component segmentation graph. The input picture is predicted by the teacher and student models to obtain a multi-component segmentation picture, and the input picture is predicted by a teacher single-organ neural network segmentation model formed by a convolutional layer and a full-link layer, wherein the formula is as follows:
Figure BDA0003553361570000061
where x is the input picture, the superscript j indicates the jth dimension,
Figure BDA0003553361570000062
is the mth teacher neural network model, δkRepresenting the softmax prediction output in the k-th dimension,
Figure BDA0003553361570000063
is the predicted probability of the kth dimension obtained through the mth teacher neural network model.
The input picture is predicted by utilizing a student multi-organ neural network segmentation model formed by the convolutional layer and the full connecting layer, and the formula is as follows:
Figure BDA0003553361570000064
wherein f istIs a neural network model of a student,
Figure BDA0003553361570000065
is the predicted probability of the kth dimension obtained by the student neural network model.
Step S2: and performing output conversion on the teacher picture segmentation picture, and expanding the teacher picture segmentation picture to multi-class output. The method for performing output conversion on the teacher picture segmentation picture and expanding the teacher picture segmentation picture to multi-class output comprises the following steps: preserving background prediction of teacher segmentation model for organ regions
Figure BDA0003553361570000071
Without change, the prediction of the organ numbered m
Figure BDA0003553361570000072
Move to the m-th dimension and then set the values of all other dimensions to 0, the formula is as follows:
Figure BDA0003553361570000073
wherein the content of the first and second substances,
Figure BDA0003553361570000074
the teacher model outputs the transformed and mapped output to the organ numbered m.
For the background region, the predicted values of all M teacher single organ segmentation models are averaged, and the formula is as follows:
Figure BDA0003553361570000075
wherein the content of the first and second substances,
Figure BDA0003553361570000076
the teacher model outputs the transformed and mapped output to the background.
Step S3: the organ and background regions are transformed on the output according to the region-based mask. The transforming the output into organ and background regions according to the region-based mask includes: a prediction mask for organ m is obtained, the formula is as follows:
Figure BDA0003553361570000077
and converting the prediction of the teacher single-organ segmentation model on the organ region, wherein the formula is as follows:
Figure BDA0003553361570000078
transforming the predictions of the student multi-organ segmentation model for the organ regions, the formula is as follows:
Figure BDA0003553361570000079
obtaining the mth teacher single organ segmentation model
Figure BDA00035533615700000710
For the mask of the background, the formula is as follows:
Figure BDA00035533615700000711
aggregating the masks of all M teacher single organ segmentation models for the background, the formula is as follows:
Figure BDA00035533615700000712
converting the prediction of the teacher single organ segmentation model for the background area, wherein the formula is as follows:
Figure BDA00035533615700000713
converting the prediction of the student multi-organ segmentation model for the background area, wherein the formula is as follows:
Figure BDA00035533615700000714
step S4: and respectively performing supervised learning on different organ areas and background areas, so that converted prediction results of the teacher model and the student models are aligned, and an efficient student multi-organ segmentation model is obtained. Respectively performing supervised learning on different organ areas and background areas to ensure that the converted prediction results of a teacher model and a student model are aligned, and obtaining an efficient student multi-organ segmentation model comprises the following steps: setting different weights for different organ areas and background areas, and aligning the converted prediction results of the teacher model and the student model, wherein the formula is as follows:
Figure BDA0003553361570000081
where λ is a hyper-parameter used to balance the weights of the organ region and the background region.
Step S5: and inputting the prediction picture to obtain an organ segmentation prediction result through a student multi-organ segmentation model. Inputting a prediction picture through a student multi-organ segmentation model, and obtaining an organ segmentation prediction result comprises the following steps: inputting a prediction picture to be predicted through a student multi-organ segmentation model, wherein the formula is as follows:
p＝δ(ft(x)) (13)
where p is the resulting multi-organ segmentation result.
The embodiment of the invention realizes the training of the multi-organ model by utilizing the learning mode of aggregating the single-organ knowledge of the multi-teacher model and based on organs and background areas, and obtains good effect on multi-organ prediction.
Based on the same concept of the above embodiments, in another embodiment of the present invention, a privacy-friendly multi-organ segmentation system based on multi-source integrated distillation is provided. Fig. 2 is a schematic diagram of the system of the present embodiment. The method comprises the following steps:
an organ segmentation model module: and inputting the pictures to obtain a multi-component segmentation chart through the teacher model and the student model.
Single organ segmentation output conversion module: and performing output conversion on the teacher picture segmentation picture, and expanding the teacher picture segmentation picture to multi-class output.
The mask definition and conversion module: the output is transformed into organ and background regions according to the region-based mask.
A supervision alignment module: and respectively performing supervised learning on different organ areas and background areas, so that converted prediction results of the teacher model and the student models are aligned, and an efficient student multi-organ segmentation model is obtained.
A prediction module: and inputting the prediction picture to obtain an organ segmentation prediction result through a student multi-organ segmentation model.
The invention also provides a multi-organ segmentation system based on multi-source integrated distillation, which comprises the following modules:
module M1: inputting pictures, and predicting by a teacher model and a student model to obtain a multi-component segmentation picture; the input picture is predicted by utilizing a teacher single-organ neural network segmentation model formed by the convolutional layer and the full-connection layer, and the formula is as follows:
Figure BDA0003553361570000082
where x is the input picture, the superscript j indicates the jth dimension,
Figure BDA0003553361570000083
is the mth teacher neural network model, δkRepresenting the softmax prediction output in the k-th dimension,
Figure BDA0003553361570000084
is the prediction probability of the kth dimension obtained through the mth teacher neural network model;
the input picture is predicted by utilizing a student multi-organ neural network segmentation model formed by the convolutional layer and the full connecting layer, and the formula is as follows:
Figure BDA0003553361570000085
wherein f istIs a neural network model of a student,
Figure BDA0003553361570000086
is the predicted probability of the kth dimension obtained by the student neural network model.
Module M2: performing output conversion on the teacher picture segmentation picture, and expanding the teacher picture segmentation picture to multi-class output; for organ regions, keeping a background prediction of teacher segmentation model
Figure BDA0003553361570000091
Without change, the prediction of the organ numbered m
Figure BDA0003553361570000092
Move to the m-th dimension and then set the values of all other dimensions to 0, the formula is as follows:
Figure BDA0003553361570000093
wherein the content of the first and second substances,
Figure BDA0003553361570000094
outputting the organ with the number of m after the teacher model outputs the conversion mapping;
for the background area, the predicted values of all M teacher single organ segmentation models are averaged, and the formula is as follows:
Figure BDA0003553361570000095
wherein the content of the first and second substances,
Figure BDA0003553361570000096
the teacher model outputs the transformed and mapped output to the background.
Module M3: performing organ and background region conversion on the output according to the region-based mask; a prediction mask for organ m is obtained, the formula is as follows:
Figure BDA0003553361570000097
and converting the prediction of the teacher single-organ segmentation model on the organ region, wherein the formula is as follows:
Figure BDA0003553361570000098
transforming the predictions of the student multi-organ segmentation model for the organ regions, the formula is as follows:
Figure BDA0003553361570000099
obtaining the mth teacher single organ segmentation model
Figure BDA00035533615700000910
For the mask of the background, the formula is as follows:
Figure BDA00035533615700000911
aggregating the masks of all M teacher single organ segmentation models for the background, the formula is as follows:
Figure BDA00035533615700000912
converting the prediction of the teacher single organ segmentation model for the background area, wherein the formula is as follows:
Figure BDA00035533615700000913
converting the prediction of the student multi-organ segmentation model for the background area, wherein the formula is as follows:
Figure BDA00035533615700000914
module M4: respectively performing supervised learning on different organ areas and background areas, and aligning converted prediction results of a teacher model and a student model to obtain a student multi-organ segmentation model; setting different weights for different organ areas and background areas, and aligning the converted prediction results of the teacher model and the student model, wherein the formula is as follows:
Figure BDA00035533615700000915
where λ is a hyper-parameter used to balance the weights of the organ region and the background region.
Module M5: inputting a prediction picture and obtaining an organ segmentation prediction result through a student multi-organ segmentation model; inputting a prediction picture to be predicted through a student multi-organ segmentation model, wherein the formula is as follows:
p＝δ(ft(x)) (13)
where p is the resulting multi-organ segmentation result.
In summary, in the above embodiments, a rough and super-segmentation result of a learnable rough and super-segmentation model is used, a high-frequency part of a high-frequency discrete learnable dictionary discrete coded picture is used, a local autoregressive model is used to complete generation from the rough and super-segmentation result to the high-frequency dictionary coding, and a high-definition picture corresponding to a final low-definition picture is generated by using a high-definition picture generation module, so that the problems of multiple organ labeling loss and data privacy are alleviated, the segmentation accuracy of each organ is improved, and a large amount of calculation time consumption is saved compared with a single organ segmentation model.
The invention can perform multi-organ segmentation by utilizing the disclosed pre-training single-organ segmentation model and the label-free multi-organ data set, thereby achieving good organ segmentation effect.
It should be noted that, the steps in the method provided by the present invention may be implemented by using corresponding modules, devices, units, and the like in the system, and those skilled in the art may implement the composition of the system by referring to the technical solution of the method, that is, the embodiment in the method may be understood as a preferred example for constructing the system, and will not be described herein again.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The above-described preferred features may be used in any combination without conflict with each other.
Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, units provided by the present invention as pure computer readable program code, the system and its various devices, modules, units provided by the present invention can be fully implemented by logically programming method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system and various devices, modules and units thereof provided by the invention can be regarded as a hardware component, and the devices, modules and units included in the system for realizing various functions can also be regarded as structures in the hardware component; means, modules, units for performing the various functions may also be regarded as structures within both software modules and hardware components for performing the method.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.

Claims (10)

1. A multi-organ segmentation method based on multi-source integrated distillation, the method comprising the steps of:
step S1: inputting pictures, and predicting by a teacher model and a student model to obtain a multi-component segmentation picture;
step S2: performing output conversion on the teacher picture segmentation picture, and expanding the teacher picture segmentation picture to multi-class output;
step S3: performing organ and background region conversion on the output according to the region-based mask;
step S4: respectively performing supervised learning on different organ areas and background areas, and aligning the converted prediction results of the teacher model and the student models to obtain a student multi-organ segmentation model;
step S5: and inputting the prediction picture to obtain an organ segmentation prediction result through a student multi-organ segmentation model.
2. The multi-organ segmentation method based on multi-source integrated distillation of claim 1, wherein the step S1 is implemented by using a teacher single-organ neural network segmentation model composed of convolutional layers and fully-connected layers to predict the input picture, and the formula is as follows:
Figure FDA0003553361560000011
where x is the input picture, the superscript j indicates the jth dimension,
Figure FDA0003553361560000012
is the mth teacher neural network model, δkTo representSoftmax predicted output in the k-th dimension,
Figure FDA0003553361560000013
is the prediction probability of the kth dimension obtained through the mth teacher neural network model;
the input picture is predicted by utilizing a student multi-organ neural network segmentation model formed by the convolutional layer and the full connecting layer, and the formula is as follows:
Figure FDA0003553361560000014
wherein f istIs a neural network model of a student,
Figure FDA0003553361560000015
is the predicted probability of the kth dimension obtained by the student neural network model.
3. The multi-organ segmentation method based on multi-source integrated distillation of claim 1, wherein the step S2 includes:
preserving background prediction of teacher segmentation model for organ regions
Figure FDA0003553361560000016
Without change, the prediction of the organ numbered m
Figure FDA0003553361560000017
Move to the m-th dimension and then set the values of all other dimensions to 0, the formula is as follows:
Figure FDA0003553361560000018
wherein the content of the first and second substances,
Figure FDA0003553361560000019
outputting the organ with the number of m after the teacher model outputs the conversion mapping;
for the background region, the predicted values of all M teacher single organ segmentation models are averaged, and the formula is as follows:
Figure FDA0003553361560000021
wherein the content of the first and second substances,
Figure FDA0003553361560000022
the teacher model outputs the transformed and mapped output to the background.
4. The multi-organ segmentation method based on multi-source integrated distillation of claim 1, wherein the step S3 includes:
a prediction mask for organ m is obtained, the formula is as follows:
Figure FDA0003553361560000023
and converting the prediction of the teacher single-organ segmentation model on the organ region, wherein the formula is as follows:
Figure FDA0003553361560000024
transforming the predictions of the student multi-organ segmentation model for the organ regions, the formula is as follows:
Figure FDA0003553361560000025
obtaining the mth teacher single organ segmentation model
Figure FDA0003553361560000026
For the mask of the background, the formula is as follows:
Figure FDA0003553361560000027
aggregating the masks of all M teacher single organ segmentation models for the background, the formula is as follows:
Figure FDA0003553361560000028
converting the prediction of the teacher single organ segmentation model for the background area, wherein the formula is as follows:
Figure FDA0003553361560000029
converting the prediction of the student multi-organ segmentation model for the background area, wherein the formula is as follows:
Figure FDA00035533615600000210
5. the multi-organ segmentation method based on multi-source integrated distillation of claim 1, wherein the step S4 sets different weights for different organ areas and background areas, and aligns the converted prediction results of teacher and student models, and the formula is as follows:
Figure FDA00035533615600000211
where λ is a hyper-parameter used to balance the weights of the organ region and the background region.
6. The multi-organ segmentation method based on multi-source integrated distillation of claim 1, wherein the step S5 is to input the prediction picture to be predicted by a student multi-organ segmentation model, and the formula is as follows:
p＝δ(ft(x)) (13)
where p is the resulting multi-organ segmentation result.
7. A multi-organ segmentation system based on multi-source integrated distillation, characterized in that the system comprises the following modules:
module M1: inputting pictures and obtaining multi-component segmentation pictures through teacher and student model prediction;
module M2: performing output conversion on the teacher picture segmentation picture, and expanding the teacher picture segmentation picture to multi-class output;
module M3: performing organ and background region conversion on the output according to the region-based mask;
module M4: respectively performing supervised learning on different organ areas and background areas, and aligning converted prediction results of a teacher model and a student model to obtain a student multi-organ segmentation model;
module M5: and inputting the prediction picture to obtain an organ segmentation prediction result through a student multi-organ segmentation model.
8. The multi-organ segmentation system based on multi-source integrated distillation of claim 7, wherein the module M1 predicts the input picture by using a teacher single-organ neural network segmentation model composed of convolution layers and full connection layers, and the formula is as follows:
Figure FDA0003553361560000031
where x is the input picture, the superscript j indicates the jth dimension,
Figure FDA0003553361560000032
is the mth teacher neural network model, δkRepresenting the softmax prediction output in the k-th dimension,
Figure FDA0003553361560000033
is the prediction probability of the kth dimension obtained through the mth teacher neural network model;
the input picture is predicted by utilizing a student multi-organ neural network segmentation model formed by the convolutional layer and the full connecting layer, and the formula is as follows:
Figure FDA0003553361560000034
wherein f istIs a neural network model of a student,
Figure FDA0003553361560000035
is the predicted probability of the kth dimension obtained by the student neural network model;
the module M2 includes:
preserving background prediction of teacher segmentation model for organ regions
Figure FDA0003553361560000036
Without change, the prediction of the organ numbered m
Figure FDA0003553361560000037
Move to the m-th dimension and then set the values of all other dimensions to 0, the formula is as follows:
Figure FDA0003553361560000038
wherein the content of the first and second substances,
Figure FDA0003553361560000039
outputting the organ with the number of m after the teacher model outputs the conversion mapping;
for the background region, the predicted values of all M teacher single organ segmentation models are averaged, and the formula is as follows:
Figure FDA00035533615600000310
wherein the content of the first and second substances,
Figure FDA00035533615600000311
the teacher model outputs the transformed and mapped output to the background.
9. The multi-organ segmentation system based on multi-source integrated distillation of claim 7, wherein the module M3 comprises:
a prediction mask for organ m is obtained, the formula is as follows:
Figure FDA0003553361560000041
and converting the prediction of the teacher single-organ segmentation model on the organ region, wherein the formula is as follows:
Figure FDA0003553361560000042
transforming the predictions of the student multi-organ segmentation model for the organ regions, the formula is as follows:
Figure FDA0003553361560000043
obtaining the mth teacher single organ segmentation model
Figure FDA0003553361560000044
For the mask of the background, the formula is as follows:
Figure FDA0003553361560000045
aggregating the masks of all M teacher single organ segmentation models for the background, the formula is as follows:
Figure FDA0003553361560000046
converting the prediction of the teacher single organ segmentation model for the background area, wherein the formula is as follows:
Figure FDA0003553361560000047
converting the prediction of the student multi-organ segmentation model for the background area, wherein the formula is as follows:
Figure FDA0003553361560000048
10. the multi-organ segmentation system based on multi-source integrated distillation of claim 7 wherein the module M4 sets different weights for different organ and background regions, aligning the transformed prediction results of teacher and student models as follows:
Figure FDA0003553361560000049
wherein λ is a hyper-parameter for balancing weights of organ regions and background regions;
the module M5 inputs a prediction picture to be predicted by a student multi-organ segmentation model, and the formula is as follows:
p＝δ(ft(x)) (13)
where p is the resulting multi-organ segmentation result.
CN202210268374.6A 2022-03-18 2022-03-18 Multi-organ segmentation method and system based on multi-source integrated distillation Pending CN114612451A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210268374.6A CN114612451A (en) 2022-03-18 2022-03-18 Multi-organ segmentation method and system based on multi-source integrated distillation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210268374.6A CN114612451A (en) 2022-03-18 2022-03-18 Multi-organ segmentation method and system based on multi-source integrated distillation

Publications (1)

Publication Number Publication Date
CN114612451A true CN114612451A (en) 2022-06-10

Family

ID=81865798

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210268374.6A Pending CN114612451A (en) 2022-03-18 2022-03-18 Multi-organ segmentation method and system based on multi-source integrated distillation

Country Status (1)

Country Link
CN (1) CN114612451A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115908823A (en) * 2023-03-09 2023-04-04 南京航空航天大学 Semantic segmentation method based on difficulty distillation

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115908823A (en) * 2023-03-09 2023-04-04 南京航空航天大学 Semantic segmentation method based on difficulty distillation
CN115908823B (en) * 2023-03-09 2023-05-12 南京航空航天大学 Semantic segmentation method based on difficulty distillation

Similar Documents

Publication Publication Date Title
Valanarasu et al. Medical transformer: Gated axial-attention for medical image segmentation
CN112651973B (en) Semantic segmentation method based on cascade of feature pyramid attention and mixed attention
Matsubara et al. Distilled split deep neural networks for edge-assisted real-time systems
CN113792113A (en) Visual language model obtaining and task processing method, device, equipment and medium
CN113705769A (en) Neural network training method and device
WO2022001805A1 (en) Neural network distillation method and device
CN111782838A (en) Image question-answering method, image question-answering device, computer equipment and medium
Li et al. Attention, suggestion and annotation: a deep active learning framework for biomedical image segmentation
CN112634296A (en) RGB-D image semantic segmentation method and terminal for guiding edge information distillation through door mechanism
Yang et al. NDNet: Narrow while deep network for real-time semantic segmentation
CN117218498B (en) Multi-modal large language model training method and system based on multi-modal encoder
CN113706544A (en) Medical image segmentation method based on complete attention convolution neural network
CN114612451A (en) Multi-organ segmentation method and system based on multi-source integrated distillation
CN113515948A (en) Language model training method, device, equipment and storage medium
CN109410189A (en) The similarity calculating method of image partition method and image, device
Yi et al. Elanet: effective lightweight attention-guided network for real-time semantic segmentation
CN116524307A (en) Self-supervision pre-training method based on diffusion model
CN113850012B (en) Data processing model generation method, device, medium and electronic equipment
Yan et al. RoboSeg: Real-time semantic segmentation on computationally constrained robots
CN114359656A (en) Melanoma image identification method based on self-supervision contrast learning and storage device
CN111652349A (en) Neural network processing method and related equipment
Hao et al. Radiographs and texts fusion learning based deep networks for skeletal bone age assessment
Hashim et al. An Optimized Image Annotation Method Utilizing Integrating Neural Networks Model and Slantlet Transformation
Pham et al. Seunet-trans: A simple yet effective unet-transformer model for medical image segmentation
CN116095183A (en) Data compression method and related equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination