CN117911844A - Multi-mode medical image labeling method and device - Google Patents
Multi-mode medical image labeling method and device Download PDFInfo
- Publication number
- CN117911844A CN117911844A CN202410319373.9A CN202410319373A CN117911844A CN 117911844 A CN117911844 A CN 117911844A CN 202410319373 A CN202410319373 A CN 202410319373A CN 117911844 A CN117911844 A CN 117911844A
- Authority
- CN
- China
- Prior art keywords
- medical image
- image data
- modal
- constraint
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000002372 labelling Methods 0.000 title claims abstract description 70
- 238000000034 method Methods 0.000 claims abstract description 34
- 238000012512 characterization method Methods 0.000 claims abstract description 31
- 238000007781 pre-processing Methods 0.000 claims abstract description 31
- 238000007670 refining Methods 0.000 claims abstract description 11
- 238000007621 cluster analysis Methods 0.000 claims abstract description 10
- 238000012545 processing Methods 0.000 claims description 24
- 239000011159 matrix material Substances 0.000 claims description 23
- 238000013528 artificial neural network Methods 0.000 claims description 18
- 238000005457 optimization Methods 0.000 claims description 18
- 238000010606 normalization Methods 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 10
- 238000012549 training Methods 0.000 claims description 10
- 238000002759 z-score normalization Methods 0.000 claims description 8
- 230000007246 mechanism Effects 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 6
- 230000006870 function Effects 0.000 claims description 5
- 230000000295 complement effect Effects 0.000 claims description 4
- 230000000750 progressive effect Effects 0.000 claims description 4
- 238000003384 imaging method Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000002595 magnetic resonance imaging Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000002591 computed tomography Methods 0.000 description 3
- 238000002597 diffusion-weighted imaging Methods 0.000 description 3
- 239000000203 mixture Substances 0.000 description 3
- 230000010412 perfusion Effects 0.000 description 3
- 238000002604 ultrasonography Methods 0.000 description 3
- 238000009792 diffusion process Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000002091 elastography Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000002600 positron emission tomography Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000002792 vascular Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Image Analysis (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
Abstract
The invention relates to the technical field of computer vision, and provides a multi-mode medical image labeling method and device, wherein the method comprises the following steps: acquiring multi-modal medical image data, the multi-modal medical image data comprising a plurality of paired samples; preprocessing and extracting features of the multi-mode medical image data in sequence to obtain a first high-level representation, and determining constraint information according to association among a plurality of paired samples, wherein the association comprises a must-connection relationship and a non-connection relationship among the plurality of paired samples; iteratively refining and fine-tuning the first high-level representation according to constraint information to obtain a second high-level representation; and carrying out cluster analysis on the second high-level characterization to obtain pseudo tag information corresponding to the multi-mode medical image data, and marking and calibrating the multi-mode medical image data according to the pseudo tag information to obtain new multi-mode medical image data. The method disclosed by the invention can be effectively generalized in weak annotation application scenes, and the efficiency and accuracy of image annotation are improved.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a multi-mode medical image labeling method and device.
Background
The multi-mode medical image representation learning aims at integrating and summarizing heterogeneous medical image groups into a unified abstract representation, so that the heterogeneous medical image groups can contain more visual, comprehensive and accurate information as much as possible, and the abstract representation marks images, thereby being beneficial to assisting doctors in making medical diagnosis.
In the related art, the method for realizing image annotation by utilizing multi-mode medical image characterization learning is mainly divided into two major categories, namely full supervision and non-supervision; although both achieve considerable results, there are still some drawbacks to be solved: (1) The full supervision method relies on a large number of image labels, and the acquisition of the image labels in a real scene needs to consume huge labor cost and capital investment; (2) The non-supervision method does not need any labeling information, but for a given image group in a real scene, a few weak labeling information is usually contained, and the non-supervision method cannot fully learn more image information, so that the extracted image features have poor characterization capability, and further the image labeling is inaccurate.
Disclosure of Invention
The invention provides a multi-mode medical image labeling method and device, which are used for solving the defects that the image labeling is inaccurate due to poor representation capability of image features extracted by an unsupervised image labeling method in the prior art, and the efficiency and the accuracy of the image labeling are improved because the acquisition of the image labeling in a real scene by the supervised image labeling method needs to consume huge labor cost and capital investment.
The invention provides a multi-mode medical image labeling method, which comprises the following steps:
acquiring multi-modal medical image data, the multi-modal medical image data comprising a plurality of paired samples;
Preprocessing and feature extraction are sequentially carried out on the multi-mode medical image data to obtain a first high-level representation, constraint information is determined according to association among the plurality of paired samples, and the association comprises a must-connection relationship and a non-connection relationship among the plurality of paired samples;
Iteratively refining and fine-tuning the first high-level representation according to the constraint information to obtain a second high-level representation;
and performing cluster analysis on the second high-level characterization to obtain pseudo tag information corresponding to the multi-modal medical image data, and marking and calibrating the multi-modal medical image data according to the pseudo tag information to obtain new multi-modal medical image data.
According to the multi-mode medical image labeling method provided by the invention, the preprocessing and feature extraction of the multi-mode medical image data sequentially comprise the following steps:
Performing target preprocessing on the multi-mode medical image data to obtain preprocessed data; the target pre-processing includes at least one of maximum and minimum normalization processing, Z-score normalization processing, and L2 norm normalization processing;
And performing end-to-end pre-training on the preprocessed data based on an automatic encoder neural network to obtain the first high-level representation.
According to the multi-mode medical image labeling method provided by the invention, the first high-level representation is iteratively refined and finely adjusted according to the constraint information, and the obtaining of the second high-level representation comprises the following steps:
Semi-supervised multi-modal learning is carried out on the optimization target according to the relaxation K-means function, the diversity self-walking learning mechanism and the constraint information, and the second high-level representation is obtained; the optimization objective mines complementary information among cross-modal samples in the progressive characterization learning process.
According to the multi-mode medical image labeling method provided by the invention, the constraint information is represented by the following formula:
wherein, For/>Sample and/>Membership of individual samples; when/>Sample and/>The samples belong to the same category, then/>=1, When/>Sample and/>The individual samples belong to different classes, then/>=0；/>For index values of sample constraints, there are arbitrary pairs of samples (/ >)，/>) When the constraint prior of the must-be-connected or the constraint prior of the not-be-connected is met, the method comprises the steps of=1; If the sample is in the pair [ ]， />) Without constraint prior, then/>=0；/>And/>Respectively is/>Sample and/>A feature representation of the sample in a v-th modality; n is the number of samples, V is the number of modes; ML is a must-connect constraint, and CL is a no-connect constraint.
According to the multi-mode medical image labeling method provided by the invention, the optimization target is represented by the following formula:
Where L is the self-supervising sample reconstruction loss, Is medical image data in the v-th mode,/>For the first high-level representation corresponding to the medical image data in the v-th mode,/>Is the feature matrix at the v-th view angle,/>For the cluster centroid matrix at the v-th view angle,/>Is a weight matrix under the v-th view angle, A is a constraint indication matrix,/>Is a super parameter, C is a sample constraint matrix,/>Whether the ith sample is allocated to the kth cluster or not, if so, the ith sample is 1, otherwise, the ith sample is 0; /(I)Is a self-step coefficient,/>The sample weight matrix is formed by the steps that F is the Frobenius norm; k represents the total number of target cluster clusters, and K is the kth target cluster; s represents a clustering indication matrix; ii wtii 1 and |wtii 2,1 are the global sparsity constraint L1 norm and the structured sparsity constraint L2,1 norm, respectively, applied to the same time.
According to the method for labeling the multi-modal medical image provided by the invention, the labeling and calibrating the multi-modal medical image data according to the pseudo tag information, and the obtaining of new multi-modal medical image data comprises the following steps:
manually calibrating and exploring the pseudo tag information to obtain the tags of the image sample group;
and obtaining the new multi-modal medical image data according to the labels of the image sample group and the multi-modal medical image data.
The invention also provides a multimode medical image labeling device, which comprises:
the data acquisition module is used for acquiring multi-mode medical image data, wherein the multi-mode medical image data comprises a plurality of paired samples;
The constraint information extraction module is used for sequentially preprocessing and extracting features of the multi-mode medical image data to obtain a first high-level representation, and determining constraint information according to the association among the plurality of paired samples, wherein the association comprises a must-connection relationship and a non-connection relationship among the plurality of paired samples;
the feature extraction module is used for iteratively refining and fine-tuning the first high-level representation according to the constraint information to obtain a second high-level representation;
And the labeling module is used for carrying out cluster analysis on the second high-level characterization to obtain pseudo tag information corresponding to the multi-modal medical image data, and labeling and calibrating the multi-modal medical image data according to the pseudo tag information to obtain new multi-modal medical image data.
According to the multi-mode medical image labeling device provided by the invention, the labeling module is specifically used for:
Performing target preprocessing on the multi-mode medical image data to obtain preprocessed data; the target pre-processing includes at least one of maximum and minimum normalization processing, Z-score normalization processing, and L2 norm normalization processing;
And performing end-to-end pre-training on the preprocessed data based on an automatic encoder neural network to obtain the first high-level representation.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the multi-mode medical image labeling method according to any one of the above when executing the program.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a multi-modal medical image labeling method as described in any of the above.
According to the multi-mode medical image labeling method and device, the multi-mode medical image data are preprocessed and feature extracted sequentially to obtain the first high-level characterization, the first high-level characterization is iteratively refined and finely adjusted according to constraint information among a plurality of paired samples to obtain the constraint information determined by the second high-level characterization, the second high-level characterization is subjected to clustering analysis to obtain pseudo tag information corresponding to the multi-mode medical image data, the multi-mode medical image data are labeled and calibrated according to the pseudo tag information to obtain new multi-mode medical image data, the new multi-mode medical image data can be effectively generalized in weak labeling application scenes, and the efficiency and the accuracy of image labeling are improved.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a multi-modal medical image labeling method according to the present invention;
FIG. 2 is a schematic flow chart of semi-supervised multi-modal learning of optimization targets provided by the invention;
FIG. 3 is a second flow chart of the method for labeling multi-modal medical images according to the present invention;
FIG. 4 is a schematic structural diagram of a multi-modal medical image labeling apparatus according to the present invention;
Fig. 5 is a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The following describes the method and apparatus for labeling multi-modal medical images according to the present invention with reference to fig. 1-4.
Fig. 1 is a schematic flow chart of a method for labeling a multi-modal medical image according to the present invention, as shown in fig. 1, the method for labeling a multi-modal medical image includes the following steps:
step 110, acquiring multi-modal medical image data, wherein the multi-modal medical image data comprises a plurality of paired samples.
In this step, the multi-modality image may refer to an image type fused with various imaging technologies such as CT (Computed Tomography, electronic computed tomography), MRI (Magnetic Resonance Imaging ), ultrasound, PET (Positron Emission Computed Tomography, positron emission tomography), and the like, the multi-modality image may also refer to a manner of integrating multiple imaging technologies in a single imaging device, the multi-modality medical image data may refer to multiple sets of medical image data of the same or different types, for example, the multi-modality MRI refers to a combination of multiple MRI sequences, which is a functional sequence such as DWI (diffusion WEIGHTED IMAGING, magnetic resonance diffusion weighted imaging), DKI (Diffusion Kurtos Isimaging, magnetic resonance diffusion kurtosis imaging), PWI (perfusion WEIGHTED IMAGING, magnetic resonance perfusion weighted imaging), and the like, which are integrated on a regular sequence basis.
In this embodiment, multi-modality CT includes pan CT, enhanced CT, CT vascular imaging (CTA), CT perfusion imaging (CTP), and the like; the multi-mode ultrasound includes common two-dimensional ultrasound, elastography, etc.
In this embodiment, the computer device stores the multi-modal medical image data distributed in each application system of the hospital and the physical examination center by local pushing or network downloading. The multi-mode medical image data generally does not contain sample labels (also called strong labels), but contains Xu Yangben pair-wise constraints (also called weak labels), namely that a must-be-connected (ML) and a unconnected (CL) respectively represent samples belonging to the same category and different categories, and the weak labels can provide additional supervision signals for a sample learning process.
Step 120, preprocessing and feature extraction are sequentially performed on the multi-mode medical image data to obtain a first high-level representation, constraint information is determined according to association among a plurality of paired samples, and the association comprises a must-connected relationship and a unconnected relationship among the plurality of paired samples.
In this step, the multi-modal medical image data includes a plurality of paired samples, and association information between different samples is not the same, for example, a must-connect relationship and a disconnection relationship between paired samples in the multi-modal medical image data; for example, the firstSample and/>If the two samples belong to the same category, the two samples are in a mandatory connection relationship, and if the two samples do not belong to the same category, the two samples are in an unfamiliar relationship.
In this step, the preprocessing includes maximum and minimum normalization processing, and may further include Z-score normalization processing, L2 norm normalization processing, or the like.
In this embodiment, features are extracted from the multi-modality medical image data, which may be high-level characterization information extracted from the multi-modality medical image data by an automatic encoder, or may be other image features extracted based on machine learning or deep learning, where the image features are used to represent the degree of information complementarity between the various modality medical image data.
In this embodiment, given medical image data of V modalities,Using paired samples present in a real scene to constrain a priori,/>Dividing X into K disjoint clusters, judging whether the constraint condition is violated when the distance between the sample and different clusters is calculated subsequently, if yes, finding the next closest cluster, and the like until finding the closest cluster meeting the constraint condition; if not, the cluster is placed.
In this embodiment, constraint information is represented by the following formula:
wherein, For/>Sample and/>Membership of individual samples; when/>Sample and/>The samples belong to the same category, then/>=1, When/>Sample and/>The individual samples belong to different classes, then/>=0；/>For index values of sample constraints, there are arbitrary pairs of samples (/ >)，/>) When the constraint prior of the must-be-connected or the constraint prior of the not-be-connected is met, the method comprises the steps of=1; If the sample is in the pair [ ]， />) Without constraint prior, then/>=0；/>And/>Respectively is/>Sample and/>A feature representation of the sample in a v-th modality; n is the number of samples, V is the number of modes; ML is a must-connect constraint, and CL is a no-connect constraint.
And 130, iteratively refining and fine-tuning the first high-level representation according to the constraint information to obtain a second high-level representation.
In this step, the first high-level representation is iteratively refined and fine-tuned by designing an optimization objective and an optimization algorithm, and the constraint condition is ensured to be satisfied in the iterative process, so as to obtain the second high-level representation.
In this embodiment, the sample constraint matrix elements constructed for technical scheme SPDMCAnd constraint indicating matrix element/>Respectively meeting corresponding constraint conditions, taking loose K-means as a base cost function, introducing a diversity self-step learning mechanism, and unifying the self-step learning mechanism to a joint optimization target to realize semi-supervised multi-mode learning.
In this embodiment, the first term of the optimization objective is self-supervised sample reconstruction loss, the second term is variable weighted relaxed K-means, and the third term is semi-supervised learning regularization term that organically unifies two different forms of pairwise constraints (ML and CL) such that model-learned multi-modal representations are as similar as possible when satisfying the ML constraint, as orthogonal as possible when satisfying the CL constraint, A contains nN/>C comprises n/>N/>; And obtaining a second high-level representation by solving the optimization target.
And 140, performing cluster analysis on the second high-level characterization to obtain pseudo tag information corresponding to the multi-mode medical image data, and marking and calibrating the multi-mode medical image data according to the pseudo tag information to obtain new multi-mode medical image data.
In this step, the clustering algorithm may be set according to user requirements, for example, according to a K-means algorithm, a Gaussian mixture model cluster, a density-based DBSCAN cluster, and the like.
In this embodiment, the multi-mode medical image data may be manually labeled by using the label information, or may be automatically labeled by designing a related labeling program or algorithm.
Specifically, labeling and calibrating the multi-modal medical image data according to the pseudo tag information to obtain new multi-modal medical image data includes: manually calibrating and exploring the pseudo tag information to obtain the tags of the image sample group; and obtaining new multi-mode medical image data according to the labels of the image sample group and the multi-mode medical image data.
According to the multi-mode medical image labeling method provided by the embodiment of the invention, the multi-mode medical image data is preprocessed and the characteristics are extracted sequentially to obtain the first high-level characterization, the first high-level characterization is iteratively refined and finely adjusted according to constraint information and the relation among a plurality of paired samples to obtain the constraint information determined by the second high-level characterization, the second high-level characterization is subjected to clustering analysis to obtain pseudo tag information corresponding to the multi-mode medical image data, the multi-mode medical image data is labeled and calibrated according to the pseudo tag information to obtain new multi-mode medical image data, the multi-mode medical image data can be effectively generalized in weak labeling application scenes, and the efficiency and the accuracy of image labeling are improved.
In some embodiments, sequentially preprocessing and feature extraction of the multi-modality medical image data includes: performing target preprocessing on the multi-mode medical image data to obtain preprocessed data; the target preprocessing comprises at least one of maximum and minimum normalization processing, Z-score normalization processing and L2 norm normalization processing; and performing end-to-end pre-training on the preprocessed data based on the automatic encoder neural network to obtain a first high-level representation.
In this embodiment, the preprocessing includes maximum and minimum normalization processing, and may include Z-score normalization processing or L2 norm normalization processing, or may be performed simultaneously.
In this embodiment, the training process of the automatic encoder neural network includes the steps of:
(1) The method comprises the steps that multi-mode medical image sample data are input into an automatic encoder neural network, an encoder of the automatic encoder neural network maps the input data to low-dimensional features, and a decoder of the automatic encoder neural network is used for obtaining reconstruction data;
(2) And performing iterative training by minimizing errors between the originally input multi-mode medical image sample data and the reconstruction data until a preset condition is met or a preset iteration number is reached.
In this embodiment, each view is designed into a corresponding auto-encoder neural networkWherein/>And/>Representing an encoder and a decoder respectively, inputting original data for end-to-end pre-training to obtain a preliminary high-level representation/>。
Specifically, the multi-modal medical image sample data is input into an automatic encoder neural network, and an encoder of the automatic encoder neural networkMapping input data to low-dimensional features, decoder using an automatic encoder neural networkObtaining reconstruction data; and then carrying out iterative training by minimizing errors between the original input multi-mode medical image sample data and the reconstruction data until a preset condition is met or the preset iteration times are reached, so as to obtain the trained automatic encoder neural network. Acquiring a preliminary high-level representation of a multi-mode medical image sample data set through a trained automatic encoder neural network, namely a first high-level representation: /(I)。
According to the multi-mode medical image labeling method provided by the embodiment of the invention, the multi-mode medical image data is subjected to target preprocessing to obtain preprocessed data, and the preprocessed data is subjected to end-to-end pretraining based on the automatic encoder neural network to obtain the first high-level representation, so that the image data quality is improved through image preprocessing, and the representation capacity of the first high-level representation is further improved.
In some embodiments, iteratively refining and fine-tuning the first high-level representation according to constraint information, the deriving the second high-level representation includes: semi-supervised multi-modal learning is carried out on the optimization target according to the relaxation K-means function, the diversity self-walking learning mechanism and the constraint information, and a second high-level representation is obtained; the optimization objective mines complementary information between cross-modal samples in the process of progressive characterization learning.
In this embodiment, the optimization objective is represented by the following formula:
Where L is the self-supervising sample reconstruction loss, Is medical image data in the v-th mode,/>For the first high-level representation corresponding to the medical image data in the v-th mode,/>Is the feature matrix at the v-th view angle,/>For the cluster centroid matrix at the v-th view angle,/>Is a weight matrix under the v-th view angle, A is a constraint indication matrix,/>Is a super parameter, C is a sample constraint matrix,/>Whether the ith sample is allocated to the kth cluster or not, if so, the ith sample is 1, otherwise, the ith sample is 0; /(I)Is a self-step coefficient,/>The sample weight matrix is formed by the steps that F is the Frobenius norm; k represents the total number of target cluster clusters, and K is the kth target cluster; s represents a clustering indication matrix; and II, W 1 and II, W 2,1 are respectively the global sparse constraint L1 norm and the structured sparse constraint L2 and 1 norm applied simultaneously, so that the model is favorable for simultaneously considering the difficulty and diversity of the sample, and the cross-modal complementary information is mined in the progressive characteristic learning process.
FIG. 2 is a schematic flow chart of semi-supervised multi-modal learning of optimization objectives according to the present invention, in which in the embodiment shown in FIG. 2, after obtaining the firstCharacterization of the individual samples in the v-th modality/>First/>Characterization of the individual samples in the v-th modality/>Then, according to constraint information between the paired samples: paired samples satisfy the ML constraint (/ >)=1，/>=1), And the paired samples satisfy CL constraint (/ >=0，/>And when the method is=1), semi-supervised multi-modal learning is carried out on the optimization target, and a corresponding second high-level representation is obtained.
According to the multi-mode medical image labeling method provided by the embodiment of the invention, the optimization target is subjected to semi-supervised multi-mode learning through the relaxation K-means function, the diversity self-learning mechanism and the constraint information to obtain the second high-level representation, the cost investment of image labeling is reduced through semi-supervised learning, the weak labeling information contained in the image is fully utilized, and the representation capability of image features is improved.
Fig. 3 is a second flow chart of the method for labeling multi-modal medical images according to the present invention, and in the embodiment shown in fig. 3, the method for labeling multi-modal medical images is specifically implemented by the following steps: step T1, acquiring multi-view data of a medical image group; step T2, constructing image composition pair constraint (including must connection and disconnection) according to the weak annotation information; step T3, carrying out data preprocessing on the multi-view data of the medical image group; step T4, performing high-level characterization learning on the preprocessed data to obtain a first high-level characterization; step T5, refining and fine-tuning the first high-level representation by combining the image composition obtained in the step T2 to obtain a second high-level representation; step T6, performing cluster analysis on the second high-level characterization to generate a pseudo tag; and step T7, manually calibrating the pseudo tag, and adjusting the error tag to finish the labeling of the multi-view data of the medical image group.
The multi-mode medical image labeling device provided by the invention is described below, and the multi-mode medical image labeling device described below and the multi-mode medical image labeling method described above can be referred to correspondingly.
Fig. 4 is a schematic structural diagram of a multi-mode medical image labeling device according to the present invention, as shown in fig. 4, the multi-mode medical image labeling device includes: a data acquisition module 410, a constraint information extraction module 420, a feature extraction module 430, and a labeling module 440.
A data acquisition module 410 for acquiring multi-modal medical image data, the multi-modal medical image data comprising a plurality of paired samples;
The constraint information extraction module 420 is configured to perform preprocessing and feature extraction on the multi-modal medical image data in sequence to obtain a first high-level representation, and determine constraint information according to correlations between a plurality of paired samples, where the correlations include a must-connected relationship and a unconnected relationship between the plurality of paired samples;
The feature extraction module 430 is configured to iteratively refine and fine tune the first high-level representation according to constraint information, so as to obtain a second high-level representation;
The labeling module 440 is configured to perform cluster analysis on the second high-level representation to obtain pseudo tag information corresponding to the multi-modal medical image data, and label and calibrate the multi-modal medical image data according to the pseudo tag information to obtain new multi-modal medical image data.
According to the multi-mode medical image labeling device provided by the embodiment of the invention, the multi-mode medical image data is preprocessed and the characteristics are extracted sequentially to obtain the first high-level characterization, the first high-level characterization is iteratively refined and finely adjusted according to constraint information and the relation among a plurality of paired samples to obtain the constraint information determined by the second high-level characterization, the second high-level characterization is subjected to clustering analysis to obtain pseudo-label information corresponding to the multi-mode medical image data, the multi-mode medical image data is labeled and calibrated according to the pseudo-label information to obtain new multi-mode medical image data, the multi-mode medical image data can be effectively generalized in weak labeling application scenes, and the efficiency and the accuracy of image labeling are improved.
In some embodiments, the labeling module is specifically configured to: performing target preprocessing on the multi-mode medical image data to obtain preprocessed data; the target preprocessing comprises at least one of maximum and minimum normalization processing, Z-score normalization processing and L2 norm normalization processing;
And performing end-to-end pre-training on the preprocessed data based on the automatic encoder neural network to obtain a first high-level representation.
According to the multi-mode medical image labeling device provided by the embodiment of the invention, the multi-mode medical image data is subjected to target preprocessing to obtain preprocessed data, and the preprocessed data is subjected to end-to-end pretraining based on the automatic encoder neural network to obtain the first high-level representation, so that the image data quality is improved through image preprocessing, and the representation capacity of the first high-level representation is further improved.
Fig. 5 is a schematic structural diagram of an electronic device according to the present invention, and as shown in fig. 5, the electronic device may include: processor 510, communication interface (Communications Interface) 520, memory 530, and communication bus 540, wherein processor 510, communication interface 520, memory 530 complete communication with each other through communication bus 540. Processor 510 may invoke logic instructions in memory 530 to perform a multi-modal medical image labeling method comprising: acquiring multi-modal medical image data, the multi-modal medical image data comprising a plurality of paired samples; preprocessing and extracting features of the multi-mode medical image data in sequence to obtain a first high-level representation, and determining constraint information according to association among a plurality of paired samples, wherein the association comprises a must-connection relationship and a non-connection relationship among the plurality of paired samples; iteratively refining and fine-tuning the first high-level representation according to constraint information to obtain a second high-level representation; and carrying out cluster analysis on the second high-level characterization to obtain pseudo tag information corresponding to the multi-mode medical image data, and marking and calibrating the multi-mode medical image data according to the pseudo tag information to obtain new multi-mode medical image data.
Further, the logic instructions in the memory 530 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a read-only memory (ROM), a random access memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention further provides a computer program product, where the computer program product includes a computer program, where the computer program can be stored on a non-transitory computer readable storage medium, and when the computer program is executed by a processor, the computer can execute the multi-mode medical image labeling method provided by the above methods, and the method includes: acquiring multi-modal medical image data, the multi-modal medical image data comprising a plurality of paired samples; preprocessing and extracting features of the multi-mode medical image data in sequence to obtain a first high-level representation, and determining constraint information according to association among a plurality of paired samples, wherein the association comprises a must-connection relationship and a non-connection relationship among the plurality of paired samples; iteratively refining and fine-tuning the first high-level representation according to constraint information to obtain a second high-level representation; and carrying out cluster analysis on the second high-level characterization to obtain pseudo tag information corresponding to the multi-mode medical image data, and marking and calibrating the multi-mode medical image data according to the pseudo tag information to obtain new multi-mode medical image data.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for labeling a multi-modal medical image, comprising:
acquiring multi-modal medical image data, the multi-modal medical image data comprising a plurality of paired samples;
Preprocessing and feature extraction are sequentially carried out on the multi-mode medical image data to obtain a first high-level representation, constraint information is determined according to association among the plurality of paired samples, and the association comprises a must-connection relationship and a non-connection relationship among the plurality of paired samples;
Iteratively refining and fine-tuning the first high-level representation according to the constraint information to obtain a second high-level representation;
and performing cluster analysis on the second high-level characterization to obtain pseudo tag information corresponding to the multi-modal medical image data, and marking and calibrating the multi-modal medical image data according to the pseudo tag information to obtain new multi-modal medical image data.
2. The method for labeling a multi-modal medical image according to claim 1, wherein the sequentially preprocessing and feature extraction of the multi-modal medical image data includes:
Performing target preprocessing on the multi-mode medical image data to obtain preprocessed data; the target pre-processing includes at least one of maximum and minimum normalization processing, Z-score normalization processing, and L2 norm normalization processing;
And performing end-to-end pre-training on the preprocessed data based on an automatic encoder neural network to obtain the first high-level representation.
3. The method of claim 1, wherein iteratively refining and fine tuning the first high-level representation according to the constraint information to obtain a second high-level representation comprises:
Semi-supervised multi-modal learning is carried out on the optimization target according to the relaxation K-means function, the diversity self-walking learning mechanism and the constraint information, and the second high-level representation is obtained; the optimization objective mines complementary information among cross-modal samples in the progressive characterization learning process.
4. The multi-modality medical image tagging method according to any one of claims 1 and 3, wherein the constraint information is represented by:
；
；
wherein, For/>Sample and/>Membership of individual samples; when/>Sample and/>The samples belong to the same category, then/>=1, When/>Sample and/>The individual samples belong to different classes, then/>=0；/>For index values of sample constraints, there are arbitrary pairs of samples (/ >)，/>) When the constraint prior of the must-be-connected or the constraint prior of the not-be-connected is met, the method comprises the steps of=1; If in the sample pair (/ >)， ) Without constraint prior, then/>=0；/>And/>Respectively is/>Sample and/>A feature representation of the sample in a v-th modality; n is the number of samples, V is the number of modes; ML is a must-connect constraint, and CL is a no-connect constraint.
5. The multi-modal medical image tagging method according to claim 3, wherein the optimization objective is represented by:
；
Where L is the self-supervising sample reconstruction loss, Is medical image data in the v-th mode,/>For the first high-level representation corresponding to the medical image data in the v-th mode,/>Is the feature matrix at the v-th view angle,/>For the cluster centroid matrix at the v-th view angle,/>Is a weight matrix under the v-th view angle, A is a constraint indication matrix,/>Is a super parameter, C is a sample constraint matrix,/>Whether the ith sample is allocated to the kth cluster or not, if so, the ith sample is 1, otherwise, the ith sample is 0; /(I)Is a self-step coefficient,/>The sample weight matrix is formed by the steps that F is the Frobenius norm; k represents the total number of target cluster clusters, and K is the kth target cluster; s represents a clustering indication matrix; ii wtii 1 and |wtii 2,1 are the global sparsity constraint L1 norm and the structured sparsity constraint L2,1 norm, respectively, applied to the same time.
6. The method for labeling multi-modal medical images according to claim 1, wherein labeling and calibrating the multi-modal medical image data according to the pseudo tag information to obtain new multi-modal medical image data includes:
manually calibrating and exploring the pseudo tag information to obtain the tags of the image sample group;
and obtaining the new multi-modal medical image data according to the labels of the image sample group and the multi-modal medical image data.
7. A multi-modal medical image labeling apparatus, comprising:
the data acquisition module is used for acquiring multi-mode medical image data, wherein the multi-mode medical image data comprises a plurality of paired samples;
The constraint information extraction module is used for sequentially preprocessing and extracting features of the multi-mode medical image data to obtain a first high-level representation, and determining constraint information according to the association among the plurality of paired samples, wherein the association comprises a must-connection relationship and a non-connection relationship among the plurality of paired samples;
the feature extraction module is used for iteratively refining and fine-tuning the first high-level representation according to the constraint information to obtain a second high-level representation;
And the labeling module is used for carrying out cluster analysis on the second high-level characterization to obtain pseudo tag information corresponding to the multi-modal medical image data, and labeling and calibrating the multi-modal medical image data according to the pseudo tag information to obtain new multi-modal medical image data.
8. The multi-modality medical image tagging device according to claim 7, wherein the tagging module is specifically configured to:
Performing target preprocessing on the multi-mode medical image data to obtain preprocessed data; the target pre-processing includes at least one of maximum and minimum normalization processing, Z-score normalization processing, and L2 norm normalization processing;
And performing end-to-end pre-training on the preprocessed data based on an automatic encoder neural network to obtain the first high-level representation.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the multimodal medical image labeling method of any of claims 1-6 when the program is executed by the processor.
10. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the multi-modality medical image labeling method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410319373.9A CN117911844A (en) | 2024-03-20 | 2024-03-20 | Multi-mode medical image labeling method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410319373.9A CN117911844A (en) | 2024-03-20 | 2024-03-20 | Multi-mode medical image labeling method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117911844A true CN117911844A (en) | 2024-04-19 |
Family
ID=90696389
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410319373.9A Pending CN117911844A (en) | 2024-03-20 | 2024-03-20 | Multi-mode medical image labeling method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117911844A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109978850A (en) * | 2019-03-21 | 2019-07-05 | 华南理工大学 | A kind of semi-supervised deep learning segmenting system of Multimodal medical image |
| CN110580695A (en) * | 2019-08-07 | 2019-12-17 | 深圳先进技术研究院 | multi-mode three-dimensional medical image fusion method and system and electronic equipment |
| CN116805380A (en) * | 2023-06-28 | 2023-09-26 | 广州医科大学附属肿瘤医院 | Multi-mode medical image fusion classification method and device, equipment and storage medium |
| CN117523356A (en) * | 2023-10-23 | 2024-02-06 | 海南大学 | Multi-view semi-supervised characterization learning method for medical image group science |
| CN117541836A (en) * | 2023-10-23 | 2024-02-09 | 海南大学 | Multi-mode medical image unsupervised characterization learning method |
-
2024
- 2024-03-20 CN CN202410319373.9A patent/CN117911844A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109978850A (en) * | 2019-03-21 | 2019-07-05 | 华南理工大学 | A kind of semi-supervised deep learning segmenting system of Multimodal medical image |
| CN110580695A (en) * | 2019-08-07 | 2019-12-17 | 深圳先进技术研究院 | multi-mode three-dimensional medical image fusion method and system and electronic equipment |
| CN116805380A (en) * | 2023-06-28 | 2023-09-26 | 广州医科大学附属肿瘤医院 | Multi-mode medical image fusion classification method and device, equipment and storage medium |
| CN117523356A (en) * | 2023-10-23 | 2024-02-06 | 海南大学 | Multi-view semi-supervised characterization learning method for medical image group science |
| CN117541836A (en) * | 2023-10-23 | 2024-02-09 | 海南大学 | Multi-mode medical image unsupervised characterization learning method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Menze et al. | The multimodal brain tumor image segmentation benchmark (BRATS) | |
| US9968257B1 (en) | Volumetric quantification of cardiovascular structures from medical imaging | |
| Ren et al. | Interleaved 3D‐CNN s for joint segmentation of small‐volume structures in head and neck CT images | |
| Shin et al. | Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning | |
| CN111488914B (en) | Alzheimer disease classification and prediction system based on multitask learning | |
| CN107016395B (en) | Identification system for sparsely expressed primary brain lymphomas and glioblastomas | |
| Yoo et al. | An example-based multi-atlas approach to automatic labeling of white matter tracts | |
| WO2018107371A1 (en) | Image searching system and method | |
| CN112037146B (en) | Automatic correction method and device for medical image artifacts and computer equipment | |
| US11430123B2 (en) | Sampling latent variables to generate multiple segmentations of an image | |
| Liu et al. | IOUC-3DSFCNN: Segmentation of brain tumors via IOU constraint 3D symmetric full convolution network with multimodal auto-context | |
| CN112766377B (en) | Left ventricle magnetic resonance image intelligent classification method, device, equipment and medium | |
| Wang et al. | Application of neuroanatomical features to tractography clustering | |
| CN116309571B (en) | Three-dimensional cerebrovascular segmentation method and device based on semi-supervised learning | |
| CN115908800A (en) | Medical image segmentation method | |
| CN115147640A (en) | Brain tumor image classification method based on improved capsule network | |
| Di Biase et al. | Connectomes for 40,000 UK Biobank participants: a multi-modal, multi-scale brain network resource | |
| CN112200810B (en) | Multi-modal automated ventricle segmentation system and method of use thereof | |
| Røvang et al. | Segmenting white matter hyperintensities on isotropic three-dimensional Fluid Attenuated Inversion Recovery magnetic resonance images: Assessing deep learning tools on a Norwegian imaging database | |
| Ghazi et al. | Deep learning methods for identification of white matter Fiber tracts: review of state-of-the-art and future prospective | |
| Saeed et al. | 3D MRU-Net: A novel mobile residual U-Net deep learning model for spine segmentation using computed tomography images | |
| CN114913174B (en) | Method, apparatus and storage medium for vascular system variation detection | |
| CN117911844A (en) | Multi-mode medical image labeling method and device | |
| CN106709921B (en) | Color image segmentation method based on space Dirichlet mixed model | |
| CN112686912B (en) | Acute stroke lesion segmentation method based on gradual learning and mixed samples |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |