CN115661377B - Self-supervision deep learning and method for constructing isotropic super-resolution three-dimensional image - Google Patents
Self-supervision deep learning and method for constructing isotropic super-resolution three-dimensional image Download PDFInfo
- Publication number
- CN115661377B CN115661377B CN202211689462.XA CN202211689462A CN115661377B CN 115661377 B CN115661377 B CN 115661377B CN 202211689462 A CN202211689462 A CN 202211689462A CN 115661377 B CN115661377 B CN 115661377B
- Authority
- CN
- China
- Prior art keywords
- loss
- batch
- mth
- training
- verification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 85
- 238000013135 deep learning Methods 0.000 title claims abstract description 45
- 238000012549 training Methods 0.000 claims abstract description 178
- 238000012795 verification Methods 0.000 claims abstract description 151
- 238000012545 processing Methods 0.000 claims abstract description 77
- 238000010200 validation analysis Methods 0.000 claims description 41
- 238000000605 extraction Methods 0.000 claims description 20
- 238000009499 grossing Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 13
- 230000000903 blocking effect Effects 0.000 claims description 8
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 2
- 238000003384 imaging method Methods 0.000 abstract description 39
- 238000011084 recovery Methods 0.000 abstract description 8
- 238000010276 construction Methods 0.000 abstract description 4
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 239000000523 sample Substances 0.000 description 23
- 238000012360 testing method Methods 0.000 description 10
- 238000000638 solvent extraction Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000001493 electron microscopy Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000004626 scanning electron microscopy Methods 0.000 description 6
- 230000008447 perception Effects 0.000 description 5
- 210000004027 cell Anatomy 0.000 description 4
- 230000007547 defect Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 239000012472 biological sample Substances 0.000 description 3
- 239000003814 drug Substances 0.000 description 3
- 238000002595 magnetic resonance imaging Methods 0.000 description 3
- 238000003801 milling Methods 0.000 description 3
- 230000000737 periodic effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000002591 computed tomography Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000008014 freezing Effects 0.000 description 2
- 238000007710 freezing Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000010884 ion-beam technique Methods 0.000 description 2
- 238000000386 microscopy Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 210000000056 organ Anatomy 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 239000011347 resin Substances 0.000 description 2
- 229920005989 resin Polymers 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 210000001519 tissue Anatomy 0.000 description 2
- 238000003325 tomography Methods 0.000 description 2
- 238000004627 transmission electron microscopy Methods 0.000 description 2
- 230000016776 visual perception Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000001218 confocal laser scanning microscopy Methods 0.000 description 1
- 238000005314 correlation function Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000018044 dehydration Effects 0.000 description 1
- 238000006297 dehydration reaction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 238000003622 knife milling Methods 0.000 description 1
- 238000001000 micrograph Methods 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 238000013421 nuclear magnetic resonance imaging Methods 0.000 description 1
- 238000010827 pathological analysis Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000001878 scanning electron micrograph Methods 0.000 description 1
- 238000010186 staining Methods 0.000 description 1
- 210000004895 subcellular structure Anatomy 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000946 synaptic effect Effects 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 238000011282 treatment Methods 0.000 description 1
- 230000037303 wrinkles Effects 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Image Analysis (AREA)
Abstract
The invention provides a self-supervision deep learning and a method for constructing an isotropic super-resolution three-dimensional image, which relates to the technical fields of three-dimensional imaging, deep learning, image processing, artificial intelligence and the like, and comprises the following steps: acquiring an anisotropic three-dimensional image; determining a training set and a verification set based on the anisotropic three-dimensional image; training the initial preset model through the training set and the verification set to obtain a plurality of verification losses and a plurality of intermediate models corresponding to the verification losses; the target model is used for obtaining an isotropic image block, and the isotropic image block is used for constructing an isotropic super-resolution three-dimensional image. The self-supervision deep learning and isotropic super-resolution three-dimensional image construction method provided by the invention is used for improving the accuracy of the model obtained through training in high-power isotropic recovery, and further improving the isotropy of the three-dimensional image obtained through the model.
Description
Technical Field
The invention relates to the technical fields of three-dimensional imaging, deep learning, image processing, artificial intelligence and the like, in particular to a method for self-supervision deep learning and construction of an isotropic super-resolution three-dimensional image.
Background
In the fields of biology and medicine, three-dimensional imaging technology is widely applied to structure and morphology research of biological samples with different dimensions such as cells, tissues, organs and the like, and currently, the three-dimensional imaging technology comprises: electron microscopy imaging techniques. The electron microscopy imaging technique includes: serial ultra-thin slice transmission electron microscopy imaging technique (Serial section transmission electron microscopy, ssTEM), serial ultra-thin slice scanning electron microscopy imaging technique (Serial section scanning electron microscopy, ssSEM), serial Block surface scanning electron microscopy imaging technique (Serial Block-Face scanning electron microscopy, SBF-SEM). Wherein the SBF-SEM performs three-dimensional imaging based on the surface of the continuous block to obtain a three-dimensional image, and the ssTEM and ssSEM perform three-dimensional imaging based on the continuous slice to obtain a three-dimensional image. The resolution of the three-dimensional images obtained by SBF-SEM, ssTEM and ssSEM exhibit anisotropy.
In order to make the resolution of a three-dimensional image isotropic, in the related art, there is provided an isotropic super-resolution reconstruction method based on deep learning in which a model is trained using an anisotropic low-resolution three-dimensional image by using the isotropic high-resolution three-dimensional image corresponding to the anisotropic low-resolution three-dimensional image as the supervision data of the model to obtain a model that can be used to obtain a three-dimensional image whose resolution exhibits isotropy.
In the related art, since it is difficult to acquire an isotropic high-resolution three-dimensional image, the number of the supervision data is insufficient, and it is difficult to apply to actual imaging; in addition, model design has limitations that make the accuracy of the trained model in high-power isotropic recovery poor, resulting in poor isotropy of the three-dimensional image obtained by the model.
Disclosure of Invention
The invention provides a self-supervision deep learning and isotropic super-resolution three-dimensional image construction method, which is used for solving the defects that in the prior art, the accuracy of a model obtained by training is poor due to the fact that the limitation on the design of the model is caused, and the isotropy difference of a three-dimensional image obtained by the model is poor, and improving the accuracy of the model obtained by training in high-power isotropy recovery is realized, so that the isotropy of the three-dimensional image obtained by the model is improved.
In a first aspect, the present invention provides a self-supervised deep learning method, including:
acquiring an anisotropic three-dimensional image;
determining a training set and a verification set based on the anisotropic three-dimensional image;
training the initial preset model through the training set and the verification set to obtain a plurality of verification losses and a plurality of intermediate models corresponding to the verification losses;
Determining a target model based on the plurality of validation losses and the plurality of intermediate models; the target model is used for obtaining an isotropic image block, and the isotropic image block is used for constructing an isotropic super-resolution three-dimensional image.
Optionally, determining the training set and the validation set based on the anisotropic three-dimensional image includes:
according to a first preset proportion, performing blocking processing on the anisotropic three-dimensional image to obtain a training image block and a residual image block; according to a second preset proportion, performing block processing on the rest image blocks to obtain verification image blocks;
partitioning the training image blocks according to a preset size to obtain a training set, and partitioning the verification image blocks to obtain a verification set; wherein the training set and the validation set include a plurality of anisotropic tiles having a preset size.
Optionally, the training set includes a plurality of training batches, each training batch includes a preset number of anisotropic tiles, the verification set includes a plurality of verification batches, and each verification batch includes a preset number of anisotropic tiles;
training the initial preset model through the training set and the verification set to obtain an (i+1) th verification loss and an intermediate model corresponding to the (i+1) th verification loss, wherein the training set and the verification set comprise the following steps:
Processing an mth training batch included in the training set through a generator in the intermediate model corresponding to the ith verification loss to obtain an mth first output batch; the method comprises the steps that an intermediate model corresponding to an ith verification loss is obtained by training an initial preset model for i times, i is an integer which is more than or equal to 1 and less than or equal to the total number of a plurality of verification losses, and m is an integer which is more than or equal to 1 and less than or equal to the total number of a plurality of training batches;
determining an unsupervised loss based on the mth training batch and the mth first output batch;
performing downsampling processing on anisotropic tiles in the mth training batch along an X axis or a Y axis of the tiles to obtain the mth preset processing batch;
processing the mth preset processing batch through a generator to obtain an mth second output batch;
determining a supervised loss based on the mth training batch and the mth second output batch;
based on the unsupervised loss and the supervised loss, adjusting model parameters of an intermediate model corresponding to the ith verification loss to obtain an intermediate model corresponding to the (i+1) th verification loss;
determining an nth second output batch corresponding to the nth verification batch in the verification set; wherein n is an integer greater than or equal to 1 and less than or equal to the total number of the plurality of validation batches;
An i+1th validation loss is determined based on the nth validation batch and the nth second output batch.
Optionally, determining an unsupervised loss based on the mth training batch and the mth first output batch comprises:
determining a symmetry loss based on the mth training batch and the mth first output batch;
determining a smoothness regular loss based on the mth first output batch;
the sum of symmetry loss and smoothness canonical loss is determined as an unsupervised loss.
Optionally, determining the symmetry loss based on the mth training batch and the mth first output batch comprises:
downsampling the isotropic image block in the m first output batch along the X axis, the Y axis or the Z axis of the image block to obtain a first false sample batch;
the method comprises the steps of performing discrimination processing on an mth training batch and a first false sample batch through a first discriminator in an intermediate model corresponding to an ith verification loss to obtain anisotropic symmetry loss;
performing rotation processing of a preset angle on isotropic tiles in the mth first output batch along the X axis or the Y axis of the tiles to obtain a second false sample batch;
the first output batch and the second false sample batch of the mth are subjected to discrimination processing through a second discriminator in the intermediate model corresponding to the ith verification loss, so that isotropic symmetry loss is obtained;
The sum of the anisotropic symmetry loss and the isotropic symmetry loss is determined as the symmetry loss.
Optionally, determining the smoothness regular loss based on the mth first output lot includes:
determining a gray-scale mean square error between two adjacent two-dimensional images of an isotropic tile included in an mth first output lot in a Z-axis direction;
performing fast Fourier transform processing on all the obtained gray level mean square errors to obtain a plurality of frequencies and a plurality of amplitude values corresponding to the frequencies;
determining the sum of amplitude values corresponding to the target frequency in the amplitude values as a step artifact loss; the target frequency is a frequency corresponding to a preset multiple in the plurality of frequencies, and the preset multiple is a specific multiple for reconstructing an isotropic super-resolution three-dimensional image based on the anisotropic three-dimensional image;
determining the total variation TV value of the mth first output batch as the image content smoothing loss;
the sum of the stepwise artifact loss and the image content smoothing loss is determined as a smoothness canonical loss.
Optionally, determining the supervised loss based on the mth training batch and the mth second output batch comprises:
determining a pixel gray value error between the mth training batch and the mth second output batch as a pixel level loss;
Feature extraction is carried out on the mth training batch through a semantic feature extraction model to obtain a first feature vector, and feature extraction is carried out on the mth second output batch to obtain a second feature vector;
determining a perceived level loss between the first feature vector and the second feature vector;
the sum of the pixel level loss and the perceived level loss is determined as a supervised loss.
Optionally, determining the target model based on the plurality of validation losses and the plurality of intermediate models includes:
determining a minimum validation loss of the plurality of validation losses;
and determining an intermediate model with the minimum verification loss from the plurality of intermediate models as a target model.
In a second aspect, the present invention provides a method for constructing an isotropic super-resolution three-dimensional image, including:
acquiring a three-dimensional image to be reconstructed;
performing blocking processing on the three-dimensional image to be reconstructed according to a preset size to obtain a plurality of anisotropic image blocks;
processing the anisotropic image blocks through a target model to obtain isotropic image blocks;
and based on the positions of the anisotropic image blocks, splicing the isotropic image blocks to obtain an isotropic super-resolution three-dimensional image.
In a third aspect, the present invention provides a self-supervised deep learning apparatus, comprising:
The first acquisition module is used for acquiring an anisotropic three-dimensional image;
a first determining module for determining a training set and a validation set based on the anisotropic three-dimensional image;
the training module is used for training the initial preset model through the training set and the verification set to obtain a plurality of verification losses and a plurality of intermediate models corresponding to the verification losses;
a second determination module for determining a target model based on the plurality of validation losses and the plurality of intermediate models; the target model is used for obtaining an isotropic image block, and the isotropic image block is used for constructing an isotropic super-resolution three-dimensional image.
Optionally, the first determining module is specifically configured to:
according to a first preset proportion, performing blocking processing on the anisotropic three-dimensional image to obtain a training image block and a residual image block; according to a second preset proportion, performing block processing on the rest image blocks to obtain verification image blocks;
partitioning the training image blocks according to a preset size to obtain a training set, and partitioning the verification image blocks to obtain a verification set; wherein the training set and the validation set include a plurality of anisotropic tiles having a preset size.
Optionally, the training set includes a plurality of training batches, each training batch includes a preset number of anisotropic tiles, the verification set includes a plurality of verification batches, and each verification batch includes a preset number of anisotropic tiles;
The training module is specifically used for:
training the initial preset model through the training set and the verification set to obtain an (i+1) th verification loss and an intermediate model corresponding to the (i+1) th verification loss, wherein the training set and the verification set comprise the following steps:
processing an mth training batch included in the training set through a generator in the intermediate model corresponding to the ith verification loss to obtain an mth first output batch; the method comprises the steps that an intermediate model corresponding to an ith verification loss is obtained by training an initial preset model for i times, i is an integer which is more than or equal to 1 and less than or equal to the total number of a plurality of verification losses, and m is an integer which is more than or equal to 1 and less than or equal to the total number of a plurality of training batches;
determining an unsupervised loss based on the mth training batch and the mth first output batch;
performing downsampling processing on anisotropic tiles in the mth training batch along an X axis or a Y axis of the tiles to obtain the mth preset processing batch;
processing the mth preset processing batch through a generator to obtain an mth second output batch;
determining a supervised loss based on the mth training batch and the mth second output batch;
based on the unsupervised loss and the supervised loss, adjusting model parameters of an intermediate model corresponding to the ith verification loss to obtain an intermediate model corresponding to the (i+1) th verification loss;
Determining an nth second output batch corresponding to the nth verification batch in the verification set; wherein n is an integer greater than or equal to 1 and less than or equal to the total number of the plurality of validation batches;
an i+1th validation loss is determined based on the nth validation batch and the nth second output batch.
Optionally, the training module is specifically configured to:
determining a symmetry loss based on the mth training batch and the mth first output batch;
determining a smoothness regular loss based on the mth first output batch;
the sum of symmetry loss and smoothness canonical loss is determined as an unsupervised loss.
Optionally, the training module is specifically configured to:
downsampling the isotropic image block in the m first output batch along the X axis, the Y axis or the Z axis of the image block to obtain a first false sample batch;
the method comprises the steps of performing discrimination processing on an mth training batch and a first false sample batch through a first discriminator in an intermediate model corresponding to an ith verification loss to obtain anisotropic symmetry loss;
performing rotation processing of a preset angle on isotropic tiles in the mth first output batch along the X axis or the Y axis of the tiles to obtain a second false sample batch;
The first output batch and the second false sample batch of the mth are subjected to discrimination processing through a second discriminator in the intermediate model corresponding to the ith verification loss, so that isotropic symmetry loss is obtained;
the sum of the anisotropic symmetry loss and the isotropic symmetry loss is determined as the symmetry loss.
Optionally, the training module is specifically configured to:
determining a gray-scale mean square error between two adjacent two-dimensional images of an isotropic tile included in an mth first output lot in a Z-axis direction;
performing fast Fourier transform processing on all the obtained gray level mean square errors to obtain a plurality of frequencies and a plurality of amplitude values corresponding to the frequencies;
determining the sum of amplitude values corresponding to the target frequency in the amplitude values as a step artifact loss; the target frequency is a frequency corresponding to a preset multiple in the plurality of frequencies, and the preset multiple is a specific multiple for reconstructing an isotropic super-resolution three-dimensional image based on the anisotropic three-dimensional image;
determining the total variation TV value of the mth first output batch as the image content smoothing loss;
the sum of the stepwise artifact loss and the image content smoothing loss is determined as a smoothness canonical loss.
Optionally, the training module is specifically configured to:
determining a pixel gray value error between the mth training batch and the mth second output batch as a pixel level loss;
feature extraction is carried out on the mth training batch through a semantic feature extraction model to obtain a first feature vector, and feature extraction is carried out on the mth second output batch to obtain a second feature vector;
determining a perceived level loss between the first feature vector and the second feature vector;
the sum of the pixel level loss and the perceived level loss is determined as a supervised loss.
Optionally, the second determining module is specifically configured to:
determining a minimum validation loss of the plurality of validation losses;
and determining an intermediate model with the minimum verification loss from the plurality of intermediate models as a target model.
In a fourth aspect, the present invention provides an apparatus for constructing an isotropic super-resolution three-dimensional image, comprising:
the second acquisition module is used for acquiring a three-dimensional image to be reconstructed;
the processing module is used for carrying out blocking processing on the three-dimensional image to be reconstructed according to the preset size to obtain a plurality of anisotropic image blocks;
the processing module is also used for processing the anisotropic image blocks through the target model to obtain isotropic image blocks;
The processing module is further used for splicing the isotropic image blocks based on the positions of the anisotropic image blocks to obtain an isotropic super-resolution three-dimensional image.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes any one of the self-supervision deep learning method or the method for constructing the isotropic super-resolution three-dimensional image when executing the program.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements any of the self-supervised deep learning methods described above or a method of constructing an isotropic super-resolution three-dimensional image.
The invention also provides a computer program product comprising a computer program which when executed by a processor implements any of the self-supervised deep learning methods described above or the method of constructing an isotropic super-resolution three-dimensional image.
The invention provides a self-supervision deep learning and method for constructing isotropic super-resolution three-dimensional images, wherein in the self-supervision deep learning method, a training set and a verification set are determined based on anisotropic three-dimensional images; training the initial preset model through the training set and the verification set to obtain a plurality of verification losses and a plurality of intermediate models corresponding to the verification losses; based on a plurality of verification losses and a plurality of intermediate models, a target model is determined, an isotropic high-resolution three-dimensional image corresponding to the anisotropic low-resolution three-dimensional image is not required to be used as supervision data of the model, the threshold and the actual workload of data collection are reduced, the self-supervision deep learning method is simpler and more efficient, the accuracy of the model obtained through training in high-power isotropy recovery is improved, and isotropy of the three-dimensional image obtained through the model is further improved.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a self-supervised deep learning method provided by the invention;
FIG. 2 is a schematic diagram of an initial default model provided by the present invention;
FIG. 3 is a schematic flow chart of obtaining the (i+1) th verification loss and the (i+1) th verification loss corresponding intermediate model according to the present invention;
FIG. 4 is a schematic flow chart of determining symmetry loss provided by the present invention;
FIG. 5 is a schematic flow chart of determining a smoothness canonical loss provided by the invention;
FIG. 6 is a schematic illustration of a stepwise artifact in a two-dimensional cross-section parallel to the axial direction provided by the present invention;
FIG. 7 is a schematic flow chart of determining a supervised loss provided by the present invention;
FIG. 8 is a schematic representation of a two-dimensional cross-section of pre-and post-reconstruction tiles provided by the present invention in a direction parallel to the axial direction;
FIG. 9 is a flow chart of a method of constructing an isotropic super-resolution three-dimensional image provided by the present invention;
FIG. 10 is a schematic representation of a two-dimensional cross-section of a pre-and post-reconstruction three-dimensional image provided by the present invention in a direction parallel to the axial direction;
FIG. 11 is a schematic diagram of a self-supervised deep learning apparatus according to the present invention;
FIG. 12 is a schematic structural view of an apparatus for constructing an isotropic super-resolution three-dimensional image according to the present invention;
fig. 13 is a schematic diagram of the physical structure of the electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The technical terms referred to in the present application will be first described.
Isotropy means that the resolution of the three-dimensional image in the X-axis, Y-axis and Z-axis is the same.
Anisotropy indicates that there is a disparity in the resolution of the three-dimensional image in the X-axis, Y-axis, and Z-axis.
The axial direction represents the Z-axis direction.
Transverse, perpendicular to the axial direction, including the X-axis direction and the Y-axis direction.
In the fields of biology and medicine, three-dimensional imaging technology is widely applied to the structure and morphology research of biological samples with different dimensions such as cells, tissues, organs and the like. Three-dimensional imaging mainly includes: confocal laser scanning fluorescence microscopy imaging, light sheet microscopy imaging, volume electron microscopy imaging, computed tomography (Computed Tomography, CT) imaging, magnetic resonance imaging (Magnetic Resonance Imaging, MRI), and the like.
The electron microscope imaging technique mainly comprises the following 5 kinds:
serial slice transmission electron tomography (Serial section electron tomography, ssET);
focused ion beam scanning electron microscopy imaging technique (Focused Ion Beam Scanning Electron Microscopy, FIB-SEM);
ssTEM；
ssSEM；
SBF-SEM。
the above 5 techniques have advantages and disadvantages in terms of imaging volume, resolution, time consumption, stability and other parameters, and these parameters are difficult to optimize at the same time, so in practical use, the parameters are usually chosen as a compromise as required.
For the study of electron microscopy, the isotropy of the resolution of the three-dimensional image is critical. Of the 5 techniques described above, ssET and FIB-SEM can achieve isotropy, and SBF-SEM, ssTEM, ssSEM cannot.
The reason why SBF-SEM, ssTEM, ssSEM cannot achieve isotropy is that:
the SBF-SEM images the surface of a continuous block obtained by milling with a hot knife, and because of the lower limit of the mechanical thickness of the milling, the imaging sampling rate of the axial direction is lower than that of the transverse direction, so that the resolution of the axial direction is lower than that of the transverse direction, and the anisotropy of the image structure information is caused;
the ssTEM and ssSEM image based on serial slices obtained from serial ultrathin sections, because of the lower limit of the mechanical thickness of the slice, the imaging sampling rate in the axial direction is lower than the imaging sampling rate in the transverse direction, resulting in lower resolution in the axial direction than in the transverse direction, resulting in anisotropy of the image structure information.
In tissue cell ultrastructural studies, the synaptic cleft between neurons is typically 5-10 nanometers (nm), the dimensions of the interaction sites between subcellular structures are typically less than 30nm, and these ultrastructures are rich in detail and morphologically complex. The resolution in the axial direction (Z-axis) being smaller than the resolution in the transverse direction (X-axis and Y-axis) by a preset factor (e.g. several times or tens of times) results in structural details being well visible in the transverse direction and being severely missing in the axial direction, which presents challenges and obstacles for three-dimensional reconstruction and analysis of ultrastructures, and related biomedical research. Therefore, reconstructing an isotropic super-resolution three-dimensional image is critical to the reconstruction and analysis of complex ultrastructures.
The existing isotropic super-resolution reconstruction method comprises the following steps: classical interpolation algorithm, and deep learning-based homodromous super-resolution reconstruction method.
Classical interpolation algorithms perform bi-linear or bi-cubic interpolation along the axis of the three-dimensional image (i.e., numerical fitting of the neighborhood space based on the known image discrete site gray values) to fill in axially missing information and improve axial resolution. The method has the advantages of simple and efficient algorithm and is also a common technology for researchers at present. The method has the defects that approximate fitting and filling are only carried out based on local image gray values, global context information and transverse high-resolution microstructure information of a three-dimensional image are not fully utilized, and compared with a true value, interpolation results still have more artifacts (such as axial blurring, structural distortion and the like), and the method is more obvious particularly in high-power isotropic reconstruction, and has more artifacts in the high-power isotropic reconstruction.
According to the method for reconstructing the isotropic super-resolution based on the deep learning, the isotropic high-resolution three-dimensional image corresponding to the anisotropic low-resolution three-dimensional image is used as the supervision data of the model, and the model is trained by adopting the anisotropic low-resolution three-dimensional image so as to obtain the model which can be used for obtaining the three-dimensional image with isotropic resolution. Because the hot knife milling or the continuous ultrathin section can cause irreversible damage to the biological sample, high-low resolution images (namely paired anisotropic low-resolution three-dimensional images and isotropic high-resolution three-dimensional images) paired along the axial direction cannot be obtained, the quantity of supervision data is insufficient, so that the accuracy of a model obtained through training is poor, and the model is difficult to apply to actual imaging; in addition, model design has limitations that result in poor accuracy of the trained model in high-power isotropic recovery, resulting in poor isotropy of the three-dimensional image obtained by the model (e.g., resulting in a three-dimensional image having a much smaller resolution in the axial direction (e.g., Z-axis) than in the lateral direction (e.g., X-axis and Y-axis). Moreover, the method is easy to generate artifacts in high-power (large difference of resolution in transverse and axial directions) isotropic reconstruction tasks.
In the application, in order to improve the accuracy of a model in high-power isotropy recovery and further improve isotropy of a three-dimensional image, the invention provides a self-supervision deep learning method.
The self-supervision deep learning method provided by the invention is described below with reference to specific embodiments.
Fig. 1 is a schematic flow chart of a self-supervised deep learning method provided by the invention. As shown in fig. 1, the method includes:
and step 101, acquiring an anisotropic three-dimensional image.
Optionally, the execution subject of the self-supervised deep learning method provided by the invention may be an electronic device, or may be a self-supervised deep learning device disposed in the electronic device, where the self-supervised deep learning device may be implemented by a combination of software and/or hardware.
The electronic device is, for example, a supercomputer, an industrial control computer, a network computer, a personal computer, or the like.
Alternatively, an anisotropic three-dimensional image can be obtained by the following processes (1-1 to 1-6).
(1-1) obtaining a resin-embedded block including the sample object.
The sample object may be, for example, a cell, a tissue, or the like.
Specifically, the resin-embedded block is obtained by subjecting the sample object to treatments such as fixation (one of chemical fixation and high-pressure freezing is selected), staining, dehydration, resin permeation, and the like.
The time required to obtain the resin-embedded block is typically 1 to 2 weeks.
(1-2) continuously ultrathin slicing the resin-embedded block by using an automatic tape collecting ultrathin slicer (ATUM or AutoCuts) to obtain continuous slices; or milling the resin embedded block by a built-in hot knife in the imaging chamber to obtain a continuous block surface.
(1-3), continuous slice or continuous block surface screening to reduce samples having wrinkles, damage, cuts, chatter, etc. defects.
(1-4) systematic inspection of the surface of the remaining serial sections or blocks after the screening of (1-3) by a three-dimensional electron microscope, in preparation for imaging.
(1-5) scanning electron microscope imaging or transmission electron microscope imaging is carried out on the continuous slice, and a large number of two-dimensional series images are obtained; or scanning electron microscope imaging is carried out on the surface of the continuous block to obtain a large number of two-dimensional series images.
And (1-6) acquiring a large number of two-dimensional images and preprocessing to obtain an anisotropic three-dimensional image.
Wherein the preprocessing comprises the following steps: splicing two-dimensional images belonging to the same continuous slice or the surface of the same continuous block, and then registering and aligning along the axial direction to obtain a three-dimensional image; and (3) performing quality inspection on the three-dimensional image, and if the three-dimensional image is in a storage defect, a stitching error and the like, returning to (1-5) for complement shooting, or replacing a problem area in the three-dimensional image by using an average image of image areas corresponding to adjacent slices, so as to finally obtain the anisotropic three-dimensional image.
Since a large-area slice or block surface is generally imaged in blocks with a certain overlapping area, it is necessary to stitch two-dimensional images belonging to the same continuous slice or the same continuous block surface.
Optionally, according to a first preset proportion, performing block processing on the anisotropic three-dimensional image to obtain a training block and a residual block;
according to a second preset proportion, performing block processing on the rest image blocks to obtain verification image blocks;
partitioning the training image blocks according to a preset size to obtain a training set, and partitioning the verification image blocks to obtain a verification set; wherein the training set and the validation set include a plurality of anisotropic tiles having a preset size.
Alternatively, the sum of the first preset ratio and the second preset ratio may be equal to 1 or may be smaller than 1.
And under the condition that the sum of the first preset proportion and the second preset proportion is smaller than 1, performing block processing on the rest image blocks according to the second preset proportion, and further obtaining the test image blocks.
Furthermore, the test image blocks can be partitioned according to a preset size to obtain a test set, and the test set comprises a plurality of anisotropic image blocks with the preset size.
For example, the size ratio of the training, validation and test tiles is 75%:15%:15%. I.e. 75% of the first preset proportion and 15% of the second preset proportion.
And step 103, training the initial preset model through the training set and the verification set to obtain a plurality of verification losses and a plurality of intermediate models corresponding to the verification losses.
Alternatively, the initial pre-set model is shown in fig. 2. Fig. 2 is a schematic structural diagram of an initial preset model provided by the present invention. As shown in fig. 2, the initial preset model includes: a generator (G), a first arbiter (D1) and a first arbiter (D2).
Alternatively, G may be a generator configured based on a 3D neural network, D1 may be a arbiter configured based on a 3D convolutional neural network, and D2 may be a arbiter configured based on a 3D neural network.
Optionally, specific network architectures for G, D1 and D2 include, but are not limited to: 3D UNet, 3D FSRCNN, EDVR, basic VSR, video Transformer, etc.
The plurality of validation losses includes all validation losses obtained from the beginning of training the initial pre-set model to the end of training. At the end of training, the parametric model of generator G converges.
For a detailed description of step 103, please refer to the embodiment of fig. 3, which is not described in detail herein.
104, determining a target model based on the verification losses and the intermediate models; the target model is used for obtaining an isotropic image block, and the isotropic image block is used for constructing an isotropic super-resolution three-dimensional image.
Specifically, determining a minimum verification loss of the plurality of verification losses; and determining an intermediate model with the minimum verification loss from the plurality of intermediate models as a target model.
Alternatively, the target model may also be tested by a test set. Specifically, a plurality of anisotropic tiles in a test set can be divided into batches (batches) according to a preset number to obtain a plurality of batches, and for each batch, the batch is input into a generator of a target model to obtain an output batch; one anisotropic tile in the batch corresponds to one isotropic tile in the output batch;
after traversing all batches, based on the positions of all anisotropic tiles, splicing the isotropic tiles in all output batches to obtain reconstructed tiles corresponding to the test set.
Optionally, the method may further include: determining one or more of the following parameters between the test tile and the reconstructed tile:
peak signal to noise ratio (Peak Signal to Noise Ratio, PSNR);
Structural similarity index (Structure Similarity Index Measure, SSIM);
multiscale structural similarity index (Multi-Scale Structure Similarity Index Measure, MS-SSIM);
perceived tile similarity (Learned Perceptual Image Patch Similarity, LPIPS);
fourier shell correlation functions (Fourier Shell Correlation, FSC); or,
determining one or more of the following parameters of the isotropically reconstructed tile:
optionally, it may also include: determining one or more of the following parameters of the reconstructed tile:
periodic stepwise artifacts;
total Variation (TV) values;
full width at half maximum (Full Width at Half Maxima, FWHM) of a particular microstructure.
Wherein, PSNR belongs to pixel level index, tends to be used for measuring gray value error of the image; SSIM and MS-SSIM belong to structural indexes and tend to be used for measuring the distortion degree of ultrastructural distortion; LPIPS belongs to a perception level index, and tends to be used for measuring visual perception differences of human eyes; TV is a common indicator for measuring smoothness of image content.
FSC is a resolution index commonly used by a freezing electron microscope and a fluorescence microscope, FWHM is a resolution index commonly used by a fluorescence microscope, and the invention expands the two resolution indexes into an electron microscope image resolution scene.
In the self-supervision deep learning method provided by the invention, a training set and a verification set are determined based on an anisotropic three-dimensional image; training the initial preset model through the training set and the verification set to obtain a plurality of verification losses and a plurality of intermediate models corresponding to the verification losses; based on a plurality of verification losses and a plurality of intermediate models, a target model is determined, an isotropic high-resolution three-dimensional image corresponding to the anisotropic low-resolution three-dimensional image is not required to be used as supervision data of the model, the threshold and the actual workload of data collection are reduced, the self-supervision deep learning method is simpler and more efficient, the accuracy of the model obtained through training in high-power isotropy recovery is improved, and isotropy of the three-dimensional image obtained through the model is further improved.
The training set comprises a plurality of training batches, and each training batch comprises a preset number of anisotropic tiles.
The verification set comprises a plurality of verification batches, and each verification batch comprises a preset number of anisotropic tiles.
Alternatively, the preset number may be 50, 100, etc., which is not limited in this application.
In the process of training the initial preset model through the training set and the verification set to obtain a plurality of verification losses and a plurality of intermediate models corresponding to the verification losses, one training and one verification can be performed through one training batch and one verification batch, so that one verification loss and the intermediate model corresponding to the verification loss are obtained.
Step 103 will be described in detail below with reference to fig. 3 by taking an example of a verification loss and an intermediate model corresponding to the verification loss.
FIG. 3 is a schematic flow chart of the present invention for obtaining the (i+1) th verification loss and the (i+1) th verification loss corresponding intermediate model. As shown in fig. 3, the method includes:
The intermediate model corresponding to the ith verification loss is obtained by training the initial preset model for i times, and i is an integer which is more than or equal to 1 and less than or equal to the total number of the verification losses.
m is an integer greater than or equal to 1 and less than or equal to the total number of the plurality of training batches.
The mth training lot is a training lot that is cycled through from multiple training lots during the i+1 training sessions.
Optionally, determining a symmetry loss based on the mth training batch and the mth first output batch; determining a smoothness regular loss based on the mth first output batch; the sum of symmetry loss and smoothness canonical loss is determined as an unsupervised loss.
Alternatively, the symmetry loss may be obtained by the method shown in FIG. 4 below, and the smoothness canonical loss may be obtained by the method shown in FIG. 5 below, which is not described in detail herein.
For example, along the X-axis of the block, the anisotropic block in the mth training batch is downsampled to obtain second-order anisotropic image blocks (Z-LR, Y-HR, X-LR) in the mth preset processing batch. Where LR denotes low resolution and HR denotes high resolution.
For example, along the Y-axis of the tile, downsampling the anisotropic tile in the mth training batch to obtain second-order anisotropic image blocks (Z-LR, Y-LR, X-HR) in the mth preset processing batch
Alternatively, the anisotropic tiles in the mth training batch may be downsampled by:
average downsampling;
drawing frames;
bicubic downsampling.
Optionally, the downsampling approach requires that the mode of the three-dimensional data (i.e., the mth training batch) be consistent in principle (e.g., a mode corresponding to FIB-SEM, ssTEM, ssSEM or SBF-SEM imaging techniques).
And 304, processing the mth preset processing batch through a generator in the intermediate model corresponding to the ith verification loss to obtain an mth second output batch.
For example, in the case of downsampling the anisotropic tiles in the mth training batch along the tile X-axis, the mth second output batch includes a plurality of first order anisotropic image blocks (Z-LR, Y-HR, X-HR).
Specifically, for a detailed description of step 305, please refer to the embodiment of fig. 6.
And 306, adjusting model parameters of the intermediate model corresponding to the ith verification loss based on the unsupervised loss and the supervised loss to obtain the intermediate model corresponding to the (i+1) th verification loss.
n is an integer greater than or equal to 1 and less than or equal to the total number of the plurality of validation batches.
Alternatively, the nth second output lot corresponding to the nth verification lot may be determined by the methods of steps 303 to 304, which are not described herein.
Alternatively, the (i+1) th verification loss may be determined by: determining a pixel gray value error between the nth verification batch and the nth second output batch; performing feature extraction on the nth verification batch through a semantic feature extraction model to obtain a third feature vector, and performing feature extraction on the nth second output batch to obtain a fourth feature vector; determining a perceptual level penalty between the third feature vector and the fourth feature vector; the sum of the pixel gray value error and the perceived level loss is determined as the i+1th verification loss.
Alternatively, the pixel gray value error between the nth verification batch and the nth second output batch may be obtained by an average absolute value error (L1 loss) algorithm or a mean square error (L2 loss) algorithm.
Alternatively, the semantic feature extraction model may be a pre-trained convolutional neural network (e.g., VGGNet, resNet, etc.).
Alternatively, the perceptual level penalty between the third feature vector and the fourth feature vector may be derived by an average absolute value error (L1 penalty) algorithm or a mean square error (L2 penalty) algorithm.
In the invention, as the resolution of the image block along the axial direction is lower, the anisotropic image block in the mth training batch is downsampled along the transverse direction (X axis or Y axis) of the image block, so that the resolution of the anisotropic image block in the mth preset processing batch is also lower, and the difference between the resolutions of the anisotropic image block in the mth preset processing batch in the axial direction and the transverse direction is reduced. Further, the mth preset processing batch is processed to obtain an mth second output batch, and the mth second output batch can be used as supervision data of the mth training batch, so that supervision loss is obtained.
In addition, the self-supervision deep learning method provided by the invention has good self-adaptation capability on specific data of different image contents in different modes, namely, model parameters aiming at the data can be learned in a training stage, the structural semantic features of the image can be effectively extracted and reconstruction can be completed, good reconstruction performance can be achieved under the condition of insufficient training data, and the problem of generalization and migration of the model can be avoided compared with a non-specific data training mode.
Data-specific) data, data for model training and model testing, are derived from the same anisotropic three-dimensional image. The model is trained, for example, by an anisotropic three-dimensional image a, which is tested by the trained model.
FIG. 4 is a schematic flow chart of determining symmetry loss provided by the present invention. As shown in fig. 4, the method includes:
For example, the isotropic tiles in the mth first output lot are downsampled along the X-axis of the tile to obtain anisotropic tiles (Z-HR, Y-HR, X-LR) in the first dummy sample lot.
For example, the isotropic tiles in the mth first output lot are downsampled along the Y-axis of the tile to obtain anisotropic tiles (Z-HR, Y-LR, X-LR) in the first dummy sample lot.
For example, the isotropic tiles in the mth first output lot are downsampled along the Z-axis of the tile to obtain anisotropic tiles (Z-LR, Y-HR, X-HR) in the first dummy sample lot.
Alternatively, the downsampling method may be any of average downsampling, frame extraction, and bicubic downsampling.
And step 402, performing discrimination processing on the mth training batch and the first false sample batch through a first discriminator in the intermediate model corresponding to the ith verification loss to obtain anisotropic symmetry loss.
Alternatively, the preset angle may be 90 degrees.
For example, the isotropic tiles in the mth first output lot are rotated 90 degrees along the X-axis or Y-axis of the tiles to obtain isotropic tiles (Z-HR, Y-HR, X-HR) in the second dummy sample lot.
And 404, judging the mth first output batch and the second false sample batch by a second discriminator in the intermediate model corresponding to the ith verification loss, so as to obtain isotropic symmetry loss.
In the invention, the model parameters of the model are updated based on symmetry loss, so that the reconstructed image can be obtained more truly in high-power isotropy recovery of the model.
FIG. 5 is a flow chart of determining a smoothness canonical loss provided by the invention. As shown in fig. 5, the method includes:
For example, an isotropic tile has L two-dimensional images in the Z-axis direction, and L-1 gray scale mean square errors can be obtained.
The target frequency is a frequency corresponding to a preset multiple among the plurality of frequencies, and the preset multiple is a specific multiple for reconstructing the isotropic super-resolution three-dimensional image based on the anisotropic three-dimensional image.
Alternatively, the preset multiple may be 2, 4, 8, 10, etc.
The stepwise artifact is an artifact which is often generated in an image when super-resolution in the axial direction is reconstructed, and is represented by, for example, a case where N times (i.e., a preset multiple) anisotropic data is reconstructed (i.e., super-resolution in the axial direction is reconstructed), and periodic abrupt structural changes (discontinuities) occur in the XZ plane or YZ plane every N pixels in the axial direction. Specifically, please refer to the embodiment of fig. 6. Fig. 6 is a schematic illustration of a stepwise artifact in a two-dimensional cross-section parallel to the axial direction (e.g., XZ plane) provided by the present invention. As can be seen from fig. 6, the pixel points are distributed in a stepwise artifact.
In the above-mentioned multiple amplitude values, the peak value appears on the frequency corresponding to N, the larger the peak value, the more serious the ladder-like artifact, therefore in the invention, the amplitude value corresponding to the target frequency in the multiple amplitude values is determined as the ladder-like artifact loss, the accuracy of training the initial preset model can be improved, and the inhibition degree of the model on the ladder-like artifact is enhanced.
The image content smoothing loss is based on the smoothing priori of the natural image, so that abnormal conditions such as checkerboard artifacts, gray abrupt changes and the like can be prevented from being generated on the XY plane, the YZ plane or the XZ plane by the image blocks in the mth first output batch, the accuracy of training an initial preset model is improved, and the avoidance degree of the model on the abnormal conditions is enhanced.
Fig. 7 is a schematic flow chart of determining a supervised loss provided by the present invention. As shown in fig. 7, the method includes:
Alternatively, the pixel gray value error between the mth training batch and the mth second output batch may be obtained by an average absolute value error (L1 loss) algorithm or a mean square error (L2 loss) algorithm.
Alternatively, the perceptual level penalty between the first feature vector and the second feature vector may be determined by a mean absolute value error (L1 penalty) algorithm or a mean square error (L2 penalty) algorithm.
It should be noted that, the present invention refers to the "sum" of losses multiple times, and may also be to perform weighted summation according to a preset weighting parameter. In practical application, the weighting parameters corresponding to each loss can be set according to the requirements.
In the invention, the supervised loss is the sum of pixel level error and perception level loss, which are in a trade-off relation, if the pixel level loss exists, the perception level error is possibly caused to be larger (the visual perception effect is poor), and if the pixel level loss is taken only, the pixel level error is possibly caused to be larger, so that the sum of the pixel level loss and the perception level loss is determined to be the supervised loss, the pixel level error and the perception level error can be restrained at the same time, and the accuracy of a model obtained through training is improved.
The deep learning model and the loss function designed by the invention have completeness, innovativeness and effectiveness, and have a remarkably better effect than the previous method in a high-isotropy reconstruction scene. The high-power anisotropic three-dimensional image widely exists in the body electron microscopy imaging, is a great technical challenge in the field, can enable the reconstruction result of the ultrastructure to be more complete and more reliable through the design of the invention, and lays a good data foundation for research and solution of life science problems, pathological diagnosis and the like in the fields of connection group science, construction biology, medicine and the like.
Fig. 8 is a schematic view of a two-dimensional cross-section of the pre-and post-reconstruction tiles provided by the present invention in a direction parallel to the axial direction. As shown in fig. 8, includes: a two-dimensional cross section parallel to the axial direction in the anisotropic image block before reconstruction and a two-dimensional cross section parallel to the axial direction in the isotropic image block after reconstruction. The two-dimensional cross section shown in fig. 8 is an XZ plane.
And reconstructing the anisotropic image block through the target model to obtain an isotropic image block.
As can be seen from fig. 8, the anisotropic tile before reconstruction lacks information along the Z-axis. The reconstructed isotropic tiles compensate for the missing information along the Z-axis, revealing more structural detail information.
Fig. 9 is a flow chart of a method for constructing an isotropic super-resolution three-dimensional image according to the present invention. As shown in fig. 9, the method includes:
Alternatively, the execution subject of the method for constructing the isotropic super-resolution three-dimensional image may be an electronic device, or may be a device for constructing the isotropic super-resolution three-dimensional image in the electronic device, where the device for constructing the isotropic super-resolution three-dimensional image may be implemented by a combination of software and/or hardware. The electronic device may be the same as or different from the electronic device in the embodiment of fig. 1 described above.
Alternatively, the three-dimensional image to be reconstructed may be the same as or different from the anisotropic three-dimensional image described above.
And 902, performing blocking processing on the three-dimensional image to be reconstructed according to a preset size to obtain a plurality of anisotropic tiles.
It should be noted that, by the generator in the object model, the plurality of anisotropic tiles are processed to obtain a plurality of isotropic tiles.
And 904, splicing the isotropic image blocks based on the positions of the anisotropic image blocks to obtain an isotropic super-resolution three-dimensional image.
In the method for constructing the isotropic super-resolution three-dimensional image, provided by the invention, the plurality of anisotropic image blocks are processed through the target model to obtain the plurality of isotropic image blocks, and then the plurality of isotropic image blocks are spliced based on the positions of the plurality of anisotropic image blocks to obtain the isotropic super-resolution three-dimensional image, so that the isotropy of the reconstructed three-dimensional image can be improved.
The application range of the method for constructing the isotropic super-resolution three-dimensional image provided by the invention comprises various biomedical field three-dimensional imaging modes including a volume electron microscope, the reconstruction of various preset times of three-dimensional images to be reconstructed can be performed, the wide super-resolution reconstruction requirement can be met, and the method particularly shows more outstanding performance advantages in high-power isotropic reconstruction.
Furthermore, in the application of the body electron microscopy, the problems of high imaging cost, long imaging time, too small imaging volume and the like of isotropic resolution imaging equipment are avoided, the cost (such as time cost and economic cost) for acquiring high-quality three-dimensional imaging data is effectively reduced, and the method has higher application value. In addition, the method for constructing the isotropic super-resolution three-dimensional image can be applied to a wider biomedical imaging mode, including but not limited to fluorescence or light sheet microscopy, electron microscopy, nuclear magnetic resonance imaging and the like.
Optionally, the method for constructing the isotropic super-resolution three-dimensional image provided by the invention further comprises the following steps:
determining one or more reconstruction indexes between the three-dimensional image to be reconstructed and the isotropic super-resolution three-dimensional image:
PSNR；
SSIM；
MS-SSIM；
LPIPS；
FSC。
optionally, the method for constructing the isotropic super-resolution three-dimensional image provided by the invention further comprises the following steps:
determining one or more reconstruction indexes of the isotropic super-resolution three-dimensional image:
periodic stepwise artifacts;
TV values;
FWHM。
fig. 10 is a schematic view of a two-dimensional cross-section of a pre-and post-reconstruction three-dimensional image provided by the present invention in parallel with the axial direction. As shown in fig. 10, includes: a two-dimensional cross section parallel to the axial direction in the anisotropic three-dimensional image before reconstruction and a two-dimensional cross section parallel to the axial direction in the isotropic super-resolution three-dimensional image after reconstruction. It can be seen from fig. 10 that the two-dimensional section parallel to the axial direction lacks information along the Z-axis before reconstruction, and that the two-dimensional section parallel to the axial direction compensates for the missing information along the Z-axis after reconstruction, revealing more structural detail information.
Fig. 11 is a schematic structural diagram of the self-supervised deep learning apparatus provided by the present invention. As shown in fig. 11, the self-supervised deep learning apparatus includes:
a first acquisition module 110 for acquiring an anisotropic three-dimensional image;
a first determining module 120 for determining a training set and a validation set based on the anisotropic three-dimensional image;
the training module 130 is configured to train the initial preset model through a training set and a verification set, so as to obtain a plurality of verification losses and a plurality of intermediate models corresponding to the verification losses;
a second determining module 140 for determining a target model based on the plurality of validation losses and the plurality of intermediate models; the target model is used for obtaining an isotropic image block, and the isotropic image block is used for constructing an isotropic super-resolution three-dimensional image.
Optionally, the first determining module 120 is specifically configured to:
according to a first preset proportion, performing blocking processing on the anisotropic three-dimensional image to obtain a training image block and a residual image block; according to a second preset proportion, performing block processing on the rest image blocks to obtain verification image blocks;
partitioning the training image blocks according to a preset size to obtain a training set, and partitioning the verification image blocks to obtain a verification set; wherein the training set and the validation set include a plurality of anisotropic tiles having a preset size.
Optionally, the training set includes a plurality of training batches, each training batch includes a preset number of anisotropic tiles, the verification set includes a plurality of verification batches, and each verification batch includes a preset number of anisotropic tiles;
the training module 130 is specifically configured to:
training the initial preset model through the training set and the verification set to obtain an (i+1) th verification loss and an intermediate model corresponding to the (i+1) th verification loss, wherein the training set and the verification set comprise the following steps:
processing an mth training batch included in the training set through a generator in the intermediate model corresponding to the ith verification loss to obtain an mth first output batch; the method comprises the steps that an intermediate model corresponding to an ith verification loss is obtained by training an initial preset model for i times, i is an integer which is more than or equal to 1 and less than or equal to the total number of a plurality of verification losses, and m is an integer which is more than or equal to 1 and less than or equal to the total number of a plurality of training batches;
determining an unsupervised loss based on the mth training batch and the mth first output batch;
performing downsampling processing on anisotropic tiles in the mth training batch along an X axis or a Y axis of the tiles to obtain the mth preset processing batch;
Processing the mth preset processing batch through a generator to obtain an mth second output batch;
determining a supervised loss based on the mth training batch and the mth second output batch;
based on the unsupervised loss and the supervised loss, adjusting model parameters of an intermediate model corresponding to the ith verification loss to obtain an intermediate model corresponding to the (i+1) th verification loss;
determining an nth second output batch corresponding to the nth verification batch in the verification set; wherein n is an integer greater than or equal to 1 and less than or equal to the total number of the plurality of validation batches;
an i+1th validation loss is determined based on the nth validation batch and the nth second output batch.
Optionally, the training module 130 is specifically configured to:
determining a symmetry loss based on the mth training batch and the mth first output batch;
determining a smoothness regular loss based on the mth first output batch;
the sum of symmetry loss and smoothness canonical loss is determined as an unsupervised loss.
Optionally, the training module 130 is specifically configured to:
downsampling the isotropic image block in the m first output batch along the X axis, the Y axis or the Z axis of the image block to obtain a first false sample batch;
The method comprises the steps of performing discrimination processing on an mth training batch and a first false sample batch through a first discriminator in an intermediate model corresponding to an ith verification loss to obtain anisotropic symmetry loss;
performing rotation processing of a preset angle on isotropic tiles in the mth first output batch along the X axis or the Y axis of the tiles to obtain a second false sample batch;
the first output batch and the second false sample batch of the mth are subjected to discrimination processing through a second discriminator in the intermediate model corresponding to the ith verification loss, so that isotropic symmetry loss is obtained;
the sum of the anisotropic symmetry loss and the isotropic symmetry loss is determined as the symmetry loss.
Optionally, the training module 130 is specifically configured to:
determining a gray-scale mean square error between two adjacent two-dimensional images of an isotropic tile included in an mth first output lot in a Z-axis direction;
performing fast Fourier transform processing on all the obtained gray level mean square errors to obtain a plurality of frequencies and a plurality of amplitude values corresponding to the frequencies;
determining the sum of amplitude values corresponding to the target frequency in the amplitude values as a step artifact loss; the target frequency is a frequency corresponding to a preset multiple in the plurality of frequencies, and the preset multiple is a specific multiple for reconstructing an isotropic super-resolution three-dimensional image based on the anisotropic three-dimensional image;
Determining the total variation TV value of the mth first output batch as the image content smoothing loss;
the sum of the stepwise artifact loss and the image content smoothing loss is determined as a smoothness canonical loss.
Optionally, the training module 130 is specifically configured to:
determining a pixel gray value error between the mth training batch and the mth second output batch as a pixel level loss;
feature extraction is carried out on the mth training batch through a semantic feature extraction model to obtain a first feature vector, and feature extraction is carried out on the mth second output batch to obtain a second feature vector;
determining a perceived level loss between the first feature vector and the second feature vector;
the sum of the pixel level loss and the perceived level loss is determined as a supervised loss.
Optionally, the second determining module 140 is specifically configured to:
determining a minimum validation loss of the plurality of validation losses;
and determining an intermediate model with the minimum verification loss from the plurality of intermediate models as a target model.
Fig. 12 is a schematic structural diagram of an apparatus for constructing an isotropic super-resolution three-dimensional image according to the present invention. As shown in fig. 12, the apparatus for constructing an isotropic super-resolution three-dimensional image includes:
A second acquisition module 210, configured to acquire a three-dimensional image to be reconstructed;
the processing module 220 is configured to perform block processing on the three-dimensional image to be reconstructed according to a preset size to obtain a plurality of anisotropic tiles;
the processing module 220 is further configured to process the plurality of anisotropic tiles through the target model to obtain a plurality of isotropic tiles;
the processing module 220 is further configured to splice the plurality of isotropic tiles based on the positions of the plurality of anisotropic tiles, so as to obtain an isotropic super-resolution three-dimensional image.
Fig. 13 is a schematic diagram of the physical structure of the electronic device provided by the present invention. As shown in fig. 13, the electronic device may include: processor 310, communication interface (Communications Interface) 320, memory 330 and communication bus 340, wherein processor 310, communication interface 320, memory 330 accomplish communication with each other through communication bus 340. The processor 310 may invoke logic instructions in the memory 330 to perform the self-supervised deep learning method described above or the method of constructing an isotropic super-resolution three-dimensional image.
Further, the logic instructions in the memory 330 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method of the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, being capable of performing the self-supervised deep learning method or the method of constructing an isotropic super-resolution three-dimensional image.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the above-described self-supervised deep learning method or the method of constructing an isotropic super-resolution three-dimensional image.
The apparatus embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on such understanding, the foregoing technical solutions may be embodied essentially or in part in the form of a software product, which may be stored in a computer-readable storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the various embodiments or methods of some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A self-supervised deep learning method, comprising:
acquiring an anisotropic three-dimensional image;
determining a training set and a verification set based on the anisotropic three-dimensional image;
training an initial preset model through the training set and the verification set to obtain a plurality of verification losses and a plurality of intermediate models corresponding to the verification losses;
determining a target model based on the plurality of validation losses and the plurality of intermediate models; the target model is used for obtaining an isotropic image block, and the isotropic image block is used for constructing an isotropic super-resolution three-dimensional image;
the training set comprises a plurality of training batches, each training batch comprises a preset number of anisotropic tiles, the verification set comprises a plurality of verification batches, and each verification batch comprises a preset number of anisotropic tiles;
training the initial preset model through the training set and the verification set to obtain an (i+1) th verification loss and an intermediate model corresponding to the (i+1) th verification loss, wherein the training set and the verification set comprise the following steps:
processing an mth training batch included in the training set through a generator in the intermediate model corresponding to the ith verification loss to obtain an mth first output batch; the intermediate model corresponding to the ith verification loss is obtained by training the initial preset model for i times, i is an integer which is more than or equal to 1 and less than or equal to the total number of the verification losses, and m is an integer which is more than or equal to 1 and less than or equal to the total number of the training batches;
Determining an unsupervised loss based on the mth training batch and the mth first output batch;
performing downsampling processing on the anisotropic image blocks in the mth training batch along the X axis or the Y axis of the image blocks to obtain an mth preset processing batch;
processing the mth preset processing batch through the generator to obtain an mth second output batch;
determining a supervised loss based on the mth training batch and the mth second output batch;
based on the unsupervised loss and the supervised loss, adjusting model parameters of an intermediate model corresponding to the ith verification loss to obtain an intermediate model corresponding to the (i+1) th verification loss;
determining an nth second output batch corresponding to the nth verification batch in the verification set; wherein n is an integer greater than or equal to 1 and less than or equal to the total number of the plurality of validation batches;
the (i+1) th validation loss is determined based on the (n) th validation lot and the (n) th second output lot.
2. The self-supervised deep learning method of claim 1, wherein said determining a training set and a validation set based on said anisotropic three-dimensional image comprises:
According to a first preset proportion, performing block processing on the anisotropic three-dimensional image to obtain a training block and a residual block; according to a second preset proportion, the rest image blocks are subjected to block processing to obtain verification image blocks;
the training image blocks are subjected to blocking processing according to a preset size to obtain the training set, and the verification image blocks are subjected to blocking processing to obtain the verification set; wherein the training set and the validation set include a plurality of anisotropic tiles having the preset size.
3. The self-supervised deep learning method of claim 1, wherein said determining an unsupervised penalty based on said mth training batch and said mth first output batch comprises:
determining a symmetry loss based on the mth training batch and the mth first output batch;
determining a smoothness regular loss based on the mth first output batch;
and determining the sum of the symmetry loss and the smoothness regular loss as the unsupervised loss.
4. The self-supervised deep learning method of claim 3, wherein said determining symmetry loss based on said mth training batch and said mth first output batch comprises:
Downsampling the isotropic image block in the m first output batch along the X axis, the Y axis or the Z axis of the image block to obtain a first false sample batch;
judging the mth training batch and the first false sample batch by a first discriminator in the intermediate model corresponding to the ith verification loss to obtain anisotropic symmetry loss;
performing rotation processing of a preset angle on isotropic tiles in the mth first output batch along the X axis or the Y axis of the tiles to obtain a second false sample batch;
judging the mth first output batch and the second false sample batch by a second discriminator in the intermediate model corresponding to the ith verification loss to obtain isotropic symmetry loss;
and determining the sum of the anisotropic symmetry loss and the isotropic symmetry loss as the symmetry loss.
5. The self-supervised deep learning method of claim 3, wherein said determining a smoothness regularization loss based on said mth first output batch comprises:
determining a gray-scale mean square error between two adjacent two-dimensional images of an isotropic tile included in the mth first output lot in the Z-axis direction;
Performing fast Fourier transform processing on all the obtained gray level mean square errors to obtain a plurality of frequencies and a plurality of amplitude values corresponding to the plurality of frequencies;
determining the sum of amplitude values corresponding to the target frequency in the amplitude values as a step artifact loss; the target frequency is a frequency corresponding to a preset multiple in the plurality of frequencies, and the preset multiple is a specific multiple for reconstructing an isotropic super-resolution three-dimensional image based on the anisotropic three-dimensional image;
determining the total variation TV value of the mth first output batch as the image content smoothing loss;
and determining the sum of the step artifact loss and the image content smoothing loss as the smoothing regular loss.
6. The self-supervised deep learning method of claim 1, wherein said determining a supervised penalty based on said mth training batch and said mth second output batch comprises:
determining a pixel gray value error between the mth training batch and the mth second output batch as a pixel level loss;
performing feature extraction on the mth training batch through a semantic feature extraction model to obtain a first feature vector, and performing feature extraction on the mth second output batch to obtain a second feature vector;
Determining a perceptual level penalty between the first feature vector and the second feature vector;
and determining a sum of the pixel level loss and the perceived level loss as the supervised loss.
7. The self-supervised deep learning method of claim 1 or 2, wherein said determining a target model based on said plurality of validation losses and said plurality of intermediate models comprises:
determining a minimum validation loss of the plurality of validation losses;
and determining an intermediate model with the minimum verification loss from the plurality of intermediate models as the target model.
8. A method of constructing an isotropic super-resolution three-dimensional image, comprising:
acquiring a three-dimensional image to be reconstructed;
performing block processing on the three-dimensional image to be reconstructed according to a preset size to obtain a plurality of anisotropic image blocks;
processing the anisotropic image blocks through a target model to obtain isotropic image blocks; wherein the target model is obtained by the self-supervised deep learning method of any of claims 1 to 7;
and based on the positions of the anisotropic image blocks, splicing the isotropic image blocks to obtain the isotropic super-resolution three-dimensional image.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the self-supervised deep learning method of any of claims 1 to 7 or the method of constructing an isotropic super resolution three dimensional image of claim 8 when the program is executed by the processor.
10. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the self-supervised deep learning method of any of claims 1 to 7 or the method of constructing an isotropic super resolution three dimensional image of claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211689462.XA CN115661377B (en) | 2022-12-28 | 2022-12-28 | Self-supervision deep learning and method for constructing isotropic super-resolution three-dimensional image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211689462.XA CN115661377B (en) | 2022-12-28 | 2022-12-28 | Self-supervision deep learning and method for constructing isotropic super-resolution three-dimensional image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115661377A CN115661377A (en) | 2023-01-31 |
| CN115661377B true CN115661377B (en) | 2023-05-05 |
Family
ID=85022431
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211689462.XA Active CN115661377B (en) | 2022-12-28 | 2022-12-28 | Self-supervision deep learning and method for constructing isotropic super-resolution three-dimensional image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115661377B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117974448A (en) * | 2024-04-02 | 2024-05-03 | 中国科学院自动化研究所 | Three-dimensional medical image isotropy super-resolution method and device |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111507462A (en) * | 2020-04-15 | 2020-08-07 | 华中科技大学鄂州工业技术研究院 | End-to-end three-dimensional medical image super-resolution reconstruction method and system |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3499459A1 (en) * | 2017-12-18 | 2019-06-19 | FEI Company | Method, device and system for remote deep learning for microscopic image reconstruction and segmentation |
| CN113436067B (en) * | 2021-05-22 | 2023-05-09 | 西北工业大学深圳研究院 | Self-learning super-resolution three-dimensional photoacoustic vessel image reconstruction method and system |
| CN114529507B (en) * | 2021-12-30 | 2024-05-17 | 广西慧云信息技术有限公司 | Visual transducer-based particle board surface defect detection method |
-
2022
- 2022-12-28 CN CN202211689462.XA patent/CN115661377B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111507462A (en) * | 2020-04-15 | 2020-08-07 | 华中科技大学鄂州工业技术研究院 | End-to-end three-dimensional medical image super-resolution reconstruction method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115661377A (en) | 2023-01-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zeng et al. | Simultaneous single-and multi-contrast super-resolution for brain MRI images based on a convolutional neural network | |
| CN108898560B (en) | Core CT image super-resolution reconstruction method based on three-dimensional convolutional neural network | |
| Darbon et al. | Fast nonlocal filtering applied to electron cryomicroscopy | |
| CN104574298B (en) | A kind of noise-reduction method of more b values diffusion weightings images based on mutual information | |
| CN100561518C (en) | Self-adaptation medical image sequence interpolation method based on area-of-interest | |
| Baselice | Ultrasound image despeckling based on statistical similarity | |
| Dolui et al. | A new similarity measure for non-local means filtering of MRI images | |
| CN113160380B (en) | Three-dimensional magnetic resonance image super-resolution reconstruction method, electronic equipment and storage medium | |
| CN115661377B (en) | Self-supervision deep learning and method for constructing isotropic super-resolution three-dimensional image | |
| CN112419320B (en) | Cross-modal heart segmentation method based on SAM and multi-layer UDA | |
| WO2016187148A1 (en) | System, method and computer accessible medium for noise estimation, noise removal and gibbs ringing removal | |
| US20220114699A1 (en) | Spatiotemporal resolution enhancement of biomedical images | |
| La Rosa et al. | MPRAGE to MP2RAGE UNI translation via generative adversarial network improves the automatic tissue and lesion segmentation in multiple sclerosis patients | |
| Sander et al. | Autoencoding low-resolution MRI for semantically smooth interpolation of anisotropic MRI | |
| CN116523739A (en) | Unsupervised implicit modeling blind super-resolution reconstruction method and device | |
| Do et al. | MRI super-resolution using 3D cycle-consistent generative adversarial network | |
| Nedjati-Gilani et al. | Regularized super-resolution for diffusion MRI | |
| CN115311135A (en) | 3 DCNN-based isotropic MRI resolution reconstruction method | |
| Xie et al. | Super-resolution reconstruction of bone micro-structure micro-CT image based on auto-encoder structure | |
| CN116797457B (en) | Method and system for simultaneously realizing super-resolution and artifact removal of magnetic resonance image | |
| Agrawal et al. | Enhancing Z-resolution in CT volumes with deep residual learning | |
| He et al. | IsoVEM: Isotropic Reconstruction for Volume Electron Microscopy Based on Transformer | |
| Kaur et al. | MR-SRNET: Transformation of low field MR images to high field MR images | |
| CN117576250B (en) | Rapid reconstruction method and system for prospective undersampled MRI Dixon data | |
| Gautam et al. | Implementation of NLM and PNLM for de-noising of MRI images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230704 Address after: 100190 No. 95 East Zhongguancun Road, Beijing, Haidian District Patentee after: INSTITUTE OF AUTOMATION, CHINESE ACADEMY OF SCIENCES Patentee after: Institute of Biophysics, Chinese Academy of Sciences Address before: 100190 No. 95 East Zhongguancun Road, Beijing, Haidian District Patentee before: INSTITUTE OF AUTOMATION, CHINESE ACADEMY OF SCIENCES |