CN114092325B - Fluorescent image super-resolution reconstruction method and device, computer equipment and medium - Google Patents
Fluorescent image super-resolution reconstruction method and device, computer equipment and medium Download PDFInfo
- Publication number
- CN114092325B CN114092325B CN202111120216.8A CN202111120216A CN114092325B CN 114092325 B CN114092325 B CN 114092325B CN 202111120216 A CN202111120216 A CN 202111120216A CN 114092325 B CN114092325 B CN 114092325B
- Authority
- CN
- China
- Prior art keywords
- original
- spectrogram
- spectrum
- image
- spectrograms
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 79
- 238000001228 spectrum Methods 0.000 claims abstract description 115
- 230000003595 spectral effect Effects 0.000 claims abstract description 30
- 238000002073 fluorescence micrograph Methods 0.000 claims abstract description 22
- 238000007781 pre-processing Methods 0.000 claims abstract description 22
- 230000009466 transformation Effects 0.000 claims abstract description 17
- 230000008859 change Effects 0.000 claims abstract description 16
- 238000001914 filtration Methods 0.000 claims abstract description 12
- 230000006870 function Effects 0.000 claims description 80
- 238000000926 separation method Methods 0.000 claims description 25
- 230000008569 process Effects 0.000 claims description 22
- 238000012545 processing Methods 0.000 claims description 17
- 239000011159 matrix material Substances 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 8
- 230000003287 optical effect Effects 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 238000012546 transfer Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 13
- 238000005286 illumination Methods 0.000 description 9
- 238000004422 calculation algorithm Methods 0.000 description 7
- 239000000523 sample Substances 0.000 description 7
- 230000035772 mutation Effects 0.000 description 6
- 230000021615 conjugation Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 238000013519 translation Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000000799 fluorescence microscopy Methods 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 4
- 230000007704 transition Effects 0.000 description 4
- 210000004027 cell Anatomy 0.000 description 3
- 235000021120 animal protein Nutrition 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000005284 excitation Effects 0.000 description 2
- 238000004091 panning Methods 0.000 description 2
- 108090000623 proteins and genes Proteins 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000011426 transformation method Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000011478 gradient descent method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 210000003463 organelle Anatomy 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000002689 soil Substances 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/14—Fourier, Walsh or analogous domain transformations, e.g. Laplace, Hilbert, Karhunen-Loeve, transforms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Theoretical Computer Science (AREA)
- Pure & Applied Mathematics (AREA)
- Mathematical Optimization (AREA)
- Mathematical Analysis (AREA)
- Data Mining & Analysis (AREA)
- Computational Mathematics (AREA)
- Algebra (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)
Abstract
The invention provides a super-resolution reconstruction method of a fluorescence image, which comprises the steps of S1 obtaining N groups of sequence original images; s2 preprocessing the edge area of each original image so that the pixels of the edge area of the original image are in a gradual change state; s3, carrying out spectrum transformation on any group of preprocessed sequence original images to obtain corresponding original spectrograms, then separating the original spectrograms and obtaining corresponding spectrum components
And
s4 creating a new blank spectrogram, combining the spectral components
Moving to the center of the blank spectrogram while simultaneously removing high-frequency components
And
respectively moving to the corresponding correct frequency domain positions in the blank spectrogram to obtain a recombined spectrogram corresponding to the set of sequence original spectrograms; s5 repeating steps S3 to S4 until the original images of the N groups of sequences are processed; s6, splicing the multiple recombined spectrograms and filtering to obtain an approximately isotropic two-dimensional frequency spectrum; s7, carrying out Fourier inverse transformation on the two-dimensional frequency spectrum to obtain a super-resolution image, thereby effectively improving the image reconstruction precision.
Description
Technical Field
The invention belongs to the field of image reconstruction, and particularly relates to a fluorescent image super-resolution reconstruction method, a fluorescent image super-resolution reconstruction device, computer equipment and a fluorescent image super-resolution reconstruction medium.
Background
The structured light illumination fluorescence microscopy technology carries high-frequency information which cannot be recorded in a sample to a visible low-pass frequency band of a microscope by modifying an illumination mode of a conventional microscope, and extracts the carried high-frequency information and reconstructs an ultra-resolution image by changing the direction and the phase of a pattern and matching with a corresponding processing algorithm. Theoretically, the resolution of images obtained by structured light illumination fluorescence microscopy can be twice as high as that of the traditional fluorescence microscope. The structured light illumination fluorescence microscopy has been widely applied in the aspects of biomedicine, cell dynamic process observation and the like due to the advantages of low requirements on a fluorescence probe, low optical power, high time resolution, high reconstruction resolution and the like.
In the structured light illumination fluorescence microscopic imaging technology, the final super-resolution image quality mainly depends on the quality of the processes of frequency spectrum separation, frequency spectrum movement and frequency spectrum splicing, wherein if errors occur in any step, even if the precision of the subsequent steps is higher, the final super-resolution image inevitably has serious stripes, artifacts and even errors.
Disclosure of Invention
The invention aims to provide a fluorescent image super-resolution reconstruction method, which can realize the final improvement of the image super-resolution precision by respectively improving the quality of each process of frequency spectrum separation, frequency spectrum movement and frequency spectrum splicing.
In order to achieve the above object, the present invention provides a fluorescence image super-resolution reconstruction method, comprising the following steps:
s1, acquiring N groups of sequence original images, wherein each group of sequence original images comprises M original images, N is the modulation direction of structured light, and each modulation direction has M phases;
s2 preprocessing the edge area of each of the original images so that the pixel sizes between the edge area and the outside and between the inside of the edge area of the original image are in a gradual change state;
s3, carrying out spectrum transformation on any group of preprocessed sequence original images to obtain corresponding original spectrogram, then separating the original spectrogram, and obtaining corresponding spectrum components
And
s4 creating a new blank spectrogram, combining the spectral components
Moving to the center of the blank spectrogram and simultaneously adding high-frequency components
And
respectively moving to the corresponding correct frequency domain positions in the blank spectrogram to obtain recombined spectrograms corresponding to the original spectrogram, wherein the moved spectral components
And
the length and the width of the blank spectrogram are within the range;
s5 repeating the steps S3 to S4 until all the original images of the N groups of sequences are processed, and acquiring a plurality of recombined spectrograms;
s6, splicing the multiple recombined spectrograms to obtain spliced spectrograms, and performing filtering operation on the spliced spectrograms to obtain approximately isotropic two-dimensional spectrums;
and S7, performing inverse Fourier transform on the two-dimensional frequency spectrum to obtain a super-resolution image.
Preferably, in step S2, the preprocessing the edge value of each original image includes the following steps:
s21, presetting an edge distance d, and determining four edge area ranges to be processed of the original image according to the edge distance d;
s22 performs the gradation process on the pixel values inside each edge area line by line or line by line in sequence until all four edge areas are processed.
Further, each original pixel value y in the edge area is respectively replaced by y' to perform a gradation process, wherein,
preferably, in step S3, a matrix is constructed and solved to obtain corresponding spectral components
And
wherein,
i is the average intensity of the excitation light, m is the modulation degree,
is the initial phase.
Further, three initial phases are set
Conversion to initial phase
Wherein
Correspondingly, the initial phase
The acquisition comprises the following steps:
s31 obtaining initial peak value p 0 ；
S32 based on the initial peak value p 0 Acquiring a corresponding neighborhood region, and acquiring the value size of a three-dimensional point pair (u, v, f) corresponding to a plurality of pixel points in the neighborhood region, wherein u and v are coordinates corresponding to each neighborhood point in a spectrogram, and distribution weight f is a parameter which can represent the distribution condition of the spoke angle phase of the region at will on the pixel points in the neighborhood region;
s33, constructing a fitting function model F (u, v, F) for fitting the variation trend of the argument phase in the neighborhood region;
s34 solving unknown quantity in the fitting function model F (u, v, F) based on the three-dimensional point pair (u, v, F) value to obtain a fitting function expression;
s35 obtaining the maximum value position p of the fitting function expression function max And calculating p max The amplitude phase of the beam is used as the final initial phase
Further, in S32, the distribution weight f is a complex modulus length τ or an argument θ of the mixed spectrum,
where τ is a + bi, and a and b are the real part and imaginary part of the complex number of the mixed spectrum at the pixel position, respectively.
Further, in step S33, the fitting function model is a two-dimensional gaussian function;
wherein k, sigma and mu are unknown quantities,
or
Or, the fitting function model is a two-dimensional diffraction intensity equation
Wherein k, alpha are unknown quantities,
or
Or, the fitting function model is F (r, F | k, γ) ═ k · J 1 (γ r) wherein J 1 (gamma r) is a first order Bessel function, k, gamma are unknowns,
or
Preferably, the step S6 is preceded by a step of determining whether all N sets of sequence original images have traversed, and if yes, the step S6 is executed; if not, repeating the steps S3 to S4 until all the N groups of sequence original images are processed.
The invention also provides a fluorescence image super-resolution reconstruction device, which comprises:
the device comprises a sequence original image acquisition module, a sequence original image acquisition module and a data processing module, wherein the sequence original image acquisition module is used for acquiring N groups of sequence original images, each group of sequence original images comprises M original images, N is the modulation direction of structured light, and each modulation direction has M phases; the preprocessing module is used for preprocessing the edge value of each original image to enable pixels in the edge area of the original image to be in a gradual change state;
the gradual change processing module is used for preprocessing the edge area of each original image to enable the pixel size between the edge area and the outside and between the inside of the edge area of each original image to be in a gradual change state;
a spectrum separation module for performing spectrum transformation on any group of preprocessed sequence original images to obtain corresponding original spectrogram, separating the original spectrogram, and obtaining corresponding spectrum components
And
a spectrum moving module for creating a blank spectrogram and dividing the spectrum components
Moving to the center of the blank spectrogram and simultaneously transmitting high-frequency components
And
respectively moving to the corresponding correct frequency domain positions in the blank spectrogram to obtain recombined spectrograms corresponding to the original spectrogram, wherein the moved spectral components
And
the length and the width of the blank spectrogram are within the range;
the frequency spectrum splicing module is used for splicing the plurality of recombined frequency spectrograms to obtain spliced frequency spectrograms, and carrying out filtering operation on the spliced frequency spectrograms to obtain approximately isotropic two-dimensional frequency spectrums;
and the inverse transformation module is used for performing inverse Fourier transformation on the two-dimensional frequency spectrum to obtain a super-resolution image.
The invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of any of the above methods when executing the computer program.
The invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of any of the methods described above.
According to the fluorescent image super-resolution reconstruction method, the fluorescent image super-resolution reconstruction device, the computer equipment and the fluorescent image super-resolution reconstruction medium, after N groups of sequence original images are obtained, all original images are respectively preprocessed, and after edge transition processing is carried out on the original images, a frequency spectrum mutation value caused by edge pixel value mutation in the frequency spectrum separation process can be eliminated, so that the risk of searching errors for peak points in initial phase estimation in medium-frequency spectrum separation is reduced, and the precision of frequency spectrum separation is improved; meanwhile, in the process of carrying out spectrum translation, the expanded spectrogram is newly built to accommodate all the moved spectrum components, so that the situation that the high-frequency components move out of the spectrogram in the process of spectrum translation is avoided, the high-frequency information of the original image is effectively reserved, and the accuracy in reconstruction is further improved.
In addition, according to the fluorescent image super-resolution reconstruction method, the fluorescent image super-resolution reconstruction device, the computer equipment and the fluorescent image super-resolution reconstruction medium, in the process of frequency spectrum separation, the traditional iterative algorithm is replaced by parameter fitting, the algorithm speed and efficiency are improved, meanwhile, the precision meets the requirements of engineering use, and the speed and the initial phase estimation precision are considered.
Drawings
FIG. 1 is a schematic diagram of a workflow of a fluorescence image super-resolution reconstruction method according to an embodiment of the present invention;
FIG. 2 is a light path diagram of a projection type SIM super-resolution microscope system based on digital micromirror DMD modulation and LED illumination; the reference numbers in the figures are: 1-LED illumination light source, 2-beam splitting prism, 3-structured light generator, 4-collimating lens, 5-illumination light filter, 6-beam splitter, 7-reflector, 8-microscope objective, 9-objective stage, 10-filter, 11-tube lens and 12-area array digital camera;
FIG. 3(a) is a pixel distribution diagram of the edge region of the original image; FIG. 3(b) is a pixel distribution diagram of the edge region after the pre-processing;
FIG. 4 is a schematic diagram illustrating an initial phase acquisition process;
fig. 5(a) to 5(c) are schematic diagrams of conventional spectrum splicing;
FIGS. 6(a) to 6(c) are schematic diagrams of the spectrum splicing according to the present invention;
FIG. 7(a) is an original image obtained by photographing with a CMOS camera at the time of gene sequencing;
FIG. 7(b) is a super-resolution image obtained by the reconstruction method of the present invention;
FIG. 8(a) is an original image of animal protein captured by a CMOS camera;
FIG. 8(b) is a super-resolution image obtained by the reconstruction method of the present invention;
fig. 9 is a schematic structural diagram of a fluorescence image super-resolution reconstruction apparatus according to an embodiment of the present invention;
fig. 10 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of the invention in any way.
Like reference numerals refer to like elements throughout the specification. The expression "and/or" includes any and all combinations of one or more of the associated listed items. In the drawings, the thickness, size, and shape of an object have been slightly exaggerated for convenience of explanation. The figures are purely diagrammatic and not drawn to scale.
It will be further understood that the terms "comprises," "comprising," "includes," "including," "has," "includes" and/or "including," when used in this specification, specify the presence of stated features, steps, integers, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, integers, operations, elements, components, and/or groups thereof.
The terms "substantially", "about" and the like as used in the specification are used as terms of approximation and not as terms of degree, and are intended to account for inherent deviations in measured or calculated values that would be recognized by one of ordinary skill in the art.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.
Example one
As shown in FIG. 1, the invention discloses a fluorescence image super-resolution reconstruction method, comprising the following steps:
s1, acquiring N groups of sequence original images, wherein each group of sequence original images comprises M original images, N is the modulation direction of structured light, and each modulation direction has M phases;
s2 preprocessing the edge area of each of the original images so that pixels between the edge area and the outside of the original image and inside the edge area are in a gradual change state;
s3, carrying out spectrum transformation on any group of preprocessed sequence original images to obtain corresponding original spectrogram, then separating the original spectrogram, and obtaining corresponding spectrum components
And
s4 creating a new blank spectrogram, combining the spectral components
Moving to the center of the blank spectrogram and simultaneously adding high-frequency components
And
respectively moving to the corresponding correct frequency domain positions in the blank spectrogram to obtain recombined spectrograms corresponding to the original spectrogram, wherein the moved spectral components
And
the length and the width of the blank spectrogram are within the range;
s5 repeating the steps S3 to S4 until all the original images of the N groups of sequences are processed, and acquiring a plurality of recombined spectrograms;
s6, splicing the multiple recombined spectrograms to obtain spliced spectrograms, and performing filtering operation on the spliced spectrograms to obtain approximately isotropic two-dimensional spectrums;
s7, carrying out Fourier inverse transformation on the two-dimensional frequency spectrum to obtain a super-resolution image.
According to the super-resolution reconstruction method for the fluorescence image, all original images are respectively preprocessed after N groups of sequence original images are obtained, and after edge transition processing is carried out on the original images, a frequency spectrum mutation value caused by edge pixel value mutation in the frequency spectrum separation process can be eliminated, so that the risk of finding errors of peak points in initial phase estimation in intermediate frequency spectrum separation is reduced, and the precision of the frequency spectrum separation is improved; meanwhile, in the process of carrying out spectrum translation, the expanded spectrogram is newly built to accommodate all the moved spectrum components, so that the situation that the high-frequency components move out of the spectrogram in the process of spectrum translation is avoided, the high-frequency information of the original image is effectively reserved, and the accuracy in reconstruction is further improved.
Preferably, in step S1, the acquiring N sets of sequence original images includes the following steps: the irradiation of the structured light on the sample has N modulation directions, each modulation direction has M phases, N is a natural number larger than or equal to 2, and each phase in each modulation direction acquires an original image, so that NxM original images are obtained.
In this embodiment, the projection type SIM super-resolution microscope system based on DMD modulation and LED illumination shown in fig. 2 may be used, first, an LED light beam with a wavelength of 405nm enters the dichroic prism 2 and irradiates the DMD, a structured light generated after being modulated by the DMD of the digital micromirror exits through the dichroic prism 2, enters the microscope objective 8 after being collimated by the collimator lens 4, and the microscope objective 8 micro-compresses and projects the structured light stripe on its focal plane; then, placing the cell sample on an object stage 9 and adjusting the cell sample to a focal plane of a microscope objective, illuminating the sample by structured light generated by micro projection, and exciting fluorescent molecules of a labeled organelle to emit light; finally, controlling the digital micromirror DMD to load 2 light fields with 6 structures, wherein the included angle of the space directions is 90 degrees, and the phase of each space direction is 0, 2 pi/3 and 4 pi/3 in sequence, and the area array digital camera respectively and correspondingly collects 6 images and stores the images in a computer; of course, it is needless to say that the digital micromirror DMD can also be controlled to sequentially load 6 structured light fields with 90 ° included angles in more spatial directions and phases of 0, 2 pi/3, and 4 pi/3 in each spatial direction, and accordingly, the area array digital camera respectively collects 9 and 12 images.
After N groups of sequence original images are obtained, all original images are respectively preprocessed by the fluorescence image super-resolution reconstruction method disclosed by the invention, so that the pixels of the edge area of each original image are in a smooth gradual change state, and the phenomenon that the pixel value mutation position is misjudged as a peak point due to the fact that the larger pixel value mutation exists in the pixels of the edge area when the subsequent frequency spectrum separation is carried out is avoided, so that the initial phase estimation is wrong, and the accuracy of the frequency spectrum separation is influenced.
As a preferable scheme, the preprocessing of the edge value of each of the original images in step S2 includes the following steps:
s21, presetting an edge distance d, and determining four to-be-processed edge area ranges of the original image according to the edge distance d;
s22 performs a gradation process on the pixel values inside each edge area row by row or column by column until all four edge areas are processed.
Further, in step S22, each original pixel value y in the edge region is replaced by y' to perform the tapering process, wherein,
wherein x is the distance between the position of the pixel point and the edge of the image.
Step S2 is further described with reference to the embodiment shown in fig. 3, as shown in fig. 3(a), the line-column pixel values of the outermost edge of the original image are all 255, and the pixels of the outer region of the original image are usually 0, if the original image is directly subjected to spectral separation, since the pixels of the outer region suddenly rise from 0 to 255 between the edge regions of the original image, in step S3, the angular phase of the peak is required to be used as the initial phase during the spectral separation process
The evaluation criterion of (2) is that the false judgment of the edge area as the peak value easily occurs, thereby leading to the initial phase
The estimation accuracy of (2) is seriously wrong, and the initial phase
The effect on super-resolution images is very serious, includingResearch shows that if the error of the initial phase exceeds 10% -20%, the contrast of the final super-resolution image is obviously reduced, and image information is largely covered by stripes, so that effective information cannot be identified. Therefore, in the embodiment shown in fig. 3, the preset edge distance d is 5, and correspondingly, the value range in the edge region is 9 × 5 and 5 × 9, and the pixels in the edge region are respectively replaced by y' from the original y, wherein
The structure after the edge region is subjected to the over-processing is shown in fig. 3(b), pixels of the processed original image are in smooth transition between the outside and the inside or between the inside, no abnormal sudden change exists, particularly, the most marginal row and column of the image are provided, the pixel value of the original image is replaced by 8 from 255, no sudden change exists between the pixel value of the original image and the external pixel value 0 of the original image, meanwhile, other pixel values y' of the edge region are obtained on the basis of the original pixel value y, the original image information is also reserved while the smooth transition is realized, and the subsequent processing is not influenced.
In step S2, the pixels in the common coverage area of each edge area may be replaced once as needed, or may be replaced multiple times.
In step S3, performing spectrum transformation on any group of preprocessed sequence original images to obtain corresponding original spectrograms, then separating the original spectrograms, and obtaining corresponding spectrum components
And
transforming the preprocessed sequence original image into an original spectrogram after Fourier transform; then, when the original spectrogram is separated, a matrix formula is constructed, and then three unknown spectral components corresponding to different directions can be solved by the matrix formula constructed as follows
And
for spectral separation.
Analyzing the above equation set, the frequency spectrum of the structured light illuminated image
Is a known quantity, H (k) is the light source transfer function OTF, and therefore, the parameter I 0 、m、
Are the values to be evaluated, but the influence of the values on the quality of the final super-resolution soil phase is different.
Average intensity of excitation light I 0 The size of the common constant factor only changes the overall gray distribution of the image linearly, and the effect can be compensated by means of image histogram normalization and the like, so that the effect is not obvious. The structured light modulation m only affects the overall intensity of the high frequency components, resulting in high and low frequency ratios, but does not cause spectral separation errors. Therefore, the precision requirement of the structured light modulation m is not high. Initial phase of structured light fringes
The effect on the super-resolution image is very severe. As described above, if the error of the initial phase exceeds 10% to 20%, the contrast of the final super-resolution image will be significantly reduced, and the image information will be largely covered by the stripes, so that the effective information cannot be identified. Therefore, the initial phase value must be accurately evaluated.
In this embodiment, three structured light stripes are initially phased
Conversion to initial phase
Wherein
I.e. assuming 2/3 pi as the phase difference between the different phases, then
Can be expressed as:
as shown in fig. 4, the initial phase
The acquisition comprises the following steps:
s31 obtaining initial peak value p 0 ；
S32 based on the initial peak value p 0 Acquiring a corresponding neighborhood region, and acquiring the value size of a three-dimensional point pair (u, v, f) corresponding to a plurality of pixel points in the neighborhood region, wherein u and v are coordinates corresponding to each neighborhood point in a spectrogram, and distribution weight f is a parameter which can represent the distribution condition of the spoke angle phase of the region at will on the pixel points in the neighborhood region;
s33, constructing a fitting function model F (u, v, F) for fitting the variation trend of the argument phase in the neighborhood region;
s34 solving unknown quantity in the fitting function model F (u, v, F) based on the three-dimensional point pair (u, v, F) value to obtain a fitting function expression;
s35 obtaining the maximum value position p of the fitting function expression function max And calculate p max The amplitude phase of the beam is used as the final initial phase
In step S31, an initial peak p may be obtained based on the existing peak estimation method 0 As a preferable scheme, in this embodiment, the initial peak value p may be obtained based on an image conjugation method or an image reconstruction transformation method 0 。
When an initial peak value p is acquired using an image conjugation method (Fair _ SIM) 0 The method comprises the following steps:
s311, converting the obtained image with the phase difference under the same angle into a frequency domain, namely converting D (r)
S312, carrying out frequency spectrum separation to obtain three frequency spectrum components
S313 pairs three spectral components according to optical system parameters
Processing to reduce interference of invalid information on the image, and respectively corresponding to the image after processing
S314 to convert the
Turning to the space domain, three components S '(r), S' (e) are formed jπp r)，S′(e -jπp r)；
S315, high-frequency and low-frequency conjugate calculation is carried out to obtain image autocorrelation c 1 、c 2 The size is measured, the conjugation result is transformed into the frequency domain after Fourier transform, and the frequency domain is recorded as
Wherein, c 1 、c 2 As an image autocorrelation calculation result, an indicates a conjugate.
S316 is in
On the spectrogram, the maximum value is found in the region within the OTF cut-off frequency, wherein a pair of maximum value points symmetrical about the frequency position of '0' is found, and 1 is taken as the initial peak value p 0 ；
Furthermore, in step S312, the spectrum separation is performed according to the following formula, specifically, the spectrum separation is obtained by matrix solving
Meanwhile, in step S313, the processing method may be selected according to the optical system parameters such as OTF and attenuation coefficient of the optical system.
Obtaining an initial peak value p when selecting an image reconstruction transform method (IRT) 0 The method can comprise the following steps: when the image phase difference at the same angle satisfies 2/3 pi, 1+ e is considered -j2π/3 +e j2π/3 The IRT method multiplies the three images by {1, e, respectively, 0 -j2π/3 ，e j2π/3 And summing to construct a recombined image, and determining an initial peak value p by applying a method similar to the image conjugation method step S15 on the frequency spectrum of the recombined image 0 。
The initial peak value p is obtained based on the image conjugation method or the image recombination transformation method 0 All the methods are prior art, and are not described herein again, and it goes without saying that besides the two methods illustrated above, other methods may be used to obtain the initial peak value p 0 And then, realizing the subsequent super-resolution reconstruction method.
Obtaining an initial peak value p 0 Thereafter, the process proceeds to step S32, where the initial peak value p is used as the basis 0 And acquiring a corresponding neighborhood region, and acquiring the value size of the three-dimensional point pairs (u, v, f) corresponding to a plurality of pixel points in the neighborhood region.
Neighborhood region, i.e. initial peak p 0 Neighborhood pixel locations on the spectral image. In order to ensure the accuracy of subsequent fitting, the neighborhood points cannot be selected too few, otherwise, too large fitting deviation may be caused by too few data samples. When selecting the neighborhood point, the distance between the neighborhood point and the initial peak position p 0 Not too far away, points that are outside a certain distance from the initial peak location point may not fit the defined fit function model. As a preferable scheme, the initial peak value p can be obtained based on discrete point taking methods such as a D neighborhood, a rectangular neighborhood (including but not limited to access methods such as a 4 neighborhood, an 8 neighborhood and a 15 neighborhood), an R neighborhood and the like 0 The neighborhood of (2). Fig. 3(a) and 3(b) are a pixel distribution diagram of an edge region of an original image and a pixel distribution diagram of a preprocessed edge region, and fig. 4 is a schematic diagram of an acquisition flow of an initial phase.
After the neighborhood region is determined, further, the value of the distribution weight f of each point in the neighborhood region is evaluated, and the distribution weight f is defined as an initial peak value p 0 And any parameter that may represent the argument phase distribution of the region at the neighborhood point position, in this embodiment, as a preferred scheme, f is defined as the complex modulus length τ of the mixed spectrum or as the argument θ.
where τ is a + bi, each pixel location in the mixed spectrum is a complex number, and a and b are the real part and imaginary part of the mixed spectrum corresponding to the complex number at the pixel location.
Solving for the initial peak p 0 And the weight distribution f of the neighborhood points, constructing a three-dimensional point pair (u, v, f) consisting of the coordinates of the points on the spectrogram and the weight points, or writing the three-dimensional point pair (u, v, f) as (r, f), wherein u and v are the coordinates corresponding to each neighborhood point in the spectrogram,
or
After obtaining the neighborhood region and the value of each neighborhood point three-dimensional point pair (u, v, F) in the neighborhood region, the method goes to step S3 to construct a fitting function model F (u, v, F) which is used for fitting the variation trend of the argument phase in the neighborhood region.
Due to the initial peak value p 0 And the weight distribution f of the neighborhood points thereof accords with a certain function rule. The method shown in the present invention utilizes a certain fitting function model to simulate the distribution of the weighted values f, and as a preferred scheme, in this embodiment, the variation trend of the argument phase in the neighborhood of the peak position can be approximately fitted based on any one of the following functions.
(a) Two-dimensional gaussian function
Wherein k, σ and mu are unknown quantities;
(b) equation of two-dimensional diffraction intensity
Wherein k and alpha are unknown quantities;
(c)F(r，f|k，γ)＝k·J 1 (γ r) wherein J 1 (γ r) is a first order Bessel function, and k, γ are unknowns.
Of course, the above 3 models can be replaced by other functional models simulating the distribution to which the weight values f are consistent.
After a proper fitting function model is defined, the step S34 is carried out, unknown quantity in the fitting function model F (u, v, F) is solved based on the value of the three-dimensional point pair (u, v, F) determined in the step S32, and a fitting function expression is obtained;
in step S34, substituting the three-dimensional point pairs (u, v, F) obtained in step S32 into the fitting model function to construct an equation set, and solving the unknown quantity in the fitting function model F (u, v, F), so as to obtain a determined fitting function expression.
In order to ensure the fitting accuracy and reliability, it is necessary to ensure that the neighborhood point selection in step S32 cannot be too few. In principle, the number of pairs of three-dimensional points (neighborhood points) must be greater than or equal to the number of unknowns in the fitting function model F (u, v, F), so that the function parameters can be solved by constructing a homogeneous equation set or by a fitting method. As a preferable scheme, in this embodiment, the number of the three-dimensional point pairs is >2 × unknown number. That is, if the fitting function model is a two-dimensional gaussian function, the number of neighborhood points is >2 × 3 — 6 because the number of unknowns is 3, and if the fitting function model is a two-dimensional diffraction intensity equation or a first-order bessel function, the number of neighborhood points is >2 × 2 — 4 because the number of unknowns is 2. By the arrangement, the number of the field points can be ensured to be sufficient, so that the function model solved by the fitting method is more accurate.
In addition, when the number of the neighborhood points is large enough, the neighborhood points can be subjected to proper filtering processing by using a random sample consensus (RANSAC) method. Considering that the data of each sampling point is not necessarily very accurate, when the number of neighborhood points is large enough, the invention further filters some neighborhood points which are too different from the distribution of the model function through a filtering algorithm, so that the reserved neighborhood points have higher fitting precision, thereby further improving the precision of the solved function model.
After step S34, all unknowns in the model function are found, and the function model becomes a clear and specific function, and then the process proceeds to step S35 to find the position of the maximum value of the function. In this embodiment, the maximum value p for determining the function can be obtained by mathematical means such as derivation or gradient descent method max The maximum value necessarily approaches p 0 And p is max With sub-pixel level accuracy, typically on the order of 0.01 pixel or even 0.001 pixel. Then further take p max The angular amplitude phase of the beam is used as the final initial phase
In the present embodiment, the method using the fair _ SIM described in step S1 is usedCalculation of p max The angular amplitude phase of the point is taken as the final initial phase
Specifically, the high frequency component obtained by separating the spectrum is shifted by p max To the correct frequency spectrum position, then the cross correlation of high and low frequency components in the OTF area is obtained, and the phase value is obtained, thereby obtaining the final initial phase
According to the initial phase estimation method, the traditional iterative algorithm is replaced by parameter fitting, the speed and the efficiency of the algorithm are improved, meanwhile, the precision meets the requirements of engineering use, two aspects of speed and initial phase estimation precision are considered, the initial phase estimation method can be used for obtaining the subsequent calculation of the initial phase as long as the initial peak and the relevant parameters of the neighborhood points of the initial peak are calculated, a large amount of iterative calculations are not needed, the speed can be effectively guaranteed, meanwhile, the algorithm can improve the initial peak position from the whole pixel to the sub-pixel level by selecting a function model which accords with the reality, and therefore the precision is effectively guaranteed.
After the spectrum separation, step S4 is performed to perform spectrum restoration, and as a preferred scheme, in this embodiment, the shifting operation is not directly performed on the three spectrum components to perform spectrum restoration, but a spectrogram is created first, and then the spectrum components are processed
Moving to the center position of the spectrogram; at the same time, high frequency components are combined
And
and respectively moving to the corresponding correct frequency domain positions in the spectrogram to perform spectrum restoration operation. Wherein, the length and width of the newly-created spectrogram are large enough to make the shifted spectral components
And
the length and the width of the blank spectrogram are within the range; the situation that the high-frequency component moves out of the spectrogram in the spectrum translation process can be avoided, so that the high-frequency information of the original image is effectively reserved, and the accuracy in reconstruction is improved, wherein in the step S4, the spectrum component is used
After moving to the center of the spectrogram, high-frequency components can be obtained
And
and multiplying the high-frequency spectrum by the impact function respectively to shift the high-frequency spectrum to the high-frequency position of the image. In this embodiment, the impulse function expression is as follows:
as shown in fig. 5(a) to 5(c), if the panning operation is performed directly with the three spectral components obtained in step S3, the spectral components are subjected to the panning operation
When the radius of the spectrum is larger than the value of P, the original spectrogram can not contain high-frequency components when the spectrum moves
And
the corresponding information also causes a lot of image information to be lost when the subsequent frequency spectrums are spliced, and further the high-precision reconstruction of the image cannot be realized.
As shown in fig. 6(a) to 6(c), in the present embodiment, the newly created spectrogram is first expanded to twice the original length and width of the previous spectrogram, and then the spectral components are expanded
Moving to the central position (0, 0) of the spectrogram, and then adding high-frequency components
And
convolution operation is respectively carried out on the high-frequency component and the unit impulse function delta, so that the centers of the high-frequency components can be respectively moved to the positions of (p, 0) and (-p, 0), and therefore frequency spectrum information is effectively reserved while frequency spectrum restoration is conveniently and rapidly achieved.
And repeating the steps from S3 to S4 until all the N groups of sequence original images are processed, and acquiring a plurality of recombined spectrograms.
In addition, as a preferable scheme, a step of judging whether all the N groups of sequence original images traverse is further included before the step S6, if yes, the step is shifted to the step S6; if not, repeating the steps S3 to S4 until all the N groups of sequence original images are processed, so as to ensure that the recombined spectrogram in each modulation direction is not missed as much as possible.
After the recombined spectrograms in multiple directions are all obtained, the step S6 is carried out to splice the recombined spectrograms to obtain spliced spectrograms, and the spliced spectrograms are subjected to filtering operation to obtain approximately isotropic two-dimensional spectrums;
because there is an overlap region generated in the stitching process, and the overlap region may cause an artifact phenomenon in the reconstruction process, a deconvolution filtering operation needs to be performed to eliminate the overlap spectrum on the spectrogram due to the stitching. In this embodiment, as a preferable scheme, step S5 in this embodiment may include the following steps:
s51, based on one of the recombined spectrograms, adding the values of the corresponding pixel positions in the other N-1 recombined spectrograms and filling the result into the recombined spectrograms serving as the basis to obtain a complete spliced spectrogram.
S52, filtering the spliced spectrogram to obtain an approximately isotropic two-dimensional spectrum.
In step S51, it is needless to say that a new spectral image with the same width and height as the reconstructed spectral image may be selected, the values of the pixel positions corresponding to the N reconstructed spectral images are added, and the result is filled into the new spectral image to form a spliced spectral image.
Further, as a preferable embodiment, in step S52, Richardson-Lucy deconvolution expression is adopted
And performing filter operation on the spliced spectrogram to eliminate overlapped spectrums on the spectrogram due to splicing so as to obtain an approximately isotropic two-dimensional spectrum. Wherein
Is the estimate of f after k iterations, is the correlation operator,
is an R _ L function, image
Is a deblurred image.
Fig. 7 and 8 are graphs showing comparison of experimental effects of imaging performed by using the super-resolution imaging method of the present invention, where fig. 7(a) is an original image obtained by using a CMOS camera for gene sequencing, fig. 8(a) is an original image of animal protein obtained by using a CMOS camera for imaging, and fig. 7(b) and 8(b) are super-resolution images obtained by using the reconstruction method of the present invention, and it can be found by comparing the two images that the resolution of the reconstructed image is 1.5 to 1.8 times that of the original image, and it can be clearly seen that the resolution can be effectively improved and the imaging quality can be improved by using the reconstruction method of the present invention.
Example two
As shown in fig. 9, the present invention also discloses a fluorescence image super-resolution reconstruction apparatus 10, which includes:
a sequence original image obtaining module 11, configured to obtain N groups of sequence original images, where each group of sequence original images includes M original images, where N is a modulation direction of structured light, and each modulation direction has M phases; the preprocessing module is used for preprocessing the edge value of each original image to enable pixels in the edge area of the original image to be in a gradual change state;
a pixel preprocessing module 12, configured to preprocess an edge area of each of the original images, so that pixels between the edge area and an outside of the original image and inside the edge area are in a gradual change state;
a spectrum separation module 13, configured to perform spectrum transformation on any group of preprocessed sequence original images to obtain corresponding original spectrograms, then separate the original spectrograms, and obtain corresponding spectrum components
And
a spectrum moving module 14 for creating a blank spectrogram and dividing the spectral components
Moving to the center of the blank spectrogram and simultaneously adding high-frequency components
And
respectively moving to the corresponding correct frequency domain positions in the blank spectrogram to obtain recombined spectrograms corresponding to the original spectrogram, wherein the moved spectral components
And
the length and the width of the blank spectrogram are within the range;
the frequency spectrum splicing module 15 is configured to splice the multiple recombined spectrograms to obtain a spliced spectrogram, and perform a filtering operation on the spliced spectrogram to obtain an approximately isotropic two-dimensional frequency spectrum;
and the inverse transformation module 16 is configured to perform inverse fourier transformation on the two-dimensional frequency spectrum to obtain a super-resolution image.
Preferably, the pixel preprocessing module 12 includes an edge region acquisition sub-module and a preprocessing sub-module; the edge region acquisition submodule is used for presetting an edge distance d and determining four edge region ranges to be processed of the original image according to the edge distance d; the preprocessing submodule is used for carrying out gradual change processing on the pixel values in each edge area row by row or column by column until the four edge areas are processed.
Further, in the pre-processing sub-module, each original pixel value y in the edge region is replaced by y' respectively for performing the gradual change processing, wherein,
as a preferred scheme, the spectrum separation module 13 includes a spectrum transformation submodule and a spectrum component obtaining submodule, where the spectrum component obtaining submodule is configured to construct a matrix and solve the matrix to obtain corresponding spectrum components
And
wherein,
as an image intensity distribution, I 0 M is the structural light modulation degree for the average intensity of the exciting light,
for the structured light fringe initial phase, H (k) is the optical transfer function.
Furthermore, the spectrum separation module 13 includes an initial phase obtaining sub-module and an initial phase converting sub-module; the initial phase conversion sub-module is used for converting three initial phases
Conversion to initial phase
Wherein
The initial phase
The acquisition submodule includes:
an initial peak value obtaining unit for obtaining an initial peak value p 0 ；
A three-dimensional point pair acquisition unit for acquiring a three-dimensional point pair based on the initial peak value p 0 Acquiring a corresponding neighborhood region, and acquiring the value size of three-dimensional point pairs (u, v, f) corresponding to a plurality of pixel points in the neighborhood region, wherein u and v are coordinates corresponding to each neighborhood point in a spectrogram, and the distribution weight f is a parameter which can represent the distribution condition of the argument phase of the region on any pixel point in the neighborhood region;
the fitting function model building unit is used for building a fitting function model F (u, v, F) which is used for fitting the variation trend of the argument phase in the neighborhood region;
a fitting function expression obtaining unit, configured to solve an unknown quantity in the fitting function model F (u, v, F) based on the three-dimensional point pair (u, v, F) value, and obtain a fitting function expression;
an initial phase obtaining unit for obtaining a maximum value position p of the fitting function expression function max And calculate p max The amplitude phase of the beam is used as the final initial phase
Further, in the three-dimensional point pair obtaining unit, the distribution weight f is a complex modulus length τ of a mixed spectrum or an argument θ, where:
a. b mixing the real and imaginary parts of the spectrum corresponding to the complex number at that pixel location, respectively.
Further, in the fitting function model constructing unit, the fitting function model is a two-dimensional gaussian function;
wherein k, sigma and mu are unknown quantities,
or
Or, the fitting function model is a two-dimensional diffraction intensity equation
Wherein k, alpha are unknown quantities,
or
Or, the fitting function model is F (r, F | k, γ) ═ k · J 1 (γ r) wherein J 1 (gamma r) is a first order Bessel function, k, gamma are unknowns,
or
Preferably, the fluorescence image super-resolution reconstruction device further comprises a judging module, wherein the judging module is used for judging whether all the N groups of sequence original images traverse, and if so, the step is switched to a frequency spectrum splicing module for processing; if not, the method is switched to a frequency spectrum moving module for processing until all the N groups of sequence original images are processed.
As a preferred scheme, in the spectrum splicing module 15, a Richardson-Lucy deconvolution expression is used
And performing filter operation on the spliced spectrogram to eliminate overlapped spectrums on the spectrogram due to splicing so as to obtain an approximately isotropic two-dimensional spectrum. Wherein
Is the estimation of f after k iterations, is the correlation operator,
is an R _ L function, image
Is a deblurred image.
EXAMPLE III
Fig. 10 is a schematic structural diagram of a computer device according to an embodiment of the present invention, for example, a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a cabinet server (including an independent server or a server cluster formed by multiple servers) that can execute a program. The computer device 20 of the present embodiment includes at least but is not limited to: a memory 21, a processor 22, which may be communicatively coupled to each other via a system bus, as shown in FIG. 10. It is noted that fig. 10 only shows a computer device 20 with components 21-22, but it is to be understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead.
In this embodiment, the memory 21 (i.e., the readable storage medium) includes a Flash memory, a hard disk, a multimedia Card, a Card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), and a Programmable Read Only Memory (PROM), and the memory 21 may also be an external storage device of the computer device 20, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card), and the like provided on the computer device 20. Of course, the memory 21 may also include both internal and external storage devices of the computer device 20. In this embodiment, the memory 21 is generally used for storing an operating system and various types of application software installed in the computer device 20, such as program codes of the fluorescence image super-resolution reconstruction apparatus in the method embodiment. Further, the memory 21 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 22 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 22 is typically used to control the overall operation of the computer device 20. In this embodiment, the processor 22 is configured to execute the program code stored in the memory 21 or process data, for example, execute the fluorescence image super-resolution reconstruction apparatus 10, so as to implement the fluorescence image super-resolution reconstruction method in the method embodiment.
Example four
The present application also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application mall, etc., having stored thereon a computer program that, when executed by a processor, performs corresponding functions. The computer-readable storage medium of the present embodiment is used for storing program codes of a fluorescence image super-resolution reconstruction apparatus, and when being executed by a processor, the program codes implement the fluorescence image super-resolution reconstruction method in the method embodiment.
It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict. The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (9)
1. A fluorescence image super-resolution reconstruction method is characterized by comprising the following steps:
s1, acquiring N groups of sequence original images, wherein each group of sequence original images comprises M original images, N is the modulation direction of structured light, and each modulation direction has M phases;
s2 preprocessing the edge area of each of the original images so that the pixel sizes between the edge area and the outside and between the inside of the edge area of the original image are in a gradual change state;
s3, carrying out spectrum transformation on any group of preprocessed sequence original images to obtain corresponding original spectrogram, then separating the original spectrogram, and obtaining corresponding spectrum components
And
s4 creating a new blank spectrogram, combining the spectral components
Moving to the center of the blank spectrogram and simultaneously adding high-frequency components
And
respectively moving to the corresponding correct frequency domain positions in the blank spectrogram to obtain recombined spectrograms corresponding to the original spectrogram, wherein the moved spectral components
And
the length and the width of the blank spectrogram are within the range;
s5 repeating the steps S3 to S4 until all the original images of the N groups of sequences are processed, and acquiring a plurality of recombined spectrograms;
s6, splicing the multiple recombined spectrograms to obtain spliced spectrograms, and filtering the spliced spectrograms to obtain approximately isotropic two-dimensional spectrums;
s7, performing inverse Fourier transform on the two-dimensional frequency spectrum to obtain a super-resolution image;
in step S3, when the original spectrogram is separated, a matrix is constructed and solved to obtain corresponding spectral components
And
wherein,
as an image intensity distribution, I 0 M is the structural light modulation degree for the average intensity of the exciting light,
for the structured light fringe initial phase, H (k) is the optical transfer function;
three initial phases are combined
Conversion to initial phase
Wherein
The initial phase
The acquisition comprises the following steps:
s31 obtaining initial peak value p 0 ；
S32 based on the initial peak value p 0 Acquiring a corresponding neighborhood region, and acquiring the value size of a three-dimensional point pair (u, v, f) corresponding to a plurality of pixel points in the neighborhood region, wherein u and v are coordinates corresponding to each neighborhood point in a spectrogram, and distribution weight f is a parameter which can represent the distribution condition of the spoke angle phase of the region at will on the pixel points in the neighborhood region;
s33, constructing a fitting function model F (u, v, F) for fitting the variation trend of the argument phase in the neighborhood region;
s34 solving unknown quantity in the fitting function model F (u, v, F) based on the three-dimensional point pair (u, v, F) value to obtain a fitting function expression;
s35 obtaining the maximum value position p of the fitting function expression function max And calculate p max The amplitude phase of the phase is used as the final initial phase
2. The method for super-resolution reconstruction of fluorescence image according to claim 1, wherein said step S2, preprocessing the edge value of each of said original images comprises the steps of:
s21, presetting an edge distance d, and determining four to-be-processed edge area ranges of the original image according to the edge distance d;
s22 performs a gradation process on the pixel values inside each edge area row by row or column by column until all four edge areas are processed.
3. The super-resolution reconstruction method of fluorescence image according to claim 1, wherein each original pixel value y in the edge region is replaced by y' for gradation process, respectively,
and x is the distance between the position of the pixel point and the edge of the image.
4. The method for super-resolution reconstruction of fluorescence image according to claim 1, wherein in S32, the distribution weight f is a complex modulus length τ of the mixed spectrum or an argument θ, wherein:
τ is a + bi, and a and b are the real part and imaginary part of the complex number of the mixed spectrum at the pixel position, respectively.
5. The method for super-resolution reconstruction of fluorescence image according to claim 1, wherein in step S33,
the fitting function model is a two-dimensional Gaussian function
Wherein k, sigma and mu are unknown quantities,
or
Or, the fitting function model is a two-dimensional diffraction intensity equation
Wherein k, alpha are unknown quantities,
or
Or, the fitting function model is F (r, F | k, γ) ═ k · J 1 (γ r) wherein J 1 (gamma r) is a first order Bessel function, k, gamma are unknowns,
or
P 0u 、P 0v Is an initial peak value P 0 The corresponding coordinates.
6. The super-resolution reconstruction method for fluorescence image according to claim 1, wherein said step S6 is preceded by a step of determining whether all N sets of original images have been traversed, if yes, proceeding to step S6; if not, repeating the steps S3 to S4 until all the N groups of sequence original images are processed.
7. A fluorescence image super-resolution reconstruction device is characterized in that: the method comprises the following steps:
the device comprises a sequence original image acquisition module, a sequence original image acquisition module and a data processing module, wherein the sequence original image acquisition module is used for acquiring N groups of sequence original images, each group of sequence original images comprises M original images, N is the modulation direction of structured light, and each modulation direction has M phases; the preprocessing module is used for preprocessing the edge value of each original image to enable pixels in the edge area of the original image to be in a gradual change state;
the pixel preprocessing module is used for preprocessing the edge area of each original image to enable the pixel size between the edge area and the outside and between the inside of the edge area of each original image to be in a gradual change state;
a spectrum separation module, which is used for carrying out spectrum transformation on any group of preprocessed sequence original images to obtain corresponding original spectrogram and then separating the original spectrogram,and obtaining corresponding spectral components
And
a spectrum moving module for creating a blank spectrogram and dividing the spectrum components
Moving to the center of the blank spectrogram and simultaneously adding high-frequency components
And
respectively moving to the corresponding correct frequency domain positions in the blank spectrogram to obtain recombined spectrograms corresponding to the original spectrogram, wherein the moved spectral components
And
the length and the width of the blank spectrogram are within the range;
the frequency spectrum splicing module is used for splicing the plurality of recombined frequency spectrograms to obtain spliced frequency spectrograms, and carrying out filtering operation on the spliced frequency spectrograms to obtain approximately isotropic two-dimensional frequency spectrums;
the inverse transformation module is used for performing inverse Fourier transformation on the two-dimensional frequency spectrum to obtain a super-resolution image;
the spectrum separation module is further configured to construct a matrix and solve the matrix to obtain corresponding spectrum components when the original spectrogram is separated
And
wherein,
as an image intensity distribution, I 0 M is the structural light modulation degree for the average intensity of the exciting light,
for the structured light fringe initial phase, H (k) is the optical transfer function;
three initial phases are combined
Conversion to initial phase
Wherein
The initial phase
The obtaining of (2) comprises:
obtaining an initial peak value p 0 ；
Based on the initial peak value p 0 Acquiring a corresponding neighborhood region, and acquiring the value size of a three-dimensional point pair (u, v, f) corresponding to a plurality of pixel points in the neighborhood region, wherein u and v are coordinates corresponding to each neighborhood point in a spectrogram, and distribution weight f is a parameter which can represent the distribution condition of the spoke angle phase of the region at will on the pixel points in the neighborhood region;
constructing a fitting function model F (u, v, F) for fitting the variation trend of the argument phase in the neighborhood region;
solving unknown quantities in the fitting function model F (u, v, F) based on the three-dimensional point pair (u, v, F) values to obtain a fitting function expression;
obtaining the maximum value position p of the fitting function expression function max And calculate p max The amplitude phase of the beam is used as the final initial phase
8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein: the processor, when executing the computer program, realizes the steps of the method of any one of claims 1 to 6.
9. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program when executed by a processor implements the steps of the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111120216.8A CN114092325B (en) | 2021-09-24 | 2021-09-24 | Fluorescent image super-resolution reconstruction method and device, computer equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111120216.8A CN114092325B (en) | 2021-09-24 | 2021-09-24 | Fluorescent image super-resolution reconstruction method and device, computer equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114092325A CN114092325A (en) | 2022-02-25 |
| CN114092325B true CN114092325B (en) | 2022-08-16 |
Family
ID=80296205
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111120216.8A Active CN114092325B (en) | 2021-09-24 | 2021-09-24 | Fluorescent image super-resolution reconstruction method and device, computer equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114092325B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114897950A (en) * | 2022-04-29 | 2022-08-12 | 上海精积微半导体技术有限公司 | Image registration and defect detection method |
| CN115100033B (en) * | 2022-05-20 | 2023-09-08 | 浙江大学 | Fluorescent microscopic image super-resolution reconstruction method and device and computing equipment |
| WO2024000288A1 (en) * | 2022-06-29 | 2024-01-04 | 深圳华大生命科学研究院 | Image stitching method, and gene sequencing system and corresponding gene sequencer |
| CN115953297B (en) * | 2022-12-27 | 2023-12-22 | 二十一世纪空间技术应用股份有限公司 | Remote sensing image super-resolution reconstruction and enhancement method and device |
| CN118311758A (en) * | 2023-01-06 | 2024-07-09 | 深圳市真迈生物科技有限公司 | Light regulator and lighting device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102998789A (en) * | 2012-12-31 | 2013-03-27 | 华中科技大学 | Super-resolution differential interference phase contrast microscopic imaging system and microscopic imaging method |
| CN107180411A (en) * | 2017-05-19 | 2017-09-19 | 中国科学院苏州生物医学工程技术研究所 | A kind of image reconstructing method and system |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105759440A (en) * | 2016-04-29 | 2016-07-13 | 西安电子科技大学 | Random scattering optical super-diffraction limit imaging method based on structured illumination |
| CN111145089B (en) * | 2019-11-18 | 2023-06-16 | 中国科学院苏州生物医学工程技术研究所 | High-fidelity image reconstruction method, system, computer equipment and storage medium |
| CN113108910B (en) * | 2021-04-22 | 2022-05-20 | 熵智科技(深圳)有限公司 | Light source spectrum image acquisition method, device, equipment and storage medium |
-
2021
- 2021-09-24 CN CN202111120216.8A patent/CN114092325B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102998789A (en) * | 2012-12-31 | 2013-03-27 | 华中科技大学 | Super-resolution differential interference phase contrast microscopic imaging system and microscopic imaging method |
| CN107180411A (en) * | 2017-05-19 | 2017-09-19 | 中国科学院苏州生物医学工程技术研究所 | A kind of image reconstructing method and system |
Non-Patent Citations (1)
| Title |
|---|
| SIM显微镜照明结构光场优化与系统点扩散函数表征;鹿伟民;《中国优秀硕士学位论文全文数据库》;20160615;1-83 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114092325A (en) | 2022-02-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114092325B (en) | Fluorescent image super-resolution reconstruction method and device, computer equipment and medium | |
| JP6752200B2 (en) | Lighting system and Fourier tycography imaging equipment | |
| CN110441271B (en) | Light field high-resolution deconvolution method and system based on convolutional neural network | |
| US6658142B1 (en) | Computerized adaptive imaging | |
| JP2017117462A (en) | Physical registration of images acquired by fourier ptychography | |
| AU2014280898A1 (en) | Reconstruction algorithm for Fourier Ptychographic imaging | |
| JP6112872B2 (en) | Imaging system, image processing method, and imaging apparatus | |
| US9031304B2 (en) | Image processing system | |
| CN108648222B (en) | Method and device for improving spatial resolution of structured light depth data | |
| US11226293B1 (en) | Structured illumination optical inspection platform for transparent materials | |
| Li et al. | Super resolution for astronomical observations | |
| CN110411981A (en) | A kind of phase imaging method based on TIE, device and readable storage medium storing program for executing | |
| CN112529794A (en) | High dynamic range structured light three-dimensional measurement method, system and medium | |
| Guo et al. | Lensfree auto-focusing imaging using nuclear norm of gradient | |
| Zimmermann et al. | Deep learning-based hyperspectral image reconstruction from emulated and real computed tomography imaging spectrometer data | |
| CN112651884B (en) | Method and device for acquiring chromatography super-resolution image and electronic equipment | |
| Buddha et al. | Estimation of point spread function of an optical microscope using stochastic minimization of least square errors | |
| CN109489816B (en) | Microscopic hyperspectral imaging platform and large-area data cube acquisition method | |
| CN114881907B (en) | Optical microscopic image multi-depth-of-field focus synthesis method and system and image processing method | |
| CN116338945A (en) | Digital adaptive optics method and apparatus for multiphoton imaging | |
| CN111063022B (en) | Phase body three-dimensional reconstruction method combining phase gradient and direct linear transformation | |
| Meißner | Determination and improvement of spatial resolution obtained by optical remote sensing systems | |
| Tadrous | A method of PSF generation for 3D brightfield deconvolution | |
| CN113808009A (en) | Peak initial phase estimation method and device, computer equipment and storage medium | |
| WO2022014579A1 (en) | Optical aberration correction program and optical wavefront estimation program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231206 Address after: 518000, Building 5, Building C, Building C, Huaqiang Creative Park, Biyan Community, Guangming Street, Guangming District, Shenzhen, Guangdong Province, China 1301 Patentee after: SHENZHEN ROBOT VISION TECHNOLOGY Co.,Ltd. Address before: 518034 703, 7th floor, Zhongdian Difu building, Zhenhua Road, Fuqiang community, Huaqiang North Street, Futian District, Shenzhen City, Guangdong Province Patentee before: SHANGZHI TECHNOLOGY (SHENZHEN) Co.,Ltd. |