CN118154411A - Digital adaptive optical architecture and system - Google Patents
Digital adaptive optical architecture and system Download PDFInfo
- Publication number
- CN118154411A CN118154411A CN202410573482.3A CN202410573482A CN118154411A CN 118154411 A CN118154411 A CN 118154411A CN 202410573482 A CN202410573482 A CN 202410573482A CN 118154411 A CN118154411 A CN 118154411A
- Authority
- CN
- China
- Prior art keywords
- image
- sub
- aperture
- sampling rate
- wavefront
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000003044 adaptive effect Effects 0.000 title claims description 15
- 230000003287 optical effect Effects 0.000 title abstract description 18
- 238000005070 sampling Methods 0.000 claims abstract description 65
- 230000004075 alteration Effects 0.000 claims abstract description 64
- 238000012545 processing Methods 0.000 claims abstract description 14
- 238000000034 method Methods 0.000 claims description 30
- 230000008707 rearrangement Effects 0.000 claims description 13
- 238000012549 training Methods 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 230000010354 integration Effects 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000003384 imaging method Methods 0.000 abstract description 15
- 238000001514 detection method Methods 0.000 abstract description 10
- 230000015572 biosynthetic process Effects 0.000 abstract description 3
- 238000003786 synthesis reaction Methods 0.000 abstract description 3
- 230000001427 coherent effect Effects 0.000 abstract description 2
- 230000006698 induction Effects 0.000 abstract description 2
- 230000009286 beneficial effect Effects 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 239000008358 core component Substances 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000004587 chromatography analysis Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000004513 sizing Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/18—Image warping, e.g. rearranging pixels individually
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20004—Adaptive image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a digital self-adaptive optical architecture and a system, which comprise the steps of rearranging pixels of an original light field image to obtain a sub-aperture image with a low spatial sampling rate; processing the sub-aperture image with the low spatial sampling rate by using a TIS algorithm to obtain a sub-aperture image with the high spatial sampling rate; estimating the space offset of the sub-aperture image with high space sampling rate by using a wave-front gradient estimation algorithm from coarse granularity to fine granularity to obtain a first wave-front gradient, and converting the first wave-front gradient into a first wave-front aberration by using a trained MLP model; and performing phase space deconvolution operation on the sub-aperture image with the high space sampling rate based on the first wavefront aberration to obtain an image reconstruction result of the local image. The invention can perform high-speed wide-view-field wavefront detection, adopts a turbulence induction scanning algorithm to improve the sampling rate, and adopts a non-coherent aperture synthesis algorithm to realize aberration removal and high-resolution imaging.
Description
Technical Field
The present invention relates to the field of light field imaging, and in particular, to a digital adaptive optical architecture and system.
Background
Optical aberrations are commonly existing in natural environments and imaging systems, and atmospheric turbulence motion, imperfect lenses and the like can cause deviation of light ray tracks, so that optical aberrations are generated, and image blurring and signal distortion are caused. The Adaptive Optics (AO) technology aims to measure and correct optical aberration caused by atmospheric turbulence or other media in real time, and is mainly applied to the fields of astronomical observation, laser communication, visual science and the like, and can remarkably improve image quality and signal stability.
The core components of the AO system include a wavefront sensor, a deformable mirror, and an advanced control system. The wavefront sensor is used for detecting the distortion condition of the light wave front entering the system in real time; the deformable mirror is used for carrying out shape adjustment according to the measurement result of the wavefront sensor so as to compensate the distortion of the wavefront of the light; the control system coordinates the wavefront sensor and the deformable mirror to ensure that the distortion is effectively corrected.
Existing AO systems have certain limitations mainly in terms of field of view and system complexity. The optical aberration has spatial non-uniformity, and the existing AO system mainly adopts a Shack-Hartmann wavefront sensor, can only measure single optical aberration in a smaller field of view (FOV), and cannot solve the problem of spatial non-uniformity. Advanced AO technologies, such as MCAO and GLAO, employ a multi-sensor approach to extend the field of view to several angles, but sacrifice some wavefront detection accuracy, resulting in increased system complexity.
Disclosure of Invention
The present invention aims to solve at least one of the technical problems in the related art to some extent.
Therefore, the invention provides a digital self-adaptive image reconstruction method, which solves the problems of wide view field, high resolution imaging and larger optical aberration and realizes high resolution imaging.
Another object of the invention is to propose a digital adaptive image reconstruction system.
To achieve the above object, an aspect of the present invention provides a digital adaptive image reconstruction method, including:
Carrying out pixel rearrangement on the original light field image to obtain a sub-aperture image with a low spatial sampling rate;
processing the sub-aperture image with the low spatial sampling rate by using a TIS algorithm to obtain a sub-aperture image with a high spatial sampling rate;
estimating the space offset of the sub-aperture image with the high space sampling rate by using a wave-front gradient estimation algorithm from coarse granularity to fine granularity to obtain a first wave-front gradient, and converting the first wave-front gradient into a first wave-front aberration by using a trained MLP model;
and carrying out phase space deconvolution operation on the sub-aperture image with the high space sampling rate based on the first wavefront aberration to obtain an image reconstruction result of the local image.
The digital self-adaptive image reconstruction method of the embodiment of the invention can also have the following additional technical characteristics:
In one embodiment of the invention, after obtaining the sub-aperture image at the low spatial sampling rate, the method further comprises:
acquiring image sequences of the sub-aperture images with the low spatial sampling rate of the same sub-aperture at different moments;
And calculating the relative coordinates between the sub-aperture images at the reference moment according to the relative offset between the image sequences.
In one embodiment of the present invention, the processing the sub-aperture image with low spatial sampling rate by using TIS algorithm to obtain a sub-aperture image with high spatial sampling rate includes:
splicing a plurality of sub-aperture images in the image sequence according to the relative coordinates of the sub-aperture images at the reference moment by using a TIS algorithm to obtain a coordinate splicing result;
and obtaining a sub-aperture image with high spatial sampling rate at the reference moment based on the coordinate splicing result.
In one embodiment of the invention, converting the first wavefront gradient to a first wavefront aberration using a trained MLP model includes:
Acquiring a second wavefront gradient obtained by a coarse-granularity-to-fine-granularity wavefront gradient estimation algorithm;
Converting the second wavefront gradient into second wavefront aberration by using a two-dimensional integration algorithm so as to fit a Zernike polynomial coefficient to be fitted, which corresponds to the second wavefront aberration;
Training the MLP model by using the second wavefront gradient to obtain a trained MLP model; the model input layer corresponds to the dimension of the second wavefront gradient, and the number of nodes of the model output layer is equal to the number of Zernike polynomial coefficients to be fitted;
And inputting a first wavefront gradient into the trained MLP model to output corresponding Zernike polynomial coefficients so as to calculate the first wavefront aberration.
In one embodiment of the present invention, after obtaining the image reconstruction result of the partial image, the method further includes:
Reconstructing each partial image to obtain image reconstruction results of all the partial images;
and splicing the image reconstruction results of all the local images to obtain the image reconstruction result of the global image.
To achieve the above object, another aspect of the present invention provides a space-time-angle fusion dynamic light field reconstruction system, including:
the image pixel rearrangement module is used for carrying out pixel rearrangement on the original light field image to obtain a sub-aperture image with a low spatial sampling rate;
the aperture image processing module is used for processing the sub-aperture image with the low spatial sampling rate by using a TIS algorithm to obtain a sub-aperture image with the high spatial sampling rate;
The wavefront aberration conversion module is used for estimating the space offset of the sub-aperture image with the high space sampling rate by utilizing a coarse-granularity to fine-granularity wavefront gradient estimation algorithm to obtain a first wavefront gradient, and converting the first wavefront gradient into first wavefront aberration by utilizing a trained MLP model;
And the local image reconstruction module is used for carrying out phase space deconvolution operation on the sub-aperture image with the high space sampling rate based on the first wavefront aberration to obtain an image reconstruction result of the local image.
According to the digital self-adaptive image reconstruction method and system, the plug-and-play wide-field wavefront sensor is used for high-speed wide-field wavefront detection, the turbulence induction scanning algorithm is adopted to improve the sampling rate, and the incoherent aperture synthesis algorithm is used, so that low-cost, wide-field and high-resolution imaging can be realized.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The foregoing and/or additional aspects and advantages of the invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a flow chart of a digital adaptive image reconstruction method according to an embodiment of the present invention;
FIG. 2 is a block diagram of a digital adaptive image reconstruction method according to an embodiment of the present invention;
FIG. 3 is a flow chart of a wavefront gradient estimation algorithm from coarse to fine granularity in accordance with an embodiment of the present invention;
FIG. 4 is a flow chart of an ISA algorithm according to an embodiment of the present invention;
Fig. 5 is a schematic structural diagram of a digital adaptive image reconstruction system according to an embodiment of the present invention.
Detailed Description
It should be noted that, without conflict, the embodiments of the present invention and features of the embodiments may be combined with each other. The invention will be described in detail below with reference to the drawings in connection with embodiments.
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
The digital adaptive image reconstruction method and system according to the embodiments of the present invention are described below with reference to the accompanying drawings.
Fig. 1 is a flow chart of a digital adaptive image reconstruction method according to an embodiment of the present invention.
As shown in fig. 1, the method includes, but is not limited to, the steps of:
s1, carrying out pixel rearrangement on an original light field image to obtain a sub-aperture image with a low spatial sampling rate;
S2, processing the sub-aperture image with the low spatial sampling rate by using a TIS algorithm to obtain a sub-aperture image with the high spatial sampling rate;
S3, estimating the space offset of the sub-aperture image with high space sampling rate by using a wave-front gradient estimation algorithm from coarse granularity to fine granularity to obtain a first wave-front gradient, and converting the first wave-front gradient into a first wave-front aberration by using a trained MLP model;
S4, performing phase space deconvolution operation on the sub-aperture image with the high space sampling rate based on the first wavefront aberration to obtain an image reconstruction result of the local image.
It will be appreciated that the present invention proposes a new DAO architecture whose core component is a plug-and-play wide-field-of-view wavefront sensor (WWS). In terms of hardware, the WWS is only slightly changed on the basis of the traditional 2D CMOS image sensor, specifically, only a Micro Lens Array (MLA) is required to be placed on an original image plane, and then the 2D CMOS image sensor is placed on a back focal plane of the MLA, so that the system complexity and the implementation cost are low; in algorithm aspect, the invention adopts a wave front gradient estimation algorithm from coarse granularity to fine granularity, and combines multi-layer perceptron (MLP) mapping to realize high-speed (30 Hz) and wide-field-of-view (over 1100 angular seconds) space non-uniform aberration detection.
Fig. 2 is a block diagram of a digital adaptive image reconstruction method of the present invention, as shown in fig. 2, including a preprocessing stage, a flow of a Turbulence Induced Scanning (TIS) algorithm, an aberration wavefront sensing, and a flow of a non-coherent aperture synthesis (ISA) algorithm.
In one embodiment of the invention, the preprocessing stage is a pixel rearrangement operation.
Specifically, the original light field image is converted into a corresponding number of sub-aperture images, each representing the projection of a light field of different angles. Minor deviations between CMOS and MLA can be corrected by sizing and rotation operations.
In one embodiment of the invention, the TIS, the left branch of fig. 2, achieves sub-pixel level image alignment and stitching.
It can be appreciated that the 2D CMOS for light field imaging records 4D light field information, and the spatial sampling rate is sacrificed while the angular information is recorded, so that the spatial sampling rate of the sub-aperture image obtained by preprocessing is too low, and TIS is used for recovering the sampling rate.
Specifically, firstly, estimating the relative offset of image sequences of the same sub-aperture at different moments to obtain the relative coordinates of the sub-aperture images; and then splicing the sub-aperture images in the sequence according to the relative coordinates of the sub-aperture images at the reference moment by using a TIS algorithm, so as to obtain the sub-aperture images at the reference moment and with high spatial sampling rate. The image quality obtained by adopting the TIS algorithm is higher, and no obvious dynamic artifact exists.
The TIS algorithm of the embodiment of the invention realizes the alignment and the splicing of the images at the sub-pixel level, wherein the algorithm input is a plurality of pictures with low spatial sampling rate, and the output is a picture with high spatial sampling rate. When the space sampling rate is low, the optical aberration caused by the rapid change of atmospheric turbulence can be regarded as the rapid scanning of the space non-uniformity, and the TIS algorithm can fuse scanning information to improve the space sampling rate.
The TIS algorithm flow may be formulated as follows:
In the method, in the process of the invention, Representing the number of image frames for TIS (the requirement is odd, then reference moment/>），Time/>Corresponding/>Sub-aperture image,/>TIS result of sub-aperture image representing reference time,/>Representing two-dimensional coordinates on an image plane,/>Representation/>And/>The relative coordinate offset between them,Representing sub-aperture image/>In the coordinate position/>Results using Bicubic Interpolation (BI)/>Representing mean square error,/>Indicating the Scatter Interpolation (SI).
The purpose of equation (1) is to calculate the relative coordinate offsets of the sub-aperture images at different moments, and the optimization problem is solved by adopting a random gradient descent algorithm (SGD); the meaning of the formula (2) is that a plurality of sub-aperture images with randomly distributed transverse coordinates are subjected to a scattered point interpolation mode to obtain gray values corresponding to the gray values on uniform grid positions, and the gray values are fused into a sub-aperture image with high resolution.
In one embodiment of the invention, the aberration wavefront sensing is the right branch of fig. 2.
Specifically, a wavefront gradient estimation algorithm from coarse granularity to fine granularity is used for estimating the spatial non-uniform offset of a plurality of different sub-aperture images at a specific moment, and the offset is the wavefront gradient field. The "coarse-grain to fine-grain" optimization strategy can significantly improve the robustness and running speed of the algorithm. The specific flow of the algorithm is shown in fig. 3, and the flow is briefly described as follows:
(1) The algorithm is used to estimate the sub-aperture image from the target To reference sub-aperture image/>According to the invention, a certain reference sub-aperture image/>, is selected for different sub-aperture images obtained by shooting at the same momentAll other sub-aperture images/>, are estimatedTo/>The spatial non-uniform transverse offset is spliced according to the position of the aperture surface, and a spatial non-uniform wave front gradient field is obtained;
(2) First referring to the flow pair in the upper half of FIG. 3 And/>Normalization is performed to remove the intensity distribution, so as to avoid failure of an estimation algorithm caused by inconsistent intensity distribution, and then the sub-aperture image/>, is normalizedAnd/>The upper estimated wavefront gradient, the optimization problem is expressed as follows:
In the middle of Representing mean square error,/>Representing bicubic interpolation,/>Representing bicubic upsampling,/>Representing two-dimensional coordinates (same size as the image) on the image plane,/>Representing two-dimensional coordinates (the size is consistent with the number of local spatially consistent regions, smaller than the image size) on the image plane,/>Represents the lateral offset;
(3) Referring to the flow in the lower half of FIG. 3, coarse-grained wavefront gradient is first optimized Will/>Is used to initialize fine-grained wavefront gradient/>Still further optimize/>After the iteration process converges,/>I.e. the/>, is sought. The coarse-granularity wavefront gradient and the fine-granularity wavefront gradient are optimized by adopting a random gradient descent algorithm, and an optimizer adopts Adam.
Further, after obtaining the wavefront gradient, an integration algorithm may be used to obtain the wavefront aberration from the wavefront gradient. In an actual application scene, the integration algorithm is slower in speed and cannot meet the real-time observation requirement, and therefore, the method designs and trains an MLP model to realize rapid conversion from gradient to aberration. The MLP is trained by adopting the supervised mode, and the trained MLP is used for reasoning instead of the integral process, and the specific flow is as follows:
1) Preparing input data: the input data is the second wavefront gradient calculated by a wavefront gradient estimation algorithm from coarse granularity to fine granularity;
2) Preparing an output to be fitted: a two-dimensional integration algorithm is operated on the second wavefront gradient to obtain second wavefront aberration, and a least square method is used for fitting Zernike polynomial coefficients corresponding to the second wavefront aberration, wherein the Zernike polynomial coefficients are output to be fitted;
3) Initializing: initializing an MLP network, adopting a structure of double hidden layers, wherein an input layer is a wave front gradient, the first hidden layer is provided with 300 nodes, the second hidden layer is provided with 500 nodes, an output layer is a Zernike polynomial coefficient, and the parameters are randomly initialized;
4) Training: the loss function adopts MSE, the optimizer adopts Adam, and the MSE loss is converged after iterative training;
5) Reasoning: the first wavefront gradient is input to the MLP, and the MLP may output Zernike polynomial coefficients of the first wavefront aberration.
Therefore, experiments show that the MLP model can better fit training data, and the performance of the MLP model on verification data is very similar to that of the training data, so that the MLP model has good generalization capability.
In one embodiment of the present invention, the specific flow of the algorithm of ISA is shown in fig. 4, and the flow is briefly described as follows:
Taking the spatial non-uniformity of aberration into consideration, reconstructing by adopting a block deconvolution mode, wherein the deconvolution method adopts a phase space deconvolution algorithm;
For the local area, the aberration is considered to be consistent, so that the obtained wavefront aberration operates a phase space deconvolution algorithm on the obtained TIS post-local sub-aperture image to obtain a reconstruction result of the local image;
And reconstructing each local part to obtain a reconstruction result of all local areas, and only splicing the local areas to obtain a reconstruction result of the global image, thereby realizing the improvement of the global resolution.
It can be known that the invention adopts ISA algorithm to realize the removal of the space inconsistent optical aberration, and remarkably improves the imaging resolution.
Illustratively, the deep neural network can replace the current aberration wavefront detection algorithm, and the accuracy and the speed of the deep neural network are to be verified. The deep neural network can replace the operation of block deconvolution in the ISA, and the imaging quality and the imaging speed of the deep neural network are to be verified.
In summary, the beneficial effects and application scenarios brought by the present invention include:
1) Low cost, wide field of view, high resolution imaging: the invention mainly solves the problems of wide-field and high-resolution imaging, and is particularly effective in the case of larger optical aberration. In addition, the low-cost characteristic of the invention is beneficial to being widely applied to astronomical observation, long-distance imaging and other scenes, and paving and bridging are explored for science.
2) Turbulence analysis: the wide field of view wavefront gradient detection is useful for analyzing atmospheric turbulence motion. For example, in gradient Detection and ranging (Slope Detection AND RANGING, SLODAR), wide field of wavefront gradient data is beneficial to improving the longitudinal resolution of turbulent flow chromatography (profiling) and robustness of analysis results.
3) Turbulence prediction: according to the taylor freeze flow hypothesis, the turbulence aberration data of the wide field of view is beneficial to improving the accuracy of turbulence aberration prediction. In free space optical communication, the pre-compensation of turbulent aberration is an important link for improving the signal-to-noise ratio and reducing the error rate, and the method is beneficial to improving the prediction precision of turbulent aberration, thereby improving the communication quality.
According to the digital self-adaptive image reconstruction method provided by the embodiment of the invention, the plug-and-play wide-field wavefront sensor is used for carrying out high-speed wide-field wavefront detection, the TIS algorithm is adopted to realize sub-pixel level image alignment and splicing, the algorithm input is a plurality of pictures with low spatial sampling rate, and the output is a picture with high spatial sampling rate. When the space sampling rate is low, the optical aberration caused by the rapid change of atmospheric turbulence can be regarded as the rapid scanning of the space non-uniformity, and the TIS algorithm can fuse scanning information to improve the space sampling rate. And the ISA algorithm is adopted to realize the removal of the spatial non-uniform optical aberration, so that the imaging resolution is obviously improved.
In order to implement the above-described embodiment, as shown in fig. 5, a digital adaptive image reconstruction system 10 is also provided in this embodiment, and the system 10 includes an image pixel rearrangement module 100, an aperture image processing module 200, a wavefront aberration conversion module 300, and a partial image reconstruction module 400.
The image pixel rearrangement module 100 is configured to perform pixel rearrangement on an original light field image to obtain a sub-aperture image with a low spatial sampling rate;
the aperture image processing module 200 is configured to process the sub-aperture image with the low spatial sampling rate by using the TIS algorithm to obtain a sub-aperture image with a high spatial sampling rate;
The wavefront aberration conversion module 300 is configured to estimate a spatial offset of the sub-aperture image with a high spatial sampling rate by using a coarse-granularity to fine-granularity wavefront gradient estimation algorithm to obtain a first wavefront gradient, and convert the first wavefront gradient into a first wavefront aberration by using a trained MLP model;
the local image reconstruction module 400 is configured to perform a phase space deconvolution operation on the sub-aperture image with the high spatial sampling rate based on the first wavefront aberration to obtain an image reconstruction result of the local image.
Further, after the image pixel rearrangement module 100, the method further includes: the coordinate calculation module is used for:
Acquiring image sequences of sub-aperture images with different moments and low spatial sampling rates of the same sub-aperture;
the relative coordinates between the sub-aperture images at the reference instant are calculated from the relative offsets between the image sequences.
Further, the aperture image processing module 200 is further configured to:
Splicing a plurality of sub-aperture images in an image sequence according to the relative coordinates of the sub-aperture images at the reference moment by using a TIS algorithm to obtain a coordinate splicing result;
and obtaining a sub-aperture image with high spatial sampling rate at the reference moment based on the coordinate splicing result.
Further, the wavefront aberration conversion module 300 is further configured to:
Acquiring a second wavefront gradient obtained by a coarse-granularity-to-fine-granularity wavefront gradient estimation algorithm;
Converting the second wavefront gradient into second wavefront aberration by using a two-dimensional integration algorithm so as to fit a Zernike polynomial coefficient to be fitted, which corresponds to the second wavefront aberration;
Training the MLP model by using the second wavefront gradient to obtain a trained MLP model; the model input layer corresponds to the dimension of the second wavefront gradient, and the number of nodes of the model output layer is equal to the number of Zernike polynomial coefficients to be fitted;
And inputting a first wavefront gradient into the trained MLP model to output corresponding Zernike polynomial coefficients so as to calculate the first wavefront aberration.
Further, after the partial image reconstruction module 400, further includes: a global image reconstruction module for:
Reconstructing each partial image to obtain image reconstruction results of all the partial images;
And splicing the image reconstruction results of all the local images to obtain the image reconstruction result of the global image.
According to the digital self-adaptive image reconstruction system provided by the embodiment of the invention, the high-speed wide-field wavefront detection is carried out through the plug-and-play wide-field wavefront sensor, the image alignment and the splicing of the sub-pixel level are realized by adopting a TIS algorithm, the algorithm input is a plurality of pictures with low spatial sampling rate, and the output is a picture with high spatial sampling rate. When the space sampling rate is low, the optical aberration caused by the rapid change of atmospheric turbulence can be regarded as the rapid scanning of the space non-uniformity, and the TIS algorithm can fuse scanning information to improve the space sampling rate. And the ISA algorithm is adopted to realize the removal of the spatial non-uniform optical aberration, so that the imaging resolution is obviously improved.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "plurality" means at least two, for example, two, three, etc., unless specifically defined otherwise.
Claims (10)
1. A method for digital adaptive image reconstruction, comprising:
Carrying out pixel rearrangement on the original light field image to obtain a sub-aperture image with a low spatial sampling rate;
processing the sub-aperture image with the low spatial sampling rate by using a TIS algorithm to obtain a sub-aperture image with a high spatial sampling rate;
estimating the space offset of the sub-aperture image with the high space sampling rate by using a wave-front gradient estimation algorithm from coarse granularity to fine granularity to obtain a first wave-front gradient, and converting the first wave-front gradient into a first wave-front aberration by using a trained MLP model;
and carrying out phase space deconvolution operation on the sub-aperture image with the high space sampling rate based on the first wavefront aberration to obtain an image reconstruction result of the local image.
2. The method of claim 1, wherein after obtaining the sub-aperture image at the low spatial sampling rate, the method further comprises:
acquiring image sequences of the sub-aperture images with the low spatial sampling rate of the same sub-aperture at different moments;
And calculating the relative coordinates between the sub-aperture images at the reference moment according to the relative offset between the image sequences.
3. The method of claim 2, wherein processing the low spatial sample rate sub-aperture image using a TIS algorithm results in a high spatial sample rate sub-aperture image, comprising:
splicing a plurality of sub-aperture images in the image sequence according to the relative coordinates of the sub-aperture images at the reference moment by using a TIS algorithm to obtain a coordinate splicing result;
and obtaining a sub-aperture image with high spatial sampling rate at the reference moment based on the coordinate splicing result.
4. The method of claim 1, wherein converting the first wavefront gradient to a first wavefront aberration using a trained MLP model comprises:
Acquiring a second wavefront gradient obtained by a coarse-granularity-to-fine-granularity wavefront gradient estimation algorithm;
Converting the second wavefront gradient into second wavefront aberration by using a two-dimensional integration algorithm so as to fit a Zernike polynomial coefficient to be fitted, which corresponds to the second wavefront aberration;
Training the MLP model by using the second wavefront gradient to obtain a trained MLP model; the model input layer corresponds to the dimension of the second wavefront gradient, and the number of nodes of the model output layer is equal to the number of Zernike polynomial coefficients to be fitted;
And inputting a first wavefront gradient into the trained MLP model to output corresponding Zernike polynomial coefficients so as to calculate the first wavefront aberration.
5. The method according to claim 1, wherein after obtaining the image reconstruction result of the partial image, the method further comprises:
Reconstructing each partial image to obtain image reconstruction results of all the partial images;
and splicing the image reconstruction results of all the local images to obtain the image reconstruction result of the global image.
6. A digital adaptive image reconstruction system, comprising:
the image pixel rearrangement module is used for carrying out pixel rearrangement on the original light field image to obtain a sub-aperture image with a low spatial sampling rate;
the aperture image processing module is used for processing the sub-aperture image with the low spatial sampling rate by using a TIS algorithm to obtain a sub-aperture image with the high spatial sampling rate;
The wavefront aberration conversion module is used for estimating the space offset of the sub-aperture image with the high space sampling rate by utilizing a coarse-granularity to fine-granularity wavefront gradient estimation algorithm to obtain a first wavefront gradient, and converting the first wavefront gradient into first wavefront aberration by utilizing a trained MLP model;
And the local image reconstruction module is used for carrying out phase space deconvolution operation on the sub-aperture image with the high space sampling rate based on the first wavefront aberration to obtain an image reconstruction result of the local image.
7. The system of claim 6, further comprising, after the image pixel rearrangement module: the coordinate calculation module is used for:
acquiring image sequences of the sub-aperture images with the low spatial sampling rate of the same sub-aperture at different moments;
And calculating the relative coordinates between the sub-aperture images at the reference moment according to the relative offset between the image sequences.
8. The system of claim 7, wherein the aperture image processing module is further configured to:
splicing a plurality of sub-aperture images in the image sequence according to the relative coordinates of the sub-aperture images at the reference moment by using a TIS algorithm to obtain a coordinate splicing result;
and obtaining a sub-aperture image with high spatial sampling rate at the reference moment based on the coordinate splicing result.
9. The system of claim 6, wherein the wavefront aberration conversion module is further configured to:
Acquiring a second wavefront gradient obtained by a coarse-granularity-to-fine-granularity wavefront gradient estimation algorithm;
Converting the second wavefront gradient into second wavefront aberration by using a two-dimensional integration algorithm so as to fit a Zernike polynomial coefficient to be fitted, which corresponds to the second wavefront aberration;
Training the MLP model by using the second wavefront gradient to obtain a trained MLP model; the model input layer corresponds to the dimension of the second wavefront gradient, and the number of nodes of the model output layer is equal to the number of Zernike polynomial coefficients to be fitted;
And inputting a first wavefront gradient into the trained MLP model to output corresponding Zernike polynomial coefficients so as to calculate the first wavefront aberration.
10. The system of claim 6, further comprising, after the partial image reconstruction module: a global image reconstruction module for:
Reconstructing each partial image to obtain image reconstruction results of all the partial images;
and splicing the image reconstruction results of all the local images to obtain the image reconstruction result of the global image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410573482.3A CN118154411B (en) | 2024-05-10 | 2024-05-10 | Digital adaptive optical architecture and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410573482.3A CN118154411B (en) | 2024-05-10 | 2024-05-10 | Digital adaptive optical architecture and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118154411A true CN118154411A (en) | 2024-06-07 |
| CN118154411B CN118154411B (en) | 2024-07-16 |
Family
ID=91290958
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410573482.3A Active CN118154411B (en) | 2024-05-10 | 2024-05-10 | Digital adaptive optical architecture and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118154411B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113396440A (en) * | 2019-02-14 | 2021-09-14 | 卡尔蔡司医疗技术公司 | System for OCT image conversion and ophthalmic image denoising and neural network thereof |
| CN115375586A (en) * | 2022-10-24 | 2022-11-22 | 中国人民解放军国防科技大学 | Method and device for sharpening high-turbulence image, computer equipment and medium |
| CN117560105A (en) * | 2023-11-13 | 2024-02-13 | 北京理工大学 | Swin-transducer-based high-capacity optical communication self-adaptive compensation method |
-
2024
- 2024-05-10 CN CN202410573482.3A patent/CN118154411B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113396440A (en) * | 2019-02-14 | 2021-09-14 | 卡尔蔡司医疗技术公司 | System for OCT image conversion and ophthalmic image denoising and neural network thereof |
| CN115375586A (en) * | 2022-10-24 | 2022-11-22 | 中国人民解放军国防科技大学 | Method and device for sharpening high-turbulence image, computer equipment and medium |
| CN117560105A (en) * | 2023-11-13 | 2024-02-13 | 北京理工大学 | Swin-transducer-based high-capacity optical communication self-adaptive compensation method |
Non-Patent Citations (3)
| Title |
|---|
| J.OSBORN 等: "First on-sky results of a neural network based tomographic reconstructor: Carmen on Canary", PROCEEDINGS OF SPIE, vol. 9148, 21 July 2014 (2014-07-21) * |
| JIAMIN WU 等: "An integrated imaging sensor for aberration-corrected 3D photography", NATURE, vol. 612, 19 October 2022 (2022-10-19) * |
| 马辰昊;付跃刚;贺文俊;王加科;董亭亭;: "双光楔微扫描哈特曼-夏克波前探测技术", 红外与激光工程, no. 09, 25 September 2015 (2015-09-25) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118154411B (en) | 2024-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111539879B (en) | Video blind denoising method and device based on deep learning | |
| CN100481937C (en) | Equipment for reconstructing high dynamic image in high resolution | |
| Shimizu et al. | Super-resolution from image sequence under influence of hot-air optical turbulence | |
| Kronander et al. | A unified framework for multi-sensor HDR video reconstruction | |
| EP2560375B1 (en) | Image processing device, image capture device, program, and image processing method | |
| CN105589210B (en) | Digital synthetic aperture imaging method based on pupil modulation | |
| CN113076685B (en) | Training method of image reconstruction model, image reconstruction method and device thereof | |
| US8565478B2 (en) | Method of improving the resolution of a moving object in a digital image sequence | |
| JP2013531268A (en) | Measuring distance using coded aperture | |
| CN115226406A (en) | Image generation device, image generation method, recording medium generation method, learning model generation device, learning model generation method, learning model, data processing device, data processing method, estimation method, electronic device, generation method, program, and non-transitory computer-readable medium | |
| Fishbain et al. | Real-time stabilization of long range observation system turbulent video | |
| CN103841297A (en) | Electronic image-stabilizing method suitable for resultant-motion camera shooting carrier | |
| CN113436130B (en) | Intelligent sensing system and device for unstructured light field | |
| CN114998141B (en) | Space environment high dynamic range imaging method based on multi-branch network | |
| CN112819697B (en) | Remote sensing image space-time fusion method and system | |
| JP7543080B2 (en) | Trained model and data processing device | |
| CN115086550B (en) | Meta imaging system | |
| US11967096B2 (en) | Methods and apparatuses of depth estimation from focus information | |
| US10288523B2 (en) | Method and device for characterising optical aberrations of an optical system | |
| CN111225139B (en) | Motion scene shooting device for acquiring complementary sequence image and restoration method | |
| JP5566199B2 (en) | Image processing apparatus, control method therefor, and program | |
| Yu et al. | Color polarization image super-resolution reconstruction via a cross-branch supervised learning strategy | |
| CN118154411B (en) | Digital adaptive optical architecture and system | |
| CN112634139B (en) | Optical field super-resolution imaging method, device and equipment | |
| KR20110088680A (en) | Image processing apparatus which can compensate a composite image obtained from a plurality of image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |