CN114967121B - Design method of end-to-end single lens imaging system - Google Patents
Design method of end-to-end single lens imaging system Download PDFInfo
- Publication number
- CN114967121B CN114967121B CN202210522840.9A CN202210522840A CN114967121B CN 114967121 B CN114967121 B CN 114967121B CN 202210522840 A CN202210522840 A CN 202210522840A CN 114967121 B CN114967121 B CN 114967121B
- Authority
- CN
- China
- Prior art keywords
- imaging system
- image
- loss
- lens imaging
- single lens
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 61
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000013461 design Methods 0.000 title claims abstract description 19
- 230000006870 function Effects 0.000 claims abstract description 47
- 238000012549 training Methods 0.000 claims abstract description 16
- 238000013135 deep learning Methods 0.000 claims abstract description 8
- 238000013528 artificial neural network Methods 0.000 claims description 33
- 230000003287 optical effect Effects 0.000 claims description 32
- 238000001914 filtration Methods 0.000 claims description 23
- 238000004088 simulation Methods 0.000 claims description 22
- 238000005457 optimization Methods 0.000 claims description 14
- 238000004422 calculation algorithm Methods 0.000 claims description 13
- 210000002569 neuron Anatomy 0.000 claims description 9
- 239000011159 matrix material Substances 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 6
- 230000003044 adaptive effect Effects 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 4
- 230000001131 transforming effect Effects 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- 230000003071 parasitic effect Effects 0.000 claims description 3
- 239000000126 substance Substances 0.000 claims description 3
- 238000012937 correction Methods 0.000 claims description 2
- 239000000654 additive Substances 0.000 claims 1
- 230000000996 additive effect Effects 0.000 claims 1
- 230000000694 effects Effects 0.000 description 11
- 238000005070 sampling Methods 0.000 description 10
- 238000009826 distribution Methods 0.000 description 9
- 230000000875 corresponding effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000004075 alteration Effects 0.000 description 4
- 238000009792 diffusion process Methods 0.000 description 4
- 238000011084 recovery Methods 0.000 description 4
- 238000011423 initialization method Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 229920003229 poly(methyl methacrylate) Polymers 0.000 description 1
- 239000004926 polymethyl methacrylate Substances 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/0012—Optical design, e.g. procedures, algorithms, optimisation routines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/043—Architecture, e.g. interconnection topology based on fuzzy logic, fuzzy membership or fuzzy inference, e.g. adaptive neuro-fuzzy inference systems [ANFIS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G06T5/70—
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention discloses a design method of an end-to-end single lens imaging system, which comprises the following steps: constructing a single-lens imaging system frame; calculating the square error loss and the additional constraint loss of the restored image and the original image, and constructing a loss function based on the square error loss and the additional constraint loss; and iteratively optimizing trainable parameters of the single-lens imaging system framework to construct the single-lens imaging system. The invention optimally designs the single-lens imaging system, so that the single-lens imaging system has a better initial structure, the training difficulty of deep learning is greatly reduced, and certain performance requirements can be met.
Description
Technical Field
The invention belongs to the technical field of computational optical imaging, and particularly relates to a design method of an end-to-end single-lens imaging system.
Background
Compared with a complex lens group, the single lens has the advantages of small size, light weight, simple structure and the like. However, the aberration of a common single lens is large, and particularly when a large field of view is imaged, the obtained image tends to be blurred. In order to realize high-quality optical system imaging, complex lens combination is often needed to correct aberration. However, the complex lens group optical system has the disadvantages of large volume, large mass, high cost and the like, and has certain limitation in the application with miniaturization requirements such as mobile phone lenses, unmanned aerial vehicle platform camera systems, remote sensing cameras and the like.
The single-lens imaging system acquires an image through a single lens, and corrects a blur caused by a single-lens aberration in the image through a post-processing algorithm, so that the method is a main method for realizing light weight of a lens, and can be applied to the fields of smart phone cameras, unmanned aerial vehicle platform camera systems, remote sensing cameras and the like which need to be miniaturized. Single lens imaging systems can be divided into discrete design systems and end-to-end design systems. The discrete design system firstly designs a single lens and then designs a post-processing recovery algorithm according to the imaging effect of the single lens. The end-to-end single lens imaging system design connects imaging simulation and restoration algorithm together, and learning training is carried out on the single lens surface shape parameters and the restoration algorithm parameters simultaneously by utilizing a deep learning technology. The end-to-end design has the advantage of global optimization compared to a discrete design.
In the existing end-to-end single lens design method, the recovery network part uses a convolution neural network, so that the position information can not be learned, and the aberration fuzzy recovery effect related to the space position is poor. Moreover, the existing methods lack constraints on additional boundary conditions such as edge thickness, center thickness and energy distribution of the single lens, which leads to that the optical lens designed by the algorithm does not meet the requirements of practical processing application.
Disclosure of Invention
The present invention is directed to a method for designing an end-to-end single lens imaging system, so as to solve the above-mentioned problems of the prior art.
In order to achieve the above object, the present invention provides a method for designing an end-to-end single lens imaging system, comprising:
calculating the square error loss and the additional constraint loss of the restored image and the original image, and constructing a loss function based on the square error loss and the additional constraint loss;
and constructing a single-lens imaging system frame, performing iterative optimization on the single-lens imaging system frame based on the deep learning and the loss function to obtain an optimized system, and taking the optimized system as a single-lens imaging system.
Optionally, the single lens imaging system frame comprises: the system comprises an optical image fuzzy simulation module, a fuzzy kernel learning module and an inverse filtering image restoration module;
the process of constructing the single lens imaging system frame comprises the following steps:
establishing a mapping equation of point spread functions of each field of view and each wave band of the single lens about surface shape parameters of the single lens, performing convolution interpolation on the point spread functions and the original image to obtain a single lens fuzzy simulation image, and constructing an optical image fuzzy simulation module;
taking a point spread function of a 0-degree view field in the point spread function as an estimated fuzzy core, constructing a neural network, correcting the estimated fuzzy core based on the neural network, acquiring a correction result, and constructing the fuzzy core learning module;
constructing the inverse filtering image restoration module based on an adaptive wiener filtering method and a fuzzy kernel output by the fuzzy kernel learning module;
and constructing the single-lens imaging system framework based on the optical image blur simulation module, the blur kernel learning module and the inverse filtering image restoration module.
Optionally, in the process of constructing the optical image blur simulation module, the point spread function is obtained based on a geometrical optics principle and a ray tracing method; and setting the aspheric surface parameters of the single lens as surface shape parameters, wherein the single lens fuzzy simulation graph is differentiable about the aspheric surface parameters.
Optionally, the neural network is a three-layer fully-connected neural network with a skip-connected structure, each layer of fully-connected layer comprising 27 × 27 neurons.
Optionally, the method for correcting the estimated blur kernel by the neural network includes: transforming the pre-estimated fuzzy kernel into a one-dimensional matrix form and inputting the one-dimensional matrix form into the neural network; and calculating the output result of each layer of neural network, converting the output result of the third layer into an image matrix form, forming a corrected fuzzy core and performing zero filling.
Optionally, an algorithm expression of the adaptive wiener filtering method is as follows:
wherein the content of the first and second substances,
representing the restored image, F (-) representing the Fourier transform, F -1 (. -) denotes the inverse Fourier transform, K is an optimizable parameter adaptively adjusted by learning training, I 1 In order to blur the simulation image with a single lens,
and the fuzzy cores are output by the fuzzy core learning module.
Optionally, the formula for calculating the square error loss mselos between the restored image and the original image is:
where m, n denotes the size of the image, I, j denotes the pixel position in the image, I 0 Representing the original image, mselos representing the squared error loss of the original image and the restored image.
Optionally, the additional constraint penalty is calculated by:
in the formula, loss 0 Represents the additional constraint penalty, k n Representing the current value of the variable, k y Represents variable threshold, sigmoid represents activation function, when k is n ≥k y When the additional loss value is not activated, when k n ＜k y The parasitic loss value is activated.
Optionally, the method for constructing the loss function based on the squared error loss and the additional constraint loss is: and carrying out weighted summation on the squared error loss and the additional constraint loss, and constructing a loss function based on the result of the weighted summation.
Optionally, the method for iteratively optimizing the single-lens imaging system frame includes:
and (3) setting the initial aspheric surface parameter and the parameter of the neural network as 0 initially, and optimizing the aspheric surface parameter, the neural network and the inverse filtering self-adaptive parameter together.
The invention has the following technical effects:
(1) The invention establishes an end-to-end single lens imaging system frame, and simultaneously optimizes the optical system surface shape parameter, the ResDNN3 neural network parameter and the noise constant parameter in the wiener filtering image restoration algorithm in the single lens imaging system frame according to the imaging effect of the system.
(2) The invention provides a full-connection neural network (ResDNN 3) with a skip connection structure, which takes an estimated fuzzy core as an input and can be used for learning and correcting the fuzzy core of an optical system.
(3) The invention adds the additional constraint loss of the optical system in the training optimization of the end-to-end single lens imaging system, and can constrain the edge thickness, the energy distribution and the like of the designed single lens.
(4) The invention provides a set of initialization method for designing the single-lens imaging system framework, so that the single-lens imaging system has a better initial structure, the system framework has a good initial value during training, and the difficulty in training and optimizing the system framework is greatly reduced.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:
FIG. 1 is a flow chart of a method for designing an end-to-end single lens imaging system in an embodiment of the invention;
fig. 2 is a structure diagram of a ResDNN3 network in an embodiment of the present invention;
FIG. 3 is a light path diagram of a single lens in an embodiment of the present invention;
fig. 4 is a comparison diagram of point spread functions of each field of view of a single lens before and after optimization in the embodiment of the present invention, where (a) is a point spread function of each field of view of an unoptimized optical system, and (b) is a point spread function of each field of view of an optical system obtained by learning optimization;
FIG. 5 is a diagram illustrating the learning optimization effect of fuzzy core according to an embodiment of the present invention, where (a) is an unoptimized estimated fuzzy core and (b) is a learning-optimized fuzzy core;
Detailed Description
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than here.
Example one
The embodiment provides a design method of an end-to-end single-lens imaging system, and the method can be effectively applied to the fields of smart phone cameras, unmanned aerial vehicle platform camera systems, remote sensing cameras and the like which need to be miniaturized imaging systems, and the flow chart is shown in fig. 1. In this embodiment, a single-lens imaging system with a focal length of 43.5mm, a clear aperture of 23.4mm, and a full field of view of 47 ° is taken as an example to describe a specific embodiment of the present invention:
the method comprises the following steps: the established end-to-end single lens imaging system framework comprises three modules: the device comprises an optical image fuzzy simulation module, a fuzzy kernel learning module and an inverse filtering image restoration module. The three modules are set up as follows:
step 1-1, an optical image fuzzy simulation module is established. Firstly, establishing a mapping equation of point spread functions of each field of view and each wave band of the single lens on surface shape parameters of the single lens, and then carrying out convolution interpolation on the point spread functions and an original image to obtain a single lens fuzzy simulation image so as to realize optical image fuzzy simulation.
The aspheric surface parameters of the single lens are set as the surface shape parameters which can be optimized, and the fuzzy image in the established image simulation model is differentiable about the aspheric surface parameters. The single lens used in the invention is a plano-convex even-order aspheric lens, the front surface of the lens is a plane, and the back surface of the lens is a curved surface using the parameters of 4 th, 6 th, 8 th and 10 th order even-order aspheric surfaces.
The invention firstly establishes a mapping equation from the light incidence position and direction to the image surface position under a single lens optical system through light tracing based on the geometrical optics principle to establish a functional relation between the surface shape parameter and the point spread function:
G(x 0 ,y 0 ,θ,{a 4 ,a 6 ,a 8 ,a 10 })＝(x 1 ,y 1 )
wherein x 0 ,y 0 Theta is the incident position and direction, { a n Is the set of surface shape parameters to be optimized, x 1 ,y 1 And G (-) is the mapping transformation from the incident position and direction of the light to the image plane position. To establish this mapping equation, the surface function F (x, y, z, a) of the optical lens is known n ) =0, refractive index of the lens material and position of the respective optical surface. The general surface shape equation for an even aspheric surface is:
wherein r is the position in the radial direction, z is the sagittal diameter at the corresponding position, c is the curvature at the vertex, r is the radius, k is the conicity, a 2 、a 4 、a 6 Etc. are aspheric coefficients. In the present example, the lens thickness d is 6mm, the apex radius of curvature r is-21.4 mm, the conicity k is 0 4 、a 6 、a 8 And a 10 Are aspheric coefficients to be optimized.
The lens material used in the examples was PMMA, which has a refractive index n at the center wavelength of the visible band 1 =1.49, space using lightThe refraction law can obtain the azimuth angle after the light is refracted, the spatial refraction law and the linear propagation law of the light can be used for solving to obtain the spatial position and the direction angle of each refraction of the light on the lens surface, and the layer-by-layer solution can be obtained by x 0 ,y 0 Theta gives x 1 ,y 1 。
7 x 7 point spread sampling areas are uniformly selected on an image surface, the size of each sampling area is 27 x 27 pixels, and the incident angle of a chief ray corresponding to the center position of each point spread sampling area is calculated. For each incidence angle, respectively creating 64 multiplied by 64 parallel sampling light ray arrays on an incidence plane, and calculating the drop point distribution of the light rays in a point diffusion sampling area, wherein the light ray quantity in each pixel block in the point diffusion sampling area is used as a parameter value of a corresponding parameter in a point diffusion function.
Setting the spatial position of each pixel center in each point diffusion sampling area of the image surface as x 2 ,y 2 And then the distance between the center of each pixel on the image plane and the light ray falling point is as follows:
then, diffusing the light ray falling point of the image plane by using a Gaussian function, and endowing different weights to the pixels according to the distance between the light ray falling point and the spatial center of the pixels:
wherein, the standard deviation of the sigma Gaussian function can be adjusted according to the pixel size of the detector. The intensity distributions of all the falling points of the sampling light rays with the same incidence angle are superposed and normalized, so that the point spread function of the corresponding position can be obtained:
and (3) respectively convolving the point spread functions with the original image, and then fusing the degraded images convolved with the point spread functions into a fuzzy simulation image by using an interpolation function and adding a noise item. This process is described as:
wherein, I 0 As an original image, I 1 SINC for single lens blur simulation image, η being noise ij And (5) representing the SINC function weight graph of the central point at the (i, j) position.
Step 1-2, establishing a fuzzy core optimization learning module. And (3) taking the point spread function of the central field obtained in the step (1-1) as an estimated fuzzy kernel, and establishing a neural network for transforming the estimated fuzzy kernel into a fuzzy kernel representing the characteristics of the whole optical system, thereby realizing the function of learning the fuzzy kernel.
Predicted fuzzy kernel psf used 00 Transforming the point spread function of the central wave band of the central field obtained in the step one into a form H of a one-dimensional matrix 0 ＝[h 0 ,h 1 ,…,h 27×27 ]And inputting the data into a ResDNN3 network established by the invention.
The structure of the ResDNN3 network established herein is a three-layer fully-connected neural network with a skip-connection structure, as shown in fig. 2, in which all neurons are connected to each other, and each layer of fully-connected layer has 27 × 27 neurons. Skipping connections means that the output of each layer of the neural network needs to be added with the input of the neural network, so that the effects of enriching network information and the like are achieved.
The skipping of the connection can introduce the information of the neuron in the previous layer in a simple manner, and hardly increases the extra computation amount, and the method is widely applied to the convolutional neural network at present, and can also exert corresponding effects in the fully-connected neural network. Order to
Represents the jth neuron in the xth fully-connected network,
represents the last oneThe ith entry in the layer input is,
represents the jth neuron in the upper layer,
and
representing the weight and bias of this connection, respectively, then the calculation of each neuron in the neural network is:
wherein the weight w i And bias b i And as an optimization variable of the neural network, the calculation result of all the neurons in each layer of the fully-connected neural network is used as the output of the layer of the neural network. Therefore, the output result of each layer of the neural network in the invention is as follows in sequence:
output result H of the third layer 3 Transformed into the form of an image matrix to form a modified blur kernel psf H3 . Zero-filling the corrected blur kernel to the blurred image I 1 The point spread function after zero padding is the same size
And (4) showing.
And 1-3, establishing an inverse filtering image restoration module. And the self-adaptive wiener filtering method is used as an inverse filtering algorithm of the image restoration module, and the fuzzy kernel used by the image restoration module is the fuzzy kernel output by the fuzzy kernel learning module.
Wiener filtering is also called minimum mean square error filtering, and is an inverse filter recovery algorithm considering noise interference. Order to
Representing a restored image, S η As power spectrum of noise, S f For the power spectrum of the degradation function, F (-) represents the Fourier transform,
and optimizing the fuzzy core output by the learning module for the fuzzy core, so that the algorithm expression for restoring the image by using the wiener filter is as follows:
wherein the content of the first and second substances,
terms are difficult to compute accurately and are often set to constants when used in the past. In the invention, the parameters are set as optimizable parameters and are adaptively adjusted through learning training. Let F -1 (DEG) is inverse Fourier transform, K value is adaptive parameter, then image is restored
The expression of (c) is:
in step 2, when m, n denotes the size of the image, i, j denotes the pixel position in the image, and mselos denotes the square error loss between the original image and the restoration image, the square error loss between the original image and the restoration image is:
additional constraint loss through the current value k of the variable for an optical system n And a set variable threshold k y Introducing the difference value, using sigmoid function as activation function, when k is n ≥k y When this loss term is inactive, when k n ＜k y The parasitic loss value is activated. The invention can specifically make additional constraint on the thickness of the edge of the lens and the energy distribution, and the general paradigm is as follows:
wherein, for a single lens, the current value k of the variable n There are two main types of lens edge thickness and energy distribution. Let R be the position in the radial direction, radius R 1 (R) and R 2 (r) is a coordinate function of the front and rear surfaces in the direction of the optical axis, r 0 The current value k of the variation of the edge thickness is the radial radius of the lens n1 The expression of (a) is:
k n1 ＝R 2 (r 0 )-R 1 (r 0 )
the system energy value can be represented by the sum of the image surface falling point values of the sampling parallel ray array, the energy sum in a point spread sampling range, or the light falling point energy sum in a smaller range or even a central pixel, and if m and n are considered energy transfer ranges, the variable current values of the energy distribution are as follows:
k n2 ＝sum(psf(x,y)),x≤m,y≤n
weighting and summing the square error loss and the additional constraint loss to obtain a loss function loss of the whole end-to-end design, and setting the weights of the square error loss and the additional constraint loss as alpha and beta respectively, then:
loss＝α*mseloss+β 1 *loss 0 (k n1 )+β 2 *loss 0 (k n2 )
step three: iterative optimization is carried out on trainable parameters of the end-to-end single-lens imaging system by utilizing a deep learning technology to obtain optimal parameters of the single-lens imaging system
The initial aspheric parameters and resDNN3 network parameters are both initialized to 0, so at the initial training of deep learning: the optical system is a standard spherical mirror, the estimated fuzzy core is a point spread function of the standard spherical mirror under a 0-degree view field, the operation result of the resDNN3 network on the estimated fuzzy core is still the point spread function of the standard spherical mirror under the 0-degree view field, and the initial value of the characteristic parameter of the wiener filtering noise is set to be 0.01. Compared with random initialization parameters, the initialization method has certain physical significance, and a good imaging effect can be obtained when a system is trained for the first time. Therefore, the single-lens imaging system has a good initial structure, and the training difficulty of deep learning is greatly reduced.
A clear scene image with an image size of 256 × 256 was selected as a data set, with 350 pictures in the training set and 50 pictures in the test set. In each iteration of deep learning training, calculating the loss value of a training set to perform gradient reduction, and updating parameters; and calculating loss values of the test set for verifying the effect of the single-lens imaging system without updating parameters. The learning rate is set to 1 × 10 -4 And repeating the iteration for 100 times, and selecting a parameter with the minimum loss value of the test set as a final optimization result.
Aspheric parameter a of single lens 4 、a 6 、a 8 And a 10 The optimized values become 1.766 multiplied by 10 respectively -5 、-9.100×10 -9 、-4.052×10 -11 And 9.894X 10 -12 Noise constant parameter K =2 × 10 optimized in wiener filtering -4 . The optical path diagram of the designed single lens is shown in fig. 3. As can be seen from the comparison between the standard spherical mirror (a) of the unoptimized optical system and the optical system (b) with optimized aspheric parameters in FIG. 4, the optimization of the end-to-end single-lens imaging system to the surface shape of the single lens makes the point spread function of the edge field of view of the single lens more concentrated and more consistent with the point spread function of the central field of view, so that a learned blur is usedAnd (4) checking the single-lens blurred image to perform inverse filtering restoration, so that a good restoration effect can be obtained. As can be seen from the comparison between the estimated blur kernel (a) and the blur kernel (b) representing the characteristics of the single lens full field obtained by learning in fig. 5, the learned blur kernel is more complex, the distribution of the blur kernel is related to the spatial position, and the ResDNN3 network has a good learning effect.
The restoration algorithm part in the single-lens imaging system can restore the blurred image acquired by the single lens to a clear image close to the original image. Measuring the similarity of the two images by using the peak signal-to-noise ratio and the structural similarity as evaluation indexes, and obtaining that the structural similarity between the blurred image obtained by the single lens and the original image is 0.63 and the peak signal-to-noise ratio is 20.93 through calculation; the structural similarity between the restored image and the original image is 0.93, the peak signal-to-noise ratio is 25.47, and the effectiveness of the design of the invention is also proved through quantitative index comparison.
The invention establishes an end-to-end single lens imaging system framework, and simultaneously optimizes the surface shape parameter of an optical system, the ResDNN3 neural network parameter and the noise constant parameter in the wiener filtering image restoration algorithm according to the imaging effect of the system.
The invention provides a full-connection neural network (ResDNN 3) with a skip connection structure, which takes an estimated fuzzy core as input and can be used for learning and correcting the fuzzy core of an optical system.
The invention adds the additional constraint loss of the optical system in the training optimization of the end-to-end single lens imaging system, and can constrain the edge thickness, the energy distribution and the like of the designed single lens.
The invention provides a set of initialization method for the designed single-lens imaging system framework, so that the system framework has a good initial value during training, and the difficulty in training and optimizing the system framework is greatly reduced.
The above description is only for the preferred embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (8)
1. A method of designing an end-to-end single lens imaging system, comprising the steps of:
calculating the square error loss and the additional constraint loss of the restored image and the original image, and constructing a loss function based on the square error loss and the additional constraint loss;
constructing a single-lens imaging system frame, carrying out iterative optimization on the single-lens imaging system frame based on deep learning and the loss function to obtain an optimized system, and taking the optimized system as a single-lens imaging system;
the single lens imaging system frame includes: the system comprises an optical image fuzzy simulation module, a fuzzy kernel learning module and an inverse filtering image restoration module;
the process of constructing the single lens imaging system frame comprises the following steps:
establishing a mapping equation of point spread functions of each view field and each wave band of the single lens about surface shape parameters of the single lens, performing convolution interpolation on the point spread functions and an original image to obtain a single lens fuzzy simulation image, and constructing an optical image fuzzy simulation module;
taking the point spread function as an estimated fuzzy core, constructing a neural network, correcting the estimated fuzzy core based on the neural network, acquiring a correction result, and constructing a fuzzy core learning module;
constructing the inverse filtering image restoration module based on an adaptive wiener filtering method and a fuzzy kernel output by the fuzzy kernel learning module;
constructing the single-lens imaging system framework based on the optical image blur simulation module, the blur kernel learning module and the inverse filtering image restoration module;
the calculation method of the additional constraint loss comprises the following steps:
in the formula, loss 0 Represents the additional constraint penalty, k n Representing the current value of the variable, k y Representing variable threshold, sigmoid represents activation function, when k is n ≥k y When the additional loss value is not activated, when k n ＜k y The parasitic loss value is activated.
2. The design method of an end-to-end single-lens imaging system according to claim 1, wherein in the process of constructing the optical image blur simulation module, the point spread function is obtained based on a geometrical optics principle and a ray tracing method; and setting the aspheric surface parameters of the single lens as surface shape parameters, wherein the single lens fuzzy simulation graph is differentiable about the aspheric surface parameters.
3. The end-to-end single lens imaging system design method of claim 1, wherein the neural network is a three-layer fully-connected neural network ResDNN3 with a skip-connected structure, each layer fully-connected layer comprising 27 x 27 neurons.
4. The method for designing an end-to-end single-lens imaging system according to claim 1, wherein the method for correcting the estimated blur kernel by the neural network comprises: transforming the pre-estimated fuzzy kernel into a one-dimensional matrix form and inputting the one-dimensional matrix form into the neural network; and calculating the output result of each layer of neural network, converting the output result of the third layer into an image matrix form, forming a corrected fuzzy core and performing zero filling.
5. The method of designing an end-to-end single lens imaging system according to claim 1, wherein the algorithm expression of the adaptive wiener filtering method is as follows:
wherein the content of the first and second substances,
representing the restored image, F () representing the Fourier transform, F -1 () Denotes the inverse Fourier transform, K is an optimizable parameter adaptively adjusted by learning training, I 1 In order to blur the simulation image with a single lens,
and the fuzzy kernel is output by the fuzzy kernel learning module.
6. The method of designing an end-to-end single lens imaging system of claim 1, wherein the equation for calculating the squared error loss mselos between the restored image and the original image is:
where m, n denotes the size of the image, I, j denotes the pixel position in the image, I 0 The original image is shown, and mselos shows the squared error loss between the original image and the restored image.
7. The method of designing an end-to-end single lens imaging system of claim 1, wherein the method of constructing a loss function based on the squared error loss and the additive constraint loss is: and performing weighted summation on the squared error loss and the additional constraint loss, and constructing a loss function based on the result of the weighted summation.
8. The method of designing an end-to-end single lens imaging system of claim 1, wherein the iterative optimization of the single lens imaging system framework is performed by: and setting the initial aspheric surface parameters and the initial parameter values of the neural network as 0, and jointly optimizing the aspheric surface parameters, the neural network and the self-adaptive parameters of the inverse filtering image restoration module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210522840.9A CN114967121B (en) | 2022-05-13 | 2022-05-13 | Design method of end-to-end single lens imaging system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210522840.9A CN114967121B (en) | 2022-05-13 | 2022-05-13 | Design method of end-to-end single lens imaging system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114967121A CN114967121A (en) | 2022-08-30 |
| CN114967121B true CN114967121B (en) | 2023-02-03 |
Family
ID=82984078
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210522840.9A Active CN114967121B (en) | 2022-05-13 | 2022-05-13 | Design method of end-to-end single lens imaging system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114967121B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116862800B (en) * | 2023-07-11 | 2024-01-30 | 哈尔滨工业大学 | Large-view-field single-lens space-variant blurred image restoration method and device |
| CN117233960B (en) * | 2023-11-15 | 2024-01-23 | 清华大学 | Optical system on-line design method and device based on intelligent light calculation |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5786582A (en) * | 1992-02-27 | 1998-07-28 | Symbol Technologies, Inc. | Optical scanner for reading and decoding one- and two-dimensional symbologies at variable depths of field |
| CN102395917A (en) * | 2009-02-17 | 2012-03-28 | 领先角膜控股有限责任公司 | Ophthalmic lens with optical sectors |
| WO2017134275A1 (en) * | 2016-02-05 | 2017-08-10 | Eidgenossische Technische Hochschule Zurich | Methods and systems for determining an optical axis and/or physical properties of a lens and use of the same in virtual imaging and head-mounted displays |
| WO2018045602A1 (en) * | 2016-09-07 | 2018-03-15 | 华中科技大学 | Blur kernel size estimation method and system based on deep learning |
| CN110009674A (en) * | 2019-04-01 | 2019-07-12 | 厦门大学 | Monocular image depth of field real-time computing technique based on unsupervised deep learning |
| CN110458901A (en) * | 2019-06-26 | 2019-11-15 | 西安电子科技大学 | A kind of optimum design method of overall importance based on the photo electric imaging system for calculating imaging |
| CN111709895A (en) * | 2020-06-17 | 2020-09-25 | 中国科学院微小卫星创新研究院 | Image blind deblurring method and system based on attention mechanism |
| CN112036137A (en) * | 2020-08-27 | 2020-12-04 | 哈尔滨工业大学（深圳） | Deep learning-based multi-style calligraphy digital ink simulation method and system |
| CN112329920A (en) * | 2020-11-06 | 2021-02-05 | 深圳先进技术研究院 | Unsupervised training method and unsupervised training device for magnetic resonance parameter imaging model |
| CN113077540A (en) * | 2021-03-31 | 2021-07-06 | 点昀技术(南通)有限公司 | End-to-end imaging equipment design method and device |
| CN113191983A (en) * | 2021-05-18 | 2021-07-30 | 陕西师范大学 | Image denoising method and device based on deep learning attention mechanism |
| WO2021218119A1 (en) * | 2020-04-30 | 2021-11-04 | 中国科学院深圳先进技术研究院 | Image toning enhancement method and method for training image toning enhancement neural network |
| CN114063282A (en) * | 2021-11-30 | 2022-02-18 | 哈尔滨工业大学 | Large-view-field single lens surface shape optimization method and device based on point spread function |
| CN114418883A (en) * | 2022-01-18 | 2022-04-29 | 北京工业大学 | Blind image deblurring method based on depth prior |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7616841B2 (en) * | 2005-06-17 | 2009-11-10 | Ricoh Co., Ltd. | End-to-end design of electro-optic imaging systems |
| KR20200094058A (en) * | 2019-01-29 | 2020-08-06 | 한국과학기술원 | Lensless Hyperspectral Imaging Method and Apparatus Therefore |
| CN113296259B (en) * | 2021-05-25 | 2022-11-08 | 中国科学院国家天文台南京天文光学技术研究所 | Super-resolution imaging method and device based on aperture modulation subsystem and deep learning |
-
2022
- 2022-05-13 CN CN202210522840.9A patent/CN114967121B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5786582A (en) * | 1992-02-27 | 1998-07-28 | Symbol Technologies, Inc. | Optical scanner for reading and decoding one- and two-dimensional symbologies at variable depths of field |
| CN102395917A (en) * | 2009-02-17 | 2012-03-28 | 领先角膜控股有限责任公司 | Ophthalmic lens with optical sectors |
| WO2017134275A1 (en) * | 2016-02-05 | 2017-08-10 | Eidgenossische Technische Hochschule Zurich | Methods and systems for determining an optical axis and/or physical properties of a lens and use of the same in virtual imaging and head-mounted displays |
| WO2018045602A1 (en) * | 2016-09-07 | 2018-03-15 | 华中科技大学 | Blur kernel size estimation method and system based on deep learning |
| CN110009674A (en) * | 2019-04-01 | 2019-07-12 | 厦门大学 | Monocular image depth of field real-time computing technique based on unsupervised deep learning |
| CN110458901A (en) * | 2019-06-26 | 2019-11-15 | 西安电子科技大学 | A kind of optimum design method of overall importance based on the photo electric imaging system for calculating imaging |
| WO2021218119A1 (en) * | 2020-04-30 | 2021-11-04 | 中国科学院深圳先进技术研究院 | Image toning enhancement method and method for training image toning enhancement neural network |
| CN111709895A (en) * | 2020-06-17 | 2020-09-25 | 中国科学院微小卫星创新研究院 | Image blind deblurring method and system based on attention mechanism |
| CN112036137A (en) * | 2020-08-27 | 2020-12-04 | 哈尔滨工业大学（深圳） | Deep learning-based multi-style calligraphy digital ink simulation method and system |
| CN112329920A (en) * | 2020-11-06 | 2021-02-05 | 深圳先进技术研究院 | Unsupervised training method and unsupervised training device for magnetic resonance parameter imaging model |
| CN113077540A (en) * | 2021-03-31 | 2021-07-06 | 点昀技术(南通)有限公司 | End-to-end imaging equipment design method and device |
| CN113191983A (en) * | 2021-05-18 | 2021-07-30 | 陕西师范大学 | Image denoising method and device based on deep learning attention mechanism |
| CN114063282A (en) * | 2021-11-30 | 2022-02-18 | 哈尔滨工业大学 | Large-view-field single lens surface shape optimization method and device based on point spread function |
| CN114418883A (en) * | 2022-01-18 | 2022-04-29 | 北京工业大学 | Blind image deblurring method based on depth prior |
Non-Patent Citations (2)
| Title |
|---|
| 基于简单透镜计算成像的图像复原重建;王新华等;《吉林大学学报(工学版)》;20170531(第03期);全文 * |
| 最大似然空间变化图像恢复算法;王治乐等;《红外与激光工程》;20120725(第07期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114967121A (en) | 2022-08-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Sun et al. | End-to-end complex lens design with differentiable ray tracing | |
| CN114967121B (en) | Design method of end-to-end single lens imaging system | |
| Sitzmann et al. | End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging | |
| US11721002B2 (en) | Imaging system and method for imaging objects with reduced image blur | |
| Elmalem et al. | Learned phase coded aperture for the benefit of depth of field extension | |
| EP1734746B1 (en) | End-to-end design of electro-optic imaging systems | |
| CN108537746B (en) | Fuzzy variable image blind restoration method based on deep convolutional network | |
| US11354537B2 (en) | Image processing apparatus, imaging apparatus, image processing method, and storage medium | |
| CN110533607A (en) | A kind of image processing method based on deep learning, device and electronic equipment | |
| Akpinar et al. | Learning wavefront coding for extended depth of field imaging | |
| CN113570508A (en) | Image restoration method and device, storage medium and terminal | |
| Chen et al. | U-net like deep autoencoders for deblurring atmospheric turbulence | |
| Shi et al. | Rapid all-in-focus imaging via physical neural network optical encoding | |
| Sweere et al. | Deep learning-based super-resolution and de-noising for XMM-newton images | |
| Yang et al. | Curriculum learning for ab initio deep learned refractive optics | |
| CN111415303A (en) | Zone plate coding aperture imaging method and device based on deep learning | |
| US20070093993A1 (en) | End-to-end design of electro-optic imaging systems using backwards ray tracing from the detector to the source | |
| Lin et al. | Non-blind optical degradation correction via frequency self-adaptive and finetune tactics | |
| CN114859550B (en) | End-to-end design method for Fresnel single-lens calculation imaging system | |
| Paxman et al. | Spatial stabilization of deep-turbulence-induced anisoplanatic blur | |
| CN116612005A (en) | Super-resolution reconstruction method and device for hyperspectral image and electronic equipment | |
| Olivas et al. | Platform motion blur image restoration system | |
| CN114494065A (en) | Image deblurring method, device and equipment and readable storage medium | |
| CN115311149A (en) | Image denoising method, model, computer-readable storage medium and terminal device | |
| Su et al. | Atmospheric turbulence degraded image restoration using a modified dilated convolutional network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |