CN114494258A - Lens aberration prediction and image reconstruction method and device - Google Patents
Lens aberration prediction and image reconstruction method and device Download PDFInfo
- Publication number
- CN114494258A CN114494258A CN202210394888.6A CN202210394888A CN114494258A CN 114494258 A CN114494258 A CN 114494258A CN 202210394888 A CN202210394888 A CN 202210394888A CN 114494258 A CN114494258 A CN 114494258A
- Authority
- CN
- China
- Prior art keywords
- aberration
- view
- picture
- image reconstruction
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10052—Images from lightfield camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20228—Disparity calculation for image-based rendering
Abstract
The application relates to the technical field of image data processing or generation, in particular to a method and a device for aberration prediction and image reconstruction of a lens, wherein the method comprises the following steps: scanning the collected data of the light field camera to obtain a multi-view picture; extracting view angle characteristics of each view angle from the multi-view-angle picture; based on the view angle characteristics of each view angle, inputting the multi-view-angle pictures into an aberration prediction model according to a preset sequence to obtain an aberration prediction result; and acquiring aberration higher than the first preset resolution according to the aberration prediction result, and obtaining an image reconstruction picture according to the aberration. Therefore, the technical problems that the related technology is based on physical means or a simple neural network, the propagation process is complex, the robustness is poor, time consumption and energy consumption are large, and the reconstruction effect is poor are solved.
Description
Technical Field
The present disclosure relates to the field of image data processing or generation, and more particularly, to a method and an apparatus for aberration prediction and image reconstruction of a lens.
Background
Two-dimensional imaging sensors have revolutionized almost all areas including industrial detection, mobile devices, automotive driving, surveillance, medical diagnostics, biology and astronomy, benefiting from the rapid growth of the semiconductor industry, with the pixel size of digital sensors growing rapidly over the past decade. However, the practical performance of most imaging systems has reached the bottleneck of optics rather than electronics, for example, for a gigapixel sensor, the number of pixels available for a normal imaging system is typically limited to the megapixel level due to optical aberrations caused by imperfect lenses or environmental disturbances, resulting in light emanating from a point being spread over a large area on a two-dimensional sensor. Meanwhile, projecting a 3D scene onto a 2D plane can result in a loss of various degrees of freedom of LF (Light Field), such as depth and local coherence. Therefore, obtaining high density depth maps using integrated sensors has been a challenge.
The related art uses scanning light field imaging, and realizes robust, general and high-performance 3D imaging at low cost based on a Digital Adaptive Optics (AO) algorithm, namely a Digital Adaptive Optics (DAO) algorithm, of the wave Optics. Specifically, the method simulates the forward and reverse processes of light propagation in a physical mode, sets an initial aberration value, calculates a lens point spread function, calculates a high-resolution picture in a deconvolution process, calculates a picture to be shot by the picture and the point spread function, compares the picture with the existing sensor signal, corrects the aberration, and continuously iterates until convergence.
However, the related art still has certain defects, which are mainly divided into two types:
firstly, based on physics, the forward and backward propagation process is complex, the computer simulation time is long, whether convergence is uncertain, the iterative processes are different for different process iteration modes, the robustness is poor, the deconvolution process consumes a lot of time and energy, and the iteration causes the process to be carried out for many times, which is unavoidable and difficult to be parallel.
Secondly, based on a simple Neural network, such as a 3D CNN (Convolutional Neural network), the network is overloaded in the whole process, and the performance of the network to predict a point spread function or an aberration is unstable or has low precision, so that the reconstruction effect is poor.
In summary, the related art cannot guarantee the parallax estimation accuracy and speed at the same time, and improvement is needed.
Disclosure of Invention
The application provides a method and a device for aberration prediction and image reconstruction of a lens, which are used for solving the technical problems of poor reconstruction effect caused by complex propagation process, poor robustness, large time consumption and energy consumption due to the fact that related technologies are based on physical means or simple neural networks.
An embodiment of a first aspect of the present application provides a method for predicting aberration of a lens and reconstructing an image, including the following steps: scanning the collected data of the light field camera to obtain a multi-view picture; extracting view characteristic of each view from the multi-view picture; based on the view angle characteristics of each view angle, inputting the multi-view-angle pictures into an aberration prediction model according to a preset sequence to obtain an aberration prediction result; and acquiring aberration higher than a first preset resolution according to the aberration prediction result, and obtaining an image reconstruction picture according to the aberration.
Optionally, in an embodiment of the application, the obtaining, according to the aberration prediction result, an aberration higher than a preset resolution includes: based on the aberration prediction result, utilizing a fitting of a Zernike polynomial to generate an aberration higher than the preset resolution.
Optionally, in an embodiment of the present application, the obtaining an image reconstruction picture according to the aberration includes: and calculating an actual point spread function by using a preset point spread function according to the aberration, and performing deconvolution on the actual point spread function and the multi-view picture to obtain an image reconstruction picture.
Optionally, in an embodiment of the application, the inputting the multi-view picture to an aberration prediction model according to a preset order based on the view characteristic of each view to obtain an aberration prediction result includes: and predicting the aberration of each view angle by using a multi-head attention mechanism, wherein the aberration prediction model learns relevant local characteristics of the aberration to obtain the aberration lower than a second preset resolution.
The second aspect of the present application provides an aberration prediction and image reconstruction apparatus for a lens, including: the scanning module is used for scanning the collected data of the light field camera to obtain a multi-view picture; an extraction module, configured to extract view features of each view from the multi-view picture; the aberration prediction module is used for inputting the multi-view pictures into an aberration prediction model according to a preset sequence based on the view angle characteristics of each view angle to obtain an aberration prediction result; and the reconstruction module is used for acquiring the aberration higher than a first preset resolution according to the aberration prediction result and obtaining an image reconstruction picture according to the aberration.
Optionally, in an embodiment of the present application, the reconstruction module includes: a fitting unit for generating an aberration higher than the preset resolution by fitting of a Zernike polynomial based on the aberration prediction result.
Optionally, in an embodiment of the present application, the reconstruction module includes: and the calculating unit is used for calculating an actual point spread function by using a preset point spread function according to the aberration, and performing deconvolution with the multi-view picture to obtain an image reconstruction picture.
Optionally, in an embodiment of the present application, the aberration prediction module includes: and the predicting unit is used for predicting the aberration of each view angle by using a multi-head attention mechanism, wherein the aberration predicting model learns the relevant local characteristics of the aberration to obtain the aberration lower than a second preset resolution.
An embodiment of a third aspect of the present application provides an electronic device, including: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the aberration prediction and image reconstruction method of the lens as described in the above embodiments.
A fourth aspect of the present application provides a computer-readable storage medium, which stores computer instructions for causing the computer to execute the aberration prediction and image reconstruction method for a lens according to the foregoing embodiments.
According to the image reconstruction method and device, the scanning light field camera can be used for obtaining the multi-view images, the view angle characteristics of each view angle are extracted, the aberration prediction result is obtained through the aberration prediction model, the image reconstruction photos are obtained according to the aberrations, high-speed image reconstruction without iteration and small in memory burden can be achieved while the parallax estimation precision is guaranteed, the performance is stable, the robustness is high, and the image reconstruction method and device can be used for achieving parallelism. Therefore, the technical problems that the related technology is based on physical means or a simple neural network, the propagation process is complex, the robustness is poor, time consumption and energy consumption are high, and the reconstruction effect is poor are solved.
Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a flowchart of a method for aberration prediction and image reconstruction of a lens according to an embodiment of the present application;
FIG. 2 is a flowchart of a method for aberration prediction and image reconstruction of a lens according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of an aberration prediction and image reconstruction apparatus for a lens according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application and should not be construed as limiting the present application.
Aberration prediction and image reconstruction methods and apparatuses of a lens according to embodiments of the present application are described below with reference to the accompanying drawings. In order to solve the technical problems that the reconstruction effect is poor due to the fact that the related technology mentioned in the background technology center is based on a physical means or a simple neural network, the propagation process is complex, the robustness is poor, time consumption and energy consumption are large, and therefore the reconstruction effect is poor, the method for predicting the aberration and reconstructing the image of the lens is provided. Therefore, the technical problems that the related technology is based on physical means or a simple neural network, the propagation process is complex, the robustness is poor, time consumption and energy consumption are large, and the reconstruction effect is poor are solved.
Specifically, fig. 1 is a schematic flowchart of a method for predicting aberration of a lens and reconstructing an image according to an embodiment of the present disclosure.
As shown in fig. 1, the aberration prediction and image reconstruction method for a lens includes the steps of:
in step S101, the collected data of the light field camera is scanned to obtain a multi-view picture.
In the actual implementation process, the embodiment of the application can obtain the data picture through the data of the scanning light field camera and align the data picture, so that the multi-view picture is obtained.
In step S102, a view feature of each view is extracted from the multi-view picture.
It can be understood that the network is a relatively common feature extractor, and can implement feature extraction in a fast parallelization manner, so that the embodiment of the application can input pictures of multiple view angles into the residual error neural network ResNet101 to extract features of each view angle.
In step S103, based on the view angle characteristics of each view angle, the multi-view images are input to the aberration prediction model according to a preset sequence, so as to obtain an aberration prediction result.
As a possible implementation manner, the embodiments of the present application may input the pictures of multiple views into the disparity prediction model transformer in a preset order, for example, from top to bottom and from left to right, and the disparity prediction model transformer predicts the gradient of each picture corresponding to the low resolution of the disparity by using a short 2-layer 2-multi-headed attention mechanism, so as to obtain the disparity prediction result.
It should be noted that the preset sequence can be set by those skilled in the art according to practical situations, and is not limited in particular here.
Optionally, in an embodiment of the present application, the inputting, based on a view characteristic of each view, the multi-view picture to the disparity prediction model according to a preset order to obtain a disparity prediction result includes: and predicting the aberration of each view angle by using a multi-head attention mechanism, wherein the aberration prediction model learns the relevant local characteristics of the aberration to obtain the aberration lower than a second preset resolution.
In the actual execution process, the aberration of each view angle can be predicted by adopting a multi-head attention mechanism, and relevant local characteristics of the aberration are learned, so that the aberration lower than a second preset resolution ratio is obtained.
It should be noted that the second preset resolution can be set by those skilled in the art according to practical situations, and is not limited in particular here.
In step S104, an aberration higher than a first preset resolution is obtained according to the aberration prediction result, and an image reconstruction picture is obtained according to the aberration.
Specifically, according to the embodiment of the application, the aberration higher than the first preset resolution can be obtained according to the aberration prediction result, an ideal point spread function is used for calculation, and deconvolution is performed on the multi-view image, so that the original image is reconstructed. According to the method and the device, the relevance of the multi-view image is processed through the aberration prediction model transform, and the attention mechanism is effectively utilized for learning, so that the gradient of the aberration is predicted more accurately and effectively, the iterative process is reduced, the consumed time of the deconvolution process is greatly shortened, the ill-conditioned problem does not exist, and the robustness is higher.
It should be noted that the first preset resolution may be set by a person skilled in the art according to practical situations, and is not limited in particular here.
Optionally, in an embodiment of the present application, obtaining the aberration higher than the preset resolution according to the aberration prediction result includes: based on the aberration prediction results, a fitting of a Zernike polynomial is utilized to generate aberrations above a preset resolution.
Further, the embodiments of the present application may perform a zernike polynomial fitting using the aberration with low resolution to generate the aberration higher than the preset resolution. The image reconstruction method and the image reconstruction device have the advantages that the image reconstruction is carried out by utilizing the Zernike polynomial, the aberration of most of lenses can be related, and therefore the image reconstruction efficiency is effectively improved.
Optionally, in an embodiment of the present application, obtaining an image reconstruction picture according to the aberration includes: and calculating an actual point spread function by using a preset point spread function according to the aberration, and performing deconvolution on the actual point spread function and the multi-view picture to obtain an image reconstruction picture.
In the actual execution process, the actual point spread function can be calculated according to the aberration by using the preset point spread function RL, and the image reconstruction picture can be obtained by performing fast deconvolution with the multi-view picture, so that the fast deployment and operation can be realized, and the image reconstruction efficiency can be effectively improved.
The aberration prediction and image reconstruction method for the lens according to the embodiment of the present application is described in detail with reference to fig. 2.
As shown in fig. 2, the embodiment of the present application includes the following steps:
step S201: scanning the collected data of the light field camera. In an actual implementation process, the embodiment of the present application may obtain a data picture by scanning data of the light field camera.
Step S202: and storing the multi-view pictures into multi-frame pictures. According to the embodiment of the application, the data pictures can be aligned, so that the multi-view pictures are obtained and then stored into the multi-frame pictures.
Step S203: the ResNet101 extracts multi-view picture features. According to the embodiment of the application, pictures of multiple view angles can be input into the residual error neural network ResNet101, and the features of each view angle are extracted.
Step S204: the Transformer predicts the phase gradient. As a possible implementation manner, the embodiments of the present application may input the pictures of multiple views into the disparity prediction model transformer in a preset order, for example, from top to bottom and from left to right, and the disparity prediction model transformer predicts the gradient of each picture corresponding to the low resolution of the disparity by using a short 2-layer 2-multi-headed attention mechanism, so as to obtain the disparity prediction result.
It should be noted that the preset sequence can be set by those skilled in the art according to practical situations, and is not limited in particular here.
Step S205: differing by the coefficients of the zernike polynomials. Further, the embodiments of the present application may perform a zernike polynomial fitting using the aberration with low resolution to generate the aberration higher than the preset resolution. The image reconstruction method and the image reconstruction device have the advantages that the image reconstruction is carried out by utilizing the Zernike polynomial, the aberration of most of lenses can be related, and therefore the image reconstruction efficiency is effectively improved.
Step S206: the actual point spread function. In the actual execution process, the actual point spread function can be calculated according to the aberration by using the preset point spread function RL, and the image reconstruction picture can be obtained by performing fast deconvolution with the multi-view picture, so that the fast deployment and operation can be realized, and the image reconstruction efficiency can be effectively improved.
Step S207: and calculating an actual picture. According to the method and the device, the image can be calculated and reconstructed through deconvolution of the multi-view image, and the image can also be calculated and reconstructed through an actual point spread function.
According to the aberration prediction and image reconstruction method for the lens, a multi-view image can be obtained by using a scanning light field camera, the view angle characteristics of each view angle are extracted, the aberration prediction result is obtained by using an aberration prediction model, and then an image reconstruction photo is obtained according to the aberration. Therefore, the technical problems that the related technology is based on physical means or a simple neural network, the propagation process is complex, the robustness is poor, time consumption and energy consumption are large, and the reconstruction effect is poor are solved.
Next, an aberration prediction and image reconstruction apparatus of a lens proposed according to an embodiment of the present application is described with reference to the drawings.
Fig. 3 is a block diagram illustrating an aberration prediction and image reconstruction apparatus for a lens according to an embodiment of the present application.
As shown in fig. 3, the aberration predicting and image reconstructing apparatus 10 of the lens includes: a scanning module 100, an extraction module 200, an aberration prediction module 300, and a reconstruction module 400.
Specifically, the scanning module 100 is configured to scan the collected data of the light field camera to obtain a multi-view picture.
An extracting module 200, configured to extract a view feature of each view from the multi-view picture.
The aberration prediction module 300 is configured to input the multi-view pictures into the aberration prediction model according to a preset sequence based on the view angle characteristics of each view angle, so as to obtain an aberration prediction result.
The reconstruction module 400 is configured to obtain an aberration higher than a first preset resolution according to the aberration prediction result, and obtain an image reconstruction picture according to the aberration.
Optionally, in an embodiment of the present application, the reconstruction module 400 includes: and a fitting unit.
And the fitting unit is used for utilizing the fitting of the Zernike polynomial to generate the aberration higher than the preset resolution ratio based on the aberration prediction result.
Optionally, in an embodiment of the present application, the reconstruction module 400 includes: and a computing unit.
The calculation unit is used for calculating an actual point spread function by using a preset point spread function according to the aberration, and performing deconvolution with the multi-view picture to obtain an image reconstruction picture.
Optionally, in an embodiment of the present application, the aberration prediction module 300 includes: and a prediction unit.
The prediction unit is used for predicting the aberration of each view angle by using a multi-head attention mechanism, wherein the aberration prediction model learns the relevant local characteristics of the aberration to obtain the aberration lower than a second preset resolution.
It should be noted that the foregoing explanation on the embodiments of the aberration prediction and image reconstruction method for a lens is also applicable to the aberration prediction and image reconstruction apparatus for a lens in this embodiment, and will not be described herein again.
According to the aberration prediction and image reconstruction device for the lens, provided by the embodiment of the application, a multi-view-angle picture can be obtained by using a scanning light field camera, the view angle characteristics of each view angle are extracted, the aberration prediction result is obtained by using an aberration prediction model, and then an image reconstruction picture is obtained according to the aberration. Therefore, the technical problems that the related technology is based on physical means or a simple neural network, the propagation process is complex, the robustness is poor, time consumption and energy consumption are large, and the reconstruction effect is poor are solved.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device may include:
The processor 402 implements the aberration prediction and image reconstruction method of the lens provided in the above-described embodiment when executing the program.
Further, the electronic device further includes:
a communication interface 403 for communication between the memory 401 and the processor 402.
A memory 401 for storing computer programs executable on the processor 402.
If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 may be connected to each other through a bus and perform communication with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 4, but this does not indicate only one bus or one type of bus.
Alternatively, in practical implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 may complete communication with each other through an internal interface.
The processor 402 may be a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement embodiments of the present Application.
The present embodiment also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the aberration prediction and image reconstruction method of a lens as above.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or N embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "N" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more N executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of implementing the embodiments of the present application.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or N wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are well known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.
Claims (10)
1. A method for predicting aberration of a lens and reconstructing an image, comprising the steps of:
scanning the collected data of the light field camera to obtain a multi-view picture;
extracting view characteristic of each view from the multi-view picture;
based on the view angle characteristics of each view angle, inputting the multi-view-angle pictures into an aberration prediction model according to a preset sequence to obtain an aberration prediction result; and
and obtaining the aberration higher than a first preset resolution according to the aberration prediction result, and obtaining an image reconstruction picture according to the aberration.
2. The method of claim 1, wherein obtaining the aberration higher than a preset resolution according to the aberration prediction result comprises:
based on the aberration prediction result, utilizing a fitting of a Zernike polynomial to generate an aberration higher than the preset resolution.
3. The method of claim 1, wherein obtaining the image reconstruction picture according to the aberration comprises:
and calculating an actual point spread function by using a preset point spread function according to the aberration, and performing deconvolution on the actual point spread function and the multi-view picture to obtain an image reconstruction picture.
4. The method according to claim 1, wherein the inputting the multi-view pictures into an aberration prediction model according to a preset order based on the view characteristics of each view to obtain an aberration prediction result comprises:
and predicting the aberration of each view angle by using a multi-head attention mechanism, wherein the aberration prediction model learns the relevant local characteristics of the aberration to obtain the aberration lower than a second preset resolution.
5. An aberration prediction and image reconstruction apparatus for a lens, comprising:
the scanning module is used for scanning the collected data of the light field camera to obtain a multi-view picture;
the extraction module is used for extracting the view angle characteristics of each view angle from the multi-view-angle picture;
the aberration prediction module is used for inputting the multi-view pictures into an aberration prediction model according to a preset sequence based on the view angle characteristics of each view angle to obtain an aberration prediction result; and
and the reconstruction module is used for acquiring the aberration higher than a first preset resolution according to the aberration prediction result and obtaining an image reconstruction picture according to the aberration.
6. The apparatus of claim 5, wherein the reconstruction module comprises:
a fitting unit for generating an aberration higher than the preset resolution by fitting of a Zernike polynomial based on the aberration prediction result.
7. The apparatus of claim 5, wherein the reconstruction module comprises:
and the calculating unit is used for calculating an actual point spread function by using a preset point spread function according to the aberration, and performing deconvolution with the multi-view picture to obtain an image reconstruction picture.
8. The apparatus of claim 5, wherein the aberration prediction module comprises:
and the predicting unit is used for predicting the aberration of each view angle by using a multi-head attention mechanism, wherein the aberration predicting model learns the relevant local characteristics of the aberration to obtain the aberration lower than a second preset resolution.
9. An electronic device, comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the aberration prediction and image reconstruction method of the lens according to any one of claims 1 to 4.
10. A computer-readable storage medium, on which a computer program is stored, the program being executable by a processor for implementing the aberration prediction and image reconstruction method of a lens according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210394888.6A CN114494258B (en) | 2022-04-15 | 2022-04-15 | Lens aberration prediction and image reconstruction method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210394888.6A CN114494258B (en) | 2022-04-15 | 2022-04-15 | Lens aberration prediction and image reconstruction method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114494258A true CN114494258A (en) | 2022-05-13 |
| CN114494258B CN114494258B (en) | 2022-08-30 |
Family
ID=81489631
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210394888.6A Active CN114494258B (en) | 2022-04-15 | 2022-04-15 | Lens aberration prediction and image reconstruction method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114494258B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023231138A1 (en) * | 2022-05-30 | 2023-12-07 | 元潼(北京)技术有限公司 | Multi-angle-of-view image super-resolution reconstruction method and apparatus based on meta-imaging |
| CN117541519A (en) * | 2024-01-09 | 2024-02-09 | 清华大学 | Optical field aberration correction method, optical field aberration correction device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030103189A1 (en) * | 2001-09-11 | 2003-06-05 | The Regents Of The University Of California | Characterizing aberrations in an imaging lens and applications to visual testing and integrated circuit mask analysis |
| US20150178923A1 (en) * | 2013-12-24 | 2015-06-25 | Lytro, Inc. | Light-field aberration correction |
| CN109640830A (en) * | 2016-07-14 | 2019-04-16 | 医视特有限公司 | Focus ultrasonic based on precedent |
| CN110675451A (en) * | 2019-09-17 | 2020-01-10 | 北京超放信息技术有限公司 | Digital self-adaptive correction method and system based on phase space optics |
| CN111915526A (en) * | 2020-08-05 | 2020-11-10 | 湖北工业大学 | Photographing method based on brightness attention mechanism low-illumination image enhancement algorithm |
| CN113256741A (en) * | 2021-07-14 | 2021-08-13 | 清华大学 | Lens calibration method and device for scanning light field imaging system |
-
2022
- 2022-04-15 CN CN202210394888.6A patent/CN114494258B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030103189A1 (en) * | 2001-09-11 | 2003-06-05 | The Regents Of The University Of California | Characterizing aberrations in an imaging lens and applications to visual testing and integrated circuit mask analysis |
| US20150178923A1 (en) * | 2013-12-24 | 2015-06-25 | Lytro, Inc. | Light-field aberration correction |
| CN109640830A (en) * | 2016-07-14 | 2019-04-16 | 医视特有限公司 | Focus ultrasonic based on precedent |
| CN110675451A (en) * | 2019-09-17 | 2020-01-10 | 北京超放信息技术有限公司 | Digital self-adaptive correction method and system based on phase space optics |
| CN111915526A (en) * | 2020-08-05 | 2020-11-10 | 湖北工业大学 | Photographing method based on brightness attention mechanism low-illumination image enhancement algorithm |
| CN113256741A (en) * | 2021-07-14 | 2021-08-13 | 清华大学 | Lens calibration method and device for scanning light field imaging system |
Non-Patent Citations (2)
| Title |
|---|
| 吴志云等: "用泽尼克多项式消除三维轮廓术中的条纹载频", 《光学学报》 * |
| 潘德馨: "光学像差的图像校正算法", 《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023231138A1 (en) * | 2022-05-30 | 2023-12-07 | 元潼(北京)技术有限公司 | Multi-angle-of-view image super-resolution reconstruction method and apparatus based on meta-imaging |
| CN117541519A (en) * | 2024-01-09 | 2024-02-09 | 清华大学 | Optical field aberration correction method, optical field aberration correction device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114494258B (en) | 2022-08-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114494258B (en) | Lens aberration prediction and image reconstruction method and device | |
| Jeon et al. | Accurate depth map estimation from a lenslet light field camera | |
| US10326981B2 (en) | Generating 3D images using multi-resolution camera set | |
| CN108833785B (en) | Fusion method and device of multi-view images, computer equipment and storage medium | |
| US20150206504A1 (en) | Unified optimization method for end-to-end camera image processing for translating a sensor captured image to a display image | |
| CN112396645B (en) | Monocular image depth estimation method and system based on convolution residual learning | |
| KR20120099713A (en) | Algorithms for estimating precise and relative object distances in a scene | |
| Atapour-Abarghouei et al. | Depthcomp: Real-time depth image completion based on prior semantic scene segmentation | |
| CN113256741B (en) | Lens calibration method and device for scanning light field imaging system | |
| Jin et al. | Light field super-resolution via attention-guided fusion of hybrid lenses | |
| Paramanand et al. | Shape from sharp and motion-blurred image pair | |
| WO2014120281A1 (en) | Increasing frame rate of an image stream | |
| CN115035235A (en) | Three-dimensional reconstruction method and device | |
| CN114640885B (en) | Video frame inserting method, training device and electronic equipment | |
| KR102319643B1 (en) | Method for processing microscopy image using artificial neural network with point spread function layer and apparatus therefore | |
| CN108366262B (en) | Motion encoder | |
| CN113298187A (en) | Image processing method and device, and computer readable storage medium | |
| TWI417810B (en) | Image enhancement method, image enhancement apparaus and image processing circuit | |
| CN114782507A (en) | Asymmetric binocular stereo matching method and system based on unsupervised learning | |
| Joseph Raj et al. | Video-rate calculation of depth from defocus on a FPGA | |
| JP2018133064A (en) | Image processing apparatus, imaging apparatus, image processing method, and image processing program | |
| Shen et al. | Non-uniform motion deblurring with kernel grid regularization | |
| Doner et al. | FPGA-based infrared image deblurring using angular position of IR detector | |
| KR20210046465A (en) | Method and apparatus for correction of aberration | |
| Chidadala et al. | Design of convolutional neural network with cuckoo search algorithm for super-resolution uhd systems on fpga |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |