CN114926339B - Light field multi-view image super-resolution reconstruction method and system based on deep learning - Google Patents
Light field multi-view image super-resolution reconstruction method and system based on deep learning Download PDFInfo
- Publication number
- CN114926339B CN114926339B CN202210599012.5A CN202210599012A CN114926339B CN 114926339 B CN114926339 B CN 114926339B CN 202210599012 A CN202210599012 A CN 202210599012A CN 114926339 B CN114926339 B CN 114926339B
- Authority
- CN
- China
- Prior art keywords
- lens surface
- resolution reconstruction
- light field
- resolution
- deep learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013135 deep learning Methods 0.000 title claims abstract description 35
- 238000000034 method Methods 0.000 title claims abstract description 25
- 238000003384 imaging method Methods 0.000 claims abstract description 98
- 230000000007 visual effect Effects 0.000 claims abstract description 17
- 238000013136 deep learning model Methods 0.000 claims description 27
- 230000004927 fusion Effects 0.000 claims description 24
- 230000003287 optical effect Effects 0.000 claims description 13
- 238000003062 neural network model Methods 0.000 claims description 10
- 230000000694 effects Effects 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 2
- 239000010902 straw Substances 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- FFBHFFJDDLITSX-UHFFFAOYSA-N benzyl N-[2-hydroxy-4-(3-oxomorpholin-4-yl)phenyl]carbamate Chemical compound OC1=C(NC(=O)OCC2=CC=CC=C2)C=CC(=C1)N1CCOCC1=O FFBHFFJDDLITSX-UHFFFAOYSA-N 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 230000005693 optoelectronics Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Studio Devices (AREA)
Abstract
The invention provides a light field multi-view image super-resolution reconstruction method and a light field multi-view image super-resolution reconstruction system based on deep learning, and belongs to the technical field of light field reconstruction. The method comprises the steps of obtaining a first imaging focus, first imaging data, a second imaging focus, second imaging data and current focal length data of a first lens surface and a second lens surface of a target object shot by a light field camera under different view angles, and obtaining reconstruction results corresponding to the different view angles, wherein the reconstruction results comprise a first resolution reconstruction result and a second resolution reconstruction result; when the first resolution ratio reconstruction result is matched with the second resolution ratio reconstruction result, starting super-resolution ratio reconstruction; otherwise, parameter adjustment is initiated. According to the technical scheme, super-resolution reconstruction of the light field image data can be realized based on difference matching of different visual angles, and when the matching reconstruction conditions are not met, the imaging parameters are adjusted in a self-adaptive mode, so that the super-resolution reconstruction effect is ensured.
Description
Technical Field
The invention belongs to the technical field of light field reconstruction, and particularly relates to a light field multi-view image super-resolution reconstruction method and system based on deep learning.
Background
The capability of the light field camera to record the light direction enables the light field camera to obtain refocused images at any positions in a geometric tracking mode, compared with the traditional shooting mode, the traditional shooting mode needs a complicated focusing process, and the light field digital refocusing is revolutionary change.
Although the light field camera can record the position and angle information of the space light rays at the same time, and has remarkable advantages in digital refocusing, full-focus image acquisition and depth estimation, the recording of multi-dimensional information causes the spatial resolution of the light field image to be low, and the research on the light field super-resolution reconstruction technology has great significance in order to fully utilize the advantages of the light field camera and overcome the problem of limited image resolution.
In the prior art, light field super-resolution reconstruction is generally implemented by acquiring image data based on a single visual angle and fusing different resolutions. However, the problem of imaging difference of the light field camera for the same target object (region) under different viewing angles is not considered in this way, and the problem of parameter adjustment of the lens array inside the light field camera itself is not considered, so that registration error or image display blur after super-resolution fusion may occur.
Disclosure of Invention
In order to solve the technical problems, the invention provides a light field multi-view image super-resolution reconstruction method and a light field multi-view image super-resolution reconstruction system based on deep learning.
The technical scheme includes that a first imaging focus, first imaging data, a second imaging focus and second imaging data of a target object on a first lens surface and current focal length data of the first lens surface and a second lens surface, which are shot by a light field camera under different view angles, are obtained, and reconstruction results corresponding to the different view angles are obtained, wherein the reconstruction results comprise a first resolution reconstruction result and a second resolution reconstruction result; when the first resolution ratio reconstruction result is matched with the second resolution ratio reconstruction result, starting super-resolution ratio reconstruction; otherwise, parameter adjustment is initiated.
According to the technical scheme, super-resolution reconstruction of the light field image data can be realized based on difference matching of different viewing angles, and imaging parameters are adjusted in a self-adaptive mode when the super-resolution reconstruction conditions are not met, so that the super-resolution reconstruction effect is ensured.
Specifically, in a first aspect of the present invention, a light field multi-view image super-resolution reconstruction method based on deep learning is provided. The method is implemented based on a light field camera comprising a first lens face and a second lens face;
at least one of the first lens surface and the second lens surface is a flexible optical lens, and the focal length of the flexible optical lens can be adjusted within a design range.
In the performing step, the method comprises:
s1: acquiring a first imaging focus F of a target object shot by the light field camera on the first lens surface under a first visual angle 1 1 And a second imaging focus on the second lens surface
S2: the current focal length J of the first lens surface 1 A current focal length J of the second lens surface 2 And said first imaging focus F 1 1 And a second imaging focus on the second lens face
Relative distance therebetween
Inputting the data as a deep learning parameter to a deep learning model, wherein the deep learning model outputs a first resolution reconstruction result of the target object;
s3: acquiring a first imaging focus F of the target object shot by the light field camera on the first lens surface under a second visual angle 1 2 And a second imaging focus on the second lens surface
S4: the current focal length J of the first lens surface 1 SaidCurrent focal length J of the second lens face 2 And the first imaging focus F 1 2 And a second imaging focus on the second lens face
Relative distance therebetween
Inputting the target object as a deep learning parameter to the deep learning model, wherein the deep learning model outputs a second resolution reconstruction result of the target object;
s5: judging whether the first resolution reconstruction result is matched with the second resolution reconstruction result, and if so, entering the step S6;
otherwise, adjusting imaging parameters and returning to the step S1;
the imaging parameters comprise one of the current focal length of the first lens surface, the current focal length of the second lens surface, the relative distance between the first lens surface and the second lens surface, or any combination thereof;
s6: and carrying out image fusion on the first resolution reconstruction result and the second resolution reconstruction result to obtain a super-resolution reconstruction result of the target object.
As a further improvement, the deep learning model is a back propagation neural network model, and the neural network model comprises an input layer, an intermediate layer and an output layer;
the input layer comprises a first node, a second node and a third node;
the first node inputs a current focal length J of the first lens surface 1 (ii) a The second nodal point inputs a current focal length J of the second lens face 2 ；
The third node inputs first imaging data of a target object shot by the light field camera on the first lens surface and second imaging data of the target object shot by the light field camera on the second lens surface;
the middle layer comprises a first fusion node and a second fusion node;
the first fusion node fuses a first imaging focus F on the first lens surface 1 1 And a second imaging focus on the second lens face
And obtaining the relative distance
The second fusion node fuses the first imaging data and the second imaging data and obtains target data to be reconstructed;
the output layer is connected with the middle layer and the input layer, the output of the middle layer is used as the input of the output layer, and the output result of the output layer is fed back and output to the input layer.
The first perspective is different from the second perspective;
in a specific selection, the first viewing angle or the second viewing angle is selected from one of the following viewing angles:
parallel to a first coordinate axis of the cartesian coordinate system, parallel to a second coordinate axis of the cartesian coordinate system, parallel to a third coordinate axis of the cartesian coordinate system.
The step S5 of determining whether the first resolution reconstruction result and the second resolution reconstruction result match includes:
judging whether the difference value of the image resolution of the same plane of the first resolution reconstruction result and the second resolution reconstruction result in a Cartesian coordinate system is within a preset range or not;
and if so, matching the first resolution reconstruction result with the second resolution reconstruction result.
More specifically, three difference values of image resolutions of three same planes of the first resolution reconstruction result and the second resolution reconstruction result in a cartesian coordinate system are obtained;
and if at least two difference values of the three difference values are within a preset range, matching the first resolution reconstruction result with the second resolution reconstruction result.
In a second aspect of the present invention, to implement the method in the first aspect, a light field multi-view image super-resolution reconstruction system based on deep learning is provided, the system is connected to a light field camera, the light field camera includes a first lens surface and a second lens surface, a focal length of at least one lens surface is adjustable, and a relative distance between the first lens surface and the second lens surface is adjustable;
in a specific structure, the system further comprises a visual angle setting unit, a deep learning unit, a matching judgment unit, a parameter adjusting unit and a super-resolution reconstruction unit.
More specifically, the functions of the above respective implementation units are implemented as follows:
the visual angle setting unit is used for setting a shooting visual angle of the light field camera, and the shooting visual angle comprises a first visual angle and a second visual angle;
the deep learning unit is used for acquiring a first imaging focus, first imaging data, a second imaging focus, second imaging data and current focal length data of the first lens surface and the second lens surface of a target object shot by the light field camera under different view angles, and acquiring reconstruction results corresponding to the different view angles, wherein the reconstruction results comprise a first resolution reconstruction result and a second resolution reconstruction result;
the matching judgment unit is used for judging whether the first resolution reconstruction result is matched with the second resolution reconstruction result;
the parameter adjusting unit is used for adjusting imaging parameters of the light field camera;
the super-resolution reconstruction unit carries out image fusion on the first resolution reconstruction result and the second resolution reconstruction result to obtain a super-resolution reconstruction result of the target object;
the imaging parameters include one of a current focal length of the first lens surface, a current focal length of the second lens surface, a relative distance of the first lens surface and the second lens surface, or any combination thereof.
More specifically, the matching judgment unit is configured to judge whether the first resolution reconstruction result and the second resolution reconstruction result match, and specifically includes:
acquiring three difference values of image resolutions of three same planes of the first resolution reconstruction result and the second resolution reconstruction result in a Cartesian coordinate system;
and if at least two difference values of the three difference values are within a preset range, matching the first resolution reconstruction result with the second resolution reconstruction result.
The super-resolution reconstruction unit and the parameter adjusting unit are communicated with the matching judging unit;
when the judgment result of the matching judgment unit is matching, starting the super-resolution reconstruction unit; otherwise, starting the parameter adjusting unit.
More specifically, the deep learning unit comprises a deep learning model; the deep learning model is a back propagation neural network model.
In a specific structure, the neural network model comprises an input layer, an intermediate layer and an output layer;
the input layer comprises a first node, a second node and a third node;
the first nodal point inputs a current focal length J of the first lens face 1 (ii) a The second node inputs a current focal length J of the second lens surface 2 ；
The third node inputs first imaging data of a target object shot by the light field camera on the first lens surface and second imaging data of the target object shot by the light field camera on the second lens surface;
the middle layer comprises a first fusion node and a second fusion node;
the first fusion node fuses a first imaging focus F on the first lens surface 1 1 And a second imaging focus on the second lens surface
And obtaining the relative distance
The second fusion node fuses the first imaging data and the second imaging data and obtains target data to be reconstructed;
the output layer is connected with the middle layer and the input layer, the output of the middle layer is used as the input of the output layer, and the output result of the output layer is fed back and output to the input layer.
According to the technical scheme, super-resolution reconstruction of the light field image data can be realized based on difference matching of different visual angles, and when the matching reconstruction conditions are not met, imaging parameters are adjusted in a self-adaptive mode, so that the super-resolution reconstruction effect is ensured; meanwhile, the neural network deep learning model with back propagation is adopted, so that the output result can reversely act on the input layer, a feedback signal is provided for the subsequent secondary visual angle adjustment, the accuracy and the efficiency of the whole reconstruction process are further ensured, and closed-loop self-learning feedback is formed.
Further advantages of the invention will be apparent in the detailed description section in conjunction with the drawings attached hereto.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a schematic flow chart illustrating steps of a light field multi-view image super-resolution reconstruction method based on deep learning according to an embodiment of the present invention;
FIG. 2 is a block diagram of a deep learning model used in various embodiments of the present invention;
FIG. 3 is a schematic diagram of the relative positions of different lenses of a light field camera as used by various embodiments of the present invention;
FIG. 4 is a schematic diagram illustrating a determination of whether reconstruction results at different viewing angles are matched according to various embodiments of the present invention;
fig. 5 is an architecture diagram of a light field multi-view image super-resolution reconstruction system based on deep learning according to an embodiment of the present invention.
Detailed Description
The invention is further described with reference to the following drawings and detailed description.
Fig. 1 shows a flow of steps of a light field multi-view image super-resolution reconstruction method based on deep learning according to an embodiment of the present invention, which includes a loop determination process of steps S1-S6.
The method of fig. 1 is implemented based on a light field camera comprising a first lens face and a second lens face.
As a specific example, various embodiments of the present invention are directed to a light field camera that includes a microlens array, but of course, other optical sensors. In general, a light field camera consists of three parts, a main lens, a microlens array, and an image sensor.
In the present invention, the adjustment of the main lens is not considered, and therefore, the subsequent embodiments or diagrams only show the microlens and the corresponding focusing mirror (also called refocusing mirror).
Therefore, the first lens surface and the second lens surface can be understood as a micro lens and a corresponding focusing lens (also called a refocusing lens), i.e. the first lens surface is a micro lens and the second lens surface is a focusing lens; or the second lens surface is a micro lens, and the first lens surface is a focusing lens.
On the basis, the steps S1-S6 are specifically realized as follows:
s1: acquiring a first imaging focus F of a target object shot by the light field camera on the first lens surface under a first visual angle 1 1 And a second imaging focus on the second lens surface
S2: the current focal length J of the first lens surface 1 A current focal length J of the second lens surface 2 And the first imaging focus F 1 1 And a second imaging focus on the second lens face
Relative distance therebetween
Inputting the data as a deep learning parameter to a deep learning model, wherein the deep learning model outputs a first resolution reconstruction result of the target object;
s3: acquiring a first imaging focus F of the target object shot by the light field camera on the first lens surface under a second visual angle 1 2 And a second imaging focus on the second lens surface
S4: the current focal length J of the first lens surface 1 A current focal length J of the second lens surface 2 And said first imaging focus F 1 2 And a second imaging focus on the second lens face
Relative distance therebetween
Inputting the target object as a deep learning parameter to the deep learning model, wherein the deep learning model outputs a second resolution reconstruction result of the target object;
s5: judging whether the first resolution reconstruction result is matched with the second resolution reconstruction result, and if so, entering the step S6;
otherwise, adjusting imaging parameters and returning to the step S1;
s6: and carrying out image fusion on the first resolution reconstruction result and the second resolution reconstruction result to obtain a super-resolution reconstruction result of the target object.
In step S5, the imaging parameter includes one of a current focal length of the first lens surface, a current focal length of the second lens surface, a relative distance between the first lens surface and the second lens surface, or any combination thereof.
FIG. 2 is a block diagram of a deep learning model used in various embodiments of the present invention.
In fig. 2, the deep learning model is a back-propagation neural network model, which includes an input layer, an intermediate layer and an output layer;
the input layer comprises a first node, a second node and a third node;
the first node inputs a current focal length J of the first lens surface 1 (ii) a The second node inputs a current focal length J of the second lens surface 2 ；
The third node inputs first imaging data of a target object shot by the light field camera on the first lens surface and second imaging data of the target object shot by the light field camera on the second lens surface;
the middle layer comprises a first fusion node and a second fusion node;
the first fusion node fuses a first imaging focus F on the first lens surface 1 1 And a second imaging focus on the second lens face
And obtaining the relative distance
The second fusion node fuses the first imaging data and the second imaging data and obtains target data to be reconstructed;
the output layer is connected with the middle layer and the input layer, the output of the middle layer is used as the input of the output layer, and the output result of the output layer is fed back and output to the input layer.
The specific deep learning realizes the correlation principle of light field reconstruction, and reference may be made to various light field deep learning reconstruction models introduced in the prior art, for example:
[1] yi Yupeng, light field image super-resolution reconstruction algorithm research based on deep learning [ D ]. Taiyuan science and technology university, 2021. DOI.
[2] Nilluxia. Three-dimensional light field accurate reproduction key technology research [ D ]. Zhejiang university, 2021.DOI.
It can be seen that the embodiment of the invention can realize super-resolution reconstruction of light field image data based on difference matching of different viewing angles, and adaptively adjust imaging parameters when the matching reconstruction conditions are not met, thereby ensuring the super-resolution reconstruction effect; meanwhile, the neural network deep learning model with back propagation is adopted, so that the output result can reversely act on the input layer, a feedback signal is provided for the subsequent secondary visual angle adjustment, the accuracy and the efficiency of the whole reconstruction process are further ensured, and closed-loop self-learning feedback is formed.
More specifically, FIG. 3 illustrates a schematic diagram of relative positions of different lenses of a light field camera as used by various embodiments of the present invention.
In fig. 3, a first imaging focus F on the first lens face is shown 1 1 And a second imaging focus on the second lens face
Current focal length J of the first lens face 1 A current focal length J of the second lens surface 2 The first imaging focus F 1 1 And a second imaging focus on the second lens face
Relative distance therebetween
In various embodiments of the present invention, the first imaging focus F, regardless of the lens array employed 1 1 And a second imaging focus on the second lens face
Relative distance therebetween
Can be fine-tuned within the design.
On the basis, as a further preference, a flexible optical lens can be adopted as the partial lens, and the focal length of the flexible optical lens can be adjusted within a design range.
For example, at least one of the first lens surface and the second lens surface is a flexible optical lens, the focal length of which is adjustable within a design range.
Or the first lens surface and the second lens surface are both flexible optical lenses, and the focal length of the flexible optical lenses can be adjusted within a design range.
How to manufacture or implement a flexible optical lens with a focal length that can be fine-tuned within a design range belongs to the prior art, and can be seen in the following:
[3] preparation and performance studies of Liupenghui, li Shi Yao, wang Wen, wen Xuyang, wu dynasty, zhou Xiong, zyong love Flexible liquid Crystal microlens arrays [ J ]. Photonic science, 2021,50 (03): 86-93.
[4] Wang Jong, li Lei optical imaging systems [ J ] based on adaptive lenses optoelectronics, 2020,40 (03): 155-163. DOI.
The invention is therefore not specifically developed in this regard, and the above cited documents are incorporated as part of the embodiments of the present invention.
As an empirical alternative of the invention, the first imaging focus F is adjusted during the adjustment process 1 1 And a second imaging focus on the second lens face
Relative distance therebetween
Current focal length J with the first lens face 1 A current focal length J of the second lens surface 2 The following constraints are satisfied:
wherein K is an adjustment coefficient;
when the first lens surface is a micro lens and the second lens surface is a focusing lens, K is larger than 1;
when the second lens surface is a micro lens and the first lens surface is a focusing lens, the 0-straw K-straw (1) is formed.
Reference is next made to fig. 4. Fig. 4 is a schematic diagram illustrating determination of whether reconstruction results at different viewing angles are matched in various embodiments of the present invention.
Specifically, the step S5 of determining whether the first resolution reconstruction result and the second resolution reconstruction result match includes:
judging whether the difference value of the image resolution of the same plane of the first resolution reconstruction result and the second resolution reconstruction result in a Cartesian coordinate system is within a preset range or not;
if yes, the first resolution reconstruction result is matched with the second resolution reconstruction result.
In various embodiments of the present invention, the first viewing angle or the second viewing angle is selected from one of the following viewing angles:
parallel to a first axis of the cartesian coordinate system, parallel to a second axis of the cartesian coordinate system and parallel to a third axis of the cartesian coordinate system.
It is understood that the first coordinate axis, the second coordinate axis, and the third coordinate axis are each one of the X-Y-Z axes.
See, as an example, fig. 4.
Assuming that the first viewing angle is a viewing angle parallel to a first coordinate axis of the cartesian coordinate system, e.g. a viewing angle parallel to the X-axis;
assuming that the second viewing angle is a viewing angle parallel to a second coordinate axis of the cartesian coordinate system, e.g. a viewing angle parallel to the Y-axis;
the first resolution reconstruction results in three planes of image resolution in cartesian coordinates:
an image resolution 1 in the XY plane, an image resolution 2 in the XZ plane, and an image resolution 3 in the YZ plane;
the second resolution reconstruction results in image resolutions in three planes also in a cartesian coordinate system:
an image resolution 11 in the XY plane, an image resolution 12 in the XZ plane, and an image resolution 13 in the XY plane;
at this time, at least two of the image resolution 1 of the XY plane, the image resolution 2 of the XZ plane, and the image resolution 3 of the YZ plane should be the same as or slightly different from at least two of the image resolution 11 of the XY plane, the image resolution 12 of the XZ plane, and the image resolution 13 of the XY plane (the difference value is within a preset range), so as to ensure that no error or blur is generated in the subsequent fusion reconstruction process.
Thus, in particular implementations, the first viewing angle is different from the second viewing angle.
More specifically, three difference values of image resolutions of three planes of the first resolution reconstruction result and the second resolution reconstruction result in a cartesian coordinate system are obtained;
and if at least two difference values of the three difference values are within a preset range, matching the first resolution reconstruction result with the second resolution reconstruction result.
The preset range can be set based on the size of the reconstruction resolution, and the higher the requirement of the reconstruction resolution is, the smaller the preset range is.
To implement the above method, referring to fig. 5, fig. 5 is an architecture diagram of a light field multi-view image super-resolution reconstruction system based on deep learning according to an embodiment of the present invention.
In fig. 5, a light field multi-view image super-resolution reconstruction system based on deep learning is shown, the system is connected with a light field camera, the light field camera comprises a first lens surface and a second lens surface, the focal length of at least one lens surface can be adjusted, and the relative distance between the first lens surface and the second lens surface can be adjusted;
fig. 5 shows that the system further comprises:
a viewing angle setting unit for setting a photographing viewing angle of the light field camera, the photographing viewing angle including a first viewing angle and a second viewing angle;
the deep learning unit is used for acquiring a first imaging focus, first imaging data, a second imaging focus and second imaging data of a target object on the first lens surface, the second imaging focus and the second imaging data of the target object, and current focal length data of the first lens surface and the second lens surface, wherein the target object is shot by the light field camera at different view angles, and acquiring reconstruction results corresponding to the different view angles, wherein the reconstruction results comprise a first resolution reconstruction result and a second resolution reconstruction result;
a matching judgment unit, configured to judge whether the first resolution reconstruction result and the second resolution reconstruction result are matched;
a parameter adjusting unit for adjusting imaging parameters of the light field camera;
a super-resolution reconstruction unit which performs image fusion on the first resolution reconstruction result and the second resolution reconstruction result to obtain a super-resolution reconstruction result of the target object;
the imaging parameters include one of a current focal length of the first lens surface, a current focal length of the second lens surface, a relative distance of the first lens surface and the second lens surface, or any combination thereof.
Preferably, the first lens surface and the second lens surface are both flexible optical lenses.
Preferably, the matching judgment unit is configured to judge whether the first resolution reconstruction result and the second resolution reconstruction result match, and specifically includes:
acquiring three difference values of image resolutions of three same planes of the first resolution reconstruction result and the second resolution reconstruction result under a Cartesian coordinate system;
and if at least two difference values of the three difference values are within a preset range, matching the first resolution reconstruction result with the second resolution reconstruction result.
The super-resolution reconstruction unit and the parameter adjusting unit are communicated with the matching judgment unit;
when the judgment result of the matching judgment unit is matching, starting the super-resolution reconstruction unit; otherwise, starting the parameter adjusting unit.
The deep learning unit comprises a deep learning model;
the deep learning model is a back propagation neural network model.
Specific deep learning model structure can be seen in fig. 2, and will not be described again.
According to the technical scheme, super-resolution reconstruction of light field image data can be realized based on difference matching of different viewing angles, and imaging parameters are adjusted in a self-adaptive manner when the super-resolution reconstruction conditions are not met, so that a super-resolution reconstruction effect is ensured; meanwhile, the back-propagation neural network deep learning model is adopted, so that the output result can reversely act on the input layer, a feedback signal is provided for the subsequent secondary adjustment of the visual angle, the accuracy and efficiency of the whole reconstruction process are further ensured, and closed-loop self-learning feedback is formed.
This section describes a number of different embodiments. It should be noted that different embodiments of the present invention can respectively solve one or more technical problems mentioned in the background art and achieve corresponding technical effects, and different combinations of the embodiments can solve all the mentioned technical problems and achieve all the technical effects;
however, it is not required that every single embodiment of the present invention solve all the technical problems or achieve all the improvements. The solution to a certain problem or the improved corresponding embodiment of a single technical effect may both constitute an independent technical solution of the present invention.
Although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that: modifications and equivalents may be made to the embodiments of the invention without departing from the spirit and scope of the invention, which is to be covered by the claims.
The present invention is not limited to the specific module structure described in the prior art. The prior art mentioned in the background section can be used as part of the invention to understand the meaning of some technical features or parameters. The scope of the present invention is defined by the claims.
Claims (7)
1. A light field multi-view image super-resolution reconstruction method based on deep learning is realized based on a light field camera, wherein the light field camera comprises a first lens surface and a second lens surface;
characterized in that the method comprises the following steps:
s1: acquiring a first imaging focus F of a target object shot by the light field camera on the first lens surface under a first view angle 1 1 And a second imaging focus F on the second lens surface 2 1 ；
S2: the current focal length J of the first lens surface 1 A current focal length J of the second lens surface 2 And the first imaging focus F 1 1 And a second imaging focus F on the second lens face 2 1 Relative distance between | F 2 1 -F 1 1 I is input into a deep learning model as a deep learning parameter, and the deep learning model outputs a first resolution reconstruction result of the target object;
s3: acquiring a first imaging focus F of the target object shot by the light field camera on the first lens surface under a second visual angle 1 2 And a second imaging focus F on the second lens surface 2 2 ；
S4: the current focal length J of the first lens surface 1 A current focal length J of the second lens surface 2 And said first imaging focus F 1 2 And a second imaging focal point F on the second lens surface 2 2 Relative distance | F between 2 2 -F 1 2 I, inputting the I as a deep learning parameter to the deep learning model, and outputting a second resolution reconstruction result of the target object by the deep learning model;
s5: judging whether the first resolution ratio reconstruction result is matched with the second resolution ratio reconstruction result, and if so, entering the step S6;
otherwise, adjusting imaging parameters and returning to the step S1;
the imaging parameters comprise one of the current focal length of the first lens surface, the current focal length of the second lens surface, the relative distance between the first lens surface and the second lens surface, or any combination thereof;
s6: performing image fusion on the first resolution reconstruction result and the second resolution reconstruction result to obtain a super-resolution reconstruction result of the target object;
the deep learning model is a neural network model of back propagation, and the neural network model comprises an input layer, a middle layer and an output layer; the output layer is connected with the middle layer and the input layer, the output of the middle layer is used as the input of the output layer, and the output result of the output layer is fed back and output to the input layer;
the step S5 of determining whether the first resolution reconstruction result and the second resolution reconstruction result match includes: acquiring three difference values of image resolutions of three same planes of the first resolution reconstruction result and the second resolution reconstruction result in a Cartesian coordinate system; and if at least two difference values in the three difference values are in a preset range, matching the first resolution ratio reconstruction result with the second resolution ratio reconstruction result.
2. The light field multi-view image super-resolution reconstruction method based on deep learning of claim 1, wherein: the input layer comprises a first node, a second node and a third node;
the first node inputs a current focal length J of the first lens surface 1 (ii) a The second node inputs a current focal length J of the second lens surface 2 ；
The third node inputs first imaging data of a target object shot by the light field camera on the first lens surface and second imaging data of the target object shot by the light field camera on the second lens surface;
the middle layer comprises a first fusion node and a second fusion node;
the first fusion node fuses a first imaging focus F on the first lens face 1 1 And a second imaging focus F on the second lens face 2 1 And obtaining the relative distance | F 2 1 -F 1 1 |；
And the second fusion node fuses the first imaging data and the second imaging data and obtains target data to be reconstructed.
3. The light field multi-view image super-resolution reconstruction method based on deep learning of claim 1, wherein: at least one of the first lens surface and the second lens surface is a flexible optical lens, and the focal length of the flexible optical lens can be adjusted within a design range.
4. The light field multi-view image super-resolution reconstruction method based on deep learning of claim 1, wherein: the first perspective is different from the second perspective;
the first viewing angle or the second viewing angle is selected from one of the following viewing angles:
parallel to a first coordinate axis of the cartesian coordinate system, parallel to a second coordinate axis of the cartesian coordinate system, parallel to a third coordinate axis of the cartesian coordinate system.
5. A light field multi-view image super-resolution reconstruction system based on deep learning is connected with a light field camera, wherein the light field camera comprises a first lens surface and a second lens surface, the focal length of at least one lens surface is adjustable, and the relative distance between the first lens surface and the second lens surface is adjustable;
characterized in that the system further comprises:
a viewing angle setting unit for setting a photographing viewing angle of the light field camera, the photographing viewing angle including a first viewing angle and a second viewing angle;
the deep learning unit is used for acquiring a first imaging focus, first imaging data, a second imaging focus and second imaging data of a target object on the first lens surface, the second imaging focus and the second imaging data of the target object, and current focal length data of the first lens surface and the second lens surface, wherein the target object is shot by the light field camera at different view angles, and acquiring reconstruction results corresponding to the different view angles, wherein the reconstruction results comprise a first resolution reconstruction result and a second resolution reconstruction result;
a matching judgment unit, configured to judge whether the first resolution reconstruction result and the second resolution reconstruction result are matched;
a parameter adjusting unit for adjusting imaging parameters of the light field camera;
a super-resolution reconstruction unit which performs image fusion on the first resolution reconstruction result and the second resolution reconstruction result to obtain a super-resolution reconstruction result of the target object;
the imaging parameters comprise one of the current focal length of the first lens surface, the current focal length of the second lens surface, the relative distance between the first lens surface and the second lens surface, or any combination thereof;
the deep learning unit comprises a deep learning model, the deep learning model is a neural network model of back propagation, and the neural network model comprises an input layer, an intermediate layer and an output layer; the output layer is connected with the middle layer and the input layer, the output of the middle layer is used as the input of the output layer, and the output result of the output layer is fed back and output to the input layer; the matching judgment unit is configured to judge whether the first resolution reconstruction result and the second resolution reconstruction result are matched, and specifically includes: acquiring three difference values of image resolutions of three same planes of the first resolution reconstruction result and the second resolution reconstruction result in a Cartesian coordinate system; and if at least two difference values in the three difference values are in a preset range, matching the first resolution reconstruction result with the second resolution reconstruction result.
6. The deep learning-based light field multi-view image super-resolution reconstruction system according to claim 5, wherein: the first lens surface and the second lens surface are both flexible optical lenses.
7. The light field multi-view image super-resolution reconstruction system based on deep learning of claim 5, wherein: the super-resolution reconstruction unit and the parameter adjusting unit are communicated with the matching judging unit;
when the judgment result of the matching judgment unit is matching, starting the super-resolution reconstruction unit; otherwise, starting the parameter adjusting unit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210599012.5A CN114926339B (en) | 2022-05-30 | 2022-05-30 | Light field multi-view image super-resolution reconstruction method and system based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210599012.5A CN114926339B (en) | 2022-05-30 | 2022-05-30 | Light field multi-view image super-resolution reconstruction method and system based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114926339A CN114926339A (en) | 2022-08-19 |
| CN114926339B true CN114926339B (en) | 2023-02-03 |
Family
ID=82812593
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210599012.5A Active CN114926339B (en) | 2022-05-30 | 2022-05-30 | Light field multi-view image super-resolution reconstruction method and system based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114926339B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112070675A (en) * | 2020-09-07 | 2020-12-11 | 武汉工程大学 | Regularization light field super-resolution method based on graph and light field microscopic device |
| CN112102173A (en) * | 2020-10-26 | 2020-12-18 | 安庆师范大学 | Optical field image angle super-resolution reconstruction method |
| CN112365400A (en) * | 2020-10-27 | 2021-02-12 | 深圳大学 | Rapid super-resolution reconstruction method for light field angle |
| CN112750076A (en) * | 2020-04-13 | 2021-05-04 | 奕目(上海)科技有限公司 | Light field multi-view image super-resolution reconstruction method based on deep learning |
| CN113256772A (en) * | 2021-05-10 | 2021-08-13 | 华中科技大学 | Double-angle light field high-resolution reconstruction system and method based on visual angle conversion |
| CN113850902A (en) * | 2021-09-13 | 2021-12-28 | 清华大学深圳国际研究生院 | Light field three-dimensional reconstruction method based on light field microscope system |
| WO2022016350A1 (en) * | 2020-07-21 | 2022-01-27 | Oppo广东移动通信有限公司 | Light field image processing method, light field image encoder and decoder, and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108280814B (en) * | 2018-02-08 | 2021-08-31 | 重庆邮电大学 | Light field image angle super-resolution reconstruction method based on perception loss |
| CN113870433A (en) * | 2021-09-09 | 2021-12-31 | 华中科技大学 | Light field super-resolution three-dimensional reconstruction method and system |
-
2022
- 2022-05-30 CN CN202210599012.5A patent/CN114926339B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112750076A (en) * | 2020-04-13 | 2021-05-04 | 奕目(上海)科技有限公司 | Light field multi-view image super-resolution reconstruction method based on deep learning |
| WO2022016350A1 (en) * | 2020-07-21 | 2022-01-27 | Oppo广东移动通信有限公司 | Light field image processing method, light field image encoder and decoder, and storage medium |
| CN112070675A (en) * | 2020-09-07 | 2020-12-11 | 武汉工程大学 | Regularization light field super-resolution method based on graph and light field microscopic device |
| CN112102173A (en) * | 2020-10-26 | 2020-12-18 | 安庆师范大学 | Optical field image angle super-resolution reconstruction method |
| CN112365400A (en) * | 2020-10-27 | 2021-02-12 | 深圳大学 | Rapid super-resolution reconstruction method for light field angle |
| CN113256772A (en) * | 2021-05-10 | 2021-08-13 | 华中科技大学 | Double-angle light field high-resolution reconstruction system and method based on visual angle conversion |
| CN113850902A (en) * | 2021-09-13 | 2021-12-28 | 清华大学深圳国际研究生院 | Light field three-dimensional reconstruction method based on light field microscope system |
Non-Patent Citations (4)
| Title |
|---|
| high-dimensional dense residual convolutional neural network for light field reconstruction;Meng nan et al.;《IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE》;20210301;第43卷(第3期);第873-886页 * |
| MPIN:a macro-pixel integration network for light field super-resolution;xinya wang et al.;《 Frontiers of Information Technology & Electronic Engineering》;20211029;第22卷(第10期);第1299-1310页 * |
| 三维光场精准再现关键技术研究;倪丽霞;《中国博士学位论文全文数据库信息科技辑》;20220115(第1期);第I138-171页 * |
| 融合全局与局部视角的光场超分辨率重建;邓武 等;《计算机应用研究》;20180408(第5期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114926339A (en) | 2022-08-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP4673202B2 (en) | Image input device | |
| US8988538B2 (en) | Image pickup apparatus and lens apparatus | |
| CN102494609B (en) | Three-dimensional photographing process based on laser probe array and device utilizing same | |
| TWI538508B (en) | Image capturing system obtaining scene depth information and focusing method thereof | |
| JP4935544B2 (en) | Imaging device | |
| JP2014010400A (en) | Imaging device and lens device | |
| CN106454318B (en) | Stereoscopic imaging method and stereoscopic imaging device | |
| CN111033344A (en) | Module with multiple cameras for integration in a mobile device | |
| JP2013048309A (en) | Image pickup apparatus | |
| US11086099B2 (en) | Light-folding camera and mobile device including the same | |
| CN109325981B (en) | Geometric parameter calibration method for micro-lens array type optical field camera based on focusing image points | |
| US8749652B2 (en) | Imaging module having plural optical units in which each of at least two optical units include a polarization filter and at least one optical unit includes no polarization filter and image processing method and apparatus thereof | |
| CN107948470B (en) | Camera module and mobile device | |
| CN113810676A (en) | Image processing apparatus, method, system, medium, and method of manufacturing learning model | |
| JP2014209169A (en) | Imaging apparatus and control method therefor, program, and storage medium | |
| US20040151380A1 (en) | Multi-layered real-time stereo matching method and system | |
| JP7288226B2 (en) | ranging camera | |
| CN111213364A (en) | Shooting equipment control method, shooting equipment control device and shooting equipment | |
| JP2009216600A (en) | Distance measuring device | |
| JP2004336239A (en) | Stereoscopic video image correcting apparatus and method therefor | |
| CN114926339B (en) | Light field multi-view image super-resolution reconstruction method and system based on deep learning | |
| Takahashi | Development of ultrawide-angle compact camera using free-form optics | |
| KR20200090391A (en) | Camera module and image generating method performed therein | |
| CN105208373B (en) | Universal integrated imaging recording device based on light field compression sampling | |
| KR20200117507A (en) | Camera module and image operating method performed therein |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |