CN115187454A - Multi-view image super-resolution reconstruction method and device based on meta-imaging - Google Patents
Multi-view image super-resolution reconstruction method and device based on meta-imaging Download PDFInfo
- Publication number
- CN115187454A CN115187454A CN202210605176.4A CN202210605176A CN115187454A CN 115187454 A CN115187454 A CN 115187454A CN 202210605176 A CN202210605176 A CN 202210605176A CN 115187454 A CN115187454 A CN 115187454A
- Authority
- CN
- China
- Prior art keywords
- image
- view
- resolution
- training
- reconstruction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000003384 imaging method Methods 0.000 title claims abstract description 30
- 238000012549 training Methods 0.000 claims abstract description 88
- 238000003062 neural network model Methods 0.000 claims abstract description 17
- 238000012216 screening Methods 0.000 claims abstract description 16
- 238000004590 computer program Methods 0.000 claims description 7
- 238000013528 artificial neural network Methods 0.000 claims description 6
- 238000005070 sampling Methods 0.000 abstract description 5
- 238000004891 communication Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000006073 displacement reaction Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000036544 posture Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
The application relates to the technical field of image reconstruction, in particular to a multi-view image super-resolution reconstruction method, a device and equipment based on meta-imaging, wherein the method comprises the following steps: collecting a multi-view image of an object to be reconstructed; screening a multi-view target image with a resolution lower than a preset resolution from the multi-view images; inputting the multi-view target image into a pre-trained image reconstruction model to obtain a multi-view reconstruction image of an object to be reconstructed, wherein the pre-trained image reconstruction model is obtained by training a plurality of images lower than a first preset resolution threshold and images correspondingly higher than a second preset resolution threshold, recovering a higher multi-view resolution image under a smaller scanning frequency by utilizing the redundant sampling characteristic among the multi-view images, and finally realizing the reconstruction of a low-resolution multi-view image by utilizing a neural network model. Therefore, the problems that the scanning times are too many, the accuracy of the recovered multi-view resolution image is low and the like in the traditional scanning light field image reconstruction are solved.
Description
Technical Field
The present application relates to the field of image reconstruction technologies, and in particular, to a multi-view image super-resolution reconstruction method and apparatus based on meta-imaging.
Background
At present, any video camera equipment is affected by physical conditions of an imaging system, shooting environment and the like, and the actually obtained images are poor in quality and low in resolution.
In the related art, the resolution of the image can be improved by reducing the pixel size, increasing the chip scale, changing the arrangement mode of the detecting elements and reconstructing the super-resolution image. The process level is improved, the size of the detecting element is reduced, the density of the detecting element of a CCD (Charge Coupled Device) camera is increased, although the purposes of reducing the size of a pixel and improving the resolution of an image can be achieved, the illumination obtained by each pixel is reduced, correspondingly, the exposure noise is more obvious, and the image quality is deteriorated.
In the related art, it is difficult to recover a high multi-view resolution image, and a high-resolution two-dimensional or three-dimensional image cannot be reconstructed, which is urgently needed to be solved.
Disclosure of Invention
The application provides a multi-view image super-resolution reconstruction method and device based on meta-imaging, electronic equipment and a storage medium, and aims to solve the problems that scanning times are too many, accuracy of recovered multi-view resolution images is low and the like when a traditional light field super-resolution works.
The embodiment of the first aspect of the application provides a multi-view image super-resolution reconstruction method based on meta-imaging, which includes the following steps: collecting a multi-view image of an object to be reconstructed; screening a multi-view target image with a resolution lower than a preset resolution from the multi-view images; and inputting the multi-view target image into a pre-trained image reconstruction model to obtain a multi-view reconstruction image of the object to be reconstructed, wherein the pre-trained image reconstruction model is obtained by training a plurality of images lower than a first preset resolution threshold value and images correspondingly higher than a second preset resolution threshold value.
Optionally, in an embodiment of the present application, before inputting the multi-view target image into a pre-trained image reconstruction model, the method further includes: acquiring a plurality of first multi-view training images, which are lower than the first preset resolution threshold, of a training reconstruction object and a plurality of second multi-view training images, which correspond to the first multi-view training images and are higher than the second preset resolution threshold, wherein the second preset resolution threshold is greater than the first preset resolution threshold; training a neural network model by taking the first multi-view training image as input and the second multi-view training image as output; and when the neural network model meets the training termination condition, generating the pre-trained image reconstruction model.
Optionally, in an embodiment of the present application, the training termination condition includes: and the resolution difference value between the output image and the input image of the neural network is greater than the preset resolution difference value or the iteration times meet the preset times.
Optionally, in an embodiment of the present application, before generating the pre-trained image reconstruction model, further including: acquiring a target resolution difference value of an input image and an output image in the image reconstruction model; and matching the difference value of the first preset resolution threshold value and the second preset resolution threshold value according to the target resolution difference value.
Optionally, in an embodiment of the present application, after obtaining the multi-view reconstructed image of the object to be reconstructed, the method further includes: and re-reconstructing the multi-view reconstructed image by using an image reconstruction algorithm to obtain a two-dimensional or three-dimensional reconstructed image of the object to be reconstructed.
The embodiment of the second aspect of the present application provides a multi-view image super-resolution reconstruction apparatus based on meta-imaging, including: the first acquisition module is used for acquiring a multi-view image of an object to be reconstructed; the screening module is used for screening multi-view target images with the resolution lower than a preset resolution from the multi-view images; the first reconstruction module is used for inputting the multi-view target image into a pre-trained image reconstruction model to obtain a multi-view reconstruction image of the object to be reconstructed, wherein the pre-trained image reconstruction model is obtained by training a plurality of images lower than a first preset resolution threshold value and images correspondingly higher than a second preset resolution threshold value.
Optionally, in an embodiment of the present application, the method further includes: a second collecting module, configured to collect, before the multi-view target image is input into a pre-trained image reconstruction model, a plurality of first multi-view training images lower than the first preset resolution threshold for training a reconstruction object and a plurality of second multi-view training images higher than the second preset resolution threshold corresponding to the first multi-view training images, where the second preset resolution threshold is greater than the first preset resolution threshold; the training module is used for training a neural network model by taking the first multi-view training image as input and the second multi-view training image as output; and the generating module is used for generating the pre-trained image reconstruction model when the neural network model meets the training termination condition.
Optionally, in an embodiment of the present application, the training termination condition includes: and the resolution difference value between the output image and the input image of the neural network is greater than the preset resolution difference value or the iteration times meet the preset times.
Optionally, in an embodiment of the present application, the method further includes: the acquisition module is used for acquiring a target resolution difference value of an input image and an output image in the image reconstruction model before the pre-trained image reconstruction model is generated; and the matching module is used for matching the difference value between the first preset resolution threshold value and the second preset resolution threshold value according to the target resolution difference value.
Optionally, in an embodiment of the present application, the method further includes: and the second reconstruction module is used for re-reconstructing the multi-view reconstructed image by using an image reconstruction algorithm after the multi-view reconstructed image of the object to be reconstructed is obtained, so as to obtain a two-dimensional or three-dimensional reconstructed image of the object to be reconstructed.
An embodiment of a third aspect of the present application provides an electronic device, including: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to perform the method for super-resolution reconstruction of multi-view images based on meta-imaging as described in the above embodiments.
A fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, where the program is executed by a processor to execute the method for super-resolution reconstruction of multi-view images based on meta-imaging as described in the foregoing embodiments.
Therefore, the embodiment of the application has the following beneficial effects:
the method comprises the steps of acquiring a multi-view image of an object to be reconstructed; screening a multi-view target image with a resolution lower than a preset resolution from the multi-view images; inputting the multi-view target image into a pre-trained image reconstruction model to obtain a multi-view reconstruction image of an object to be reconstructed, wherein the pre-trained image reconstruction model is obtained by training a plurality of images lower than a first preset resolution threshold and images correspondingly higher than a second preset resolution threshold, recovering a higher multi-view resolution image under a smaller scanning frequency by utilizing the redundant sampling characteristic among the multi-view images, and finally realizing the reconstruction of a low-resolution multi-view image by utilizing a neural network model. Therefore, the problems that the scanning times are too many, the accuracy of the recovered multi-view resolution image is low and the like in the traditional light field super-resolution work are solved.
Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a flowchart of a method for super-resolution reconstruction of a multi-view image based on meta-imaging according to an embodiment of the present application;
FIG. 2 is a schematic diagram illustrating an implementation logic of a training and prediction process for an image reconstruction model according to an embodiment of the present application;
fig. 3 is an exemplary diagram of a multi-view image super-resolution reconstruction apparatus based on meta-imaging according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Description of reference numerals: a first acquisition module-100, a screening module-200, a first reconstruction module-300, a memory-401, a processor-402, and a communication interface-403.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application and should not be construed as limiting the present application.
The following describes a multi-view image super-resolution reconstruction method, an apparatus, an electronic device, and a storage medium based on meta-imaging according to embodiments of the present application with reference to the drawings. In order to solve the problems mentioned in the background art, the present application provides a multi-view image super-resolution reconstruction method based on meta-imaging, in which method, the embodiments of the present application collect multi-view images of an object to be reconstructed; screening a multi-view target image with a resolution lower than a preset resolution from the multi-view images; inputting the multi-view target image into a pre-trained image reconstruction model to obtain a multi-view reconstruction image of an object to be reconstructed, wherein the pre-trained image reconstruction model is obtained by training a plurality of images lower than a first preset resolution threshold and images correspondingly higher than a second preset resolution threshold, recovering a higher multi-view resolution image under a smaller scanning frequency by utilizing the redundant sampling characteristic among the multi-view images, and finally realizing the reconstruction of a low-resolution multi-view image by utilizing a neural network model. Therefore, the problems that the scanning times are too many, the accuracy of the recovered multi-view resolution image is low and the like in the traditional light field super-resolution work are solved
Specifically, fig. 1 is a flowchart of a multi-view image super-resolution reconstruction method based on meta-imaging according to an embodiment of the present disclosure.
As shown in fig. 1, the method for super-resolution reconstruction of multi-view images based on meta-imaging comprises the following steps:
in step S101, multi-view images of an object to be reconstructed are acquired.
In order to reconstruct an object to be reconstructed, the embodiment of the application can acquire multi-view images of the object to be reconstructed, and acquire a plurality of multi-view images by acquiring the object to be reconstructed in multiple angles and multiple postures. There are various ways to capture the image, such as capturing by a camera or an imaging system with a lens, and the like, and the method is not particularly limited.
In step S102, a multi-view target image lower than a preset resolution is screened from the multi-view images.
It can be appreciated that the embodiments of the present application can improve the reconstruction of a low-resolution multi-view image into a high-resolution reconstructed image. The embodiment of the application can screen the multi-view images after acquiring the multi-view images of the object to be reconstructed, and screen out a plurality of multi-view images with resolution lower than preset resolution for reconstruction.
Specifically, in the embodiment of the present application, in order to improve the effect of super-resolution reconstruction of a multi-view image, a plurality of images with relatively low resolution as much as possible may be input. In particular, these multiple relatively low resolution images are characterized by: the multi-view image is generated by at least 2 or more scans.
In step S103, the multi-view target image is input into a pre-trained image reconstruction model to obtain a multi-view reconstructed image of the object to be reconstructed, where the pre-trained image reconstruction model is obtained by training a plurality of images lower than a first preset resolution threshold and an image correspondingly higher than a second preset resolution threshold.
And inputting the multi-view target image which is subjected to screening and is lower than the preset resolution in the multi-view image as an input image into a pre-trained image fusion model to obtain a multi-view reconstructed image. The multi-view image super-resolution reconstruction of the multi-view target image is realized through the neural network model.
It will be appreciated by those skilled in the art that in embodiments of the present application, the image reconstruction model described above may be implemented using a convolutional neural network based on a 3d net and attention mechanism. Specifically, a multi-view image with relatively low resolution is input as a three-dimensional image stack, characteristics of each view and different views are learned by three-dimensional convolution, and a true value is finally output as a super-resolution multi-view image. Wherein the attention mechanism module is arranged in a cascade of down-sampling on the left side of the 3d net and deconvolution subtraction on the right side. In a specific implementation process, a technician may also adopt other super-resolution image reconstruction strategies based on deep learning, which is not specifically limited herein.
Optionally, in an embodiment of the present application, before inputting the multi-view target image into the pre-trained image reconstruction model, the method further includes: collecting a plurality of first multi-view training images which are lower than a first preset resolution threshold value and used for training a reconstructed object and a plurality of second multi-view training images which correspond to the first multi-view training images and are higher than a second preset resolution threshold value, wherein the second preset resolution threshold value is larger than the first preset resolution threshold value; training a neural network model by taking the first multi-view training image as input and the second multi-view training image as output; and when the neural network model meets the training termination condition, generating a pre-trained image reconstruction model.
Specifically, a supervised training mode is adopted in the training process of the image reconstruction model, and all input images of the model training have corresponding result labels. It should be noted that the input data for model training is a plurality of multi-view images with relatively low resolution. In particular, these multiple relatively low-resolution multi-view images are characterized by: the multi-view image is generated at least by 2 or more scans.
The output data of the training, i.e. the labels, should be relatively high resolution multi-view images. The characteristics of the multi-view high-resolution images are as follows: the high-resolution multi-view image should have a one-to-one correspondence with the plurality of input low-resolution multi-view images. During training, the resolution factor of the improvement is determined by training data and can be 2 times, 3 times, 4 times, 5 times and the like.
Optionally, in an embodiment of the present application, the training termination condition includes: the resolution difference value between the output image and the input image of the neural network is larger than the preset resolution difference value or the iteration times meet the preset times.
And inputting the low-resolution multi-view image training data into the model to obtain a high-resolution multi-view reconstructed image. In the training process, a person skilled in the art can decide whether to terminate the training of the model according to actual conditions, such as the difference or multiple of the resolution between the output image and the input image. For example, when the output image resolution is 5 times the input image resolution, training of the model is terminated. In addition, the skilled person can also give appropriate weight to the difference or multiple of the average resolution between the output image and the input image as a hyperparameter to combine with some loss functions, and continuously update each parameter through back propagation, so that after the loss function value reaches a certain range, the model training is finished.
Optionally, in an embodiment of the present application, before generating the pre-trained image reconstruction model, the method further includes: acquiring a target resolution difference value of an input image and an output image in an image reconstruction model; and matching the difference value of the first preset resolution threshold value and the second preset resolution threshold value according to the target resolution difference value.
It can be understood that, during training, a plurality of first multi-view training images lower than the first preset resolution threshold are input, and a plurality of second multi-view training images higher than the second preset resolution threshold corresponding to the first multi-view training images are output. The embodiment of the application can adjust the size of the first preset resolution threshold value and the second preset resolution threshold value during model training according to the input domain output requirement of the image reconstruction model, and as a specific adjusting mode, the embodiment of the application can adjust the difference value of the first preset resolution threshold value and the second preset resolution threshold value according to the difference value of the input resolution and the inattentive resolution.
Optionally, in an embodiment of the present application, after obtaining the multi-view reconstructed image of the object to be reconstructed, the method further includes: and re-reconstructing the multi-view reconstructed image by using an image reconstruction algorithm to obtain a two-dimensional or three-dimensional reconstructed image of the object to be reconstructed.
It should be noted that after the multi-view reconstructed image of the object to be reconstructed is obtained by inputting the multiple multi-view images into the pre-trained image reconstruction model, the embodiment of the application may further reconstruct the multi-view reconstructed image by using an image reconstruction algorithm to obtain a two-dimensional or three-dimensional reconstructed image of the object to be reconstructed. For example, if the resolution of the multi-view image output after the light field is collected is increased, if the resolution of each view needs to be increased by 3 times (three times for horizontal and vertical), the number of scanning times theoretically needs to reach 3 × 3 times. It can be understood that each pixel is densely sampled 3 x 3 times, divided into 3 x 3 sub-pixels. In the embodiment of the present application, only 2 or 3 scans need to be used to achieve the resolution of 3 × 3 scans, and a high-resolution two-dimensional or three-dimensional image is finally reconstructed, where the scans may be performed in an image plane scanning manner, which includes but is not limited to displacement through a microlens array, scanning by a photosensor, or vibration and displacement of a light path itself. The logic for executing the above-mentioned training and prediction process of the image reconstruction model is shown in fig. 2.
According to the multi-view image super-resolution reconstruction method based on the meta-imaging, which is provided by the embodiment of the application, multi-view images of an object to be reconstructed are collected; screening a multi-view target image with a resolution lower than a preset resolution from the multi-view images; and inputting the multi-view target image into a pre-trained image reconstruction model to obtain a multi-view reconstruction image of the object to be reconstructed, wherein the pre-trained image reconstruction model is obtained by training a plurality of images lower than a first preset resolution threshold and images correspondingly higher than a second preset resolution threshold. Therefore, the high multi-view resolution image can be recovered under a small scanning frequency, and the high-resolution two-dimensional or three-dimensional image can be finally reconstructed.
The multi-view image super-resolution reconstruction device based on meta-imaging proposed according to the embodiments of the present application is described next with reference to the accompanying drawings.
Fig. 3 is a block diagram of a multi-view image super-resolution reconstruction apparatus based on meta-imaging according to an embodiment of the present application.
As shown in fig. 3, the multi-view image super-resolution reconstruction apparatus 10 based on meta-imaging includes: a first acquisition module 100, a screening module 200, and a first reconstruction module 300.
The first acquisition module is used for acquiring a multi-view image of an object to be reconstructed. And the screening module is used for screening the multi-view target image with the resolution ratio lower than the preset resolution ratio from the multi-view images. The first reconstruction module is used for inputting the multi-view target images into a pre-trained image reconstruction model to obtain multi-view reconstructed images of the object to be reconstructed, wherein the pre-trained image reconstruction model is obtained by training a plurality of images lower than a first preset resolution threshold and images correspondingly higher than a second preset resolution threshold.
Optionally, in an embodiment of the present application, the method further includes: the second acquisition module is used for acquiring a plurality of first multi-view training images lower than a first preset resolution threshold value of a training reconstruction object and a plurality of second multi-view training images higher than a second preset resolution threshold value corresponding to the first multi-view training images before inputting the multi-view target images into a pre-trained image reconstruction model, wherein the second preset resolution threshold value is larger than the first preset resolution threshold value; the training module is used for training the neural network model by taking the first multi-view training image as input and the second multi-view training image as output; and the generating module is used for generating a pre-trained image reconstruction model when the neural network model meets the training termination condition.
Optionally, in an embodiment of the present application, the training termination condition includes: the resolution difference value between the output image and the input image of the neural network is larger than the preset resolution difference value or the iteration times meet the preset times.
Optionally, in an embodiment of the present application, the method further includes: the acquisition module is used for acquiring a target resolution difference value of an input image and an output image in the image reconstruction model before generating a pre-trained image reconstruction model; and the matching module is used for matching the difference value of the first preset resolution threshold value and the second preset resolution threshold value according to the target resolution difference value.
Optionally, in an embodiment of the present application, the method further includes: and the second reconstruction module is used for re-reconstructing the multi-view reconstructed image by using an image reconstruction algorithm after the multi-view reconstructed image of the object to be reconstructed is obtained, so as to obtain a two-dimensional or three-dimensional reconstructed image of the object to be reconstructed.
It should be noted that the foregoing explanation on the embodiment of the multi-view image super-resolution reconstruction method based on meta-imaging is also applicable to the multi-view image super-resolution reconstruction apparatus based on meta-imaging in this embodiment, and details are not repeated here.
According to the multi-view image super-resolution reconstruction device based on the meta-imaging, which is provided by the embodiment of the application, multi-view images of an object to be reconstructed are collected; screening a multi-view target image with a resolution lower than a preset resolution from the multi-view images; and inputting the multi-view target image into a pre-trained image reconstruction model to obtain a multi-view reconstruction image of the object to be reconstructed, wherein the pre-trained image reconstruction model is obtained by training a plurality of images lower than a first preset resolution threshold and images correspondingly higher than a second preset resolution threshold. And recovering a higher multi-view resolution image under fewer scanning times by utilizing the redundant sampling characteristic among the multi-view images, and finally realizing the reconstruction of the low-resolution multi-view image by utilizing a neural network model.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device may include:
The processor 402 executes the program to implement the multi-view image super-resolution reconstruction method based on meta-imaging provided in the above-described embodiments.
Further, the electronic device further includes:
a communication interface 403 for communication between the memory 401 and the processor 402.
A memory 401 for storing computer programs executable on the processor 402.
If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 may be connected to each other through a bus and perform communication with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 4, but this does not indicate only one bus or one type of bus.
Optionally, in a specific implementation, if the memory 401, the processor 402, and the communication interface 403 are integrated on a chip, the memory 401, the processor 402, and the communication interface 403 may complete mutual communication through an internal interface.
The processor 402 may be a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement embodiments of the present Application.
The present embodiment also provides a computer-readable storage medium, on which a computer program is stored, wherein the program is executed by a processor to implement the above meta-imaging based multi-view image super-resolution reconstruction method.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or N embodiments or examples. Moreover, various embodiments or examples and features of various embodiments or examples described in this specification can be combined and combined by one skilled in the art without being mutually inconsistent.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "N" means at least two, e.g., two, three, etc., unless explicitly defined otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more N executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of implementing the embodiments of the present application.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are well known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried out in the method of implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and the program, when executed, includes one or a combination of the steps of the method embodiments.
Claims (12)
1. A multi-view image super-resolution reconstruction method based on meta-imaging is characterized by comprising the following steps:
collecting a multi-view image of an object to be reconstructed;
screening a multi-view target image with a resolution lower than a preset resolution from the multi-view images;
and inputting the multi-view target image into a pre-trained image reconstruction model to obtain a multi-view reconstruction image of the object to be reconstructed, wherein the pre-trained image reconstruction model is obtained by training a plurality of images lower than a first preset resolution threshold value and images correspondingly higher than a second preset resolution threshold value.
2. The method of claim 1, wherein before inputting the multi-view target image into a pre-trained image reconstruction model, further comprising:
collecting a plurality of first multi-view training images of a training reconstruction object, wherein the first multi-view training images are lower than the first preset resolution threshold value, and a plurality of second multi-view training images which correspond to the first multi-view training images and are higher than the second preset resolution threshold value, wherein the second preset resolution threshold value is larger than the first preset resolution threshold value;
training a neural network model by taking the first multi-view training image as input and the second multi-view training image as output;
and when the neural network model meets the training termination condition, generating the pre-trained image reconstruction model.
3. The method of claim 2, wherein the training termination condition comprises:
and the resolution difference value between the output image and the input image of the neural network is greater than the preset resolution difference value or the iteration times meet the preset times.
4. The method of claim 2, further comprising, prior to generating the pre-trained image reconstruction model:
acquiring a target resolution difference value of an input image and an output image in the image reconstruction model;
and matching the difference value of the first preset resolution threshold value and the second preset resolution threshold value according to the target resolution difference value.
5. The method according to any one of claims 1-4, further comprising, after obtaining the multi-view reconstructed image of the object to be reconstructed:
and re-reconstructing the multi-view reconstructed image by using an image reconstruction algorithm to obtain a two-dimensional or three-dimensional reconstructed image of the object to be reconstructed.
6. A multi-view image super-resolution reconstruction device based on meta-imaging is characterized by comprising:
the first acquisition module is used for acquiring a multi-view image of an object to be reconstructed;
the screening module is used for screening multi-view target images with the resolution lower than a preset resolution from the multi-view images;
the first reconstruction module is used for inputting the multi-view target image into a pre-trained image reconstruction model to obtain a multi-view reconstruction image of the object to be reconstructed, wherein the pre-trained image reconstruction model is obtained by training a plurality of images lower than a first preset resolution threshold value and images correspondingly higher than a second preset resolution threshold value.
7. The apparatus of claim 6, further comprising:
a second collecting module, configured to collect, before the multi-view target image is input into a pre-trained image reconstruction model, a plurality of first multi-view training images lower than the first preset resolution threshold for training a reconstruction object and a plurality of second multi-view training images higher than the second preset resolution threshold corresponding to the first multi-view training images, where the second preset resolution threshold is greater than the first preset resolution threshold;
the training module is used for training a neural network model by taking the first multi-view training image as input and the second multi-view training image as output;
and the generating module is used for generating the pre-trained image reconstruction model when the neural network model meets the training termination condition.
8. The apparatus of claim 7, wherein the training termination condition comprises:
and the resolution difference value between the output image and the input image of the neural network is greater than the preset resolution difference value or the iteration times meet the preset times.
9. The apparatus of claim 7, further comprising:
the acquisition module is used for acquiring a target resolution difference value of an input image and an output image in the image reconstruction model before the pre-trained image reconstruction model is generated;
and the matching module is used for matching the difference value of the first preset resolution threshold value and the second preset resolution threshold value according to the target resolution difference value.
10. The apparatus according to any one of claims 6-9, further comprising, after obtaining the multi-view reconstructed image of the object to be reconstructed:
and the second reconstruction module is used for reconstructing the multi-view reconstructed image by using an image reconstruction algorithm to obtain a two-dimensional or three-dimensional reconstructed image of the object to be reconstructed.
11. An electronic device, comprising: memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the meta-imaging based multi-view image super-resolution reconstruction method according to any one of claims 1 to 5.
12. A computer-readable storage medium, on which a computer program is stored, which program is executable by a processor for implementing a method for super-resolution reconstruction of multi-view images based on meta-imaging as claimed in any one of claims 1 to 5.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210605176.4A CN115187454A (en) | 2022-05-30 | 2022-05-30 | Multi-view image super-resolution reconstruction method and device based on meta-imaging |
| PCT/CN2022/103912 WO2023231138A1 (en) | 2022-05-30 | 2022-07-05 | Multi-angle-of-view image super-resolution reconstruction method and apparatus based on meta-imaging |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210605176.4A CN115187454A (en) | 2022-05-30 | 2022-05-30 | Multi-view image super-resolution reconstruction method and device based on meta-imaging |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115187454A true CN115187454A (en) | 2022-10-14 |
Family
ID=83513545
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210605176.4A Pending CN115187454A (en) | 2022-05-30 | 2022-05-30 | Multi-view image super-resolution reconstruction method and device based on meta-imaging |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN115187454A (en) |
| WO (1) | WO2023231138A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115909255A (en) * | 2023-01-05 | 2023-04-04 | 北京百度网讯科技有限公司 | Image generation method, image segmentation method, image generation device, image segmentation device, vehicle-mounted terminal and medium |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7447382B2 (en) * | 2004-06-30 | 2008-11-04 | Intel Corporation | Computing a higher resolution image from multiple lower resolution images using model-based, robust Bayesian estimation |
| CN108074218B (en) * | 2017-12-29 | 2021-02-23 | 清华大学 | Image super-resolution method and device based on light field acquisition device |
| US10891527B2 (en) * | 2019-03-19 | 2021-01-12 | Mitsubishi Electric Research Laboratories, Inc. | Systems and methods for multi-spectral image fusion using unrolled projected gradient descent and convolutinoal neural network |
| CN112750076B (en) * | 2020-04-13 | 2022-11-15 | 奕目(上海)科技有限公司 | Light field multi-view image super-resolution reconstruction method based on deep learning |
| CN113763248A (en) * | 2021-09-08 | 2021-12-07 | 深圳前海微众银行股份有限公司 | Super-resolution image reconstruction method, device, equipment and storage medium |
| CN113870433A (en) * | 2021-09-09 | 2021-12-31 | 华中科技大学 | Light field super-resolution three-dimensional reconstruction method and system |
| CN114494258B (en) * | 2022-04-15 | 2022-08-30 | 清华大学 | Lens aberration prediction and image reconstruction method and device |
| CN114549731B (en) * | 2022-04-22 | 2022-09-16 | 清华大学 | Method and device for generating visual angle image, electronic equipment and storage medium |
-
2022
- 2022-05-30 CN CN202210605176.4A patent/CN115187454A/en active Pending
- 2022-07-05 WO PCT/CN2022/103912 patent/WO2023231138A1/en unknown
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115909255A (en) * | 2023-01-05 | 2023-04-04 | 北京百度网讯科技有限公司 | Image generation method, image segmentation method, image generation device, image segmentation device, vehicle-mounted terminal and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023231138A1 (en) | 2023-12-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5909540B2 (en) | Image processing display device | |
| CN111709895A (en) | Image blind deblurring method and system based on attention mechanism | |
| US9066034B2 (en) | Image processing apparatus, method and program with different pixel aperture characteristics | |
| EP3644270A1 (en) | Image processing device, imaging device, image processing method, program, and storage medium | |
| JP7206386B2 (en) | Image processing model training method, image processing method, network device, and storage medium | |
| CN106683174B (en) | Three-dimensional reconstruction method and device of binocular vision system and binocular vision system | |
| KR20110032402A (en) | Method and apparatus for blending multiple images | |
| CN103828344A (en) | Image processing apparatus, image processing method and program, and image pickup apparatus including image processing apparatus | |
| CN110599400A (en) | EPI-based light field image super-resolution method | |
| WO2007042073A1 (en) | Image processing method and system | |
| JP2005018535A (en) | Signal processor, signal processing method, program, and recording medium | |
| CN112862689A (en) | Image super-resolution reconstruction method and system | |
| CN112837245A (en) | Dynamic scene deblurring method based on multi-mode fusion | |
| CN115187454A (en) | Multi-view image super-resolution reconstruction method and device based on meta-imaging | |
| CN114494258B (en) | Lens aberration prediction and image reconstruction method and device | |
| Chang et al. | Beyond camera motion blur removing: How to handle outliers in deblurring | |
| CN111369443A (en) | Zero-order learning super-resolution method for optical field cross-scale | |
| WO2003003296A1 (en) | Neural network based element image pre-processor and method of pre-processing using a neural network | |
| JP7398938B2 (en) | Information processing device and its learning method | |
| CN111353982B (en) | Depth camera image sequence screening method and device | |
| Zheng et al. | DomainPlus: cross transform domain learning towards high dynamic range imaging | |
| CN117274514A (en) | Remote sensing image generation method and device based on ground-air visual angle geometric transformation | |
| Vien et al. | Bidirectional motion estimation with cyclic cost volume for high dynamic range imaging | |
| CN114119428B (en) | Image deblurring method and device | |
| CN116309066A (en) | Super-resolution imaging method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |