CN113096010A

CN113096010A - Image reconstruction method and apparatus, and storage medium

Info

Publication number: CN113096010A
Application number: CN202110292950.6A
Authority: CN
Inventors: 卓海杰
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2021-03-18
Filing date: 2021-03-18
Publication date: 2021-07-09

Abstract

The embodiment of the application discloses an image reconstruction method, image reconstruction equipment and a storage medium, wherein the method comprises the following steps: in a first working mode, carrying out image acquisition processing on a target object at a target position through an image sensor to obtain a first image corresponding to the target object; in a second working mode, carrying out image acquisition processing on a target object at a target position through an image sensor to obtain a second image corresponding to the target object; wherein the resolution of the first image is less than the resolution of the second image; training the initial image reconstruction model based on the first image and the second image to obtain a super-resolution image reconstruction model; and reconstructing the image to be detected according to the super-resolution image reconstruction model to obtain a target high-resolution image corresponding to the image to be detected.

Description

Image reconstruction method and apparatus, and storage medium

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to an image reconstruction method and apparatus, and a storage medium.

Background

When portable electronic equipment such as a smart phone and a tablet uses a camera to shoot images, the performance of the image sensor is limited, the shot images often have the problems of low definition and low resolution, the image quality is poor, and the requirements of users are difficult to meet.

In order to obtain a clear high-resolution image, a super-resolution image reconstruction method is mostly adopted at present to reconstruct a high-resolution image with higher definition from an unclear low-resolution image so as to improve the image quality. However, the super-resolution image reconstruction method adopted in the related art is generally not ideal in reconstruction effect, so that the quality of the high-resolution image obtained by reconstruction is poor.

Disclosure of Invention

The embodiment of the application provides an image reconstruction method, image reconstruction equipment and a storage medium, the reconstruction effect is better, and high-quality high-resolution images can be obtained.

The technical scheme of the embodiment of the application is realized as follows:

in a first aspect, an embodiment of the present application provides an image reconstruction method, where the method includes:

in a first working mode, acquiring and processing a target object at a target position through an image sensor to obtain a first image corresponding to the target object;

in a second working mode, acquiring and processing the target object at the target position through the image sensor to obtain a second image corresponding to the target object; wherein the resolution of the first image is less than the resolution of the second image;

training an initial image reconstruction model based on the first image and the second image to obtain a super-resolution image reconstruction model;

and reconstructing the image to be detected according to the super-resolution image reconstruction model to obtain a target high-resolution image corresponding to the image to be detected.

In a second aspect, an embodiment of the present application provides an image reconstruction apparatus, including: a first acquisition unit, a training unit and a reconstruction unit,

the first acquisition unit is used for acquiring and processing an image of a target object at a target position through an image sensor in a first working mode to obtain a first image corresponding to the target object;

the first acquisition unit is further configured to, in a second working mode, perform image acquisition processing on the target object at the target position through the image sensor to obtain a second image corresponding to the target object; wherein the resolution of the first image is less than the resolution of the second image;

the training unit is used for training an initial image reconstruction model based on the first image and the second image to obtain a super-resolution image reconstruction model;

and the reconstruction unit is used for reconstructing the image to be detected according to the super-resolution image reconstruction model to obtain a target high-resolution image corresponding to the image to be detected.

In a third aspect, an embodiment of the present application provides an image reconstruction apparatus, which includes a processor, and a memory storing instructions executable by the processor, and when the instructions are executed by the processor, the image reconstruction method as described above is implemented.

In a fourth aspect, the present application provides a computer-readable storage medium, on which a program is stored, for use in an image reconstruction apparatus, where the program, when executed by a processor, implements the image reconstruction method as described above.

The embodiment of the application provides an image reconstruction method and equipment, and a storage medium, wherein the image reconstruction equipment can acquire and process a target object at a target position through an image sensor in a first working mode to obtain a first image corresponding to the target object; in a second working mode, acquiring and processing a target object at a target position through an image sensor to obtain a second image corresponding to the target object; wherein the resolution of the first image is less than the resolution of the second image; training the initial image reconstruction model based on the first image and the second image to obtain a super-resolution image reconstruction model; and reconstructing the image to be detected according to the super-resolution image reconstruction model to obtain a target high-resolution image corresponding to the image to be detected. That is to say, in the embodiment of the present application, the terminal performs image acquisition processing on the same target object at the same position through the same image sensor in different working modes to obtain a first image with low resolution and a second image with high resolution of the target object; and then based on a deep learning mode, performing model training processing by using the first image and the second image to obtain a super-resolution image reconstruction model capable of reconstructing a high-resolution image with higher definition from a low-resolution image. Therefore, in the application, the model training sample data acquired by the image reconstruction device is obtained by switching different camera image showing modes to acquire the same target object existing in the real world at the same shooting position and in the same shooting environment by the same image sensor, so that the quality of the training sample data is higher, the reconstruction effect of the super-resolution image reconstruction model acquired based on the training sample data is better, and the high-quality high-resolution image with higher definition can be acquired.

Drawings

FIG. 1A is a schematic illustration of Binning;

FIG. 1B is a schematic diagram of Remosaic;

fig. 2 is a first schematic flow chart illustrating an implementation of an image reconstruction method according to an embodiment of the present application;

fig. 3 is a schematic network structure diagram of an image reconstruction model according to an embodiment of the present application;

fig. 4 is a schematic flow chart illustrating an implementation process of the image reconstruction method according to the embodiment of the present application;

fig. 5 is a schematic effect diagram of a super-resolution image reconstruction model provided in an embodiment of the present application;

fig. 6 is a schematic flow chart illustrating an implementation of the image reconstruction method according to the embodiment of the present application;

fig. 7 is a fourth schematic flowchart of an image reconstruction method according to an embodiment of the present application;

fig. 8 is a schematic flow chart illustrating an implementation of the image reconstruction method according to the embodiment of the present application;

fig. 9 is a schematic structural diagram of a first image reconstruction device according to the present application;

fig. 10 is a schematic structural diagram of a second image reconstruction apparatus according to the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the specific embodiments described herein are illustrative of the relevant application and are not limiting of the application. It should be noted that, for the convenience of description, only the parts related to the related applications are shown in the drawings.

Before further detailed description of the embodiments of the present invention, terms and expressions mentioned in the embodiments of the present invention are explained, and the terms and expressions mentioned in the embodiments of the present invention are applied to the following explanations.

1) Pixel four in one (Binning): the four-in-one operation is performed on each pixel, and four adjacent pixels in the initial image are combined into one pixel block to generate a result image with the resolution of 12 MP.

Specifically, fig. 1A is a schematic diagram of Binning, where a square lattice filled with left oblique lines represents a pixel G, a square lattice filled with right oblique lines represents a pixel R, and a square lattice filled with cross oblique lines represents a pixel B, as shown in fig. 1A, 4G pixels are converged into one G pixel, and R and B are the same, and the resolution is reduced to 1/4, so that the purpose of outputting the graph of the 48M image sensor to 12M, which makes the subsequent algorithm processing more time-friendly and memory-friendly.

In addition, although the signal-to-noise ratio of the resulting image obtained by the Binning mode is high and the noise is small, the image definition is insufficient and the detail resolution capability is poor.

2) Pixel rearrangement (Remosaic): the initial image was converted to Bayer format by the re-mosaic algorithm to obtain a resultant image with a resolution of 48M.

Specifically, fig. 1B is a schematic diagram of Remosaic, where a square lattice filled with oblique lines on the left represents a green pixel tree, a square lattice filled with oblique lines on the right represents a Red pixel Red, and a square lattice filled with oblique lines on the cross represents a Blue pixel Blue, as shown in fig. 1B, the original four same G/R/B pixels are rearranged, and interpolation is performed at corresponding positions. Because the resolution is not lost in the Remosaic mode, a result image obtained through the Remosaic mode has higher definition and stronger detail resolution capability than a Binning mode, but because the four-in-one operation is not carried out, the single pixel size is smaller and the light sensing capability is weak, so that the signal-to-noise ratio of the result image is lower and the noise point is larger.

3) Super Resolution (Super-Resolution): the resolution of the original image is improved by a hardware or software method, and super-resolution reconstruction is performed by obtaining a high-resolution image through a series of low-resolution images.

4) Super-resolution degradation model

The hyper-division task is a comprehensive task, can be regarded as a combination of deblurring, denoising and upsampling, and can be expressed by a mathematical expression:

Y＝downsample(KX+N) (1)

in formula (1), X is a high-resolution image, K is a blur kernel, N is noise, Y is a low-resolution image, and downsample is an image downsampling. When a super-resolution reconstruction algorithm based on deep learning is carried out, real hyper-resolution data is needed, and a high-resolution image X and a low-resolution image satisfying real image noise distribution N and real amplification blur K are needed to obtain a real hyper-resolution data pair.

At present, with the large-scale use of portable electronic devices, more and more people commonly use portable electronic devices such as smart phones and tablets to take pictures, but the performance of image sensors is limited when the portable electronic devices such as smart phones and tablets take pictures by using cameras, and the taken pictures often have the problems of low definition and low resolution, and the quality of the pictures is poor, so that the requirements of users are difficult to meet.

In order to obtain a clear high-resolution image, a super-resolution image reconstruction method is mostly adopted at present to reconstruct a high-resolution image with higher definition from an unclear low-resolution image so as to improve the image quality.

The existing super-Resolution reconstruction algorithm based on deep learning includes two types, one is the construction of a super-Resolution network model directly based on a public data set, such as DIV2K, but since a Low Resolution image (LR) in DIV2K is directly down-sampled by a High Resolution image (HR), referring to formula (1), only a degradation process of downsample is considered, and it is not considered that the LR of the real world usually also includes fuzzy degradation K and noise degradation N. Furthermore, the super-resolution network model constructed based on the public data set only has good effect on the public data set, and in a real image, the network model is usually poor or even has no effect.

Because the super-resolution reconstruction algorithm based on deep learning has a high degree of dependence on data, and the quality of the data directly determines the upper limit of the super-resolution effect, the CameraSR proposes a data acquisition scheme based on near-far shooting in order to solve the problems caused by the above public data set. Firstly, printing a target image and placing the target image at a far position, shooting for the first time and placing a mobile phone at a position A far away from the target, wherein the shot image is used as LR; and B, in the second shooting, the mobile phone is placed at a position B close to the target, and the shot image is taken as HR. Since both the two shots vary in position and environment, the LR and HR need to be positionally aligned and color calibrated. Referring to equation (1), in this case, since LR shooting is a real shooting environment, noise and blur of the LR shooting itself coincide with the real world. As for the down-sampling downsample, simulation is performed by the disparity in the distance between two shots.

However, when data acquisition is performed by far and near shooting to construct a network model, there are several main defects:

(1) because the change of the surrounding environment is ensured to be small, the image of the real world needs to be printed by a camera and put on a baffle for shooting, the difference between the image and the image of the real world is large, and the difference can cause the data distribution of the training data set and the data distribution of the real test set to be inconsistent, thereby causing poor over-distinguishing effect.

(2) The proposal has quite strict requirements on the precision of the device, not only the position of each movement needs to be accurately measured, but also the 3A of the mobile phone needs to be locked to ensure that the brightness and the color of the two images are similar as much as possible, in addition, the laboratory environment needs to be shot to ensure that the indoor environments of the two shots are consistent,

(3) the image shot twice still needs to be subjected to pixel alignment and color correction due to position deviation, and the alignment pressure is still relatively large due to inconsistency of the two shooting positions.

In summary, the super-resolution image reconstruction method adopted in the related art is generally not ideal in reconstruction effect, so that the quality of the high-resolution image obtained by reconstruction is poor.

In view of this, in order to solve the problems of the existing hyper-variability algorithm, embodiments of the present application provide an image reconstruction method and apparatus, and a storage medium, and specifically, a terminal performs image acquisition processing on the same target object at the same position in different working modes through the same image sensor to obtain a first image with low resolution and a second image with high resolution of the target object; and then based on a deep learning mode, performing model training processing by using the first image and the second image to obtain a super-resolution image reconstruction model capable of reconstructing a high-resolution image with higher definition from a low-resolution image. Therefore, in the application, the model training sample data acquired by the image reconstruction device is obtained by switching different camera image showing modes to acquire the same target object existing in the real world at the same shooting position and in the same shooting environment by the same image sensor, so that the quality of the training sample data is higher, the reconstruction effect of the super-resolution image reconstruction model acquired based on the training sample data is better, and the high-quality high-resolution image with higher definition can be acquired.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

An embodiment of the present application provides a method for calculating a pressure field, fig. 2 is a schematic implementation flow diagram of an image reconstruction method provided in the embodiment of the present application, and as shown in fig. 2, in the embodiment of the present application, a method for performing image reconstruction by an image reconstruction device may include the following steps:

step 101, in a first working mode, image acquisition processing is performed on a target object at a target position through an image sensor, so as to obtain a first image corresponding to the target object.

In the embodiment of the application, the image reconstruction device may perform image acquisition processing on a target object at a target position through the image sensor in a first working mode, so as to obtain a first image corresponding to the target object.

It should be understood that in the embodiments of the present application, the image reconstruction device may be a mobile electronic device, and may also be a non-mobile electronic device. The mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted terminal, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), etc.; the non-mobile electronic device may be a Personal Computer (PC), a Television (TV), a teller machine, a self-service machine, or the like; the embodiments of the present application are not particularly limited.

In an embodiment of the present application, an execution subject of the image reconstruction method provided in the present application may be an image reconstruction device, or a Central Processing Unit (CPU) of the image reconstruction device, or a control module in the image reconstruction device for executing the image reconstruction method.

Here, in the embodiment of the present application, an image reconstruction method executed by an image reconstruction device is taken as an example, and the image processing method provided in the embodiment of the present application is described.

It should be noted that, in the embodiment of the present application, the image reconstruction device may be an electronic device having an operating system. The operating system may be an Android (Android) operating system, an apple IOS operating system, or other possible operating systems, and the embodiments of the present application are not particularly limited.

It should be noted that, in the embodiment of the present application, the first operation mode refers to an image rendering mode of an image sensor in an image reconstruction apparatus. Specifically, the first operating mode may be a drawing mode in which the resolution of the acquired image is low. Such as Binning model.

It should be understood that the image sensor refers to a functional module for image acquisition in the image reconstruction device, i.e. a camera. Optionally, the image sensor may be a telephoto camera in the device, or may be a main camera in the device.

It should be understood that the target object refers to a target object for image acquisition. Alternatively, the target object may be a dynamically movable human or animal present in the real world; or static standing building or trees.

It is to be understood that the target position is used to characterize the distance of the target object from the image sensor in the image reconstruction device.

Specifically, in the embodiment of the present application, after selecting the first working mode with a lower resolution of the obtained image, the image sensor in the image reconstruction device performs image acquisition processing on the target object with the distance from the convergence target, so as to obtain the first image with the lower resolution. For example, a mobile phone camera takes a picture of a building which is 10 meters away from each other in a Binning mode to obtain a picture of the building with low resolution.

Further, in the embodiment of the present application, after the image reconstruction device performs image acquisition processing on the target object in the first working mode to obtain the first image, the image reconstruction device may further switch to the second working mode to continue the image acquisition processing on the target object.

102, in a second working mode, carrying out image acquisition processing on a target object at a target position through an image sensor to obtain a second image corresponding to the target object; wherein the resolution of the first image is less than the resolution of the second image.

In an embodiment of the application, after obtaining the first image of the target object in the first operating mode, the image reconstruction device may continue to perform image acquisition processing on the same target object at the same position through the same image sensor in the second operating mode to obtain a second image corresponding to the target object.

It should be noted that, in the embodiment of the present application, the second operation mode refers to another image drawing mode adopted by the image sensor in the image reconstruction apparatus, which is distinguished from the first operation mode. Specifically, the second operation mode may be an image drawing mode in which the resolution of the acquired image is high. Such as Remosaic mode.

It should be noted that, when the image reconstruction device continues to perform the image acquisition processing in the second operating mode, the image sensor, the target object, and the target position capable of representing the distance from the target object to the image sensor are all the same as the parameters of the image acquisition in the first operating mode. That is, the image reconstruction apparatus acquires two images of the target object having different resolutions by switching the operation mode under the same photographing position and photographing environment.

It can be understood that if the image sensor performs image acquisition processing by switching the operation modes, the longer the time for switching the first operation mode and the second operation mode is, the target object state may change, which may result in a difference between the captured images. Therefore, in the embodiment of the present application, after the image sensor performs image capturing in the first operating mode to obtain the first image, the image sensor needs to switch to the second operating mode for continuing to perform image capturing to obtain the second image in a short time, that is, the operating mode is switched rapidly to implement continuous shooting of two images.

It should be noted that, in the embodiment of the present application, a low-resolution image of a target object is obtained in the first operation mode, and a high-resolution image of the same target object is obtained in the second operation mode.

It can be understood that, in the embodiment of the present application, the terminal may also perform image acquisition processing on the target object to obtain a high-resolution image in the second working mode; and then switching to a second working mode to obtain a low-resolution image of the target object. That is, the sequence of the image pattern when the image sensor operates is not limited, that is, step 102 may be before step 101, and step 102 may also be after step 101.

Further, in the embodiment of the present application, the image reconstruction apparatus may perform image acquisition on a variety of target objects by using the above image acquisition method to obtain low resolution images and high resolution images corresponding to the variety of target objects, i.e. low resolution-high resolution image pairs.

Alternatively, the image sensor may be a Quad Bayer image sensor. That is, a Quad Bayer image sensor, which acquires an image using a Quad Bayer color filter array, may be provided in the electronic device.

It should be noted that, in the embodiment of the present application, the first image and the second image are both image files in a color YUV format.

Here, the YUV format is a color coding method. YUV is a kind of compiled true-color space (color space), and the proper terms such as Y' UV, YUV, YCbCr, YPbPr, etc. may be called YUV, overlapping with each other. "Y" represents brightness (Luma or Luma), i.e., a gray scale value; "U" and "V" represent Chrominance (or Chroma) and are used to describe the color and saturation of an image for specifying the color of a pixel.

Further, in the embodiment of the present application, after obtaining the first image with low resolution and the second image with high resolution of the same target object, the image reconstruction apparatus may perform a training process of the model based on the image data.

And 103, training the initial image reconstruction model based on the first image and the second image to obtain a super-resolution image reconstruction model.

In the embodiment of the present application, after obtaining the first image with low resolution and the second image with high resolution of the same target object, the image reconstruction device may further perform training processing on the initial image reconstruction model based on the image data, so as to obtain the super-resolution image reconstruction model.

Specifically, in the embodiment of the present application, the image reconstruction device may use the first image and the second image as model training data, specifically, use the first image with low resolution as an initial model input, and use the second image with high resolution as a reference map for supervised learning, thereby obtaining a super-resolution image reconstruction model.

It can be understood that, based on the deep learning algorithm, in order to obtain the super-resolution image reconstruction model, a researcher needs to train the super-resolution image reconstruction model through a large number of low-resolution-high-resolution images, and the better the training data is, the better the model effect is.

Because the training data adopted by the method is that the same camera is used for plotting, the shooting position and the shooting environment are the same, and the target object is a person or a still object existing in the real environment and better accords with the real image data. Therefore, the super-resolution image reconstruction model further obtained by using the training data has better effect.

It should be noted that, in the embodiment of the present application, the structure of the network model adopts a U-shaped network structure, that is, a network structure similar to U-net. Specifically, the method includes performing path compression operation through the convolutional layer to perform down-sampling, namely, performing image resolution step-by-step reduction on an input image, and then performing path expansion operation through the deconvolution layer to obtain a low-resolution image after down-sampling, and performing resolution step-by-step increment to further obtain a corresponding high-resolution image.

It should be noted that, in the embodiment of the present application, in addition to the conventional convolution layer and the conventional deconvolution layer, an Information multi-distillation block (IMDB) module and an S2db module are newly added to the neural network structure of the initial image reconstruction model

Specifically, fig. 3 is a schematic diagram of a network structure of an image reconstruction model proposed in the embodiment of the present application, and as shown in fig. 3, an input LR image is first up-sampled and then input to the convolutional layer conv3, that is, a 3 × 3 convolutional layer is subjected to processing such as feature extraction, and the number of channels is changed from 1 to 4. Then, the spatial resolution is converted into the number of channels by the S2db module, the image size becomes smaller, and the number of channels is changed from 4 to 64. Further, the image obtained by the S2db module is continuously encapsulated by the IMDB module, and the number of channels is not changed. Then, the picture downsampling and packaging are carried out through a second S2db module and a second IMDB module, the image size is smaller, and the number of channels is changed from 64 to 256; and then, the image is subjected to channel number-to-spatial resolution conversion and restoration operation through a d2s module, and then is packaged again through a third IMDB module. Further, the concat module continues to perform superposition of the channel number, including the channel number 64 corresponding to the first IMDB module and the channel number 64 corresponding to the third IMDB module, and then performs superposition processing, so as to obtain a channel number of 128. Then, the number of channels is reduced by the convolutional layer conv1, the spatial resolution is converted into the number of channels by the third S2d module, the output result of the third S2d module is overlapped with the output result of the first convolutional layer conv3, the overlapped result is subjected to feature extraction and channel number reduction by the second convolutional layer conv3 again, the number of channels is changed from 64 to 4, and then the HR image with the number of channels corresponding to the LR image being 1 is further obtained.

It is to be understood that, since the presence of the fine jitter is not avoided when the low resolution image and the high resolution image of the same target object are obtained through different camera imaging modes, in the embodiment of the present application, in order to solve the effect caused by the fine jitter in the two shots, the image reconstruction apparatus may perform the image alignment process on the first image and the second image of the target object before obtaining the first image and the second image.

Specifically, fig. 4 is a schematic view of a second implementation flow of the image reconstruction method provided in the embodiment of the present application, and as shown in fig. 4, in the embodiment of the present application, the method for training the initial image reconstruction model based on the first image and the second image to obtain the super-resolution image reconstruction model may include the following steps:

and 103a, aligning the first image and the second image to obtain an aligned first image and an aligned second image.

And 103b, training the initial image reconstruction model based on the aligned first image and the aligned second image to obtain a super-resolution image reconstruction model.

Specifically, in order to obtain better training data and ensure that the effect of the image reconstruction model is better, the image reconstruction device may perform pixel alignment processing on the first image with the low resolution and the second image with the high resolution, so as to obtain the first image with the low resolution after alignment and the second image with the high resolution after alignment, and perform model training through the first image after alignment and the second image after alignment. Namely, the aligned first image is used as an initial model input, and the aligned second image is used as a reference image for supervised learning, so as to obtain a super-resolution image reconstruction model with better effect.

On the other hand, the image reconstruction device may perform color correction processing on the two images on the basis of performing image alignment, and further perform model training processing on the processed two images.

Further, in the embodiment of the present application, after obtaining the super-resolution image reconstruction model, the image reconstruction device may further perform reconstruction processing on the low-resolution image through the image reconstruction model.

And 104, reconstructing the image to be detected according to the super-resolution image reconstruction model to obtain a target high-resolution image corresponding to the image to be detected.

In the embodiment of the application, after the image reconstruction device obtains the super-resolution image reconstruction model, the image reconstruction processing can be performed on the image to be detected through the model, and then the high-resolution image corresponding to the image to be detected is obtained.

Specifically, the image reconstruction device may input the image to be measured as a model, and then output and obtain a high-resolution image corresponding to the image to be measured.

It can be understood that, since the super-resolution image reconstruction model obtained based on the deep learning algorithm uses the Binning mode to obtain a low-resolution image as an input and uses the high-resolution image obtained in the Remosaic mode as a reference image for supervised learning, the high-resolution image finally output by the model is actually the image obtained in the Remosaic image mode.

In the use process of the model, the image to be measured may be an image obtained by using a Binning mode or a Remosaic mode. If the image to be detected is an image obtained in a Binning mode, inputting the image to be detected into a super-resolution image reconstruction model, and obtaining a high-resolution image with the resolution being greater than that of the image to be detected; and if the image to be detected is the image obtained in the Binning mode, inputting the image to be detected into the super-resolution image reconstruction model, and obtaining a high-resolution image with the resolution basically the same as that of the image to be detected.

That is, regardless of the image to be measured, a target image with a high resolution can be obtained as long as the image to be measured is input to the super-resolution measurement model.

For example, fig. 5 is a schematic diagram of an effect of a super-resolution image reconstruction model provided in an embodiment of the present application, where fig. 5(a) is an image to be measured input by a model, and fig. 5(b) is an image output by the model, it can be seen that the input image of fig. 5(a) has a poor definition and is blurred, and after the super-resolution image reconstruction is performed, the image of fig. 5(b) has a higher definition.

Therefore, the super-resolution image reconstruction is completed by adopting the improved network structure similar to the U-net, and the relatively linear super-resolution effect is obtained. The simple and efficient data acquisition mode and the network model structure enable the deployment of the super-resolution image reconstruction algorithm based on deep learning on a mobile phone to be possible.

The embodiment of the application provides an image reconstruction method, wherein image reconstruction equipment acquires and processes the same target object at the same position in different working modes through the same image sensor so as to obtain a first image with low resolution and a second image with high resolution of the target object; and then based on a deep learning mode, performing model training processing by using the first image and the second image to obtain a super-resolution image reconstruction model capable of reconstructing a high-resolution image with higher definition from a low-resolution image. Therefore, in the application, the model training sample data acquired by the image reconstruction device is obtained by switching different camera image showing modes to acquire the same target object existing in the real world at the same shooting position and in the same shooting environment by the same image sensor, so that the quality of the training sample data is higher, the reconstruction effect of the super-resolution image reconstruction model acquired based on the training sample data is better, and the high-quality high-resolution image with higher definition can be acquired.

Based on the foregoing embodiment, in an embodiment of the present application, fig. 6 is a schematic flow chart illustrating an implementation of an image reconstruction method provided in the embodiment of the present application, and as shown in fig. 6, an image reconstruction device performs training processing on an initial image reconstruction model based on a first image and a second image, and a method for obtaining a super-resolution image reconstruction model includes the following steps:

and 103c, inputting the first image into the initial image reconstruction model, and outputting a super-resolution image prediction result.

And 103d, training based on the preset target loss function, the super-resolution image prediction result and the second image to obtain a super-resolution image reconstruction model.

It can be understood that the model input end of the super-resolution image reconstruction model finally obtained by the application corresponds to a low-resolution image, and the output end corresponds to a high-resolution image.

Specifically, in the embodiment of the present application, when performing the training process of the graph reconstruction model based on the low-resolution first image and the high-resolution second image, the image reconstruction device may input the low-resolution first image as model input data to the initial image reconstruction model, and further obtain the initial high-resolution image output by the model, that is, the super-resolution prediction image.

Further, in the process of inputting the first image with low resolution to the initial image reconstruction model for machine learning to train the model, the image reconstruction apparatus may introduce a target loss function, such as setting the loss function as a norm loss, and further perform model training processing based on the target loss function, the second image serving as the reference image, and the model training result super-resolution predicted image to obtain the super-resolution image reconstruction model.

Here, the image reconstruction apparatus may compare the second image, which is a reference image, with the model training result super-resolution prediction image each time using the target loss function, and then determine whether the current model training result satisfies an expected requirement based on the comparison result.

Specifically, fig. 7 is a fourth flowchart illustrating an image reconstruction method according to an embodiment of the present application, and as shown in fig. 7, in the embodiment of the present application, the method for obtaining a super-resolution image reconstruction model by performing the training process on the basis of a preset target loss function, a super-resolution predicted image and a second image by an image reconstruction device includes the following steps:

and step 103d1, calculating the difference value between the super-resolution predicted image and the second image through the target loss function.

And step 103d2, training according to the difference value to obtain a super-resolution image reconstruction model.

Here, the image reconstruction device may read the target loss function, and then perform model training processing in combination with the output result of the model in the training process and the reference image until the training result satisfies a certain condition, so as to obtain the super-resolution image reconstruction model.

Specifically, the image reconstruction device is preset to a loss threshold, such as a minimum absolute value error threshold. Using the target loss function, the image reconstruction device may calculate a difference value between the super-resolution predicted image and the second image, and then compare the difference value with a preset loss threshold value until the comparison result is that the difference value is less than or equal to the preset loss threshold value, such as the absolute value of the difference value is less than the minimum absolute value error threshold value.

Further, the image reconstruction apparatus may determine that the super-resolution predicted image output by the current model is substantially the same as the second image serving as the reference image, that is, the model obtained by the current training is the super-resolution image reconstruction model satisfying the condition.

The embodiment of the application provides an image reconstruction method, model training sample data obtained by image reconstruction equipment is obtained by switching different camera image drawing modes to acquire images of a same target object existing in the real world at the same shooting position and the same shooting environment by a same image sensor, so that a low-resolution-high-resolution image pair is obtained, the quality of the training sample data is higher, the reconstruction effect of a super-resolution image reconstruction model obtained based on the training sample data is better, and a high-quality high-resolution image with higher definition can be obtained.

Based on the foregoing embodiment, in an embodiment of the present application, fig. 8 is a schematic flow chart of an implementation process of an image reconstruction method provided in the embodiment of the present application, and as shown in fig. 8, before reconstructing a to-be-detected image according to a super-resolution image reconstruction model and obtaining a target high resolution image corresponding to the to-be-detected image, that is, before step 104, the method for image reconstruction by an image reconstruction device further includes the following steps:

and 105, carrying out image acquisition processing on the target object at the target position through the first image sensor to obtain a third image corresponding to the target object.

Step 106, carrying out image acquisition processing on the target object at the target position through a second image sensor to obtain a fourth image corresponding to the target object; wherein the resolution of the third image is less than the resolution of the fourth image.

And step 107, training the initial image reconstruction model based on the third image and the fourth image to obtain a super-resolution image reconstruction model.

Here, in addition to the step 101-.

It should be noted that, compared with the method for obtaining the low-resolution-high-resolution image pair by switching the operation mode, the method for obtaining the low-resolution-high-resolution image pair by switching the image sensor has the following common points: still carrying out image acquisition processing on the same target object at the same shooting position and in the same shooting environment; the difference is as follows: the image sensor needs to be changed, but the pattern of the image sensor does not need to be changed any more, i.e. the pattern of the two image sensors is the same.

Specifically, similar to the principle of step 101-102, the image reconstruction device may perform image acquisition processing on the target object at the target shooting position through the first image sensor with a poor imaging effect to obtain a third image with a low resolution; furthermore, the image reconstruction device can switch the image sensors, and the second image sensor with a good imaging effect is used for acquiring and processing the image of the same target object at the same shooting position, so that a fourth image with high resolution is obtained.

It can be understood that, taking a mobile phone as an example, the mobile phone usually includes 2 to 4 cameras, and for a relatively distant scene, the imaging effect of the telephoto camera is significantly better than that of the main camera, so that the image reconstruction device can shoot an image as LR through the main camera, and then shoot an image as HR through the telephoto camera for the same shooting object at the same shooting position and in the same shooting environment.

Further, the image reconstruction device may perform training processing on the initial image reconstruction model by using the obtained third image and fourth image to obtain a super-resolution image reconstruction model.

Here, in order to solve the influence of the fine jitter in the two shots, the image reconstruction device may perform processing such as pixel alignment and color correction on the third image and the fourth image, and then perform model training processing based on the processed third image and fourth image.

The embodiment of the application provides an image reconstruction method, model training sample data obtained by image reconstruction equipment is obtained by image sensors with different effect effects under the same shooting position and the same shooting environment, image acquisition is carried out on the same target object existing in the real world, and then a low-resolution-high-resolution image pair is obtained, the quality of the training sample data is higher, further the reconstruction effect of a super-resolution image reconstruction model obtained based on the training sample data is better, and a high-quality high-resolution image with higher definition can be obtained.

Based on the foregoing embodiment, in another embodiment of the present application, fig. 9 is a schematic structural diagram of a composition of the image reconstruction apparatus 10, as shown in fig. 9, the image reconstruction apparatus 10 may include a first acquisition unit 11, a training unit 12, a reconstruction unit 13, and a second acquisition unit 14,

the first acquisition unit 11 is configured to, in a first working mode, perform image acquisition processing on a target object at a target position through an image sensor to obtain a first image corresponding to the target object;

the first acquisition unit 11 is further configured to, in a second working mode, perform image acquisition processing on the target object at the target position through the image sensor to obtain a second image corresponding to the target object; wherein the resolution of the first image is less than the resolution of the second image;

the training unit 12 is configured to perform training processing on an initial image reconstruction model based on the first image and the second image to obtain a super-resolution image reconstruction model;

and the reconstruction unit 13 is configured to reconstruct the image to be detected according to the super-resolution image reconstruction model, and obtain a target high-resolution image corresponding to the image to be detected.

Further, in the embodiment of the present application, the first operation mode is a Binning mode; the second working mode is a Remosaic mode.

Further, in the embodiment of the present application, the super-resolution image reconstruction model adopts a U-shaped neural network structure.

Further, in an embodiment of the present application, the training unit 12 is specifically configured to perform alignment processing on the first image and the second image to obtain an aligned first image and an aligned second image; and training the initial image reconstruction model based on the aligned first image and the aligned second image to obtain the super-resolution image reconstruction model.

Further, in an embodiment of the present application, the training unit 12 is further specifically configured to input the first image into the initial image reconstruction model, and output a super-resolution prediction image; and performing the training processing based on a preset target loss function, the super-resolution prediction image and the second image to obtain the super-resolution image reconstruction model.

Further, in the embodiment of the present application, the training unit 12 is further specifically configured to calculate a difference value between the super-resolution predicted image and the second image through the target loss function; and performing the training processing according to the difference value to obtain the super-resolution image reconstruction model.

Further, in an embodiment of the present application, the second acquiring unit 14 is configured to, before reconstructing the image to be detected according to the super-resolution image reconstruction model to obtain a target high-resolution image corresponding to the image to be detected, perform image acquisition processing on the target object at the target position through the first image sensor to obtain a third image corresponding to the target object.

Further, in the embodiment of the present application, the second acquiring unit 14 is further configured to perform image acquisition processing on the target object at the target position through a second image sensor, so as to obtain a fourth image corresponding to the target object; wherein the resolution of the third image is less than the resolution of the fourth image.

Further, in the embodiment of the present application, the training unit 12 is further configured to perform training processing on the initial image reconstruction model based on the third image and the fourth image, so as to obtain the super-resolution image reconstruction model.

Further, in an embodiment of the present application, the second image, the third image, and the fourth image are YUV format image files.

In an embodiment of the present application, further, fig. 10 is a schematic diagram of a composition structure of the image reconstruction apparatus provided in the present application, as shown in fig. 10, the image reconstruction apparatus 10 provided in the embodiment of the present application may further include a processor 15 and a memory 16 storing instructions executable by the processor 15, and further, the image reconstruction apparatus 10 may further include a communication interface 17, and a bus 18 for connecting the processor 15, the memory 16, and the communication interface 17.

In an embodiment of the present Application, the Processor 15 may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a ProgRAMmable Logic Device (PLD), a Field ProgRAMmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that the electronic devices for implementing the above processor functions may be other devices, and the embodiments of the present application are not limited in particular. The pressure field computing device 10 may further comprise a memory 16, which memory 16 may be connected to the processor 15, wherein the memory 16 is configured to store executable program code comprising computer operating instructions, and wherein the memory 16 may comprise a high speed RAM memory and may further comprise a non-volatile memory, such as at least two disk memories.

In the embodiment of the present application, the bus 18 is used to connect the communication interface 17, the processor 15, and the memory 16 and the intercommunication among these devices.

In an embodiment of the present application, the memory 16 is used for storing instructions and data.

Further, in an embodiment of the present application, the processor 15 is configured to, in a first working mode, perform image acquisition processing on a target object at a target position through an image sensor, and obtain a first image corresponding to the target object; in a second working mode, image acquisition processing is carried out on the target object at the target position through the image sensor, and a second image corresponding to the target object is obtained; wherein the resolution of the first image is less than the resolution of the second image; training an initial image reconstruction model based on the first image and the second image to obtain a super-resolution image reconstruction model; and reconstructing the image to be detected according to the super-resolution image reconstruction model to obtain a target high-resolution image corresponding to the image to be detected.

In practical applications, the Memory 16 may be a volatile Memory (volatile Memory), such as a Random-Access Memory (RAM); or a non-volatile Memory (non-volatile Memory), such as a Read-Only Memory (ROM), a flash Memory (flash Memory), a Hard Disk (Hard Disk Drive, HDD) or a Solid-State Drive (SSD); or a combination of the above types of memories and provides instructions and data to the processor 15.

In addition, each functional module in this embodiment may be integrated into one recommendation unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware or a form of a software functional module.

Based on the understanding that the technical solution of the present embodiment essentially or a part contributing to the prior art, or all or part of the technical solution, may be embodied in the form of a software product stored in a storage medium, and include several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the method of the present embodiment. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The embodiment of the application provides an image reconstruction device, which can acquire and process a target object at a target position through an image sensor in a first working mode to obtain a first image corresponding to the target object; in a second working mode, acquiring and processing a target object at a target position through an image sensor to obtain a second image corresponding to the target object; wherein the resolution of the first image is less than the resolution of the second image; training the initial image reconstruction model based on the first image and the second image to obtain a super-resolution image reconstruction model; and reconstructing the image to be detected according to the super-resolution image reconstruction model to obtain a target high-resolution image corresponding to the image to be detected. That is to say, in the embodiment of the present application, the terminal performs image acquisition processing on the same target object at the same position through the same image sensor in different working modes to obtain a first image with low resolution and a second image with high resolution of the target object; and then based on a deep learning mode, performing model training processing by using the first image and the second image to obtain a super-resolution image reconstruction model capable of reconstructing a high-resolution image with higher definition from a low-resolution image. Therefore, in the application, the model training sample data acquired by the image reconstruction device is obtained by switching different camera image showing modes to acquire the same target object existing in the real world at the same shooting position and in the same shooting environment by the same image sensor, so that the quality of the training sample data is higher, the reconstruction effect of the super-resolution image reconstruction model acquired based on the training sample data is better, and the high-quality high-resolution image with higher definition can be acquired.

An embodiment of the present application provides a computer-readable storage medium, on which a program is stored, which when executed by a processor implements the image reconstruction method as described above.

Specifically, the program instructions corresponding to an image reconstruction method in the present embodiment may be stored on a storage medium such as an optical disc, a hard disc, a usb disk, or the like, and when the program instructions corresponding to an image reconstruction method in the storage medium are read or executed by an electronic device, the method includes the following steps:

in a first working mode, carrying out image acquisition processing on a target object at a target position through an image sensor to obtain a first image corresponding to the target object;

in a second working mode, image acquisition processing is carried out on the target object at the target position through the image sensor, and a second image corresponding to the target object is obtained; wherein the resolution of the first image is less than the resolution of the second image;

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of implementations of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart block or blocks and/or flowchart block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks in the flowchart and/or block diagram block or blocks. The above description is only a preferred embodiment of the present application, and is not intended to limit the scope of the present application.

Claims

1. A method of image reconstruction, the method comprising:

2. A further method according to claim 1, wherein the first operating mode is a pixel-by-pixel Binning mode; the second operating mode is a pixel rearrangement Remosaic mode.

3. The method of claim 1, wherein the super-resolution image reconstruction model adopts a U-shaped neural network structure.

4. The method according to claim 1 or 3, wherein the training of the initial image reconstruction model based on the first image and the second image to obtain the super-resolution image reconstruction model comprises:

aligning the first image and the second image to obtain an aligned first image and an aligned second image;

and training the initial image reconstruction model based on the aligned first image and the aligned second image to obtain the super-resolution image reconstruction model.

5. The method according to claim 1 or 3, wherein the training of the initial image reconstruction model based on the first image and the second image to obtain the super-resolution image reconstruction model comprises:

inputting the first image into the initial image reconstruction model, and outputting a super-resolution predicted image;

and performing the training processing based on a preset target loss function, the super-resolution prediction image and the second image to obtain a super-resolution image reconstruction model.

6. The method according to claim 5, wherein the performing the training process based on the preset target loss function, the super-resolution predicted image and the second image to obtain the super-resolution image reconstruction model comprises:

calculating a difference value between the super-resolution predicted image and the second image through the target loss function;

and performing the training processing according to the difference value to obtain the super-resolution image reconstruction model.

7. The method according to claim 1, wherein before the reconstructing the image to be measured according to the super-resolution image reconstruction model to obtain the target high-resolution image corresponding to the image to be measured, the method further comprises:

acquiring and processing an image of the target object at the target position through a first image sensor to obtain a third image corresponding to the target object;

acquiring and processing an image of the target object at the target position through a second image sensor to obtain a fourth image corresponding to the target object; wherein the resolution of the third image is less than the resolution of the fourth image;

and training the initial image reconstruction model based on the third image and the fourth image to obtain the super-resolution image reconstruction model.

8. The method of claim 7, wherein the first image, the second image, the third image, and the fourth image are color YUV format image files.

9. An image reconstruction apparatus characterized by comprising: a first acquisition unit, a training unit and a reconstruction unit,

10. An image reconstruction device comprising a processor, a memory having stored thereon instructions executable by the processor, the instructions when executed by the processor implementing the method of any one of claims 1-8.

11. A computer-readable storage medium, on which a program is stored, for use in an image reconstruction device, characterized in that the program, when being executed by a processor, is adapted to carry out the method of any one of claims 1 to 8.