CN111754406B - Image resolution processing method, device, equipment and readable storage medium - Google Patents
Image resolution processing method, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN111754406B CN111754406B CN202010578505.1A CN202010578505A CN111754406B CN 111754406 B CN111754406 B CN 111754406B CN 202010578505 A CN202010578505 A CN 202010578505A CN 111754406 B CN111754406 B CN 111754406B
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- low
- frequency residual
- downsampled
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 31
- 238000005070 sampling Methods 0.000 claims description 46
- 238000013527 convolutional neural network Methods 0.000 claims description 31
- 238000000926 separation method Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 2
- 238000000034 method Methods 0.000 abstract description 70
- 230000000007 visual effect Effects 0.000 abstract description 12
- 230000008569 process Effects 0.000 description 23
- 230000006870 function Effects 0.000 description 13
- 230000005540 biological transmission Effects 0.000 description 9
- 238000004891 communication Methods 0.000 description 6
- 230000004913 activation Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 3
- 238000000605 extraction Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001575 pathological effect Effects 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
- G06T3/4076—Super resolution, i.e. output image resolution higher than sensor resolution by iteratively correcting the provisional high resolution image using the original low-resolution image
Abstract
The application discloses an image resolution processing method, device, equipment and readable storage medium, wherein the method comprises the steps of obtaining a low-resolution image to be processed, wherein a high-frequency residual error is embedded in the low-resolution image; separating a high-frequency residual error and a downsampled image from the low-resolution image; upsampling the downsampled image to obtain a high resolution image; the high resolution image is combined with the high frequency residual to obtain a high resolution output image. And (3) separating a high-frequency residual error and a downsampled image from the low-resolution image, combining the high-resolution image obtained by upsampling the downsampled image with the high-frequency residual error to obtain a high-resolution output image, and introducing the high-frequency residual error to enable the upsampled image to recover missing detail information, so that the visual quality and fidelity of the image are higher.
Description
Technical Field
The present invention relates to the field of image processing, and in particular, to a method, an apparatus, a device, and a readable storage medium for processing image resolution.
Background
Image downsampling and upsampling are also commonly referred to as image downsampling and image super-resolution, and refer to the process of acquiring a low resolution image from a high resolution image and acquiring a high resolution image from a low resolution image, respectively.
In conventional image upsampling or super resolution techniques, an existing low resolution image is enlarged and reconstructed to obtain a high quality enlarged image. However, because the image upsampling needs to recover the high-frequency detail information which is already lost in the low-resolution image, which is a typical pathological problem, the current super-resolution algorithm cannot accurately recover the high-frequency detail lost in the image downsampling process, so that the effect of upsampling is poor due to the loss of the detail of the finally amplified recovered image.
Disclosure of Invention
The main purpose of the present application is to provide an image resolution processing method, apparatus, device and readable storage medium, which are aimed at solving the problems of low visual quality and fidelity of the restored image after the current upsampling process.
In order to achieve the above object, the present application provides an image resolution processing method, which includes the following steps:
acquiring a low-resolution image to be processed, wherein the low-resolution image is embedded with a high-frequency residual error;
separating a high-frequency residual error and a downsampled image from the low-resolution image;
upsampling the downsampled image to obtain a high resolution image;
the high resolution image is combined with the high frequency residual to obtain a high resolution output image.
Optionally, before the step of acquiring the low resolution image to be processed embedded with the high frequency residual, the method further includes:
acquiring an original high-resolution image;
downsampling the original high-resolution image to obtain a downsampled image;
performing up-sampling prediction processing on the down-sampled image to obtain a high-resolution restored image;
obtaining the high-frequency residual error according to the original high-resolution image and the high-resolution restored image;
embedding the high frequency residual into the downsampled image to obtain the low resolution image with the high frequency residual embedded therein.
Optionally, the step of downsampling the original high resolution image to obtain a downsampled image includes:
inputting the original high-resolution image into a downsampling network based on a convolutional neural network;
and according to a preset resolution value, downsampling the original high-resolution image into a downsampled image through the downsampling network.
Optionally, the step of embedding the high frequency residual into the downsampled image to obtain the low resolution image with the high frequency residual embedded therein comprises:
according to the downsampling multiple, obtaining a preset number of low-resolution high-frequency residuals consistent with the resolution of the downsampled image;
inputting the low-resolution high-frequency residual error and the downsampled image into a convolutional neural network;
and outputting the low-resolution image embedded with the high-frequency residual error through the convolution data network.
Optionally, the step of obtaining a preset number of low resolution high frequency residuals consistent with the resolution of the downsampled image according to the downsampling multiple includes:
if the downsampling multiple is an integer multiple, selecting a first preset number of initial pixel points in the high-frequency residual error according to the downsampling multiple, and selecting a second preset number of interval pixel points;
and forming the number of the interval pixel points into a third preset number of low-resolution high-frequency residual errors.
Optionally, the step of obtaining a preset number of low resolution high frequency residuals consistent with the resolution of the downsampled image according to the downsampling multiple includes:
if the downsampling multiple is a non-integer multiple, determining a near integer multiple matched with the downsampling multiple;
downsampling the high-frequency residual errors according to the approximate integer multiple to obtain a fourth preset number of candidate low-resolution high-frequency residual errors;
bilinear interpolation is performed on each of the candidate low resolution high frequency residuals to obtain a low resolution high frequency residual consistent with the resolution of the downsampled image.
Optionally, the step of performing upsampling processing on the downsampled image to obtain a high resolution image includes:
inputting the downsampled image into an upsampling network based on a convolutional neural network;
the downsampled image is processed according to the upsampling network to obtain a high resolution image.
The application also provides an image resolution processing apparatus, including:
the acquisition module is used for acquiring a low-resolution image to be processed, wherein the low-resolution image is embedded with a high-frequency residual error;
the separation module is used for separating a high-frequency residual error and a downsampled image from the low-resolution image;
the up-sampling module is used for up-sampling the down-sampled image to acquire a high-resolution image;
and the combining module is used for combining the high-resolution image with the high-frequency residual error to acquire a high-resolution output image.
The present application also provides an image resolution processing apparatus including: the image resolution processing method comprises the steps of a memory, a processor and an image resolution processing program which is stored in the memory and can run on the processor, wherein the image resolution processing program is executed by the processor to realize the image resolution processing method.
The present application also provides a readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the image resolution processing method as described above.
The method comprises the steps of obtaining a low-resolution image to be processed, wherein a high-frequency residual error is embedded in the low-resolution image; separating a high-frequency residual error and a downsampled image from the low-resolution image; upsampling the downsampled image to obtain a high resolution image; the high resolution image is combined with the high frequency residual to obtain a high resolution output image. And (3) separating a high-frequency residual error and a downsampled image from the low-resolution image, combining the high-resolution image obtained by upsampling the downsampled image with the high-frequency residual error to obtain a high-resolution output image, and introducing the high-frequency residual error to enable the upsampled image to recover missing detail information, so that the visual quality and fidelity of the image are higher.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.
FIG. 1 is a schematic diagram of a device structure of a hardware operating environment according to an embodiment of the present application;
FIG. 2 is a flowchart of a first embodiment of an image resolution processing method according to the present application;
FIG. 3 is a flowchart illustrating steps before step S10 in FIG. 2 in a second embodiment of an image resolution processing method according to the present application;
FIG. 4 is a flowchart for refining step S15 of FIG. 3 in a fourth embodiment of the image resolution processing method of the present application;
fig. 5 is a schematic system architecture diagram of an embodiment of an image resolution processing apparatus according to the present application.
The realization, functional characteristics and advantages of the present application will be further described with reference to the embodiments, referring to the attached drawings.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
In the following description, suffixes such as "module", "component", or "unit" for representing elements are used only for facilitating the description of the present invention, and have no specific meaning per se. Thus, "module," "component," or "unit" may be used in combination.
As shown in fig. 1, fig. 1 is a schematic diagram of a terminal structure of a hardware running environment according to an embodiment of the present application.
The terminal in the embodiment of the application is an image resolution processing device.
As shown in fig. 1, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
Optionally, the terminal may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and so on. Among other sensors, such as light sensors, motion sensors, and other sensors. In particular, the light sensor may comprise an ambient light sensor, which may adjust the brightness of the display screen according to the brightness of ambient light, and a proximity sensor, which may turn off the display screen and/or the backlight when the terminal device is moved to the ear. Of course, the terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and the like, which are not described herein.
It will be appreciated by those skilled in the art that the terminal structure shown in fig. 1 is not limiting of the terminal and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
As shown in fig. 1, an operating system, a network communication module, a user interface module, and an image resolution processing program may be included in the memory 1005 as one type of computer storage medium.
In the terminal shown in fig. 1, the network interface 1004 is mainly used for connecting to a background server and performing data communication with the background server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be configured to call an image resolution processing program stored in the memory 1005 and perform the following operations:
acquiring a low-resolution image to be processed, wherein the low-resolution image is embedded with a high-frequency residual error;
separating a high-frequency residual error and a downsampled image from the low-resolution image;
upsampling the downsampled image to obtain a high resolution image;
the high resolution image is combined with the high frequency residual to obtain a high resolution output image.
Based on the above-mentioned terminal hardware structure, various embodiments of the present application are presented.
The application provides an image resolution processing method.
Referring to fig. 2, in a first embodiment of an image resolution processing method, the method includes:
step S10, obtaining a low-resolution image to be processed, wherein the low-resolution image is embedded with a high-frequency residual error;
image downsampling and upsampling are also commonly referred to as image downsampling and image super-resolution, and refer to the process of acquiring a low resolution image from a high resolution image and acquiring a high resolution image from a low resolution image, respectively. The high resolution image cannot be used due to hardware conditions of the device or transmission conditions of file transmission, or based on an active selection of the user, and needs to be processed into a low resolution image through downsampling, but then the low resolution image may need to be resampled into the high resolution image. Particularly, in order to reduce the size of a transmission file in the image transmission process, a high-resolution image is changed into a low-resolution image through downsampling and is transmitted to a receiving end, and the receiving end needs to amplify and reconstruct the transmitted low-resolution image through an upsampling method after receiving the transmitted low-resolution image, but the high-frequency detail information which is lost in the low-resolution image is difficult to recover, so that the visual quality and fidelity obtained through upsampling are low at present. Meanwhile, the current up-sampling method is usually aimed at a fixed or simple down-sampling model, and after the down-sampling model is replaced, the performance of the corresponding up-sampling model is greatly reduced. In the application, when the low-resolution image is required to be restored into the high-resolution image by an up-sampling method through the low-resolution image embedded with the high-frequency residual, the high-frequency residual can supplement high-frequency details missing in the low-resolution image. The high-frequency residual is embedded in the low-resolution image, so that the high-frequency residual does not need to be additionally transmitted, and the visual effect of the low-resolution image is not influenced. The high-frequency residual error is obtained by carrying out one-time up-sampling prediction on the low-resolution image after the initial high-resolution image is subjected to down-sampling based on a prediction mechanism, and comparing the obtained corresponding restored high-resolution image with the initial high-resolution image. The downsampling in the application can be a traditional downsampling method such as extracting pixel points in an image, or can be a downsampling network based on a convolutional neural network, and likewise, the upsampling method can also be a traditional upsampling method such as an interpolation method, or can also be an upsampling network based on the convolutional neural network. The downsampling method and the upsampling method adopted in the present application are not limited, and of course, it is preferable to use a downsampling network and an upsampling network based on a convolutional neural network.
Step S20, separating a high-frequency residual error and a downsampled image from the low-resolution image;
the low resolution image includes a downsampled image and a high frequency residual, and in the upsampling process, only the downsampled image is upsampled, so the high frequency residual in the low resolution image and the sampled image are separated. The method for extracting the high-frequency residual error in the low-resolution image is based on a convolutional neural network, the neural network comprises an input layer, a plurality of convolutional layers and residual error connection from the input layer to the output of the convolutional layers, specifically, the input layer is a 3*3-sized convolutional layer, the input layer converts the low-resolution image into 64 feature maps, each convolutional layer is a 3*3-sized convolutional layer, a linear rectification function is used as an activation function, the number of feature channels is also 64, and the convolutional layers are used for feature extraction and reconstruction. The residual connection directly transmits the input image with the first resolution to the output position of the convolution layer, the output layer is a double-branch output layer, one branch is two convolution layers with the size of 3*3, the other branch sequentially comprises a convolution layer with the size of 3*3, an up-sampling layer and a convolution layer with the size of the other, and the two channels reconstruct the characteristic diagram into a down-sampling image and a high-frequency residual error respectively.
Step S30, up-sampling the down-sampled image to obtain a high-resolution image;
and for the downsampled image separated from the low-resolution image, performing upsampling processing on the downsampled image by an upsampling method to obtain the high-resolution image. The up-sampling method can be an interpolation method or an up-sampling network method based on a convolutional neural network. The interpolation method is simpler to implement, but the result obtained by the up-sampling network method based on the convolutional neural network is more accurate, so the up-sampling network method based on the convolutional neural network is preferable in the application.
Step S40, combining the high-resolution image with the high-frequency residual to obtain a high-resolution output image;
finally, the high-resolution image obtained by the up-sampling processing is combined with the high-frequency residual error to obtain a high-resolution output image, the high-frequency details lost by the down-sampling processing are absent in the high-resolution image obtained by the up-sampling processing, and the high-frequency residual error contains corresponding details, so that the finally obtained high-resolution image has corresponding details, better visual effect and higher fidelity.
In the embodiment, a low-resolution image with a high-frequency residual error embedded therein to be processed is acquired; separating a high-frequency residual error and a downsampled image from the low-resolution image; upsampling the downsampled image to obtain a high resolution image; the high resolution image is combined with the high frequency residual to obtain a high resolution output image. And (3) separating a high-frequency residual error and a downsampled image from the low-resolution image, combining the high-resolution image obtained by upsampling the downsampled image with the high-frequency residual error to obtain a high-resolution output image, and introducing the high-frequency residual error to enable the upsampled image to recover missing detail information, so that the visual quality and fidelity of the image are higher.
Further, referring to fig. 2 and 3, on the basis of the first embodiment of the image resolution processing method of the present application, there is provided a second embodiment of the image resolution processing method which, in the second embodiment,
before step S10, the method further includes:
step S11, acquiring an original high-resolution image;
in practical use of the technical scheme in the application, it is preferable to cooperatively learn the downsampling process and the upsampling process, combine the downsampling process and the upsampling process to perform resolution processing on the image, acquire a low-resolution image for a high-resolution image by a downsampling method and acquire a high-frequency residual error based on a prediction mechanism, embed the high-frequency residual error into the low-resolution image, and then restore the low-resolution image embedded with the high-frequency residual error into the high-resolution image by an upsampling method. Thus, in this application, the downsampling and upsampling processes constitute a complete image resolution process. The original high-resolution image is acquired, and the original high-resolution image can be a high-resolution image required to be transmitted or can be other high-resolution images. The original high resolution image contains all the due image information, especially detail information.
Step S12, carrying out downsampling processing on the original high-resolution image to obtain a downsampled image;
and carrying out downsampling processing on the original high-resolution image to obtain a downsampled image. The downsampling method may be a conventional downsampling method, for example, a method of combining a plurality of pixels in an area into one pixel by a weighted average method to reduce the resolution of an image, or selecting a specific pixel among all pixels of the image according to a certain rule to obtain a downsampled image with low resolution. Meanwhile, the downsampling method can also be a downsampling network based on a convolutional neural network. It is preferred in this application to take a downsampling network.
Step S13, up-sampling prediction processing is carried out on the down-sampling image so as to obtain a high-resolution restored image;
the upsampling prediction for the downsampled image is either a super-resolution reconstruction or an upsampling process, i.e. a process of restoring the low resolution image to a high resolution image. The upsampling process may be performed by conventional upsampling methods such as interpolation, or may be performed by upsampling networks based on convolutional neural networks. When an upsampling network is employed, the sub-upsampling network is the same as in the first embodiment. However, the up-sampling method is not improved, so that the high-resolution image obtained by the up-sampling method still has the problem of corresponding detail loss.
Step S14, obtaining the high-frequency residual error according to the original high-resolution image and the high-resolution restored image;
and comparing the high-resolution restored image obtained through up-sampling with the original high-resolution image to obtain a high-frequency residual, wherein the high-frequency residual is the detail information which cannot be restored in the up-sampling process, and the missing detail information can be supplemented into the image again through the high-frequency residual.
Step S15, embedding the high-frequency residual into the downsampled image to obtain the low-resolution image embedded with the high-frequency residual;
for the high-frequency residual and the downsampled image, because the space information redundancy and the information entropy redundancy exist in the downsampled image, the high-frequency residual can be embedded into the downsampled image without increasing the storage consumption and the transmission consumption of the image in the transmission process. Meanwhile, a convolutional neural network method is generally adopted for embedding the high-frequency residual error into the downsampled image.
The steps in the first embodiment and the steps in the second embodiment can be combined to form an optimal complete scheme in the application, namely, a downsampling method and an upsampling method are combined, downsampling is performed on a high-resolution image to obtain a low-resolution image, then upsampling prediction is performed to obtain a high-frequency residual, the low-resolution image is embedded into the high-frequency residual for transmission and storage, when the resolution of the low-resolution image needs to be amplified and reconstructed, the high-frequency residual can be utilized for restoring the missing high-frequency details after upsampling reduction, the visual quality and fidelity of the finally obtained high-resolution image are better, meanwhile, the selection of the downsampling method and the upsampling method is not limited, and only the upsampling prediction needs to be increased to obtain the high-frequency residual and the high-frequency residual is embedded into the low-resolution image.
In this embodiment, the high-frequency residual is obtained by upsampling prediction and embedded in the image, so as to provide residual information for obtaining a high-quality high-resolution image later, without increasing consumption in the image storage and transmission process.
Further, on the basis of the above-described embodiments of the image resolution processing method of the present application, there is provided a third embodiment of the image resolution processing method, in the third embodiment,
step S12 includes:
step A1, inputting the original high-resolution image into a downsampling network based on a convolutional neural network;
step A2, according to a preset resolution value, downsampling the original high-resolution image into a downsampled image through the downsampling network;
in this embodiment of the present application, downsampling is performed by a downsampling network based on a convolutional neural network, which in turn comprises an input layer, a number of convolutional layers, a downsampling layer, a number of convolutional layers, a residual connection from the input layer back to the convolutional layers back, and an output layer. The input layer is a 3*3 convolution layer used for converting the original high-resolution image into 64 feature images; the convolution layer before the downsampling layer is a convolution layer with the size of 3*3, a linear rectification function is used as an activation function, and the number of characteristic channels is 64 and is used for extracting and reconstructing the characteristics; the downsampling layer is a bilinear interpolation layer and is used for reducing the resolution of the feature map to a specified size; the convolution layer after the downsampling layer is a convolution layer with the size of 3*3, a linear rectification function is also used as an activation function, and the number of characteristic channels is also 64, so that the feature extraction and reconstruction are further carried out on the feature image after the dimension reduction; the residual connection comprises a downsampling layer and a convolution layer with the size of 3*3, the downsampling layer in the residual connection is also a bilinear interpolation layer, the activation function in the residual connection is also a linear rectification function, the number of characteristic channels is 64, and the residual connection is used for ensuring the accuracy of a final output result after the input images are transmitted to the convolution layer; the output layer is also a 3*3-sized convolutional layer for reconstructing the 64 feature maps into a downsampled image. In the down-sampling process, down-sampling is performed through a down-sampling network according to a preset resolution value.
In this embodiment, the downsampling network downsamples the original high-resolution image, so that the acquired downsampled image has better visual quality.
Further, referring to fig. 2 and 4, on the basis of the above-described embodiments of the image resolution processing method of the present application, there is provided a fourth embodiment of the image resolution processing method, in the fourth embodiment,
step S15 includes:
step S151, obtaining a preset number of low-resolution high-frequency residuals consistent with the resolution of the downsampled image according to the downsampled multiple;
according to the resolution ratio of the downsampled image and the original high-resolution image, downsampling multiple can be obtained, if the resolution ratio of the original high-resolution image is 1920 x 1080, the resolution ratio of the downsampled image is 960 x 540, the downsampling multiple is 2 x 2, according to the downsampling multiple, the high-frequency residual is changed into a low-resolution high-frequency residual which is the same as the resolution ratio of the downsampled image, and meanwhile, the number of the acquired low-resolution high-frequency residual is more than one so as to ensure that the complete high-frequency residual can be reserved, and meanwhile, the resolution ratio of the high-frequency residual is consistent with that of the downsampled image so as to ensure that the visual quality of the downsampled image is not influenced after the downsampling image is embedded.
Step S152, inputting the low-resolution high-frequency residual error and the downsampled image into a convolutional neural network;
step S153, outputting the low-resolution image embedded with the high-frequency residual error through the convolution data network;
the convolutional neural network for embedding the high-frequency residual into the downsampled image comprises an input layer, a plurality of convolutional layers, a residual connection from the input layer to the output of the convolutional layers and an output layer. The input layer is a convolution layer with a size of 3*3 and is used for converting the downsampled image and the low-resolution high-frequency residual image into 64 characteristic images; the convolution layer is a 3*3-sized convolution layer, and takes a linear rectification function as an activation function for feature extraction and reconstruction; the output layer is a 3*3-sized convolutional layer used to reconstruct the 64 feature maps into a low resolution image with embedded high frequency residuals.
In this embodiment, the high-frequency residual information is embedded into the downsampled image through the convolutional neural network, so that the prompt that the downsampled image contains the high-frequency residual information does not increase extra storage and transmission consumption, and the visual effect of the downsampled image is not affected.
Further, on the basis of the above-described embodiments of the image resolution processing method of the present application, there is provided a fifth embodiment of the image resolution processing method, in the fifth embodiment,
step S151 includes:
step B1, if the downsampling multiple is an integer multiple, selecting a first preset number of initial pixel points in the high-frequency residual error according to the downsampling multiple, and selecting a second preset number of interval pixel points;
step B2, the number of the interval pixel points is formed into a third preset number of low-resolution high-frequency residual errors;
when the downsampling multiple is an integer multiple, a plurality of low-resolution high-frequency residuals are obtained by adopting a sampling method with corresponding pixel points at intervals, and the resolution of each low-resolution residual is consistent with that of the downsampled image. If the downsampled image is reduced by two times in the horizontal direction of the original high-resolution image, a low-resolution high-frequency residual is obtained from the first pixel point at the upper left corner by sampling every 1 pixel point, another low-resolution high-frequency residual is obtained from the second pixel point at the upper left corner by sampling every 1 point, and thus a preset number of low-resolution high-frequency residuals are obtained, wherein the preset number is related to the downsampling multiple, and the like for other downsampling integer multiples.
Wherein in another possible scenario, step S151 comprises:
step B3, if the downsampling multiple is a non-integer multiple, determining a near integer multiple matched with the downsampling multiple;
step B4, according to the approximate integer multiple, downsampling the high-frequency residual errors to obtain a fourth preset number of candidate low-resolution high-frequency residual errors;
step B5, bilinear interpolation is carried out on each candidate low-resolution high-frequency residual error so as to obtain a low-resolution high-frequency residual error consistent with the resolution of the downsampled image;
if the downsampling multiple is not the integer multiple, firstly determining the matched approximate integer multiple, if the downsampling multiple is 2.1 times, then the approximate integer multiple is 2 times, firstly acquiring candidate low-resolution high-frequency residual errors according to the approximate integer multiple according to the method in the steps B1 to B2, and then using bilinear interpolation for each candidate low-resolution high-frequency residual error to ensure that the resolution of the final low-resolution high-frequency residual error is consistent with the resolution of the downsampled image.
In this embodiment, the sampling method that the high-frequency residual is changed to the low resolution is determined according to the downsampling multiple, so as to ensure that the resolution of the high-frequency residual is consistent with that of the downsampled image.
Further, on the basis of the above-described embodiments of the image resolution processing method of the present application, there is provided a sixth embodiment of the image resolution processing method, in the sixth embodiment,
step S30 includes:
step C1, inputting the downsampled image into an upsampling network based on a convolutional neural network;
step C2, processing the downsampled image according to the upsampling network to obtain a high-resolution image;
in this embodiment, an upsampling network based on convolutional neural network is used for the upsampling process of the downsampled image. The up-sampling network sequentially comprises an up-sampling layer, an input layer, a plurality of residual blocks, a residual connection from the back of the input layer to the back of the output of the residual blocks and an output layer. The upsampling layer is a bilinear interpolation layer and is used for interpolating and amplifying the downsampled image to the size of the appointed resolution; the input layer is a 3*3 convolution layer used for converting the interpolated image into 64 feature images; each residual block comprises a 64-channel 3*3 convolution layer, a linear rectification function, another 64-channel 3*3 convolution layer and a residual connection connecting the two ends; the output layer is a 3*3 sized convolutional layer for reconstructing the 64 feature maps into a high resolution image. The upsampling network in this embodiment may also be used for upsampling prediction in the second embodiment.
When a convolutional neural network-based downsampling network and an upsampling network are used, the upsampling network and the downsampling network may be trained. The training of the up-sampling network and the down-sampling network can be pre-independent training or can be end-to-end training combined into the whole network. In the training process, a high-quality high-resolution image is used as a training set, meanwhile, the constraint condition in the training process is that a finally obtained high-resolution output image is consistent with an original high-resolution image, and meanwhile, a low-resolution image embedded with a high-frequency residual error is visually consistent with an original downsampled image, namely, the basic content of an original image cannot be changed due to the embedded residual error component. The loss function L for the training process may be:
L＝L 1 (X OUT ,X)+λL SSIM (Y*,Y)
wherein L is 1 For L1 loss, L SSIM The SSIM losses are common loss functions in both image enhancement and reconstruction problems. X is X OUT For a high resolution output image, X is the original high resolution image, Y is the low resolution image embedded with the high frequency residual, and Y is the downsampled image.
In the embodiment, the resolution of the downsampled image is amplified and reconstructed through the upsampling network based on the convolutional neural network, so that the acquired high-resolution image has better visual quality.
In addition, referring to fig. 5, an embodiment of the present application further proposes an image resolution processing apparatus, including:
the first acquisition module is used for acquiring a low-resolution image to be processed, wherein the low-resolution image is embedded with a high-frequency residual error;
the separation module is used for separating a high-frequency residual error and a downsampled image from the low-resolution image;
the up-sampling module is used for up-sampling the down-sampled image to obtain a high-resolution image;
and the combining module is used for combining the high-resolution image with the high-frequency residual error to acquire a high-resolution output image.
Optionally, the image resolution processing apparatus further includes:
the second acquisition module is used for acquiring an original high-resolution image;
the downsampling module is used for downsampling the original high-resolution image to obtain a downsampled image;
the prediction module is used for carrying out up-sampling prediction processing on the down-sampling image so as to obtain a high-resolution restored image;
the residual error acquisition module is used for acquiring the high-frequency residual error according to the original high-resolution image and the high-resolution restored image;
and the embedded module is used for embedding the high-frequency residual error into the downsampled image to acquire the low-resolution image embedded with the high-frequency residual error.
Optionally, the downsampling module is further configured to:
inputting the original high-resolution image into a downsampling network based on a convolutional neural network;
and according to a preset resolution value, downsampling the original high-resolution image into a downsampled image through the downsampling network.
Optionally, the embedded module is further configured to:
according to the downsampling multiple, obtaining a preset number of low-resolution high-frequency residuals consistent with the resolution of the downsampled image;
inputting the low-resolution high-frequency residual error and the downsampled image into a convolutional neural network;
and outputting the low-resolution image embedded with the high-frequency residual error through the convolution data network.
Optionally, the image resolution processing apparatus further includes:
the sampling module is used for selecting a first preset number of initial pixel points in the high-frequency residual error according to the downsampling multiple if the downsampling multiple is an integer multiple, and selecting a second preset number of interval pixel points;
and forming the number of the interval pixel points into a third preset number of low-resolution high-frequency residual errors.
Optionally, the sampling module is further configured to:
if the downsampling multiple is a non-integer multiple, determining a near integer multiple matched with the downsampling multiple;
downsampling the high-frequency residual errors according to the approximate integer multiple to obtain a fourth preset number of candidate low-resolution high-frequency residual errors;
bilinear interpolation is performed on each of the candidate low resolution high frequency residuals to obtain a low resolution high frequency residual consistent with the resolution of the downsampled image.
Optionally, the upsampling module is further configured to:
inputting the downsampled image into an upsampling network based on a convolutional neural network;
the downsampled image is processed according to the upsampling network to obtain a high resolution image.
The expansion content of the specific embodiments of the apparatus and the readable storage medium (i.e., the computer readable storage medium) of the present application is substantially the same as the embodiments of the image resolution processing method described above, and will not be described herein.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present invention and the scope of the claims, which are to be protected by the present invention.
Claims (7)
1. An image resolution processing method, characterized in that the image resolution processing method comprises the steps of:
acquiring a low-resolution image to be processed, wherein the low-resolution image is embedded with a high-frequency residual error;
separating a high-frequency residual error and a downsampled image from the low-resolution image;
upsampling the downsampled image to obtain a high resolution image;
combining the high resolution image with the high frequency residual to obtain a high resolution output image;
before the step of obtaining the low-resolution image to be processed, the low-resolution image embedded with the high-frequency residual error further comprises the following steps:
acquiring an original high-resolution image;
downsampling the original high-resolution image to obtain a downsampled image;
performing up-sampling prediction processing on the down-sampled image to obtain a high-resolution restored image;
obtaining the high-frequency residual error according to the original high-resolution image and the high-resolution restored image;
embedding the high frequency residual into the downsampled image to obtain the low resolution image with the high frequency residual embedded therein;
wherein the step of embedding the high frequency residual into the downsampled image to obtain the low resolution image with the high frequency residual embedded therein comprises:
obtaining a downsampling multiple according to the downsampled image and the original high-resolution image;
according to the downsampling multiple, a preset number of low-resolution high-frequency residuals which are consistent with the resolution of the downsampled image are obtained;
inputting the low-resolution high-frequency residual error and the downsampled image into a convolutional neural network;
outputting the low-resolution image embedded with the high-frequency residual error through the convolutional neural network;
wherein, the step of obtaining a preset number of low resolution high frequency residuals consistent with the resolution of the downsampled image according to the downsampling multiple includes:
if the downsampling multiple is an integer multiple, selecting a first preset number of initial pixel points in the high-frequency residual error according to the downsampling multiple, and selecting a second preset number of interval pixel points;
and forming the number of the interval pixel points into a third preset number of low-resolution high-frequency residual errors.
2. The image resolution processing method according to claim 1, wherein the step of downsampling the original high-resolution image to obtain a downsampled image comprises:
inputting the original high-resolution image into a downsampling network based on a convolutional neural network;
and according to a preset resolution value, downsampling the original high-resolution image into a downsampled image through the downsampling network.
3. The image resolution processing method according to claim 1, wherein the step of acquiring a preset number of low resolution high frequency residuals in accordance with the resolution of the downsampled image based on the downsampling multiple includes:
if the downsampling multiple is a non-integer multiple, determining a near integer multiple matched with the downsampling multiple;
downsampling the high-frequency residual errors according to the approximate integer multiple to obtain a fourth preset number of candidate low-resolution high-frequency residual errors;
bilinear interpolation is performed on each of the candidate low resolution high frequency residuals to obtain a low resolution high frequency residual consistent with the resolution of the downsampled image.
4. The image resolution processing method according to claim 1, wherein the step of upsampling the downsampled image to obtain a high resolution image comprises:
inputting the downsampled image into an upsampling network based on a convolutional neural network;
the downsampled image is processed according to the upsampling network to obtain a high resolution image.
5. An image resolution processing apparatus, characterized in that the image resolution processing apparatus comprises:
the acquisition module is used for acquiring a low-resolution image to be processed, wherein the low-resolution image is embedded with a high-frequency residual error;
the separation module is used for separating a high-frequency residual error and a downsampled image from the low-resolution image;
the up-sampling module is used for up-sampling the down-sampled image to obtain a high-resolution image;
a combining module for combining the high resolution image with the high frequency residual to obtain a high resolution output image;
the acquisition module is further configured to acquire an original high-resolution image, perform downsampling processing on the original high-resolution image to acquire a downsampled image, perform upsampling prediction processing on the downsampled image to acquire a high-resolution restored image, obtain the high-frequency residual error according to the original high-resolution image and the high-resolution restored image, and embed the high-frequency residual error into the downsampled image to acquire the low-resolution image embedded with the high-frequency residual error;
the acquisition module is further configured to obtain a downsampling multiple according to the downsampled image and the original high-resolution image, acquire a preset number of low-resolution high-frequency residuals consistent with the resolution of the downsampled image according to the downsampling multiple, input the low-resolution high-frequency residuals and the downsampled image into a convolutional neural network, and output the low-resolution image embedded with the high-frequency residuals through the convolutional neural network;
the acquisition module is further configured to select a first preset number of starting pixel points from the high-frequency residual according to the downsampling multiple if the downsampling multiple is an integer multiple, and select a second preset number of interval pixel points; and forming the number of the interval pixel points into a third preset number of low-resolution high-frequency residual errors.
6. An image resolution processing apparatus, characterized in that the image resolution processing apparatus comprises: memory, a processor and an image resolution processing program stored on the memory and executable on the processor, which when executed by the processor, implements the steps of the image resolution processing method according to any one of claims 1 to 4.
7. A readable storage medium, characterized in that the readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the image resolution processing method according to any one of claims 1 to 4.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010578505.1A CN111754406B (en) | 2020-06-22 | 2020-06-22 | Image resolution processing method, device, equipment and readable storage medium |
| PCT/CN2020/111777 WO2021258530A1 (en) | 2020-06-22 | 2020-08-27 | Image resolution processing method, device, apparatus, and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010578505.1A CN111754406B (en) | 2020-06-22 | 2020-06-22 | Image resolution processing method, device, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111754406A CN111754406A (en) | 2020-10-09 |
| CN111754406B true CN111754406B (en) | 2024-02-23 |
Family
ID=72676553
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010578505.1A Active CN111754406B (en) | 2020-06-22 | 2020-06-22 | Image resolution processing method, device, equipment and readable storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111754406B (en) |
| WO (1) | WO2021258530A1 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113313691A (en) * | 2021-06-03 | 2021-08-27 | 上海市第一人民医院 | Thyroid color Doppler ultrasound processing method based on deep learning |
| CN114331849B (en) * | 2022-03-15 | 2022-06-10 | 之江实验室 | Cross-mode nuclear magnetic resonance hyper-resolution network and image super-resolution method |
| CN114742738A (en) * | 2022-04-11 | 2022-07-12 | 北京字节跳动网络技术有限公司 | Image processing method, image processing device, storage medium and electronic equipment |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103607591A (en) * | 2013-10-28 | 2014-02-26 | 四川大学 | Image compression method combining super-resolution reconstruction |
| WO2015180053A1 (en) * | 2014-05-28 | 2015-12-03 | 北京大学深圳研究生院 | Method and apparatus for rapidly reconstructing super-resolution image |
| CN107018422A (en) * | 2017-04-27 | 2017-08-04 | 四川大学 | Still image compression method based on depth convolutional neural networks |
| CN107181949A (en) * | 2017-06-23 | 2017-09-19 | 四川大学 | A kind of compression of images framework of combination super-resolution and residual coding technology |
| CN108734660A (en) * | 2018-05-25 | 2018-11-02 | 上海通途半导体科技有限公司 | A kind of image super-resolution rebuilding method and device based on deep learning |
| CN109525859A (en) * | 2018-10-10 | 2019-03-26 | 腾讯科技（深圳）有限公司 | Model training, image transmission, image processing method and relevant apparatus equipment |
| WO2019153671A1 (en) * | 2018-02-11 | 2019-08-15 | 深圳创维－Rgb电子有限公司 | Image super-resolution method and apparatus, and computer readable storage medium |
| WO2019192588A1 (en) * | 2018-04-04 | 2019-10-10 | 华为技术有限公司 | Image super resolution method and device |
| CN110636289A (en) * | 2019-09-27 | 2019-12-31 | 北京金山云网络技术有限公司 | Image data transmission method, system, device, electronic equipment and storage medium |
| CN111179177A (en) * | 2019-12-31 | 2020-05-19 | 深圳市联合视觉创新科技有限公司 | Image reconstruction model training method, image reconstruction method, device and medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101320423A (en) * | 2008-06-26 | 2008-12-10 | 复旦大学 | Low resolution gait recognition method based on high-frequency super-resolution |
| CN102915527A (en) * | 2012-10-15 | 2013-02-06 | 中山大学 | Face image super-resolution reconstruction method based on morphological component analysis |
| CN103116880A (en) * | 2013-01-16 | 2013-05-22 | 杭州电子科技大学 | Image super resolution rebuilding method based on sparse representation and various residual |
| CN106981046B (en) * | 2017-03-21 | 2019-10-11 | 四川大学 | Single image super resolution ratio reconstruction method based on multi-gradient constrained regression |
| CN107358575A (en) * | 2017-06-08 | 2017-11-17 | 清华大学 | A kind of single image super resolution ratio reconstruction method based on depth residual error network |
| EP3567549A1 (en) * | 2018-05-07 | 2019-11-13 | Technische Universität München | Depth super-resolution from shading |
-
2020
- 2020-06-22 CN CN202010578505.1A patent/CN111754406B/en active Active
- 2020-08-27 WO PCT/CN2020/111777 patent/WO2021258530A1/en active Application Filing
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103607591A (en) * | 2013-10-28 | 2014-02-26 | 四川大学 | Image compression method combining super-resolution reconstruction |
| WO2015180053A1 (en) * | 2014-05-28 | 2015-12-03 | 北京大学深圳研究生院 | Method and apparatus for rapidly reconstructing super-resolution image |
| CN107018422A (en) * | 2017-04-27 | 2017-08-04 | 四川大学 | Still image compression method based on depth convolutional neural networks |
| CN107181949A (en) * | 2017-06-23 | 2017-09-19 | 四川大学 | A kind of compression of images framework of combination super-resolution and residual coding technology |
| WO2019153671A1 (en) * | 2018-02-11 | 2019-08-15 | 深圳创维－Rgb电子有限公司 | Image super-resolution method and apparatus, and computer readable storage medium |
| WO2019192588A1 (en) * | 2018-04-04 | 2019-10-10 | 华为技术有限公司 | Image super resolution method and device |
| CN108734660A (en) * | 2018-05-25 | 2018-11-02 | 上海通途半导体科技有限公司 | A kind of image super-resolution rebuilding method and device based on deep learning |
| CN109525859A (en) * | 2018-10-10 | 2019-03-26 | 腾讯科技（深圳）有限公司 | Model training, image transmission, image processing method and relevant apparatus equipment |
| CN110636289A (en) * | 2019-09-27 | 2019-12-31 | 北京金山云网络技术有限公司 | Image data transmission method, system, device, electronic equipment and storage medium |
| CN111179177A (en) * | 2019-12-31 | 2020-05-19 | 深圳市联合视觉创新科技有限公司 | Image reconstruction model training method, image reconstruction method, device and medium |
Non-Patent Citations (1)
| Title |
|---|
| 基于残差字典的超分辨率图像重构;李剑飞;《中国优秀硕士学位论文全文数据库 (信息科技辑)》;第I138-963页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111754406A (en) | 2020-10-09 |
| WO2021258530A1 (en) | 2021-12-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11354785B2 (en) | Image processing method and device, storage medium and electronic device | |
| CN111754406B (en) | Image resolution processing method, device, equipment and readable storage medium | |
| CN110136066B (en) | Video-oriented super-resolution method, device, equipment and storage medium | |
| CN108492249B (en) | Single-frame super-resolution reconstruction method based on small convolution recurrent neural network | |
| US11538136B2 (en) | System and method to process images of a video stream | |
| CN111784570A (en) | Video image super-resolution reconstruction method and device | |
| CN111784571A (en) | Method and device for improving image resolution | |
| CN111447359B (en) | Digital zoom method, system, electronic device, medium, and digital imaging device | |
| CN107220934B (en) | Image reconstruction method and device | |
| CN111951164A (en) | Image super-resolution reconstruction network structure and image reconstruction effect analysis method | |
| CN114298900A (en) | Image super-resolution method and electronic equipment | |
| CN111738951A (en) | Image processing method and device | |
| CN111161386A (en) | Ultrasonic image rendering method and device and ultrasonic equipment | |
| CN113628115A (en) | Image reconstruction processing method and device, electronic equipment and storage medium | |
| WO2024032331A9 (en) | Image processing method and apparatus, electronic device, and storage medium | |
| CN110677676B (en) | Video encoding method and apparatus, video decoding method and apparatus, and storage medium | |
| CN114494022B (en) | Model training method, super-resolution reconstruction method, device, equipment and medium | |
| US20110317773A1 (en) | Method for downsampling images | |
| CN114742738A (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| CN114266697A (en) | Image processing and model training method and device, electronic equipment and storage medium | |
| CN114240750A (en) | Video resolution improving method and device, storage medium and electronic equipment | |
| CN115272667A (en) | Farmland image segmentation model training method and device, electronic equipment and medium | |
| CN115272132A (en) | Image processing method, device, equipment and storage medium | |
| CN112419146A (en) | Image processing method and device and terminal equipment | |
| CN111738958B (en) | Picture restoration method and device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |