CN111754406A - Image resolution processing method, device and equipment and readable storage medium - Google Patents
Image resolution processing method, device and equipment and readable storage medium Download PDFInfo
- Publication number
- CN111754406A CN111754406A CN202010578505.1A CN202010578505A CN111754406A CN 111754406 A CN111754406 A CN 111754406A CN 202010578505 A CN202010578505 A CN 202010578505A CN 111754406 A CN111754406 A CN 111754406A
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- sampling
- low
- frequency residual
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 32
- 238000005070 sampling Methods 0.000 claims abstract description 165
- 238000000034 method Methods 0.000 claims abstract description 69
- 238000012545 processing Methods 0.000 claims abstract description 44
- 230000008569 process Effects 0.000 claims description 23
- 238000013527 convolutional neural network Methods 0.000 claims description 21
- 238000013528 artificial neural network Methods 0.000 claims description 8
- 238000000926 separation method Methods 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 2
- 230000000007 visual effect Effects 0.000 abstract description 12
- 230000006870 function Effects 0.000 description 13
- 230000005540 biological transmission Effects 0.000 description 8
- 238000012549 training Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 230000004913 activation Effects 0.000 description 5
- 238000000605 extraction Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 3
- 230000003321 amplification Effects 0.000 description 3
- 238000003199 nucleic acid amplification method Methods 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001575 pathological effect Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
- G06T3/4076—Super resolution, i.e. output image resolution higher than sensor resolution by iteratively correcting the provisional high resolution image using the original low-resolution image
Abstract
The application discloses an image resolution processing method, an image resolution processing device, image resolution processing equipment and a readable storage medium, wherein the method comprises the steps of obtaining a low-resolution image to be processed, wherein a high-frequency residual error is embedded in the low-resolution image; separating a high-frequency residual error and a down-sampled image from the low-resolution image; performing up-sampling processing on the down-sampled image to acquire a high-resolution image; combining the high resolution image with the high frequency residual to obtain a high resolution output image. And (3) separating a high-frequency residual error and a down-sampled image from the low-resolution image, combining the high-resolution image obtained by up-sampling the down-sampled image with the high-frequency residual error to obtain a high-resolution output image, introducing the high-frequency residual error to enable the up-sampled image to recover missing detail information, and enabling the visual quality and the fidelity of the image to be higher.
Description
Technical Field
The present invention relates to the field of image processing, and in particular, to a method, an apparatus, a device, and a readable storage medium for processing image resolution.
Background
Image down-sampling and up-sampling, also commonly referred to as image resolution reduction and image super-resolution, refer to the process of acquiring a low resolution image from a high resolution image and a high resolution image from a low resolution image, respectively.
In conventional image up-sampling or super-resolution techniques, an existing low-resolution image is enlarged and reconstructed to obtain a high-quality enlarged image. However, since image up-sampling requires restoring missing high-frequency detail information in a low-resolution image, which is a typical pathological problem, the current super-resolution algorithm cannot accurately restore the missing high-frequency detail in the image down-sampling process, so that the detail of the restored image after final amplification is missing, and the up-sampling processing effect is poor.
Disclosure of Invention
The present application mainly aims to provide an image resolution processing method, an image resolution processing apparatus, an image resolution processing device, and a readable storage medium, and aims to solve the problem that the restored image after the upsampling processing is low in visual quality and fidelity at present.
In order to achieve the above object, the present application provides an image resolution processing method, including:
acquiring a low-resolution image to be processed, wherein a high-frequency residual error is embedded in the low-resolution image;
separating a high-frequency residual error and a down-sampled image from the low-resolution image;
performing up-sampling processing on the down-sampled image to acquire a high-resolution image;
combining the high resolution image with the high frequency residual to obtain a high resolution output image.
Optionally, before the step of acquiring the low-resolution image to be processed with the high-frequency residual embedded therein, the method further includes:
acquiring an original high-resolution image;
performing down-sampling processing on the original high-resolution image to obtain a down-sampled image;
performing upsampling prediction processing on the downsampled image to obtain a high-resolution restored image;
obtaining the high-frequency residual error according to the original high-resolution image and the high-resolution restoration image;
embedding the high frequency residual into the downsampled image to obtain the low resolution image embedded with the high frequency residual.
Optionally, the step of performing down-sampling processing on the original high-resolution image to obtain a down-sampled image includes:
inputting the original high-resolution image into a convolution neural network-based downsampling network;
and according to a preset resolution value, down-sampling the original high-resolution image into a down-sampled image through the down-sampling network.
Optionally, the embedding the high frequency residual into the downsampled image to obtain the low resolution image embedded with the high frequency residual comprises:
acquiring a preset number of low-resolution high-frequency residual errors consistent with the resolution of the down-sampled image according to the down-sampling multiple;
inputting the low-resolution high-frequency residual and the downsampled image into a convolutional neural network;
outputting the low resolution image with the embedded high frequency residual error through the convolutional data network.
Optionally, the step of obtaining a preset number of low-resolution high-frequency residuals consistent with the resolution of the down-sampled image according to the down-sampling multiple includes:
if the down-sampling multiple is an integral multiple, selecting a first preset number of initial pixel points from the high-frequency residual error according to the down-sampling multiple, and selecting a second preset number of interval pixel points;
and forming a third preset number of low-resolution high-frequency residual errors by the number of the interval pixels.
Optionally, the step of obtaining a preset number of low-resolution high-frequency residuals consistent with the resolution of the down-sampled image according to the down-sampling multiple includes:
if the down-sampling multiple is a non-integral multiple, determining a nearly integral multiple matched with the down-sampling multiple;
according to the approximate integral multiple, performing down-sampling on the high-frequency residual error to obtain a fourth preset number of candidate low-resolution high-frequency residual errors;
and carrying out bilinear interpolation on each candidate low-resolution high-frequency residual error to obtain a low-resolution high-frequency residual error consistent with the resolution of the down-sampled image.
Optionally, the step of performing an upsampling process on the downsampled image to obtain a high-resolution image comprises:
inputting the downsampled image into an upsampling network based on a convolutional neural network;
and processing the down-sampled image according to the up-sampling network to obtain a high-resolution image.
The present application also provides an image resolution processing apparatus, including:
the acquisition module is used for acquiring a low-resolution image to be processed, wherein the low-resolution image is embedded with a high-frequency residual error;
the separation module is used for separating a high-frequency residual error and a down-sampled image from the low-resolution image;
the up-sampling module is used for carrying out up-sampling processing on the down-sampled image so as to obtain a high-resolution image;
a combining module to combine the high resolution image with the high frequency residual to obtain a high resolution output image.
The present application also provides an image resolution processing apparatus, including: a memory, a processor and an image resolution processing program stored on the memory and executable on the processor, the image resolution processing program when executed by the processor implementing the steps of the image resolution processing method as described above.
The present application also provides a readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the image resolution processing method as described above.
Acquiring a low-resolution image to be processed, wherein a high-frequency residual error is embedded in the low-resolution image; separating a high-frequency residual error and a down-sampled image from the low-resolution image; performing up-sampling processing on the down-sampled image to acquire a high-resolution image; combining the high resolution image with the high frequency residual to obtain a high resolution output image. And (3) separating a high-frequency residual error and a down-sampled image from the low-resolution image, combining the high-resolution image obtained by up-sampling the down-sampled image with the high-frequency residual error to obtain a high-resolution output image, introducing the high-frequency residual error to enable the up-sampled image to recover missing detail information, and enabling the visual quality and the fidelity of the image to be higher.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
Fig. 1 is a schematic device structure diagram of a hardware operating environment according to an embodiment of the present application;
FIG. 2 is a flowchart illustrating a first embodiment of an image resolution processing method according to the present application;
FIG. 3 is a flowchart illustrating steps before step S10 in FIG. 2 according to a second embodiment of the image resolution processing method of the present application;
FIG. 4 is a flowchart of a refinement of step S15 in FIG. 3 according to a fourth embodiment of the image resolution processing method of the present application;
fig. 5 is a schematic system structure diagram of an embodiment of an image resolution processing apparatus according to the present application.
The implementation, functional features and advantages of the objectives of the present application will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in itself. Thus, "module", "component" or "unit" may be used mixedly.
As shown in fig. 1, fig. 1 is a schematic terminal structure diagram of a hardware operating environment according to an embodiment of the present application.
The terminal in the embodiment of the application is image resolution processing equipment.
As shown in fig. 1, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a storage device separate from the processor 1001.
Optionally, the terminal may further include a camera, a Radio Frequency (RF) circuit, a sensor, an audio circuit, a WiFi module, and the like. Such as light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor that adjusts the brightness of the display screen according to the brightness of ambient light, and a proximity sensor that turns off the display screen and/or the backlight when the terminal device is moved to the ear. Of course, the terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which are not described herein again.
Those skilled in the art will appreciate that the terminal structure shown in fig. 1 is not intended to be limiting and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a kind of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and an image resolution processing program.
In the terminal shown in fig. 1, the network interface 1004 is mainly used for connecting to a backend server and performing data communication with the backend server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be configured to call the image resolution processing program stored in the memory 1005 and perform the following operations:
acquiring a low-resolution image to be processed, wherein a high-frequency residual error is embedded in the low-resolution image;
separating a high-frequency residual error and a down-sampled image from the low-resolution image;
performing up-sampling processing on the down-sampled image to acquire a high-resolution image;
combining the high resolution image with the high frequency residual to obtain a high resolution output image.
Based on the above terminal hardware structure, various embodiments of the present application are provided.
The application provides an image resolution processing method.
Referring to fig. 2, in a first embodiment of an image resolution processing method, the method includes:
step S10, acquiring a low-resolution image to be processed, wherein the low-resolution image is embedded with a high-frequency residual error;
image down-sampling and up-sampling, also commonly referred to as image resolution reduction and image super-resolution, refer to the process of acquiring a low resolution image from a high resolution image and a high resolution image from a low resolution image, respectively. The high-resolution image cannot be used due to hardware conditions of the device or transmission conditions of file transmission, or based on active selection by the user, the high-resolution image needs to be processed into a low-resolution image by down-sampling, but the low-resolution image may need to be re-up-sampled into a high-resolution image later. Especially, in the process of image transmission, in order to reduce the size of a transmission file, a high-resolution image is converted into a low-resolution image through down-sampling and transmitted to a receiving end, the receiving end needs to amplify and reconstruct the transmitted low-resolution image through an up-sampling method, but high-frequency detail information which is lacked in the low-resolution image is difficult to restore, and therefore the visual quality and the fidelity which are obtained through up-sampling at present are low. Meanwhile, the existing up-sampling method is usually directed at a fixed or simple down-sampling model, and after the down-sampling model is replaced, the performance of the corresponding up-sampling model is greatly reduced. In the application, the low-resolution image embedded with the high-frequency residual error is used, and when the low-resolution image needs to be restored to the high-resolution image through an up-sampling method, the high-frequency residual error can supplement the missing high-frequency details of the image in the low-resolution image. The high-frequency residual error is embedded into the low-resolution image, the high-frequency residual error does not need to be additionally transmitted, and the visual effect of the low-resolution image cannot be influenced. The acquisition of the high-frequency residual error is based on a prediction mechanism, the low-resolution image obtained by performing down-sampling on the initial high-resolution image is subjected to one-time up-sampling prediction, and the high-frequency residual error can be obtained by acquiring a corresponding restored high-resolution image and comparing the restored high-resolution image with the initial high-resolution image. For the down-sampling in the present application, the down-sampling may be a conventional down-sampling method such as extracting a pixel point in an image, or may be a down-sampling network based on a convolutional neural network, and similarly, the up-sampling method may also be a conventional up-sampling method such as an interpolation method, or may be an up-sampling network based on a convolutional neural network. In the present application, the down-sampling method and the up-sampling method are not limited, and it is, of course, preferable to use a down-sampling network and an up-sampling network based on a convolutional neural network.
Step S20, separating high-frequency residual error and down-sampled image from the low-resolution image;
the low-resolution image comprises a down-sampling image and a high-frequency residual error, and in the up-sampling process, only the down-sampling image needs to be subjected to up-sampling processing, so that the high-frequency residual error in the low-resolution image and the sampling image are separated. The extraction of the high-frequency residual error in the low-resolution image is performed by a method based on a convolutional neural network, wherein the neural network comprises an input layer, a plurality of convolutional layers and a residual error connection from the input layer to the convolutional layers after output, specifically, the input layer is a convolutional layer with the size of 3 × 3, the input layer converts the low-resolution image into 64 feature maps, each convolutional layer is also a convolutional layer with the size of 3 × 3, simultaneously, a linear rectification function is used as an activation function, the number of feature channels is also 64, and the convolutional layers are used for feature extraction and reconstruction. The residual connection can directly transmit the input first resolution image to the output position of the convolution layer, the output layer is a double-branch output layer, one branch is two convolution layers with the size of 3 x 3, the other branch sequentially comprises one convolution layer with the size of 3 x 3, an upper sampling layer and another convolution layer with the size of the other branch, and the two channels respectively reconstruct the characteristic image into a down-sampling image and a high-frequency residual.
Step S30, performing upsampling processing on the downsampled image to obtain a high-resolution image;
and for the down-sampling image separated from the low-resolution image, performing up-sampling processing on the down-sampling image by using an up-sampling method to obtain a high-resolution image. The upsampling method may be an interpolation method, or an upsampling network method based on a convolutional neural network. The interpolation method is simpler to implement, but the result obtained by the convolution neural network-based up-sampling network method is more accurate, so the convolution neural network-based up-sampling network method is preferable in the application.
Step S40, combining the high resolution image with the high frequency residual to obtain a high resolution output image;
and finally, combining the high-resolution image obtained by the up-sampling treatment with the high-frequency residual error to obtain a high-resolution output image, wherein the high-resolution image obtained by the up-sampling treatment lacks high-frequency details lost by the down-sampling treatment, and the high-frequency residual error contains corresponding details, so that the finally obtained high-resolution image has corresponding details, the visual effect is better, and the fidelity is higher.
In the embodiment, a low-resolution image embedded with a high-frequency residual error to be processed is obtained; separating a high-frequency residual error and a down-sampled image from the low-resolution image; performing up-sampling processing on the down-sampled image to acquire a high-resolution image; combining the high resolution image with the high frequency residual to obtain a high resolution output image. And (3) separating a high-frequency residual error and a down-sampled image from the low-resolution image, combining the high-resolution image obtained by up-sampling the down-sampled image with the high-frequency residual error to obtain a high-resolution output image, introducing the high-frequency residual error to enable the up-sampled image to recover missing detail information, and enabling the visual quality and the fidelity of the image to be higher.
Further, referring to fig. 2 and 3, on the basis of the first embodiment of the image resolution processing method of the present application, there is provided a second embodiment of the image resolution processing method, in which,
before step S10, the method further includes:
step S11, acquiring an original high-resolution image;
in practical use of the technical scheme in the application, preferably, a down-sampling process and an up-sampling process are cooperatively learned and combined to perform resolution processing on an image, a low-resolution image is obtained for a high-resolution image through a down-sampling method, a high-frequency residual is obtained based on a prediction mechanism, the high-frequency residual is embedded in the low-resolution image, and then the low-resolution image embedded with the high-frequency residual is restored to the high-resolution image through an up-sampling method. Therefore, in the present application, the down-sampling and up-sampling processes constitute a complete image resolution processing process. The original high-resolution image is obtained, and the original high-resolution image can be a high-resolution image which needs to be transmitted or other high-resolution images. The original high-resolution image contains all the necessary image information, in particular detail information.
Step S12, down-sampling the original high-resolution image to obtain a down-sampled image;
and performing down-sampling processing on the original high-resolution image to obtain a down-sampled image. The down-sampling method may be a conventional down-sampling method, such as combining a plurality of pixel points in a region into one pixel point by a weighted average method to reduce the resolution of the image, or selecting a specific pixel point from all the pixel points of the image according to a certain rule to obtain a low-resolution down-sampled image. Meanwhile, the down-sampling method can also be a down-sampling network based on a convolutional neural network. It is preferred in this application to employ a downsampling network.
Step S13, performing upsampling prediction processing on the downsampled image to obtain a high-resolution restored image;
the up-sampling prediction of the down-sampled image is a super-resolution reconstruction and an up-sampling process, namely a process of restoring a low-resolution image into a high-resolution image. The upsampling process may use a conventional upsampling method such as interpolation, or may use an upsampling network based on a convolutional neural network. When an upsampling network is employed, the secondary upsampling network is the same as the upsampling network in the first embodiment. However, the up-sampling method itself is not improved, so that the high-resolution image obtained by the up-sampling method still has the problem of corresponding detail loss.
Step S14, obtaining the high-frequency residual error according to the original high-resolution image and the high-resolution restored image;
the high-resolution restored image obtained by up-sampling is compared with the original high-resolution image to obtain a high-frequency residual error, wherein the high-frequency residual error is detail information which cannot be restored in the up-sampling process, and the missing detail information can be replenished into the image again through the high-frequency residual error.
Step S15, embedding the high-frequency residual error into the down-sampling image to obtain the low-resolution image embedded with the high-frequency residual error;
for the high-frequency residual error and the down-sampled image, because the spatial information redundancy and the information entropy redundancy exist in the down-sampled image, the high-frequency residual error can be embedded into the down-sampled image without increasing the storage consumption and the transmission consumption of the image in the transmission process. Meanwhile, a convolution neural network method is generally adopted for embedding the high-frequency residual error into the downsampled image.
The steps in the first embodiment and the steps in the second embodiment are combined to form an optimal complete scheme in the application, that is, a down-sampling method and an up-sampling method are combined, down-sampling is performed on a high-resolution image to obtain a low-resolution image, then a high-frequency residual is obtained through up-sampling prediction, the high-frequency residual is embedded into the low-resolution image to be transmitted and stored, when the low-resolution image needs to be subjected to resolution amplification and reconstruction, the high-frequency residual can be used for restoring the high-frequency details which are lacked after up-sampling restoration, the visual quality and the fidelity of the finally obtained high-resolution image are better, meanwhile, the selection of the down-sampling method and the up-sampling method is not limited, and only the up-sampling prediction needs to be added to obtain the high-frequency residual and the high-frequency.
In the embodiment, the high-frequency residual is obtained through upsampling prediction and embedded in the image, so that residual information is provided for obtaining high-quality high-resolution images later, and meanwhile, the consumption in the image storage and transmission process is not increased.
Further, on the basis of the above-described embodiments of the image resolution processing method of the present application, there is provided a third embodiment of the image resolution processing method which, in the third embodiment,
step S12 includes:
step A1, inputting the original high-resolution image into a convolution neural network-based downsampling network;
step A2, according to a preset resolution value, down-sampling the original high-resolution image into a down-sampled image through the down-sampling network;
in this embodiment of the present application, the downsampling is performed through a downsampling network based on a convolutional neural network, and the downsampling network sequentially includes an input layer, a plurality of convolutional layers, a downsampling layer, a plurality of convolutional layers, a residual connection from the back of the input layer to the back of the convolutional layers, and an output layer. Wherein, the input layer is a convolution layer with the size of 3 x 3 and is used for converting an original high-resolution image into 64 characteristic maps; the convolution layer in front of the down-sampling layer is a convolution layer with the size of 3 x 3, a linear rectification function is used as an activation function, the number of characteristic channels is 64, and the characteristic channels are used for characteristic extraction and reconstruction; the down-sampling layer is a bilinear interpolation layer and is used for reducing the resolution of the characteristic diagram to a specified size; the convolution layer after the down-sampling layer is a convolution layer with the size of 3 x 3, a linear rectification function is also used as an activation function, the number of characteristic channels is also 64, and the method is used for further carrying out characteristic extraction and reconstruction on the characteristic image after the dimensionality reduction; the residual error connection comprises a down-sampling layer and a convolution layer with the size of 3 x 3, the down-sampling layer in the residual error connection is also a bilinear interpolation layer, an activation function in the residual error connection is also a linear rectification function, the number of characteristic channels is 64, and the residual error connection is used for ensuring the accuracy of the final output result after the input images are transmitted to the convolution layer; the output slice is also a 3 x 3 sized convolution slice for reconstructing the 64 feature maps into a down-sampled image. And in the down-sampling process, down-sampling is carried out through a down-sampling network according to a preset resolution value.
In the embodiment, the original high-resolution image is down-sampled through the down-sampling network, so that the obtained down-sampled image has better visual quality.
Further, referring to fig. 2 and 4, on the basis of the above-described embodiments of the image resolution processing method of the present application, there is provided a fourth embodiment of the image resolution processing method, in which,
step S15 includes:
step S151, acquiring a preset number of low-resolution high-frequency residuals consistent with the resolution of the down-sampled image according to the down-sampling multiple;
according to the downsampling image and the resolution ratio of the original high-resolution image, the downsampling multiple can be obtained, for example, if the resolution ratio of the original high-resolution image is 1920 × 1080 and the resolution ratio of the downsampling image is 960 × 540, the downsampling multiple is 2 × 2, according to the downsampling multiple, the high-frequency residual error is changed into the low-resolution high-frequency residual error which is the same as the resolution ratio of the downsampling image, meanwhile, the number of the obtained low-resolution high-frequency residual errors is more than one to ensure that the complete high-frequency residual error can be kept, and meanwhile, the resolution ratio of the high-frequency residual error is consistent with the downsampling image to ensure that the visual quality of the downsampling image cannot be influenced after embedding.
Step S152, inputting the low-resolution high-frequency residual error and the downsampled image into a convolutional neural network;
step S153, outputting the low-resolution image embedded with the high-frequency residual error through the convolution data network;
the convolutional neural network for embedding high-frequency residual errors into a down-sampled image comprises an input layer, a plurality of convolutional layers, a residual error connection from the input layer to the convolutional layers and an output layer. The input layer is a convolution layer with the size of 3 x 3 and is used for converting the down-sampled image and the low-resolution high-frequency residual image into 64 feature images; the convolution layer is a convolution layer with the size of 3 x 3, and a linear rectification function is used as an activation function for feature extraction and reconstruction; the output layer is a 3 × 3 convolutional layer for reconstructing 64 feature maps into a low resolution image with high frequency residuals embedded therein.
In the embodiment, the high-frequency residual error information is embedded into the downsampled image through the convolutional neural network, extra storage and transmission consumption are not increased when the downsampled image contains the prompt of the high-frequency residual error information, and the visual effect of the downsampled image is not influenced.
Further, on the basis of the above-described embodiments of the image resolution processing method of the present application, there is provided a fifth embodiment of the image resolution processing method which, in the fifth embodiment,
step S151 includes:
step B1, if the down-sampling multiple is an integral multiple, selecting a first preset number of initial pixel points in the high-frequency residual error according to the down-sampling multiple, and selecting a second preset number of interval pixel points;
step B2, forming a third preset number of low-resolution high-frequency residuals by the number of the interval pixels;
and when the down-sampling multiple is integral multiple, acquiring a plurality of low-resolution high-frequency residual errors by adopting a method of sampling at intervals of corresponding pixel points, wherein the resolution of each low-resolution residual error is consistent with that of the down-sampled image. If the down-sampled image is reduced by two times in the horizontal direction of the original high-resolution image, sampling from the first pixel point at the upper left corner every other 1 pixel point to obtain a low-resolution high-frequency residual error, and sampling from the second pixel point at the upper left corner every other 1 point to obtain another low-resolution high-frequency residual error, so as to obtain a preset number of low-resolution high-frequency residual errors, wherein the preset number is related to the down-sampling multiple, and the integer multiples of other down-sampling are analogized in this way.
In another possible solution, step S151 includes:
step B3, if the down-sampling multiple is non-integral multiple, determining the approximate integral multiple matched with the down-sampling multiple;
step B4, according to the approximate integral multiple, down-sampling the high-frequency residual error to obtain a fourth preset number of candidate low-resolution high-frequency residual errors;
step B5, carrying out bilinear interpolation on each candidate low-resolution high-frequency residual error to obtain a low-resolution high-frequency residual error consistent with the resolution of the down-sampled image;
if the down-sampling multiple is not the integral multiple, determining that the matched approximate integral multiple is firstly determined, if the down-sampling multiple is 2.1 times, the approximate integral multiple is 2 times, firstly obtaining candidate low-resolution high-frequency residual errors according to the approximate integral multiple and the method in the steps B1-B2, and then using bilinear interpolation on each candidate low-resolution high-frequency residual error to ensure that the resolution of the final low-resolution high-frequency residual error is consistent with that of the down-sampled image.
In this embodiment, a sampling method for changing the high-frequency residual error into the low resolution error is determined according to the down-sampling multiple, so that the resolution of the high-frequency residual error is consistent with that of the down-sampled image.
Further, on the basis of the above-described embodiments of the image resolution processing method of the present application, there is provided a sixth embodiment of the image resolution processing method, in which,
step S30 includes:
step C1, inputting the down-sampled image into an up-sampling network based on a convolutional neural network;
step C2, processing the down-sampled image according to the up-sampling network to obtain a high resolution image;
in this embodiment, an upsampling network based on a convolutional neural network is used for the upsampling process of the downsampled image. The up-sampling network comprises an up-sampling layer, an input layer, a plurality of residual blocks, a residual connection from the back of the input layer to the back of the output of the residual blocks and an output layer in sequence. The up-sampling layer is a bilinear interpolation layer and is used for interpolating and amplifying the down-sampling image to a specified resolution; the input layer is a convolution layer with the size of 3 x 3 and is used for converting the interpolated image into 64 characteristic graphs; each residual block comprises a 64-channel 3 x 3 convolution layer, a linear rectification function, another 64-channel 3 x 3 convolution layer and a residual connection connecting two ends; the output layer is a 3 x 3 sized convolutional layer used to reconstruct the 64 signatures into a high resolution image. The upsampling network in the present embodiment can also be used for the upsampling prediction in the second embodiment.
When using a convolutional neural network based downsampling network as well as an upsampling network, the upsampling network as well as the downsampling network may be trained. The training of the up-sampling network and the down-sampling network can be independent training in advance, or can be combined into the whole network for end-to-end training. In the training process, a high-quality high-resolution image is used as a training set, and the constraint condition in the training process is mainly that a finally obtained high-resolution output image is required to be consistent with an original high-resolution image, and simultaneously, a low-resolution image embedded with a high-frequency residual error is required to be visually consistent with an original downsampled image, namely, the basic content of an original image cannot be changed due to the embedded residual error component. For the loss function L in the training process, it can be:
L＝L1(XOUT,X)+λLSSIM(Y*,Y)
wherein L is1Is L1 lossLose, LSSIMThe SSIM losses are all common loss functions in image enhancement and reconstruction problems. XOUTFor the high resolution output image, X is the original high resolution image, Y is the low resolution image with the high frequency residual embedded therein, and Y is the down sampled image.
In the embodiment, resolution amplification and reconstruction of the downsampled image are performed through the upsampling network based on the convolutional neural network, and the obtained high-resolution image has better visual quality.
Further, with reference to fig. 5, an embodiment of the present application also proposes an image resolution processing apparatus including:
the first acquisition module is used for acquiring a low-resolution image to be processed, wherein a high-frequency residual error is embedded in the low-resolution image;
the separation module is used for separating a high-frequency residual error and a down-sampled image from the low-resolution image;
the up-sampling module is used for carrying out up-sampling processing on the down-sampled image so as to obtain a high-resolution image;
a combining module to combine the high resolution image with the high frequency residual to obtain a high resolution output image.
Optionally, the image resolution processing apparatus further includes:
the second acquisition module is used for acquiring an original high-resolution image;
the down-sampling module is used for carrying out down-sampling processing on the original high-resolution image so as to obtain a down-sampled image;
the prediction module is used for carrying out up-sampling prediction processing on the down-sampled image so as to obtain a high-resolution restored image;
a residual error obtaining module, configured to obtain the high-frequency residual error according to the original high-resolution image and the high-resolution restored image;
and the embedded module is used for embedding the high-frequency residual error into the down-sampled image so as to obtain the low-resolution image embedded with the high-frequency residual error.
Optionally, the down-sampling module is further configured to:
inputting the original high-resolution image into a convolution neural network-based downsampling network;
and according to a preset resolution value, down-sampling the original high-resolution image into a down-sampled image through the down-sampling network.
Optionally, the inline module is further configured to:
acquiring a preset number of low-resolution high-frequency residual errors consistent with the resolution of the down-sampled image according to the down-sampling multiple;
inputting the low-resolution high-frequency residual and the downsampled image into a convolutional neural network;
outputting the low resolution image with the embedded high frequency residual error through the convolutional data network.
Optionally, the image resolution processing apparatus further includes:
the sampling module is used for selecting a first preset number of initial pixel points from the high-frequency residual error according to the down-sampling multiple if the down-sampling multiple is an integral multiple, and selecting a second preset number of interval pixel points;
and forming a third preset number of low-resolution high-frequency residual errors by the number of the interval pixels.
Optionally, the sampling module is further configured to:
if the down-sampling multiple is a non-integral multiple, determining a nearly integral multiple matched with the down-sampling multiple;
according to the approximate integral multiple, performing down-sampling on the high-frequency residual error to obtain a fourth preset number of candidate low-resolution high-frequency residual errors;
and carrying out bilinear interpolation on each candidate low-resolution high-frequency residual error to obtain a low-resolution high-frequency residual error consistent with the resolution of the down-sampled image.
Optionally, the upsampling module is further configured to:
inputting the downsampled image into an upsampling network based on a convolutional neural network;
and processing the down-sampled image according to the up-sampling network to obtain a high-resolution image.
The specific implementation of the apparatus and the readable storage medium (i.e., the computer readable storage medium) of the present application is basically the same as the embodiments of the image resolution processing method, and is not described herein again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (10)
1. An image resolution processing method, characterized by comprising the steps of:
acquiring a low-resolution image to be processed, wherein a high-frequency residual error is embedded in the low-resolution image;
separating a high-frequency residual error and a down-sampled image from the low-resolution image;
performing up-sampling processing on the down-sampled image to acquire a high-resolution image;
combining the high resolution image with the high frequency residual to obtain a high resolution output image.
2. The image resolution processing method according to claim 1, wherein the step of acquiring the low resolution image to be processed embedded with the high frequency residual is preceded by:
acquiring an original high-resolution image;
performing down-sampling processing on the original high-resolution image to obtain a down-sampled image;
performing upsampling prediction processing on the downsampled image to obtain a high-resolution restored image;
obtaining the high-frequency residual error according to the original high-resolution image and the high-resolution restoration image;
embedding the high frequency residual into the downsampled image to obtain the low resolution image embedded with the high frequency residual.
3. The image resolution processing method according to claim 2, wherein the step of down-sampling the original high resolution image to obtain a down-sampled image comprises:
inputting the original high-resolution image into a convolution neural network-based downsampling network;
and according to a preset resolution value, down-sampling the original high-resolution image into a down-sampled image through the down-sampling network.
4. The image resolution processing method according to claim 2, wherein the embedding the high frequency residual into the down-sampled image to obtain the low resolution image embedded with the high frequency residual comprises:
acquiring a preset number of low-resolution high-frequency residual errors consistent with the resolution of the down-sampled image according to the down-sampling multiple;
inputting the low-resolution high-frequency residual and the downsampled image into a convolutional neural network;
outputting the low resolution image with the embedded high frequency residual error through the convolutional data network.
5. The image resolution processing method according to claim 4, wherein the step of obtaining a preset number of low-resolution high-frequency residuals consistent with the resolution of the down-sampled image according to the down-sampling multiple comprises:
if the down-sampling multiple is an integral multiple, selecting a first preset number of initial pixel points from the high-frequency residual error according to the down-sampling multiple, and selecting a second preset number of interval pixel points;
and forming a third preset number of low-resolution high-frequency residual errors by the number of the interval pixels.
6. The image resolution processing method according to claim 4, wherein the step of obtaining a preset number of low-resolution high-frequency residuals consistent with the resolution of the down-sampled image according to the down-sampling multiple comprises:
if the down-sampling multiple is a non-integral multiple, determining a nearly integral multiple matched with the down-sampling multiple;
according to the approximate integral multiple, performing down-sampling on the high-frequency residual error to obtain a fourth preset number of candidate low-resolution high-frequency residual errors;
and carrying out bilinear interpolation on each candidate low-resolution high-frequency residual error to obtain a low-resolution high-frequency residual error consistent with the resolution of the down-sampled image.
7. The image resolution processing method according to claim 1, wherein the step of performing the up-sampling process on the down-sampled image to acquire the high-resolution image includes:
inputting the downsampled image into an upsampling network based on a convolutional neural network;
and processing the down-sampled image according to the up-sampling network to obtain a high-resolution image.
8. An image resolution processing apparatus characterized by comprising:
the acquisition module is used for acquiring a low-resolution image to be processed, wherein the low-resolution image is embedded with a high-frequency residual error;
the separation module is used for separating a high-frequency residual error and a down-sampled image from the low-resolution image;
the up-sampling module is used for carrying out up-sampling processing on the down-sampled image so as to obtain a high-resolution image;
a combining module to combine the high resolution image with the high frequency residual to obtain a high resolution output image.
9. An image resolution processing apparatus characterized by comprising: memory, a processor and an image resolution processing program stored on the memory and executable on the processor, the image resolution processing program when executed by the processor implementing the steps of the image resolution processing method according to any one of claims 1 to 7.
10. A readable storage medium, characterized in that the readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps of the image resolution processing method according to any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010578505.1A CN111754406B (en) | 2020-06-22 | 2020-06-22 | Image resolution processing method, device, equipment and readable storage medium |
| PCT/CN2020/111777 WO2021258530A1 (en) | 2020-06-22 | 2020-08-27 | Image resolution processing method, device, apparatus, and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010578505.1A CN111754406B (en) | 2020-06-22 | 2020-06-22 | Image resolution processing method, device, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111754406A true CN111754406A (en) | 2020-10-09 |
| CN111754406B CN111754406B (en) | 2024-02-23 |
Family
ID=72676553
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010578505.1A Active CN111754406B (en) | 2020-06-22 | 2020-06-22 | Image resolution processing method, device, equipment and readable storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111754406B (en) |
| WO (1) | WO2021258530A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113313691A (en) * | 2021-06-03 | 2021-08-27 | 上海市第一人民医院 | Thyroid color Doppler ultrasound processing method based on deep learning |
| WO2023197805A1 (en) * | 2022-04-11 | 2023-10-19 | 北京字节跳动网络技术有限公司 | Image processing method and apparatus, and storage medium and electronic device |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114331849B (en) * | 2022-03-15 | 2022-06-10 | 之江实验室 | Cross-mode nuclear magnetic resonance hyper-resolution network and image super-resolution method |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103607591A (en) * | 2013-10-28 | 2014-02-26 | 四川大学 | Image compression method combining super-resolution reconstruction |
| WO2015180053A1 (en) * | 2014-05-28 | 2015-12-03 | 北京大学深圳研究生院 | Method and apparatus for rapidly reconstructing super-resolution image |
| CN107018422A (en) * | 2017-04-27 | 2017-08-04 | 四川大学 | Still image compression method based on depth convolutional neural networks |
| CN107181949A (en) * | 2017-06-23 | 2017-09-19 | 四川大学 | A kind of compression of images framework of combination super-resolution and residual coding technology |
| CN108734660A (en) * | 2018-05-25 | 2018-11-02 | 上海通途半导体科技有限公司 | A kind of image super-resolution rebuilding method and device based on deep learning |
| CN109525859A (en) * | 2018-10-10 | 2019-03-26 | 腾讯科技（深圳）有限公司 | Model training, image transmission, image processing method and relevant apparatus equipment |
| WO2019153671A1 (en) * | 2018-02-11 | 2019-08-15 | 深圳创维－Rgb电子有限公司 | Image super-resolution method and apparatus, and computer readable storage medium |
| WO2019192588A1 (en) * | 2018-04-04 | 2019-10-10 | 华为技术有限公司 | Image super resolution method and device |
| CN110636289A (en) * | 2019-09-27 | 2019-12-31 | 北京金山云网络技术有限公司 | Image data transmission method, system, device, electronic equipment and storage medium |
| CN111179177A (en) * | 2019-12-31 | 2020-05-19 | 深圳市联合视觉创新科技有限公司 | Image reconstruction model training method, image reconstruction method, device and medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101320423A (en) * | 2008-06-26 | 2008-12-10 | 复旦大学 | Low resolution gait recognition method based on high-frequency super-resolution |
| CN102915527A (en) * | 2012-10-15 | 2013-02-06 | 中山大学 | Face image super-resolution reconstruction method based on morphological component analysis |
| CN103116880A (en) * | 2013-01-16 | 2013-05-22 | 杭州电子科技大学 | Image super resolution rebuilding method based on sparse representation and various residual |
| CN106981046B (en) * | 2017-03-21 | 2019-10-11 | 四川大学 | Single image super resolution ratio reconstruction method based on multi-gradient constrained regression |
| CN107358575A (en) * | 2017-06-08 | 2017-11-17 | 清华大学 | A kind of single image super resolution ratio reconstruction method based on depth residual error network |
| EP3567549A1 (en) * | 2018-05-07 | 2019-11-13 | Technische Universität München | Depth super-resolution from shading |
-
2020
- 2020-06-22 CN CN202010578505.1A patent/CN111754406B/en active Active
- 2020-08-27 WO PCT/CN2020/111777 patent/WO2021258530A1/en active Application Filing
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103607591A (en) * | 2013-10-28 | 2014-02-26 | 四川大学 | Image compression method combining super-resolution reconstruction |
| WO2015180053A1 (en) * | 2014-05-28 | 2015-12-03 | 北京大学深圳研究生院 | Method and apparatus for rapidly reconstructing super-resolution image |
| CN107018422A (en) * | 2017-04-27 | 2017-08-04 | 四川大学 | Still image compression method based on depth convolutional neural networks |
| CN107181949A (en) * | 2017-06-23 | 2017-09-19 | 四川大学 | A kind of compression of images framework of combination super-resolution and residual coding technology |
| WO2019153671A1 (en) * | 2018-02-11 | 2019-08-15 | 深圳创维－Rgb电子有限公司 | Image super-resolution method and apparatus, and computer readable storage medium |
| WO2019192588A1 (en) * | 2018-04-04 | 2019-10-10 | 华为技术有限公司 | Image super resolution method and device |
| CN108734660A (en) * | 2018-05-25 | 2018-11-02 | 上海通途半导体科技有限公司 | A kind of image super-resolution rebuilding method and device based on deep learning |
| CN109525859A (en) * | 2018-10-10 | 2019-03-26 | 腾讯科技（深圳）有限公司 | Model training, image transmission, image processing method and relevant apparatus equipment |
| CN110636289A (en) * | 2019-09-27 | 2019-12-31 | 北京金山云网络技术有限公司 | Image data transmission method, system, device, electronic equipment and storage medium |
| CN111179177A (en) * | 2019-12-31 | 2020-05-19 | 深圳市联合视觉创新科技有限公司 | Image reconstruction model training method, image reconstruction method, device and medium |
Non-Patent Citations (1)
| Title |
|---|
| 李剑飞: "基于残差字典的超分辨率图像重构", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》, pages 138 - 963 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113313691A (en) * | 2021-06-03 | 2021-08-27 | 上海市第一人民医院 | Thyroid color Doppler ultrasound processing method based on deep learning |
| WO2023197805A1 (en) * | 2022-04-11 | 2023-10-19 | 北京字节跳动网络技术有限公司 | Image processing method and apparatus, and storage medium and electronic device |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021258530A1 (en) | 2021-12-30 |
| CN111754406B (en) | 2024-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11354785B2 (en) | Image processing method and device, storage medium and electronic device | |
| CN111754406B (en) | Image resolution processing method, device, equipment and readable storage medium | |
| CN110322400B (en) | Image processing method and device, image processing system and training method thereof | |
| CN108492249B (en) | Single-frame super-resolution reconstruction method based on small convolution recurrent neural network | |
| CN111784571A (en) | Method and device for improving image resolution | |
| US11538136B2 (en) | System and method to process images of a video stream | |
| CN111784570A (en) | Video image super-resolution reconstruction method and device | |
| WO2022166298A1 (en) | Image processing method and apparatus, and electronic device and readable storage medium | |
| CN107220934B (en) | Image reconstruction method and device | |
| CN113939845A (en) | Method, system and computer readable medium for improving image color quality | |
| CN111951164A (en) | Image super-resolution reconstruction network structure and image reconstruction effect analysis method | |
| CN112188236A (en) | Video interpolation frame model training method, video interpolation frame generation method and related device | |
| CN113628115A (en) | Image reconstruction processing method and device, electronic equipment and storage medium | |
| CN113837980A (en) | Resolution adjusting method and device, electronic equipment and storage medium | |
| CN110751251B (en) | Method and device for generating and transforming two-dimensional code image matrix | |
| CN110677676B (en) | Video encoding method and apparatus, video decoding method and apparatus, and storage medium | |
| CN114742738A (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| CN114240750A (en) | Video resolution improving method and device, storage medium and electronic equipment | |
| CN115272667A (en) | Farmland image segmentation model training method and device, electronic equipment and medium | |
| CN114266697A (en) | Image processing and model training method and device, electronic equipment and storage medium | |
| KR20200067114A (en) | Apparatus for transmitting image | |
| US9911178B2 (en) | System and method for content-adaptive super-resolution via cross-scale self-learning | |
| CN112419146A (en) | Image processing method and device and terminal equipment | |
| CN111798385A (en) | Image processing method and device, computer readable medium and electronic device | |
| CN114286113B (en) | Image compression recovery method and system based on multi-head heterogeneous convolution self-encoder |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |