CN111429371A - Image processing method and device and terminal equipment - Google Patents
Image processing method and device and terminal equipment Download PDFInfo
- Publication number
- CN111429371A CN111429371A CN202010207474.9A CN202010207474A CN111429371A CN 111429371 A CN111429371 A CN 111429371A CN 202010207474 A CN202010207474 A CN 202010207474A CN 111429371 A CN111429371 A CN 111429371A
- Authority
- CN
- China
- Prior art keywords
- image
- frequency
- low
- trained
- frequency image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 19
- 238000000034 method Methods 0.000 claims abstract description 43
- 238000012545 processing Methods 0.000 claims abstract description 25
- 238000000605 extraction Methods 0.000 claims description 35
- 238000006243 chemical reaction Methods 0.000 claims description 32
- 230000006870 function Effects 0.000 claims description 32
- 238000012549 training Methods 0.000 claims description 30
- 238000004590 computer program Methods 0.000 claims description 23
- 230000004927 fusion Effects 0.000 claims description 23
- 238000004422 calculation algorithm Methods 0.000 claims description 9
- 230000003321 amplification Effects 0.000 claims description 6
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 6
- 230000000694 effects Effects 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 9
- 230000000007 visual effect Effects 0.000 description 7
- 230000004044 response Effects 0.000 description 4
- 238000012935 Averaging Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000002349 favourable effect Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The application is applicable to the technical field of image processing, and provides an image processing method, an image processing device and terminal equipment, wherein the image processing method comprises the following steps: acquiring an image to be reconstructed; extracting high-frequency components and low-frequency components of the image to be reconstructed to obtain a first high-frequency image consisting of the high-frequency components and a first low-frequency image consisting of the low-frequency components; inputting the first high-frequency image into a trained high-frequency image generation network to obtain a second high-frequency image output by the high-frequency image generation network; inputting the first low-frequency image into a trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network; and generating a reconstructed image according to the second high-frequency image and the second low-frequency image, wherein the resolution of the reconstructed image is higher than that of the image to be reconstructed. By the method, the image with better effect can be reconstructed.
Description
Technical Field
The present application belongs to the field of image processing technologies, and in particular, to an image processing method, an image processing apparatus, a terminal device, and a computer-readable storage medium.
Background
Image resolution refers to the ability of the sensor to view or measure the smallest object, depending on the pixel size. Digital images recorded as two-dimensional signals are in most applications always required to have a higher resolution. In the past decades, imaging technology has been rapidly developed and image resolution has reached a new level, but in many applications it is still desirable to improve image resolution better. For example, digital surveillance products often choose to sacrifice image resolution to some extent to ensure long-term stable operation of the recording device and proper frame rate of the dynamic scene; similar situations also exist in the field of remote sensing: there is always a trade-off between spatial, spectral and image resolution; for medical imaging, however, it remains a challenge how to extract a three-dimensional model of the human structure with the first image while reducing radiation.
At present, the resolution of an image can be improved by a super-resolution method.
Existing super-resolution methods can be roughly divided into two categories, namely traditional interpolation methods and methods based on deep learning. The traditional interpolation algorithm has been developed for decades, but in the Single Image Super-Resolution (SISR) field, the effect is far less than that of the deep learning method. Therefore, many new algorithms use data-driven deep learning models to reconstruct the details needed for an image to achieve accurate super-resolution.
Most of the current improvements of SISR based on depth learning methods are directed to model structure or training methods, which are very effective when performing Super-Resolution on a low-Resolution (L ow Resolution, L R) image obtained by simulation, but the effect is significantly reduced when processing a truly acquired second image, for example, when processing a smartphone-acquired image by using a current SOTA (state of the art, most advanced) Super-Resolution model ESRGAN (x.wang et al), "ESRGAN: Enhanced Super-Resolution general adaptive Networks" in ECCV 2018 workhop), the reconstructed image has obvious over-fitting and blurring details, and the effect is even inferior to that of the low-Resolution image (for example, fig. 1(a) is an original image, fig. 1(b) is an image after-segmentation), because when processing a second image acquired by using a similar smartphone, the image itself contains a lot of noise and image processing process, and therefore, the Super-Resolution image quality is improved as much as a result of the Super-Resolution model and the Super-Resolution model.
Therefore, a new technical solution is needed to solve the above technical problems.
Disclosure of Invention
The embodiment of the application provides an image processing method, which can solve the problem that the pseudo-details generated after an image is reconstructed by the conventional method are too much.
In a first aspect, an embodiment of the present application provides an image processing method, including:
acquiring an image to be reconstructed;
extracting a high-frequency component and a low-frequency component of the image to be reconstructed to obtain a first high-frequency image composed of the high-frequency component and a first low-frequency image composed of the low-frequency component, wherein the high-frequency component is a frequency component larger than or equal to a preset frequency threshold, and the low-frequency component is a frequency component smaller than the preset frequency threshold;
inputting the first high-frequency image into a trained high-frequency image generation network to obtain a second high-frequency image output by the high-frequency image generation network;
inputting the first low-frequency image into a trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;
and generating a reconstructed image according to the second high-frequency image and the second low-frequency image, wherein the resolution of the reconstructed image is higher than that of the image to be reconstructed.
In a second aspect, an embodiment of the present application provides an image processing apparatus, including:
the image acquisition unit to be reconstructed is used for acquiring an image to be reconstructed;
the high-low frequency image extraction unit is used for extracting a high-frequency component and a low-frequency component of the image to be reconstructed to obtain a first high-frequency image consisting of the high-frequency component and a first low-frequency image consisting of the low-frequency component, wherein the high-frequency component is a frequency component which is greater than or equal to a preset frequency threshold, and the low-frequency component is a frequency component which is less than the preset frequency threshold;
a second high-frequency image generating unit, which is used for inputting the first high-frequency image into the trained high-frequency image generating network to obtain a second high-frequency image output by the high-frequency image generating network;
the second low-frequency image generation unit is used for inputting the first low-frequency image into the trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;
and the image reconstruction unit is used for generating a reconstructed image according to the second high-frequency image and the second low-frequency image, and the resolution of the reconstructed image is higher than that of the image to be reconstructed.
In a third aspect, an embodiment of the present application provides a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to the first aspect when executing the computer program.
In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the method according to the first aspect.
In a fifth aspect, the present application provides a computer program product, which when run on a terminal device, causes the terminal device to execute the method of the first aspect.
Compared with the prior art, the embodiment of the application has the advantages that:
because the high-frequency component and the low-frequency component of the image are processed separately, the noise and the artifact can be suppressed while the details are enhanced, and a hyper-resolution image with less noise and artifact and clearer details can be reconstructed.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the embodiments or the description of the prior art will be briefly described below.
FIGS. 1(b) and 1(a) are schematic diagrams comparing a reconstructed image with an original image provided by the prior art;
fig. 2 is a flowchart of an image processing method according to an embodiment of the present application;
FIG. 3 is a flowchart of another image processing method according to an embodiment of the present disclosure;
fig. 4 is a schematic flowchart of a method for generating a network training according to an embodiment of the present application;
FIG. 5 is a schematic structural diagram of a high-frequency image generation network according to an embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of a discriminant network to be trained according to an embodiment of the present disclosure;
fig. 7 is a schematic flowchart of another method for generating a network training according to an embodiment of the present application;
fig. 8 is a block diagram of an image processing apparatus according to a second embodiment of the present application;
fig. 9 is a schematic structural diagram of a terminal device according to a third embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to" determining "or" in response to detecting ". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," "fourth," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.
Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.
The first embodiment is as follows:
at present, image super-resolution improvement based on a deep learning method is usually performed by aiming at a model structure or a training method, and the method has a good effect when super-resolution processing is performed on a second image obtained by simulation, but the effect is remarkably reduced when the second image obtained really is processed. In order to solve the technical problem, when the high-frequency image generation network and the low-frequency image generation network are trained, the adopted training samples directly select the real second image and the real first image which are acquired by the terminal device, so that the robustness of the trained high-frequency image generation network and the trained low-frequency image generation network in the process of performing the hyper-resolution processing on the real second image is ensured. In addition, when the high-frequency image generation network and the low-frequency image generation network are trained, the high-frequency component and the low-frequency component of the second image and the first image are respectively extracted, and then the high-frequency image consisting of the high-frequency component and the low-frequency image consisting of the low-frequency component are processed. Namely, the high-frequency image generated by the network is generated according to the trained high-frequency image, and the low-frequency image generated by the network is generated according to the trained low-frequency image, so that the hyper-resolution image with less noise and artifact and clearer detail can be reconstructed.
How to reconstruct the image to be reconstructed by using the trained generating network is described in detail below, where the generating network includes a high-frequency image generating network and a low-frequency image generating network.
Fig. 2 shows a flowchart of an image processing method provided in an embodiment of the present application, which is detailed as follows:
step S21, acquiring an image to be reconstructed;
in this embodiment, an image to be reconstructed acquired by the terminal device is acquired, and the resolution of the image to be reconstructed is low. Specifically, an image captured or captured by a terminal device (such as a mobile phone) may be used as the image to be reconstructed, or an image captured and transmitted by another terminal device may be used as the image to be reconstructed.
Step S22, extracting a high-frequency component and a low-frequency component of the image to be reconstructed to obtain a first high-frequency image composed of the high-frequency component and a first low-frequency image composed of the low-frequency component, wherein the high-frequency component refers to a frequency component greater than or equal to a preset frequency threshold, and the low-frequency component refers to a frequency component smaller than the preset frequency threshold;
specifically, high-frequency components and low-frequency components of an image to be reconstructed are extracted by a filter. For example, a low-pass filter (e.g., a gaussian low-pass filter) is used to extract a low-frequency component of the image to be reconstructed, and then the low-frequency component is subtracted from the image to be reconstructed to obtain a high-frequency component of the image to be reconstructed. In some embodiments, an averaging filter may also be employed to extract high and low frequency components of the image to be reconstructed.
If a gaussian low-pass filter is used to extract the low-frequency component, it can be extracted by:
XD＝wL*X (1)
wherein ". x" denotes a convolution operation, wLRepresenting a Gaussian low-pass kernel, X representing the image to be reconstructed, XDRepresenting the low frequency component for X. The wLCan adopt 3x3 or can adopt 5x5, as an example, when wLAt the time of 5x5, the thickness of the film,
after the low frequency component is extracted, the corresponding high frequency component is obtained by subtracting the low frequency component from the original image:
XH＝X-XD(3)
wherein, XHRepresenting the high frequency components of X.
Step S23, inputting the first high-frequency image into the trained high-frequency image generation network to obtain a second high-frequency image output by the high-frequency image generation network;
step S24, inputting the first low-frequency image into the trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;
step S25, generating a reconstructed image according to the second high frequency image and the second low frequency image, where the resolution of the reconstructed image is higher than the resolution of the image to be reconstructed.
Specifically, the data corresponding to the second high-frequency image is used as the high-frequency data of the reconstructed image, and the data corresponding to the second low-frequency image is used as the low-frequency data of the reconstructed image, so that the reconstructed image is obtained.
In the embodiment of the application, because the high-frequency component and the low-frequency component of the image are processed separately, the noise and the artifact can be suppressed while the details are enhanced, so that a super-resolution image with less noise and artifact and clearer details can be reconstructed.
Fig. 3 is a flowchart illustrating another image processing method provided in an embodiment of the present application, in this embodiment, in order to reduce the operation consumption of the network and increase the processing speed, only data of the Y channel of the image to be reconstructed is processed, which is detailed as follows:
step S31, acquiring an image to be reconstructed;
step S32, converting the color space of the image to be reconstructed into a YUV color space, extracting a high frequency component and a low frequency component of a Y channel of the image to be reconstructed, to obtain a first high frequency image composed of the high frequency component of the Y channel, and a first low frequency image composed of the low frequency component of the Y channel, where the high frequency component is a frequency component greater than or equal to a preset frequency threshold, and the low frequency component is a frequency component less than the preset frequency threshold.
In this embodiment, the color space of the image to be reconstructed is usually a red, green and blue (RGB) color space, and in order to reduce the influence of the processing process on the color of the image, the RGB color space is converted into a luminance and chrominance (YUV) color space. After converting into YUV color space, extracting the Y channel of the YUV color space, and if NV21 standard parameters are adopted, extracting the Y channel according to the following formula (4):
in the step, a Y channel of an image to be reconstructed is extracted according to a formula (4) to obtain a gray image corresponding to data of the Y channel, and then low-frequency components and high-frequency components corresponding to the gray image are extracted according to the formulas (1) to (3) to further obtain a first low-frequency image composed of the extracted low-frequency components and a first high-frequency image composed of the corresponding high-frequency components.
Step S33, inputting the first high-frequency image into the trained high-frequency image generation network to obtain a second high-frequency image output by the high-frequency image generation network;
step S34, inputting the first low-frequency image into the trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;
step S35, generating a reconstructed image according to the second high frequency image and the second low frequency image, where the resolution of the reconstructed image is higher than the resolution of the image to be reconstructed.
In the embodiment of the application, only the data of the Y channel of the image to be reconstructed is extracted, so that the data amount required to be processed by the high-frequency image generation network and the low-frequency image generation network is reduced, the running consumption of the network can be effectively reduced, and the processing speed is increased. Moreover, the RGB color space is converted into the YUV color space, so that the problem of color cast caused by directly processing the data of the RGB three channels can be avoided.
The method for training a generation network provided in the embodiment of the present application will be described below, where the generation network includes a high-frequency image generation network and a low-frequency image generation network, and see fig. 4 for details:
step S41, a training sample set is obtained, where the training sample set stores: the method comprises the steps that a first image and a second image which correspond to the first image in a one-to-one mode are acquired by terminal equipment, wherein the resolution of the first image is higher than that of the second image;
wherein, this terminal equipment includes: mobile phones, cameras, tablet computers and the like.
In this embodiment, the first image and the second image stored in the training sample set are both real images acquired by the terminal device, that is, it is ensured that the samples used for training have real noise. The mapping table may be stored after the correspondence between the first image and the second image is determined, and then the second image corresponding to the first image may be determined by looking up the mapping table, or the first image corresponding to the second image may be determined. In order to make the trained high-frequency image generation network and low-frequency image generation network have universality, the diversity of the sample needs to be enriched as much as possible, for example, the sample comprises both an image with rich background and an image with single background; both well-lit and low-lit images; including both normally exposed and abnormally exposed images, and the like.
In addition, in order to make the difference between the content of the image included in the first image and the content included in the second image corresponding to the first image smaller, it is necessary to ensure that the time difference between the time when the terminal device acquires the first image and the time when the terminal device acquires the second image is sufficiently small.
Step S42, acquiring a first image from the training sample set, and extracting a third high-frequency image composed of high-frequency components of the first image and a third low-frequency image composed of low-frequency components of the first image;
the high-frequency component refers to a frequency component greater than or equal to a preset frequency threshold, and the low-frequency component refers to a frequency component smaller than the preset frequency threshold.
Specifically, high-frequency components and low-frequency components of the first image are extracted by a filter. For example, a low-pass filter (e.g., a gaussian low-pass filter) is used to extract the low-frequency component of the first image, and the first image is subtracted from the low-frequency component to obtain the high-frequency component of the first image.
Step S43, acquiring a second image corresponding to the first image from the training sample set, and extracting a fourth high-frequency image composed of high-frequency components of the second image and a fourth low-frequency image composed of low-frequency components of the second image;
the extraction of the low frequency component and the high frequency component of the second image is similar to the extraction of the low frequency component and the high frequency component of the first image in the above step, and details are not repeated here.
Step S44, inputting the fourth high-frequency image into a high-frequency image generation network to be trained to obtain a fifth high-frequency image output by the high-frequency image generation network to be trained;
step S45, if a change in an output value of a loss function of the high-frequency image generation network to be trained is within a preset range, stopping training of the high-frequency image generation network to be trained to obtain the trained high-frequency image generation network, otherwise, adjusting parameters of the high-frequency image generation network to be trained, and returning to execute a step of inputting the fourth high-frequency image into the high-frequency image generation network to be trained and subsequent steps, wherein a loss function of the high-frequency image generation network to be trained includes a discriminant network to be trained, the discriminant network to be trained is trained by the loss function of the discriminant network to be trained, and inputs of the loss function of the discriminant network to be trained include the third high-frequency image and the fifth high-frequency image;
the high-frequency image generation network and the discrimination network form a generation countermeasure network system, namely the high-frequency image generation network and the discrimination network are trained mutually.
In this embodiment, if the absolute value difference between adjacent output values of the loss function of the high-frequency image generation network to be trained is within a preset range (the preset range is a smaller range, for example, [0, 5]), it indicates that the loss function reaches a convergence state, and at this time, training of the high-frequency image generation network to be trained is stopped, that is, the high-frequency image generation network trained for the last time is used as the high-frequency image generation network after training. Of course, if the absolute value difference between the adjacent output values of the loss function is not within the preset range, the steps after step S42, step S43 and step S43 are performed.
In some embodiments, the loss function of the high frequency image generation network to be trained includes 3 parts L representing the loss of contentcontentL showing visual differencesperceptualAnd L of semantic losstextureOf course, if the loss function of the high-frequency image generation network to be trained includes 3 parts listed above, the variation of the output value of the loss function within a preset range may mean that the sum of the variations of the 3 parts is within a preset range, or that the variation of any one part is within a preset range, such as LcontentL in a predetermined rangeperceptualA change in the output value of (a) within a preset range, and LtextureIs within a preset range.
Wherein G isHGenerating a network, x, for the high-frequency image to be trainedH (i)Representing the ith fourth high frequency image,
denotes the ith fifth high frequency image, m denotes the number of the fourth high frequency images, GTH (i)Representing the ith third high frequency image, "| | | |" is L1 norm it is noted that in some embodiments, the L1 norm may also be transformed to a L2 norm.
LperceptualThe visual difference between the input image and the output image representing the high-frequency image generation network, where φ () represents the feature (such as the feature of the relu3_3 layer, or the feature of the relu4_3 layer, or the feature of the relu2_2 layer, or the feature of the relu1_2 layer, etc.) obtained by the image passing through the VGG network (just in Johnson et al, 'Perceptional L oss for Real-time Style Transfer and Super-Resolution' in ECCV 2016), C, H, W represents x, respectivelyH (i)The number of channels, the height and the width.
In some embodiments, the VGG network derived features may also be derived by learning the Perceptual Image Patch Similarity (L earned Perceptual Image Patch Similarity, L PIPS).
LtextureRepresenting a discriminant network (D) to be trainedH) Judge the generated the firstThe score of the five high frequency images is closer to the real image GTHThe higher the score obtained. Because a feature map is obtained through the discriminant network to be trained, the loss function (loss) of semantic loss of a single map is the result of feature map averaging, namely mean () in a formula represents averaging the feature map.
Loss function (loss) L for training discriminant networks to be trainedD：
Step S46, inputting the fourth low-frequency image into a low-frequency image generation network to be trained to obtain a fifth low-frequency image output by the low-frequency image generation network to be trained;
step S47, if the change of the output value of the loss function of the low-frequency image generation network to be trained is within a preset range, stopping training the low-frequency image generation network to be trained to obtain the trained low-frequency image generation network, otherwise, adjusting the parameters of the low-frequency image generation network to be trained, and returning to execute step S46 and step S47, wherein the input of the loss function of the low-frequency image generation network to be trained includes the third low-frequency image and the fifth low-frequency image.
Wherein the loss function of the low-frequency image generation network to be trained comprises L 'representing the content loss'contentAnd L 'representing visual differences'perceptualWherein, L'contentAnd L'perceptualReference is made to the above formula (5) and formula (6).
In the embodiment of the application, because the samples of the training high-frequency image generation network and the training low-frequency image generation network are real images collected by the terminal equipment, the images output by the training high-frequency image generation network and the training low-frequency image generation network are more accurate and meet the requirements. In addition, when the high-frequency image generation network and the low-frequency image generation network are trained, the high-frequency component and the low-frequency component of the image are respectively extracted, and then the high-frequency image consisting of the high-frequency component and the low-frequency image consisting of the low-frequency component are processed, so that artifacts generated by image post-processing acquired by terminal equipment (such as a smart phone) and noise left after image compression are mostly concentrated in a low-frequency region of the image, and details which need to be enhanced are concentrated in a high-frequency region of the image, so that the noise and the artifacts can be suppressed while the details are enhanced by separately processing the high-frequency component and the low-frequency component of the image. Namely, the high-frequency image generated by the network is generated according to the trained high-frequency image, and the low-frequency image generated by the network is generated according to the trained low-frequency image, so that the hyper-resolution image with less noise and artifact and clearer detail can be reconstructed.
Fig. 5 is a structural diagram illustrating a trained high-frequency image generation network according to an embodiment of the present application, where in fig. 5, the trained high-frequency image generation network includes: the device comprises a first feature extraction layer, a first feature conversion layer, a magnification layer and a first fusion layer, wherein M represents that the first feature conversion layer has M layers. And the second high-frequency image respectively passes through the first feature extraction layer, the first feature conversion layer, the amplification layer and the first fusion layer, and then a third high-frequency image is obtained.
Specifically, the first feature extraction layer is used for extracting features; the first feature conversion layer is used for combining the extracted features; the amplifying layer is used for amplifying the combined features; the first fusion layer is used for fusing the amplified characteristics of at least 2 channels into one image to be output.
The amplification layer realizes an amplification function through an interpolation algorithm, and the first feature extraction layer, the first feature conversion layer and the first fusion layer realize corresponding functions through a convolution algorithm. Due to the combination of the interpolation algorithm and the convolution algorithm, the third high-frequency image can reduce the checkerboard noise.
In some embodiments, the first feature extraction layer comprises at least 2 layers, and the number of channels of the next layer is greater than the number of channels of the previous layer; the first characteristic conversion layer at least comprises 2 layers, and the number of channels of each layer is kept unchanged; the convolution kernel of the first feature extraction layer and the convolution kernel of the first fusion layer are both larger than the convolution kernel of the first feature conversion layer, and the image scaling ratios corresponding to the first feature extraction layer, the first fusion layer and the first feature conversion layer are all larger than or equal to 1.
Because the convolution kernel of the first feature extraction layer and the convolution kernel of the first fusion layer are both larger than the convolution kernel of the first feature conversion layer, the obtained third high-frequency image has a larger visual field and truer details. In addition, since the image needs to be subjected to the super-resolution processing, it is necessary to ensure that the image scaling cannot be smaller than 1.
The first feature extraction layer is a process of extracting low-level features of the image by using a convolution kernel of a large receptive field, wherein the low-level features comprise gradient, brightness, size relationship and the like. In order to extract more details, the number of layers of the first feature extraction layer is limited to include at least 2 layers. Assuming that the first feature extraction layer has 2 layers, the number of channels of the 2 nd layer is greater than that of the 1 st layer. Table 1 below illustrates specific parameters of the first feature extraction layer in one example.
Table 1:
the first feature conversion layer is a process of obtaining high-level features through nonlinear combination of the low-level features extracted by the first feature extraction layer, and the high-level features comprise structures, shapes and the like. The more the number of layers is, the higher the non-linearity degree of the features is, the more complex image structure can be expressed, and the more the reality of the reconstructed image is improved. Since the smaller the convolution kernel is, the faster the processing speed is, and the main purpose of the first feature conversion layer is to combine the obtained low-level features, it is not necessary to obtain more information through more channels, therefore, the first feature conversion layer is set to select a convolution with a small convolution kernel, a constant size (x1), a constant number of channels, and M times of repetition. Here we choose M ═ 16, and specific parameters can be referred to in table 2.
Table 2:
in which the enlargement layer is not a convolution layer, but an interpolation enlargement layer, which enlarges the width and height of the features obtained by the first feature conversion layer to a desired ratio, for example, to 2 times (x2), and the first fusion layer immediately follows the enlargement layer in order to fuse the features of the plurality of channels into one image output of one channel, and the parameters of the two layers can be referred to table 3.
Table 3:
it should be noted that the structure of the low-frequency image generation network to be trained is similar to that of the high-frequency image generation network to be trained, and specifically, refer to fig. 5.
In some embodiments, the discriminative network to be trained comprises: the image fusion system comprises a second feature extraction layer used for extracting features, a second feature conversion layer used for combining the extracted features, and a second fusion layer used for fusing the features of at least 2 channels into one image to be output, wherein image scaling ratios corresponding to the second feature extraction layer, the second feature conversion layer and the second fusion layer are all smaller than 1.
Specifically, fig. 6 shows a schematic structural diagram of a to-be-trained discrimination network, and in fig. 6, the first high-frequency image and the third high-frequency image are respectively input into the to-be-trained discrimination network, and after passing through the second feature extraction layer, the second feature conversion layer, and the second fusion layer, are input into a loss function of the to-be-trained discrimination network for comparison, so as to determine whether to continue training the to-be-trained discrimination network.
It should be noted that the functions of the second feature extraction layer, the second feature conversion layer, and the second fusion layer are similar to the functions of the first feature extraction layer, the first feature conversion layer, and the first fusion layer, respectively, and are not described herein again.
In this embodiment, since the to-be-trained discrimination network mainly distinguishes the fifth high-frequency image and the third high-frequency image generated by the to-be-trained high-frequency image generation network, it is not necessary to acquire more information favorable for the reality of the constructed image, that is, it is not necessary to set more layers for the second feature extraction layer and the second feature conversion layer, which is also favorable for increasing the operation speed. In addition, the scaling ratios of the images corresponding to the second feature extraction layer, the second feature conversion layer and the second fusion layer are all smaller than 1, so that the operation speed is improved.
Specifically, the number of layers of the second feature extraction layer may be set to 1, and the parameters of the second feature extraction layer may refer to table 4.
Table 4:
the number of layers of the second feature conversion layer may be set to 2, and the parameters of the second feature conversion layer may refer to table 5.
Table 5:
the second fusion layer immediately follows the second feature conversion layer, and the specific parameters thereof can be referred to table 6:
table 6:
fig. 7 is a flowchart illustrating another training method for generating a network according to an embodiment of the present application.
In fig. 7, the resolution of the first picture is higher than the resolution of the second picture. The first picture is converted into a YUV color space through color space conversion operation, data of a Y channel is extracted, high-frequency components and low-frequency components in the Y channel are extracted through a low-pass filter, and a third high-frequency picture composed of the high-frequency components and a third low-frequency picture composed of the low-frequency components are obtained.
And the second picture is also converted into a YUV color space through color space conversion operation, data of a Y channel is extracted, and a high-frequency component and a low-frequency component in the Y channel are extracted through a low-pass filter, so that a fourth high-frequency picture consisting of the high-frequency component and a fourth low-frequency picture consisting of the low-frequency component are obtained. And the fourth high-frequency picture passes through a high-frequency image generation network to be trained to obtain a fifth high-frequency picture output by the high-frequency image generation network to be trained. And the fourth low-frequency picture passes through the low-frequency image generation network to be trained to obtain a fifth low-frequency picture output by the low-frequency image generation network to be trained.
And taking the third low-frequency picture and the fifth low-frequency picture as the input of the formula (5) and the formula (6), calculating the content loss and the visual difference of the low-frequency image generation network, and judging whether to continue training the low-frequency image generation network according to the calculation result.
And taking the third high-frequency picture and the fifth high-frequency picture as the input of the formula (5), the formula (6), the formula (7) and the formula (8), calculating the content loss, the visual difference and the semantic loss of the high-frequency image generation network, calculating the output value of a loss function of the discrimination network, and judging whether to continue training the high-frequency image generation network according to the calculated content loss, the visual difference, the semantic loss and the output value of the loss function of the discrimination network.
Example two:
fig. 8 shows a block diagram of an image processing apparatus according to an embodiment of the present application, which corresponds to the image processing method described in the first embodiment, and only shows portions related to the embodiment of the present application for convenience of description.
Referring to fig. 8, the image processing apparatus 8 includes: an image acquisition unit 81 to be reconstructed, a high-low frequency image extraction unit 82, a second high-frequency image generation unit 83, a second low-frequency image generation unit 84, and an image reconstruction unit 85, wherein:
an image acquisition unit 81 to be reconstructed, configured to acquire an image to be reconstructed;
a high-low frequency image extracting unit 82, configured to extract a high-frequency component and a low-frequency component of the image to be reconstructed, so as to obtain a first high-frequency image composed of the high-frequency component and a first low-frequency image composed of the low-frequency component, where the high-frequency component is a frequency component greater than or equal to a preset frequency threshold, and the low-frequency component is a frequency component smaller than the preset frequency threshold;
a second high-frequency image generating unit 83 for inputting the first high-frequency image into the trained high-frequency image generating network to obtain a second high-frequency image output by the high-frequency image generating network;
a second low-frequency image generating unit 84, configured to input the first low-frequency image into the trained low-frequency image generation network, so as to obtain a second low-frequency image output by the low-frequency image generation network;
an image reconstructing unit 85, configured to generate a reconstructed image according to the second high-frequency image and the second low-frequency image, where a resolution of the reconstructed image is higher than a resolution of the image to be reconstructed.
In some embodiments, in order to reduce the operation consumption of the network and increase the processing speed, only the data of the Y channel of the image is processed, in this case, the high-frequency and low-frequency image extraction unit 82 is specifically configured to:
and converting the color space of the image to be reconstructed into a YUV color space, extracting the high-frequency component and the low-frequency component of the Y channel of the image to be reconstructed, and obtaining a first high-frequency image consisting of the high-frequency component of the Y channel and a first low-frequency image consisting of the low-frequency component of the Y channel.
In the embodiment of the application, because the high-frequency component and the low-frequency component of the image are processed separately, the noise and the artifact can be suppressed while the details are enhanced, so that a super-resolution image with less noise and artifact and clearer details can be reconstructed.
It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.
Example three:
fig. 9 is a schematic structural diagram of a terminal device according to a third embodiment of the present application. As shown in fig. 9, the terminal device 9 of this embodiment includes: at least one processor 90 (only one processor is shown in fig. 9), a memory 91, and a computer program 92 stored in the memory 91 and executable on the at least one processor 90, the processor 90 implementing the steps in any of the various method embodiments described above when executing the computer program 92:
acquiring an image to be reconstructed;
extracting a high-frequency component and a low-frequency component of the image to be reconstructed to obtain a first high-frequency image composed of the high-frequency component and a first low-frequency image composed of the low-frequency component, wherein the high-frequency component is a frequency component larger than or equal to a preset frequency threshold, and the low-frequency component is a frequency component smaller than the preset frequency threshold;
inputting the first high-frequency image into a trained high-frequency image generation network to obtain a second high-frequency image output by the high-frequency image generation network;
inputting the first low-frequency image into a trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;
and generating a reconstructed image according to the second high-frequency image and the second low-frequency image, wherein the resolution of the reconstructed image is higher than that of the image to be reconstructed.
The terminal device 9 may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The terminal device may include, but is not limited to, a processor 90, a memory 91. Those skilled in the art will appreciate that fig. 9 is only an example of the terminal device 9, and does not constitute a limitation to the terminal device 9, and may include more or less components than those shown, or combine some components, or different components, for example, and may further include an input/output device, a network access device, and the like.
The Processor 90 may be a Central Processing Unit (CPU), and the Processor 90 may be other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 91 may in some embodiments be an internal storage unit of the terminal device 9, such as a hard disk or a memory of the terminal device 9, the memory 91 may in other embodiments also be an external storage device of the terminal device 9, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), etc. further, the memory 91 may also comprise both an internal storage unit and an external storage device of the terminal device 9, the memory 91 is used for storing an operating system, applications, a Boot loader (Boot L loader), data and other programs, such as program codes of the computer program, etc. the memory 91 may also be used for temporarily storing data that has been or will be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
An embodiment of the present application further provides a network device, where the network device includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, the processor implementing the steps of any of the various method embodiments described above when executing the computer program.
The embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the steps in the above-mentioned method embodiments.
The embodiments of the present application provide a computer program product, which when running on a mobile terminal, enables the mobile terminal to implement the steps in the above method embodiments when executed.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing apparatus/terminal apparatus, a recording medium, computer Memory, Read-Only Memory (ROM), random-access Memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc. In certain jurisdictions, computer-readable media may not be an electrical carrier signal or a telecommunications signal in accordance with legislative and patent practice.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/network device and method may be implemented in other ways. For example, the above-described apparatus/network device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implementing, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (10)
1. An image processing method, comprising:
acquiring an image to be reconstructed;
extracting a high-frequency component and a low-frequency component of the image to be reconstructed to obtain a first high-frequency image composed of the high-frequency component and a first low-frequency image composed of the low-frequency component, wherein the high-frequency component is a frequency component larger than or equal to a preset frequency threshold, and the low-frequency component is a frequency component smaller than the preset frequency threshold;
inputting the first high-frequency image into a trained high-frequency image generation network to obtain a second high-frequency image output by the high-frequency image generation network;
inputting the first low-frequency image into a trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;
and generating a reconstructed image according to the second high-frequency image and the second low-frequency image, wherein the resolution of the reconstructed image is higher than that of the image to be reconstructed.
2. The image processing method according to claim 1, wherein said extracting high frequency components and low frequency components of the image to be reconstructed to obtain a first high frequency image composed of the high frequency components and a first low frequency image composed of the low frequency components comprises:
and converting the color space of the image to be reconstructed into a YUV color space, extracting the high-frequency component and the low-frequency component of the Y channel of the image to be reconstructed, and obtaining a first high-frequency image consisting of the high-frequency component of the Y channel and a first low-frequency image consisting of the low-frequency component of the Y channel.
3. The image processing method of claim 1, wherein the high frequency image generation network is trained by:
acquiring a training sample set, wherein the training sample set stores: the method comprises the steps that a first image and a second image are acquired by terminal equipment, wherein the first image corresponds to the first image one by one, and the resolution of the first image is higher than that of the second image;
acquiring a first image from the training sample set, and extracting a third high-frequency image formed by high-frequency components of the first image;
acquiring a second image corresponding to the first image from the training sample set, and extracting a fourth high-frequency image formed by high-frequency components of the second image;
inputting the fourth high-frequency image into a high-frequency image generation network to be trained to obtain a fifth high-frequency image output by the high-frequency image generation network to be trained;
if the variation of the output value of the loss function of the high-frequency image generation network to be trained is within a preset range, stopping training the high-frequency image generation network to be trained to obtain the trained high-frequency image generation network, otherwise, adjusting the parameters of the high-frequency image generation network to be trained, and returning to execute the step of inputting the fourth high-frequency image into the high-frequency image generation network to be trained and the subsequent steps, wherein the loss function of the high-frequency image generation network to be trained comprises a discriminant network to be trained, the discriminant network to be trained is trained through the loss function of the discriminant network to be trained, and the input of the loss function of the discriminant network to be trained comprises the third high-frequency image and the fifth high-frequency image.
4. The image processing method of any of claims 1 to 3, wherein the trained high frequency image generation network comprises: the system comprises a first feature extraction layer for extracting features, a first feature conversion layer for combining the extracted features, an amplification layer for amplifying the combined features, and a first fusion layer for fusing the amplified features of at least 2 channels into an image and outputting the image; the amplification layer realizes an amplification function through an interpolation algorithm, and the first feature extraction layer, the first feature conversion layer and the first fusion layer realize corresponding functions through a convolution algorithm.
5. The image processing method of claim 4, wherein the first feature extraction layer comprises at least 2 layers, and the number of channels of a subsequent layer is greater than the number of channels of a previous layer;
the first characteristic conversion layer at least comprises 2 layers, and the number of channels of each layer is kept unchanged;
the convolution kernel of the first feature extraction layer and the convolution kernel of the first fusion layer are both larger than the convolution kernel of the first feature conversion layer, and the image scaling ratios corresponding to the first feature extraction layer, the first fusion layer and the first feature conversion layer are all larger than or equal to 1.
6. The image processing method of claim 3, wherein the loss function of the high frequency image generation network to be trained further comprises L representing a loss of contentcontent：
Wherein G isHGenerating a network, x, for the high-frequency image to be trainedH (i)Denotes the ith fourth high frequency image, m denotes the number of the fourth high frequency images in a batch, GTH (i)Representing the ith third high-frequency image, "| | | |" is a norm of L1.
7. The image processing method of claim 3, wherein the discriminant network to be trained comprises: the image fusion system comprises a second feature extraction layer used for extracting features, a second feature conversion layer used for combining the extracted features, and a second fusion layer used for fusing the features of at least 2 channels into one image to be output, wherein image scaling ratios corresponding to the second feature extraction layer, the second feature conversion layer and the second fusion layer are all smaller than 1.
8. An image processing apparatus characterized by comprising:
the image acquisition unit to be reconstructed is used for acquiring an image to be reconstructed;
the high-low frequency image extraction unit is used for extracting a high-frequency component and a low-frequency component of the image to be reconstructed to obtain a first high-frequency image consisting of the high-frequency component and a first low-frequency image consisting of the low-frequency component, wherein the high-frequency component is a frequency component which is greater than or equal to a preset frequency threshold, and the low-frequency component is a frequency component which is less than the preset frequency threshold;
a second high-frequency image generating unit, which is used for inputting the first high-frequency image into the trained high-frequency image generating network to obtain a second high-frequency image output by the high-frequency image generating network;
the second low-frequency image generation unit is used for inputting the first low-frequency image into the trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;
and the image reconstruction unit is used for generating a reconstructed image according to the second high-frequency image and the second low-frequency image, and the resolution of the reconstructed image is higher than that of the image to be reconstructed.
9. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010207474.9A CN111429371B (en) | 2020-03-23 | 2020-03-23 | Image processing method and device and terminal equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010207474.9A CN111429371B (en) | 2020-03-23 | 2020-03-23 | Image processing method and device and terminal equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111429371A true CN111429371A (en) | 2020-07-17 |
| CN111429371B CN111429371B (en) | 2023-09-29 |
Family
ID=71548666
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010207474.9A Active CN111429371B (en) | 2020-03-23 | 2020-03-23 | Image processing method and device and terminal equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111429371B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113079378A (en) * | 2021-04-15 | 2021-07-06 | 杭州海康威视数字技术股份有限公司 | Image processing method and device and electronic equipment |
| CN113362249A (en) * | 2021-06-24 | 2021-09-07 | 平安普惠企业管理有限公司 | Text image synthesis method and device, computer equipment and storage medium |
| CN113781347A (en) * | 2021-09-13 | 2021-12-10 | Oppo广东移动通信有限公司 | Image processing method, image processing device, electronic equipment and computer readable storage medium |
| US20230021463A1 (en) * | 2021-07-21 | 2023-01-26 | Black Sesame International Holding Limited | Multi-frame image super resolution system |
| CN115769247A (en) * | 2020-07-27 | 2023-03-07 | 华为技术有限公司 | Image enhancement method and device |
| US11790489B2 (en) | 2020-04-07 | 2023-10-17 | Samsung Electronics Co., Ltd. | Systems and method of training networks for real-world super resolution with unknown degradations |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050013509A1 (en) * | 2003-07-16 | 2005-01-20 | Ramin Samadani | High resolution image reconstruction |
| JP2018081461A (en) * | 2016-11-15 | 2018-05-24 | 日本放送協会 | Super resolution apparatus and program |
| CN108154474A (en) * | 2017-12-22 | 2018-06-12 | 浙江大华技术股份有限公司 | A kind of super-resolution image reconstruction method, device, medium and equipment |
| CN109978762A (en) * | 2019-02-27 | 2019-07-05 | 南京信息工程大学 | A kind of super resolution ratio reconstruction method generating confrontation network based on condition |
| CN110310227A (en) * | 2019-06-27 | 2019-10-08 | 电子科技大学 | A kind of image super-resolution rebuilding method decomposed based on high and low frequency information |
| CN110473144A (en) * | 2019-08-07 | 2019-11-19 | 南京信息工程大学 | A kind of image super-resolution rebuilding method based on laplacian pyramid network |
| CN110660038A (en) * | 2019-09-09 | 2020-01-07 | 山东工商学院 | Multispectral image and panchromatic image fusion method based on generation countermeasure network |
| CN110827200A (en) * | 2019-11-04 | 2020-02-21 | Oppo广东移动通信有限公司 | Image super-resolution reconstruction method, image super-resolution reconstruction device and mobile terminal |
-
2020
- 2020-03-23 CN CN202010207474.9A patent/CN111429371B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050013509A1 (en) * | 2003-07-16 | 2005-01-20 | Ramin Samadani | High resolution image reconstruction |
| JP2018081461A (en) * | 2016-11-15 | 2018-05-24 | 日本放送協会 | Super resolution apparatus and program |
| CN108154474A (en) * | 2017-12-22 | 2018-06-12 | 浙江大华技术股份有限公司 | A kind of super-resolution image reconstruction method, device, medium and equipment |
| CN109978762A (en) * | 2019-02-27 | 2019-07-05 | 南京信息工程大学 | A kind of super resolution ratio reconstruction method generating confrontation network based on condition |
| CN110310227A (en) * | 2019-06-27 | 2019-10-08 | 电子科技大学 | A kind of image super-resolution rebuilding method decomposed based on high and low frequency information |
| CN110473144A (en) * | 2019-08-07 | 2019-11-19 | 南京信息工程大学 | A kind of image super-resolution rebuilding method based on laplacian pyramid network |
| CN110660038A (en) * | 2019-09-09 | 2020-01-07 | 山东工商学院 | Multispectral image and panchromatic image fusion method based on generation countermeasure network |
| CN110827200A (en) * | 2019-11-04 | 2020-02-21 | Oppo广东移动通信有限公司 | Image super-resolution reconstruction method, image super-resolution reconstruction device and mobile terminal |
Non-Patent Citations (2)
| Title |
|---|
| 彭晏飞;高艺;杜婷婷;桑雨;訾玲玲;: "生成对抗网络的单图像超分辨率重建方法", 计算机科学与探索, no. 09, pages 1612 - 1619 * |
| 王先傲，林乐平，欧阳宁: "基于生成对抗网络的人脸超分辨率重建算法", vol. 40, no. 01, pages 49 - 53 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11790489B2 (en) | 2020-04-07 | 2023-10-17 | Samsung Electronics Co., Ltd. | Systems and method of training networks for real-world super resolution with unknown degradations |
| CN115769247A (en) * | 2020-07-27 | 2023-03-07 | 华为技术有限公司 | Image enhancement method and device |
| CN113079378A (en) * | 2021-04-15 | 2021-07-06 | 杭州海康威视数字技术股份有限公司 | Image processing method and device and electronic equipment |
| CN113362249A (en) * | 2021-06-24 | 2021-09-07 | 平安普惠企业管理有限公司 | Text image synthesis method and device, computer equipment and storage medium |
| CN113362249B (en) * | 2021-06-24 | 2023-11-24 | 广州云智达创科技有限公司 | Text image synthesis method, text image synthesis device, computer equipment and storage medium |
| US20230021463A1 (en) * | 2021-07-21 | 2023-01-26 | Black Sesame International Holding Limited | Multi-frame image super resolution system |
| US12100120B2 (en) * | 2021-07-21 | 2024-09-24 | Black Sesame Technologies Inc. | Multi-frame image super resolution system |
| CN113781347A (en) * | 2021-09-13 | 2021-12-10 | Oppo广东移动通信有限公司 | Image processing method, image processing device, electronic equipment and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111429371B (en) | 2023-09-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Wang et al. | An experiment-based review of low-light image enhancement methods | |
| CN111429371B (en) | Image processing method and device and terminal equipment | |
| US11151690B2 (en) | Image super-resolution reconstruction method, mobile terminal, and computer-readable storage medium | |
| CN113034358B (en) | Super-resolution image processing method and related device | |
| CN108337551B (en) | Screen recording method, storage medium and terminal equipment | |
| CN111079764B (en) | Low-illumination license plate image recognition method and device based on deep learning | |
| Huang et al. | Towards low light enhancement with raw images | |
| CN112602088B (en) | Method, system and computer readable medium for improving quality of low light images | |
| CN105049718A (en) | Image processing method and terminal | |
| CN110136055B (en) | Super resolution method and device for image, storage medium and electronic device | |
| CN111784570A (en) | Video image super-resolution reconstruction method and device | |
| CN111131688B (en) | Image processing method and device and mobile terminal | |
| WO2023005818A1 (en) | Noise image generation method and apparatus, electronic device, and storage medium | |
| CN110766153A (en) | Neural network model training method and device and terminal equipment | |
| Liao et al. | Efficient image dehazing using boundary conditions and local contrast | |
| Liu et al. | Learning noise-decoupled affine models for extreme low-light image enhancement | |
| JP4241774B2 (en) | Image processing apparatus, image processing method, and program | |
| CN115471413A (en) | Image processing method and device, computer readable storage medium and electronic device | |
| CN111080683B (en) | Image processing method, device, storage medium and electronic equipment | |
| Liu et al. | Blind quality assessment for tone-mapped images based on local and global features | |
| CN111222446B (en) | Face recognition method, face recognition device and mobile terminal | |
| CN110738625B (en) | Image resampling method, device, terminal and computer readable storage medium | |
| CN103077396B (en) | The vector space Feature Points Extraction of a kind of coloured image and device | |
| CN113628148A (en) | Infrared image noise reduction method and device | |
| Sun et al. | Fractal pyramid low-light image enhancement network with illumination information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |