CN115311138A - Image super-resolution method and device - Google Patents
Image super-resolution method and device Download PDFInfo
- Publication number
- CN115311138A CN115311138A CN202210800137.XA CN202210800137A CN115311138A CN 115311138 A CN115311138 A CN 115311138A CN 202210800137 A CN202210800137 A CN 202210800137A CN 115311138 A CN115311138 A CN 115311138A
- Authority
- CN
- China
- Prior art keywords
- image
- domain
- conversion module
- super
- resolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000006243 chemical reaction Methods 0.000 claims abstract description 68
- 238000012549 training Methods 0.000 claims abstract description 50
- 230000006870 function Effects 0.000 claims description 22
- 230000015556 catabolic process Effects 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 18
- 238000006731 degradation reaction Methods 0.000 claims description 18
- 238000013527 convolutional neural network Methods 0.000 claims description 13
- 125000004122 cyclic group Chemical group 0.000 claims description 10
- 230000003042 antagnostic effect Effects 0.000 claims description 5
- 230000004913 activation Effects 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 2
- 235000009508 confectionery Nutrition 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 17
- 230000005540 biological transmission Effects 0.000 description 14
- 238000004891 communication Methods 0.000 description 12
- 238000012545 processing Methods 0.000 description 10
- 238000003860 storage Methods 0.000 description 10
- 230000000694 effects Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 238000009826 distribution Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 238000013434 data augmentation Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The invention relates to an image super-resolution method and device. The image super-resolution method comprises the following steps: acquiring a training data set, wherein the training data set comprises a plurality of image data pairs, and each image data pair comprises a source image corresponding to a source domain and a target image corresponding to a target domain; training the cycle-generating confrontation network model using a training data set: the loop generation confrontation network model comprises a first domain conversion module for converting the image from a source domain to a target domain, and a second domain conversion module for converting the image from the target domain to the source domain; after training is finished, inputting a first image to be processed to a first domain conversion module, and outputting a second image to be processed through the first domain conversion module; and inputting the second image to be processed to a pre-trained super-resolution model, and outputting the super-resolution image through the super-resolution model.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an image super-resolution method and an image super-resolution device.
Background
In image transmission, in order to increase transmission speed and reduce requirements on a transmission network, an original image to be transmitted is compressed on one side of a sending device to reduce the resolution of the original image, so that the pixel quantity to be transmitted is reduced, and then the compressed low-resolution image is sent to a receiving device; after receiving the low-resolution image, the receiving device reconstructs the received low-resolution image by using an image super-resolution technology to restore the original image with high resolution.
The image super-resolution technology is an important task in artificial intelligence and computer vision, and aims to recover a real high-resolution image from a low-resolution image. With the development of deep learning technology in recent years, many image super-resolution methods based on deep learning are proposed and achieve good performance. Existing methods based on deep learning generally employ a fixed degradation manner to generate a low-resolution image from a high-definition image, so as to generate a pair of "low-resolution image-high-resolution image" image for training a super-resolution model. However, real-world images often contain complex degradation, such as complex situations of combination of motion, compression, noise and the like, and when the domain (data distribution) of the image to be processed is different from the domain (data distribution) of the low-resolution image in the previous training of the super-resolution model, the performance of the super-resolution model is sharply reduced, and the super-resolution model is difficult to adapt to the complex degradation in the real scene. Therefore, it is necessary to provide a new image super-resolution method.
Disclosure of Invention
The invention aims to provide a new technical scheme about image super-resolution, which can adapt to complex degradation in a real scene.
According to a first aspect of the present invention, there is provided an image super-resolution method, comprising:
acquiring a training data set, wherein the training data set comprises a plurality of image data pairs, each image data pair comprises a source image corresponding to a source domain and a target image corresponding to a target domain, the source domain is a domain of a first image to be processed, the target domain is a domain of training data of a super-resolution model, and the pixel sizes of the source image and the target image are the same;
training a cycle-generating confrontation network model using the training data set: the loop generation countermeasure network model includes a first domain conversion module for converting an image from a source domain to a target domain, and a second domain conversion module for converting an image from a target domain to a source domain;
after the training is finished, inputting a first image to be processed to a first domain conversion module, and outputting a second image to be processed through the first domain conversion module;
and inputting the second image to be processed to a pre-trained super-resolution model, and outputting the super-resolution image through the super-resolution model.
According to a second aspect of the present invention, there is also provided an image super-resolution device comprising a memory for storing a computer program and a processor for executing the image super-resolution method according to the first aspect of the present invention under the control of the computer program.
The image super-resolution method and the device can convert the domain of the image to be processed based on the loop generation countermeasure network so as to adapt to the trained super-resolution model, thereby effectively improving the performance of the trained super-resolution model.
Other features of the present invention and advantages thereof will become apparent from the following detailed description of exemplary embodiments of the invention, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a schematic diagram of an application scenario of the image super-resolution method according to the embodiment of the present disclosure;
FIG. 2 is a step diagram of an image super resolution method according to an embodiment;
FIG. 3 is a schematic diagram of a cycle generating confrontation network model, according to an embodiment;
4 (a) -4 (b) are schematic diagrams of a cycle comparison learning process according to an embodiment;
FIG. 5 is a schematic diagram of an image super resolution process according to an embodiment;
fig. 6 is a schematic diagram of a hardware configuration of an image super-resolution device according to an embodiment.
Detailed Description
Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
Techniques, methods and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be discussed further in subsequent figures.
< hardware configuration >
Fig. 1 is a schematic view of an application scenario to which the image super-resolution method according to the embodiment of the present disclosure may be applied.
As shown in fig. 1, the application scenario includes an image sending device 1000 and an image receiving device 2000, and the image sending device 1000 and the image receiving device 2000 are connected in a wired or wireless communication manner for image transmission.
When transmitting an image, the image sending apparatus 1000 may compress an original image to be transmitted to obtain a compressed low-resolution image, so as to reduce the amount of pixels to be transmitted; the compressed low-resolution image is then transmitted to the image receiving apparatus 2000. After receiving the low-resolution image, the image receiving apparatus 2000 reconstructs an original image with high resolution from the low-resolution image based on the image super-resolution method according to the embodiment of the present disclosure, and completes image restoration.
The image transmitted by the image transmitting apparatus 1000 may be an image of each frame of a video, or may be any other image, and is not limited herein. In the case where the image transmission apparatus 1000 transmits a video file including a plurality of frames of images, the image receiving device 2000 can obtain a high-resolution video file for playing after completing image restoration.
The image transmission apparatus 1000 may include a processor 1100, a memory 1200, an interface device 1300, and a communication device 1400.
The image receiving apparatus 2000 may also include a processor 2100, a memory 2200, an interface device 2300, and a communication device 2400.
The processors 1100, 1200 are used to execute computer programs, which may be written in instruction sets of architectures such as x86, arm, RISC, MIPS, SSE, and the like. The memory 1200, 2200 includes, for example, a ROM (read only memory), a RAM (random access memory), a nonvolatile memory such as a hard disk, and the like. The interface devices 1300, 2300 include, for example, a USB interface, a video interface, a network interface, and the like. The communication devices 1400 and 2400 can perform wired or wireless communication, for example, the communication device 1400 may include at least one short-range communication module, for example, any module that performs short-range wireless communication based on short-range wireless communication protocols such as the Hilink protocol, wiFi (IEEE 802.11 protocol), mesh, bluetooth, zigBee, thread, Z-Wave, NFC, UWB, liFi, and the like, and the communication device 1400 may also include a long-range communication module, for example, any module that performs WLAN, GPRS, 2G/3G/4G/5G long-range communication.
The memory 1200 of the image transmission apparatus 1000 is used to store a computer program for controlling the processor 1100 to operate to perform at least the following steps: compressing an original image to be transmitted to obtain a compressed low-resolution image; and transmitting the compressed low-resolution image to an image receiving device. The skilled person can design a computer program according to the method steps and how the computer program controls the processor to operate, which is well known in the art and therefore not described in detail here.
The memory 2200 of the image receiving apparatus 2000 is configured to store a computer program for controlling the processor 2100 to operate to execute an image super-resolution method according to any embodiment of the present disclosure to reconstruct a high-resolution original image from a received low-resolution image, completing image restoration. The skilled person can design a computer program according to the method steps and how the computer program controls the processor to operate, which is well known in the art and therefore not described in detail here.
The image transmitting apparatus 1000 and the image receiving apparatus 2000 may be any electronic apparatuses with image processing capability, for example, any types of user terminal apparatuses, servers, and the like, and are not limited herein.
< method example >
The embodiment of the disclosure provides an image super-resolution system, which comprises two parts, namely a loop generation confrontation network model and a pre-trained super-resolution model. By using the image super-resolution system, the first to-be-processed image with low resolution can be restored to obtain a super-resolution image corresponding to the first to-be-processed image.
Referring first to fig. 3, a cycle generation countermeasure network model provided by an embodiment of the present disclosure is described.
The loop generation countermeasure network model includes a first domain conversion module, a second domain conversion module, a first domain discriminator, and a second domain discriminator.
The output of the first domain conversion module is connected with the input of the second domain conversion module, and the output of the second domain conversion module is connected with the input of the first domain conversion module to form a circulation network. The first domain conversion module is used for converting the image from a source domain to a target domain, and the second domain conversion module is used for converting the image from the target domain to the source domain.
An input of the first domain discriminator is connected to an input of the first domain conversion module and to an output of the second domain conversion module, and an input of the second domain discriminator is connected to an output of the first domain conversion module and to an input of the second domain conversion module. The first domain discriminator is for discriminating an ability of the second domain conversion module to convert the image from the target domain to the source domain. The second domain discriminator is for discriminating an ability of the first domain conversion module to convert the image from the source domain into the target domain.
In one example, the first domain conversion module and the second domain conversion module employ a first convolutional neural network of the same structure. For example, the first convolutional neural network includes 10 convolutional layers, where the first 9 convolutional layers have 64 3x3x64 convolutional kernels and the last 1 convolutional layer has 13 x3x64 convolutional kernel. Each convolutional layer of the first convolutional neural network uses a ReLU (Linear rectification) function as an activation function. The first domain conversion module and the second domain conversion module do not share parameters in training, and the first domain conversion module after training is used for performing domain conversion on the first image to be processed.
In one example, the first domain discriminator and the second domain discriminator employ a second convolutional neural network of the same structure. For example, the second convolutional neural network includes 5 convolutional layers with a step size of 1. The convolutional layers except the last convolutional layer of the second convolutional neural network adopt a Leaky RecU (Leaky Rectified Linear Unit) function as an activation function.
Referring to fig. 2, 3 and 5, a method for providing super-resolution of an image according to an embodiment of the present disclosure will be described. The image super-resolution method includes steps S210-S240.
Step S210, a training data set is obtained. The training data set comprises a plurality of image data pairs, each image data pair comprises a source image corresponding to a source domain and a target image corresponding to a target domain, the source domain is a domain of a first image to be processed, the target domain is a domain of training data of a super-resolution model, and the pixel sizes of the source image and the target image are the same.
In one example, the source image and the target image are generated based on the first image to be processed. In one example, the acquiring of the training data set includes steps S310-314.
In step S310, a plurality of original images are cut from the first image to be processed, and a plurality of image groups are constructed by using the plurality of original images.
In one example, a plurality of original images may be cut out from the first image to be processed in a random cropping manner, and the original images may be subjected to data expansion to obtain a larger number of original images. The data augmentation mode comprises randomly selecting 90 degrees, 180 degrees and 270 degrees for rotation, randomly turning left and right, randomly turning up and down and the like.
Each image group comprises a first original image and a second original image respectively, and the pixel size of the first original image of the same group is smaller than that of the second original image. In one example, the aspect ratio of the first original image and the second original image of the same group is the same.
In step S312, the second original image is input to the degradation model, and the degradation image is output through the degradation model. The domain of the degraded image is the domain of the training data of the super-resolution model, and the pixel size of the degraded image is the same as the pixel size of the first original image.
In one example, the degradation model is a degradation model used when generating the training data of the super-resolution model, and thus, the domain of the degraded image output by the degradation model is the domain of the training data of the super-resolution model.
And step S314, taking the first original image as a source image and the degraded image as a target image to construct an image data pair.
Finally, the image data pair consisting of the source image and the target image can be described by equation (1):
where Di represents the ith image data pair in the training dataset,
representing the i-th degraded image,
representing the ith source image, N represents training dataTotal number of image data pairs in the set.
Step S220, training the circularly generated confrontation network model by using the training data set.
The image domain distribution can be learned by a large number of intra-block countermeasures of the image itself. The embodiment of the disclosure adopts a loop generation confrontation network model, a first domain conversion module is used for converting an image from a source domain to a target domain, a second domain conversion module is used for converting the image from the target domain to the source domain, and the first domain conversion module and the second domain conversion module are trained in a loop confrontation mode.
In one example, the training of the cycle-generated confrontation network model using the training data set includes steps S410-414.
Step S410, inputting a source image into the first domain conversion module to obtain a first intermediate image. Then, the first intermediate image is input to a second domain conversion module to obtain a second intermediate image.
Step S412, the target image is input to the second domain conversion module, and a third intermediate image is obtained. And then, inputting the third intermediate image into the first domain conversion module to obtain a fourth intermediate image.
In step S414, the source image and the third intermediate image are input to the first domain discriminator, and the target image and the first intermediate image are input to the second domain discriminator.
The computation loop generates a total loss against the network model, and the training is stopped when the total loss falls below a preset threshold.
In one example, training for the cycle generation confrontation network model first uses the confrontation loss L gan And (6) carrying out constraint. In one example, the loss L is combated gan Determined according to equation (2):
L gan ＝E[D T (x t )-1] 2 +E[D S (x s )-1] 2 formula (2)
Wherein E represents the expectation operation, D S Denotes a first domain discriminator, D T Denotes a second domain discriminator, x s Representing a source image, x t RepresentAn object image.
In calculating the antagonistic loss L gan Before, the first domain discriminator D needs to be completed S And a second domain discriminator D T The training of (3).
Wherein the loss function used to train the first domain discriminator
Can be represented by formula (3):
wherein the loss function used for training the second domain discriminator
Can be represented by formula (4):
wherein E represents the desired operation, D S Denotes a first domain discriminator, D T Representing a second domain discriminator, DAM 1 Representing a first domain conversion module, DAM 2 Denotes a second domain conversion module, x s Representing a source image, x t Representing the target image.
Using antagonistic losses L only gan The first domain conversion module and the second domain conversion module are more distorted, and the first domain conversion module and the second domain conversion module need to be further constrained to maintain the integrity of the image structure, so the embodiment of the disclosure further introduces the loop consistency loss L cycle 。
In one example, the cycle consistency penalty L cycle Including the pixel-by-pixel penalty between the source image and the second intermediate image and the pixel-by-pixel penalty between the target image and the fourth intermediate image, i.e. the sum of these two penalties. In one example, L1 loss (mean absolute error loss) or L2 loss (absolute mean square error loss) is usedThe direct output of the first domain conversion module and the second domain conversion module is constrained, that is, the pixel-by-pixel loss between the source image and the second intermediate image is the mean absolute error loss or the absolute mean square error loss, and the pixel-by-pixel loss between the destination image and the fourth intermediate image is the mean absolute error loss or the absolute mean square error loss.
In one example, the cycle consistency penalties L cycle Determined according to equation (5):
L cycle ＝||x t -DAM 1 (DAM 2 (x t ))|| 1 +||x s -DAM 2 (DAM 1 (x s ))|| 1 formula (5)
Wherein, "| | I Lily 1 "denotes the L1 loss function or the L2 loss function, DAM 1 Representing a first domain conversion module, DAM 2 Denotes a second domain conversion module, x s Representing a source image, x t Representing the target image.
In one example, a cyclic contrast loss L is further introduced cll . Loss of cyclic contrast L cll The method is a loss caused by contrast learning, and the purpose of the contrast learning is to shorten the distance between a positive sample and a current output result and to lengthen the distance between a negative sample and the current output result in a vector characterization space. Referring to fig. 4 (a) and 4 (b), the input and output of 2 domain conversion modules are subjected to circular contrast training. The first domain conversion module inputs the third intermediate image and outputs the fourth intermediate image, the second domain conversion module inputs the first intermediate image and outputs the second intermediate image, and the contrast loss L is reduced in the cycle cll May be moved away from the third intermediate image (negative examples) and closer to the target image (positive examples), and may be moved away from the first intermediate image (negative examples) and closer to the target image (positive examples).
In one example, the loss of contrast L of the cycle cll Determined according to equation (6):
wherein, "| | I Lily F "is the matrix norm. F i And extracting functions for the features of the ith layer of the pre-trained deep convolutional neural network, wherein the functions are used for extracting the depth features to distinguish the domain differences among different images. The deep convolutional neural network is another network that has been trained, independent of the cycle-generated countermeasure network model of the embodiments of the present disclosure. The deep convolutional neural network and the ith layer can be determined by a person skilled in the art according to actual requirements. In the disclosed embodiment, the deep convolutional neural network may be, for example, a VGG19 network, and the i-th layer may be, for example, layers 5-10 of the VGG19 network. x is the number of s Representing a source image, x t A representation of the target image is shown,
a first intermediate image is represented which is,
a second intermediate image is represented which is,
a third intermediate image is represented which is,
representing a fourth intermediate image.
In one example, the loop generates a total loss L against the network model all Determined according to equation (7):
l all ＝L gan +λ 1 *L cycle +λ 2 *L cll formula (7)
Wherein L is all Total loss to cycle the antagonistic network model, L gan To combat losses, L cycle For cyclic consistency loss, L cll For cyclic contrast loss, λ 1 And λ 2 Is a preset hyper-parameter.
Step S230, after the training is completed, inputting the first image to be processed to the first domain conversion module, and outputting the second image to be processed through the first domain conversion module.
After being trained, the first domain conversion module has the capability of converting the first image to be processed from its source domain into the target domain, i.e. from its source domain into the domain of the training data of the super-resolution model. Therefore, the domain of the second image to be processed output by the first domain conversion module is the target domain.
And S240, inputting the second image to be processed to the pre-trained super-resolution model, and outputting the super-resolution image through the super-resolution model.
The domain of the second image to be processed is a target domain, namely the domain of the training data of the super-resolution model, so that the super-resolution model can well reconstruct the second image to be processed into the super-resolution image.
According to the image super-resolution method, the domain of the image to be processed is converted based on the loop generation countermeasure network so as to adapt to the trained super-resolution model, and therefore the performance of the trained super-resolution model is effectively improved.
According to the image super-resolution method, in the training process of the cyclic generation type confrontation network, the source domain and the target domain of the image to be processed are subjected to confrontation training, cyclic contrast loss is introduced, and accuracy of domain conversion is guaranteed.
The image super-resolution method of the embodiment of the disclosure can use the image to be processed as self-supervision, and convert the image to be processed into the target domain without losing information.
The image super-resolution method disclosed by the embodiment of the disclosure has a good super-resolution effect on the image to be processed in the real world. According to the image super-resolution method disclosed by the embodiment of the disclosure, for the to-be-processed image obtained by artificial degradation, when the degradation mode is different from that of the training data of the super-resolution model, the cyclic generation type countermeasure network can also convert the to-be-processed image obtained by the different degradation modes into the target domain (the domain of the training data of the super-resolution model), so that the method is suitable for the trained super-resolution model, and a good recovery effect is obtained.
A large number of experiments prove that the super-resolution method disclosed by the embodiment of the disclosure can be used for widely improving the performance of the existing super-resolution model and the visual quality of a super-resolution image. The inventor uses a large number of test images to test the super-resolution method of the embodiment of the disclosure on a Y channel of a YCbCr space, and evaluation indexes use peak signal-to-noise ratio and structural similarity, so that good effects are obtained on the two evaluation indexes. In addition, the super-resolution method of the embodiment of the disclosure can achieve a good visual effect by performing a visual effect test on the test image and the super-resolution image finally output by using the super-resolution method. The super-resolution method has a good effect on recovering the banded edge information in the image. The super-resolution method disclosed by the embodiment of the disclosure can effectively recover clear and natural textures, and meanwhile, obvious visual misalignment and fake information cannot be introduced.
In one embodiment, the image super-resolution method of any of the above embodiments can be applied to, but is not limited to, the field of image transmission. The method is applied to the field of image transmission, an original high-definition image can be compressed at one end of an image sending device, a low-resolution image is obtained by compressing the resolution of the original high-definition image, then the low-resolution image is transmitted to an image receiving device in a wired or wireless mode, the image super-resolution method of any embodiment is implemented at one side of the image receiving device, the super-resolution image recovered by the low-resolution image is obtained, and the super-resolution image can be very similar to the original high-definition image. Thus, the transmission speed can be increased without affecting the effect of using the transmitted image on the image receiving apparatus side.
< apparatus embodiment >
Fig. 6 is a block schematic diagram of an image super-resolution device according to an embodiment. As shown in fig. 6, the image super-resolution apparatus 800 may include a processor 810 and a memory 820, the memory 820 being used for storing a computer program, the processor 810 being used for executing an image super-resolution method according to any embodiment of the present disclosure under the control of the computer program.
The image super-resolution apparatus 800 may be included in the image receiving device 2000 shown in fig. 1, or may be any other device capable of performing image processing, and is not limited herein.
The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therewith for causing a processor to implement various aspects of the present invention.
The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as a punch card or an in-groove protruding structure with instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be interpreted as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or an electrical signal transmitted through an electrical wire.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.
Computer program instructions for carrying out operations of the present invention may be assembler instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), with state information of computer-readable program instructions, which can execute the computer-readable program instructions.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.
Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or technical improvements to the market, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.
Claims (10)
1. An image super-resolution method, comprising:
acquiring a training data set, wherein the training data set comprises a plurality of image data pairs, each image data pair comprises a source image corresponding to a source domain and a target image corresponding to a target domain, the source domain is a domain of a first image to be processed, the target domain is a domain of training data of a super-resolution model, and the pixel sizes of the source image and the target image are the same;
training a cycle-generating confrontation network model using the training data set: the loop generation countermeasure network model includes a first domain conversion module for converting an image from a source domain to a target domain, and a second domain conversion module for converting an image from a target domain to a source domain;
after the training is finished, inputting a first image to be processed to a first domain conversion module, and outputting a second image to be processed through the first domain conversion module;
and inputting the second image to be processed to a pre-trained super-resolution model, and outputting a super-resolution image through the super-resolution model.
2. The method according to claim 1, characterized in that said source image and said target image are generated based on a first image to be processed;
the acquiring a training data set comprises:
cutting a plurality of original images from a first image to be processed, and constructing a plurality of image groups by using the plurality of original images, wherein each image group respectively comprises a first original image and a second original image, and the pixel size of the first original image is smaller than that of the second original image;
inputting a second original image into a degradation model, and outputting a degradation image through the degradation model, wherein the domain of the degradation image is the domain of training data of the super-resolution model, and the pixel size of the degradation image is the same as that of the first original image;
and taking the first original image as a source image and the degraded image as a target image to construct an image data pair.
3. The method of claim 1, wherein the loop-generated countermeasure network model includes a first domain evaluator and a second domain evaluator; the first domain discriminator is used for discriminating the capability of the second domain conversion module to convert the image from the target domain to the source domain, and the second domain discriminator is used for discriminating the capability of the first domain conversion module to convert the image from the source domain to the target domain;
the training of a cycle-generated confrontation network model using the training data set includes:
inputting a source image into a first domain conversion module to obtain a first intermediate image;
inputting the first intermediate image into a second domain conversion module to obtain a second intermediate image;
inputting the target image into a second domain conversion module to obtain a third intermediate image;
inputting the third intermediate image into the first domain conversion module to obtain a fourth intermediate image;
inputting the source image and the third intermediate image to a first domain discriminator;
the target image and the first intermediate image are input to a second domain evaluator.
4. The method of claim 1, wherein the total loss of the cycle-generating antagonistic network model is determined according to the following equation:
L all ＝L gan +λ 1 *L cycle +λ 2 *L cll
wherein L is all To total loss, L gan To combat losses, L cycle For cyclic consistency loss, L cll For cyclic contrast loss, λ 1 And λ 2 Is a preset hyper-parameter.
5. The method of claim 4, wherein the antagonistic loss is determined according to the following equation:
L gan ＝E[D T (x t )-1] 2 +E[D S (x s )-1] 2
wherein E represents the expectation operation, D S Denotes a first domain discriminator, D T Denotes a second domain discriminator, x s Representing a source image, x t Representing the target image.
6. The method of claim 4, wherein the cyclical consistency loss comprises a pixel-by-pixel loss between the source image and the second intermediate image, and a pixel-by-pixel loss between the destination image and the fourth image.
7. The method of claim 4, characterized by hindering: the cycle consistency loss is determined according to the following equation:
L cycle ＝||x t -DAM 1 (DAM 2 (x t ))|| 1 +||x s -DAM 2 (DAM 1 (x s ))|| 1
wherein, "| | I Lily 1 "denotes the L1 loss function or the L2 loss function, DAM 1 Representing a first domain conversion module, DAM 2 Denotes a second domain conversion module, x s Representing a source image, x t Representing the target image.
8. The method of claim 4, wherein the cyclic contrast loss is determined according to the following equation:
wherein, "| | | purple sweet F "is the matrix norm; f i Extracting a function for the features of the ith layer of the deep convolutional neural network; x is the number of s Representing a source image, x t A representation of the target image is shown,
a first intermediate image is represented which is,
a second intermediate image is represented which is,
a third intermediate image is represented which is,
representing a fourth intermediate image.
9. The method of claim 1, wherein the first domain conversion module and the second domain conversion module employ a first convolutional neural network of the same structure;
each convolutional layer of the first convolutional neural network employs a ReLU function as an activation function.
10. An image super-resolution device, comprising a memory for storing a computer program and a processor for executing the image super-resolution method according to any one of claims 1 to 9 under the control of the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210800137.XA CN115311138B (en) | 2022-07-06 | 2022-07-06 | Image super-resolution method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210800137.XA CN115311138B (en) | 2022-07-06 | 2022-07-06 | Image super-resolution method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115311138A true CN115311138A (en) | 2022-11-08 |
| CN115311138B CN115311138B (en) | 2023-06-23 |
Family
ID=83856467
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210800137.XA Active CN115311138B (en) | 2022-07-06 | 2022-07-06 | Image super-resolution method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115311138B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020041517A1 (en) * | 2018-08-21 | 2020-02-27 | The Salk Institute For Biological Studies | Systems and methods for enhanced imaging and analysis |
| CN111507909A (en) * | 2020-03-18 | 2020-08-07 | 南方电网科学研究院有限责任公司 | Method and device for clearing fog image and storage medium |
| CN111899172A (en) * | 2020-07-16 | 2020-11-06 | 武汉大学 | Vehicle target detection method oriented to remote sensing application scene |
| CN113160234A (en) * | 2021-05-14 | 2021-07-23 | 太原理工大学 | Unsupervised remote sensing image semantic segmentation method based on super-resolution and domain self-adaptation |
-
2022
- 2022-07-06 CN CN202210800137.XA patent/CN115311138B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020041517A1 (en) * | 2018-08-21 | 2020-02-27 | The Salk Institute For Biological Studies | Systems and methods for enhanced imaging and analysis |
| CN111507909A (en) * | 2020-03-18 | 2020-08-07 | 南方电网科学研究院有限责任公司 | Method and device for clearing fog image and storage medium |
| CN111899172A (en) * | 2020-07-16 | 2020-11-06 | 武汉大学 | Vehicle target detection method oriented to remote sensing application scene |
| CN113160234A (en) * | 2021-05-14 | 2021-07-23 | 太原理工大学 | Unsupervised remote sensing image semantic segmentation method based on super-resolution and domain self-adaptation |
Non-Patent Citations (6)
| Title |
|---|
| HONGYANG ZHOU 等: "Real-World Image Super-Resolution Via Spatio-Temporal Correction Network" * |
| MINGZHENG HOU 等: "Semi-supervised image super-resolution with attention CycleGAN" * |
| XIAOBIN ZHU 等: "Video super-resolution based on a spatio-temporal matching network" * |
| 祝晓斌 等: "A brief survey on deep learning based image super-resolution" * |
| 蔡文郁 等: "基于循环生成对抗网络的超分辨率重建算法研究" * |
| 辛元雪 等: "基于改进增强型超分辨率生成对抗网络的图像超分辨率重建算法" * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115311138B (en) | 2023-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022267641A1 (en) | Image defogging method and system based on cyclic generative adversarial network | |
| Daras et al. | Intermediate layer optimization for inverse problems using deep generative models | |
| US10924755B2 (en) | Real time end-to-end learning system for a high frame rate video compressive sensing network | |
| Wang et al. | Event-driven video frame synthesis | |
| CN109949222B (en) | Image super-resolution reconstruction method based on semantic graph | |
| CN115115918B (en) | Visual learning method based on multi-knowledge fusion | |
| GB2548749A (en) | Online training of hierarchical algorithms | |
| CN116681584A (en) | Multistage diffusion image super-resolution algorithm | |
| KS et al. | Deep multi-stage learning for hdr with large object motions | |
| Khan et al. | Sparse to dense depth completion using a generative adversarial network with intelligent sampling strategies | |
| US20220335560A1 (en) | Watermark-Based Image Reconstruction | |
| CN117441333A (en) | Configurable location for inputting auxiliary information of image data processing neural network | |
| CN115311138B (en) | Image super-resolution method and device | |
| CN116402719A (en) | Human blind face image recovery system and method based on potential diffusion model | |
| CN114119428B (en) | Image deblurring method and device | |
| CN114501031B (en) | Compression coding and decompression method and device | |
| Zhang et al. | Image super-resolution via RL-CSC: when residual learning meets convolutional sparse coding | |
| CN113902647A (en) | Image deblurring method based on double closed-loop network | |
| EP4272437A1 (en) | Independent positioning of auxiliary information in neural network based picture processing | |
| EP4391533A1 (en) | Feature map encoding method and apparatus and feature map decoding method and apparatus | |
| CN116580269B (en) | Method for training model, method for processing image, electronic device and storage medium | |
| US20240221230A1 (en) | Feature map encoding and decoding method and apparatus | |
| Shen et al. | Generating Human Faces from Cartoon Emojis | |
| CN116016954A (en) | Image encoding method, image decoding device, and computer storage medium | |
| CN117956178A (en) | Video encoding method and device, and video decoding method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |