WO2016019484A1 - An apparatus and a method for providing super-resolution of a low-resolution image - Google Patents

An apparatus and a method for providing super-resolution of a low-resolution image Download PDF

Info

Publication number
WO2016019484A1
WO2016019484A1 PCT/CN2014/000755 CN2014000755W WO2016019484A1 WO 2016019484 A1 WO2016019484 A1 WO 2016019484A1 CN 2014000755 W CN2014000755 W CN 2014000755W WO 2016019484 A1 WO2016019484 A1 WO 2016019484A1
Authority
WO
WIPO (PCT)
Prior art keywords
resolution
image
low
sub
ground truth
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2014/000755
Other languages
French (fr)
Inventor
Xiaoou Tang
Chao Dong
Chen Change Loy
Kaiming He
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to PCT/CN2014/000755 priority Critical patent/WO2016019484A1/en
Priority to CN201480082564.2A priority patent/CN106796716B/en
Publication of WO2016019484A1 publication Critical patent/WO2016019484A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Definitions

  • the present application generally relates to a field of image processing, more particularly, to an apparatus and a method for providing super-resolution of a low-resolution image.
  • SR Super-resolution
  • SR is a class of techniques that enhance the resolution of an imaging system.
  • Recent state-of-the-art methods for super-resolution of a single image are mostly example-based. These methods either exploit internal similarities of the same image, or learn mapping functions from external low-resolution and high-resolution exemplar pairs.
  • the external example-based methods are often provided with abundant samples, but are challenged by the difficulties of modeling the data effectively and compactly.
  • CNN Convolutional neural networks
  • an apparatus for providing super-resolution of a low-resolution image may comprise: a patch extracting and representing device comprising a first set of filters configured to extract patches from the low-resolution image and represent each of the extracted patches as a high dimensional vector, a mapping device comprising a second set of filters configured to map nonlinearly each of the high dimensional vectors onto a high-resolution patch-wise representation and an aggregating device configured to aggregate the high-resolution patch-wise representations to generate a high-resolution image for the low-resolution image.
  • a method for providing super-resolution of a low-resolution image may comprise: extracting patches from the low-resolution image and representing each of the extracted patches as a high dimensional vector by a patch extracting and representing device comprising a first set of filters; mapping nonlinearly each of the high dimensional vectors onto a high-resolution patch-wise representation by a mapping device comprising a second set of filters; and aggregating the high-resolution patch- wise representations to generate a high-resolution image.
  • a method for training a convolutional neural network system for providing super-resolution of a low-resolution image may comprise: 1) sampling a low-resolution sub-image and its corresponding ground truth high-resolution sub-image from a predetermined training set; 2) reconstructing the low-resolution sub-image to a high-resolution sub-image by the convolutional neural network system; 3) generating a reconstruction error by comparing dissimilarity between the reconstructed high-resolution sub-image and the ground truth high-resolution sub-image; 4) back-propagating the reconstruction error through the convolutional neural network system so as to adjust weights on connections between neurons of the convolutional neural network system; and repeating steps l)-4) until an average value of the reconstruction error is lower than a preset threshold.
  • an apparatus for providing super-resolution of a low-resolution image may comprise a reconstructing unit configured to reconstruct the low-resolution image to a high-resolution image based on predetermined parameters and a training unit configured to train the convolutional neural network system with a predetermined training set so as to determine the parameters used by the reconstructing unit.
  • the reconstructing unit may comprise: a patch extracting and representing device configured to extract patches from the low-resolution image and represent each of the extracted patches as a high dimensional vector based on the predetermined parameters; a mapping device configured to map nonlinearly each of the high dimensional vectors onto a high-resolution patch-wise representation; and a aggregating device configured to aggregate the high-resolution patch-wise representations to generate a high-resolution image.
  • the patch extracting and representing device, the mapping device and the aggregating device comprise a plurality of convolutional layers, and the plurality of convolutional layers are sequentially connected to each other to form a convolutional neural network system.
  • the present application does not explicitly learn the dictionaries or manifolds for modeling the patch space. These are implicitly achieved via the convolutional layers. Furthermore, the patch extraction and aggregation are also formulated as convolutional layers, so are involved in the optimization. In the method and apparatus of the present application, the entire convolutional neural network is fully obtained through training, with little pre/post-processing. With a lightweight structure, the apparatus and method of the present application have achieved superior performance than the state-of-the-art methods.
  • Fig. 1 is a schematic diagram illustrating an apparatus for providing super-resolution of a low-resolution image consistent with an embodiment of the present application.
  • Fig. 2 is a schematic diagram illustrating an apparatus for providing super-resolution of a low-resolution image consistent with another embodiment of the present application.
  • FIG. 3 is a schematic diagram illustrating a convolutional neural network system, consistent with some disclosed embodiments.
  • Fig. 4. is a schematic diagram illustrating a training unit of the apparatus, consistent with some disclosed embodiments.
  • Fig. 5. is a schematic diagram illustrating a training set preparation device of the training unit, consistent with some disclosed embodiments.
  • Fig. 6 is a schematic diagram illustrating an apparatus for providing super-resolution of a low-resolution image when it is implemented in software, consistent with some disclosed embodiments.
  • Fig. 7 is a schematic flowchart illustrating a method for providing super-resolution of a low-resolution image, consistent with some disclosed embodiments.
  • FIG. 8 is a schematic flowchart illustrating a method for training a convolutional neural network system for providing super-resolution of a low-resolution image, consistent with some disclosed embodiments.
  • FIG. 1 is a schematic diagram illustrating an exemplary apparatus 1000 for providing super-resolution of a low-resolution image consistent with some disclosed embodiments.
  • the apparatus 1000 may comprise a patch extracting and representing device 100, a mapping device 200 and an aggregating device 300.
  • the patch extracting and representing device 100 may comprise a first set of filters configured to extract patches from the low-resolution image and represent each of the extracted patches as a high dimensional vector.
  • the mapping device 200 may comprise a second set of filters configured to map nonlinearly each of the high dimensional vectors onto a high-resolution patch-wise.
  • the aggregating device 300 may be configured to aggregate the high-resolution patch-wise representations to generate a high-resolution image for the low-resolution image.
  • the first set of filters is configured to extract patches from the low-resolution image and represent each of the extracted patches as a high dimensional vector by rule of a first nonlinear function of first parameters, in which the first parameters are determined from predetermined parameters associated with the low-resolution image.
  • the second set of filters is configured to map nonlinearly each of the high dimensional vectors onto a high-resolution patch-wise representation by rule of a second nonlinear function of second parameters, in which the second parameters are determined from predetermined parameters associated with the high dimensional vector.
  • the first set of filters, the second set of filters and the aggregating device as mentioned in the above will be further discussed in detail.
  • the low-resolution image is denoted by F
  • the high-resolution image is denoted by F(Y) which is as similar as possible to a ground truth high-resolution image X.
  • the first set of filters is configured to extract patches from the low-resolution image Y and represent each of the extracted patches as a high dimensional vector.
  • these vectors comprise a set of feature maps, of which the number equals to the dimensionality of the vectors.
  • a popular strategy in image restoration is to densely extract patches and then represent them by a set of pre-trained bases such as PCA (Principal Component Analysis), DCT (Discrete Cosine Transformation), Haar, etc.
  • the first set of filters may be simulated as an operation F .
  • Fj(Y) F' (W ! * Y + Bj) , (1) where W ⁇ and B ⁇ represent the filters and biases respectively.
  • F'(x) is a nonlinear function e.g., max(0,x), tanh(x) or l/(l+exp(-x)).
  • W ⁇ is of a size cx lx 1 xtii, where c is the number of channels in the input image, for example, if the input image is a color image, then c is 3, and i is the spatial size of a filter, and ti ⁇ is the number of filters.
  • W ⁇ applies ti ⁇ convolutions on the image, and each convolution has a kernel size cx/ix i.
  • the output is composed of ti ⁇ feature maps.
  • B ⁇ is an ⁇ -dimensional vector, whose each element is associated with a filter.
  • the second set of filters is configured to map nonlinearly each of the high dimensional vectors onto another high-dimensional vector.
  • the first set of filters extracts an ni-dimensional feature for each patch.
  • the second set of filters maps each of these ni-dimensional vectors into an n 2 -dimensional vector.
  • Each mapped vector is conceptually the representation of a high-resolution patch.
  • the second set of filters may be simulated as an operation F 2 :
  • F 2 (Y) F"(W2*F 1 (Y) + B 2 ), (2) where W 2 is of a size tiixl xl xn 2 , and B 2 is a n 2 -dimensional vector.
  • F"(x) is a nonlinear function e.g., max(0,x), tanh(x) or l/(l+exp(-x)).
  • each of the output n 2 -dimensional vectors is conceptually a representation of a high-resolution patch that will be used for reconstruction.
  • the aggregating device 300 aggregates the high-resolution patch-wise representations to generate a high-resolution image.
  • the aggregating device 300 may be simulated as an operation F3:
  • F(Y) W 3 * F 2 (Y) + B 3 , (3) where W3 is of a size n2Xf. 3 X.f3Xc, and B3 is a c-dimensional vector.
  • the filters may act like an averaging filter. If the representations of the high-resolution patches are in some other domains (e.g., coefficients in terms of some bases), W3 may behave as first projecting the coefficients onto the image domain and then averaging. In either way, W3 is a set of linear filters.
  • the apparatus 1000 may further comprise a comparing device (not shown) which is configured to sample a ground truth high-resolution image corresponding to the low-resolution image from a predetermined training set and compare dissimilarity between the aggregated high-resolution image and the sampled ground truth high-resolution image to generate a reconstruction error.
  • the reconstruction error comprises a mean squared error.
  • the reconstruction error is back-propagated in order to determine the parameters, i.e., Wi t W2 , W3, B ⁇ t B2 , and B3.
  • the apparatus 1000 may further comprise an upscaling unit (not shown), and the upscaling unit is configured to upscale the low-resolution image to a predetermined size.
  • the low-resolution image may be upscaled by using bicubic interpolation.
  • the upscaling is the only pre-processing in the embodiment.
  • Fig. 2 is a schematic diagram illustrating an apparatus 1000' for providing super-resolution of a low-resolution image consistent with another embodiment of the present application.
  • the apparatus 1000' may comprise a reconstructing unit 100' and a training unit 200' .
  • the reconstructing unit 100' is configured to reconstruct the low-resolution image to a high-resolution image based on predetermined parameters.
  • the reconstructing unit 100' may further comprise a patch extracting and representing device 110', a mapping device 120' and a aggregating device 130' .
  • the patch extracting and representing device 110', the mapping device 120' and the aggregating device 130' may connect together to form a convolutional neural network system.
  • Fig. 3 illustrates the layer configuration of the convolutional neural network system in mathematic simulation model.
  • each of the patch extracting and representing device 110', the mapping device 120' and the aggregating device 130' may be simulated as at least one convolutional layer, respectively. Different operations are performed at different convolutional layers, respectively.
  • the patch extracting and representing unit 110' is configured to extract patches from the low-resolution image and represent each of the extracted patches as a high dimensional vector based on the predetermined parameters. This is equivalent to convolving the image by a set of filters as mentioned above.
  • the mapping device 120' is configured to nonlinearly map each of the high dimensional vectors onto a high-resolution patch-wise representation. This is equivalent to applying a second set of filters as mentioned above which have a trivial spatial support lxl. Alternatively, it is possible to add more convolutional layers (whose spatial supports are lxl) to increase the non-linearity. But this can significantly increase the complexity of the convolutional neural network system, and thus demands more training data and time.
  • the aggregating device 130' is configured to aggregate the high-resolution patch-wise representations to generate a high-resolution image.
  • the training unit 200' is configured to train the convolutional neural network system with a predetermined training set so as to determine the parameters, for example W ⁇ , W 2 , W 3 , B B 2 , B 3 , used by the reconstructing unit.
  • the training unit 200' may comprise a sampling device 210', a comparing device 220', and a back-propagating device 230' .
  • the sampling device 210' may be configured to sample a low-resolution sub-image and its corresponding ground truth high-resolution sub-image from a predetermined training set and input the low-resolution sub-image to the convolutional neural network system.
  • “sub-images” means these samples are treated as small “images” rather than “patches”, in the sense that "patches” are overlapping and require some averaging as post-processing but "sub-images" need not.
  • the comparing device 220' may be configured to compare dissimilarity between the reconstructed high-resolution sub-image based on the input low-resolution sub-image from the convolutional neural network system and the corresponding ground truth high-resolution sub-image to generate a reconstruction error.
  • the reconstruction error may comprise a mean squared error, and the error is minimized by using stochastic gradient descent with the standard back propagation.
  • the back-propagating device 230' is configured to back-propagate the reconstruction error through the convolutional neural network system so as to adjust weights on connections between neurons of the convolutional neural network system.
  • the convolutional neural network system do not preclude the usage of other kinds of reconstruction error, if only the reconstruction error are derivable. If a better perceptually motivated metric is given during the training, it is flexible for the convolutional neural network system to adapt to that metric.
  • the training unit 200' may further comprise a training set preparation device configured to prepare the predetermined training set for training the convolutional neural network system.
  • Fig. 5 is a schematic diagram illustrating the training set preparation device of the training unit 200' .
  • the training set preparation device may comprise a cropper 241 ', a low-resolution sub-image generator 242', a pairing device 243' and a collector 244' .
  • the cropper 24 may be configured to crop randomly a plurality of sub-images from a randomly selected training image to generate a set of ground truth high-resolution sub-images. For example, the cropper 24 may crop n sub-images of mxm pixels each.
  • the low-resolution sub-image generator 242' may be configured to generate a set of low-resolution sub-images based on the set of ground truth high-resolution sub-images.
  • the pairing device 243' may be configured to pair each of the ground truth high-resolution sub-images with a corresponding low-resolution sub-image.
  • the collector 244' may be configured to collect all the pairs to form the predetermined training set.
  • the low-resolution sub-image generator 242' may comprise a blurring device, a sampling device and an upscaling device.
  • the blurring device may be configured to blur each of the ground truth high-resolution sub-images by a Gaussian kernel.
  • the sampling device may be configured to sample the blurred ground truth high-resolution sub-image by a predetermined scaling factor.
  • the upscaling device may be configured to upscale the sampled ground truth high-resolution sub-image by a predetermined scaling factor to generate the set of low-resolution sub-images.
  • the apparatus 1000 and 1000' may be implemented using certain hardware, software, or a combination thereof.
  • the embodiments of the present invention may be adapted to a computer program product embodied on one or more computer readable storage media (comprising but not limited to disk storage, CD-ROM, optical memory and the like) containing computer program codes.
  • Fig. 6 is a schematic diagram illustrating an apparatus 1000 and 1000' for providing super-resolution of a low-resolution image when it is implemented in software, consistent with some disclosed embodiments.
  • apparatus 1000 and 1000' may include a general purpose computer, a computer cluster, a mainstream computer, a computing device dedicated for providing online contents, or a computer network comprising a group of computers operating in a centralized or distributed fashion.
  • apparatus 1000 and 1000' may include one or more processors (processors 102, 104, 106 etc.), a memory 112, a storage device 116, and a bus to facilitate information exchange among various devices of apparatus 1000.
  • Processors 102-106 may include a central processing unit (“CPU"), a graphic processing unit (“GPU”), or other suitable information processing devices.
  • processors 102-106 can include one or more printed circuit boards, and/or one or more microprocessor chips. Processors 102-106 can execute sequences of computer program instructions to perform various methods that will be explained in greater detail below.
  • Memory 112 can include, among other things, a random access memory (“RAM”) and a read-only memory (“ROM”). Computer program instructions can be stored, accessed, and read from memory 112 for execution by one or more of processors 102-106. For example, memory 112 may store one or more software applications. Further, memory 112 may store an entire software application or only a part of a software application that is executable by one or more of processors 102-106. It is noted that although only one block is shown in Fig. 1, memory 112 may include multiple physical devices installed on a central computing device or on different computing devices.
  • Fig. 7 is a schematic flowchart illustrating a method 2000 for providing super-resolution of a low-resolution image, consistent with some disclosed embodiments.
  • the method 2000 may be described in detail with respect to Fig. 7.
  • patches are extracted from the low-resolution image and each of the extracted patches is represented as a high dimensional vector by the patch extracting and representing device comprising the first set of filters.
  • these vectors comprise a set of feature maps, of which the number equals to the dimensionality of the vectors.
  • a popular strategy in image restoration is to densely extract patches and then represent them by a set of pre-trained bases such as PCA, DCT, Haar, etc.
  • each of the high dimensional vectors is mapped nonlinearly onto a high-resolution patch-wise representation by a mapping device comprising a second set of filters.
  • the first set of filters extracts an ni-dimensional feature for each patch.
  • the second set of filters maps each of these ni-dimensional vectors into an n 2 -dimensional vector.
  • each mapped vector is conceptually the representation of a high-resolution patch. These vectors comprise another set of feature maps.
  • step S230 the high-resolution patch-wise representations are aggregated to generate a high-resolution image.
  • these steps S210-S230 may be simulated by the above-mentioned formulae (l)-(3).
  • the patches may be extracted from the low-resolution image and representing each of the extracted patches as a high dimensional vector by rule of a first nonlinear function of first parameters, in which the first parameters are determined from predetermined parameters associated with the low-resolution image.
  • F'(first parameters) may be max(0, first parameters), tan zf first parameters) or l/( l+exp( -first parameters )).
  • each of the high dimensional vectors onto a high-resolution patch-wise representation may be mapped nonlinearly by rule of a function of a second nonlinear function of second parameters, in which the second parameters are determined from predetermined parameters associated with the high dimensional vector.
  • F"(second parameters) may be max(0, second parameters), tanh( second parameters) or l/( 1 + exp( -second parameters ) ) .
  • the method 2000 may further comprise a step of sampling a ground truth high-resolution image corresponding to the low-resolution image from a predetermined training set and a step of comparing dissimilarity between the aggregated high-resolution image and the corresponding ground truth high-resolution image to generate a reconstruction error.
  • the reconstruction error is back-propagated in order to determine the parameters, i.e., Wi t W 2, W3, B ⁇ t B 2, and B 3 .
  • the method 2000 further comprises a step of preparing the predetermined training set.
  • a plurality of sub-images is first cropped from a randomly selected training image to generate a set of ground truth high-resolution sub-images. For example, n sub-images of fflXffl pixels each may be cropped.
  • a set of low-resolution sub-images are generated based on the set of ground truth high-resolution sub-images.
  • each of the ground truth high-resolution sub-images is paired with a corresponding low-resolution sub-image. Then, all the pairs are collected to form the predetermined training set.
  • each of the ground truth high-resolution sub-images is blurred by a Gaussian kernel, and the blurred ground truth high-resolution sub-image is downsampled by a predetermined scaling factor. Then, the downsampled sub-images are upscaled by the same predetermined scaling factor to generate the set of low-resolution sub-images.
  • the method 2000 may further comprise a step of upscaling the low-resolution image to a predetermined size (not shown) before the patches are extracted from the low-resolution image.
  • a method 3000 for training a convolutional neural network system for providing super-resolution of a low-resolution image is illustrated.
  • the method 3000 may be described in detail with respect to Fig. 8.
  • a low-resolution sub-image and its corresponding ground truth high-resolution sub-image are sampled from a predetermined training set at step S310.
  • a high-resolution sub-image is reconstructed from the low-resolution sub-image by the convolutional neural network system.
  • a reconstruction error is generated by comparing dissimilarity between the reconstructed high-resolution sub-image and the ground truth high-resolution sub-image.
  • the reconstruction error is back-propagated through the convolutional neural network system so as to adjust weights on connections between neurons of the convolutional neural network system. Repeating steps S310-S340 until an average value of the reconstruction error is lower than a preset threshold, for example, half of the mean square error between the low-resolution sub-images and high-resolution sub-image in the predetermined training set.
  • the present application does not explicitly learn the dictionaries or manifolds for modeling the patch space. These are implicitly achieved via the convolutional layers. Furthermore, the patch extraction and aggregation are also formulated as convolutional layers, so are involved in the optimization. In the method and apparatus of the present application, the entire convolutional neural network is fully obtained through training, with little pre/post-processing. With a lightweight structure, the apparatus and method of the present application have achieved superior performance than the state-of-the-art methods.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

Disclosed is an apparatus for providing super-resolution of a low-resolution image. The apparatus may comprise: a patch extracting and representing device comprising a first set of filters configured to extract patches from the low-resolution image and represent each of the extracted patches as a high dimensional vector, a mapping device comprising a second set of filters configured to map nonlinearly each of the high dimensional vectors onto a high-resolution patch-wise representation and an aggregating device configured to aggregate the high-resolution patch-wise representations to generate a high-resolution image for the low-resolution image. A method for providing super-resolution of a low-resolution image is also disclosed.

Description

AN APPARATUS AND A METHOD FOR PROVIDING SUPER-RESOLUTION OF A
LOW-RESOLUTION IMAGE
Technical Field
[0001] The present application generally relates to a field of image processing, more particularly, to an apparatus and a method for providing super-resolution of a low-resolution image.
Background
[0002] Super-resolution (SR) is a class of techniques that enhance the resolution of an imaging system. Recent state-of-the-art methods for super-resolution of a single image are mostly example-based. These methods either exploit internal similarities of the same image, or learn mapping functions from external low-resolution and high-resolution exemplar pairs. The external example-based methods are often provided with abundant samples, but are challenged by the difficulties of modeling the data effectively and compactly.
[0003] A category of state-of-the-art SR approaches learn a mapping between low/high-resolution patches. These studies vary on how to learn a compact dictionary or manifold space to relate low/high-resolution patches, and on how representation schemes can be conducted in such spaces. In these methods, the patches are the focus of the optimization; the patch extraction and aggregation steps are considered as pre/post-processing and handled separately.
[0004] Convolutional neural networks (CNN) date back decades and have recently shown an explosive popularity partially due to its success in image classification. The convolutional neural network is usually applied for natural image denoising and removing noisy patterns (dirt/rain), but not for the image super-resolution problem.
Summary
[0005] According to an embodiment of the present application, disclosed is an apparatus for providing super-resolution of a low-resolution image. The apparatus may comprise: a patch extracting and representing device comprising a first set of filters configured to extract patches from the low-resolution image and represent each of the extracted patches as a high dimensional vector, a mapping device comprising a second set of filters configured to map nonlinearly each of the high dimensional vectors onto a high-resolution patch-wise representation and an aggregating device configured to aggregate the high-resolution patch-wise representations to generate a high-resolution image for the low-resolution image.
[0006] According to an embodiment of the present application, disclosed is a method for providing super-resolution of a low-resolution image, the method may comprise: extracting patches from the low-resolution image and representing each of the extracted patches as a high dimensional vector by a patch extracting and representing device comprising a first set of filters; mapping nonlinearly each of the high dimensional vectors onto a high-resolution patch-wise representation by a mapping device comprising a second set of filters; and aggregating the high-resolution patch- wise representations to generate a high-resolution image.
[0007] According to an embodiment of the present application, disclosed is a method for training a convolutional neural network system for providing super-resolution of a low-resolution image, and the method may comprise: 1) sampling a low-resolution sub-image and its corresponding ground truth high-resolution sub-image from a predetermined training set; 2) reconstructing the low-resolution sub-image to a high-resolution sub-image by the convolutional neural network system; 3) generating a reconstruction error by comparing dissimilarity between the reconstructed high-resolution sub-image and the ground truth high-resolution sub-image; 4) back-propagating the reconstruction error through the convolutional neural network system so as to adjust weights on connections between neurons of the convolutional neural network system; and repeating steps l)-4) until an average value of the reconstruction error is lower than a preset threshold.
[0008] According to an embodiment of the present application, disclosed is an apparatus for providing super-resolution of a low-resolution image, the apparatus may comprise a reconstructing unit configured to reconstruct the low-resolution image to a high-resolution image based on predetermined parameters and a training unit configured to train the convolutional neural network system with a predetermined training set so as to determine the parameters used by the reconstructing unit. The reconstructing unit may comprise: a patch extracting and representing device configured to extract patches from the low-resolution image and represent each of the extracted patches as a high dimensional vector based on the predetermined parameters; a mapping device configured to map nonlinearly each of the high dimensional vectors onto a high-resolution patch-wise representation; and a aggregating device configured to aggregate the high-resolution patch-wise representations to generate a high-resolution image. The patch extracting and representing device, the mapping device and the aggregating device comprise a plurality of convolutional layers, and the plurality of convolutional layers are sequentially connected to each other to form a convolutional neural network system.
[0009] In contrast to existing methods, the present application does not explicitly learn the dictionaries or manifolds for modeling the patch space. These are implicitly achieved via the convolutional layers. Furthermore, the patch extraction and aggregation are also formulated as convolutional layers, so are involved in the optimization. In the method and apparatus of the present application, the entire convolutional neural network is fully obtained through training, with little pre/post-processing. With a lightweight structure, the apparatus and method of the present application have achieved superior performance than the state-of-the-art methods.
Brief Description of the Drawing
[0010] Exemplary non-limiting embodiments of the present invention are described below with reference to the attached drawings. The drawings are illustrative and generally not to an exact scale. The same or similar elements on different figures are referenced with the same reference numbers.
[0011] Fig. 1 is a schematic diagram illustrating an apparatus for providing super-resolution of a low-resolution image consistent with an embodiment of the present application.
[0012] Fig. 2 is a schematic diagram illustrating an apparatus for providing super-resolution of a low-resolution image consistent with another embodiment of the present application.
[0013] Fig. 3 is a schematic diagram illustrating a convolutional neural network system, consistent with some disclosed embodiments.
[0014] Fig. 4. is a schematic diagram illustrating a training unit of the apparatus, consistent with some disclosed embodiments.
[0015] Fig. 5. is a schematic diagram illustrating a training set preparation device of the training unit, consistent with some disclosed embodiments.
[0016] Fig. 6 is a schematic diagram illustrating an apparatus for providing super-resolution of a low-resolution image when it is implemented in software, consistent with some disclosed embodiments.
[0017] Fig. 7 is a schematic flowchart illustrating a method for providing super-resolution of a low-resolution image, consistent with some disclosed embodiments.
[0018] Fig. 8 is a schematic flowchart illustrating a method for training a convolutional neural network system for providing super-resolution of a low-resolution image, consistent with some disclosed embodiments.
Detailed Description
[0019] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When appropriate, the same reference numbers are used throughout the drawings to refer to the same or like parts. Fig. 1 is a schematic diagram illustrating an exemplary apparatus 1000 for providing super-resolution of a low-resolution image consistent with some disclosed embodiments.
[0020] Referring to Fig. 1, where the apparatus 1000 is implemented by the hardware, it may comprise a patch extracting and representing device 100, a mapping device 200 and an aggregating device 300. In the embodiment shown in Fig. l, the patch extracting and representing device 100 may comprise a first set of filters configured to extract patches from the low-resolution image and represent each of the extracted patches as a high dimensional vector. The mapping device 200 may comprise a second set of filters configured to map nonlinearly each of the high dimensional vectors onto a high-resolution patch-wise. The aggregating device 300 may be configured to aggregate the high-resolution patch-wise representations to generate a high-resolution image for the low-resolution image.
[0021] According to the embodiment, the first set of filters is configured to extract patches from the low-resolution image and represent each of the extracted patches as a high dimensional vector by rule of a first nonlinear function of first parameters, in which the first parameters are determined from predetermined parameters associated with the low-resolution image.
[0022] According to the embodiment, the second set of filters is configured to map nonlinearly each of the high dimensional vectors onto a high-resolution patch-wise representation by rule of a second nonlinear function of second parameters, in which the second parameters are determined from predetermined parameters associated with the high dimensional vector.
[0023] Hereinafter, the first set of filters, the second set of filters and the aggregating device as mentioned in the above will be further discussed in detail. For convenience of description, the low-resolution image is denoted by F, and the high-resolution image is denoted by F(Y) which is as similar as possible to a ground truth high-resolution image X.
[0024] The first set of filters is configured to extract patches from the low-resolution image Y and represent each of the extracted patches as a high dimensional vector. In an embodiment, these vectors comprise a set of feature maps, of which the number equals to the dimensionality of the vectors. A popular strategy in image restoration is to densely extract patches and then represent them by a set of pre-trained bases such as PCA (Principal Component Analysis), DCT (Discrete Cosine Transformation), Haar, etc.
[0025] According to an embodiment, the first set of filters may be simulated as an operation F .
Fj(Y) = F' (W! * Y + Bj) , (1) where W\ and B\ represent the filters and biases respectively. Here, F'(x) is a nonlinear function e.g., max(0,x), tanh(x) or l/(l+exp(-x)). In the embodiment, W\ is of a size cx lx 1 xtii, where c is the number of channels in the input image, for example, if the input image is a color image, then c is 3, and i is the spatial size of a filter, and ti\ is the number of filters. Intuitively, W\ applies ti\ convolutions on the image, and each convolution has a kernel size cx/ix i. The output is composed of ti\ feature maps. B\ is an πι-dimensional vector, whose each element is associated with a filter.
[0026] The second set of filters is configured to map nonlinearly each of the high dimensional vectors onto another high-dimensional vector. In the embodiment, the first set of filters extracts an ni-dimensional feature for each patch. The second set of filters maps each of these ni-dimensional vectors into an n2-dimensional vector. Each mapped vector is conceptually the representation of a high-resolution patch. These vectors comprise another set of feature maps.
[0027] According to an embodiment, the second set of filters may be simulated as an operation F2:
F2(Y) = F"(W2*F1(Y) + B2), (2) where W2 is of a size tiixl xl xn2, and B2 is a n2-dimensional vector. Here F"(x) is a nonlinear function e.g., max(0,x), tanh(x) or l/(l+exp(-x)). In the embodiment, each of the output n2-dimensional vectors is conceptually a representation of a high-resolution patch that will be used for reconstruction.
[0028] Alternatively, it is possible to add more convolutional layers (whose spatial supports are lxl) to increase the non-linearity. But this can significantly increase the complexity of the convolutional neural network system, and thus demands more training data and time.
[0029] The aggregating device 300 aggregates the high-resolution patch-wise representations to generate a high-resolution image. The aggregating device 300 may be simulated as an operation F3:
F(Y) = W3 * F2(Y) + B3, (3) where W3 is of a size n2Xf. 3 X.f3Xc, and B3 is a c-dimensional vector.
[0030] If the representations of the high-resolution patches are in the image domain (i.e., each representation can be simply reshaped to form the patch), the filters may act like an averaging filter. If the representations of the high-resolution patches are in some other domains (e.g., coefficients in terms of some bases), W3 may behave as first projecting the coefficients onto the image domain and then averaging. In either way, W3 is a set of linear filters.
[0031] According to the embodiment, the apparatus 1000 may further comprise a comparing device (not shown) which is configured to sample a ground truth high-resolution image corresponding to the low-resolution image from a predetermined training set and compare dissimilarity between the aggregated high-resolution image and the sampled ground truth high-resolution image to generate a reconstruction error. For example, the reconstruction error comprises a mean squared error. The reconstruction error is back-propagated in order to determine the parameters, i.e., Wit W2, W3, B\t B2, and B3.
[0032] In one embodiment, the apparatus 1000 may further comprise an upscaling unit (not shown), and the upscaling unit is configured to upscale the low-resolution image to a predetermined size. For example, the low-resolution image may be upscaled by using bicubic interpolation. The upscaling is the only pre-processing in the embodiment.
[0033] Fig. 2 is a schematic diagram illustrating an apparatus 1000' for providing super-resolution of a low-resolution image consistent with another embodiment of the present application. As shown in Fig. 2, where the apparatus 1000' is implemented by the hardware, it may comprise a reconstructing unit 100' and a training unit 200' . The reconstructing unit 100' is configured to reconstruct the low-resolution image to a high-resolution image based on predetermined parameters. [0034] According to an embodiment shown in Fig. 2, the reconstructing unit 100' may further comprise a patch extracting and representing device 110', a mapping device 120' and a aggregating device 130' . In the embodiment, the patch extracting and representing device 110', the mapping device 120' and the aggregating device 130' may connect together to form a convolutional neural network system. Fig. 3 illustrates the layer configuration of the convolutional neural network system in mathematic simulation model. In one embodiment, each of the patch extracting and representing device 110', the mapping device 120' and the aggregating device 130' may be simulated as at least one convolutional layer, respectively. Different operations are performed at different convolutional layers, respectively.
[0035] In the embodiment, the patch extracting and representing unit 110' is configured to extract patches from the low-resolution image and represent each of the extracted patches as a high dimensional vector based on the predetermined parameters. This is equivalent to convolving the image by a set of filters as mentioned above.
[0036] The mapping device 120' is configured to nonlinearly map each of the high dimensional vectors onto a high-resolution patch-wise representation. This is equivalent to applying a second set of filters as mentioned above which have a trivial spatial support lxl. Alternatively, it is possible to add more convolutional layers (whose spatial supports are lxl) to increase the non-linearity. But this can significantly increase the complexity of the convolutional neural network system, and thus demands more training data and time.
[0037] The aggregating device 130' is configured to aggregate the high-resolution patch-wise representations to generate a high-resolution image.
[0038] In an embodiment, the training unit 200' is configured to train the convolutional neural network system with a predetermined training set so as to determine the parameters, for example W\, W2, W3, B B2, B3, used by the reconstructing unit. According to the embodiment as shown in Fig. 5, the training unit 200' may comprise a sampling device 210', a comparing device 220', and a back-propagating device 230' .
[0039] The sampling device 210' may be configured to sample a low-resolution sub-image and its corresponding ground truth high-resolution sub-image from a predetermined training set and input the low-resolution sub-image to the convolutional neural network system. Here, "sub-images" means these samples are treated as small "images" rather than "patches", in the sense that "patches" are overlapping and require some averaging as post-processing but "sub-images" need not.
[0040] The comparing device 220' may be configured to compare dissimilarity between the reconstructed high-resolution sub-image based on the input low-resolution sub-image from the convolutional neural network system and the corresponding ground truth high-resolution sub-image to generate a reconstruction error. For example, the reconstruction error may comprise a mean squared error, and the error is minimized by using stochastic gradient descent with the standard back propagation.
[0041] The back-propagating device 230' is configured to back-propagate the reconstruction error through the convolutional neural network system so as to adjust weights on connections between neurons of the convolutional neural network system.
[0042] It shall be noticed that the convolutional neural network system do not preclude the usage of other kinds of reconstruction error, if only the reconstruction error are derivable. If a better perceptually motivated metric is given during the training, it is flexible for the convolutional neural network system to adapt to that metric.
[0043] In one embodiment, the training unit 200' may further comprise a training set preparation device configured to prepare the predetermined training set for training the convolutional neural network system. Fig. 5 is a schematic diagram illustrating the training set preparation device of the training unit 200' . As shown, the training set preparation device may comprise a cropper 241 ', a low-resolution sub-image generator 242', a pairing device 243' and a collector 244' .
[0044] The cropper 24 may be configured to crop randomly a plurality of sub-images from a randomly selected training image to generate a set of ground truth high-resolution sub-images. For example, the cropper 24 may crop n sub-images of mxm pixels each. The low-resolution sub-image generator 242' may be configured to generate a set of low-resolution sub-images based on the set of ground truth high-resolution sub-images. The pairing device 243' may be configured to pair each of the ground truth high-resolution sub-images with a corresponding low-resolution sub-image. The collector 244' may be configured to collect all the pairs to form the predetermined training set.
[0045] According to an embodiment, the low-resolution sub-image generator 242' may comprise a blurring device, a sampling device and an upscaling device. The blurring device may be configured to blur each of the ground truth high-resolution sub-images by a Gaussian kernel. The sampling device may be configured to sample the blurred ground truth high-resolution sub-image by a predetermined scaling factor. The upscaling device may be configured to upscale the sampled ground truth high-resolution sub-image by a predetermined scaling factor to generate the set of low-resolution sub-images.
[0046] It shall be appreciated that the apparatus 1000 and 1000' may be implemented using certain hardware, software, or a combination thereof. In addition, the embodiments of the present invention may be adapted to a computer program product embodied on one or more computer readable storage media (comprising but not limited to disk storage, CD-ROM, optical memory and the like) containing computer program codes. Fig. 6 is a schematic diagram illustrating an apparatus 1000 and 1000' for providing super-resolution of a low-resolution image when it is implemented in software, consistent with some disclosed embodiments.
[0047] In the case that the apparatus 1000 and 1000' are implemented with software, the apparatus 1000 and 1000' may include a general purpose computer, a computer cluster, a mainstream computer, a computing device dedicated for providing online contents, or a computer network comprising a group of computers operating in a centralized or distributed fashion. As shown in Fig. 4, apparatus 1000 and 1000' may include one or more processors (processors 102, 104, 106 etc.), a memory 112, a storage device 116, and a bus to facilitate information exchange among various devices of apparatus 1000. Processors 102-106 may include a central processing unit ("CPU"), a graphic processing unit ("GPU"), or other suitable information processing devices. Depending on the type of hardware being used, processors 102-106 can include one or more printed circuit boards, and/or one or more microprocessor chips. Processors 102-106 can execute sequences of computer program instructions to perform various methods that will be explained in greater detail below.
[0048] Memory 112 can include, among other things, a random access memory ("RAM") and a read-only memory ("ROM"). Computer program instructions can be stored, accessed, and read from memory 112 for execution by one or more of processors 102-106. For example, memory 112 may store one or more software applications. Further, memory 112 may store an entire software application or only a part of a software application that is executable by one or more of processors 102-106. It is noted that although only one block is shown in Fig. 1, memory 112 may include multiple physical devices installed on a central computing device or on different computing devices.
[0049] Fig. 7 is a schematic flowchart illustrating a method 2000 for providing super-resolution of a low-resolution image, consistent with some disclosed embodiments. Hereafter, the method 2000 may be described in detail with respect to Fig. 7.
[0050] At step S210, patches are extracted from the low-resolution image and each of the extracted patches is represented as a high dimensional vector by the patch extracting and representing device comprising the first set of filters. In an embodiment, these vectors comprise a set of feature maps, of which the number equals to the dimensionality of the vectors. A popular strategy in image restoration is to densely extract patches and then represent them by a set of pre-trained bases such as PCA, DCT, Haar, etc.
[0051] At step S220, each of the high dimensional vectors is mapped nonlinearly onto a high-resolution patch-wise representation by a mapping device comprising a second set of filters. In the embodiment, the first set of filters extracts an ni-dimensional feature for each patch. The second set of filters maps each of these ni-dimensional vectors into an n2-dimensional vector. In an embodiment, each mapped vector is conceptually the representation of a high-resolution patch. These vectors comprise another set of feature maps.
[0052] At step S230, the high-resolution patch-wise representations are aggregated to generate a high-resolution image. In an embodiment, these steps S210-S230 may be simulated by the above-mentioned formulae (l)-(3).
[0053] According to an embodiment, the patches may be extracted from the low-resolution image and representing each of the extracted patches as a high dimensional vector by rule of a first nonlinear function of first parameters, in which the first parameters are determined from predetermined parameters associated with the low-resolution image. For example, F'(first parameters) may be max(0, first parameters), tan zf first parameters) or l/( l+exp( -first parameters )).
[0054] According to an embodiment, each of the high dimensional vectors onto a high-resolution patch-wise representation may be mapped nonlinearly by rule of a function of a second nonlinear function of second parameters, in which the second parameters are determined from predetermined parameters associated with the high dimensional vector. For example, F"(second parameters) may be max(0, second parameters), tanh( second parameters) or l/( 1 + exp( -second parameters ) ) .
[0055] According to an embodiment, after the high-resolution patch-wise representations are aggregated to generate a high-resolution image, the method 2000 may further comprise a step of sampling a ground truth high-resolution image corresponding to the low-resolution image from a predetermined training set and a step of comparing dissimilarity between the aggregated high-resolution image and the corresponding ground truth high-resolution image to generate a reconstruction error. The reconstruction error is back-propagated in order to determine the parameters, i.e., Wit W2, W3, B\t B2, and B3.
[0056] According to an embodiment, before a ground truth high-resolution image corresponding to the low-resolution image is sampled from a predetermined training set, the method 2000 further comprises a step of preparing the predetermined training set. In particular, a plurality of sub-images is first cropped from a randomly selected training image to generate a set of ground truth high-resolution sub-images. For example, n sub-images of fflXffl pixels each may be cropped. Next, a set of low-resolution sub-images are generated based on the set of ground truth high-resolution sub-images. Then, each of the ground truth high-resolution sub-images is paired with a corresponding low-resolution sub-image. Then, all the pairs are collected to form the predetermined training set.
[0057] Following is a possible way of generating low-resolution sub-images. For example, each of the ground truth high-resolution sub-images is blurred by a Gaussian kernel, and the blurred ground truth high-resolution sub-image is downsampled by a predetermined scaling factor. Then, the downsampled sub-images are upscaled by the same predetermined scaling factor to generate the set of low-resolution sub-images. [0058] In one embodiment, the method 2000 may further comprise a step of upscaling the low-resolution image to a predetermined size (not shown) before the patches are extracted from the low-resolution image.
[0059] According to an embodiment, a method 3000 for training a convolutional neural network system for providing super-resolution of a low-resolution image is illustrated. Hereafter, the method 3000 may be described in detail with respect to Fig. 8.
[0060] As shown in Fig. 8, a low-resolution sub-image and its corresponding ground truth high-resolution sub-image are sampled from a predetermined training set at step S310. As step S320, a high-resolution sub-image is reconstructed from the low-resolution sub-image by the convolutional neural network system. At step S330, a reconstruction error is generated by comparing dissimilarity between the reconstructed high-resolution sub-image and the ground truth high-resolution sub-image. At step S340, the reconstruction error is back-propagated through the convolutional neural network system so as to adjust weights on connections between neurons of the convolutional neural network system. Repeating steps S310-S340 until an average value of the reconstruction error is lower than a preset threshold, for example, half of the mean square error between the low-resolution sub-images and high-resolution sub-image in the predetermined training set.
[0061] In contrast to existing methods, the present application does not explicitly learn the dictionaries or manifolds for modeling the patch space. These are implicitly achieved via the convolutional layers. Furthermore, the patch extraction and aggregation are also formulated as convolutional layers, so are involved in the optimization. In the method and apparatus of the present application, the entire convolutional neural network is fully obtained through training, with little pre/post-processing. With a lightweight structure, the apparatus and method of the present application have achieved superior performance than the state-of-the-art methods.
[0062] Although the preferred examples of the present invention have been described, those skilled in the art can make variations or modifications to these examples upon knowing the basic inventive concept. The appended claims are intended to be considered as comprising the preferred examples and all the variations or modifications fell into the scope of the present invention.
[0063] Obviously, those skilled in the art can make variations or modifications to the present invention without departing the spirit and scope of the present invention. As such, if these variations or modifications belong to the scope of the claims and equivalent technique, they may also fall into the scope of the present invention.

Claims

What is claimed is:
1. An apparatus for providing super-resolution of a low-resolution image, comprising: a patch extracting and representing device comprising a first set of filters configured to extract patches from the low-resolution image and represent each of the extracted patches as a high dimensional vector;
a mapping device comprising a second set of filters configured to map nonlinearly each of the high dimensional vectors onto a high-resolution patch- wise representation; and
an aggregating device configured to aggregate the high-resolution patch-wise representations to generate a high-resolution image for the low-resolution image.
2. An apparatus according to claim 1, wherein the first set of filters is configured to extract the patches from the low-resolution image and represent each of the extracted patches as the high dimensional vector by rule of a first nonlinear function of first parameters, wherein the first parameters are determined from predetermined parameters associated with the low-resolution image.
3. An apparatus according to claim 1 or 2, wherein the second set of filters is configured to map nonlinearly each of the high dimensional vectors onto the high-resolution patch-wise representation by rule of a second nonlinear function of second parameters, wherein the second parameters are determined from predetermined parameters associated with the high dimensional vector.
4. An apparatus according to claim 3, further comprising:
a comparing device configured to sample a ground truth high-resolution image corresponding to the low-resolution image from a predetermined training set and compare a dissimilarity between the aggregated high-resolution image and the corresponding ground truth high-resolution image to generate a reconstruction error, wherein the reconstruction error is back-propagated in order to determine the first and second parameters.
5. An apparatus according to claim 4, further comprising a training set preparation device, wherein the training set preparation device further comprises:
a cropper configured to crop randomly a plurality of sub-images from a randomly selected training image to generate a set of ground truth high-resolution sub-images;
a low-resolution sub-image generator configured to generate a set of low-resolution sub-images based on the set of ground truth high-resolution sub-images;
a pairing device configured to pair each of the ground truth high-resolution sub-images with a corresponding low-resolution sub-image; and
a collector configured to collect the paired ground truth high-resolution sub-images and the low-resolution sub-image to form the predetermined training set.
6. An apparatus according to claim 5, wherein the low-resolution sub-image generator further comprises:
a blurring device configured to blur each of the ground truth high-resolution sub-images by a Gaussian kernel;
a downsampling device configured to downsample the blurred ground truth high-resolution sub-image by a predetermined scaling factor; and
an upscaling device configured to upscale the downsampled ground truth high-resolution sub-images by a predetermined scaling factor to generate the set of low-resolution sub-images.
7. An apparatus according to claim 1, further comprising:
an upscaling unit configured to upscale the low-resolution image to a predetermined size.
8. An apparatus according to claim 4, wherein the reconstruction error comprises a mean squared error.
9. A method for providing super-resolution of a low-resolution image, comprising:
extracting patches from the low-resolution image and representing each of the extracted patches as a high dimensional vector by a patch extracting and representing device comprising a first set of filters;
mapping nonlinearly each of the high dimensional vectors onto a high-resolution patch-wise representation by a mapping device comprising a second set of filters; and
aggregating the high-resolution patch-wise representations to generate a high-resolution image.
10. A method according to claim 9, wherein the extracting patches from the low-resolution image and representing each of the extracted patches as a high dimensional vector further comprises:
extracting patches from the low-resolution image and representing each of the extracted patches as the high dimensional vector by rule of a first nonlinear function of first parameters, wherein the first parameters are determined from predetermined parameters associated with the low-resolution image.
11. A method according to claim 9 or 10, wherein the mapping nonlinearly each of the high dimensional vectors onto a high-resolution patch-wise representation further comprises: mapping nonlinearly each of the high dimensional vectors onto the high-resolution patch-wise representation by rule of a second nonlinear function of second parameters, wherein the second parameters are determined from predetermined parameters associated with the high dimensional vector.
12. A method according to claim 11, further comprising:
sampling a ground truth high-resolution image corresponding to the low-resolution image from a predetermined training set; and comparing a dissimilarity between the aggregated high-resolution image and the corresponding ground truth high-resolution image to generate a reconstruction error, wherein the reconstruction error is back-propagated in order to determine the first and second parameters.
13. A method according to claim 12, wherein before sampling a ground truth high-resolution image corresponding to the low-resolution image from a predetermined training set, further comprising:
cropping randomly a plurality of sub-images from a randomly selected training image to generate a set of ground truth high-resolution sub-images;
generating a set of low-resolution sub-images based on the set of ground truth high-resolution sub-images;
pairing each of the ground truth high-resolution sub-images with a corresponding low-resolution sub-image; and
collecting the paired ground truth high-resolution sub-images and the low-resolution sub-image to form the predetermined training set.
14. A method according to claim 13, wherein the step of generating a set of low-resolution sub-images based on the set of ground truth high-resolution sub-images further comprises:
blurring each of the ground truth high-resolution sub-images by a Gaussian kernel;
downsampling the blurred ground truth high-resolution sub-image by a predetermined scaling factor; and
upscaling the downsampled ground truth high-resolution sub-images by a predetermined scaling factor to generate the set of low-resolution sub-images.
15. A method according to claim 9, before extracting patches from the low-resolution image, further comprising:
upscaling the low-resolution image to a predetermined size.
16. A method according to claim 12, wherein the reconstruction error comprises a mean squared error.
17. A method for training a convolutional neural network system for providing super-resolution of a low-resolution image, comprising:
1) sampling a low-resolution sub-image and its corresponding ground truth high-resolution sub-image from a predetermined training set;
2) reconstructing the low-resolution sub-image to a high-resolution sub-image by the convolutional neural network system;
3) generating a reconstruction error by comparing dissimilarity between the reconstructed high-resolution sub-image and the ground truth high-resolution sub-image;
4) back-propagating the reconstruction error through the convolutional neural network system so as to adjust weights on connections between neurons of the convolutional neural network system; and
repeating steps l)-4) until an average value of the reconstruction error is lower than a preset threshold.
18. An apparatus for providing super-resolution of a low-resolution image, comprising: a reconstructing unit configured to reconstruct the low-resolution image to a high-resolution image based on predetermined parameters, wherein the reconstructing unit comprises:
a patch extracting and representing device configured to extract patches from the low-resolution image and represent each of the extracted patches as a high dimensional vector based on the predetermined parameters;
a mapping device configured to map nonlinearly each of the high dimensional vectors onto a high-resolution patch- wise representation; and
an aggregating device configured to aggregate the high-resolution patch- wise representations to generate a high-resolution image; wherein the patch extracting and representing device, the mapping device and the aggregating device comprise a plurality of convolutional layers, the plurality of convolutional layers are sequentially connected to each other to form a convolutional neural network system;
a training unit configured to train the convolutional neural network system with a predetermined training set so as to modify the predetermined parameters used by the reconstructing unit.
PCT/CN2014/000755 2014-08-08 2014-08-08 An apparatus and a method for providing super-resolution of a low-resolution image Ceased WO2016019484A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2014/000755 WO2016019484A1 (en) 2014-08-08 2014-08-08 An apparatus and a method for providing super-resolution of a low-resolution image
CN201480082564.2A CN106796716B (en) 2014-08-08 2014-08-08 For providing the device and method of super-resolution for low-resolution image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2014/000755 WO2016019484A1 (en) 2014-08-08 2014-08-08 An apparatus and a method for providing super-resolution of a low-resolution image

Publications (1)

Publication Number Publication Date
WO2016019484A1 true WO2016019484A1 (en) 2016-02-11

Family

ID=55262969

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2014/000755 Ceased WO2016019484A1 (en) 2014-08-08 2014-08-08 An apparatus and a method for providing super-resolution of a low-resolution image

Country Status (2)

Country Link
CN (1) CN106796716B (en)
WO (1) WO2016019484A1 (en)

Cited By (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106251289A (en) * 2016-07-21 2016-12-21 北京邮电大学 A kind of based on degree of depth study and the video super-resolution method for reconstructing of self-similarity
CN106447609A (en) * 2016-08-30 2017-02-22 上海交通大学 Image super-resolution method based on depth convolutional neural network
CN106991646A (en) * 2017-03-28 2017-07-28 福建帝视信息科技有限公司 A kind of image super-resolution method based on intensive connection network
CN108537733A (en) * 2018-04-11 2018-09-14 南京邮电大学 Super resolution ratio reconstruction method based on multipath depth convolutional neural networks
CN108615222A (en) * 2018-04-17 2018-10-02 中国矿业大学 A kind of depth convolutional network image super-resolution system based on multipair multi-connection
CN108694700A (en) * 2017-04-10 2018-10-23 三星电子株式会社 System and method for deep learning image super-resolution
CN109325915A (en) * 2018-09-11 2019-02-12 合肥工业大学 A super-resolution reconstruction method for low-resolution surveillance video
CN109360148A (en) * 2018-09-05 2019-02-19 北京悦图遥感科技发展有限公司 Remote sensing image super-resolution reconstruction method and device based on hybrid random downsampling
CN109801214A (en) * 2018-05-29 2019-05-24 京东方科技集团股份有限公司 Image reconstruction device and method, equipment, computer readable storage medium
US10410322B2 (en) 2017-04-05 2019-09-10 Here Global B.V. Deep convolutional image up-sampling
CN110310227A (en) * 2019-06-27 2019-10-08 电子科技大学 An image super-resolution reconstruction method based on high and low frequency information decomposition
CN110322528A (en) * 2019-06-26 2019-10-11 浙江大学 Nuclear magnetic resonance brain image reconstructing blood vessel method based on 3T, 7T
CN110390638A (en) * 2019-07-22 2019-10-29 北京工商大学 A kind of high-resolution three-dimension voxel model method for reconstructing
WO2019231235A1 (en) * 2018-05-29 2019-12-05 엘지전자 주식회사 Signal processing device and image display device comprising same
CN110675324A (en) * 2018-07-02 2020-01-10 上海寰声智能科技有限公司 4K ultra-high definition image sharpening processing method
CN110738099A (en) * 2019-08-30 2020-01-31 中山大学 A low-resolution pedestrian re-identification method based on adaptive dual-branch network
CN110738601A (en) * 2019-10-23 2020-01-31 智慧视通(杭州)科技发展有限公司 low-resolution face image super-resolution reconstruction method based on three-dimensional face model
CN111461990A (en) * 2020-04-03 2020-07-28 华中科技大学 A step-by-step approach to super-resolution imaging based on deep learning
CN111563843A (en) * 2020-04-30 2020-08-21 苏州大学 A kind of image super-resolution reconstruction method, system and related device
CN111696034A (en) * 2020-05-11 2020-09-22 青岛小鸟看看科技有限公司 Image processing method and device and electronic equipment
CN112184555A (en) * 2020-10-21 2021-01-05 天津大学 Stereo image super-resolution reconstruction method based on deep interactive learning
CN112598575A (en) * 2020-12-22 2021-04-02 电子科技大学 Image information fusion and super-resolution reconstruction method based on feature processing
CN112825271A (en) * 2019-11-21 2021-05-21 通用电气精准医疗有限责任公司 System and method for reconstructing medical images using a deep neural network and recursive extraction of measurement data
CN112907441A (en) * 2020-12-29 2021-06-04 中央财经大学 Space downscaling method based on super-resolution of underground water satellite image
EP3714404A4 (en) * 2017-11-26 2021-08-18 Yeda Research and Development Co. Ltd SIGNAL ENHANCEMENT AND MANIPULATION USING A SPECIFIC DEEP SIGNAL NETWORK
CN115601242A (en) * 2022-12-13 2023-01-13 电子科技大学（Cn） A lightweight image super-resolution reconstruction method suitable for hardware deployment
WO2025041222A1 (en) * 2023-08-21 2025-02-27 Nec Corporation Image matching apparatus, image matching method, training apparatus, training method, and non-transitory computer-readable medium

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102351349B1 (en) * 2017-06-30 2022-01-13 케이엘에이 코포레이션 Generation of high resolution images from low resolution images for semiconductor applications
CN108416733A (en) * 2018-02-07 2018-08-17 杭州电子科技大学 One kind not reducing quality and quick image super-resolution method
CN108765343B (en) * 2018-05-29 2021-07-20 Oppo（重庆）智能科技有限公司 Image processing method, device, terminal and computer-readable storage medium
CN112598579B (en) * 2020-12-28 2024-08-27 苏州科达特种视讯有限公司 Monitoring scene-oriented image super-resolution method, device and storage medium
TWI879417B (en) * 2023-04-20 2025-04-01 創惟科技股份有限公司 Image processing method and image processing device

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020172434A1 (en) * 2001-04-20 2002-11-21 Mitsubishi Electric Research Laboratories, Inc. One-pass super-resolution images
CN101299235A (en) * 2008-06-18 2008-11-05 中山大学 Method for reconstructing human face super resolution based on core principle component analysis
CN101639937A (en) * 2009-09-03 2010-02-03 复旦大学 Super-resolution method based on artificial neural network
CN101710386A (en) * 2009-12-25 2010-05-19 西安交通大学 Super-resolution face recognition method based on relevant characteristic and non-liner mapping
CN102306306A (en) * 2011-08-30 2012-01-04 西安交通大学 Character recognition method for carrying out super-resolution reconstruction by utilizing characteristics of local binary pattern and radial basis function
CN102306374A (en) * 2011-08-30 2012-01-04 西安交通大学 Method for rebuilding super-resolution human face image by position block nonlinear mapping
CN103500165A (en) * 2013-08-21 2014-01-08 新华通讯社 High-dimensional vector quantity search method combining clustering and double key values

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102880878B (en) * 2011-07-15 2015-05-06 富士通株式会社 Single-image-based super-resolution analytical method and system
CN102354394B (en) * 2011-09-22 2015-03-11 中国科学院深圳先进技术研究院 Image super-resolution method and system
CN103839242A (en) * 2014-01-15 2014-06-04 中国科学院电子学研究所 Rapid image super-resolution improvement method based on high-dimensional indexing

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020172434A1 (en) * 2001-04-20 2002-11-21 Mitsubishi Electric Research Laboratories, Inc. One-pass super-resolution images
CN101299235A (en) * 2008-06-18 2008-11-05 中山大学 Method for reconstructing human face super resolution based on core principle component analysis
CN101639937A (en) * 2009-09-03 2010-02-03 复旦大学 Super-resolution method based on artificial neural network
CN101710386A (en) * 2009-12-25 2010-05-19 西安交通大学 Super-resolution face recognition method based on relevant characteristic and non-liner mapping
CN102306306A (en) * 2011-08-30 2012-01-04 西安交通大学 Character recognition method for carrying out super-resolution reconstruction by utilizing characteristics of local binary pattern and radial basis function
CN102306374A (en) * 2011-08-30 2012-01-04 西安交通大学 Method for rebuilding super-resolution human face image by position block nonlinear mapping
CN103500165A (en) * 2013-08-21 2014-01-08 新华通讯社 High-dimensional vector quantity search method combining clustering and double key values

Cited By (45)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106251289A (en) * 2016-07-21 2016-12-21 北京邮电大学 A kind of based on degree of depth study and the video super-resolution method for reconstructing of self-similarity
CN106447609A (en) * 2016-08-30 2017-02-22 上海交通大学 Image super-resolution method based on depth convolutional neural network
CN106991646A (en) * 2017-03-28 2017-07-28 福建帝视信息科技有限公司 A kind of image super-resolution method based on intensive connection network
CN106991646B (en) * 2017-03-28 2020-05-26 福建帝视信息科技有限公司 Image super-resolution method based on dense connection network
US10410322B2 (en) 2017-04-05 2019-09-10 Here Global B.V. Deep convolutional image up-sampling
US10621696B2 (en) 2017-04-05 2020-04-14 Here Global B.V. Deep convolutional image up-sampling
CN108694700A (en) * 2017-04-10 2018-10-23 三星电子株式会社 System and method for deep learning image super-resolution
CN108694700B (en) * 2017-04-10 2023-12-01 三星电子株式会社 Systems and methods for deep learning image super-resolution
US11907835B2 (en) 2017-11-26 2024-02-20 Yeda Research And Development Co. Ltd. Signal enhancement and manipulation using a signal-specific deep network
EP3714404A4 (en) * 2017-11-26 2021-08-18 Yeda Research and Development Co. Ltd SIGNAL ENHANCEMENT AND MANIPULATION USING A SPECIFIC DEEP SIGNAL NETWORK
CN108537733A (en) * 2018-04-11 2018-09-14 南京邮电大学 Super resolution ratio reconstruction method based on multipath depth convolutional neural networks
CN108537733B (en) * 2018-04-11 2022-03-11 南京邮电大学 Super-resolution reconstruction method based on multi-path deep convolutional neural network
CN108615222A (en) * 2018-04-17 2018-10-02 中国矿业大学 A kind of depth convolutional network image super-resolution system based on multipair multi-connection
CN109801214A (en) * 2018-05-29 2019-05-24 京东方科技集团股份有限公司 Image reconstruction device and method, equipment, computer readable storage medium
WO2019231235A1 (en) * 2018-05-29 2019-12-05 엘지전자 주식회사 Signal processing device and image display device comprising same
CN110675324A (en) * 2018-07-02 2020-01-10 上海寰声智能科技有限公司 4K ultra-high definition image sharpening processing method
CN110675324B (en) * 2018-07-02 2023-10-10 上海寰声智能科技有限公司 4K ultra-high definition image sharpening processing method
CN109360148A (en) * 2018-09-05 2019-02-19 北京悦图遥感科技发展有限公司 Remote sensing image super-resolution reconstruction method and device based on hybrid random downsampling
CN109360148B (en) * 2018-09-05 2023-11-07 北京悦图遥感科技发展有限公司 Remote sensing image super-resolution reconstruction method and device based on hybrid random downsampling
CN109325915A (en) * 2018-09-11 2019-02-12 合肥工业大学 A super-resolution reconstruction method for low-resolution surveillance video
CN109325915B (en) * 2018-09-11 2022-11-08 合肥工业大学 A super-resolution reconstruction method for low-resolution surveillance video
CN110322528A (en) * 2019-06-26 2019-10-11 浙江大学 Nuclear magnetic resonance brain image reconstructing blood vessel method based on 3T, 7T
CN110310227B (en) * 2019-06-27 2020-09-08 电子科技大学 Image super-resolution reconstruction method based on high-low frequency information decomposition
CN110310227A (en) * 2019-06-27 2019-10-08 电子科技大学 An image super-resolution reconstruction method based on high and low frequency information decomposition
CN110390638B (en) * 2019-07-22 2023-07-28 北京工商大学 A High Resolution 3D Voxel Model Reconstruction Method
CN110390638A (en) * 2019-07-22 2019-10-29 北京工商大学 A kind of high-resolution three-dimension voxel model method for reconstructing
CN110738099A (en) * 2019-08-30 2020-01-31 中山大学 A low-resolution pedestrian re-identification method based on adaptive dual-branch network
CN110738099B (en) * 2019-08-30 2022-06-07 中山大学 A low-resolution pedestrian re-identification method based on adaptive dual-branch network
CN110738601A (en) * 2019-10-23 2020-01-31 智慧视通(杭州)科技发展有限公司 low-resolution face image super-resolution reconstruction method based on three-dimensional face model
CN112825271A (en) * 2019-11-21 2021-05-21 通用电气精准医疗有限责任公司 System and method for reconstructing medical images using a deep neural network and recursive extraction of measurement data
CN111461990A (en) * 2020-04-03 2020-07-28 华中科技大学 A step-by-step approach to super-resolution imaging based on deep learning
CN111461990B (en) * 2020-04-03 2022-03-18 华中科技大学 A step-by-step approach to super-resolution imaging based on deep learning
CN111563843A (en) * 2020-04-30 2020-08-21 苏州大学 A kind of image super-resolution reconstruction method, system and related device
CN111563843B (en) * 2020-04-30 2023-11-28 苏州大学 An image super-resolution reconstruction method, system and related devices
CN111696034B (en) * 2020-05-11 2023-10-27 青岛小鸟看看科技有限公司 Image processing methods, devices and electronic equipment
CN111696034A (en) * 2020-05-11 2020-09-22 青岛小鸟看看科技有限公司 Image processing method and device and electronic equipment
CN112184555B (en) * 2020-10-21 2022-07-19 天津大学 A Stereo Image Super-Resolution Reconstruction Method Based on Deep Interactive Learning
CN112184555A (en) * 2020-10-21 2021-01-05 天津大学 Stereo image super-resolution reconstruction method based on deep interactive learning
CN112598575B (en) * 2020-12-22 2022-05-03 电子科技大学 An image information fusion and super-resolution reconstruction method based on feature processing
CN112598575A (en) * 2020-12-22 2021-04-02 电子科技大学 Image information fusion and super-resolution reconstruction method based on feature processing
CN112907441B (en) * 2020-12-29 2023-05-30 中央财经大学 Space downscaling method based on super-resolution of ground water satellite image
CN112907441A (en) * 2020-12-29 2021-06-04 中央财经大学 Space downscaling method based on super-resolution of underground water satellite image
CN115601242B (en) * 2022-12-13 2023-04-18 电子科技大学 A lightweight image super-resolution reconstruction method suitable for hardware deployment
CN115601242A (en) * 2022-12-13 2023-01-13 电子科技大学（Cn） A lightweight image super-resolution reconstruction method suitable for hardware deployment
WO2025041222A1 (en) * 2023-08-21 2025-02-27 Nec Corporation Image matching apparatus, image matching method, training apparatus, training method, and non-transitory computer-readable medium

Also Published As

Publication number Publication date
CN106796716B (en) 2018-11-16
CN106796716A (en) 2017-05-31

Similar Documents

Publication Publication Date Title
CN106796716B (en) For providing the device and method of super-resolution for low-resolution image
Lee et al. Local texture estimator for implicit representation function
Anwar et al. Densely residual laplacian super-resolution
Lan et al. MADNet: A fast and lightweight network for single-image super resolution
Dong et al. Learning a deep convolutional network for image super-resolution
Lai et al. Deep laplacian pyramid networks for fast and accurate super-resolution
CN111369440B (en) Model training and image super-resolution processing method, device, terminal and storage medium
Kim et al. Accurate image super-resolution using very deep convolutional networks
Zhang et al. Residual dense network for image super-resolution
Li et al. FilterNet: Adaptive information filtering network for accurate and fast image super-resolution
Brifman et al. Turning a denoiser into a super-resolver using plug and play priors
Shi et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network
CN109146788B (en) Super-resolution image reconstruction method and device based on deep learning
Yu et al. A unified learning framework for single image super-resolution
CN114565528B (en) A remote sensing image denoising method and system based on multi-scale and attention mechanism
CN108765343B (en) Image processing method, device, terminal and computer-readable storage medium
CN116188272B (en) Two-stage depth network image super-resolution reconstruction method suitable for multiple fuzzy cores
CN113538246A (en) Remote sensing image super-resolution reconstruction method based on unsupervised multi-stage fusion network
CN111932480A (en) Deblurred video recovery method and device, terminal equipment and storage medium
Liu et al. Multi-scale residual hierarchical dense networks for single image super-resolution
Deshpande et al. SURVEY OF SUPER RESOLUTION TECHNIQUES.
CN118628366A (en) Hyperspectral and multispectral image fusion method and system based on self-learning coupled diffusion posterior sampling
CN118967476A (en) A hyperspectral image fusion method, system, device and medium based on deep prior and dual attention residual network
Qin et al. Single-image super-resolution with multilevel residual attention network
Wang et al. Large receptive field convolutional neural network for image super-resolution

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14899412

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14899412

Country of ref document: EP

Kind code of ref document: A1