WO2020000877A1 - 用于生成图像的方法和装置 - Google Patents

用于生成图像的方法和装置 Download PDF

Info

Publication number
WO2020000877A1
WO2020000877A1 PCT/CN2018/116332 CN2018116332W WO2020000877A1 WO 2020000877 A1 WO2020000877 A1 WO 2020000877A1 CN 2018116332 W CN2018116332 W CN 2018116332W WO 2020000877 A1 WO2020000877 A1 WO 2020000877A1
Authority
WO
WIPO (PCT)
Prior art keywords
pixel
image
matrix
target
value
Prior art date
Application number
PCT/CN2018/116332
Other languages
English (en)
French (fr)
Inventor
余林韵
李磊
尹海斌
姜东�
Original Assignee
北京字节跳动网络技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京字节跳动网络技术有限公司 filed Critical 北京字节跳动网络技术有限公司
Publication of WO2020000877A1 publication Critical patent/WO2020000877A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20024—Filtering details
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning

Definitions

  • Embodiments of the present application relate to the field of computer technology, and in particular, to a method and an apparatus for generating an image.
  • High resolution means that the pixel density in the image is high, which can provide more details, which are indispensable in many practical applications.
  • high-resolution medical images are very helpful for doctors to make a correct diagnosis; using high-resolution satellite images can easily distinguish similar objects from similar ones; if high-resolution images can be provided, computer vision The performance of pattern recognition will be greatly improved.
  • the existing method for generating a high-resolution image usually directly learns a mapping relationship from a low-resolution image to a high-resolution image, and processes the original image based on the mapping relationship to generate a high-resolution image.
  • the embodiments of the present application provide a method and a device for generating an image.
  • an embodiment of the present application provides a method for generating an image.
  • the method includes: interpolating a target image to generate an interpolated image; performing super-resolution reconstruction on the interpolated image to generate a reconstructed image; The pixel value of the pixel is used to perform pixel compensation on the reconstructed image to generate a high-resolution image.
  • performing pixel compensation on the reconstructed image based on the pixel values of the pixels in the target image includes: dividing the pixel value into a plurality of continuous value ranges; for a value range of the multiple value ranges, Determine the average of the pixel values within the range of the target image and the reconstructed image, and perform pixel compensation on the reconstructed image based on the determined averages.
  • performing pixel compensation on the reconstructed image based on the determined average value includes: for a value range in a plurality of value ranges, in response to determining a target image, pixels in the reconstructed image within the value range The mean values of the values are different, and the value range is determined as a candidate value range; the target value range is determined from the determined candidate value ranges, and the pixel values in the reconstructed image within the target value range are compensated to After the compensation, the mean value of the pixel values in the target value range in the reconstructed image is equal to the mean value of the pixel values in the target value range in the target image after compensation.
  • selecting the target value range from the determined candidate value ranges includes: determining whether there are continuous candidate value ranges and the number of continuous candidate value ranges is not less than a preset value; if yes, changing The candidate value range among the continuous candidate value ranges is determined as the target value range.
  • performing pixel compensation on the reconstructed image based on the pixel value of the pixel in the target image includes: for the target image and the pixel in the reconstructed image, determining the pixel based on the pixel value comparison between the pixel and an adjacent pixel For each category, the average value of the pixel values of pixels belonging to the category in the target image and the reconstructed image is determined separately. In response to determining that the average values are different, the pixel values of pixels in the reconstructed image that belong to the category are compensated so that The average value of the pixel values of the pixels belonging to the category in the reconstructed image is equal to the average value of the pixel values of the pixels belonging to the category in the target image after compensation.
  • performing super-resolution reconstruction on the interpolated image to generate a reconstructed image includes: for a pixel in the interpolated image, extracting a first pixel matrix centered on the pixel, and performing principal component analysis on the first pixel matrix, Obtain a target matrix; for a pixel in the interpolated image, based on the target matrix corresponding to the pixel, select a filter from a pre-generated filter set, extract a second pixel matrix centered on the pixel, and use the selected filter to The two-pixel matrix is convolved to obtain a high-resolution pixel value corresponding to the pixel; the obtained high-resolution pixel values are summarized to generate a reconstructed image.
  • the filter set is generated by the following steps: extracting a set of high-resolution image samples, and sequentially downsampling and interpolating the high-resolution image samples in the set of high-resolution image samples; for the high-resolution after interpolation
  • For the pixels in the image sample a third pixel matrix centered on the pixel is extracted, and the principal component analysis is performed on the third pixel matrix to obtain a target matrix sample.
  • the obtained target matrix samples are classified and trained with each type of target matrix.
  • the filters corresponding to the samples are aggregated into a filter set.
  • classifying the obtained target matrix samples, training filters corresponding to each type of target matrix samples, and summarizing the trained filters into a filter set including: obtaining the obtained target matrix The samples are multiplied with the preset matrix to divide the target matrix samples with the same result of the dot multiplication into one class. For each type of target matrix sample, the pixels in the high-resolution image samples after interpolation are extracted and extracted. A fourth pixel matrix centered on a pixel, taking the fourth pixel matrix as an input, and outputting a high-resolution pixel corresponding to the pixel as an output, and training to obtain a filter corresponding to this type of target matrix sample.
  • performing a principal component analysis on the first pixel matrix to obtain a target matrix includes: determining a covariance matrix of the first pixel matrix; determining a feature value and a feature vector of the covariance matrix; and from the determined feature value The target feature value is selected, and the feature vector corresponding to the target feature value is formed into a feature matrix; the first pixel matrix is multiplied with the feature matrix to obtain the target matrix.
  • selecting a filter from a pre-generated filter set based on the target matrix corresponding to the pixel includes: for the pixel in the interpolated image, the target matrix corresponding to the pixel and The preset matrix performs a dot multiplication operation, and a filter corresponding to the dot multiplication operation result is selected from a pre-generated filter set.
  • an embodiment of the present application provides an apparatus for generating an image.
  • the apparatus includes: an interpolation unit configured to interpolate a target image to generate an interpolated image; and a reconstruction unit configured to superimpose the interpolated image.
  • Resolution reconstruction to generate a reconstructed image;
  • compensation unit configured to perform pixel compensation on the reconstructed image based on the pixel values of pixels in the target image to generate a high-resolution image.
  • the compensation unit includes: a dividing module configured to divide the pixel value into a plurality of continuous value ranges; and a first determining module configured to divide the value range of the plurality of value ranges into Determine the average value of the pixel values in the target image and the reconstructed image within the value range; the first compensation module is configured to perform pixel compensation on the reconstructed image based on the determined average value.
  • the first compensation module includes: a first determination submodule configured to, for a value range in a plurality of value ranges, in response to determining a target image and reconstructing an image within the value range in the value range The mean value of the pixel values is different, and the value range is determined as a candidate value range; the compensation submodule is configured to determine a target value range from the determined candidate value range, and the target value range in the reconstructed image The pixel values in the target image are compensated so that the average value of the pixel values in the target image range in the reconstructed image is equal to the average of the pixel values in the target image range in the target image after compensation.
  • the compensation sub-module is further configured to: determine whether there are continuous candidate value ranges and the number of continuous candidate value ranges is not less than a preset value; if yes, the candidates in the continuous candidate value ranges are The value range is determined as the target value range.
  • the compensation unit includes: a second determination module configured to determine, for a pixel in the target image and the reconstructed image, a category of the pixel based on a comparison of pixel values of the pixel with neighboring pixels; a second compensation A module configured to determine, for each category, an average value of pixel values of pixels belonging to the category in a target image and a reconstructed image, and in response to determining that the average values are different, compensate the pixel values of pixels belonging to the category in the reconstructed image, In this way, the average value of the pixel values of pixels belonging to the category in the reconstructed image is equal to the average value of the pixel values of pixels belonging to the category in the target image after compensation.
  • the reconstruction unit includes: an analysis module configured to extract a first pixel matrix centered on the pixel in the interpolated image, and perform a principal component analysis on the first pixel matrix to obtain a target matrix; select A module configured to select a filter from a pre-generated filter set for a pixel in an interpolated image based on a target matrix corresponding to the pixel, extract a second pixel matrix centered on the pixel, and use the selected filter pair to The second pixel matrix is convolved to obtain a high-resolution pixel value corresponding to the pixel; the generating module is configured to summarize the obtained high-resolution pixel values to generate a reconstructed image.
  • the filter set is generated by the following steps: extracting a set of high-resolution image samples, and sequentially downsampling and interpolating the high-resolution image samples in the set of high-resolution image samples; for the high-resolution after interpolation
  • For the pixels in the image sample a third pixel matrix centered on the pixel is extracted, and the principal component analysis is performed on the third pixel matrix to obtain a target matrix sample.
  • the obtained target matrix samples are classified and trained with each type of target matrix.
  • the filters corresponding to the samples are aggregated into a filter set.
  • classifying the obtained target matrix samples, training filters corresponding to each type of target matrix samples, and summarizing the trained filters into a filter set including: obtaining the obtained target matrix The samples are multiplied with the preset matrix to divide the target matrix samples with the same result of the dot multiplication into one class. For each type of target matrix sample, the pixels in the high-resolution image samples after interpolation are extracted and extracted. A fourth pixel matrix centered on a pixel, taking the fourth pixel matrix as an input, and outputting a high-resolution pixel corresponding to the pixel as an output, and training to obtain a filter corresponding to this type of target matrix sample.
  • the analysis module includes: a second determination submodule configured to determine a covariance matrix of the first pixel matrix; a third determination submodule configured to determine a feature value and a feature vector of the covariance matrix; A sub-module configured to select a target feature value from the determined feature values and form a feature matrix corresponding to the target feature value; a multiplication sub-module is configured to multiply the first pixel matrix and the feature matrix to obtain Target matrix.
  • the selection module is further configured to perform a dot multiplication operation on a pixel in the interpolated image with a preset matrix corresponding to the pixel, and select a dot multiplication operation result from a pre-generated filter set. Corresponding filter.
  • an embodiment of the present application provides an electronic device, including: one or more processors; a storage device storing one or more programs thereon; when one or more programs are processed by one or more processors Execution causes one or more processors to implement the method as in any one of the methods for generating an image.
  • an embodiment of the present application provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method as in any one of the methods for generating an image is implemented.
  • the method and device for generating an image provided in the embodiments of the present application, perform interpolation on a target image to generate an interpolated image, and then perform super-resolution reconstruction on the interpolated image to generate a reconstructed image.
  • pixel compensation is performed on the reconstructed image to generate a high-resolution image. Therefore, based on super-resolution reconstruction, compensation is performed on the reconstructed image, which improves the effect of generating a high-resolution image.
  • FIG. 1 is an exemplary system architecture diagram to which an embodiment of the present application can be applied;
  • FIG. 1 is an exemplary system architecture diagram to which an embodiment of the present application can be applied;
  • FIG. 2 is a flowchart of an embodiment of a method for generating an image according to the present application
  • FIG. 3 is a schematic diagram of an application scenario of a method for generating an image according to the present application
  • FIG. 4 is a flowchart of still another embodiment of a method for generating an image according to the present application.
  • FIG. 5 is a flowchart of an embodiment of a method for generating a filter set according to the present application
  • FIG. 6 is a schematic structural diagram of an embodiment of an apparatus for generating an image according to the present application.
  • FIG. 7 is a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application.
  • the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105.
  • the network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105.
  • the network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
  • the user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages (such as image processing requests) and the like.
  • Various communication client applications such as image processing applications, video playback applications, information browsing applications, and social platform software, can be installed on the terminal devices 101, 102, and 103.
  • the terminal devices 101, 102, and 103 may be hardware or software.
  • the terminal devices 101, 102, and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.
  • the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. It is not specifically limited here.
  • the server 105 may be a server that provides various services, such as an image processing server for performing image processing.
  • the image processing server may perform processing such as interpolation and analysis on the received target image and other data, and feed back the processing result (such as a high-resolution image) to the terminal device.
  • the server may be hardware or software.
  • the server can be implemented as a distributed server cluster consisting of multiple servers or as a single server.
  • the server can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. It is not specifically limited here.
  • the method for generating an image provided by the embodiment of the present application is generally executed by the server 105, and accordingly, the apparatus for generating an image is generally set in the server 105.
  • the terminal devices 101, 102, and 103 can also directly perform processing such as interpolation and analysis on their stored target images.
  • the method for generating images provided by the embodiments of this application can also be performed by the terminal devices 101, 102, and 103.
  • the exemplary system architecture 100 at this time may not include the network 104 and the server 105 described above.
  • terminal devices, networks, and servers in FIG. 1 are merely exemplary. Depending on the implementation needs, there can be any number of terminal devices, networks, and servers.
  • a flowchart 200 of one embodiment of a method for generating an image according to the present application is shown.
  • the method for generating an image includes the following steps:
  • Step 201 Interpolate the target image to generate an interpolated image.
  • an execution subject of the method for generating an image may first extract a target image, where the target image may be various images to be subjected to super-resolution reconstruction.
  • the above target image may be a face image, an article image, a landscape image, and the like.
  • the target image may be stored locally in advance, or may be sent by other electronic devices (for example, the terminal devices 101, 102, and 103 shown in FIG. 1) through a wired connection or a wireless connection.
  • the above wireless connection methods may include, but are not limited to, 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods now known or developed in the future.
  • the above-mentioned execution subject may use various existing image interpolation methods to interpolate the target image to enlarge the target image to a target size (for example, zoom in to 2 times, 3 times, or 4 times).
  • the above-mentioned target image interpolation may be performed using nearest neighbor interpolation, bilinear interpolation, bisquare interpolation, bicubic interpolation, or other high-order interpolation methods.
  • image interpolation is the process of generating high-resolution images from low-resolution images, and can be used to recover information lost in the images. It should be noted that the above-mentioned various image interpolation methods are well-known technologies that are widely studied and applied at present, and will not be repeated here.
  • the interpolation processing on the target image is performed first, and the resolution of the target image can be initially improved.
  • the subsequent image processing steps can be performed to improve the effect of generating a high-resolution image.
  • Step 202 Super-resolution reconstruction is performed on the interpolation image to generate a reconstructed image.
  • the above-mentioned executing subject may perform super-resolution reconstruction on the above-mentioned interpolation image by using various super-variability reconstruction methods to generate a reconstructed image.
  • Super-Resolution is to increase the resolution of the original image through hardware or software. The process of obtaining a high-resolution image from a low-resolution image is super-resolution reconstruction.
  • the above-mentioned execution subject may use a deep learning method to perform super-resolution reconstruction on the interpolation image.
  • the execution subject may input the interpolation image into a pre-trained image processing model to obtain a reconstructed image output from the image processing model.
  • the image processing model can be used for super-resolution reconstruction of an image.
  • the above image processing model can be obtained by training as follows: In the first step, multiple sets of training samples are extracted. Each of the aforementioned training samples may include a high-resolution image and a low-resolution image after processing the high-resolution image.
  • the second step is to use the machine learning method to take the low-resolution images in each group of training samples as input and the high-resolution images in this group of training samples as output to train to obtain an image processing model.
  • the image processing model can be trained using various existing model structures.
  • SRCNN Super-Resolution Convolutional Neural Network
  • the SRCNN may include three convolutional layers, and an MSE (Mean Square Error) function may be used as a loss function.
  • MSE Mel Square Error
  • the above-mentioned execution subject may use an existing super-resolution reconstruction tool (for example, an image sharpening tool RAISR (Rapid and Accurate Image SuperResolution)) to perform super-resolution on the interpolation image. reconstruction.
  • an existing super-resolution reconstruction tool for example, an image sharpening tool RAISR (Rapid and Accurate Image SuperResolution)
  • Step 203 Perform pixel compensation on the reconstructed image based on the pixel values of the pixels in the target image to generate a high-resolution image.
  • the above-mentioned execution subject may perform pixel compensation on the reconstructed image based on the pixel values of the pixels in the target image to generate a high-resolution image.
  • the pixels in the reconstructed image may be compensated by using various pixel compensation methods.
  • the above-mentioned execution body may perform pixel compensation on the reconstructed image according to the following steps:
  • the first step is to divide the pixel value into multiple continuous value ranges.
  • the pixel value is usually expressed by one byte after quantization.
  • the gray value with continuous black-gray-white changes is quantized into 256 gray levels, and the range of gray values is 0 to 255. Therefore, 256 values from 0 to 255 are usually used to identify the pixel value of a pixel.
  • the 256 pixel values from 0 to 255 can be divided into multiple continuous value ranges. For example, it can be divided into 32 value ranges.
  • the eight pixel values from 0 to 7 are in the first value range; the eight pixel values from 8 to 15 are in the second value range; and so on.
  • the execution body may first determine that the pixel value in the target image (the original target image before interpolation) is in the first value. The average value of the pixel values of the pixels in the range, and determine the average value of the pixel values of the pixels in the reconstructed image with pixel values within the first range of values.
  • the execution subject may determine the average value of the pixel values of the pixels in the target image whose pixel values are within the second value range, and determine the reconstruction The average value of the pixel values of the pixels in the image whose pixel values are within the second value range. And so on, until all 32 value ranges are processed.
  • the third step is to perform pixel compensation on the reconstructed image based on the determined average value.
  • the execution subject may perform pixel compensation on the reconstructed image in various manners based on the determined average value.
  • the execution subject may determine whether the average value of the pixel values in the target image and the reconstructed image within the value range is the same. In response to the determination being the same, the compensation of the pixel values in the value range in the reconstructed image is not performed. In response to the determination being different, the pixel values in the reconstructed image within the value range may be compensated, so that the average value after compensation is equal to the average value of the pixel values of the target image within the value range.
  • the reconstructed image is in the first value range.
  • the average value of the pixel values within is 4, and each pixel value in the first value range in the target image may be increased by 1 as a compensation value.
  • performing pixel compensation on the reconstructed image based on the determined average value may also be performed by the following steps:
  • the execution subject may determine whether the average values of the pixel values in the target image and the reconstructed image within the value range are the same. In response to the determination being the same, the compensation of the pixel values in the value range in the reconstructed image is not performed. In response to the determination being different, the value range may be determined as a candidate value range.
  • the second step is to determine the target value range from the determined candidate value range, and to compensate the pixel values in the reconstructed image within the target value range, so that the reconstructed image is within the target value range.
  • the average value of the pixel values of is equal to the average value of the pixel values in the target image within the target value range after compensation.
  • the target value range can be filtered from the determined candidate value range through various preset conditions.
  • the candidate value range may be used as the target value range.
  • the candidate value range in the foregoing continuous candidate value ranges may be determined as the target value range.
  • the first value range corresponding to the eight pixel values from 0 to 7 the second value range corresponding to the eight pixel values from 8 to 15, and the third value range corresponding to the eight pixel values from 16 to 23.
  • the fourth value ranges corresponding to the eight pixel values from 24 to 31 are candidate value ranges, and the four candidate value ranges are four consecutive candidate value ranges. Therefore, these four candidate value ranges can be used as the target value ranges.
  • the above-mentioned execution body may perform pixel compensation on the reconstructed image according to the following steps:
  • a pixel type is determined based on a comparison of pixel values of the pixel with neighboring pixels.
  • neighboring pixels can be determined in four different ways.
  • the pixel can be used as the center, and the left and right pixels adjacent to the pixel can be determined as the neighboring pixels of the pixel.
  • the pixel value of the pixel may be recorded as b; the pixel value of the left pixel is recorded as a; and the pixel value of the right pixel is recorded as c.
  • the pixel can be used as the center, and the upper and lower pixels adjacent to the pixel can be determined as the adjacent pixels of the pixel.
  • the pixel value of the pixel may be referred to as b; the pixel value of the upper pixel may be referred to as a; and the pixel value of the lower pixel may be referred to as c.
  • the pixel as the center may be used to determine an upper left pixel and a lower right pixel adjacent to the pixel as neighboring pixels of the pixel.
  • the pixel value of the pixel may be recorded as b; the pixel value of the upper left pixel is recorded as a; and the pixel value of the lower right pixel is recorded as c.
  • the pixel may be used as the center, and an upper right pixel and a lower left pixel adjacent to the pixel may be determined as neighboring pixels of the pixel.
  • the pixel value of the pixel may be referred to as b; the pixel value of the upper right pixel may be referred to as a; and the pixel value of the lower left pixel may be referred to as c.
  • the execution subject may select any one of the methods described above. For each pixel (pixel value b) in the target image and the reconstructed image, determine the neighboring pixels of the pixel (pixel value a by using the selected method). And c). Then, based on the comparison of the pixel values of the pixel with neighboring pixels, the category of the pixel is determined.
  • the average values of the pixel values of the pixels belonging to the category in the target image and the reconstructed image are determined separately.
  • compensation of the pixel values of the category in the above-mentioned reconstructed image is not performed.
  • the pixel values of pixels belonging to the category in the reconstructed image may be compensated, so that the mean values of the pixel values of pixels belonging to the category in the reconstructed image belong to the category in the target image after compensation The average of the pixel values of the pixels is equal.
  • the above-mentioned execution body may perform pixel compensation on the reconstructed image according to the following steps: First, the pixel values of each pixel in the original target image may be extracted to determine the average pixel value of the target image. . Then, the pixel value of each pixel in the reconstructed image is extracted, and the average pixel value of the reconstructed image is determined. Thereafter, it is determined whether the average pixel value of the target image and the average pixel value of the reconstructed image are the same. If they are different, the pixels of the reconstructed image are compensated so that the compensated average pixel value of the reconstructed image is equal to the average pixel value of the target image.
  • FIG. 3 is a schematic diagram of an application scenario of the method for generating an image according to this embodiment.
  • a user first sends an image processing request to an image processing server using a terminal device, and the image processing request includes a target image 301 to be subjected to super-resolution image reconstruction.
  • the image processing server receives the target image 301, it first interpolates the target image 301 to obtain an interpolated image. Then, super-resolution reconstruction is performed on the interpolation image to generate a reconstructed image. Finally, pixel compensation is performed on the reconstructed image to generate a high-resolution image 302.
  • the target image is interpolated to generate an interpolated image, and then the interpolated image is super-resolution reconstructed to generate a reconstructed image. Finally, based on the pixel values of the pixels in the target image, The reconstructed image is pixel-compensated to generate a high-resolution image. Therefore, in the process of generating a high-resolution image, first performing interpolation processing on the target image can initially improve the resolution of the target image. On the basis of the above-mentioned interpolation image, the subsequent image processing steps can be performed to improve the effect of generating a high-resolution image. At the same time, based on the super-resolution reconstruction, the reconstructed image is compensated, which improves the effect of high-resolution image generation.
  • the process 400 of the method for generating an image includes the following steps:
  • Step 401 Interpolate the target image to generate an interpolated image.
  • an execution subject of the method for generating an image may first extract a target image, where the target image may be various images to be subjected to super-resolution reconstruction. After extracting the target image, the above-mentioned execution subject may use various existing image interpolation methods to interpolate the target image to enlarge the target image to a target size (for example, zoom in to 2 times, 3 times, or 4 times).
  • the above-mentioned target image interpolation may be performed using nearest neighbor interpolation, bilinear interpolation, bisquare interpolation, bicubic interpolation, or other high-order interpolation methods.
  • the interpolation processing on the target image is performed first, and the resolution of the target image can be initially improved.
  • the subsequent image processing steps can be performed to improve the effect of generating a high-resolution image.
  • Step 402 For a pixel in the interpolation image, extract a first pixel matrix centered on the pixel, and perform principal component analysis on the first pixel matrix to obtain a target matrix.
  • the above-mentioned execution subject may first extract a first pixel matrix centered on the pixel.
  • the first pixel matrix may include pixel values of pixels in a square area (for example, a 3 ⁇ 3 image patch) with the pixel as a center.
  • the execution body may perform principal component analysis (PCA) on the first pixel matrix to obtain a target matrix.
  • PCA principal component analysis
  • the covariance matrix of the first pixel matrix may be determined first.
  • the eigenvalues and eigenvectors of the covariance matrix can be determined.
  • the first pixel matrix can be projected into the space formed by the feature vectors, and the matrix obtained after the projection can be determined as the target matrix.
  • principal component analysis is also called principal component analysis, which aims to use the idea of dimensionality reduction to transform multiple indicators into a few comprehensive indicators.
  • principal component analysis is a technique that simplifies data sets. It is a linear transformation that transforms the data into a new coordinate system.
  • Principal component analysis can be used to reduce the dimensionality of the data set while maintaining the feature that the variance of the data set contributes the most. This is done by preserving low-order principal components and ignoring higher-order principal components. Such low-order components can often retain the most important aspects of the data.
  • the first pixel matrix centered on the pixel may be a square area centered on the pixel (for example, a 3 ⁇ 3 image A matrix corresponding to a patch).
  • the values in the first pixel matrix may correspond to the pixels in the square area one by one, that is, the values in the i-th row and the j-th column of the first pixel matrix are the pixel values of the pixels in the i-th row and the j-th column in the square area.
  • i is an integer not less than 1 and not more than the number of rows of the first pixel matrix
  • j is an integer not less than 1 and not more than the number of columns of the first pixel matrix.
  • the first pixel matrix centered on the pixel can be obtained in the following manner: In the first step, the pixel centered on the pixel is extracted. The pixel values of the pixels in the square area generate a pixel matrix in which the values in the matrix correspond one-to-one with the pixels in the square area. It should be noted that, for some pixels (such as pixels located at the edge of the image), there are no corresponding pixel values at certain positions in the pixel matrix of the pixels (for example, pixels located at the upper edge of the image, pixels of the pixel The first row of the matrix has no corresponding pixel value).
  • the values of these positions can be set to preset values (for example, 0).
  • the pixel matrix is converted into a row vector. Since a row vector is a special form of a matrix, the vector can be determined as the first pixel matrix centered on the pixel. As an example, for a certain pixel in the above-mentioned interpolation image, a 3 ⁇ 3 patch pixel centered on the pixel can be extracted to generate a 3 ⁇ 3 (3 rows and 3 columns) pixel matrix. Then, the 3 ⁇ 3 pixel matrix may be converted into a row vector, and the row vector may be determined as a first pixel matrix of 1 ⁇ 9 (1 row and 9 columns).
  • a principal component analysis of the first pixel matrix of the pixel may be performed by using the following steps:
  • the first step is to determine the covariance matrix of the first pixel matrix.
  • the covariance matrix of the matrix is a 9 ⁇ 9 matrix.
  • the second step is to determine the eigenvalues and eigenvectors of the covariance matrix.
  • each feature value may correspond to a feature vector.
  • the calculation method of the covariance matrix, the calculation method of the eigenvalues of the matrix, and the calculation method of the eigenvectors of the matrices are well-known techniques widely studied and applied in the field of mathematics at present, and will not be repeated here.
  • a target feature value is selected from the determined feature values, and a feature vector corresponding to the target feature value is formed into a feature matrix.
  • the execution subject may select the target feature value from the determined feature values by using various selection methods. For example, a preset number of feature vectors can be selected as the target feature vector from the feature vectors corresponding to the feature values in the order of the feature values from large to small, and the target feature vectors are sequentially combined to obtain the The transpose is determined as a feature matrix.
  • the covariance matrix is a 9 ⁇ 9 matrix.
  • the execution body determines that the matrix has 9 eigenvalues and 9 corresponding eigenvectors
  • the eigenvectors can be sorted in the order of the eigenvalues from large to small. Then, the first 8 feature vectors can be selected and combined to obtain an 8 ⁇ 9 (8 rows and 9 columns) matrix.
  • the transpose of the matrix can be determined as a feature matrix, and the feature matrix is a 9 ⁇ 8 (9 rows and 8 columns) matrix.
  • the fourth step is to multiply the first pixel matrix and the feature matrix to obtain the target matrix.
  • the first pixel matrix is a matrix of 1 ⁇ 9 (1 row and 9 columns)
  • the feature matrix is a matrix of 9 ⁇ 8 (9 rows and 8 columns).
  • the two matrices are multiplied to obtain 1 ⁇ 8 (1 row and 8 columns).
  • Target matrix is 1 ⁇ 8 (1 row and 8 columns).
  • the principal component analysis is used to process the first pixel matrix, and the first pixel matrix is reduced in dimension.
  • the important features in the first pixel matrix of each pixel can be retained, so that The difference is more obvious, so that the pixels in the interpolated image can be classified more accurately.
  • Step 403 For a pixel in the interpolation image, based on the target matrix corresponding to the pixel, a filter is selected from a pre-generated filter set, a second pixel matrix centered on the pixel is extracted, and the selected first The two-pixel matrix is convolved to obtain a high-resolution pixel value corresponding to the pixel.
  • the execution subject may select a filter from a pre-generated filter set based on a target matrix corresponding to the pixel.
  • a filter set may be stored in the execution body in advance.
  • Each filter in the above filter set may correspond to a category of pixels.
  • the above-mentioned execution body may analyze or calculate the target matrix of each pixel, determine the category of each pixel according to the analysis result or the calculation result, and then select a corresponding filter for each pixel.
  • the execution body may substitute the target matrix into a preset formula or function for calculation, and obtain a calculation result (for example, a value).
  • each filter in the filter set may be a parameter matrix or a parameter vector.
  • a filter is used to perform a convolution calculation on the target matrix of a pixel to obtain a high-resolution pixel value of the pixel.
  • the above-mentioned execution body can extract a second pixel matrix centered on the pixel.
  • the second pixel matrix may include pixel values of pixels in a square area (for example, a 7 ⁇ 7 image patch) with the pixel as a center.
  • the above-mentioned second pixel matrix may be a 1 ⁇ 49 row vector.
  • the selected second filter may be used to convolve the second pixel matrix to obtain a high-resolution pixel value corresponding to the pixel.
  • the size of the second pixel matrix of the pixel may be the same as or different from the size of the first pixel matrix, which is not limited herein.
  • the category of the pixel is determined based on the principal component analysis, a filter corresponding to the category is selected, and a high-resolution pixel value is calculated, so that each pixel in the above-mentioned interpolation image can be more accurately classified.
  • the execution body may perform a dot multiplication operation on a target matrix corresponding to the pixel and a preset matrix, and use a dot multiplication operation result as the The category of the pixel. Then, a filter corresponding to a dot multiplication operation result is selected from a pre-generated filter set. What needs to be explained here is that the total number of categories of pixels and the dot multiplication result of each category can be determined in advance based on the analysis processing of a large number of image samples, and a filter corresponding to each category can be generated in advance.
  • the above-mentioned execution body may store or extract a correspondence relationship between each filter and a dot multiplication operation result.
  • FIG. 5 shows a flowchart of an embodiment of a method for generating a filter set according to the present application.
  • the method 500 for generating a filter set includes the following steps:
  • Step 501 Extract a set of high-resolution image samples, and sequentially down-sample and interpolate the high-resolution image samples in the set of high-resolution image samples.
  • each high-resolution sample in the above-mentioned high-resolution image sample set it may be down-sampled first, and then the down-sampled high-resolution samples are interpolated.
  • the downsampling multiple can be set in advance.
  • 2x downsampling of a high-resolution image sample may be converting an image in a 2 ⁇ 2 image block in the high-resolution image sample into a pixel, and the pixel value of the pixel is equal to the 2 ⁇ 2 image block.
  • the same interpolation method as in step 401 may be used for interpolation, and details are not described herein again.
  • Step 502 For a pixel in the high-resolution image sample after interpolation, extract a third pixel matrix centered on the pixel, and perform principal component analysis on the third pixel matrix to obtain a target matrix sample.
  • a third pixel matrix centered on the pixel can be extracted, and the third pixel matrix is subjected to principal component analysis to obtain a target matrix sample.
  • the third pixel matrix extraction step for pixels in the high-resolution image sample after interpolation is basically the same as the first pixel matrix extraction step for the above-mentioned interpolation image;
  • the steps of the target matrix sample are basically the same as the operation of performing the principal component analysis on the first pixel matrix, and are not repeated here.
  • the number of rows and columns of the first pixel matrix may be the same as the number of rows and columns of the third pixel matrix, respectively.
  • each of the first pixel matrix and the third pixel matrix may be a 1 ⁇ 9 matrix.
  • Step 503 Classify the obtained target matrix samples.
  • each obtained target matrix sample may be substituted into a preset formula or function for calculation, to obtain a calculation result (for example, a numerical value), and the target matrix samples corresponding to the same calculation result are classified into the same class.
  • a calculation result for example, a numerical value
  • Each category can be characterized by a calculation result.
  • a point multiplication operation may first be performed on each obtained target matrix sample and a preset matrix. Then, the target matrix samples with the same result of the dot multiplication operation are divided into one class.
  • Step 504 Train filters corresponding to each type of target matrix sample, and aggregate the trained filters into a filter set.
  • a machine learning method can be used to train the filter corresponding to that type.
  • a fourth pixel matrix centered on the pixel may be first extracted, and the fourth pixel may be A matrix is used as an input, and a high-resolution pixel corresponding to the pixel is used as an output, and a filter corresponding to this type of target matrix sample is obtained by training using a machine learning method.
  • the step of extracting the fourth pixel matrix is basically the same as the step of extracting the second pixel matrix described above, and details are not described herein again.
  • the number of rows and columns of the second pixel matrix may be the same as the number of rows and columns of the fourth pixel matrix, respectively.
  • each of the second pixel matrix and the fourth pixel matrix may be a 1 ⁇ 49 matrix.
  • a correspondence relationship between a point multiplication operation result and a filter may also be established.
  • the result of the dot multiplication operation can be used as the key of the key-value pair
  • the parameter vector or parameter matrix used to characterize the filter can be used as the value of the key-value pair. relationship.
  • the dimensionality reduction classification of pixels by principal component analysis technology can preserve the important characteristics of each pixel, so that the differences between different pixels are more obvious, and the pixel classification can be more accurate. Filter training on this basis can improve the pertinence and accuracy of the filter.
  • Step 404 Summarize the obtained high-resolution pixel values to generate a reconstructed image.
  • the above-mentioned execution body can summarize the high-resolution pixel values corresponding to the obtained pixels. Generate a reconstructed image.
  • Step 405 Divide the pixel value into multiple continuous value ranges.
  • the pixel value can be divided into a plurality of continuous value range images.
  • the prime value is usually expressed in one byte after quantization.
  • the gray value with continuous black-gray-white changes is quantized to 256 gray levels, and the range of gray values is 0 to 255. Therefore, 256 values from 0 to 255 are usually used to identify the pixel value of a pixel.
  • the 256 pixel values from 0 to 255 can be divided into multiple continuous value ranges. For example, it can be divided into 32 value ranges.
  • the eight pixel values from 0 to 7 are in the first value range; the eight pixel values from 8 to 15 are in the second value range; and so on.
  • Step 406 For the value ranges of the multiple value ranges, determine the average values of the pixel values in the target image and the reconstructed image within the value ranges.
  • the execution subject determines the average value of the pixel values in the target image and the reconstructed image within the value range for the value ranges in the multiple value ranges.
  • the execution body may first determine that the pixel value in the target image (the original target image before interpolation) is in the first value.
  • the average value of the pixel values of the pixels in the range and determine the average value of the pixel values of the pixels in the reconstructed image with pixel values within the first range of values.
  • the execution subject may determine the average value of the pixel values of the pixels in the target image whose pixel values are within the second value range, and determine the reconstruction The average value of the pixel values of the pixels in the image whose pixel values are within the second value range. And so on, until all 32 value ranges are processed.
  • Step 407 For a value range among a plurality of value ranges, in response to determining that the average value of pixel values in the target image and the reconstructed image within the value range is different, determine the value range as a candidate value range.
  • the execution subject may determine whether the average values of the pixel values in the target image and the reconstructed image within the value ranges are the same. In response to the determination being the same, the compensation of the pixel values in the value range in the reconstructed image is not performed. In response to the determination being different, the value range may be determined as a candidate value range.
  • Step 408 Determine a target value range from the determined candidate value range, and compensate the pixel values in the reconstructed image within the target value range, so that the average value of the pixel values in the reconstructed image within the target value range After compensation, the average value of the pixel values within the target value range in the target image is equal to generate a high-resolution image.
  • the above-mentioned execution subject may determine the target value range from the determined candidate value range, and compensate the pixel values within the target value range in the reconstructed image to make the target value in the reconstructed image.
  • the average value of the pixel values in the range is equal to the average value of the pixel values in the target image in the target value range after compensation.
  • the first value range corresponding to the eight pixel values from 0 to 7 the second value range corresponding to the eight pixel values from 8 to 15, and the third value range corresponding to the eight pixel values from 16 to 23.
  • the fourth value ranges corresponding to the eight pixel values from 24 to 31 are candidate value ranges, and the four candidate value ranges are four consecutive candidate value ranges. Therefore, these four candidate value ranges can be used as the target value ranges.
  • the process 400 of the method for generating an image in this embodiment highlights the use of a principal component analysis method to determine a target matrix and selects a filter to generate a reconstruction The image step, while highlighting the filter set generation step. Therefore, in the solution described in this embodiment, the pixels are dimensionally reduced and classified by the principal component analysis technology, and important features of each pixel can be retained, so that differences between different pixels are more obvious, and thus pixel classification can be more accurate. Filter training on this basis can improve the pertinence and accuracy of the filter. Using the trained filters to generate the reconstructed image further improves the effect of reconstructed image generation. Based on the above, pixel compensation is performed on the reconstructed image, which further improves the effect of high-resolution image generation.
  • this application provides an embodiment of an apparatus for generating an image.
  • the apparatus embodiment corresponds to the method embodiment shown in FIG. 2.
  • the device can be specifically applied to various electronic devices.
  • the apparatus 600 for generating an image includes: an interpolation unit 601 configured to interpolate a target image to generate an interpolated image; and a reconstruction unit 602 configured to perform the interpolation image described above.
  • Super-resolution reconstruction to generate a reconstructed image;
  • a compensation unit 603 is configured to perform pixel compensation on the reconstructed image based on the pixel values of the pixels in the target image to generate a high-resolution image.
  • the compensation unit 603 may include a division module, a first determination module, and a first compensation module (not shown in the figure).
  • the above-mentioned dividing module may be configured to divide the pixel value into a plurality of continuous value ranges.
  • the first determining module may be configured to determine an average value of pixel values in the target image and the reconstructed image within the value range for the value ranges in the multiple value ranges.
  • the first compensation module may be configured to perform pixel compensation on the reconstructed image based on the determined average value.
  • the foregoing first compensation module may include a first determination submodule and a compensation submodule (not shown in the figure).
  • the first determining sub-module may be configured to, for the value ranges in the multiple value ranges, in response to determining that the average values of pixel values within the value ranges in the target image and the reconstructed image are different, This value range is determined as a candidate value range.
  • the compensation sub-module may be configured to determine a target value range from the determined candidate value range, and compensate the pixel values in the reconstructed image within the target value range, so that the reconstructed image includes the target in the target. After compensation, the average value of the pixel values in the value range is equal to the average value of the pixel values in the target image in the target value range.
  • the compensation sub-module may be further configured to determine whether there are continuous candidate value ranges and the number of the continuous candidate value ranges is not less than a preset value; if yes, the The candidate value range among the continuous candidate value ranges is determined as the target value range.
  • the compensation unit 603 may include a second determination module and a second compensation module (not shown in the figure).
  • the second determining module may be configured to determine, for a pixel in the target image and the reconstructed image, a category of the pixel based on a comparison of pixel values of the pixel with neighboring pixels.
  • the second compensation module is configured to determine, for each category, an average value of pixel values of pixels belonging to the category in the target image and the reconstructed image respectively, and in response to determining that the average values are different, the pixels in the reconstructed image that belong to the category are The pixel values of the pixel are compensated so that the average value of the pixel values of the pixels belonging to the category in the reconstructed image is equal to the average value of the pixel values of the pixels belonging to the category in the target image.
  • the reconstruction unit 602 may include an analysis module, a selection module, and a generation module (not shown in the figure).
  • the analysis module may be configured to extract a first pixel matrix centered on the pixel for the pixel in the interpolation image, and perform a principal component analysis on the first pixel matrix to obtain a target matrix.
  • the above selection module may be configured to select a filter from a pre-generated filter set for a pixel in the above-mentioned interpolation image based on a target matrix corresponding to the pixel, and extract a second pixel matrix centered on the pixel.
  • the filter convolves the second pixel matrix to obtain a high-resolution pixel value corresponding to the pixel.
  • the generating module may be configured to aggregate the obtained high-resolution pixel values to generate a reconstructed image.
  • the foregoing filter set may be generated by the following steps: extracting a set of high-resolution image samples, and sequentially down-sampling the high-resolution image samples in the above-mentioned high-resolution image sample set. And interpolation; for a pixel in the high-resolution image sample after interpolation, extract a third pixel matrix centered on the pixel, and perform principal component analysis on the third pixel matrix to obtain a target matrix sample; and obtain the target matrix sample
  • the samples are classified, the filters corresponding to each type of target matrix samples are trained, and the trained filters are aggregated into a filter set.
  • the obtained target matrix samples are classified in the above-mentioned generating step of the filter set, and a filter corresponding to each type of target matrix samples is trained, and the trained The filters are aggregated into a filter set, which may include: performing a dot multiplication operation on the obtained target matrix samples and a preset matrix, and dividing the target matrix samples with the same result of the dot multiplication operation into one class; corresponding to each type of target matrix samples Pixel in the high-resolution image sample after interpolation, extract a fourth pixel matrix centered on the pixel, take the above-mentioned fourth pixel matrix as an input, and use the high-resolution pixel corresponding to the pixel as an output.
  • the filter corresponding to this type of target matrix sample may include: performing a dot multiplication operation on the obtained target matrix samples and a preset matrix, and dividing the target matrix samples with the same result of the dot multiplication operation into one class; corresponding to each type of target matrix samples Pixel in the high-resolution image sample after interpolation,
  • the foregoing analysis module may include a second determination submodule, a third determination submodule, a composition submodule, and a composition submodule (not shown in the figure).
  • the second determination sub-module may be configured to determine a covariance matrix of the first pixel matrix.
  • the third determination sub-module may be configured to determine an eigenvalue and a eigenvector of the covariance matrix.
  • the composition sub-module may be configured to select a target feature value from the determined feature values, and form a feature matrix corresponding to the target feature value.
  • the multiplication sub-module may be configured to multiply the first pixel matrix and the feature matrix to obtain a target matrix.
  • the selection module may be further configured to perform a dot multiplication operation on a pixel in the interpolation image with a preset matrix corresponding to the pixel, and from a pre-generated filter The filter set corresponds to the result of the dot product operation.
  • the apparatus provided by the above embodiment of the present application performs interpolation on the target image by the interpolation unit 601 to generate an interpolation image, and then the reconstruction unit 602 performs super-resolution reconstruction on the interpolation image to generate a reconstructed image. Finally, the compensation unit 603 is based on the target The pixel values of the pixels in the image are used to perform pixel compensation on the reconstructed image to generate a high-resolution image. Based on the super-resolution reconstruction, the reconstructed image is compensated to improve the effect of generating a high-resolution image.
  • FIG. 7 illustrates a schematic structural diagram of a computer system 700 suitable for implementing an electronic device according to an embodiment of the present application.
  • the electronic device shown in FIG. 7 is only an example, and should not impose any limitation on the functions and scope of use of the embodiments of the present application.
  • the computer system 700 includes a central processing unit (CPU) 701, which can be loaded into a random access memory (RAM) 703 from a program stored in a read-only memory (ROM) 702 or from a storage portion 708 Instead, perform various appropriate actions and processes.
  • RAM random access memory
  • ROM read-only memory
  • various programs and data required for the operation of the system 700 are also stored.
  • the CPU 701, ROM 702, and RAM 703 are connected to each other through a bus 704.
  • An input / output (I / O) interface 705 is also connected to the bus 704.
  • the following components are connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), and the speaker; a storage portion 708 including a hard disk and the like And a communication section 709 including a network interface card such as a LAN card, a modem, and the like.
  • the communication section 709 performs communication processing via a network such as the Internet.
  • the driver 710 is also connected to the I / O interface 705 as needed.
  • a removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.
  • the process described above with reference to the flowchart may be implemented as a computer software program.
  • embodiments of the present disclosure include a computer program product including a computer program carried on a computer-readable medium, the computer program containing program code for performing a method shown in a flowchart.
  • the computer program may be downloaded and installed from a network through the communication section 709, and / or installed from a removable medium 711.
  • CPU central processing unit
  • the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two.
  • the computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programming read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal that is included in baseband or propagated as part of a carrier wave, and which carries computer-readable program code. Such a propagated data signal may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable medium may send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.
  • Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
  • each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, which contains one or more functions to implement a specified logical function Executable instructions.
  • the functions labeled in the blocks may also occur in a different order than those labeled in the drawings. For example, two blocks represented one after the other may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved.
  • each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts can be implemented by a dedicated hardware-based system that performs the specified function or operation , Or it can be implemented with a combination of dedicated hardware and computer instructions.
  • the units described in the embodiments of the present application may be implemented by software or hardware.
  • the described unit may also be provided in a processor, for example, it may be described as: a processor includes an interpolation unit, a reconstruction unit, and a compensation unit. Among them, the names of these units do not constitute a limitation on the unit itself in some cases.
  • the interpolation unit can also be described as "a unit that interpolates a target image and generates an interpolation image.”
  • the present application further provides a computer-readable medium, which may be included in the device described in the foregoing embodiments; or may exist alone without being assembled into the device.
  • the computer-readable medium carries one or more programs.
  • the device is caused to: interpolate a target image to generate an interpolated image; and perform super-resolution reconstruction on the interpolated image, Generate a reconstructed image; perform pixel compensation on the reconstructed image based on the pixel values of the pixels in the target image to generate a high-resolution image.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

本申请实施例公开了用于生成图像的方法和装置。该方法的一具体实施方式包括：对目标图像进行插值，生成插值图像；对该插值图像进行超分辨率重建，生成重建图像；基于该目标图像中的像素的像素值，对该重建图像进行像素补偿，生成高分辨率图像。该实施方式提高了高分辨率图像生成的效果。

Description

用于生成图像的方法和装置
本专利申请要求于2018年6月26日提交的、申请号为201810669838.8、申请人为北京字节跳动网络技术有限公司、发明名称为“用于生成图像的方法和装置”的中国专利申请的优先权，该申请的全文以引用的方式并入本申请中。
技术领域
本申请实施例涉及计算机技术领域，具体涉及用于生成图像的方法和装置。
背景技术
在电子图像应用领域，人们经常期望得到高分辨率图像。高分辨率意味着图像中的像素密度高，能够提供更多的细节，而这些细节在许多实际应用中不可或缺。例如，高分辨率医疗图像对于医生做出正确的诊断是非常有帮助的；使用高分辨率卫星图像就很容易从相似物中区别相似的对象；如果能够提供高分辨的图像，计算机视觉中的模式识别的性能就会大大提高。
现有的生成高分辨率图像的方法，通常是直接学习由低分辨率图像到高分辨率图像的映射关系，基于该映射关系对原始图像进行处理，生成高分辨率图像。
发明内容
本申请实施例提出了用于生成图像的方法和装置。
第一方面，本申请实施例提供了一种用于生成图像的方法，该方法包括：对目标图像进行插值，生成插值图像；对插值图像进行超分辨率重建，生成重建图像；基于目标图像中的像素的像素值，对重建图像进行像素补偿，生成高分辨率图像。
在一些实施例中，基于目标图像中的像素的像素值，对重建图像进行像素补偿，包括：将像素值划分为连续的多个取值范围；对于多个取值范围中的取值范围，分别确定目标图像、重建图像中在该取值范围内的像素值的均值；基于所确定的均值，对重建图像进行像素补偿。
在一些实施例中，基于所确定的均值，对重建图像进行像素补偿，包括：对于多个取值范围中的取值范围，响应于确定目标图像、重建图像中在该取值范围内的像素值的均值不相同，将该取值范围确定为候选取值范围；从所确定的候选取值范围中确定目标取值范围，对重建图像中在目标取值范围内的像素值进行补偿，以使重建图像中在目标取值范围内的像素值的均值在补偿后与目标图像中在目标取值范围内的像素值的均值相等。
在一些实施例中，从所确定的候选取值范围中选取目标取值范围，包括：确定是否存在连续的候选取值范围且连续的候选取值范围的数量不小于预设数值；若是，将连续的候选取值范围中的候选取值范围确定为目标取值范围。
在一些实施例中，基于目标图像中的像素的像素值，对重建图像进行像素补偿，包括：对于目标图像、重建图像中的像素，基于该像素与相邻像素的像素值比较，确定该像素的类别；对于每一类别，分别确定目标图像、重建图像中属于该类别的像素的像素值的均值，响应于确定均值不同，将重建图像中属于该类别的像素的像素值进行补偿，以使重建图像中属于该类别的像素的像素值的均值在补偿后与目标图像中属于该类别的像素的像素值的均值相等。
在一些实施例中，对插值图像进行超分辨率重建，生成重建图像，包括：对于插值图像中的像素，提取以该像素为中心的第一像素矩阵，对第一像素矩阵进行主成分分析，得到目标矩阵；对于插值图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值；将所得到的高分辨率像素值进行汇总，生成重建图像。
在一些实施例中，滤波器集合通过如下步骤生成：提取高分辨率图像样本集合，对高分辨率图像样本集合中的高分辨率图像样本依次进行下采样和插值；对于插值后的高分辨率图像样本中的像素，提取以该像素为中心的第三像素矩阵，对第三像素矩阵进行主成分分析，得到目标矩阵样本；对所得到的目标矩阵样本进行分类，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合。
在一些实施例中，对所得到的目标矩阵样本进行分类，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合，包括：将所得到的目标矩阵样本与预设矩阵进行点乘运算，将点乘运算结果相同的目标矩阵样本划分为一类；对于每一类目标矩阵样本对应的、插值后的高分辨率图像样本中的像素，提取以该像素为中心的第四像素矩阵，将第四像素矩阵作为输入，将该像素对应的高分辨率像素作为输出，训练得到与该类目标矩阵样本对应的滤波器。
在一些实施例中，对第一像素矩阵进行主成分分析，得到目标矩阵，包括：确定第一像素矩阵的协方差矩阵；确定协方差矩阵的特征值和特征向量；从所确定的特征值中选取目标特征值，将目标特征值对应的特征向量组成特征矩阵；将第一像素矩阵与特征矩阵相乘，得到目标矩阵。
在一些实施例中，对于插值图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，包括：对于插值图像中的像素，将该像素对应的目标矩阵与预设矩阵进行点乘运算，从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。
第二方面，本申请实施例提供了一种用于生成图像的装置，该装置包括：插值单元，被配置成对目标图像进行插值，生成插值图像；重建单元，被配置成对插值图像进行超分辨率重建，生成重建图像；补偿单元，被配置成基于目标图像中的像素的像素值，对重建图像进行像素补偿，生成高分辨率图像。
在一些实施例中，补偿单元包括：划分模块，被配置成将像素值划分为连续的多个取值范围；第一确定模块，被配置成对于多个取值 范围中的取值范围，分别确定目标图像、重建图像中在该取值范围内的像素值的均值；第一补偿模块，被配置成基于所确定的均值，对重建图像进行像素补偿。
在一些实施例中，第一补偿模块，包括：第一确定子模块，被配置成对于多个取值范围中的取值范围，响应于确定目标图像、重建图像中在该取值范围内的像素值的均值不相同，将该取值范围确定为候选取值范围；补偿子模块，被配置成从所确定的候选取值范围中确定目标取值范围，对重建图像中在目标取值范围内的像素值进行补偿，以使重建图像中在目标取值范围内的像素值的均值在补偿后与目标图像中在目标取值范围内的像素值的均值相等。
在一些实施例中，补偿子模块进一步被配置成：确定是否存在连续的候选取值范围且连续的候选取值范围的数量不小于预设数值；若是，将连续的候选取值范围中的候选取值范围确定为目标取值范围。
在一些实施例中，补偿单元，包括：第二确定模块，被配置成对于目标图像、重建图像中的像素，基于该像素与相邻像素的像素值比较，确定该像素的类别；第二补偿模块，被配置成对于每一类别，分别确定目标图像、重建图像中属于该类别的像素的像素值的均值，响应于确定均值不同，将重建图像中属于该类别的像素的像素值进行补偿，以使重建图像中属于该类别的像素的像素值的均值在补偿后与目标图像中属于该类别的像素的像素值的均值相等。
在一些实施例中，重建单元包括：分析模块，被配置成对于插值图像中的像素，提取以该像素为中心的第一像素矩阵，对第一像素矩阵进行主成分分析，得到目标矩阵；选取模块，被配置成对于插值图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值；生成模块，被配置成将所得到的高分辨率像素值进行汇总，生成重建图像。
在一些实施例中，滤波器集合通过如下步骤生成：提取高分辨率图像样本集合，对高分辨率图像样本集合中的高分辨率图像样本依次 进行下采样和插值；对于插值后的高分辨率图像样本中的像素，提取以该像素为中心的第三像素矩阵，对第三像素矩阵进行主成分分析，得到目标矩阵样本；对所得到的目标矩阵样本进行分类，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合。
在一些实施例中，对所得到的目标矩阵样本进行分类，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合，包括：将所得到的目标矩阵样本与预设矩阵进行点乘运算，将点乘运算结果相同的目标矩阵样本划分为一类；对于每一类目标矩阵样本对应的、插值后的高分辨率图像样本中的像素，提取以该像素为中心的第四像素矩阵，将第四像素矩阵作为输入，将该像素对应的高分辨率像素作为输出，训练得到与该类目标矩阵样本对应的滤波器。
在一些实施例中，分析模块包括：第二确定子模块，被配置成确定第一像素矩阵的协方差矩阵；第三确定子模块，被配置成确定协方差矩阵的特征值和特征向量；组成子模块，被配置成从所确定的特征值中选取目标特征值，将目标特征值对应的特征向量组成特征矩阵；相乘子模块，被配置成将第一像素矩阵与特征矩阵相乘，得到目标矩阵。
在一些实施例中，选取模块进一步配配置成：对于插值图像中的像素，将该像素对应的目标矩阵与预设矩阵进行点乘运算，从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。
第三方面，本申请实施例提供了一种电子设备，包括：一个或多个处理器；存储装置，其上存储有一个或多个程序，当一个或多个程序被一个或多个处理器执行，使得一个或多个处理器实现如用于生成图像的方法中任一实施例的方法。
第四方面，本申请实施例提供了一种计算机可读介质，其上存储有计算机程序，该程序被处理器执行时实现如用于生成图像的方法中任一实施例的方法。
本申请实施例提供的用于生成图像的方法和装置，通过对目标图像进行插值，以便生成插值图像，而后对插值图像进行超分辨率重建， 生成重建图像，最后基于目标图像中的像素的像素值，对重建图像进行像素补偿，生成高分辨率图像，从而，在进行超分辨率重建的基础上，对重建图像进行补偿，提高了高分辨率图像生成的效果。
附图说明
通过阅读参照以下附图所作的对非限制性实施例所作的详细描述，本申请的其它特征、目的和优点将会变得更明显：
图1是本申请的一个实施例可以应用于其中的示例性系统架构图；
图2是根据本申请的用于生成图像的方法的一个实施例的流程图；
图3是根据本申请的用于生成图像的方法的一个应用场景的示意图；
图4是根据本申请的用于生成图像的方法的又一个实施例的流程图；
图5图是根据本申请的用于生成滤波器集合的方法的一个实施例的流程图；
图6是根据本申请的用于生成图像的装置的一个实施例的结构示意图；
图7是适于用来实现本申请实施例的电子设备的计算机系统的结构示意图。
具体实施方式
下面结合附图和实施例对本申请作进一步的详细说明。可以理解的是，此处所描述的具体实施例仅仅用于解释相关发明，而非对该发明的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与有关发明相关的部分。
需要说明的是，在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。下面将参考附图并结合实施例来详细说明本申请。
如图1所示，系统架构100可以包括终端设备101、102、103，网络104和服务器105。网络104用以在终端设备101、102、103和服务器105之间提供通信链路的介质。网络104可以包括各种连接类型，例如有线、无线通信链路或者光纤电缆等等。
用户可以使用终端设备101、102、103通过网络104与服务器105交互，以接收或发送消息(例如图像处理请求)等。终端设备101、102、103上可以安装有各种通讯客户端应用，例如图像处理类应用、视频播放类应用、资讯浏览类应用、社交平台软件等。
终端设备101、102、103可以是硬件，也可以是软件。当终端设备101、102、103为硬件时，可以是具有显示屏并且支持网页浏览的各种电子设备，包括但不限于智能手机、平板电脑、膝上型便携计算机和台式计算机等等。当终端设备101、102、103为软件时，可以安装在上述所列举的电子设备中。其可以实现成多个软件或软件模块(例如用来提供分布式服务)，也可以实现成单个软件或软件模块。在此不做具体限定。
服务器105可以是提供各种服务的服务器，例如用于进行图像处理的图像处理服务器。图像处理服务器可以对接收到的目标图像等数据进行插值、分析等处理，并将处理结果(例如高分辨率图像)反馈给终端设备。
需要说明的是，服务器可以是硬件，也可以是软件。当服务器为硬件时，可以实现成多个服务器组成的分布式服务器集群，也可以实现成单个服务器。当服务器为软件时，可以实现成多个软件或软件模块(例如用来提供分布式服务)，也可以实现成单个软件或软件模块。在此不做具体限定。
需要指出的是，本申请实施例所提供的用于生成图像的方法一般由服务器105执行，相应地，用于生成图像的装置一般设置于服务器105中。
需要指出的是，终端设备101、102、103也可以直接对其所存储的目标图像进行插值、分析等处理，此时，本申请实施例所提供的用于生成图像的方法也可以由终端设备101、102、103执行，此时的示 例性系统架构100可以不包含上述网络104和服务器105。
应该理解，图1中的终端设备、网络和服务器的数目仅仅是示意性的。根据实现需要，可以具有任意数目的终端设备、网络和服务器。
继续参考图2，示出了根据本申请的用于生成图像的方法的一个实施例的流程200。该用于生成图像的方法，包括以下步骤：
步骤201，对目标图像进行插值，生成插值图像。
在本实施例中，用于生成图像的方法的执行主体(例如图1所示的服务器105)可以首先提取目标图像，其中，上述目标图像可以是各种待进行超分辨率重建的图像。例如，上述目标图像可以是人脸图像、物品图像、景观图像等等。上述目标图像可以预先存储于本地，也可以是由其他电子设备(例如图1所示的终端设备101、102、103)通过有线连接或者无线连接方式发送的。其中，上述无线连接方式可以包括但不限于3G/4G连接、WiFi连接、蓝牙连接、WiMAX连接、Zigbee连接、UWB(ultra wideband)连接、以及其他现在已知或将来开发的无线连接方式。
在提取目标图像之后，上述执行主体可以利用各种现有的图像插值方式对上述目标图像进行插值，以将目标图像放大至目标尺寸(如放大至2倍、3倍、4倍)。此处，可以采用诸如最近邻插值、双线性插值、双平方插值、双立方插值或者其他高阶插值方法等进行上述目标图像的插值。实践中，图像插值是从低分辨率图像生成高分辨率图像的过程，可以用以恢复图像中所丢失的信息。需要说明的是，上述各种图像插值方式是目前广泛研究和应用的公知技术，在此不再赘述。
此处，在进行高分辨率图像生成的过程中，首先执行对目标图像的插值的处理，可以初步地提高目标图像的分辨率。在上述插值图像的基础上，再进行后续图像处理步骤，可以提高高分辨率图像生成的效果。
步骤202，对插值图像进行超分辨率重建，生成重建图像。
在本实施例中，上述执行主体可以利用各种超分变率重建方法对上述插值图像进行超分辨率重建，生成重建图像。实践中，超分辨率 (Super-Resolution)即通过硬件或软件的方法提高原有图像的分辨率。通过低分辨率的图像来得到一幅高分辨率的图像过程就是超分辨率重建。
在本实施例的一些可选的实现方式中，上述执行主体可以采用深度学习的方法，对插值图像进行超分辨率重建。作为示例，上述执行主体可以将上述插值图像输入至预先训练的图像处理模型，得到上述图像处理模型输出的重建图像。其中，上述图像处理模型可以用于进行图像的超分辨率重建。此处，上述图像处理模型可以通过如下步骤训练得到：第一步，提取多组训练样本。其中，上述每一组训练样本可以包括一个高分辨率图像和经过对该高分辨率图像处理后的低分辨率图像。第二步，利用机器学习方法，将每一组训练样本中的低分辨率图像作为输入，将该组训练样本中的高分辨率图像作为输出，训练得到图像处理模型。此处，可以使用各种现有的模型结构进行图像处理模型的训练。作为示例，可以采用SRCNN(Super-Resolution Convolutional Neural Network，超分辨率卷积神经网络)。其中，SRCNN可以包含三个卷积层，可以采用MSE(Mean Square Error，均方误差)函数作为损失函数。需要说明的是，利用机器学习方法进行模型训练的具体操作方法是目前广泛研究和应用的公知技术，在此不再赘述。
在本实施例的一些可选的实现方式中，上述执行主体可以使用现有的超分辨率重建工具(例如，图像锐化工具RAISR(Rapid and Accurate Image Super Resolution))对插值图像进行超分辨率重建。
步骤203，基于目标图像中的像素的像素值，对重建图像进行像素补偿，生成高分辨率图像。
在本实施例中，上述执行主体可以基于目标图像中的像素的像素值，对重建图像进行像素补偿，生成高分辨率图像。此处，可以利用各种像素补偿方式对重建图像中的像素进行补偿。
在本实施例的一些可选的实现方式中，上述执行主体可以按照如下步骤对重建图像进行像素补偿：
第一步，将像素值划分为连续的多个取值范围。实践中，像素值量化后通常用一个字节来表示。如把有黑-灰-白连续变化的灰度值量 化为256个灰度级，灰度值的范围为0至255。因此，通常利用0至255这256个数值来标识像素的像素值。此处，可以将0至255这256个像素值划分为连续的多个取值范围。例如，可以划分为32个取值范围。其中，0至7这8个像素值为第一取值范围；8至15这8个像素值为第二取值范围；以此类推。
第二步，对于上述多个取值范围中的取值范围，分别确定上述目标图像、上述重建图像中在该取值范围内的像素值的均值。继续上例进行描述，对于0至7这8个像素值对应的第一取值范围，上述执行主体可以首先确定上述目标图像(插值前的原始的目标图像)中像素值在该第一取值范围内的像素的像素值的均值，并确定上述重建图像中像素值在该第一取值范围内的像素的像素值的均值。之后，对于8至15这8个像素值对应的第二取值范围，上述执行主体可以确定上述目标图像中像素值在该第二取值范围内的像素的像素值的均值，并确定上述重建图像中像素值在该第二取值范围内的像素的像素值的均值。以此类推，直至对于32个取值范围均处理完毕。
第三步，基于所确定的均值，对上述重建图像进行像素补偿。此处，上述执行主体基于所确定的均值，可以利用各种方式对上述重建图像进行像素补偿。作为示例，对于上述多个取值范围中的取值范围，上述执行主体可以确定上述目标图像、上述重建图像中在该取值范围内的像素值的均值是否相同。响应于确定相同，则不进行上述重建图像中在该取值范围内的像素值的补偿。响应于确定不相同，可以对重建图像中在该取值范围内的像素值进行补偿，使补偿后的均值等于上述目标图像在该取值范围内的像素值的均值。例如，对于0至7这8个像素值对应的第一取值范围，上述目标图像中在该第一取值范围内的像素值的均为5，上述重建图像中在该第一取值范围内的像素值的均值为4，则可以将上述目标图像中在该第一取值范围内的各个像素值增加1作为补偿值。
在本实施例的一些可选的实现方式中，上述第三步中，基于所确定的均值，对上述重建图像进行像素补偿，还可以通过如下步骤执行：
第一步，对于上述多个取值范围中的取值范围，上述执行主体可 以确定上述目标图像、上述重建图像中在该取值范围内的像素值的均值是否相同。响应于确定相同，则不进行上述重建图像中在该取值范围内的像素值的补偿。响应于确定不相同，可以将该取值范围确定为候选取值范围。
第二步，从所确定的候选取值范围中确定目标取值范围，对上述重建图像中在上述目标取值范围内的像素值进行补偿，以使上述重建图像中在上述目标取值范围内的像素值的均值在补偿后与上述目标图像中在上述目标取值范围内的像素值的均值相等。此处，可以通过各种预设条件从所确定的候选取值范围中筛选出目标取值范围。
作为示例，对于某一候选取值范围，若上述目标图像在该候选取值范围内的像素值的均值与上述重建图像中在该候选取值范围内的像素值的均值的差大于预设数值(例如2)，可以将该候选取值范围作为目标取值范围。
作为又一示例，首先确定是否存在连续的候选取值范围且上述连续的候选取值范围的数量不小于预设数值(例如4)。若是，可以将上述连续的候选取值范围中的候选取值范围确定为目标取值范围。作为示例，0至7这8个像素值对应的第一取值范围、8至15这8个像素值对应的第二取值范围、16至23这8个像素值对应的第三取值范围、24至31这8个像素值对应的第四取值范围均为候选取值范围，且这四个候选取值范围为连续的四个候选取值范围。因此，可以将这四个候选取值范围作为目标取值范围。
在本实施例的一些可选的实现方式中，上述执行主体可以按照如下步骤对重建图像进行像素补偿：
第一步，对于上述目标图像、上述重建图像中的像素，基于该像素与相邻像素的像素值比较，确定该像素的类别。
此处，对于某一个像素而言，可以按照四种不同的方式确定相邻像素。第一种方式，可以该像素为中心，将与该像素相邻的左右两个像素确定为该像素的相邻像素。此时，可以将该像素的像素值记为b；将左边像素的像素值记为a；将右边像素的像素值记为c。第二种方式，可以该像素为中心，将与该像素相邻的上下两个像素确定为该像素的 相邻像素。此时，可以将该像素的像素值记为b；将上边像素的像素值记为a；将下边像素的像素值记为c。第三种方式，可以以该像素为中心，将与该像素相邻的左上方像素和右下方像素确定为该像素的相邻像素。此时，可以将该像素的像素值记为b；将左上方像素的像素值记为a；将右下方像素的像素值记为c。第四种方式，可以以该像素为中心，将与该像素相邻的右上方像素和左下方像素确定为该像素的相邻像素。此时，可以将该像素的像素值记为b；将右上方像素的像素值记为a；将左下方像素的像素值记为c。
此处，上述执行主体可以选取上述任一种方式，对于上述目标图像、上述重建图像中每一个像素(像素值为b)，利用所选取的方式确定该像素的相邻像素(像素值为a和c)。而后，基于该像素与相邻像素的像素值比较，确定该像素的类别。具体地，可以将像素值满足b<a＝c的像素作为第一类别；将像素值满足b＝c<a的像素作为第二类别；将像素值满足b＝a<c的像素作为第三类别；将像素值满足b＝a>c的像素作为第四类别；将像素值满足b＝c>a的像素作为第五类别；将像素值满足b>a＝c的像素作为第六类别。
第二步，对于每一类别，分别确定上述目标图像、上述重建图像中属于该类别的像素的像素值的均值。响应于确定相同，则不进行上述重建图像中的该类别的像素值的补偿。响应于确定均值不同，可以将上述重建图像中属于该类别的像素的像素值进行补偿，以使上述重建图像中属于该类别的像素的像素值的均值在补偿后与上述目标图像中属于该类别的像素的像素值的均值相等。
在本实施例的一些可选的实现方式中，上述执行主体可以按照如下步骤对重建图像进行像素补偿：首先，可以提取原始的目标图像中的各像素的像素值，确定目标图像的平均像素值。而后，提取重建图像中的各像素的像素值，确定重建图像的平均像素值。之后，确定目标图像的平均像素值与重建图像的平均像素值的平均像素值是否相同。若不同，则对重建图像的像素进行补偿，以使重建图像的补偿后的平均像素值与目标图像的平均像素值相等。
继续参见图3，图3是根据本实施例的用于生成图像的方法的应用场景的一个示意图。在图3的应用场景中，用户首先使用终端设备向图像处理服务器发送了一个图像处理请求，该图像处理请求中包含待进行超分辨率图像重建的目标图像301。图像处理服务器接收到该目标图像301后，首先对该目标图像301进行插值，得到插值图像。而后，对上述插值图像进行超分辨率重建，生成重建图像。最后，对上述重建图像进行像素补偿，生成高分辨率图像302。
本申请的上述实施例提供的方法，通过对目标图像进行插值，以便生成插值图像，而后对上述插值图像进行超分辨率重建，生成重建图像，最后基于上述目标图像中的像素的像素值，对上述重建图像进行像素补偿，生成高分辨率图像。从而，在进行高分辨率图像生成的过程中，首先执行对目标图像的插值的处理，可以初步地提高目标图像的分辨率。在上述插值图像的基础上，再进行后续图像处理步骤，可以提高高分辨率图像生成的效果。同时，在进行超分辨率重建的基础上，对重建图像进行补偿，提高了高分辨率图像生成的效果。
进一步参考图4，其示出了用于生成图像的方法的又一个实施例的流程400。该用于生成图像的方法的流程400，包括以下步骤：
步骤401，对目标图像进行插值，生成插值图像。
在本实施例中，用于生成图像的方法的执行主体(例如图1所示的服务器105)可以首先提取目标图像，其中，上述目标图像可以是各种待进行超分辨率重建的图像。在提取目标图像之后，上述执行主体可以利用各种现有的图像插值方式对上述目标图像进行插值，以将目标图像放大至目标尺寸(如放大至2倍、3倍、4倍)。此处，可以采用诸如最近邻插值、双线性插值、双平方插值、双立方插值或者其他高阶插值方法等进行上述目标图像的插值。
此处，在进行高分辨率图像生成的过程中，首先执行对目标图像的插值的处理，可以初步地提高目标图像的分辨率。在上述插值图像的基础上，再进行后续图像处理步骤，可以提高高分辨率图像生成的效果。
步骤402，对于插值图像中的像素，提取以该像素为中心的第一像素矩阵，对第一像素矩阵进行主成分分析，得到目标矩阵。
在本实施例中，对于插值图像中的像素，上述执行主体可以首先提取以该像素为中心的第一像素矩阵。其中，上述第一像素矩阵中可以包含以该像素为中心的正方形区域(例如3×3的图像块(patch))内的像素的像素值。而后，上述执行主体可以对上述第一像素矩阵进行主成分分析(Principal Components Analysis，PCA)，得到目标矩阵。具体地，可以首先确定第一像素矩阵的协方差矩阵。之后，可以确定协方差矩阵的特征值和特征向量。最后，可以将第一像素矩阵投影至特征向量构成的空间中，将投影后所得到的矩阵确定为目标矩阵。实践中，主成分分析也称主分量分析，旨在利用降维的思想，把多指标转化为少数几个综合指标。在统计学中，主成分分析是一种简化数据集的技术。它是一个线性变换，把数据变换到一个新的坐标系统中。主成分分析可以用于减少数据集的维数，同时保持数据集的对方差贡献最大的特征。这是通过保留低阶主成分，忽略高阶主成分做到的。这样低阶成分往往能够保留住数据的最重要方面。由此，利用主成分分析的方式对第一像素矩阵进行处理，可以保留各个像素的第一像素矩阵中的重要的特征，从而使不同的第一像素矩阵的差异更为明显，因而可以实现更加准确地对上述插值图像内像素的分类。
在本实施例的一些可选的实现方式中，对于上述插值图像中的每一个像素，以该像素为中心的第一像素矩阵可以是以该像素为中心的正方形区域(例如3×3的图像块(patch))相对应的矩阵。第一像素矩阵中的数值可以与上述正方形区域中的像素一一对应，即第一像素矩阵的第i行第j列的数值为上述正方形区域中第i行第j列像素的像素值。其中，上述i为不小于1且不大于第一像素矩阵的行数的整数，上述j为不小于1且不大于第一像素矩阵的列数的整数。需要说明的是，对于某一些像素(例如位于图像边缘的像素)，该像素的第一像素矩阵中的某些位置不存在与之对应的像素值(例如，位于图像上边缘的像素，以该像素为中心的第一像素矩阵的第一行没有与之相对应的像素值)，此时，可以将这些位置的数值设置为预设值(例如0)。
在本实施例的一些可选的实现方式中，对于上述插值图像中的每一个像素，以该像素为中心的第一像素矩阵可以通过如下方式得到：第一步，提取以该像素为中心的正方形区域中的像素的像素值，生成矩阵中的数值与该正方形区域中的像素一一对应的像素矩阵。需要说明的是，对于某一些像素(例如位于图像边缘的像素)，该像素的像素矩阵中的某些位置不存在与之对应的像素值(例如，位于图像上边缘的像素，该像素的像素矩阵的第一行没有与之相对应的像素值)，此时，可以将这些位置的数值设置为预设值(例如0)。第二步，将该像素矩阵转换为行向量。由于行向量是矩阵的一种特殊的形式，因此，可以将该向量确定为以该像素为中心的第一像素矩阵。作为示例，对于上述插值图像中的某一个像素，可以提取以该像素为中心的3×3的patch的像素，生成3×3(3行3列)的像素矩阵。而后，可以将该3×3的像素矩阵转换成行向量，将该行向量确定为1×9(1行9列)的第一像素矩阵。
在本实施例的一些可选的实现方式中，对于上述插值图像中的每一个像素，可以利用如下步骤对该像素的第一像素矩阵进行主成分分析：
第一步，确定第一像素矩阵的协方差矩阵。作为示例，根据协方差矩阵的计算方法，若第一像素矩阵为1×9(1行9列)的矩阵，则该矩阵的协方差矩阵为9×9的矩阵。
第二步，确定协方差矩阵的特征值和特征向量。其中，每一个特征值可以对应一个特征向量。此处，协方差矩阵的计算方法、矩阵的特征值的计算方法、矩阵的特征向量的计算方法是目前数学领域广泛研究和应用的公知技术，在此不再赘述。
第三步，从所确定的特征值中选取目标特征值，将目标特征值对应的特征向量组成特征矩阵。此处，上述执行主体可以利用各种选取方式从所确定的特征值中选取目标特征值。例如，可以按照特征值从大到小的顺序，从特征值对应的特征向量中，依次选取预设数量的特征向量作为目标特征向量，并依次将目标特征向量进行组合，将所得到的矩阵的转置确定为特征矩阵。作为示例，协方差矩阵为9×9的矩 阵。上述执行主体确定出该矩阵存在9个特征值和9个对应的特征向量后，可以按照特征值从大到小的顺序对特征向量进行排序。而后，可以选取前8个特征向量进行组合，得到8×9(8行9列)的矩阵。之后，可以对该矩阵的转置确定为特征矩阵，该特征矩阵为9×8(9行8列)的矩阵。
第四步，将第一像素矩阵与特征矩阵相乘，得到目标矩阵。
作为示例，第一像素矩阵为1×9(1行9列)的矩阵，特征矩阵为9×8(9行8列)的矩阵，两矩阵相乘后，得到1×8(1行8列)的目标矩阵。
由此，利用主成分分析的方式对第一像素矩阵进行处理，对第一像素矩阵进行降维，可以保留各个像素的第一像素矩阵中的重要的特征，从而使不同的第一像素矩阵的差异更为明显，因而可以实现更加准确地对插值图像内像素的分类。
步骤403，对于插值图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对上述第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值。
在本实施例中，对于上述插值图像中的像素，上述执行主体可以基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器。此处，上述执行主体中可以预先存储有滤波器集合。上述滤波器集合中的每一个滤波器可以与像素的一个类别相对应。上述执行主体可以通过对各个像素的目标矩阵进行分析或计算，根据分析结果或者计算结果，确定各个像素的类别，进而针对每一个像素进行相应的滤波器的选取。作为示例，上述执行主体可以将上述目标矩阵代入预设的公式或者函数进行计算，得到计算结果(例如一个数值)。不同的计算结果可以对应有不同的滤波器，上述执行主体可以基于所得到的计算结果选取对应的滤波器。需要说明的是，滤波器集合中的每一个滤波器可以是一个参数矩阵或者参数向量。利用滤波器对某一个像素的目标矩阵进行卷积计算，可以得到该像素的高分辨率像素值。
对于每一个像素，在选取滤波器之后，上述执行主体可以提取以 该像素为中心的第二像素矩阵。其中，上述第二像素矩阵中可以包含以该像素为中心的正方形区域(例如7×7的图像块(patch))内的像素的像素值。作为示例，上述第二像素矩阵可以是1×49的行向量。之后，可以利用选取的滤波器对上述第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值。需要说明的是，对于某一个像素，该像素的第二像素矩阵的尺寸可以与第一像素矩阵的尺寸相同或不同，此处不作限定。
由此，在主成分分析的基础上确定像素的类别，选取与该类别对应的滤波器，进行高分辨率像素值的计算，从而可以实现对上述插值图像内各像素更加精确的分类。
在本实施例的一些可选的实现方式中，对于上述插值图像中的每一个像素，上述执行主体可以将该像素对应的目标矩阵与预设矩阵进行点乘运算，将点乘运算结果作为该像素的类别。而后，从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。此处需要说明的是，可以预先基于大量图像样本的分析处理而预先确定像素的类别总数和每一个类别的点乘运算结果，并预先生成与每一个类别相对应的滤波器。上述执行主体可以存储或提取各个滤波器与点乘运算结果的对应关系。
在本实施例的一些可选的实现方式中，对于步骤403中上述的滤波器集合，其生成步骤可以参照图5。图5给出了根据本申请的用于生成滤波器集合的方法的一个实施例的流程图。该用于生成滤波器集合的方法500，包括以下步骤：
步骤501，提取高分辨率图像样本集合，对高分辨率图像样本集合中的高分辨率图像样本依次进行下采样和插值。
此处，对于上述高分辨率图像样本集合中的每一个高分辨率样本，均可以先对其进行下采样，之后对下采样后的高分辨率样本插值。
此处，下采样的倍数可以预先设定。作为示例，对高分辨率图像样本进行2倍下采样，可以是把高分辨率图像样本中2×2的图像块内的图像转换为一个像素，该像素的像素值等于2×2的图像块中的所有像素的像素值的均值。此处，可以采用与步骤401中相同的插值方式 进行插值，此处不再赘述。
步骤502，对于插值后的高分辨率图像样本中的像素，提取以该像素为中心的第三像素矩阵，对第三像素矩阵进行主成分分析，得到目标矩阵样本。
此处，对于插值后的高分辨率图像样本中的每一个像素，可以提取以该像素为中心的第三像素矩阵，对上述第三像素矩阵进行主成分分析，得到目标矩阵样本。需要说明的是，对插值后的高分辨率图像样本中的像素进行第三像素矩阵提取步骤与对上述插值图像进行第一像素矩阵的提取步骤基本相同；对第三像素矩阵进行主成分分析得到目标矩阵样本的步骤与对第一像素矩阵进行主成分分析的操作基本相同，此处不再赘述。
在一些可选的实现方式中，第一像素矩阵的行数、列数可以分别与第三像素矩阵的行数、列数相同。作为示例，第一像素矩阵、第三像素矩阵均可以为1×9的矩阵。
步骤503，对所得到的目标矩阵样本进行分类。
此处，可以将所得到的各个目标矩阵样本代入到预设的公式或者函数进行计算，得到计算结果(例如数值)，将相同计算结果对应的目标矩阵样本划分为同一类。每一个类别可以用一个计算结果来表征。
在一些可选的实现方式中，可以首先将所得到的各个目标矩阵样本与预设矩阵进行点乘运算。之后，将点乘运算结果相同的目标矩阵样本划分为一类。
步骤504，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合。
此处，对于每一类目标矩阵样本，可以利用机器学习方法进行该类对应的滤波器的训练。
在一些可选的实现方式中，对于每一类目标矩阵样本对应的、插值后的高分辨率图像样本中的像素，可以首先提取以该像素为中心的第四像素矩阵，将上述第四像素矩阵作为输入，将该像素对应的高分辨率像素作为输出，利用机器学习方法训练得到与该类目标矩阵样本对应的滤波器。此处，第四像素矩阵的提取步骤与上述中提取第二像 素矩阵的步骤基本相同，此处不再赘述。
在一些可选的实现方式中，第二像素矩阵的行数、列数可以分别与第四像素矩阵的行数、列数相同。作为示例，第二像素矩阵、第四像素矩阵均可以为1×49的矩阵。
在一些可选的实现方式中，还可以建立点乘运算结果与滤波器的对应关系。作为示例，可以以点乘运算结果为键值对的键，以用于表征滤波器的参数向量或者参数矩阵为键值对的值，利用键值对形式表征点乘运算结果与滤波器的对应关系。
由此，通过主成分分析技术对像素进行降维分类，可以保留各个像素的重要的特征，从而使不同的像素的差异更为明显，因而可以使像素分类更加准确。在此基础上进行滤波器训练，可以提高滤波器的针对性和准确性。
回到用于生成图像的方法的流程400，在步骤403得到插值图像中的各像素的高分辨率像素值后，继续参见如下步骤：
步骤404，将所得到的高分辨率像素值进行汇总，生成重建图像。
在本实施例中，由于根据插值图像中的每一个像素均可以计算得到一个对应的高分辨率像素值，因此，上述执行主体可以将所得到的各个像素对应的高分辨率像素值进行汇总，生成重建图像。
步骤405，将像素值划分为连续的多个取值范围。
在本实施例中，可以将像素值划分为连续的多个取值范围像。实践中，素值量化后通常用一个字节来表示。如把有黑-灰-白连续变化的灰度值量化为256个灰度级，灰度值的范围为0至255。因此，通常利用0至255这256个数值来标识像素的像素值。此处，可以将0至255这256个像素值划分为连续的多个取值范围。例如，可以划分为32个取值范围。其中，0至7这8个像素值为第一取值范围；8至15这8个像素值为第二取值范围；以此类推。
步骤406，对于上述多个取值范围中的取值范围，分别确定目标图像、上述重建图像中在该取值范围内的像素值的均值。
在本实施例中，上述执行主体对于上述多个取值范围中的取值范围，分别确定上述目标图像、上述重建图像中在该取值范围内的像素 值的均值。继续上例进行描述，对于0至7这8个像素值对应的第一取值范围，上述执行主体可以首先确定上述目标图像(插值前的原始的目标图像)中像素值在该第一取值范围内的像素的像素值的均值，并确定上述重建图像中像素值在该第一取值范围内的像素的像素值的均值。之后，对于8至15这8个像素值对应的第二取值范围，上述执行主体可以确定上述目标图像中像素值在该第二取值范围内的像素的像素值的均值，并确定上述重建图像中像素值在该第二取值范围内的像素的像素值的均值。以此类推，直至对于32个取值范围均处理完毕。
步骤407，对于多个取值范围中的取值范围，响应于确定目标图像、重建图像中在该取值范围内的像素值的均值不相同，将该取值范围确定为候选取值范围。
在本实施例中，对于上述多个取值范围中的取值范围，上述执行主体可以确定上述目标图像、上述重建图像中在该取值范围内的像素值的均值是否相同。响应于确定相同，则不进行上述重建图像中在该取值范围内的像素值的补偿。响应于确定不相同，可以将该取值范围确定为候选取值范围。
步骤408，从所确定的候选取值范围中确定目标取值范围，对重建图像中在目标取值范围内的像素值进行补偿，以使重建图像中在目标取值范围内的像素值的均值在补偿后与目标图像中在目标取值范围内的像素值的均值相等，生成高分辨率图像。
在本实施例中，上述执行主体可以从所确定的候选取值范围中确定目标取值范围，对重建图像中在目标取值范围内的像素值进行补偿，以使重建图像中在目标取值范围内的像素值的均值在补偿后与目标图像中在目标取值范围内的像素值的均值相等。作为示例，首先确定是否存在连续的候选取值范围且上述连续的候选取值范围的数量不小于预设数值(例如4)。若是，可以将上述连续的候选取值范围中的候选取值范围确定为目标取值范围。作为示例，0至7这8个像素值对应的第一取值范围、8至15这8个像素值对应的第二取值范围、16至23这8个像素值对应的第三取值范围、24至31这8个像素值对应的第四取值范围均为候选取值范围，且这四个候选取值范围为连续的四 个候选取值范围。因此，可以将这四个候选取值范围作为目标取值范围。
从图4中可以看出，与图2对应的实施例相比，本实施例中的用于生成图像的方法的流程400突出了利用主成分分析方法确定目标矩阵、并选取滤波器以生成重建图像的步骤，同时突出了滤波器集合的生成步骤。由此，本实施例描述的方案，通过主成分分析技术对像素进行降维分类，可以保留各个像素的重要的特征，从而使不同的像素的差异更为明显，因而可以使像素分类更加准确。在此基础上进行滤波器训练，可以提高滤波器的针对性和准确性。利用所训练的滤波器进行重建图像的生成，进一步提高了重建图像生成的效果。在以上基础上，对重建图像进行像素补偿，更进一步提高了高分辨率图像生成的效果。
进一步参考图6，作为对上述各图所示方法的实现，本申请提供了一种用于生成图像的装置的一个实施例，该装置实施例与图2所示的方法实施例相对应，该装置具体可以应用于各种电子设备中。
如图6所示，本实施例所述的用于生成图像的装置600包括：插值单元601，被配置成对目标图像进行插值，生成插值图像；重建单元602，被配置成对上述插值图像进行超分辨率重建，生成重建图像；补偿单元603，被配置成基于上述目标图像中的像素的像素值，对上述重建图像进行像素补偿，生成高分辨率图像。
在本实施例的一些可选的实现方式中，上述补偿单元603可以包括划分模块、第一确定模块和第一补偿模块(图中未示出)。其中，上述划分模块可以被配置成将像素值划分为连续的多个取值范围。上述第一确定模块可以被配置成对于上述多个取值范围中的取值范围，分别确定上述目标图像、上述重建图像中在该取值范围内的像素值的均值。上述第一补偿模块可以被配置成基于所确定的均值，对上述重建图像进行像素补偿。
在本实施例的一些可选的实现方式中，上述第一补偿模块可以包括第一确定子模块和补偿子模块(图中未示出)。其中，上述第一确定 子模块可以被配置成对于上述多个取值范围中的取值范围，响应于确定上述目标图像、上述重建图像中在该取值范围内的像素值的均值不相同，将该取值范围确定为候选取值范围。上述补偿子模块可以被配置成从所确定的候选取值范围中确定目标取值范围，对上述重建图像中在上述目标取值范围内的像素值进行补偿，以使上述重建图像中在上述目标取值范围内的像素值的均值在补偿后与上述目标图像中在上述目标取值范围内的像素值的均值相等。
在本实施例的一些可选的实现方式中，上述补偿子模块可以进一步被配置成确定是否存在连续的候选取值范围且上述连续的候选取值范围的数量不小于预设数值；若是，将上述连续的候选取值范围中的候选取值范围确定为目标取值范围。
在本实施例的一些可选的实现方式中，上述补偿单元603可以包括第二确定模块和第二补偿模块(图中未示出)。其中，上述第二确定模块可以被配置成对于上述目标图像、上述重建图像中的像素，基于该像素与相邻像素的像素值比较，确定该像素的类别。上述第二补偿模块被配置成对于每一类别，分别确定上述目标图像、上述重建图像中属于该类别的像素的像素值的均值，响应于确定均值不同，将上述重建图像中属于该类别的像素的像素值进行补偿，以使重建图像中属于该类别的像素的像素值的均值在补偿后与上述目标图像中属于该类别的像素的像素值的均值相等。
在本实施例的一些可选的实现方式中，上述重建单元602可以包括分析模块、选取模块和生成模块(图中未示出)。其中，上述分析模块可以被配置成对于上述插值图像中的像素，提取以该像素为中心的第一像素矩阵，对上述第一像素矩阵进行主成分分析，得到目标矩阵。上述选取模块可以被配置成对于上述插值图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对上述第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值。上述生成模块可以被配置成将所得到的高分辨率像素值进行汇总，生成重建图像。
在本实施例的一些可选的实现方式中，上述滤波器集合可以通过 如下步骤生成：提取高分辨率图像样本集合，对上述高分辨率图像样本集合中的高分辨率图像样本依次进行下采样和插值；对于插值后的高分辨率图像样本中的像素，提取以该像素为中心的第三像素矩阵，对上述第三像素矩阵进行主成分分析，得到目标矩阵样本；对所得到的目标矩阵样本进行分类，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合。
在本实施例的一些可选的实现方式中，上述滤波器集合的生成步骤中的对所得到的目标矩阵样本进行分类，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合，可以包括：将所得到的目标矩阵样本与预设矩阵进行点乘运算，将点乘运算结果相同的目标矩阵样本划分为一类；对于每一类目标矩阵样本对应的、插值后的高分辨率图像样本中的像素，提取以该像素为中心的第四像素矩阵，将上述第四像素矩阵作为输入，将该像素对应的高分辨率像素作为输出，训练得到与该类目标矩阵样本对应的滤波器。
在本实施例的一些可选的实现方式中，上述分析模块可以包括第二确定子模块、第三确定子模块、组成子模块和组成子模块(图中未示出)。其中，上述第二确定子模块可以被配置成确定上述第一像素矩阵的协方差矩阵。上述第三确定子模块可以被配置成确定上述协方差矩阵的特征值和特征向量。上述组成子模块可以被配置成从所确定的特征值中选取目标特征值，将上述目标特征值对应的特征向量组成特征矩阵。上述相乘子模块可以被配置成将上述第一像素矩阵与上述特征矩阵相乘，得到目标矩阵。
在本实施例的一些可选的实现方式中，上述选取模块可以进一步配配置成对于上述插值图像中的像素，将该像素对应的目标矩阵与预设矩阵进行点乘运算，从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。
本申请的上述实施例提供的装置，通过插值单元601对目标图像进行插值，以便生成插值图像，而后重建单元602对上述插值图像进行超分辨率重建，生成重建图像，最后补偿单元603基于上述目标图像中的像素的像素值，对上述重建图像进行像素补偿，生成高分辨率 图像，从而在进行超分辨率重建的基础上，对重建图像进行补偿，提高了高分辨率图像生成的效果。
下面参考图7，其示出了适于用来实现本申请实施例的电子设备的计算机系统700的结构示意图。图7示出的电子设备仅仅是一个示例，不应对本申请实施例的功能和使用范围带来任何限制。
如图7所示，计算机系统700包括中央处理单元(CPU)701，其可以根据存储在只读存储器(ROM)702中的程序或者从存储部分708加载到随机访问存储器(RAM)703中的程序而执行各种适当的动作和处理。在RAM 703中，还存储有系统700操作所需的各种程序和数据。CPU 701、ROM 702以及RAM 703通过总线704彼此相连。输入/输出(I/O)接口705也连接至总线704。
以下部件连接至I/O接口705：包括键盘、鼠标等的输入部分706；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分707；包括硬盘等的存储部分708；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分709。通信部分709经由诸如因特网的网络执行通信处理。驱动器710也根据需要连接至I/O接口705。可拆卸介质711，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器710上，以便于从其上读出的计算机程序根据需要被安装入存储部分708。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信部分709从网络上被下载和安装，和/或从可拆卸介质711被安装。在该计算机程序被中央处理单元(CPU)701执行时，执行本申请的方法中限定的上述功能。需要说明的是，本申请所述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、 装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本申请中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本申请中，计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、RF等等，或者上述的任意合适的组合。
附图中的流程图和框图，图示了按照本申请各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本申请实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。所描述的单元也可以设置在处理 器中，例如，可以描述为：一种处理器包括插值单元、重建单元和补偿单元。其中，这些单元的名称在某种情况下并不构成对该单元本身的限定，例如，插值单元还可以被描述为“对目标图像进行插值，生成插值图像的单元”。
作为另一方面，本申请还提供了一种计算机可读介质，该计算机可读介质可以是上述实施例中描述的装置中所包含的；也可以是单独存在，而未装配入该装置中。上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该装置执行时，使得该装置：对目标图像进行插值，生成插值图像；对该插值图像进行超分辨率重建，生成重建图像；基于该目标图像中的像素的像素值，对该重建图像进行像素补偿，生成高分辨率图像。
以上描述仅为本申请的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本申请中所涉及的发明范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述发明构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本申请中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。

Claims (22)

  1. 一种用于生成图像的方法，包括：
    对目标图像进行插值，生成插值图像；
    对所述插值图像进行超分辨率重建，生成重建图像；
    基于所述目标图像中的像素的像素值，对所述重建图像进行像素补偿，生成高分辨率图像。
  2. 根据权利要求1所述的用于生成图像的方法，其中，所述基于所述目标图像中的像素的像素值，对所述重建图像进行像素补偿，包括：
    将像素值划分为连续的多个取值范围；
    对于所述多个取值范围中的取值范围，分别确定所述目标图像、所述重建图像中在该取值范围内的像素值的均值；
    基于所确定的均值，对所述重建图像进行像素补偿。
  3. 根据权利要求2所述的用于生成图像的方法，其中，所述基于所确定的均值，对所述重建图像进行像素补偿，包括：
    对于所述多个取值范围中的取值范围，响应于确定所述目标图像、所述重建图像中在该取值范围内的像素值的均值不相同，将该取值范围确定为候选取值范围；
    从所确定的候选取值范围中确定目标取值范围，对所述重建图像中在所述目标取值范围内的像素值进行补偿，以使所述重建图像中在所述目标取值范围内的像素值的均值在补偿后与所述目标图像中在所述目标取值范围内的像素值的均值相等。
  4. 根据权利要求3所述的用于生成图像的方法，其中，所述从所确定的候选取值范围中选取目标取值范围，包括：
    确定是否存在连续的候选取值范围且所述连续的候选取值范围的数量不小于预设数值；
    若是，将所述连续的候选取值范围中的候选取值范围确定为目标取值范围。
  5. 根据权利要求1所述的用于生成图像的方法，其中，所述基于所述目标图像中的像素的像素值，对所述重建图像进行像素补偿，包括：
    对于所述目标图像、所述重建图像中的像素，基于该像素与相邻像素的像素值比较，确定该像素的类别；
    对于每一类别，分别确定所述目标图像、所述重建图像中属于该类别的像素的像素值的均值，响应于确定均值不同，将所述重建图像中属于该类别的像素的像素值进行补偿，以使所述重建图像中属于该类别的像素的像素值的均值在补偿后与所述目标图像中属于该类别的像素的像素值的均值相等。
  6. 根据权利要求1所述的用于生成图像的方法，其中，所述对所述插值图像进行超分辨率重建，生成重建图像，包括：
    对于所述插值图像中的像素，提取以该像素为中心的第一像素矩阵，对所述第一像素矩阵进行主成分分析，得到目标矩阵；
    对于所述插值图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对所述第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值；
    将所得到的高分辨率像素值进行汇总，生成重建图像。
  7. 根据权利要求6所述的用于生成图像的方法，其中，所述滤波器集合通过如下步骤生成：
    提取高分辨率图像样本集合，对所述高分辨率图像样本集合中的高分辨率图像样本依次进行下采样和插值；
    对于插值后的高分辨率图像样本中的像素，提取以该像素为中心的第三像素矩阵，对所述第三像素矩阵进行主成分分析，得到目标矩 阵样本；
    对所得到的目标矩阵样本进行分类，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合。
  8. 根据权利要求7所述的用于生成图像的方法，其中，所述对所得到的目标矩阵样本进行分类，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合，包括：
    将所得到的目标矩阵样本与预设矩阵进行点乘运算，将点乘运算结果相同的目标矩阵样本划分为一类；
    对于每一类目标矩阵样本对应的、插值后的高分辨率图像样本中的像素，提取以该像素为中心的第四像素矩阵，将所述第四像素矩阵作为输入，将该像素对应的高分辨率像素作为输出，训练得到与该类目标矩阵样本对应的滤波器。
  9. 根据权利要求6所述的用于生成图像的方法，其中，所述对所述第一像素矩阵进行主成分分析，得到目标矩阵，包括：
    确定所述第一像素矩阵的协方差矩阵；
    确定所述协方差矩阵的特征值和特征向量；
    从所确定的特征值中选取目标特征值，将所述目标特征值对应的特征向量组成特征矩阵；
    将所述第一像素矩阵与所述特征矩阵相乘，得到目标矩阵。
  10. 根据权利要求6所述的用于生成图像的方法，其中，所述对于所述插值图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，包括：
    对于所述插值图像中的像素，将该像素对应的目标矩阵与预设矩阵进行点乘运算，从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。
  11. 一种用于生成图像的装置，包括：
    插值单元，被配置成对目标图像进行插值，生成插值图像；
    重建单元，被配置成对所述插值图像进行超分辨率重建，生成重建图像；
    补偿单元，被配置成基于所述目标图像中的像素的像素值，对所述重建图像进行像素补偿，生成高分辨率图像。
  12. 根据权利要求11所述的用于生成图像的装置，其中，所述补偿单元包括：
    划分模块，被配置成将像素值划分为连续的多个取值范围；
    第一确定模块，被配置成对于所述多个取值范围中的取值范围，分别确定所述目标图像、所述重建图像中在该取值范围内的像素值的均值；
    第一补偿模块，被配置成基于所确定的均值，对所述重建图像进行像素补偿。
  13. 根据权利要求12所述的用于生成图像的装置，其中，所述第一补偿模块，包括：
    第一确定子模块，被配置成对于所述多个取值范围中的取值范围，响应于确定所述目标图像、所述重建图像中在该取值范围内的像素值的均值不相同，将该取值范围确定为候选取值范围；
    补偿子模块，被配置成从所确定的候选取值范围中确定目标取值范围，对所述重建图像中在所述目标取值范围内的像素值进行补偿，以使所述重建图像中在所述目标取值范围内的像素值的均值在补偿后与所述目标图像中在所述目标取值范围内的像素值的均值相等。
  14. 根据权利要求13所述的用于生成图像的装置，其中，所述补偿子模块进一步被配置成：
    确定是否存在连续的候选取值范围且所述连续的候选取值范围的数量不小于预设数值；
    若是，将所述连续的候选取值范围中的候选取值范围确定为目标 取值范围。
  15. 根据权利要求11所述的用于生成图像的装置，其中，所述补偿单元，包括：
    第二确定模块，被配置成对于所述目标图像、所述重建图像中的像素，基于该像素与相邻像素的像素值比较，确定该像素的类别；
    第二补偿模块，被配置成对于每一类别，分别确定所述目标图像、所述重建图像中属于该类别的像素的像素值的均值，响应于确定均值不同，将所述重建图像中属于该类别的像素的像素值进行补偿，以使所述重建图像中属于该类别的像素的像素值的均值在补偿后与所述目标图像中属于该类别的像素的像素值的均值相等。
  16. 根据权利要求11所述的用于生成图像的装置，其中，所述重建单元包括：
    分析模块，被配置成对于所述插值图像中的像素，提取以该像素为中心的第一像素矩阵，对所述第一像素矩阵进行主成分分析，得到目标矩阵；
    选取模块，被配置成对于所述插值图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对所述第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值；
    生成模块，被配置成将所得到的高分辨率像素值进行汇总，生成重建图像。
  17. 根据权利要求16所述的用于生成图像的装置，其中，所述滤波器集合通过如下步骤生成：
    提取高分辨率图像样本集合，对所述高分辨率图像样本集合中的高分辨率图像样本依次进行下采样和插值；
    对于插值后的高分辨率图像样本中的像素，提取以该像素为中心的第三像素矩阵，对所述第三像素矩阵进行主成分分析，得到目标矩 阵样本；
    对所得到的目标矩阵样本进行分类，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合。
  18. 根据权利要求17所述的用于生成图像的装置，其中，所述对所得到的目标矩阵样本进行分类，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合，包括：
    将所得到的目标矩阵样本与预设矩阵进行点乘运算，将点乘运算结果相同的目标矩阵样本划分为一类；
    对于每一类目标矩阵样本对应的、插值后的高分辨率图像样本中的像素，提取以该像素为中心的第四像素矩阵，将所述第四像素矩阵作为输入，将该像素对应的高分辨率像素作为输出，训练得到与该类目标矩阵样本对应的滤波器。
  19. 根据权利要求16所述的用于生成图像的装置，其中，所述分析模块包括：
    第二确定子模块，被配置成确定所述第一像素矩阵的协方差矩阵；
    第三确定子模块，被配置成确定所述协方差矩阵的特征值和特征向量；
    组成子模块，被配置成从所确定的特征值中选取目标特征值，将所述目标特征值对应的特征向量组成特征矩阵；
    相乘子模块，被配置成将所述第一像素矩阵与所述特征矩阵相乘，得到目标矩阵。
  20. 根据权利要求16所述的用于生成图像的装置，其中，所述选取模块进一步配配置成：
    对于所述插值图像中的像素，将该像素对应的目标矩阵与预设矩阵进行点乘运算，从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。
  21. 一种电子设备，包括：
    一个或多个处理器；
    存储装置，其上存储有一个或多个程序，
    当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如权利要求1-10中任一所述的方法。
  22. 一种计算机可读介质，其上存储有计算机程序，其中，该程序被处理器执行时实现如权利要求1-10中任一所述的方法。
PCT/CN2018/116332 2018-06-26 2018-11-20 用于生成图像的方法和装置 WO2020000877A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810669838.8A CN108921801B (zh) 2018-06-26 2018-06-26 用于生成图像的方法和装置
CN201810669838.8 2018-06-26

Publications (1)

Publication Number Publication Date
WO2020000877A1 true WO2020000877A1 (zh) 2020-01-02

Family

ID=64421320

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/116332 WO2020000877A1 (zh) 2018-06-26 2018-11-20 用于生成图像的方法和装置

Country Status (2)

Country Link
CN (1) CN108921801B (zh)
WO (1) WO2020000877A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113807365A (zh) * 2021-09-15 2021-12-17 广东电网有限责任公司 一种电缆图像的特征提取方法、装置、电子设备及介质
CN114119367A (zh) * 2021-11-17 2022-03-01 西安工业大学 一种分区域的同步相移干涉图超分辨重建的插值方法

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110111258A (zh) * 2019-05-14 2019-08-09 武汉高德红外股份有限公司 基于多核处理器的红外超分辨率重建图像方法及系统
CN110503618A (zh) * 2019-08-30 2019-11-26 维沃移动通信有限公司 图像处理方法及电子设备

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101312529A (zh) * 2007-05-24 2008-11-26 华为技术有限公司 生成上下采样滤波器及实现编码的方法、系统和装置
CN102915527A (zh) * 2012-10-15 2013-02-06 中山大学 基于形态学成分分析的人脸图像超分辨率重建方法
EP3264741A1 (en) * 2016-06-30 2018-01-03 Thomson Licensing Plenoptic sub aperture view shuffling with improved resolution

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103685867B (zh) * 2012-09-12 2017-07-14 富士通株式会社 背光补偿方法和装置
CN103500445B (zh) * 2013-09-22 2016-05-04 华南理工大学 一种彩色视频的超分辨率处理方法
CN105635732B (zh) * 2014-10-30 2018-12-14 联想(北京)有限公司 自适应样点补偿编码、对视频码流进行解码的方法及装置

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101312529A (zh) * 2007-05-24 2008-11-26 华为技术有限公司 生成上下采样滤波器及实现编码的方法、系统和装置
CN102915527A (zh) * 2012-10-15 2013-02-06 中山大学 基于形态学成分分析的人脸图像超分辨率重建方法
EP3264741A1 (en) * 2016-06-30 2018-01-03 Thomson Licensing Plenoptic sub aperture view shuffling with improved resolution

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113807365A (zh) * 2021-09-15 2021-12-17 广东电网有限责任公司 一种电缆图像的特征提取方法、装置、电子设备及介质
CN114119367A (zh) * 2021-11-17 2022-03-01 西安工业大学 一种分区域的同步相移干涉图超分辨重建的插值方法
CN114119367B (zh) * 2021-11-17 2024-04-09 西安工业大学 一种分区域的同步相移干涉图超分辨重建的插值方法

Also Published As

Publication number Publication date
CN108921801A (zh) 2018-11-30
CN108921801B (zh) 2020-01-07

Similar Documents

Publication Publication Date Title
WO2020000877A1 (zh) 用于生成图像的方法和装置
US10296813B2 (en) Training method and apparatus for neural network for image recognition
CN113870104A (zh) 超分辨率图像重建
CN111028153A (zh) 图像处理和神经网络训练方法、装置及计算机设备
WO2020062494A1 (zh) 图像处理方法和装置
CN110809126A (zh) 一种基于自适应可变形卷积的视频帧插值方法及系统
WO2018225133A1 (ja) 画像処理装置、画像処理方法、および画像処理プログラム
Chen et al. MICU: Image super-resolution via multi-level information compensation and U-net
Rajput Mixed Gaussian-impulse noise robust face hallucination via noise suppressed low-and-high resolution space-based neighbor representation
CN114418853A (zh) 基于相似图像检索的图像超分辨率优化方法、介质及设备
WO2020000878A1 (zh) 用于生成图像的方法和装置
Wang et al. Deep arbitrary-scale image super-resolution via scale-equivariance pursuit
Wang et al. Image super-resolution using only low-resolution images
WO2017070841A1 (zh) 图像处理方法和装置
Fang et al. Learning explicit smoothing kernels for joint image filtering
JP5892592B2 (ja) 超解像画像処理装置及び超解像画像処理用コードブック作成装置
Wang et al. Efficient super-resolution using mobilenetv3
Rajput et al. OEINR-RFH: Outlier elimination based iterative neighbor representation for robust face hallucination
Zhou et al. Single‐frame image super‐resolution inspired by perceptual criteria
Huan et al. Remote sensing image reconstruction using an asymmetric multi-scale super-resolution network
Su et al. Single image super-resolution based on space structure learning
Liu et al. CNN-Enhanced graph attention network for hyperspectral image super-resolution using non-local self-similarity
Lee et al. GPU-based real-time super-resolution system for high-quality UHD video up-conversion
KR102624154B1 (ko) 이미지 복원 방법 및 장치
CN106780331B (zh) 一种新的基于邻域嵌入的超分辨率方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18923782

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 03.05.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 18923782

Country of ref document: EP

Kind code of ref document: A1