WO2020000878A1 - 用于生成图像的方法和装置 - Google Patents

用于生成图像的方法和装置 Download PDF

Info

Publication number
WO2020000878A1
WO2020000878A1 PCT/CN2018/116333 CN2018116333W WO2020000878A1 WO 2020000878 A1 WO2020000878 A1 WO 2020000878A1 CN 2018116333 W CN2018116333 W CN 2018116333W WO 2020000878 A1 WO2020000878 A1 WO 2020000878A1
Authority
WO
WIPO (PCT)
Prior art keywords
pixel
matrix
target
image
resolution
Prior art date
Application number
PCT/CN2018/116333
Other languages
English (en)
French (fr)
Inventor
余林韵
李磊
尹海斌
朱文佳
姜东�
Original Assignee
北京字节跳动网络技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京字节跳动网络技术有限公司 filed Critical 北京字节跳动网络技术有限公司
Publication of WO2020000878A1 publication Critical patent/WO2020000878A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Definitions

  • Embodiments of the present application relate to the field of computer technology, and in particular, to a method and an apparatus for generating an image.
  • High resolution means that the pixel density in the image is high, which can provide more details, which are indispensable in many practical applications.
  • high-resolution medical images are very helpful for doctors to make a correct diagnosis; using high-resolution satellite images can easily distinguish similar objects from similar ones; if high-resolution images can be provided, computer vision The performance of pattern recognition will be greatly improved.
  • the existing method for generating a high-resolution image usually directly learns a mapping relationship from a low-resolution image to a high-resolution image, and processes the original image based on the mapping relationship to generate a high-resolution image.
  • the embodiments of the present application provide a method and a device for generating an image.
  • an embodiment of the present application provides a method for generating an image.
  • the method includes: interpolating a target image; and for a pixel in the target image after interpolation, extracting a first pixel matrix centered on the pixel , Performing a principal component analysis on the first pixel matrix to obtain a target matrix; for a pixel in the target image after interpolation, based on the target matrix corresponding to the pixel, a filter is selected from a pre-generated filter set, and the pixel is extracted as The second pixel matrix in the center is convolved with the selected filter to obtain a high-resolution pixel value corresponding to the pixel; the obtained high-resolution pixel values are summarized to generate a high-resolution image .
  • the filter set is generated by the following steps: extracting a set of high-resolution image samples, and sequentially downsampling and interpolating the high-resolution image samples in the set of high-resolution image samples; for the high-resolution after interpolation For the pixels in the image sample, a third pixel matrix centered on the pixel is extracted, and the third pixel matrix is subjected to principal component analysis to obtain a target matrix sample; the obtained target matrix sample is classified; training is performed with each type of target matrix The filters corresponding to the samples are aggregated into a filter set.
  • classifying the obtained target matrix samples includes: performing a dot multiplication operation on the obtained target matrix samples and a preset matrix; and classifying target matrix samples with the same result of the dot multiplication operation into one class.
  • training a filter corresponding to each type of target matrix sample, and summarizing the trained filters into a filter set including: a high resolution after interpolation corresponding to each type of target matrix sample For a pixel in an image sample, a fourth pixel matrix centered on the pixel is extracted, the fourth pixel matrix is taken as an input, and a high-resolution pixel corresponding to the pixel is used as an output.
  • the machine learning method is used to obtain a sample of the target matrix Corresponding filter.
  • the number of rows and columns of the first pixel matrix are the same as the number of rows and columns of the third pixel matrix, and the number of rows and columns of the second pixel matrix are respectively the number of rows and columns of the fourth pixel matrix.
  • the number of columns is the same.
  • the method further includes: establishing a correspondence between a point multiplication operation result and a filter.
  • performing a principal component analysis on the first pixel matrix to obtain a target matrix includes: determining a covariance matrix of the first pixel matrix; determining a feature value and a feature vector of the covariance matrix; and from the determined feature value The target feature value is selected, and the feature vector corresponding to the target feature value is formed into a feature matrix; the first pixel matrix is multiplied with the feature matrix to obtain the target matrix.
  • selecting a filter from a pre-generated filter set based on the target matrix corresponding to the pixel includes: for the pixel in the interpolated target image, changing the The target matrix corresponding to the pixel performs a dot multiplication operation with a preset matrix, and a filter corresponding to the dot multiplication operation result is selected from a pre-generated filter set.
  • an embodiment of the present application provides an apparatus for generating an image.
  • the apparatus includes: an interpolation unit configured to interpolate a target image; and an analysis unit configured to interpolate pixels in the target image after interpolation. , Extracting a first pixel matrix centered on the pixel, and performing principal component analysis on the first pixel matrix to obtain a target matrix; a selecting unit is configured to, for the pixels in the target image after interpolation, based on the target matrix corresponding to the pixel , Selecting a filter from a pre-generated filter set, extracting a second pixel matrix centered on the pixel, and using the selected filter to convolve the second pixel matrix to obtain a high-resolution pixel value corresponding to the pixel A generating unit configured to aggregate the obtained high-resolution pixel values to generate a high-resolution image.
  • the filter set is generated by the following steps: extracting a set of high-resolution image samples, and sequentially downsampling and interpolating the high-resolution image samples in the set of high-resolution image samples; for the high-resolution after interpolation For the pixels in the image sample, a third pixel matrix centered on the pixel is extracted, and the third pixel matrix is subjected to principal component analysis to obtain a target matrix sample; the obtained target matrix sample is classified; training is performed with each type of target matrix The filters corresponding to the samples are aggregated into a filter set.
  • classifying the obtained target matrix samples includes: performing a dot multiplication operation on the obtained target matrix samples and a preset matrix; and classifying target matrix samples with the same result of the dot multiplication operation into one class.
  • training a filter corresponding to each type of target matrix sample and summarizing the trained filters into a filter set, including: a high resolution after interpolation corresponding to each type of target matrix sample
  • a high resolution after interpolation corresponding to each type of target matrix sample For a pixel in an image sample, a fourth pixel matrix centered on the pixel is extracted, the fourth pixel matrix is taken as an input, and a high-resolution pixel corresponding to the pixel is used as an output.
  • the machine learning method is used to obtain a sample of the target matrix of this type. Corresponding filter.
  • the number of rows and columns of the first pixel matrix are the same as the number of rows and columns of the third pixel matrix, and the number of rows and columns of the second pixel matrix are respectively the number of rows and columns of the fourth pixel matrix.
  • the number of columns is the same.
  • the apparatus further includes: a establishing unit configured to establish a correspondence between a point multiplication operation result and a filter.
  • the analysis unit includes: a first determination module configured to determine a covariance matrix of the first pixel matrix; a second determination module configured to determine a feature value and a feature vector of the covariance matrix; a composition module, It is configured to select a target feature value from the determined feature values, and compose a feature vector corresponding to the target feature value into a feature matrix; a multiplication module is configured to multiply the first pixel matrix and the feature matrix to obtain a target matrix.
  • the selecting unit is further configured to: for a pixel in the target image after interpolation, perform a point multiplication operation on a target matrix corresponding to the pixel and a preset matrix, and select an AND point from a pre-generated filter set. The filter corresponding to the multiplication result.
  • an embodiment of the present application provides an electronic device, including: one or more processors; a storage device storing one or more programs thereon; when one or more programs are processed by one or more processors Execution causes one or more processors to implement the method as in any one of the methods for generating an image.
  • an embodiment of the present application provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the method as in any one of the methods for generating an image is implemented.
  • the method and device for generating an image obtained in the embodiments of the present application obtain the pixel by interpolating the target image, and then processing the first pixel matrix of each pixel in the target image after interpolation by using a principal component analysis method.
  • the high-resolution pixel values corresponding to each pixel are finally summarized to generate a high-resolution image. Therefore, in the process of generating a high-resolution image, processing of a pixel matrix using a principal component analysis method can more accurately select a filter corresponding to each pixel, thereby improving the effect of generating a high-resolution image.
  • FIG. 1 is an exemplary system architecture diagram to which an embodiment of the present application can be applied;
  • FIG. 1 is an exemplary system architecture diagram to which an embodiment of the present application can be applied;
  • FIG. 2 is a flowchart of an embodiment of a method for generating an image according to the present application
  • FIG. 3 is a schematic diagram of an application scenario of a method for generating an image according to the present application
  • FIG. 4 is a flowchart of still another embodiment of a method for generating an image according to the present application.
  • 5 and 7 are flowcharts of an embodiment of a method for generating a filter set according to the present application.
  • FIG. 6 is a schematic structural diagram of an embodiment of an apparatus for generating an image according to the present application.
  • FIG. 7 is a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application.
  • FIG. 1 illustrates an exemplary system architecture 100 to which the method for generating an image or the apparatus for generating an image of the present application can be applied.
  • the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105.
  • the network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105.
  • the network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
  • the user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages (such as image processing requests) and the like.
  • Various communication client applications such as image processing applications, video playback applications, information browsing applications, and social platform software, can be installed on the terminal devices 101, 102, and 103.
  • the terminal devices 101, 102, and 103 may be hardware or software.
  • the terminal devices 101, 102, and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.
  • the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. It is not specifically limited here.
  • the server 105 may be a server that provides various services, such as an image processing server for performing image processing.
  • the image processing server may perform processing such as interpolation and analysis on the received target image and other data, and feed back the processing result (such as a high-resolution image) to the terminal device.
  • the server may be hardware or software.
  • the server can be implemented as a distributed server cluster consisting of multiple servers or as a single server.
  • the server can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. It is not specifically limited here.
  • the method for generating an image provided by the embodiment of the present application is generally executed by the server 105, and accordingly, the apparatus for generating an image is generally set in the server 105.
  • the terminal devices 101, 102, and 103 can also directly perform processing such as interpolation and analysis on their stored target images.
  • the method for generating images provided by the embodiments of the present application can also be performed by the terminal devices. 101, 102, and 103.
  • the exemplary system architecture 100 at this time may not include the network 104 and the server 105 described above.
  • terminal devices, networks, and servers in FIG. 1 are merely exemplary. Depending on the implementation needs, there can be any number of terminal devices, networks, and servers.
  • a flowchart 200 of one embodiment of a method for generating an image according to the present application is shown.
  • the method for generating an image includes the following steps:
  • Step 201 Interpolate the target image.
  • an execution subject of the method for generating an image may first extract a target image, where the target image may be various images to be subjected to super-resolution reconstruction.
  • the above target image may be a face image, an article image, a landscape image, and the like.
  • the target image may be stored locally in advance, or may be sent by other electronic devices (for example, the terminal devices 101, 102, and 103 shown in FIG. 1) through a wired connection or a wireless connection.
  • the above wireless connection methods may include, but are not limited to, 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods now known or developed in the future.
  • the above-mentioned execution subject may use various existing image interpolation methods to interpolate the target image to enlarge the target image to a target size (for example, zoom in to 2 times, 3 times, or 4 times).
  • the above-mentioned target image interpolation may be performed using nearest neighbor interpolation, bilinear interpolation, bisquare interpolation, bicubic interpolation, or other high-order interpolation methods.
  • image interpolation is the process of generating high-resolution images from low-resolution images, and can be used to recover information lost in the images. It should be noted that the above-mentioned various image interpolation methods are well-known technologies that are widely studied and applied at present, and will not be repeated here.
  • the interpolation processing on the target image is performed first, and the resolution of the target image can be initially improved. Based on the interpolated target image, subsequent image processing steps can be performed to improve the effect of high-resolution image generation.
  • Step 202 For a pixel in the target image after interpolation, extract a first pixel matrix centered on the pixel, and perform principal component analysis on the first pixel matrix to obtain a target matrix.
  • the execution subject may first extract a first pixel matrix centered on the pixel.
  • the first pixel matrix may include pixel values of pixels in a square region (for example, a 3 ⁇ 3 patch) with the pixel as a center.
  • the execution body may perform principal component analysis (PCA) on the first pixel matrix to obtain a target matrix.
  • PCA principal component analysis
  • the covariance matrix of the first pixel matrix may be determined first.
  • the eigenvalues and eigenvectors of the covariance matrix can be determined.
  • the first pixel matrix can be projected into the space formed by the feature vectors, and the matrix obtained after the projection can be determined as the target matrix.
  • principal component analysis is also called principal component analysis, which aims to use the idea of dimensionality reduction to transform multiple indicators into a few comprehensive indicators.
  • principal component analysis is a technique that simplifies data sets. It is a linear transformation that transforms the data into a new coordinate system.
  • Principal component analysis can be used to reduce the dimensionality of the data set while maintaining the feature that the variance of the data set contributes the most. This is done by preserving low-order principal components and ignoring higher-order principal components. Such low-order components can often retain the most important aspects of the data.
  • the first pixel matrix centered on the pixel may be a square area centered on the pixel (for example, 3 ⁇ 3 Corresponding to the image patches (patch).
  • the values in the first pixel matrix may correspond to the pixels in the square area one by one, that is, the values in the i-th row and the j-th column of the first pixel matrix are the pixel values of the pixels in the i-th row and the j-th column in the square area.
  • i is an integer not less than 1 and not more than the number of rows of the first pixel matrix
  • j is an integer not less than 1 and not more than the number of columns of the first pixel matrix.
  • the values of these positions can be set to preset values (for example, 0).
  • the first pixel matrix centered on the pixel can be obtained in the following manner:
  • the pixel is extracted as The pixel values of the pixels in the square area in the center generate a pixel matrix in which the values in the matrix correspond one-to-one with the pixels in the square area.
  • the values of these positions can be set to preset values (for example, 0).
  • the pixel matrix is converted into a row vector. Since a row vector is a special form of a matrix, the vector can be determined as the first pixel matrix centered on the pixel. As an example, for a certain pixel in the target image after interpolation, a 3 ⁇ 3 patch pixel centered on the pixel can be extracted to generate a 3 ⁇ 3 (3 rows and 3 columns) pixel matrix. Then, the 3 ⁇ 3 pixel matrix may be converted into a row vector, and the row vector may be determined as a first pixel matrix of 1 ⁇ 9 (1 row and 9 columns).
  • Step 203 For a pixel in the target image after interpolation, based on the target matrix corresponding to the pixel, select a filter from a pre-generated filter set, extract a second pixel matrix centered on the pixel, and use the selected filter.
  • the second pixel matrix is convolved to obtain a high-resolution pixel value corresponding to the pixel.
  • the execution subject may select a filter from a pre-generated filter set based on a target matrix corresponding to the pixel.
  • a filter set may be stored in the execution body in advance.
  • Each filter in the above filter set may correspond to a category of pixels.
  • the above-mentioned execution body may analyze or calculate the target matrix of each pixel, determine the category of each pixel according to the analysis result or the calculation result, and then select a corresponding filter for each pixel.
  • the execution body may substitute the target matrix into a preset formula or function for calculation, and obtain a calculation result (for example, a value).
  • each filter in the filter set may be a parameter matrix or a parameter vector.
  • a filter is used to perform a convolution calculation on the target matrix of a pixel to obtain a high-resolution pixel value of the pixel.
  • the above-mentioned execution body may extract a second pixel matrix centered on the pixel.
  • the second pixel matrix may include pixel values of pixels in a square area (for example, a 7 ⁇ 7 image patch) with the pixel as a center.
  • the second pixel matrix may be a 7 ⁇ 7 matrix or a 1 ⁇ 49 row vector.
  • the selected second filter may be used to convolve the second pixel matrix to obtain a high-resolution pixel value corresponding to the pixel.
  • the size of the second pixel matrix of the pixel may be the same as or different from the size of the first pixel matrix, which is not limited herein.
  • the category of the pixels is determined based on the principal component analysis, and a filter corresponding to the category is selected to calculate high-resolution pixel values, so that each pixel in the target image after interpolation can be more accurately classified.
  • the execution body may perform a dot multiplication operation on a target matrix corresponding to the pixel and a preset matrix, and perform the dot multiplication operation result. As the category of this pixel. Then, a filter corresponding to a dot multiplication operation result can be selected from a pre-generated filter set. What needs to be explained here is that the total number of categories of pixels and the dot multiplication result of each category can be determined in advance based on the analysis processing of a large number of image samples, and a filter corresponding to each category can be generated in advance.
  • the execution body described above can store and extract the correspondence between each filter and the result of the dot multiplication operation.
  • step 204 the obtained high-resolution pixel values are summarized to generate a high-resolution image.
  • each pixel in the target image after interpolation can calculate a corresponding high-resolution pixel value
  • the above-mentioned execution body can perform the high-resolution pixel value corresponding to each pixel obtained. Summarized to produce high-resolution images.
  • FIG. 3 is a schematic diagram of an application scenario of the method for generating an image according to this embodiment.
  • a user first sends an image processing request to an image processing server using a terminal device, and the image processing request includes a target image 301 to be subjected to super-resolution image reconstruction.
  • the image processing server After receiving the target image 301, the image processing server first performs interpolation on the target image 301. Then, for each pixel in the interpolated image, a first pixel matrix centered on the pixel is extracted, and principal component analysis is performed on the first pixel matrix to obtain a target matrix.
  • the image processing server summarizes the obtained high-resolution pixel values to generate a high-resolution image 302.
  • the method provided by the foregoing embodiment of the present application obtains a target matrix corresponding to the pixel by interpolating the target image, and then processing the first pixel matrix of each pixel in the target image after interpolation by using a principal component analysis method. Then, the second pixel matrix of the pixel is convolved by using a filter selected based on the target matrix to obtain a high-resolution pixel value corresponding to the pixel, so as to obtain a height corresponding to each pixel in the target image after interpolation.
  • the resolution pixel values are finally summarized to generate a high-resolution image. Therefore, in the process of generating a high-resolution image, the interpolation processing of the target image is first performed, and the resolution of the target image can be initially improved.
  • the principal component analysis method is used to process the first pixel matrix of each pixel, and important features in the first pixel matrix of each pixel can be retained, so that different first The difference of one pixel matrix is more obvious, so the target matrix determined by this method can more accurately select the filter corresponding to each pixel, thereby further improving the effect of high-resolution image generation.
  • the process 400 of the method for generating an image includes the following steps:
  • Step 401 Interpolate a target image.
  • an execution subject of the method for generating an image may first extract a target image, where the target image may be various images to be subjected to super-resolution reconstruction. After extracting the target image, the above-mentioned execution subject may use various existing image interpolation methods to interpolate the target image to enlarge the target image to a target size (for example, zoom in to 2 times, 3 times, or 4 times).
  • the above-mentioned target image interpolation may be performed using nearest neighbor interpolation, bilinear interpolation, bisquare interpolation, bicubic interpolation, or other high-order interpolation methods.
  • the interpolation processing on the target image is performed first, and the resolution of the target image can be initially improved. Based on the interpolated target image, subsequent image processing steps can be performed to improve the effect of high-resolution image generation.
  • Step 402 For a pixel in the target image after interpolation, extract a first pixel matrix centered on the pixel.
  • the above-mentioned execution body may obtain the first pixel matrix centered on the pixel according to the following steps:
  • the first step is to extract pixel values of pixels in a square area centered on the pixel, and generate a pixel matrix in which the values in the matrix correspond one-to-one with the pixels in the square area. It should be noted that, for some pixels (such as pixels located at the edge of the image), there are no corresponding pixel values at certain positions in the pixel matrix of the pixels (for example, pixels located at the upper edge of the image, pixels of the pixel The first row of the matrix has no corresponding pixel value). At this time, the values of these positions can be set to preset values (for example, 0).
  • the pixel matrix is converted into a row vector, and the row vector is determined as a first pixel matrix.
  • a 3 ⁇ 3 patch pixel centered on the pixel can be extracted to generate a 3 ⁇ 3 (3 rows and 3 columns) pixel matrix.
  • the 3 ⁇ 3 pixel matrix may be converted into a row vector, and the row vector may be determined as a first pixel matrix of 1 ⁇ 9 (1 row and 9 columns).
  • Step 403 Determine a covariance matrix of the first pixel matrix.
  • the execution body may determine a covariance matrix of a first pixel matrix of the pixel. According to the calculation method of the covariance matrix, if the first pixel matrix is a 1 ⁇ 9 (1 row and 9 columns) matrix, the covariance matrix of the matrix is a 9 ⁇ 9 matrix.
  • Step 404 Determine the eigenvalues and eigenvectors of the covariance matrix.
  • the above-mentioned execution subject may determine a feature value and a feature vector of a covariance matrix corresponding to the pixel.
  • each feature value may correspond to a feature vector.
  • the calculation method of the covariance matrix the calculation method of the eigenvalues of the matrix, and the calculation method of the eigenvectors of the matrices are well-known techniques widely studied and applied in the field of mathematics at present, and will not be repeated here.
  • Step 405 Select a target feature value from the determined feature values, and form a feature matrix corresponding to the feature vector corresponding to the target feature value.
  • the above-mentioned execution subject may use various selection methods to select a target feature value from the determined feature values, and form a feature matrix corresponding to the target feature value into a feature matrix.
  • the execution body may select a preset number of feature vectors from the feature vectors corresponding to the feature values in the order of the feature values from large to small as the target feature vectors, and sequentially combine the target feature vectors to The transpose of the obtained matrix is determined as the feature matrix.
  • the covariance matrix is a 9 ⁇ 9 matrix.
  • the execution body determines that the matrix has 9 eigenvalues and 9 corresponding eigenvectors
  • the eigenvectors can be sorted in the order of the eigenvalues from large to small.
  • the first 8 feature vectors can be selected and combined to obtain an 8 ⁇ 9 (8 rows and 9 columns) matrix.
  • the transpose of the matrix can be determined as a feature matrix, and the feature matrix is a 9 ⁇ 8 (9 rows and 8 columns) matrix.
  • Step 406 Multiply the first pixel matrix and the feature matrix to obtain a target matrix.
  • the execution body may multiply a first pixel matrix of the pixel by a feature matrix corresponding to the pixel to obtain a target matrix corresponding to the pixel.
  • the first pixel matrix is a matrix of 1 ⁇ 9 (1 row and 9 columns)
  • the feature matrix is a matrix of 9 ⁇ 8 (9 rows and 8 columns). The two matrices are multiplied to obtain 1 ⁇ 8 (1 row and 8 columns). ) Target matrix.
  • the principal component analysis is used to process the first pixel matrix, and the first pixel matrix is reduced in dimension.
  • the important features in the first pixel matrix of each pixel can be retained, so that The difference is more obvious, so it can achieve more accurate classification of pixels in the target image after interpolation.
  • Step 407 For a pixel in the target image after interpolation, perform a dot multiplication operation on a target matrix corresponding to the pixel and a preset matrix, and select a filter corresponding to a dot multiplication operation result from a pre-generated filter set.
  • a point multiplication operation is performed on the target matrix corresponding to the pixel and a preset matrix, and a corresponding multiplication result is selected from a pre-generated filter set. Filter.
  • FIG. 5 shows a flowchart of an embodiment of a method for generating a filter set according to the present application.
  • the method 500 for generating a filter set includes the following steps:
  • Step 501 Extract a set of high-resolution image samples, and sequentially down-sample and interpolate the high-resolution image samples in the set of high-resolution image samples.
  • each high-resolution sample in the above-mentioned high-resolution image sample set it may be down-sampled first, and then the down-sampled high-resolution samples are interpolated.
  • the downsampling multiple can be set in advance.
  • performing 2x downsampling on a high-resolution image sample may be converting an image in a 2 ⁇ 2 image block in the high-resolution image sample into a pixel whose pixel value is equal to the 2 ⁇ 2 image block The mean of the pixel values of all the pixels.
  • the same interpolation method as in step 401 may be used for interpolation, and details are not described herein again.
  • Step 502 For a pixel in the high-resolution image sample after interpolation, extract a third pixel matrix centered on the pixel, and perform principal component analysis on the third pixel matrix to obtain a target matrix sample.
  • a third pixel matrix centered on the pixel can be extracted, and the third pixel matrix is subjected to principal component analysis to obtain a target matrix sample.
  • the third pixel matrix extraction step for pixels in the high-resolution image sample after interpolation is basically the same as the first pixel matrix extraction step for the interpolation target image described in step 402;
  • the steps of performing a principal component analysis on the pixel matrix to obtain a target matrix sample are basically the same as the operations of steps 403 to 406, and are not repeated here.
  • the number of rows and columns of the first pixel matrix may be the same as the number of rows and columns of the third pixel matrix, respectively.
  • each of the first pixel matrix and the third pixel matrix may be a 1 ⁇ 9 matrix.
  • Step 503 Classify the obtained target matrix samples.
  • each obtained target matrix sample may be substituted into a preset formula or function for calculation, to obtain a calculation result (for example, a numerical value), and the target matrix samples corresponding to the same calculation result are classified into the same class.
  • a calculation result for example, a numerical value
  • Each category can be characterized by a calculation result.
  • a point multiplication operation may first be performed on each obtained target matrix sample and a preset matrix. Then, the target matrix samples with the same result of the dot multiplication operation are divided into one class.
  • Step 504 Train filters corresponding to each type of target matrix sample, and aggregate the trained filters into a filter set.
  • a machine learning method can be used to train the filter corresponding to that type.
  • Each filter here can be a parameter vector.
  • a fourth pixel matrix centered on the pixel may be first extracted, and the fourth pixel may be A matrix is used as an input, and a high-resolution pixel corresponding to the pixel is used as an output, and a filter corresponding to this type of target matrix sample is obtained by training using a machine learning method.
  • the step of extracting the fourth pixel matrix is basically the same as the step of extracting the second pixel matrix in step 203, and details are not described herein again.
  • a correspondence relationship between a point multiplication operation result and a filter may also be established.
  • the result of the dot multiplication operation can be used as the key of the key-value pair
  • the parameter vector or parameter matrix used to characterize the filter can be used as the value of the key-value pair. relationship.
  • the principal component analysis technology is used to reduce the dimensions of the pixels and then classify them.
  • the important features of each pixel can be retained, so that the differences between different pixels are more obvious, and the pixel classification can be more accurate. Filter training on this basis can improve the pertinence and accuracy of the filter.
  • Step 408 Extract a second pixel matrix centered on the pixel, and convolve the second pixel matrix by using the selected filter to obtain a high-resolution pixel value corresponding to the pixel.
  • the execution subject described above may extract a second pixel matrix centered on the pixel. Then, the selected pixel is used to convolve the second pixel matrix corresponding to the pixel to obtain a high-resolution pixel value corresponding to the pixel.
  • the number of rows and columns of the second pixel matrix may be the same as the number of rows and columns of the fourth pixel matrix, respectively.
  • each of the second pixel matrix and the fourth pixel matrix may be a 1 ⁇ 49 matrix.
  • step 409 the obtained high-resolution pixel values are summarized to generate a high-resolution image.
  • each pixel in the target image after interpolation can calculate a corresponding high-resolution pixel value
  • the above-mentioned execution body can perform the high-resolution pixel value corresponding to each pixel obtained. Summarized to produce high-resolution images.
  • the process 400 of the method for generating an image in this embodiment highlights the steps of generating the filter set. Therefore, in the solution described in this embodiment, the pixels are dimensionally reduced and classified by the principal component analysis technology, and important features of each pixel can be retained, so that differences between different pixels are more obvious, and thus pixel classification can be more accurate. Filter training on this basis can improve the pertinence and accuracy of the filter. The use of the trained filters to generate high-resolution images further improves the effect of high-resolution image generation.
  • this application provides an embodiment of an apparatus for generating an image.
  • the apparatus embodiment corresponds to the method embodiment shown in FIG. 2.
  • the device can be specifically applied to various electronic devices.
  • the apparatus 600 for generating an image includes: an interpolation unit 601 configured to perform interpolation on a target image; and an analysis unit 602 configured to perform interpolation on pixels in the target image after interpolation , Extracting a first pixel matrix centered on the pixel, and performing principal component analysis on the first pixel matrix to obtain a target matrix; a selecting unit 603 is configured to, for the pixels in the target image after interpolation, based on the corresponding pixel values The target matrix, a filter is selected from a pre-generated filter set, a second pixel matrix centered on the pixel is extracted, and the above-mentioned second pixel matrix is convolved by using the selected filter to obtain a high resolution corresponding to the pixel Rate pixel value; generating unit 604 is configured to summarize the obtained high-resolution pixel values to generate a high-resolution image.
  • the foregoing filter set may be generated by the following steps: extracting a set of high-resolution image samples, and sequentially down-sampling the high-resolution image samples in the above-mentioned high-resolution image sample set. And interpolation; for a pixel in the high-resolution image sample after interpolation, extract a third pixel matrix centered on the pixel, and perform principal component analysis on the third pixel matrix to obtain a target matrix sample; and obtain the target matrix sample
  • the samples are classified; the filters corresponding to each type of target matrix sample are trained, and the trained filters are aggregated into a filter set.
  • the above-mentioned classification of the obtained target matrix samples may include: performing a point comparison between the obtained target matrix samples and a preset matrix.
  • Multiplication operation Divide the target matrix samples with the same result of the dot multiplication operation into one class.
  • training the filters corresponding to each type of target matrix sample in the above-mentioned filter set generation step, and summing the trained filters into a filter set may include: : For each pixel in the high-resolution image sample corresponding to each type of target matrix sample, a fourth pixel matrix centered on the pixel is extracted, and the fourth pixel matrix is used as an input, and the height corresponding to the pixel is The resolution pixels are used as the output, and the filters corresponding to this type of target matrix samples are obtained by training using machine learning methods.
  • the number of rows and columns of the first pixel matrix may be the same as the number of rows and columns of the third pixel matrix, and the number of rows and columns of the second pixel matrix may be respectively It is the same as the number of rows and columns of the fourth pixel matrix.
  • the apparatus may further include a establishing unit (not shown in the figure).
  • the establishing unit may be configured to establish a correspondence between a point multiplication operation result and a filter.
  • the foregoing analysis unit 602 may include a first determination module, a second determination module, a composition module, and a multiplication module (not shown in the figure).
  • the first determination module may be configured to determine a covariance matrix of the first pixel matrix.
  • the second determination module may be configured to determine an eigenvalue and a eigenvector of the covariance matrix.
  • the composition module may be configured to select a target feature value from the determined feature values, and form a feature matrix corresponding to the target feature value.
  • the multiplication module may be configured to multiply the first pixel matrix and the feature matrix to obtain a target matrix.
  • the above-mentioned selecting unit 603 may be further configured to: for a pixel in the target image after interpolation, perform a dot product operation on the target matrix corresponding to the pixel and a preset matrix, A filter corresponding to a point multiplication operation result is selected from a pre-generated filter set.
  • the device provided by the foregoing embodiment of the present application uses the interpolation unit 601 to interpolate the target image, and the analysis unit 602 uses the principal component analysis method to process the first pixel matrix of each pixel in the interpolated target image to obtain The target matrix corresponding to the pixel, and then the selection unit 603 uses the filter selected based on the target matrix to convolve the second pixel matrix of the pixel to obtain a high-resolution pixel value corresponding to the pixel, so as to obtain the value after interpolation.
  • the high-resolution pixel values corresponding to each pixel in the target image are finally generated by the generating unit 604 to generate a high-resolution image.
  • the principal component analysis method is used to determine the target matrix of each pixel, and important features in the first pixel matrix of each pixel can be retained, so that The difference is more obvious, so the target matrix determined by this method can more accurately select the filter corresponding to each pixel, thereby improving the effect of high-resolution image generation.
  • FIG. 7 illustrates a schematic structural diagram of a computer system 700 suitable for implementing an electronic device according to an embodiment of the present application.
  • the electronic device shown in FIG. 7 is only an example, and should not impose any limitation on the functions and scope of use of the embodiments of the present application.
  • the computer system 700 includes a central processing unit (CPU) 701, which can be based on a program stored in a read-only memory (ROM) 702 or a program loaded from a storage section 708 into a random access memory (RAM) 703. Instead, perform various appropriate actions and processes.
  • ROM read-only memory
  • RAM random access memory
  • various programs and data required for the operation of the system 700 are also stored.
  • the CPU 701, ROM 702, and RAM 703 are connected to each other through a bus 704.
  • An input / output (I / O) interface 705 is also connected to the bus 704.
  • the following components are connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), and the speaker; a storage portion 708 including a hard disk and the like And a communication section 709 including a network interface card such as a LAN card, a modem, and the like.
  • the communication section 709 performs communication processing via a network such as the Internet.
  • the driver 710 is also connected to the I / O interface 705 as needed.
  • a removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read out therefrom is installed into the storage section 708 as needed.
  • the process described above with reference to the flowchart may be implemented as a computer software program.
  • embodiments of the present disclosure include a computer program product including a computer program carried on a computer-readable medium, the computer program containing program code for performing a method shown in a flowchart.
  • the computer program may be downloaded and installed from a network through the communication section 709, and / or installed from a removable medium 711.
  • CPU central processing unit
  • the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two.
  • the computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programming read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal that is included in baseband or propagated as part of a carrier wave, and which carries computer-readable program code. Such a propagated data signal may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, and the computer-readable medium may send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.
  • Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
  • each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, which contains one or more functions to implement a specified logical function Executable instructions.
  • the functions labeled in the blocks may also occur in a different order than those labeled in the drawings. For example, two blocks represented one after the other may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved.
  • each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts can be implemented by a dedicated hardware-based system that performs the specified function or operation , Or it can be implemented with a combination of dedicated hardware and computer instructions.
  • the units described in the embodiments of the present application may be implemented by software or hardware.
  • the described unit may also be provided in a processor, for example, it may be described as: a processor includes an interpolation unit, an analysis unit, a selection unit, and a generation unit.
  • a processor includes an interpolation unit, an analysis unit, a selection unit, and a generation unit.
  • the names of these units do not constitute a limitation on the unit itself in some cases.
  • the interpolation unit can also be described as a “unit that interpolates the target image”.
  • the present application further provides a computer-readable medium, which may be included in the device described in the foregoing embodiments; or may exist alone without being assembled into the device.
  • the computer-readable medium carries one or more programs.
  • the device is caused to: interpolate a target image; for pixels in the target image after interpolation, extract the pixels
  • the first pixel matrix is the center, and the principal component analysis is performed on the first pixel matrix to obtain the target matrix.
  • the pixels in the interpolated target image based on the target matrix corresponding to the pixel, it is selected from the pre-generated filter set.
  • a filter extracting a second pixel matrix centered on the pixel, and convolving the second pixel matrix with the selected filter to obtain a high-resolution pixel value corresponding to the pixel;
  • the values are summarized to produce a high-resolution image.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Editing Of Facsimile Originals (AREA)

Abstract

本申请实施例公开了用于生成图像的方法和装置。该方法的一具体实施方式包括：对目标图像进行插值；对于插值后的目标图像中的像素，提取以该像素为中心的第一像素矩阵，对该第一像素矩阵进行主成分分析，得到目标矩阵；对于插值后的目标图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对该第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值；将所得到的高分辨率像素值进行汇总，生成高分辨率图像。该实施方式提升了超分辨率图像生成的效果。

Description

用于生成图像的方法和装置
本专利申请要求于2018年6月26日提交的、申请号为201810668219.7、申请人为北京字节跳动网络技术有限公司、发明名称为“用于生成图像的方法和装置”的中国专利申请的优先权，该申请的全文以引用的方式并入本申请中。
技术领域
本申请实施例涉及计算机技术领域，具体涉及用于生成图像的方法和装置。
背景技术
在电子图像应用领域，人们经常期望得到高分辨率图像。高分辨率意味着图像中的像素密度高，能够提供更多的细节，而这些细节在许多实际应用中不可或缺。例如，高分辨率医疗图像对于医生做出正确的诊断是非常有帮助的；使用高分辨率卫星图像就很容易从相似物中区别相似的对象；如果能够提供高分辨的图像，计算机视觉中的模式识别的性能就会大大提高。
现有的生成高分辨率图像的方法，通常是直接学习由低分辨率图像到高分辨率图像的映射关系，基于该映射关系对原始图像进行处理，生成高分辨率图像。
发明内容
本申请实施例提出了用于生成图像的方法和装置。
第一方面，本申请实施例提供了一种用于生成图像的方法，该方法包括：对目标图像进行插值；对于插值后的目标图像中的像素，提取以该像素为中心的第一像素矩阵，对第一像素矩阵进行主成分分析，得到目标矩阵；对于插值后的目标图像中的像素，基于该像素对应的 目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值；将所得到的高分辨率像素值进行汇总，生成高分辨率图像。
在一些实施例中，滤波器集合通过如下步骤生成：提取高分辨率图像样本集合，对高分辨率图像样本集合中的高分辨率图像样本依次进行下采样和插值；对于插值后的高分辨率图像样本中的像素，提取以该像素为中心的第三像素矩阵，对第三像素矩阵进行主成分分析，得到目标矩阵样本；对所得到的目标矩阵样本进行分类；训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合。
在一些实施例中，对所得到的目标矩阵样本进行分类，包括：将所得到的目标矩阵样本与预设矩阵进行点乘运算；将点乘运算结果相同的目标矩阵样本划分为一类。
在一些实施例中，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合，包括：对于每一类目标矩阵样本对应的、插值后的高分辨率图像样本中的像素，提取以该像素为中心的第四像素矩阵，将第四像素矩阵作为输入，将该像素对应的高分辨率像素作为输出，利用机器学习方法训练得到与该类目标矩阵样本对应的滤波器。
在一些实施例中，第一像素矩阵的行数、列数分别与第三像素矩阵的行数、列数相同，第二像素矩阵的行数、列数分别与第四像素矩阵的行数、列数相同。
在一些实施例中，该方法还包括：建立点乘运算结果与滤波器的对应关系。
在一些实施例中，对第一像素矩阵进行主成分分析，得到目标矩阵，包括：确定第一像素矩阵的协方差矩阵；确定协方差矩阵的特征值和特征向量；从所确定的特征值中选取目标特征值，将目标特征值对应的特征向量组成特征矩阵；将第一像素矩阵与特征矩阵相乘，得到目标矩阵。
在一些实施例中，对于插值后的目标图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，包括：对于插值后的目标图像中的像素，将该像素对应的目标矩阵与预设矩阵进行点乘运算，从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。
第二方面，本申请实施例提供了一种用于生成图像的装置，该装置包括：插值单元，被配置成对目标图像进行插值；分析单元，被配置成对于插值后的目标图像中的像素，提取以该像素为中心的第一像素矩阵，对第一像素矩阵进行主成分分析，得到目标矩阵；选取单元，被配置成对于插值后的目标图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值；生成单元，被配置成将所得到的高分辨率像素值进行汇总，生成高分辨率图像。
在一些实施例中，滤波器集合通过如下步骤生成：提取高分辨率图像样本集合，对高分辨率图像样本集合中的高分辨率图像样本依次进行下采样和插值；对于插值后的高分辨率图像样本中的像素，提取以该像素为中心的第三像素矩阵，对第三像素矩阵进行主成分分析，得到目标矩阵样本；对所得到的目标矩阵样本进行分类；训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合。
在一些实施例中，对所得到的目标矩阵样本进行分类，包括：将所得到的目标矩阵样本与预设矩阵进行点乘运算；将点乘运算结果相同的目标矩阵样本划分为一类。
在一些实施例中，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合，包括：对于每一类目标矩阵样本对应的、插值后的高分辨率图像样本中的像素，提取以该像素为中心的第四像素矩阵，将第四像素矩阵作为输入，将该像素对应的高分辨率像素作为输出，利用机器学习方法训练得到与该类目标矩阵样本对应的滤波器。
在一些实施例中，第一像素矩阵的行数、列数分别与第三像素矩阵的行数、列数相同，第二像素矩阵的行数、列数分别与第四像素矩阵的行数、列数相同。
在一些实施例中，该装置还包括：建立单元，被配置成建立点乘运算结果与滤波器的对应关系。
在一些实施例中，分析单元包括：第一确定模块，被配置成确定第一像素矩阵的协方差矩阵；第二确定模块，被配置成确定协方差矩阵的特征值和特征向量；组成模块，被配置成从所确定的特征值中选取目标特征值，将目标特征值对应的特征向量组成特征矩阵；相乘模块，被配置成将第一像素矩阵与特征矩阵相乘，得到目标矩阵。
在一些实施例中，选取单元进一步被配置成：对于插值后的目标图像中的像素，将该像素对应的目标矩阵与预设矩阵进行点乘运算，从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。
第三方面，本申请实施例提供了一种电子设备，包括：一个或多个处理器；存储装置，其上存储有一个或多个程序，当一个或多个程序被一个或多个处理器执行，使得一个或多个处理器实现如用于生成图像的方法中任一实施例的方法。
第四方面，本申请实施例提供了一种计算机可读介质，其上存储有计算机程序，该程序被处理器执行时实现如用于生成图像的方法中任一实施例的方法。
本申请实施例提供的用于生成图像的方法和装置，通过对目标图像进行插值，而后利用主成分分析方式对插值后的目标图像中的每一像素的第一像素矩阵进行处理，得到该像素对应的目标矩阵，之后利用基于该目标矩阵所选取的滤波器对该像素的第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值，从而得到与插值后的目标图像中的各个像素对应的高分辨率像素值，最后汇总生成高分辨率图像。由此，在高分辨率图像生成的过程中，利用主成分分析方式进行像素矩阵的处理，可以更加准确地选取各像素对应的滤波器，从而提升了高分辨率图像生成的效果。
附图说明
通过阅读参照以下附图所作的对非限制性实施例所作的详细描述，本申请的其它特征、目的和优点将会变得更明显：
图1是本申请的一个实施例可以应用于其中的示例性系统架构图；
图2是根据本申请的用于生成图像的方法的一个实施例的流程图；
图3是根据本申请的用于生成图像的方法的一个应用场景的示意图；
图4是根据本申请的用于生成图像的方法的又一个实施例的流程图；
图5图7是根据本申请的用于生成滤波器集合的方法的一个实施例的流程图；
图6是根据本申请的用于生成图像的装置的一个实施例的结构示意图；
图7是适于用来实现本申请实施例的电子设备的计算机系统的结构示意图。
具体实施方式
下面结合附图和实施例对本申请作进一步的详细说明。可以理解的是，此处所描述的具体实施例仅仅用于解释相关发明，而非对该发明的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与有关发明相关的部分。
需要说明的是，在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。下面将参考附图并结合实施例来详细说明本申请。
图1示出了可以应用本申请的用于生成图像的方法或用于生成图像的装置的示例性系统架构100。
如图1所示，系统架构100可以包括终端设备101、102、103，网络104和服务器105。网络104用以在终端设备101、102、103和 服务器105之间提供通信链路的介质。网络104可以包括各种连接类型，例如有线、无线通信链路或者光纤电缆等等。
用户可以使用终端设备101、102、103通过网络104与服务器105交互，以接收或发送消息(例如图像处理请求)等。终端设备101、102、103上可以安装有各种通讯客户端应用，例如图像处理类应用、视频播放类应用、资讯浏览类应用、社交平台软件等。
终端设备101、102、103可以是硬件，也可以是软件。当终端设备101、102、103为硬件时，可以是具有显示屏并且支持网页浏览的各种电子设备，包括但不限于智能手机、平板电脑、膝上型便携计算机和台式计算机等等。当终端设备101、102、103为软件时，可以安装在上述所列举的电子设备中。其可以实现成多个软件或软件模块(例如用来提供分布式服务)，也可以实现成单个软件或软件模块。在此不做具体限定。
服务器105可以是提供各种服务的服务器，例如用于进行图像处理的图像处理服务器。图像处理服务器可以对接收到的目标图像等数据进行插值、分析等处理，并将处理结果(例如高分辨率图像)反馈给终端设备。
需要说明的是，服务器可以是硬件，也可以是软件。当服务器为硬件时，可以实现成多个服务器组成的分布式服务器集群，也可以实现成单个服务器。当服务器为软件时，可以实现成多个软件或软件模块(例如用来提供分布式服务)，也可以实现成单个软件或软件模块。在此不做具体限定。
需要指出的是，本申请实施例所提供的用于生成图像的方法一般由服务器105执行，相应地，用于生成图像的装置一般设置于服务器105中。
需要指出的是，终端设备101、102、103也可以直接对其所存储的目标图像进行插值、分析等处理，此时，本申请实施例所提供的用于生成图像的方法也可以由终端设备101、102、103执行，此时的示例性系统架构100可以不包含上述网络104和服务器105。
应该理解，图1中的终端设备、网络和服务器的数目仅仅是示意 性的。根据实现需要，可以具有任意数目的终端设备、网络和服务器。
继续参考图2，示出了根据本申请的用于生成图像的方法的一个实施例的流程200。该用于生成图像的方法，包括以下步骤：
步骤201，对目标图像进行插值。
在本实施例中，用于生成图像的方法的执行主体(例如图1所示的服务器105)可以首先提取目标图像，其中，上述目标图像可以是各种待进行超分辨率重建的图像。例如，上述目标图像可以是人脸图像、物品图像、景观图像等等。上述目标图像可以预先存储于本地，也可以是由其他电子设备(例如图1所示的终端设备101、102、103)通过有线连接或者无线连接方式发送的。其中，上述无线连接方式可以包括但不限于3G/4G连接、WiFi连接、蓝牙连接、WiMAX连接、Zigbee连接、UWB(ultra wideband)连接、以及其他现在已知或将来开发的无线连接方式。
在提取目标图像之后，上述执行主体可以利用各种现有的图像插值方式对上述目标图像进行插值，以将目标图像放大至目标尺寸(如放大至2倍、3倍、4倍)。此处，可以采用诸如最近邻插值、双线性插值、双平方插值、双立方插值或者其他高阶插值方法等进行上述目标图像的插值。实践中，图像插值是从低分辨率图像生成高分辨率图像的过程，可以用以恢复图像中所丢失的信息。需要说明的是，上述各种图像插值方式是目前广泛研究和应用的公知技术，在此不再赘述。
此处，在进行高分辨率图像生成的过程中，首先执行对目标图像的插值的处理，可以初步地提高目标图像的分辨率。在插值后的目标图像的基础上，再进行后续图像处理步骤，可以提升高分辨率图像生成的效果。
步骤202，对于插值后的目标图像中的像素，提取以该像素为中心的第一像素矩阵，对第一像素矩阵进行主成分分析，得到目标矩阵。
在本实施例中，对于插值后的目标图像中的像素，上述执行主体可以首先提取以该像素为中心的第一像素矩阵。其中，上述第一像素矩阵中可以包含以该像素为中心的正方形区域(例如3×3的图像块 (patch))内的像素的像素值。
而后，上述执行主体可以对上述第一像素矩阵进行主成分分析(Principal Components Analysis，PCA)，得到目标矩阵。具体地，可以首先确定第一像素矩阵的协方差矩阵。之后，可以确定协方差矩阵的特征值和特征向量。最后，可以将第一像素矩阵投影至特征向量构成的空间中，将投影后所得到的矩阵确定为目标矩阵。
实践中，主成分分析也称主分量分析，旨在利用降维的思想，把多指标转化为少数几个综合指标。在统计学中，主成分分析是一种简化数据集的技术。它是一个线性变换，把数据变换到一个新的坐标系统中。主成分分析可以用于减少数据集的维数，同时保持数据集的对方差贡献最大的特征。这是通过保留低阶主成分，忽略高阶主成分做到的。这样低阶成分往往能够保留住数据的最重要方面。由此，利用主成分分析的方式对第一像素矩阵进行处理，可以保留各个像素的第一像素矩阵中的重要的特征，从而使不同的第一像素矩阵的差异更为明显，因而可以实现更加准确地对插值后的目标图像内像素的分类。
在本实施例的一些可选的实现方式中，对于插值后的目标图像中的每一个像素，以该像素为中心的第一像素矩阵可以是以该像素为中心的正方形区域(例如3×3的图像块(patch))相对应的矩阵。第一像素矩阵中的数值可以与上述正方形区域中的像素一一对应，即第一像素矩阵的第i行第j列的数值为上述正方形区域中第i行第j列像素的像素值。其中，上述i为不小于1且不大于第一像素矩阵的行数的整数，上述j为不小于1且不大于第一像素矩阵的列数的整数。需要说明的是，对于某一些像素(例如位于图像边缘的像素)，该像素的第一像素矩阵中的某些位置不存在与之对应的像素值(例如，位于图像上边缘的像素，以该像素为中心的第一像素矩阵的第一行没有与之相对应的像素值)，此时，可以将这些位置的数值设置为预设值(例如0)。
在本实施例的一些可选的实现方式中，对于插值后的目标图像中的每一个像素，以该像素为中心的第一像素矩阵可以通过如下方式得到：第一步，提取以该像素为中心的正方形区域中的像素的像素值，生成矩阵中的数值与该正方形区域中的像素一一对应的像素矩阵。需 要说明的是，对于某一些像素(例如位于图像边缘的像素)，该像素的像素矩阵中的某些位置不存在与之对应的像素值(例如，位于图像上边缘的像素，该像素的像素矩阵的第一行没有与之相对应的像素值)，此时，可以将这些位置的数值设置为预设值(例如0)。第二步，将该像素矩阵转换为行向量。由于行向量是矩阵的一种特殊的形式，因此，可以将该向量确定为以该像素为中心的第一像素矩阵。作为示例，对于插值后的目标图像中的某一个像素，可以提取以该像素为中心的3×3的patch的像素，生成3×3(3行3列)的像素矩阵。而后，可以将该3×3的像素矩阵转换成行向量，将该行向量确定为1×9(1行9列)的第一像素矩阵。
步骤203，对于插值后的目标图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值。
在本实施例中，对于插值后的目标图像中的像素，上述执行主体可以基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器。此处，上述执行主体中可以预先存储有滤波器集合。上述滤波器集合中的每一个滤波器可以与像素的一个类别相对应。上述执行主体可以通过对各个像素的目标矩阵进行分析或计算，根据分析结果或者计算结果，确定各个像素的类别，进而针对每一个像素进行相应的滤波器的选取。作为示例，上述执行主体可以将上述目标矩阵代入预设的公式或者函数进行计算，得到计算结果(例如一个数值)。不同的计算结果可以对应有不同的滤波器，上述执行主体可以基于所得到的计算结果选取对应的滤波器。需要说明的是，滤波器集合中的每一个滤波器可以是一个参数矩阵或者参数向量。利用滤波器对某一个像素的目标矩阵进行卷积计算，可以得到该像素的高分辨率像素值。
对于每一个像素，在选取滤波器之后，上述执行主体可以提取以该像素为中心的第二像素矩阵。其中，上述第二像素矩阵中可以包含以该像素为中心的正方形区域(例如7×7的图像块(patch))内的像素的像素值。作为示例，第二像素矩阵可以是7×7的矩阵，也可以是 1×49的行向量。之后，可以利用选取的滤波器对上述第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值。需要说明的是，对于某一个像素，该像素的第二像素矩阵的尺寸可以与第一像素矩阵的尺寸相同或不同，此处不作限定。
由此，在主成分分析的基础上确定像素的类别，选取与该类别对应的滤波器，进行高分辨率像素值的计算，从而可以实现对插值后的目标图像内各像素更加精确的分类。
在本实施例的一些可选的实现方式中，对于插值后的目标图像中的每一个像素，上述执行主体可以将该像素对应的目标矩阵与预设矩阵进行点乘运算，将点乘运算结果作为该像素的类别。而后，可以从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。此处需要说明的是，可以预先基于大量图像样本的分析处理而预先确定像素的类别总数和每一个类别的点乘运算结果，并预先生成与每一个类别相对应的滤波器。上述执行主体可以存储和提取各个滤波器与点乘运算结果的对应关系。
步骤204，将所得到的高分辨率像素值进行汇总，生成高分辨率图像。
在本实施例中，由于插值后的目标图像中的每一个像素均可以计算得到一个对应的高分辨率像素值，因此，上述执行主体可以将所得到的各个像素对应的高分辨率像素值进行汇总，生成高分辨率图像。
继续参见图3，图3是根据本实施例的用于生成图像的方法的应用场景的一个示意图。在图3的应用场景中，用户首先使用终端设备向图像处理服务器发送了一个图像处理请求，该图像处理请求中包含待进行超分辨率图像重建的目标图像301。图像处理服务器接收到该目标图像301后，首先对该目标图像301进行插值。而后，对插值后的图像中的每一个像素，提取以该像素为中心的第一像素矩阵，对该第一像素矩阵进行主成分分分析，得到目标矩阵。之后，选取该目标矩阵对应的滤波器，利用该滤波器对该第一像素矩阵进行卷积，得到该像素的高分辨率像素值。最后，图像处理服务器将所得到的高分辨 率像素值进行汇总，生成高分辨率图像302。
本申请的上述实施例提供的方法，通过对目标图像进行插值，而后利用主成分分析方式对插值后的目标图像中的每一像素的第一像素矩阵进行处理，得到该像素对应的目标矩阵，之后利用基于该目标矩阵所选取的滤波器对该像素的第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值，从而得到与插值后的目标图像中的各个像素对应的高分辨率像素值，最后汇总生成高分辨率图像。由此，在进行高分辨率图像生成的过程中，首先执行对目标图像的插值的处理，可以初步地提高目标图像的分辨率。在插值后的目标图像的基础上，再进行后续图像处理步骤，可以提升高分辨率图像生成的效果。此外，在高分辨率图像生成的过程中，利用主成分分析方式对每一像素的的第一像素矩阵进行处理，可以保留各个像素的第一像素矩阵中的重要的特征，从而使不同的第一像素矩阵的差异更为明显，因而利用该方式所确定目标矩阵可以更加准确地选取各像素对应的滤波器，从而进一步提升了高分辨率图像生成的效果。
进一步参考图4，其示出了用于生成图像的方法的又一个实施例的流程400。该用于生成图像的方法的流程400，包括以下步骤：
步骤401，对目标图像进行插值。
在本实施例中，用于生成图像的方法的执行主体(例如图1所示的服务器105)可以首先提取目标图像，其中，上述目标图像可以是各种待进行超分辨率重建的图像。在提取目标图像之后，上述执行主体可以利用各种现有的图像插值方式对上述目标图像进行插值，以将目标图像放大至目标尺寸(如放大至2倍、3倍、4倍)。此处，可以采用诸如最近邻插值、双线性插值、双平方插值、双立方插值或者其他高阶插值方法等进行上述目标图像的插值。此处，在进行高分辨率图像生成的过程中，首先执行对目标图像的插值的处理，可以初步地提高目标图像的分辨率。在插值后的目标图像的基础上，再进行后续图像处理步骤，可以提升高分辨率图像生成的效果。
步骤402，对于插值后的目标图像中的像素，提取以该像素为中 心的第一像素矩阵。
在本实施例中，对于插值后的目标图像中的每一个像素，上述执行主体可以按照如下步骤得到以该像素为中心的第一像素矩阵：
第一步，提取以该像素为中心的正方形区域中的像素的像素值，生成矩阵中的数值与该正方形区域中的像素一一对应的像素矩阵。需要说明的是，对于某一些像素(例如位于图像边缘的像素)，该像素的像素矩阵中的某些位置不存在与之对应的像素值(例如，位于图像上边缘的像素，该像素的像素矩阵的第一行没有与之相对应的像素值)，此时，可以将这些位置的数值设置为预设值(例如0)。
第二步，将该像素矩阵转换为行向量，将该行向量确定为第一像素矩阵。作为示例，对于插值后的目标图像中的某一个像素，可以提取以该像素为中心的3×3的patch的像素，生成3×3(3行3列)的像素矩阵。而后，可以将该3×3的像素矩阵转换成行向量，将该行向量确定为1×9(1行9列)的第一像素矩阵。
步骤403，确定第一像素矩阵的协方差矩阵。
在本实施例中，对于插值后的目标图像中的每一个像素，上述执行主体可以确定该像素的第一像素矩阵的协方差矩阵。根据协方差矩阵的计算方法，若第一像素矩阵为1×9(1行9列)的矩阵，则该矩阵的协方差矩阵为9×9的矩阵。
步骤404，确定协方差矩阵的特征值和特征向量。
在本实施例中，对于插值后的目标图像中的每一个像素，上述执行主体可以确定该像素对应的协方差矩阵的特征值和特征向量。其中，每一个特征值可以对应一个特征向量。
此处，协方差矩阵的计算方法、矩阵的特征值的计算方法、矩阵的特征向量的计算方法是目前数学领域广泛研究和应用的公知技术，在此不再赘述。
步骤405，从所确定的特征值中选取目标特征值，将目标特征值对应的特征向量组成特征矩阵。
在本实施例中，对于插值后的目标图像中的每一个像素，上述执行主体可以利用各种选取方式从所确定的特征值中选取目标特征值， 将目标特征值对应的特征向量组成特征矩阵。此处，上述执行主体可以按照特征值从大到小的顺序，从特征值对应的特征向量中，依次选取预设数量的特征向量作为目标特征向量，并依次将目标特征向量进行组合，将所得到的矩阵的转置确定为特征矩阵。
作为示例，协方差矩阵为9×9的矩阵。上述执行主体确定出该矩阵存在9个特征值和9个对应的特征向量后，可以按照特征值从大到小的顺序对特征向量进行排序。而后，可以选取前8个特征向量进行组合，得到8×9(8行9列)的矩阵。之后，可以对该矩阵的转置确定为特征矩阵，该特征矩阵为9×8(9行8列)的矩阵。
步骤406，将第一像素矩阵与特征矩阵相乘，得到目标矩阵。
在本实施例中，对于插值后的目标图像中的每一个像素，上述执行主体可以将该像素的第一像素矩阵与该像素对应的特征矩阵相乘，得到该像素对应的目标矩阵。作为示例，第一像素矩阵为1×9(1行9列)的矩阵，特征矩阵为9×8(9行8列)的矩阵，两矩阵相乘后，得到1×8(1行8列)的目标矩阵。
由此，利用主成分分析的方式对第一像素矩阵进行处理，对第一像素矩阵进行降维，可以保留各个像素的第一像素矩阵中的重要的特征，从而使不同的第一像素矩阵的差异更为明显，因而可以实现更加准确地对插值后的目标图像内像素的分类。
步骤407，对于插值后的目标图像中的像素，将该像素对应的目标矩阵与预设矩阵进行点乘运算，从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。
在本实施例中，对于插值后的目标图像中的每一个像素，将该像素对应的目标矩阵与预设矩阵进行点乘运算，从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。
在本实施例中，对于步骤407中所述的滤波器集合，其生成步骤可以参照图5。图5给出了根据本申请的用于生成滤波器集合的方法的一个实施例的流程图。该用于生成滤波器集合的方法500，包括以下步骤：
步骤501，提取高分辨率图像样本集合，对高分辨率图像样本集 合中的高分辨率图像样本依次进行下采样和插值。
此处，对于上述高分辨率图像样本集合中的每一个高分辨率样本，均可以先对其进行下采样，之后对下采样后的高分辨率样本插值。
此处，下采样的倍数可以预先设定。例如，对高分辨率图像样本进行2倍下采样，可以是把高分辨率图像样本中2×2的图像块内的图像转换为一个像素，该像素的像素值等于2×2的图像块中的所有像素的像素值的均值。此处，可以采用与步骤401中相同的插值方式进行插值，此处不再赘述。
步骤502，对于插值后的高分辨率图像样本中的像素，提取以该像素为中心的第三像素矩阵，对第三像素矩阵进行主成分分析，得到目标矩阵样本。
此处，对于插值后的高分辨率图像样本中的每一个像素，可以提取以该像素为中心的第三像素矩阵，对上述第三像素矩阵进行主成分分析，得到目标矩阵样本。需要说明的是，对插值后的高分辨率图像样本中的像素进行第三像素矩阵提取步骤与步骤402所述的对插值后的目标图像进行第一像素矩阵的提取步骤基本相同；对第三像素矩阵进行主成分分析得到目标矩阵样本的步骤与步骤403-步骤406的操作基本相同，此处不再赘述。
在一些可选的实现方式中，第一像素矩阵的行数、列数可以分别与第三像素矩阵的行数、列数相同。作为示例，第一像素矩阵、第三像素矩阵均可以为1×9的矩阵。
步骤503，对所得到的目标矩阵样本进行分类。
此处，可以将所得到的各个目标矩阵样本代入到预设的公式或者函数进行计算，得到计算结果(例如数值)，将相同计算结果对应的目标矩阵样本划分为同一类。每一个类别可以用一个计算结果来表征。
在一些可选的实现方式中，可以首先将所得到的各个目标矩阵样本与预设矩阵进行点乘运算。之后，将点乘运算结果相同的目标矩阵样本划分为一类。
步骤504，训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合。
此处，对于每一类目标矩阵样本，可以利用机器学习方法进行该类对应的滤波器的训练。此处的每一个滤波器可以是一个参数向量。
在一些可选的实现方式中，对于每一类目标矩阵样本对应的、插值后的高分辨率图像样本中的像素，可以首先提取以该像素为中心的第四像素矩阵，将上述第四像素矩阵作为输入，将该像素对应的高分辨率像素作为输出，利用机器学习方法训练得到与该类目标矩阵样本对应的滤波器。此处，第四像素矩阵的提取步骤与步骤203中提取第二像素矩阵的步骤基本相同，此处不再赘述。
在一些可选的实现方式中，还可以建立点乘运算结果与滤波器的对应关系。作为示例，可以以点乘运算结果为键值对的键，以用于表征滤波器的参数向量或者参数矩阵为键值对的值，利用键值对形式表征点乘运算结果与滤波器的对应关系。
由此，通过主成分分析技术对像素进行降维，进而进行分类，可以保留各个像素的重要的特征，从而使不同的像素的差异更为明显，因而可以使像素分类更加准确。在此基础上进行滤波器训练，可以提高滤波器的针对性和准确性。
回到用于生成图像的方法的流程400，在步骤407选取滤波器后，继续参见如下步骤：
步骤408，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值。
在本实施例中，对于插值后的目标图像中的每一个像素，上述执行主体可以提取以该像素为中心的第二像素矩阵。之后，利用选取的滤波器对该像素对应的第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值。
在一些可选的实现方式中，第二像素矩阵的行数、列数可以分别与第四像素矩阵的行数、列数相同。作为示例，第二像素矩阵、第四像素矩阵均可以为1×49的矩阵。
步骤409，将所得到的高分辨率像素值进行汇总，生成高分辨率图像。
在本实施例中，由于插值后的目标图像中的每一个像素均可以计 算得到一个对应的高分辨率像素值，因此，上述执行主体可以将所得到的各个像素对应的高分辨率像素值进行汇总，生成高分辨率图像。
从图4中可以看出，与图2对应的实施例相比，本实施例中的用于生成图像的方法的流程400突出了滤波器集合的生成步骤。由此，本实施例描述的方案，通过主成分分析技术对像素进行降维分类，可以保留各个像素的重要的特征，从而使不同的像素的差异更为明显，因而可以使像素分类更加准确。在此基础上进行滤波器训练，可以提高滤波器的针对性和准确性。利用所训练的滤波器进行高分辨率图像的生成，进一步提升了高分辨率图像生成的效果。
进一步参考图6，作为对上述各图所示方法的实现，本申请提供了一种用于生成图像的装置的一个实施例，该装置实施例与图2所示的方法实施例相对应，该装置具体可以应用于各种电子设备中。
如图6所示，本实施例所述的用于生成图像的装置600包括：插值单元601，被配置成对目标图像进行插值；分析单元602，被配置成对于插值后的目标图像中的像素，提取以该像素为中心的第一像素矩阵，对上述第一像素矩阵进行主成分分析，得到目标矩阵；选取单元603，被配置成对于插值后的目标图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对上述第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值；生成单元604，被配置成将所得到的高分辨率像素值进行汇总，生成高分辨率图像。
在本实施例的一些可选的实现方式中，上述滤波器集合可以通过如下步骤生成：提取高分辨率图像样本集合，对上述高分辨率图像样本集合中的高分辨率图像样本依次进行下采样和插值；对于插值后的高分辨率图像样本中的像素，提取以该像素为中心的第三像素矩阵，对上述第三像素矩阵进行主成分分析，得到目标矩阵样本；对所得到的目标矩阵样本进行分类；训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合。
在本实施例的一些可选的实现方式中，在上述滤波器集合的生成 步骤中，上述对所得到的目标矩阵样本进行分类，可以包括：将所得到的目标矩阵样本与预设矩阵进行点乘运算；将点乘运算结果相同的目标矩阵样本划分为一类。
在本实施例的一些可选的实现方式中，上述滤波器集合的生成步骤中的训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合，可以包括：对于每一类目标矩阵样本对应的、插值后的高分辨率图像样本中的像素，提取以该像素为中心的第四像素矩阵，将上述第四像素矩阵作为输入，将该像素对应的高分辨率像素作为输出，利用机器学习方法训练得到与该类目标矩阵样本对应的滤波器。
在本实施例的一些可选的实现方式中，第一像素矩阵的行数、列数可以分别与第三像素矩阵的行数、列数相同，第二像素矩阵的行数、列数可以分别与第四像素矩阵的行数、列数相同。
在本实施例的一些可选的实现方式中，该装置还可以包括建立单元(图中未示出)。其中，上述建立单元可以被配置成建立点乘运算结果与滤波器的对应关系。
在本实施例的一些可选的实现方式中，上述分析单元602可以包括第一确定模块、第二确定模块、组成模块和相乘模块(图中未示出)。其中，上述第一确定模块可以被配置成确定上述第一像素矩阵的协方差矩阵。上述第二确定模块可以被配置成确定上述协方差矩阵的特征值和特征向量。上述组成模块可以被配置成从所确定的特征值中选取目标特征值，将上述目标特征值对应的特征向量组成特征矩阵。上述相乘模块可以被配置成将上述第一像素矩阵与上述特征矩阵相乘，得到目标矩阵。
在本实施例的一些可选的实现方式中，上述选取单元603可以进一步被配置成：对于插值后的目标图像中的像素，将该像素对应的目标矩阵与预设矩阵进行点乘运算，从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。
本申请的上述实施例提供的装置，通过插值单元601对目标图像进行插值，而后分析单元602利用主成分分析方式对插值后的目标图 像中的每一像素的第一像素矩阵进行处理，得到该像素对应的目标矩阵，之后选取单元603利用基于该目标矩阵所选取的滤波器对该像素的第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值，从而得到与插值后的目标图像中的各个像素对应的高分辨率像素值，最后生成单元604汇总生成高分辨率图像。由此，在高分辨率图像的生成过程中，利用主成分分析方式确定每一像素的目标矩阵，可以保留各个像素的第一像素矩阵中的重要的特征，从而使不同的第一像素矩阵的差异更为明显，因而利用该方式所确定目标矩阵可以更加准确地选取各像素对应的滤波器，从而提升了高分辨率图像生成的效果。
下面参考图7，其示出了适于用来实现本申请实施例的电子设备的计算机系统700的结构示意图。图7示出的电子设备仅仅是一个示例，不应对本申请实施例的功能和使用范围带来任何限制。
如图7所示，计算机系统700包括中央处理单元(CPU)701，其可以根据存储在只读存储器(ROM)702中的程序或者从存储部分708加载到随机访问存储器(RAM)703中的程序而执行各种适当的动作和处理。在RAM 703中，还存储有系统700操作所需的各种程序和数据。CPU 701、ROM 702以及RAM 703通过总线704彼此相连。输入/输出(I/O)接口705也连接至总线704。
以下部件连接至I/O接口705：包括键盘、鼠标等的输入部分706；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分707；包括硬盘等的存储部分708；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分709。通信部分709经由诸如因特网的网络执行通信处理。驱动器710也根据需要连接至I/O接口705。可拆卸介质711，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器710上，以便于从其上读出的计算机程序根据需要被安装入存储部分708。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程 序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信部分709从网络上被下载和安装，和/或从可拆卸介质711被安装。在该计算机程序被中央处理单元(CPU)701执行时，执行本申请的方法中限定的上述功能。需要说明的是，本申请所述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本申请中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本申请中，计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、RF等等，或者上述的任意合适的组合。
附图中的流程图和框图，图示了按照本申请各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。 例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本申请实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。所描述的单元也可以设置在处理器中，例如，可以描述为：一种处理器包括插值单元、分析单元、选取单元和生成单元。其中，这些单元的名称在某种情况下并不构成对该单元本身的限定，例如，插值单元还可以被描述为“对目标图像进行插值的单元”。
作为另一方面，本申请还提供了一种计算机可读介质，该计算机可读介质可以是上述实施例中描述的装置中所包含的；也可以是单独存在，而未装配入该装置中。上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该装置执行时，使得该装置：对目标图像进行插值；对于插值后的目标图像中的像素，提取以该像素为中心的第一像素矩阵，对该第一像素矩阵进行主成分分析，得到目标矩阵；对于插值后的目标图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对该第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值；将所得到的高分辨率像素值进行汇总，生成高分辨率图像。
以上描述仅为本申请的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本申请中所涉及的发明范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述发明构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本申请中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。

Claims (18)

  1. 一种用于生成图像的方法，包括：
    对目标图像进行插值；
    对于插值后的目标图像中的像素，提取以该像素为中心的第一像素矩阵，对所述第一像素矩阵进行主成分分析，得到目标矩阵；
    对于插值后的目标图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对所述第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值；
    将所得到的高分辨率像素值进行汇总，生成高分辨率图像。
  2. 根据权利要求1所述的用于生成图像的方法，其中，所述滤波器集合通过如下步骤生成：
    提取高分辨率图像样本集合，对所述高分辨率图像样本集合中的高分辨率图像样本依次进行下采样和插值；
    对于插值后的高分辨率图像样本中的像素，提取以该像素为中心的第三像素矩阵，对所述第三像素矩阵进行主成分分析，得到目标矩阵样本；
    对所得到的目标矩阵样本进行分类；
    训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合。
  3. 根据权利要求2所述的用于生成图像的方法，其中，所述对所得到的目标矩阵样本进行分类，包括：
    将所得到的目标矩阵样本与预设矩阵进行点乘运算；
    将点乘运算结果相同的目标矩阵样本划分为一类。
  4. 根据权利要求2所述的用于生成图像的方法，其中，所述训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤 波器集合，包括：
    对于每一类目标矩阵样本对应的、插值后的高分辨率图像样本中的像素，提取以该像素为中心的第四像素矩阵，将所述第四像素矩阵作为输入，将该像素对应的高分辨率像素作为输出，利用机器学习方法训练得到与该类目标矩阵样本对应的滤波器。
  5. 根据权利要求4所述的用于生成图像的方法，其中，第一像素矩阵的行数、列数分别与第三像素矩阵的行数、列数相同，第二像素矩阵的行数、列数分别与第四像素矩阵的行数、列数相同。
  6. 根据权利要求4所述的用于生成图像的方法，其中，所述方法还包括：
    建立点乘运算结果与滤波器的对应关系。
  7. 根据权利要求1所述的用于生成图像的方法，其中，所述对所述第一像素矩阵进行主成分分析，得到目标矩阵，包括：
    确定所述第一像素矩阵的协方差矩阵；
    确定所述协方差矩阵的特征值和特征向量；
    从所确定的特征值中选取目标特征值，将所述目标特征值对应的特征向量组成特征矩阵；
    将所述第一像素矩阵与所述特征矩阵相乘，得到目标矩阵。
  8. 根据权利要求1所述的用于生成图像的方法，其中，所述对于插值后的目标图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，包括：
    对于插值后的目标图像中的像素，将该像素对应的目标矩阵与预设矩阵进行点乘运算，从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。
  9. 一种用于生成图像的装置，包括：
    插值单元，被配置成对目标图像进行插值；
    分析单元，被配置成对于插值后的目标图像中的像素，提取以该像素为中心的第一像素矩阵，对所述第一像素矩阵进行主成分分析，得到目标矩阵；
    选取单元，被配置成对于插值后的目标图像中的像素，基于该像素对应的目标矩阵，从预先生成的滤波器集合中选取滤波器，提取以该像素为中心的第二像素矩阵，利用选取的滤波器对所述第二像素矩阵进行卷积，得到与该像素对应的高分辨率像素值；
    生成单元，被配置成将所得到的高分辨率像素值进行汇总，生成高分辨率图像。
  10. 根据权利要求9所述的用于生成图像的装置，其中，所述滤波器集合通过如下步骤生成：
    提取高分辨率图像样本集合，对所述高分辨率图像样本集合中的高分辨率图像样本依次进行下采样和插值；
    对于插值后的高分辨率图像样本中的像素，提取以该像素为中心的第三像素矩阵，对所述第三像素矩阵进行主成分分析，得到目标矩阵样本；
    对所得到的目标矩阵样本进行分类；
    训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合。
  11. 根据权利要求10所述的用于生成图像的装置，其中，所述对所得到的目标矩阵样本进行分类，包括：
    将所得到的目标矩阵样本与预设矩阵进行点乘运算；
    将点乘运算结果相同的目标矩阵样本划分为一类。
  12. 根据权利要求10所述的用于生成图像的装置，其中，所述训练与每一类目标矩阵样本相对应的滤波器，将所训练的滤波器汇总为滤波器集合，包括：
    对于每一类目标矩阵样本对应的、插值后的高分辨率图像样本中的像素，提取以该像素为中心的第四像素矩阵，将所述第四像素矩阵作为输入，将该像素对应的高分辨率像素作为输出，利用机器学习方法训练得到与该类目标矩阵样本对应的滤波器。
  13. 根据权利要求12所述的用于生成图像的装置，其中，第一像素矩阵的行数、列数分别与第三像素矩阵的行数、列数相同，第二像素矩阵的行数、列数分别与第四像素矩阵的行数、列数相同。
  14. 根据权利要求12所述的用于生成图像的装置，其中，所述装置还包括：
    建立单元，被配置成建立点乘运算结果与滤波器的对应关系。
  15. 根据权利要求9所述的用于生成图像的装置，其中，所述分析单元包括：
    第一确定模块，被配置成确定所述第一像素矩阵的协方差矩阵；
    第二确定模块，被配置成确定所述协方差矩阵的特征值和特征向量；
    组成模块，被配置成从所确定的特征值中选取目标特征值，将所述目标特征值对应的特征向量组成特征矩阵；
    相乘模块，被配置成将所述第一像素矩阵与所述特征矩阵相乘，得到目标矩阵。
  16. 根据权利要求9所述的用于生成图像的装置，其中，所述选取单元进一步被配置成：
    对于插值后的目标图像中的像素，将该像素对应的目标矩阵与预设矩阵进行点乘运算，从预先生成的滤波器集合中选取与点乘运算结果相对应的滤波器。
  17. 一种电子设备，包括：
    一个或多个处理器；
    存储装置，其上存储有一个或多个程序，
    当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如权利要求1-8中任一所述的方法。
  18. 一种计算机可读介质，其上存储有计算机程序，其中，该程序被处理器执行时实现如权利要求1-8中任一所述的方法。
PCT/CN2018/116333 2018-06-26 2018-11-20 用于生成图像的方法和装置 WO2020000878A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810668219.7 2018-06-26
CN201810668219.7A CN108776954B (zh) 2018-06-26 2018-06-26 用于生成图像的方法和装置

Publications (1)

Publication Number Publication Date
WO2020000878A1 true WO2020000878A1 (zh) 2020-01-02

Family

ID=64025599

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/116333 WO2020000878A1 (zh) 2018-06-26 2018-11-20 用于生成图像的方法和装置

Country Status (2)

Country Link
CN (1) CN108776954B (zh)
WO (1) WO2020000878A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108776954B (zh) * 2018-06-26 2020-04-28 北京字节跳动网络技术有限公司 用于生成图像的方法和装置
CN111951167B (zh) * 2020-08-25 2021-05-18 深圳思谋信息科技有限公司 超分辨率图像重建方法、装置、计算机设备和存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101312529A (zh) * 2007-05-24 2008-11-26 华为技术有限公司 生成上下采样滤波器及实现编码的方法、系统和装置
CN102915527A (zh) * 2012-10-15 2013-02-06 中山大学 基于形态学成分分析的人脸图像超分辨率重建方法
EP3264741A1 (en) * 2016-06-30 2018-01-03 Thomson Licensing Plenoptic sub aperture view shuffling with improved resolution
CN108776954A (zh) * 2018-06-26 2018-11-09 北京字节跳动网络技术有限公司 用于生成图像的方法和装置

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10339633B2 (en) * 2015-11-04 2019-07-02 Peking University Shenzhen Graduate School Method and device for super-resolution image reconstruction based on dictionary matching
US10147167B2 (en) * 2015-11-25 2018-12-04 Heptagon Micro Optics Pte. Ltd. Super-resolution image reconstruction using high-frequency band extraction
CN105678697B (zh) * 2015-12-30 2018-10-12 北京工业大学 一种基于dct域本征变换的人脸图像超分辨率重建方法
CN106934766B (zh) * 2017-03-15 2020-04-21 西安理工大学 一种基于稀疏表示的红外图像超分辨率重建方法
CN107507134B (zh) * 2017-09-21 2020-09-11 大连理工大学 基于卷积神经网络的超分辨率方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101312529A (zh) * 2007-05-24 2008-11-26 华为技术有限公司 生成上下采样滤波器及实现编码的方法、系统和装置
CN102915527A (zh) * 2012-10-15 2013-02-06 中山大学 基于形态学成分分析的人脸图像超分辨率重建方法
EP3264741A1 (en) * 2016-06-30 2018-01-03 Thomson Licensing Plenoptic sub aperture view shuffling with improved resolution
CN108776954A (zh) * 2018-06-26 2018-11-09 北京字节跳动网络技术有限公司 用于生成图像的方法和装置

Also Published As

Publication number Publication date
CN108776954B (zh) 2020-04-28
CN108776954A (zh) 2018-11-09

Similar Documents

Publication Publication Date Title
US11734851B2 (en) Face key point detection method and apparatus, storage medium, and electronic device
US10853916B2 (en) Convolution deconvolution neural network method and system
WO2020000877A1 (zh) 用于生成图像的方法和装置
US20230206396A1 (en) Image super-resolution reconstructing
US9171226B2 (en) Image matching using subspace-based discrete transform encoded local binary patterns
WO2020062494A1 (zh) 图像处理方法和装置
US20220138500A1 (en) Unsupervised super-resolution training data construction
CN112990219B (zh) 用于图像语义分割的方法和装置
Zhang et al. Impact of deep learning-based image super-resolution on binary signal detection
WO2020000878A1 (zh) 用于生成图像的方法和装置
WO2022081226A1 (en) Dual-stage system for computational photography, and technique for training same
WO2017070841A1 (zh) 图像处理方法和装置
Witwit et al. Global motion based video super-resolution reconstruction using discrete wavelet transform
Wang et al. Deep arbitrary-scale image super-resolution via scale-equivariance pursuit
Wang et al. Image super-resolution using only low-resolution images
Rajput et al. OEINR-RFH: Outlier elimination based iterative neighbor representation for robust face hallucination
CN114926666A (zh) 一种图像数据处理方法及装置
CN113688928A (zh) 图像匹配方法、装置、电子设备和计算机可读介质
Huan et al. Remote sensing image reconstruction using an asymmetric multi-scale super-resolution network
Su et al. Single image super-resolution based on space structure learning
Hu et al. Single MR image super-resolution via mixed self-similarity attention network
JP2019125128A (ja) 情報処理装置、制御方法、及びプログラム
Tsai et al. EESRGAN: Efficient & Effective Super-Resolution Generative Adversarial Network
CN112581363A (zh) 图像超分辨率重建方法、装置、电子设备及存储介质
Zhang et al. Global context-aware feature modulation networks for unified multi-scale super-resolution

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18923867

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 03.05.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 18923867

Country of ref document: EP

Kind code of ref document: A1