WO2017070841A1 - Image processing method and apparatus - Google Patents

Abstract

Provided in the embodiments of the present invention are an image processing method and apparatus, the image processing method comprising: on the basis of image features, classifying an image sample set to obtain image sample subsets, the image sample subsets comprising high resolution image samples and corresponding low resolution image samples; on the basis of a first and a second target function and the low and high resolution image samples, acquiring a first and a second feature matrix and a mapping relationship matrix between the high and low resolution image samples corresponding to the image sample subsets; determining an image sample subset corresponding to an image to be processed; and, on the basis of the first target function and the first feature matrix, the second feature matrix, and the mapping relationship matrix between the high and low resolution image samples corresponding to the image sample subset corresponding to the image to be processed, acquiring a high resolution image corresponding to the image to be processed.

Description

Image processing method and device Technical field

Embodiments of the present invention relate to image processing technologies, and in particular, to an image processing method and apparatus.

Background technique

In modern society, people are increasingly demanding high-quality, high-definition image information. For example, in home digital media applications, 1080p and 4K x 2K TVs are becoming more and more popular. However, most TV programs and DVD-format videos are SD-format video, which requires super-resolution technology to standardize formats. Video is converted to high-definition or ultra-high-definition video; in addition, in network video applications, due to the bandwidth, many online videos are of poor quality, and they should be displayed on large-screen mobile terminals, computers or digital TVs. When you can use the super resolution technology. In addition, image super-resolution has important applications in medical imaging, remote sensing satellites and other fields.

According to the number of input low-resolution images, the super-resolution technology can be divided into two categories: super-resolution technology based on multi-frame image reconstruction and super-resolution based on single-frame image learning. Super-resolution technology based on single-frame image learning has wider applicability and flexibility than super-resolution technology based on multi-frame low-resolution images. A representative technical solution uses sparse coding to perform image super-resolution, which is to force the corresponding high-low resolution image blocks to share the same sparse representation: by performing sparse constraint before regularization, low-resolution image The block is regarded as an over-complete dictionary for encoding, and then the sparse expression coefficient is obtained. The high-resolution image block corresponding to the linear combination of the coefficients can be used to complete the image super-resolution reconstruction. However, the potential assumption that high and low resolution image blocks have the "same sparse representation" is difficult to achieve under practical conditions.

Summary of the invention

Embodiments of the present invention provide an image processing method and apparatus for improving high-resolution image reconstruction quality. With the image processing apparatus of the present invention or using the image processing method of the present invention, a corresponding output image having a higher resolution can be obtained for each input image.

In a first aspect, an embodiment of the present invention provides an image processing method, including: a sample training phase and an image reconstruction phase.

The sample training phase can be performed offline or online, usually in an offline manner.

During the sample training phase:

In the first step, the samples in the randomly obtained sample set may be referred to as high-resolution image samples, and the same low-resolution image samples are obtained by using the same image processing methods such as downsampling and blur. It should be understood that in order to obtain a better training effect, the samples within the randomly obtained sample set are more suitable for selecting a richly detailed image with a higher original resolution. The downsampling or blurring is a typical image processing method, and a degraded image is obtained by selectively discarding the information of the original image. For the present embodiment, the degraded image is a low resolution image. It should be understood that in order to achieve this, any well-known technique known to those skilled in the art can be selected, and is not limited to the downsampling or fuzzy image processing method.

In the second step, the sample set is divided into a plurality of sample subsets by using the same image feature as a classification criterion. It should be understood that the image samples in each subset have the same image characteristics, and the so-called image samples are a pair of image samples, ie high resolution image samples and their corresponding low resolution image samples. The so-called image features include visual features of images, statistical features, transform coefficient features, algebraic features, etc. Corresponding, feature extraction methods, including principal component analysis, support vector machines, etc., "Research on Image Feature Extraction Methods" (Digital Object Uniform Identifier (GOI): CNKI: CDMD: 2.2007.058439) The full text is introduced here. The first chapter and the second chapter of the paper give an example description of the image features and extraction methods. It should be understood that the image features and the extraction method of the present invention may be any well-known technologies known to those skilled in the art, such as the above-mentioned image features and extraction methods, and are not limited. Generally, this step is implemented by a clustering method. Clustering, also known as group analysis, is a statistical analysis technique that divides research objects into clusters of relatively homogeneous nature. Commonly used clustering methods include partitioning methods, hierarchical methods, density-based methods, grid-based methods, model-based methods, transformation-based methods, etc., "Cluster-based image classification and segmentation algorithms" (GOI) :CNKI: CDMD:2.1012.023680) The full text is introduced here. The second chapter to the fifth chapter of the paper give an example description of the clustering method. It should be understood that the clustering method of the present invention may be any well-known technology known to those skilled in the art, such as the above clustering method, and is not limited.

In the third step, for each sub-set, corresponding to each image feature, the high-resolution image samples and the low-resolution image samples are independently trained under the constraint of the first objective function, correspondingly, Obtaining a first feature matrix corresponding to the low-resolution image sample, which may be referred to as a low-resolution dictionary, obtains a second feature matrix corresponding to the high-resolution image sample, which may be referred to as a high-resolution dictionary. It should be understood that the training process for high resolution image samples and low resolution image samples is independent, in no particular order.

In some embodiments, the first objective function is min(||y-Dα|| _F +λ||α|| _F ), where y represents a low resolution image sample of the sample subset and D represents a sample subset The first characteristic matrix, α represents a matrix of expression coefficients corresponding to low-resolution image samples, ||·|| _F represents an exemplary operation, min represents a minimum operation, and λ is a regularization constant parameter; according to the first objective function and low And obtaining the first feature matrix corresponding to the subset of the image samples, comprising: iteratively updating D and α by a preset number of times under the constraint of the first objective function, to obtain a first feature matrix satisfying the first objective function.

In some embodiments, the first objective function is min(||y-Dα|| _F +λ||α|| _F ), where y represents a high resolution image sample of the sample subset and D represents a sample subset a second feature matrix, α represents a matrix of expression coefficients corresponding to high-resolution image samples, ||·|| _F represents an exemplary operation, min represents a minimum operation, and λ is a regularization constant parameter; correspondingly, according to the first target The function and the high-resolution image sample obtain a second feature matrix corresponding to the subset of the image samples, comprising: iteratively updating D and α by a preset number of times under the constraint of the first objective function, obtaining a second satisfying the first objective function Feature matrix.

It should be understood that because the training process of the high resolution image samples and the low resolution image samples is independent, the training parameters D and a that obtain the first feature matrix and obtain the second feature matrix may be different.

In this step, the training of the high-resolution image samples and the low-resolution image samples is performed independently, and the high-resolution dictionary and the low-resolution dictionary are independently obtained.

In the fourth step, for each subset, high-resolution and low-resolution image samples in the subset are encoded according to the high-resolution and low-resolution dictionary of the set image feature, and a high-resolution expression coefficient matrix is obtained. The low-resolution expression coefficient matrix is obtained, and the mapping relationship matrix between the high-low resolution image samples of the image sample subset is obtained by the high-resolution expression coefficient matrix and the low-resolution expression coefficient matrix.

In some embodiments, according to the first feature matrix, encoding the low resolution image samples under the constraint of the first objective function, obtaining a first expression coefficient matrix, ie, a low resolution expression coefficient matrix; according to the second feature matrix, Under the constraint of the first objective function, the high-resolution image samples are encoded to obtain a second expression coefficient matrix, that is, a high-resolution expression coefficient matrix; in the second objective function

Under the constraint, obtain a mapping relationship matrix, where α _l represents the first expression coefficient matrix, α _h represents the second expression coefficient matrix, and M represents the mapping relationship matrix,

Represents a full 1 matrix, ||·|| _{F table} demonstrates the number operation, and min represents the minimum value operation.

At this point, the sample training phase is completed, and each sample subset is trained, that is, a high-resolution dictionary, a low-resolution dictionary, and a mapping matrix between high- and low-resolution image samples of each image feature. The parameters obtained by the above training will be applied to the image reconstruction stage.

In the image reconstruction phase:

In the first step, determining a subset of image samples corresponding to the image to be processed, that is, determining image features of the image to be processed, for selecting a suitable high-resolution dictionary, a low-resolution dictionary, and a mapping matrix between high- and low-resolution image samples. deal with.

In some embodiments, the step may be decomposed into image features for extracting an image to be processed; comparing image features of the image to be processed with image features of each subset of image samples in the image sample set; determining image samples corresponding to the image to be processed Subsets are the smallest subset of samples.

In the sample training phase, the image feature extraction and clustering correlation methods have been detailed. In order to obtain a better classification effect, it is more suitable to determine the image sample subset corresponding to the image to be processed by using the method consistent with the sample training phase.

In the second step, the mapping relationship matrix between the high-resolution dictionary, the low-resolution dictionary, and the high-low resolution image samples to which the image sample subset of the image to be processed belongs is applied to the image to be processed, and the first stage used in the sample training phase is used. An objective function obtains a high resolution image corresponding to the image to be processed.

In some embodiments, according to the first feature matrix, that is, the low-resolution dictionary, under the constraint of the first objective function, the image to be processed is encoded, and the third expression coefficient matrix corresponding to the image to be processed is obtained, that is, the low-resolution expression coefficient. a matrix; according to the mapping relationship matrix and the third expression coefficient matrix, under the constraint of the second objective function, obtaining a fourth expression coefficient matrix corresponding to the image to be processed, that is, a high-resolution expression coefficient matrix; a fourth expression coefficient matrix and the second The feature matrix, that is, the high-resolution dictionary multiplication, obtains high-frequency components of the image to be processed; the high-frequency component and the enlarged image to be processed are added to obtain a high-resolution image corresponding to the image to be processed.

In a second aspect, an embodiment of the present invention provides an image processing apparatus, including:

a first classification module, configured to classify the image sample set according to the image feature, to obtain a plurality of image sample subsets, wherein the image sample subset includes a high resolution image sample and a low resolution image sample, the low resolution image The sample is obtained by downsampling the high resolution image sample; a first obtaining module, configured to obtain an image sample according to the first objective function and the low resolution image sample a first feature matrix corresponding to the subset; a second acquiring module, configured to obtain a second feature matrix corresponding to the subset of image samples according to the first objective function and the high-resolution image sample; and a third acquiring module, configured to The feature matrix, the second feature matrix, the low resolution image sample, the high resolution image sample, obtain a mapping relationship matrix between the high and low resolution image samples of the image sample subset; the second classification module is configured to determine the corresponding image to be processed a fourth image acquisition module, configured to: according to the first objective function and the determined mapping relationship between the first feature matrix, the second feature matrix, and the high-low resolution image samples corresponding to the image sample subset corresponding to the image to be processed A matrix that obtains a high resolution image corresponding to the image to be processed.

In a third aspect, an embodiment of the present invention provides an apparatus for processing an image, the apparatus comprising a processor configured to: the operation is the method of operation described in the first aspect.

In a fourth aspect, an embodiment of the present invention provides a computer readable storage medium storing instructions that, when executed, are used by one or more processors of a device that processes an image to perform the following operations: The operation is the method of operation described in the first aspect.

The technical solution described in the present invention can also be used for image deblurring processing. Specifically, with the image processing apparatus of the present invention or the image processing method of the present invention, a corresponding relatively clear output image can be obtained for each blurred input image.

In a fifth aspect, an embodiment of the present invention provides an image processing method, including:

Sorting the image sample set according to the image feature to obtain a plurality of image sample subsets, wherein the image sample subset includes a clear image sample and a blurred image sample, wherein the blurred image sample is obtained by blurring the clear image sample; Obtaining a first feature matrix corresponding to the image sample subset according to the objective function and the blurred image sample; obtaining a second feature matrix corresponding to the image sample subset according to the first objective function and the clear image sample; according to the first feature matrix, the second Feature matrix, blurred image sample, clear image sample, obtaining a clear relationship between the image sample subset and a mapping relationship matrix between the blurred image samples; determining a subset of image samples corresponding to the image to be processed; according to the first objective function and the determined image to be processed Corresponding image feature subsets corresponding to the first feature matrix, the second feature matrix, and the mapping matrix between the clear and blurred image samples, to obtain a clear image corresponding to the image to be processed.

In a sixth aspect, an embodiment of the present invention provides an image processing apparatus, including:

a first classification module, configured to classify the image sample set according to the image feature, to obtain a plurality of image sample subsets, wherein the image sample subset includes a clear image sample and a blurred image sample, and the blurred image sample passes the clear Obtaining image samples; first acquisition module, for Obtaining, according to the first objective function and the blurred image sample, a first feature matrix corresponding to the subset of image samples; and a second acquiring module, configured to obtain a second feature matrix corresponding to the subset of image samples according to the first objective function and the clear image sample a third obtaining module, configured to obtain a mapping relationship matrix between the clear and blurred image samples of the image sample subset according to the first feature matrix, the second feature matrix, the blurred image sample, and the clear image sample; the second classification module uses Determining a subset of image samples corresponding to the image to be processed; a fourth acquiring module, configured to: according to the first objective function and the determined image feature subset corresponding to the image to be processed, the first feature matrix, the second feature matrix, and the clear The mapping relationship matrix between the blurred image samples obtains a clear image corresponding to the image to be processed.

In a seventh aspect, an embodiment of the present invention provides an apparatus for processing an image, the apparatus including a processor configured to: the operation is the operation method described in the fifth aspect.

In an eighth aspect, an embodiment of the present invention provides a computer readable storage medium storing instructions that, when executed, are used by one or more processors of a device that processes an image to perform the following operations: The operation is the method of operation described in the fifth aspect.

The technical solution described in the present invention can also be applied to recovery processing of other types of image degradation. In particular, with the image processing apparatus of the present invention or using the image processing method of the present invention, a corresponding de-degraded output image can be obtained for each degraded input image.

The above various aspects and implementations of the present invention will be described in detail in the Detailed Description.

DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief description of the drawings used in the embodiments or the prior art description will be briefly described below. Obviously, the drawings in the following description It is a certain embodiment of the present invention, and other drawings can be obtained from those skilled in the art without any inventive labor.

1 is a schematic flowchart of an embodiment of an image processing method according to the present invention;

2 is a schematic diagram of a manner of extracting an image block according to the present invention;

Figure 3 is a schematic block diagram of an embodiment of an image processing apparatus of the present invention;

4 is a schematic block diagram of another embodiment of an image processing apparatus according to the present invention;

detailed description

The technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. It is a partial embodiment of the invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

1 is a flowchart of Embodiment 1 of an image processing method according to the present invention. As shown in FIG. 1, the method 1000 of this embodiment includes:

S1100: classify an image sample set according to image features, to obtain a subset of image samples, wherein the image sample subset includes high-resolution image samples and corresponding low-resolution image samples;

One possible way is:

Select multiple high-resolution images with rich texture details to form a training library.

The same blur and down sampling operation is performed on each of the image high resolution images X in the training library to generate a corresponding low frequency image Y with high frequency detail loss, and the low resolution image Y is enlarged to a high resolution by interpolation. Images get the same size

It should be understood that the specific downsampling operation method for generating the low resolution image Y here is not limited, and a high resolution image is generated.

The specific interpolation method is not limited.

Subtracting the high resolution image X from its corresponding low resolution image interpolation and magnifying image

Get high frequency detail map

The high frequency detail is the extracted high resolution image feature.

One or two stepwise operator pairs in horizontal and vertical directions

Filtering is performed to obtain four filtered images, and the first and second step information is the extracted low-resolution image features.

High frequency detail image

Perform sufficient sampling to collect N sizes as

The image block, where n is the width of the image block, and N is a positive integer, which can be preset and not limited.

The same image block size is sampled at the same position of the four images generated by the filtering, and the acquisition is completed, and the training sample set can be obtained.

Where y _i represents a column vector jointly formed by four image blocks acquired at the same position on the four filtered images, and x _i represents a column vector formed by the image block acquired on the high-frequency detail image at the corresponding position.

Low resolution image block by clustering algorithm

Gathered into K class, K cluster centers are

According to the clustering result, according to the correspondence between the high and low resolution image blocks

Divided into the corresponding categories, thus generating a subset of each sample.

It should be understood that the above clustering algorithm has classified image blocks having the same feature into one class, and does not limit a specific clustering algorithm.

S1200. Obtain a first feature matrix corresponding to the subset of image samples according to the first objective function and the low resolution image sample.

Specifically, the first objective function is min(||y-Dα|| _F +λ||α|| _F ), where y represents a low-resolution image sample of the sample subset, and D represents the sample a first feature matrix of the subset, α represents a matrix of expression coefficients corresponding to the low-resolution image samples, ||·|| _F represents an exemplary operation, min represents a minimum operation, and λ is a regularization constant parameter;

Correspondingly, the obtaining, according to the first objective function and the low-resolution image sample, the first feature matrix corresponding to the subset of image samples, comprising: preset times under the constraint of the first objective function The D and α are iteratively updated to obtain the first feature matrix that satisfies the first objective function.

One possible way is:

First step, low resolution image block for the ith subset

Perform principal component analysis, extract the principal component of the first m-dimensional to obtain the matrix P _l ⁱ , and use P _l ⁱ to initialize the low-resolution dictionary in this embodiment.

Among them, principal component analysis refers to a multivariate statistical analysis method that uses multiple variables to linearly transform to select fewer important variables.

Second step, fixed

by

Update α _l , equivalent, calculate

The third step is to fix α _l by

Update

Equivalent

The fourth step, after iterating the second step and the third step N times, obtaining a low resolution dictionary

Where N is a positive integer and is not limited.

It should be understood that, in this embodiment, the norm constraint of the L2 norm is used as the first objective function, and the L1 norm is also used as the norm constraint of the first objective function, that is, the objective function is min(||y-Dα| | ₁ +λ||α|| ₁ ), not limited. Among them, the L2 norm refers to the square root of the sum of the squares of the elements of the vector, and the L1 norm refers to the sum of the absolute values of the elements in the vector.

It should be understood that the first step in this embodiment uses the principal component of the m-dimensional pre-mesh of the low-resolution image block set to obtain the matrix P _l ⁱ as the initial value of the low-resolution dictionary, and then trains to obtain an under-complete dictionary, which may also be adopted. Correct

The principal component analysis is performed to obtain the orthogonal matrix P _l ⁱ as the initial value of the low-resolution dictionary, and then the training dictionary is obtained, which is not limited.

S1300. Obtain a second feature matrix corresponding to the subset of image samples according to the first objective function and the high resolution image sample.

Specifically, the first objective function is min(||y-Dα|| _F +λ||α|| _F ), where y represents a high-resolution image sample of the sample subset, and D represents the sample a second feature matrix of the subset, α represents a matrix of expression coefficients corresponding to the high-resolution image samples, ||·|| _F represents an exemplary operation, min represents a minimum operation, and λ is a regularization constant parameter;

Correspondingly, the obtaining, according to the first objective function and the high-resolution image sample, the second feature matrix corresponding to the subset of image samples, comprising: pre-constraining under the constraint of the first objective function The set number of iterations updates D and α to obtain the second feature matrix that satisfies the first objective function.

The implementation of the S1300 is similar to that of the S1200, and is not described here.

It should be understood that the S1300 and S1200 are completely independent processes, and the training of the low-resolution dictionary and the training of the high-resolution dictionary are completely independent, and there is no order, and it can also be performed in parallel.

By independently training and obtaining low-resolution and high-resolution dictionaries, the constraints on the same sparse representation are appropriately relaxed, and the sample information can be more flexibly utilized, thereby improving the accuracy of generating high-frequency components of the image by expressing coefficients, thereby increasing the accuracy. Resolution image reconstruction quality.

S1400. Obtain a mapping relationship between high and low resolution image samples of the subset of image samples according to the first feature matrix, the second feature matrix, the low resolution image sample, and the high resolution image sample. matrix;

Specifically, according to the first feature matrix, encoding the low resolution image sample to obtain a first expression coefficient matrix under the constraint of the first objective function; according to the second feature matrix, in the Encoding the high resolution image sample to obtain a second expression coefficient matrix under the constraint of an objective function; in the second objective function

Obtaining the mapping relationship matrix, wherein α _l represents the first expression coefficient matrix, α _h represents the second expression coefficient matrix, and M represents the mapping relationship matrix,

Represents a full 1 matrix, ||·|| _{F table} demonstrates the number operation, and min represents the minimum value operation.

One possible way is:

a first step, according to the first feature matrix

Encoding the low resolution image sample y _i under the constraint of the first objective function to obtain a first expression coefficient matrix α _l , equivalent, calculated by the following formula

a second step, according to the second feature matrix

Encoding the high resolution image sample y _i under the constraint of the first objective function, obtaining a second expression coefficient matrix α _h , equivalent, calculated by the following formula

It should be understood that the regular parameter constant λ in the second step and the λ in the first step may be the same or different and are not limited.

The third step, in the second objective function

Obtaining the mapping relationship matrix, wherein α _l represents the first expression coefficient matrix, α _h represents the second expression coefficient matrix, and M represents the mapping relationship matrix,

Represents an all-one matrix,

Indicates the L2 norm operation, and min denotes the minimum value operation.

S1500: Determine a subset of image samples corresponding to the image to be processed;

Specifically, extracting image features of the image to be processed; comparing image features of the image to be processed with image features of each subset of image samples in the image sample set; and determining image samples corresponding to the image to be processed The subset is the subset of samples with the least difference.

One possible way is:

Enlarge the low resolution image Y by interpolation to obtain an image

image

The resolution is the resolution of the target high resolution image.

One or two stepwise operator pairs in horizontal and vertical directions

Filtering is performed to obtain four filtered images.

For each filtered image, start from the upper left corner and from left to right in the raster scan order, and extract the size from top to bottom.

The image block, the starting point of the extracted image block starts from the last s pixel of the previous adjacent image block, where n is the width of the image block, and s is a positive integer, which can be preset, not limited, and FIG. 2 shows the image block. Schematic diagram of the starting position of the extraction.

Combining image blocks extracted corresponding to each image in the four filtered images into a column vector y _i ,

i = 1, 2, ..., N _t , where N _t is the total number of image blocks extracted from the input image.

According to the objective function

Calculating the image feature of the current image feature y _i and each image sample subset, that is, the distance between the cluster centers c _j , and taking the image sample subset k to which the cluster center closest to the cluster belongs, as the image sample corresponding to the image to be processed Subcollection.

It should be understood that other clustering algorithms or constraints may be selected in this step, which are not limited. Feasibly, the clustering algorithm in this step is consistent with the clustering algorithm used in S1100.

S1600. Obtain the image to be processed according to the mapping relationship matrix between the first feature matrix, the second feature matrix, and the high-low resolution image samples corresponding to the first target function and the image sample subset corresponding to the image to be processed. Corresponding high resolution image;

Specifically, according to the first feature matrix, under the constraint of the first objective function, encoding the image to be processed, acquiring a third expression coefficient matrix corresponding to the image to be processed; according to the mapping relationship matrix and Obtaining, according to the second objective function, a fourth expression coefficient matrix corresponding to the image to be processed; the fourth expression coefficient matrix and the second feature matrix are multiplied, Obtaining a high frequency component of the image to be processed; adding the high frequency component and the enlarged image to be processed to obtain a high resolution image corresponding to the image to be processed.

One possible way is:

In the objective function

Under the constraint, the expression coefficient α of the image to be processed y is calculated according to the low-resolution dictionary D. This step is consistent with S1200, and the specific calculation process is also consistent with S1200, and will not be described again.

Since the training process of the high-low resolution dictionary is no longer coupled, the expression coefficient of the training image and the expression coefficient of the image can be used, the same objective function can be used, and the accuracy of generating high-frequency components of the image by the expression coefficient is improved, thereby improving the high resolution. Image reconstruction quality.

In the objective function

Under the constraint, according to the mapping relationship matrix M, the expression coefficient α _{l of the} image to be processed, the expression coefficient α _h corresponding to the target high-resolution image is calculated, which is equivalent,

It should be understood that, in this embodiment, the expression coefficient corresponding to the target high-resolution image is calculated, and a linear regression algorithm is used, and vector regression (SVR), Ridge Regression, or other nonlinear regression algorithms may also be used, without limitation. .

Α _h and D _h The expression dictionary high resolution coefficient corresponding to a high resolution image, the high-frequency component calculation image to be processed x _h, i.e. x _h = D _{h α h.}

The high-frequency component of the image to be processed refers to the to-be-processed image after being enlarged by an algorithm such as upsampling. A digital representation of the image detail portion that is missing relative to the target high resolution image.

After obtaining the high-frequency components of all the extracted image blocks, the high-frequency components of the image to be processed are arranged in the order in which the corresponding image blocks are extracted, and the overlapping portions are averaged as the high-frequency components of the target high-resolution image. Image X _h .

According to the enlarged image to be processed

And the high-frequency component image X _h to obtain the target high-resolution image X, that is,

In order to better prove the advantages of the embodiments of the present invention over the prior art, the method using the embodiment of the present invention and the method in the prior art are tested, and the results are shown in Table 1.

Table 1 simulation results

It can be seen from Table 1 that the image processing method according to the embodiment of the present invention is superior to other prior art reconstruction effects.

In the embodiment of the present invention, by independently training and obtaining a low-resolution and high-resolution dictionary, the constraints on the same sparse representation are appropriately relaxed, and the sample information can be utilized more flexibly, and at the same time, the solution of the training process of the high-low resolution dictionary is obtained. Coupling, the process of reconstructing the image and the process of sample training can be completely consistent. The expression coefficient of the training image and the expression coefficient of the image generated during the reconstruction phase use the same objective function, the same calculation and derivation process, and improve the reliability of the training result. Sex, which improves the accuracy of generating high-frequency components of images by expression coefficients and improves the quality of high-resolution image reconstruction.

In another embodiment of the present invention, the present invention is also applicable to image deblurring processing, specifically:

With respect to the training sample set including the high resolution image samples and the corresponding low resolution image samples, the training sample set for image deblurring processing includes clear image samples and corresponding blurred image samples. In a specific implementation, the clear image corresponds to the high resolution image, the blurred image corresponds to the low resolution image, and the process of obtaining the clear image from the blurred image corresponds to the process of obtaining the high resolution image from the low resolution image. The method is the same as that of Embodiment 1, and will not be described again.

FIG. 3 is a block diagram of a second embodiment of an image processing apparatus according to the present invention. As shown in FIG. 3, the apparatus 10 of the embodiment includes:

The first classification module 11 is configured to classify the image sample set according to the image feature to obtain a subset of the image samples, wherein the image sample subset includes the high resolution image sample and the corresponding low resolution image sample;

a first acquiring module 12, configured to obtain, according to the first objective function and the low-resolution image sample, a first feature matrix corresponding to the subset of image samples;

The second obtaining module 13 is configured to obtain, according to the first objective function and the high-resolution image sample, a second feature matrix corresponding to the subset of image samples;

The third obtaining module 14 is configured to obtain high and low resolutions of the subset of image samples according to the first feature matrix, the second feature matrix, the low resolution image sample, and the high resolution image sample. a mapping relationship matrix between image samples;

a second classification module 15 configured to determine a subset of image samples corresponding to the image to be processed;

The fourth obtaining module 16 is configured to: according to the first objective function and the mapping relationship matrix between the first feature matrix, the second feature matrix, and the high and low resolution image samples corresponding to the image sample subset corresponding to the image to be processed, Obtaining a high resolution image corresponding to the image to be processed.

It should be understood that the image processing apparatus 10 according to an embodiment of the present invention may correspond to the method 1000 of performing image processing in the embodiment of the present invention, and the above and other operations and/or functions of the respective modules in the image processing apparatus 10 are respectively implemented for The corresponding processes of the respective methods in FIG. 1 are not described herein for the sake of brevity.

In the embodiment of the present invention, by independently training and obtaining a low-resolution and high-resolution dictionary, the constraints on the same sparse representation are appropriately relaxed, and the sample information can be utilized more flexibly, and at the same time, the solution of the training process of the high-low resolution dictionary is obtained. Coupling, the process of reconstructing the image and the process of sample training can be completely consistent. The expression coefficient of the training image and the expression coefficient of the image generated during the reconstruction phase make Using the same objective function, the same calculation and derivation process, the reliability of the training result is improved, and the accuracy of generating high-frequency components of the image by the expression coefficient is improved, and the quality of high-resolution image reconstruction is improved.

4 is a block diagram of a third embodiment of an image processing apparatus according to the present invention. As shown in FIG. 4, the apparatus 20 of the present embodiment includes a processor 21, a memory 22, and a bus system 23. The processor 21 and the memory 22 are connected by a bus system 23 for storing instructions for executing instructions stored by the memory 22. The memory 22 of the image processing device 20 stores the program code, and the processor 21 can call the program code stored in the memory 22 to perform an operation of classifying the image sample set according to the image feature to obtain a subset of the image samples, wherein the image sample subset Include a high resolution image sample and a corresponding low resolution image sample; obtain a first feature matrix corresponding to the image sample subset according to the first objective function and the low resolution image sample; according to the first objective function and the high resolution image sample Obtaining a second feature matrix corresponding to the image sample subset; obtaining mapping between high and low resolution image samples of the image sample subset according to the first feature matrix, the second feature matrix, the low resolution image sample, and the high resolution image sample a relationship matrix; determining a subset of image samples corresponding to the image to be processed; and mapping between the first feature matrix, the second feature matrix, and the high-low resolution image samples corresponding to the image object subset corresponding to the image to be processed according to the first objective function Matrix, obtaining the high resolution corresponding to the image to be processed Like.

It should be understood that, in the embodiment of the present invention, the processor 21 may be a central processing unit ("CPU"), and the processor 21 may also be other general-purpose processors and digital signal processors (DSPs). , an application specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, and the like. The general purpose processor may be a microprocessor or the processor or any conventional processor or the like.

The memory 22 can include read only memory and random access memory and provides instructions and data to the processor 21. A portion of the memory 22 may also include a non-volatile random access memory. For example, the memory 22 can also store information of the device type.

The bus system 23 may include a power bus, a control bus, a status signal bus, and the like in addition to the data bus. However, for clarity of description, various buses are labeled as the bus system 23 in the figure.

In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 21 or an instruction in the form of software. The steps of the method disclosed in connection with the embodiments of the present invention may be It is implemented directly as a hardware processor, or by a combination of hardware and software modules in the processor. The software module can be located in a conventional storage medium such as random access memory, flash memory, read only memory, programmable read only memory or electrically erasable programmable memory, registers, and the like. The storage medium is located in the memory 22, and the processor 21 reads the information in the memory 22 and combines the hardware to perform the steps of the above method. To avoid repetition, it will not be described in detail here.

Optionally, as an embodiment, the processor 21 is specifically configured to: the first objective function is min(||y-Dα|| _F +λ||α|| _F ), where y represents a sample subset Low-resolution image samples, D represents the first feature matrix of the sample subset, α represents the expression coefficient matrix corresponding to the low-resolution image samples, ||·|| _{F table} demonstrates the number operation, min represents the minimum value operation, λ is Regularizing a constant parameter; correspondingly, obtaining a first feature matrix corresponding to the subset of image samples according to the first objective function and the low-resolution image sample, comprising: iteratively updating D with a preset number of times under the constraint of the first objective function And α, obtain a first feature matrix that satisfies the first objective function.

Optionally, as an embodiment, the processor 21 is specifically configured to: the first objective function is min(||y-Dα|| _F +λ||α|| _F ), where y represents a sample subset High-resolution image samples, D represents the second feature matrix of the sample subset, α represents the expression coefficient matrix corresponding to the high-resolution image samples, ||·|| _{F table} demonstrates the number operation, min represents the minimum value operation, λ is Regularizing a constant parameter; correspondingly, obtaining a second feature matrix corresponding to the subset of image samples according to the first objective function and the high-resolution image sample, comprising: iteratively updating D by a preset number of times under the constraint of the first objective function And α, obtain a second characteristic matrix that satisfies the first objective function.

Optionally, as an embodiment, the processor 21 is configured to: according to the first feature matrix, encode the low resolution image sample under the constraint of the first objective function, to obtain the first expression coefficient matrix; The feature matrix, under the constraint of the first objective function, encodes the high resolution image sample to obtain the second expression coefficient matrix; in the second objective function

Under the constraint, obtain the mapping relationship matrix, where α _l represents the first expression coefficient matrix, α _h represents the second expression coefficient matrix, and M represents the mapping relationship matrix,

Represents a full 1 matrix, ||·|| _{F table} demonstrates the number operation, and min represents the minimum value operation.

Optionally, as an embodiment, the processor 21 is specifically configured to: extract image features of an image to be processed; compare image features of the image to be processed with image features of each subset of image samples in the image sample set; The subset of image samples corresponding to the image to be processed is the subset of samples with the smallest difference.

Optionally, as an embodiment, the processor 21 is specifically configured to: according to the first feature matrix, Encoding the image to be processed under the constraint of the first objective function, obtaining a third expression coefficient matrix corresponding to the image to be processed; obtaining the image to be processed under the constraint of the second objective function according to the mapping relationship matrix and the third expression coefficient matrix Corresponding fourth expression coefficient matrix; the fourth expression coefficient matrix and the second feature matrix are multiplied to obtain a high frequency component of the image to be processed; the high frequency component and the enlarged image to be processed are added to obtain a corresponding image to be processed Resolution image.

It should be understood that the image processing apparatus 20 according to an embodiment of the present invention may correspond to the method 1000 of performing image processing in the embodiment of the present invention, and the above and other operations and/or functions of the respective modules in the image processing apparatus 20 are respectively implemented for The corresponding processes of the respective methods in FIG. 1 are not described herein for the sake of brevity.

In the embodiment of the present invention, by independently training and obtaining a low-resolution and high-resolution dictionary, the constraints on the same sparse representation are appropriately relaxed, and the sample information can be utilized more flexibly, and at the same time, the solution of the training process of the high-low resolution dictionary is obtained. Coupling, the process of reconstructing the image and the process of sample training can be completely consistent. The expression coefficient of the training image and the expression coefficient of the image generated during the reconstruction phase use the same objective function, the same calculation and derivation process, and improve the reliability of the training result. Sex, which improves the accuracy of generating high-frequency components of images by expression coefficients and improves the quality of high-resolution image reconstruction.

It is to be understood that the phrase "one embodiment" or "an embodiment" or "an" Thus, "in one embodiment" or "in an embodiment" or "an" In addition, these particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

In the various embodiments of the present invention, it should be understood that the size of the sequence numbers of the above processes does not mean the order of execution, and the order of execution of each process should be determined by its function and internal logic, and should not be taken to the embodiments of the present invention. The implementation process constitutes any limitation.

Additionally, the terms "system" and "network" are used interchangeably herein. It should be understood that the term "and/or" herein is merely an association relationship describing an associated object, indicating that there may be three relationships, for example, A and/or B, which may indicate that A exists separately, and A and B exist simultaneously. There are three cases of B alone. In addition, the character "/" in this article generally indicates that the contextual object is an "or" relationship.

In the embodiments provided herein, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined from A. However, it should also be understood that determining B from A does not mean that B is only determined based on A, and that B can also be determined based on A and/or other information.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both, for clarity of hardware and software. Interchangeability, the composition and steps of the various examples have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods for implementing the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present invention.

A person skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the system, the device and the unit described above can refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

An integrated unit, if implemented in the form of a software functional unit and sold or used as a standalone product, can be stored in a computer readable storage medium. Based on such understanding, the present invention The technical solution in essence or the part contributing to the prior art or part of the technical solution may be embodied in the form of a software product stored in a storage medium, including a plurality of instructions for making one The computer device (which may be a personal computer, server, or network device, etc.) performs all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: a USB flash drive, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a disk or a CD. A variety of media that can store program code.

The above is only a few embodiments of the present invention, and various modifications and changes may be made thereto without departing from the spirit and scope of the invention. A person skilled in the art can understand that the features of the embodiments or different embodiments can be combined with each other to form a new embodiment without conflict.

Claims (12)

1. An image processing method, comprising:

   Sorting the image sample set according to the image feature to obtain a plurality of image sample subsets, wherein the image sample subset includes high resolution image samples and low resolution image samples, the low resolution image samples being High-resolution image sample down sampling;

   Obtaining, according to the first objective function and the low resolution image sample, a first feature matrix corresponding to the subset of image samples;

   Obtaining, according to the first objective function and the high resolution image sample, a second feature matrix corresponding to the subset of image samples;

   Obtaining, according to the first feature matrix, the second feature matrix, the low resolution image sample, and the high resolution image sample, a mapping relationship matrix between high and low resolution image samples of the image sample subset;

   Determining a subset of image samples corresponding to the image to be processed;

   Obtaining the image to be processed according to the first objective function and the mapping relationship matrix between the first feature matrix, the second feature matrix, and the high and low resolution image samples corresponding to the determined image sample subset corresponding to the image to be processed. Corresponding high resolution image.
2. The method of claim 1 wherein said first objective function is min(||y-Dα|| _F +λ||α|| _F ), wherein y represents a low of said subset of samples a resolution image sample, D represents a first feature matrix of the sample subset, α represents a matrix of expression coefficients corresponding to the low-resolution image samples, ||·|| _F represents an exemplary operation, and min represents a minimum operation , λ is a regularization constant parameter;

   Correspondingly, the obtaining, according to the first objective function and the low-resolution image sample, the first feature matrix corresponding to the subset of image samples, comprising: preset times under the constraint of the first objective function The D and α are iteratively updated to obtain the first feature matrix that satisfies the first objective function.
3. The method according to claim 1 or 2, wherein the obtaining a second feature corresponding to the subset of image samples according to the first objective function and the high resolution image sample The matrix includes: after the constraint of the first objective function, iteratively updates D and α by a preset number of times to obtain the second feature matrix satisfying the first objective function.
4. The method according to any one of claims 1 to 3, wherein said first feature matrix, said second feature matrix, said low resolution image sample, said high resolution image sample Obtaining a mapping relationship matrix between the high and low resolution image samples of the subset of image samples, including:

   Decoding, according to the first feature matrix, the low resolution image samples under the constraint of the first objective function to obtain a first expression coefficient matrix; and according to the second feature matrix, in the first objective function Encoding the high resolution image sample to obtain a second expression coefficient matrix; in the second objective function

   

   Obtaining the mapping relationship matrix, wherein α _l represents the first expression coefficient matrix, α _h represents the second expression coefficient matrix, and M represents the mapping relationship matrix,

   

   Represents a full 1 matrix, ||·|| _{F table} demonstrates the number operation, and min represents the minimum value operation.
5. The method according to any one of claims 1 to 4, wherein the determining a subset of image samples corresponding to the image to be processed comprises:

   Extracting image features of the image to be processed;

   Comparing the difference between the image feature of the image to be processed and the image feature of each subset of image samples in the image sample set;

   Determining, by the subset of image samples corresponding to the image to be processed, the subset of samples having the smallest difference.
6. The method according to any one of claims 1 to 5, wherein the first feature matrix and the second feature matrix corresponding to the image sample subset corresponding to the image to be processed according to the first objective function Obtaining a high-resolution image corresponding to the image to be processed, and obtaining a high-resolution image corresponding to the image to be processed, including:

   Encoding the to-be-processed image to obtain a third expression coefficient matrix corresponding to the to-be-processed image according to the first feature matrix; and according to the mapping relationship matrix and the a matrix of expression coefficients, under the constraint of the second objective function, obtaining a fourth expression coefficient matrix corresponding to the image to be processed; the fourth expression coefficient matrix and the second feature matrix are multiplied to obtain the High frequency component of the image to be processed; said high frequency component and said amplified The images to be processed are added to obtain a high resolution image corresponding to the image to be processed.
7. An image processing apparatus comprising:

   a first classification module, configured to classify the image sample set according to the image feature to obtain a plurality of image sample subsets, wherein the image sample subset includes a high resolution image sample and a low resolution image sample, the low resolution Rate image samples are obtained by downsampling the high resolution image samples;

   a first acquiring module, configured to obtain, according to the first objective function and the low-resolution image sample, a first feature matrix corresponding to the subset of image samples;

   a second acquiring module, configured to obtain, according to the first objective function and the high-resolution image sample, a second feature matrix corresponding to the subset of image samples;

   a third acquiring module, configured to obtain high and low resolution images of the subset of image samples according to the first feature matrix, the second feature matrix, the low resolution image sample, and the high resolution image sample a mapping relationship matrix between samples;

   a second classification module, configured to determine a subset of image samples corresponding to the image to be processed;

   a fourth acquiring module, configured to: according to the first objective function and the determined image feature subset corresponding to the image to be processed, a first relationship matrix, a second feature matrix, and a mapping relationship matrix between high and low resolution image samples Obtaining a high resolution image corresponding to the image to be processed.
8. The apparatus according to claim 7, wherein the first obtaining module comprises:

   The first objective function is min(||y-Dα|| _F +λ||α|| _F ), where y represents a low-resolution image sample of the sample subset, and D represents the sample subset a first characteristic matrix, α represents a matrix of expression coefficients corresponding to the low-resolution image samples, an example of the ||·|| _F table operation, a minimum operation of min, and a regularization constant parameter of λ;

   Correspondingly, the first acquiring module is specifically configured to: after the constraint of the first objective function, iteratively update D and α by a preset number of times to obtain the first feature matrix that satisfies the first objective function.
9. The apparatus according to claim 7 or 8, wherein the second obtaining module is configured to: after the constraint of the first objective function, iteratively update D and α by a preset number of times to obtain the first objective function. The second feature matrix.
10. The apparatus according to any one of claims 7 to 9, wherein the third obtaining module is specifically configured to:

    Decoding, according to the first feature matrix, the low resolution image samples under the constraint of the first objective function to obtain a first expression coefficient matrix; and according to the second feature matrix, in the first objective function Encoding the high resolution image sample to obtain a second expression coefficient matrix; in the second objective function

    

    Obtaining the mapping relationship matrix, wherein α _l represents the first expression coefficient matrix, α _h represents the second expression coefficient matrix, and M represents the mapping relationship matrix,

    

    Represents a full 1 matrix, ||·|| _{F table} demonstrates the number operation, and min represents the minimum value operation.
11. The apparatus according to any one of claims 7 to 10, wherein the second classification module is specifically configured to:

    Extracting image features of the image to be processed;

    Comparing the difference between the image feature of the image to be processed and the image feature of each subset of image samples in the image sample set;

    Determining, by the subset of image samples corresponding to the image to be processed, the subset of samples having the smallest difference.
12. The apparatus according to any one of claims 7 to 11, wherein the fourth obtaining module is specifically configured to:

    Encoding the to-be-processed image to obtain a third expression coefficient matrix corresponding to the to-be-processed image according to the first feature matrix; and according to the mapping relationship matrix and the a matrix of expression coefficients, under the constraint of the second objective function, obtaining a fourth expression coefficient matrix corresponding to the image to be processed; the fourth expression coefficient matrix and the second feature matrix are multiplied to obtain the a high frequency component of the image to be processed; the high frequency component and the enlarged image to be processed are added to obtain a high resolution image corresponding to the image to be processed.

Images