CN107977929B - Image super-resolution processing method and device - Google Patents
Image super-resolution processing method and device Download PDFInfo
- Publication number
- CN107977929B CN107977929B CN201610920030.3A CN201610920030A CN107977929B CN 107977929 B CN107977929 B CN 107977929B CN 201610920030 A CN201610920030 A CN 201610920030A CN 107977929 B CN107977929 B CN 107977929B
- Authority
- CN
- China
- Prior art keywords
- dictionary
- resolution
- block
- blocking
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 11
- 238000012545 processing Methods 0.000 claims abstract description 88
- 238000000034 method Methods 0.000 claims abstract description 17
- 230000000903 blocking effect Effects 0.000 claims description 109
- 238000000638 solvent extraction Methods 0.000 claims description 14
- 230000002194 synthesizing effect Effects 0.000 claims description 13
- 238000010276 construction Methods 0.000 claims description 10
- 230000011218 segmentation Effects 0.000 claims description 9
- 238000005192 partition Methods 0.000 claims description 8
- 238000012935 Averaging Methods 0.000 claims description 5
- 230000015572 biosynthetic process Effects 0.000 claims description 3
- 238000003786 synthesis reaction Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 9
- 238000010586 diagram Methods 0.000 description 16
- 230000006870 function Effects 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 3
- 241001025261 Neoraja caerulea Species 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Editing Of Facsimile Originals (AREA)
Abstract
The invention discloses an image super-resolution processing method and device. The method comprises the following steps: constructing a plurality of dictionaries according to the low-resolution images and the corresponding high-resolution images; adaptively selecting a dictionary corresponding to an input image; and performing super-resolution processing on the input image by adopting a self-adaptive selected dictionary. According to the invention, the traditional single dictionary is divided into a plurality of dictionaries according to different scenes and different block sizes, so that the reconstruction accuracy is greatly improved, and the image super-resolution processing effect is improved.
Description
Technical Field
The present invention relates to the field of image processing, and in particular, to a method and an apparatus for processing super-resolution of an image.
Background
The current 4K image super-resolution technology comprises three technologies, namely, implementing super-resolution processing of an image by using a specific interpolation method, training a dictionary by using a sparse coding method and researching the expression form of high-resolution details of the image under low resolution to construct a super-resolution processing model of the image, and learning the super-resolution processing model of the image by using a CNN method and a large number of high-resolution images and low-resolution images.
Although the interpolation method has a fast processing speed, the effect is poor, and the sparse coding method has a better effect than the interpolation method, but needs to be trained in advance, and the CNN (convolutional Neural Network) method has a better effect than the former two methods, but needs to be trained in advance by using a large amount of data, and the training time is long.
Disclosure of Invention
In view of the above technical problems, the present invention provides an image super-resolution processing method and apparatus, which can perform super-resolution processing on an image according to different block sizes and different scenes, thereby greatly improving reconstruction accuracy.
According to an aspect of the present invention, there is provided an image super-resolution processing method including:
constructing a plurality of dictionaries according to the low-resolution images and the corresponding high-resolution images;
adaptively selecting a dictionary corresponding to an input image;
and performing super-resolution processing on the input image by adopting a self-adaptive selected dictionary.
In one embodiment of the present invention, the constructing the plurality of dictionaries from the low-resolution images and the corresponding high-resolution images comprises:
constructing a plurality of dictionaries by using a plurality of groups of low-resolution image blocks and corresponding high-resolution image detail blocks;
and clustering the pixel information of the low-resolution blocks aiming at each dictionary to obtain a plurality of sub-dictionaries.
In one embodiment of the present invention, the constructing the plurality of dictionaries using the plurality of sets of low-resolution image blocks and the corresponding high-resolution image detail blocks includes:
carrying out scene classification on the low-resolution images;
partitioning each low-resolution image according to different partition sizes;
determining a high-resolution detail block corresponding to each low-resolution image block;
and forming a dictionary by all low-resolution image blocks and corresponding high-resolution detail blocks of the same scene and the same block size.
In an embodiment of the present invention, the determining the high-resolution detail block corresponding to each low-resolution image block includes:
determining a high-resolution image block corresponding to each low-resolution image block;
and carrying out difference on the interpolation image blocks of the high-resolution image block and the low-resolution image block to obtain a high-resolution detail block.
In one embodiment of the present invention, the step of adaptively selecting the dictionary corresponding to the input image comprises:
determining a scene of an input image;
determining an optimal blocking mode of an input image, and blocking the input image according to the optimal blocking mode;
and determining a dictionary corresponding to each image block according to the scene and the block size of each image block after the block division.
In an embodiment of the present invention, the step of determining an optimal blocking mode of the input image and blocking the input image according to the optimal blocking mode includes:
performing first blocking on an input image according to a blocking mode with the largest blocking size in a plurality of pre-constructed dictionaries;
determining whether secondary blocking is needed or not and the number of blocks of the secondary blocking are needed according to texture complexity of each image block subjected to the primary blocking, wherein the size of the image block subjected to the secondary blocking meets the size of the blocks in a plurality of pre-constructed dictionaries;
and if the second time of blocking is determined to be needed, performing second time of blocking according to the number of the second time of blocking.
In an embodiment of the present invention, the super-resolution processing of the input image using the adaptively selected dictionary includes:
performing super-resolution processing on each small block in the input image by adopting a set of a plurality of sub-dictionaries in the self-adaptive selected dictionary;
and synthesizing all the small blocks subjected to super-resolution processing into a picture.
In one embodiment of the present invention, the super-resolution processing of each small block in the input image using the adaptively selected set of sub-dictionaries in the dictionary comprises:
averaging low-resolution blocks of each sub-dictionary in the corresponding dictionary to obtain a dictionary center, and taking the sub-dictionary with the dictionary center most similar to the input image block as an initial dictionary, wherein the dictionary radius R of the initial dictionary is 0, and the corresponding dictionary is a self-adaptive selected dictionary;
reconstructing an input image block according to a low-resolution image block in a current dictionary;
acquiring a reconstruction error according to the difference value of the reconstruction image block and the input image block;
performing super-resolution processing on an input image block according to a high-resolution detail block in a current dictionary under the condition that the reconstruction error is not larger than a preset threshold value or the current dictionary already contains all sub-dictionaries of a corresponding dictionary;
and under the condition that the reconstruction error is larger than a preset threshold value, adding a fixed value on the radius of the current dictionary, synthesizing all sub dictionaries corresponding to the dictionary centers with the distance smaller than R from the original dictionary center into a new current dictionary, and then executing the step of reconstructing the input image block according to the low-resolution image block in the current dictionary.
According to another aspect of the present invention, there is provided an image super-resolution processing apparatus including a multi-dictionary constructing module, a dictionary selecting module, and a super-resolution processing module, wherein:
the multi-dictionary building module is used for building a plurality of dictionaries according to the low-resolution images and the corresponding high-resolution images;
the dictionary selection module is used for adaptively selecting a dictionary corresponding to the input image;
and the super-resolution processing module is used for performing super-resolution processing on the input image by adopting the self-adaptive selected dictionary.
In one embodiment of the invention, the multiple dictionary building module comprises a multiple dictionary building unit and a sub-dictionary generating unit, wherein:
the multi-dictionary construction unit is used for constructing a plurality of dictionaries by using a plurality of groups of low-resolution image blocks and corresponding high-resolution image detail blocks;
and the sub-dictionary generating unit is used for clustering each dictionary through the pixel information of the low-resolution block to obtain a plurality of sub-dictionaries.
In an embodiment of the present invention, the multi-dictionary building unit includes a scene classification sub-module, a segmentation sub-module, a detail block determination sub-module, and a dictionary building sub-module, wherein:
the scene classification submodule is used for carrying out scene classification on the low-resolution images;
the segmentation submodule is used for segmenting each low-resolution image according to different segmentation sizes;
the detail block determination submodule is used for determining a high-resolution detail block corresponding to each low-resolution image block;
and the dictionary construction submodule is used for constructing a dictionary by all low-resolution image blocks and corresponding high-resolution detail blocks of the same scene and the same block size.
In an embodiment of the present invention, the detail block determination sub-module is configured to determine a high-resolution image block corresponding to each low-resolution image block; and carrying out difference on the interpolation image blocks of the high-resolution image block and the low-resolution image block to obtain a high-resolution detail block.
In one embodiment of the present invention, the dictionary selection module includes a scene determination unit, a blocking manner determination unit, and a dictionary determination unit, wherein:
a scene determination unit for determining a scene of an input image;
the device comprises a blocking mode determining unit, a blocking mode determining unit and a blocking mode determining unit, wherein the blocking mode determining unit is used for determining the optimal blocking mode of an input image and blocking the input image according to the optimal blocking mode;
and the dictionary determining unit is used for determining the dictionary corresponding to the image block according to the scene and the block size of each image block after the block division.
In an embodiment of the present invention, the blocking mode determining unit includes a first blocking sub-module, a second blocking judgment sub-module, and a second blocking sub-module, wherein:
the first blocking submodule is used for carrying out first blocking on the input image according to a blocking mode with the largest blocking size in a plurality of pre-constructed dictionaries;
the second partitioning judgment sub-module is used for determining whether second partitioning is needed or not and the number of partitions of the second partitioning according to the texture complexity of each image block after the first partitioning, wherein the size of each image block after the second partitioning meets the size of the partitions in a plurality of pre-constructed dictionaries;
and the second blocking submodule is used for carrying out second blocking according to the number of the blocks of the second blocking under the condition that the second blocking judgment submodule determines that the second blocking is required.
In one embodiment of the present invention, a super-resolution processing module includes a super-resolution processing unit and an image synthesizing unit, wherein:
the super-resolution processing unit is used for performing super-resolution processing on each small block in the input image by adopting a set of a plurality of sub-dictionaries in the self-adaptive selected dictionary;
and the image synthesis unit is used for synthesizing all the small blocks subjected to super-resolution processing into a picture.
In one embodiment of the present invention, the super-resolution processing unit includes an initial dictionary determination submodule, a reconstruction error acquisition submodule, a super-resolution processing submodule, and a new dictionary determination submodule, wherein:
the initial dictionary determining submodule is used for averaging the low-resolution blocks of each sub-dictionary in the corresponding dictionary to obtain a dictionary center, the sub-dictionary with the dictionary center most similar to the input image block is used as the initial dictionary, the dictionary radius R of the initial dictionary is 0, and the corresponding dictionary is a self-adaptive selected dictionary;
the reconstruction submodule is used for reconstructing the input image block according to the low-resolution image block in the current dictionary;
the reconstruction error acquisition submodule is used for acquiring a reconstruction error according to the difference value of the reconstruction image block and the input image block;
the super-resolution processing sub-module is used for carrying out super-resolution processing on the input image block according to the high-resolution detail block in the current dictionary under the condition that the reconstruction error is not larger than a preset threshold value or the current dictionary already contains all sub-dictionaries of the corresponding dictionary;
and the new dictionary determining submodule is used for adding a fixed value on the radius of the current dictionary under the condition that the reconstruction error is larger than a preset threshold value, synthesizing all sub-dictionaries corresponding to the dictionary centers with the distance smaller than R from the original dictionary center into a new current dictionary, and then indicating the reconstruction submodule to carry out the operation of reconstructing the input image block according to the low-resolution image block in the current dictionary.
According to the invention, the traditional single dictionary is divided into a plurality of dictionaries according to different scenes and different block sizes, so that the reconstruction accuracy is greatly improved, and the image super-resolution processing effect is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic diagram of an embodiment of the image super-resolution processing method of the present invention.
Fig. 2 is a schematic diagram of an embodiment of the image super-resolution processing apparatus according to the present invention.
FIG. 3 is a diagram of a multi-dictionary building block in accordance with an embodiment of the present invention.
FIG. 4 is a diagram of a multi-dictionary building unit in one embodiment of the present invention.
FIG. 5 is a diagram of a dictionary selection module in accordance with one embodiment of the present invention.
Fig. 6 is a schematic diagram of a blocking mode determining unit according to an embodiment of the present invention.
Fig. 7 is a diagram illustrating a super-resolution processing module according to an embodiment of the invention.
Fig. 8 is a schematic diagram of a super-resolution processing unit in an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
Fig. 1 is a schematic diagram of an embodiment of the image super-resolution processing method of the present invention. Preferably, this embodiment can be executed by the image super-resolution processing apparatus of the present invention.
The core idea of the image super-resolution processing method is to construct a plurality of dictionaries for self-adaptive selection to perform image super-resolution processing. As shown in FIG. 1, the method comprises three steps of multi-dictionary construction, adaptive dictionary selection and super-resolution processing.
The three steps shown in FIG. 1 will be specifically described below.
In one embodiment of the present invention, step 1 may comprise:
and 11, constructing a plurality of dictionaries by using the plurality of groups of low-resolution image blocks and the corresponding high-resolution image detail blocks.
In an embodiment of the present invention, step 11 may further include:
and step 111, carrying out scene classification on the original low-resolution image BOW (bag of words) and SVM (Support Vector Machine) methods. If the original image belongs to a scene, all the blocks of the image belong to the scene.
And step 112, partitioning each low-resolution image according to different partition sizes.
For example, in one specific example, step 112 may comprise: each low resolution image is partitioned into 4 × 4 pixel blocks, 8 × 8 pixel blocks, and 16 × 16 pixel blocks.
In step 113, a high resolution detail block corresponding to each low resolution image block is determined.
In a specific example of the present invention, step 113 may further include:
step 1131, determine a high resolution image block corresponding to each low resolution image block.
Step 1132, subtracting the interpolated image blocks of the high-resolution image block and the low-resolution image block to obtain a high-resolution detail block.
And step 114, forming a dictionary by all low-resolution image blocks and corresponding high-resolution detail blocks of the same scene and the same block size.
And step 12, clustering each dictionary through the pixel information of the low-resolution block to obtain a plurality of sub-dictionaries.
And 2, adaptively selecting a dictionary corresponding to the input image.
In one embodiment of the present invention, step 2 may comprise:
step 21, the scene of the input image is determined.
All blocks of an input image are super-resolution processed by using the dictionary of the scene, but the block sizes of the small block selection dictionary of the same image are not necessarily the same.
And step 22, determining the optimal blocking mode of the input image, and blocking the input image according to the optimal blocking mode.
In one embodiment of the present invention, step 22 may further include:
and step 221, performing first blocking on the input image according to a blocking mode with the largest blocking size in a plurality of pre-constructed dictionaries.
For example: assuming that there are three dictionaries with different block sizes, which are respectively partitioned according to 4 × 4 pixel blocks, 8 × 8 pixel blocks, and 16 × 16 pixel blocks, the input image is first partitioned according to 16 × 16 pixel blocks.
For example: for the aforementioned dictionary having three different block sizes, after the input image is blocked by 16 × 16 pixel blocks, whether to further divide it into 8 × 8 or 4 × 4 pixel blocks is determined by texture complexity. For each 16 × 16 pixel block, there are three further blocking approaches: if the blocking mode is x and the number of the blocks is n, x is 1 and represents no blocking, namely n is 1; x-2 represents a pixel block divided into 8 × 8, and n-4; x-3 represents a 4 × 4 pixel block, and n-16.
In one embodiment of the present invention, step 222 may be determined according to equation (1).
Wherein
Denotes the ith block after x blocks, H denotes texture complexity, S denotes block size, λ is a constant,
for the best partitioning, the texture complexity function is shown in equation (2), and the partition size function of the previous embodiment is shown in equation (3). When the texture complexity is calculated, a color distribution histogram of the small blocks needs to be obtained first, k in formula (2) represents a bin in the histogram, and p (k) represents the probability of the bin.
And 223, if it is determined that the second blocking is required, performing the second blocking according to the number of the second blocking.
And step 23, determining a dictionary corresponding to each image block according to the scene and the block size of each image block after the block division.
And 3, performing super-resolution processing on the input image by adopting the self-adaptive selected dictionary.
For each small block of the input image, the corresponding dictionary has already been selected in step 2, but the low resolution block is reconstructed not directly from the dictionary but from a set of several sub-dictionaries in the dictionary, which have already been explained in step 1.
In one embodiment of the present invention, step 3 may comprise:
and step 31, performing super-resolution processing on each small block in the input image by adopting a set of a plurality of sub-dictionaries in the self-adaptive selected dictionary.
In an embodiment of the present invention, step 31 may further include:
and 311, averaging the low-resolution blocks of each sub-dictionary in the corresponding dictionary to obtain a dictionary center, and taking the sub-dictionary with the dictionary center most similar to the input image block as an initial dictionary, wherein the dictionary radius R of the initial dictionary is 0, and the corresponding dictionary is a self-adaptive selected dictionary.
And step 312, reconstructing the input image block according to the low-resolution image block in the current dictionary.
In one example of the present invention, step 312 may include: and reconstructing the input image block according to the low-resolution image block in the current dictionary by adopting a sparse coding method.
In one specific example of the present invention, step 312 may include: the input image block is reconstructed according to equation (4).
In equation (4), the input image block is P, and the low resolution image block in the dictionary is Li，αiIs LiThe corresponding reconstruction coefficients.
In a specific example of the present invention, step 313 may include: the reconstruction error e is obtained according to equation (5).
And step 314, performing super-resolution processing on the input image block according to the high-resolution detail block in the current dictionary under the condition that the reconstruction error e is not larger than the preset threshold value T or the current dictionary already contains all the sub-dictionaries of the corresponding dictionary.
In a specific example of the present invention, the super-resolution processing the input image block according to the high-resolution detail block in the current dictionary in step 314 may include: the super-resolution processing is performed according to the formula (6).
In formula (6), HR (P) is the high resolution block obtained, B is the bicubic interpolation function, HiIs LiCorresponding high resolution detail image blocks.
And step 32, synthesizing all the small blocks subjected to super-resolution processing into a picture.
The image super-resolution processing method provided by the embodiment of the invention is a sparse coding image super-resolution processing method based on multi-dictionary construction and self-adaptive dictionary selection, and a traditional single dictionary is divided into a plurality of dictionaries according to different scenes and different block sizes, so that the accuracy of image super-resolution reconstruction is greatly improved, and the image super-resolution processing effect is improved.
In the above embodiment of the present invention, the scale of each dictionary is much smaller than that of a conventional single dictionary, and the speed of finding the reconstruction coefficient is much faster, so that the speed of image super-resolution processing is improved.
The embodiment of the invention can perform super-resolution processing on the image according to different block sizes and different scenes, has good image super-resolution processing effect and high speed, and is very suitable for super-resolution processing of blue-ray movies into application scenes of 4K videos.
Fig. 2 is a schematic diagram of an embodiment of the image super-resolution processing apparatus according to the present invention. As shown in fig. 2, the image super-resolution processing apparatus includes a multi-dictionary constructing module 100, a dictionary selecting module 200, and a super-resolution processing module 300, wherein:
a multi-dictionary construction module 100 for constructing a plurality of dictionaries from the low-resolution images and the corresponding high-resolution images.
A dictionary selection module 200 for adaptively selecting a dictionary corresponding to the input image.
And a super-resolution processing module 300, configured to perform super-resolution processing on the input image by using the adaptively selected dictionary.
The structure and function of the multi-dictionary constructing module 100, the dictionary selecting module 200 and the super-resolution processing module 300 in the embodiment of fig. 2 are further explained by specific examples.
FIG. 3 is a diagram of a multi-dictionary building block in accordance with an embodiment of the present invention. As shown in fig. 3, the multi-dictionary building module 100 of the embodiment of fig. 2 may include a multi-dictionary building unit 110 and a sub-dictionary generating unit 120, where:
a multi-dictionary construction unit 110 for constructing a plurality of dictionaries using the plurality of sets of low-resolution image blocks and the corresponding high-resolution image detail blocks.
And the sub-dictionary generating unit 120 is configured to cluster the pixel information of the low-resolution block for each dictionary to obtain a plurality of sub-dictionaries.
FIG. 4 is a diagram of a multi-dictionary building unit in one embodiment of the present invention. As shown in fig. 4, the multi-dictionary building unit 110 of the embodiment of fig. 3 may include a scene classification sub-module 111, a segmentation sub-module 112, a detail block determination sub-module 113, and a dictionary building sub-module 114, where:
and the scene classification submodule 111 is used for carrying out scene classification on the low-resolution images.
A segmentation sub-module 112 for segmenting each low resolution image according to a different segmentation size.
And a detail block determination sub-module 113 for determining a high resolution detail block corresponding to each low resolution image block.
In an embodiment of the present invention, the detail block determination sub-module 113 may be specifically configured to determine a high-resolution image block corresponding to each low-resolution image block; and carrying out difference on the interpolation image blocks of the high-resolution image block and the low-resolution image block to obtain a high-resolution detail block.
And the dictionary construction sub-module 114 is used for constructing a dictionary by all the low-resolution image blocks and the corresponding high-resolution detail blocks of the same scene and the same block size.
FIG. 5 is a diagram of a dictionary selection module in accordance with one embodiment of the present invention. As shown in fig. 5, the dictionary selection module 200 of the embodiment of fig. 2 may include a scene determination unit 210, a block mode determination unit 220, and a dictionary determination unit 230, where:
a scene determination unit 210 for determining a scene of the input image.
The block mode determining unit 220 is configured to determine an optimal block mode of the input image, and block the input image according to the optimal block mode.
The dictionary determining unit 230 is configured to determine a dictionary corresponding to each image block according to the scene and the block size of each image block after the block division.
Fig. 6 is a schematic diagram of a blocking mode determining unit according to an embodiment of the present invention. As shown in fig. 6, the blocking manner determining unit 220 in the embodiment of fig. 5 may include a first blocking sub-module 221, a second blocking sub-module 222, and a second blocking sub-module 223, where:
the first blocking submodule 221 is configured to perform first blocking on the input image according to a blocking manner with a largest blocking size in a plurality of pre-constructed dictionaries.
The second blocking judgment sub-module 222 is configured to determine, through texture complexity, whether second blocking is required or not and the number of blocks of the second blocking are required for each image block after the first blocking, where the size of the image block after the second blocking meets the size of blocks in a plurality of pre-constructed dictionaries.
The second blocking sub-module 223 is configured to perform second blocking according to the number of blocks of the second blocking when the second blocking judgment sub-module 222 determines that the second blocking is required.
Fig. 7 is a diagram illustrating a super-resolution processing module according to an embodiment of the invention. As shown in fig. 7, the super-resolution processing module 300 of the embodiment of fig. 2 may include a super-resolution processing unit 310 and an image synthesizing unit 320, wherein:
a super-resolution processing unit 310 for performing super-resolution processing on each small block in the input image using a set of a plurality of sub-dictionaries in the adaptively selected dictionary.
And an image synthesizing unit 320, configured to synthesize all the super-resolution processed small blocks into one image.
Fig. 8 is a schematic diagram of a super-resolution processing unit in an embodiment of the present invention. As shown in fig. 8, the super-resolution processing unit 310 of the embodiment of fig. 2 may include an initial dictionary determination sub-module 311, a reconstruction sub-module 312, a reconstruction error acquisition sub-module 313, a super-resolution processing sub-module 314, and a new dictionary determination sub-module 315, wherein:
the initial dictionary determining submodule 311 is configured to average the low-resolution blocks of each sub-dictionary in the corresponding dictionary to obtain a dictionary center, and use the sub-dictionary with the dictionary center most similar to the input image block as the initial dictionary, where a dictionary radius R of the initial dictionary is 0, and the corresponding dictionary is a self-adaptive selected dictionary.
And the reconstructing sub-module 312 is configured to reconstruct the input image block according to the low-resolution image block in the current dictionary.
And the reconstruction error obtaining submodule 313 is used for obtaining a reconstruction error according to the difference value between the reconstructed image block and the input image block.
A super-resolution processing sub-module 314 for performing super-resolution processing on the input image block according to the high-resolution detail block in the current dictionary if the reconstruction error is not greater than the predetermined threshold or the current dictionary already contains all sub-dictionaries of the corresponding dictionary.
And the new dictionary determining sub-module 315 is configured to, when the reconstruction error is greater than a predetermined threshold, add a fixed value to the radius of the current dictionary, combine all sub-dictionaries corresponding to the dictionary center whose distance from the original dictionary center is less than R into a new current dictionary, and then instruct the reconstruction sub-module 312 to perform an operation of reconstructing the input image block according to the low-resolution image block in the current dictionary.
The image super-resolution processing device provided by the embodiment of the invention is a sparse coding image super-resolution processing device based on multi-dictionary construction and adaptive dictionary selection, and a traditional single dictionary is divided into a plurality of dictionaries according to different scenes and different block sizes, so that the accuracy of image super-resolution reconstruction is greatly improved, and the image super-resolution processing effect is improved.
In the above embodiment of the present invention, the scale of each dictionary is much smaller than that of a conventional single dictionary, and the speed of finding the reconstruction coefficient is much faster, so that the speed of image super-resolution processing is improved.
The embodiment of the invention can perform super-resolution processing on the image according to different block sizes and different scenes, has good image super-resolution processing effect and high speed, and is very suitable for super-resolution processing of blue-ray movies into application scenes of 4K videos.
The functional units of the multi-dictionary building module 100, the dictionary selection module 200, the super-resolution processing module 300, etc. described above may be implemented as a general purpose processor, a Programmable Logic Controller (PLC), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any suitable combination thereof, for performing the functions described herein.
Thus far, the present invention has been described in detail. Some details well known in the art have not been described in order to avoid obscuring the concepts of the present invention. It will be fully apparent to those skilled in the art from the foregoing description how to practice the presently disclosed embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The description of the present invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to practitioners skilled in this art. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
Claims (6)
1. An image super-resolution processing method is characterized by comprising the following steps:
constructing a plurality of dictionaries according to the low-resolution images and the corresponding high-resolution images;
adaptively selecting a dictionary corresponding to an input image;
performing super-resolution processing on an input image by adopting a self-adaptive selected dictionary;
wherein constructing a plurality of dictionaries from the low resolution images and the corresponding high resolution images comprises:
constructing a plurality of dictionaries by using a plurality of groups of low-resolution image blocks and corresponding high-resolution image detail blocks;
clustering is carried out on each dictionary through pixel information of the low-resolution blocks to obtain a plurality of sub-dictionaries;
wherein constructing a plurality of dictionaries using the plurality of sets of low-resolution image blocks and the corresponding high-resolution image detail blocks comprises:
carrying out scene classification on the low-resolution images;
partitioning each low-resolution image according to different partition sizes;
determining a high-resolution detail block corresponding to each low-resolution image block;
forming a dictionary by all low-resolution image blocks and corresponding high-resolution detail blocks of the same scene and the same block size;
wherein the determining the high-resolution detail block corresponding to each low-resolution image block comprises:
determining a high-resolution image block corresponding to each low-resolution image block;
carrying out difference on interpolation image blocks of the high-resolution image block and the low-resolution image block to obtain a high-resolution detail block;
the super-resolution processing of the input image by adopting the adaptively selected dictionary comprises the following steps:
performing super-resolution processing on each small block in the input image by adopting a set of a plurality of sub-dictionaries in the self-adaptive selected dictionary;
synthesizing all the small blocks subjected to super-resolution processing into a picture;
performing super-resolution processing on each small block in the input image by adopting a set of a plurality of sub-dictionaries in the adaptively selected dictionary comprises the following steps:
averaging low-resolution blocks of each sub-dictionary in the corresponding dictionary to obtain a dictionary center, and taking the sub-dictionary with the dictionary center most similar to the input image block as an initial dictionary, wherein the dictionary radius R of the initial dictionary is 0, and the corresponding dictionary is a self-adaptive selected dictionary;
reconstructing an input image block according to a low-resolution image block in a current dictionary;
acquiring a reconstruction error according to the difference value of the reconstruction image block and the input image block;
performing super-resolution processing on an input image block according to a high-resolution detail block in a current dictionary under the condition that the reconstruction error is not larger than a preset threshold value or the current dictionary already contains all sub-dictionaries of a corresponding dictionary;
and under the condition that the reconstruction error is larger than a preset threshold value, adding a fixed value on the radius of the current dictionary, synthesizing all sub dictionaries corresponding to the dictionary centers with the distance smaller than R from the original dictionary center into a new current dictionary, and then executing the step of reconstructing the input image block according to the low-resolution image block in the current dictionary.
2. The method of claim 1, wherein the step of adaptively selecting a dictionary corresponding to the input image comprises:
determining a scene of an input image;
determining an optimal blocking mode of an input image, and blocking the input image according to the optimal blocking mode;
and determining a dictionary corresponding to each image block according to the scene and the block size of each image block after the block division.
3. The method of claim 2, wherein the step of determining an optimal blocking mode for the input image and blocking the input image according to the optimal blocking mode comprises:
performing first blocking on an input image according to a blocking mode with the largest blocking size in a plurality of pre-constructed dictionaries;
determining whether secondary blocking is needed or not and the number of blocks of the secondary blocking are needed according to texture complexity of each image block subjected to the primary blocking, wherein the size of the image block subjected to the secondary blocking meets the size of the blocks in a plurality of pre-constructed dictionaries;
and if the second time of blocking is determined to be needed, performing second time of blocking according to the number of the second time of blocking.
4. An image super-resolution processing apparatus comprising a multi-dictionary constructing module, a dictionary selecting module, and a super-resolution processing module, wherein:
the multi-dictionary building module is used for building a plurality of dictionaries according to the low-resolution images and the corresponding high-resolution images;
the dictionary selection module is used for adaptively selecting a dictionary corresponding to the input image;
the super-resolution processing module is used for performing super-resolution processing on the input image by adopting a self-adaptive selected dictionary;
the multi-dictionary building module comprises a multi-dictionary building unit and a sub-dictionary generating unit, wherein:
the multi-dictionary construction unit is used for constructing a plurality of dictionaries by using a plurality of groups of low-resolution image blocks and corresponding high-resolution image detail blocks;
the sub-dictionary generating unit is used for clustering each dictionary through the pixel information of the low-resolution block to obtain a plurality of sub-dictionaries;
the multi-dictionary building unit comprises a scene classification submodule, a segmentation submodule, a detail block determining submodule and a dictionary building submodule, wherein:
the scene classification submodule is used for carrying out scene classification on the low-resolution images;
the segmentation submodule is used for segmenting each low-resolution image according to different segmentation sizes;
the detail block determination submodule is used for determining a high-resolution detail block corresponding to each low-resolution image block;
the dictionary construction submodule is used for constructing a dictionary by all low-resolution image blocks and corresponding high-resolution detail blocks in the same scene and the same block size;
the detail block determination submodule is used for determining a high-resolution image block corresponding to each low-resolution image block; and the interpolation image blocks of the high-resolution image block and the low-resolution image block are subjected to subtraction to obtain a high-resolution detail block;
wherein, the super-resolution processing module includes a super-resolution processing unit and an image synthesis unit, wherein:
the super-resolution processing unit is used for performing super-resolution processing on each small block in the input image by adopting a set of a plurality of sub-dictionaries in the self-adaptive selected dictionary;
the image synthesis unit is used for synthesizing all the super-resolution processed small blocks into a picture;
the super-resolution processing unit comprises an initial dictionary determining submodule, a reconstruction error acquisition submodule, a super-resolution processing submodule and a new dictionary determining submodule, wherein:
the initial dictionary determining submodule is used for averaging the low-resolution blocks of each sub-dictionary in the corresponding dictionary to obtain a dictionary center, the sub-dictionary with the dictionary center most similar to the input image block is used as the initial dictionary, the dictionary radius R of the initial dictionary is 0, and the corresponding dictionary is a self-adaptive selected dictionary;
the reconstruction submodule is used for reconstructing the input image block according to the low-resolution image block in the current dictionary;
the reconstruction error acquisition submodule is used for acquiring a reconstruction error according to the difference value of the reconstruction image block and the input image block;
the super-resolution processing sub-module is used for carrying out super-resolution processing on the input image block according to the high-resolution detail block in the current dictionary under the condition that the reconstruction error is not larger than a preset threshold value or the current dictionary already contains all sub-dictionaries of the corresponding dictionary;
and the new dictionary determining submodule is used for adding a fixed value on the radius of the current dictionary under the condition that the reconstruction error is larger than a preset threshold value, synthesizing all sub-dictionaries corresponding to the dictionary centers with the distance smaller than R from the original dictionary center into a new current dictionary, and then indicating the reconstruction submodule to carry out the operation of reconstructing the input image block according to the low-resolution image block in the current dictionary.
5. The apparatus of claim 4, wherein the dictionary selection module comprises a scene determination unit, a block mode determination unit, and a dictionary determination unit, wherein:
a scene determination unit for determining a scene of an input image;
the device comprises a blocking mode determining unit, a blocking mode determining unit and a blocking mode determining unit, wherein the blocking mode determining unit is used for determining the optimal blocking mode of an input image and blocking the input image according to the optimal blocking mode;
and the dictionary determining unit is used for determining the dictionary corresponding to the image block according to the scene and the block size of each image block after the block division.
6. The apparatus of claim 5, wherein the blocking mode determining unit comprises a first blocking submodule, a second blocking judgment submodule, and a second blocking submodule, wherein:
the first blocking submodule is used for carrying out first blocking on the input image according to a blocking mode with the largest blocking size in a plurality of pre-constructed dictionaries;
the second partitioning judgment sub-module is used for determining whether second partitioning is needed or not and the number of partitions of the second partitioning according to the texture complexity of each image block after the first partitioning, wherein the size of each image block after the second partitioning meets the size of the partitions in a plurality of pre-constructed dictionaries;
and the second blocking submodule is used for carrying out second blocking according to the number of the blocks of the second blocking under the condition that the second blocking judgment submodule determines that the second blocking is required.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610920030.3A CN107977929B (en) | 2016-10-21 | 2016-10-21 | Image super-resolution processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610920030.3A CN107977929B (en) | 2016-10-21 | 2016-10-21 | Image super-resolution processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107977929A CN107977929A (en) | 2018-05-01 |
| CN107977929B true CN107977929B (en) | 2021-02-23 |
Family
ID=62003836
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610920030.3A Active CN107977929B (en) | 2016-10-21 | 2016-10-21 | Image super-resolution processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107977929B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109064399B (en) * | 2018-07-20 | 2023-01-24 | 广州视源电子科技股份有限公司 | Image super-resolution reconstruction method and system, computer device and storage medium thereof |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20080078217A (en) * | 2007-02-22 | 2008-08-27 | 정태우 | Method for indexing object in video, method for annexed service using index of object and apparatus for processing video |
| CN102547301A (en) * | 2010-09-30 | 2012-07-04 | 苹果公司 | System and method for processing image data using an image signal processor |
| CN102722876A (en) * | 2012-05-29 | 2012-10-10 | 杭州电子科技大学 | Residual-based ultra-resolution image reconstruction method |
| CN102968766A (en) * | 2012-11-23 | 2013-03-13 | 上海交通大学 | Dictionary database-based adaptive image super-resolution reconstruction method |
| CN103093444A (en) * | 2013-01-17 | 2013-05-08 | 西安电子科技大学 | Image super-resolution reconstruction method based on self-similarity and structural information constraint |
| CN105046672A (en) * | 2015-06-30 | 2015-11-11 | 北京工业大学 | Method for image super-resolution reconstruction |
| CN105389778A (en) * | 2015-11-04 | 2016-03-09 | 北京大学深圳研究生院 | Image super resolution reconstruction method and device based on dictionary matching |
| CN105825473A (en) * | 2015-12-24 | 2016-08-03 | 三维通信股份有限公司 | Image restoration method through adaptively switching analysis sparse regularization and synthesis sparse regularization |
| CN105844590A (en) * | 2016-03-23 | 2016-08-10 | 武汉理工大学 | Image super-resolution reconstruction method and system based on sparse representation |
-
2016
- 2016-10-21 CN CN201610920030.3A patent/CN107977929B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20080078217A (en) * | 2007-02-22 | 2008-08-27 | 정태우 | Method for indexing object in video, method for annexed service using index of object and apparatus for processing video |
| CN102547301A (en) * | 2010-09-30 | 2012-07-04 | 苹果公司 | System and method for processing image data using an image signal processor |
| CN102722876A (en) * | 2012-05-29 | 2012-10-10 | 杭州电子科技大学 | Residual-based ultra-resolution image reconstruction method |
| CN102968766A (en) * | 2012-11-23 | 2013-03-13 | 上海交通大学 | Dictionary database-based adaptive image super-resolution reconstruction method |
| CN103093444A (en) * | 2013-01-17 | 2013-05-08 | 西安电子科技大学 | Image super-resolution reconstruction method based on self-similarity and structural information constraint |
| CN105046672A (en) * | 2015-06-30 | 2015-11-11 | 北京工业大学 | Method for image super-resolution reconstruction |
| CN105389778A (en) * | 2015-11-04 | 2016-03-09 | 北京大学深圳研究生院 | Image super resolution reconstruction method and device based on dictionary matching |
| CN105825473A (en) * | 2015-12-24 | 2016-08-03 | 三维通信股份有限公司 | Image restoration method through adaptively switching analysis sparse regularization and synthesis sparse regularization |
| CN105844590A (en) * | 2016-03-23 | 2016-08-10 | 武汉理工大学 | Image super-resolution reconstruction method and system based on sparse representation |
Non-Patent Citations (1)
| Title |
|---|
| 超分辨率图像重建方法综述;苏衡 等;《自动化学报》;20130831;第39卷(第8期);第1202-1213页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107977929A (en) | 2018-05-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110324664B (en) | Video frame supplementing method based on neural network and training method of model thereof | |
| CN112396645B (en) | Monocular image depth estimation method and system based on convolution residual learning | |
| CN109101975B (en) | Image semantic segmentation method based on full convolution neural network | |
| CN109712165B (en) | Similar foreground image set segmentation method based on convolutional neural network | |
| CN104008538B (en) | Based on single image super-resolution method | |
| KR20220066945A (en) | Image processing method, apparatus, electronic device and computer readable storage medium | |
| EP3963516B1 (en) | Teaching gan (generative adversarial networks) to generate per-pixel annotation | |
| CN110689599B (en) | 3D visual saliency prediction method based on non-local enhancement generation countermeasure network | |
| Yin et al. | Highly accurate image reconstruction for multimodal noise suppression using semisupervised learning on big data | |
| US10198801B2 (en) | Image enhancement using self-examples and external examples | |
| Cao et al. | Image Super-Resolution via Adaptive $\ell _ {p}(0< p< 1) $ Regularization and Sparse Representation | |
| EP3507773A1 (en) | Systems and methods for providing convolutional neural network based image synthesis using stable and controllable parametric models, a multiscale synthesis framework and novel network architectures | |
| Wen et al. | VIDOSAT: High-dimensional sparsifying transform learning for online video denoising | |
| CN114339409B (en) | Video processing method, device, computer equipment and storage medium | |
| CN111476719A (en) | Image processing method, image processing device, computer equipment and storage medium | |
| CN114240735B (en) | Arbitrary style migration method, system, storage medium, computer equipment and terminal | |
| CN109447897B (en) | Real scene image synthesis method and system | |
| Miandji et al. | Compressive image reconstruction in reduced union of subspaces | |
| WO2021114870A1 (en) | Parallax estimation system and method, electronic device and computer-readable storage medium | |
| CN110809126A (en) | Video frame interpolation method and system based on adaptive deformable convolution | |
| CN113313810A (en) | 6D attitude parameter calculation method for transparent object | |
| US20220335685A1 (en) | Method and apparatus for point cloud completion, network training method and apparatus, device, and storage medium | |
| CN116205820A (en) | Image enhancement method, target identification method, device and medium | |
| CN107977929B (en) | Image super-resolution processing method and device | |
| CN107085725B (en) | Method for clustering image areas through LLC based on self-adaptive codebook |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |