CN112950496A - Image sample set generation method, image sample set generation device and non-volatile computer-readable storage medium - Google Patents
Image sample set generation method, image sample set generation device and non-volatile computer-readable storage medium Download PDFInfo
- Publication number
- CN112950496A CN112950496A CN202110172289.5A CN202110172289A CN112950496A CN 112950496 A CN112950496 A CN 112950496A CN 202110172289 A CN202110172289 A CN 202110172289A CN 112950496 A CN112950496 A CN 112950496A
- Authority
- CN
- China
- Prior art keywords
- image
- fuzzy core
- sub
- blur kernel
- kernel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000012545 processing Methods 0.000 claims abstract description 25
- 230000009466 transformation Effects 0.000 claims description 37
- 238000013519 translation Methods 0.000 claims description 19
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 15
- 238000000844 transformation Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 8
- 125000004122 cyclic group Chemical group 0.000 abstract description 3
- 239000011159 matrix material Substances 0.000 description 17
- 238000004422 calculation algorithm Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 5
- 238000012549 training Methods 0.000 description 5
- 230000001131 transforming effect Effects 0.000 description 5
- 238000009825 accumulation Methods 0.000 description 4
- 230000011218 segmentation Effects 0.000 description 4
- 241001465754 Metazoa Species 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000004091 panning Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000010008 shearing Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/02—Affine transformations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/60—Rotation of whole images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The application provides a generation method and a generation device of an image sample set and a computer readable storage medium. The generation method comprises the following steps: acquiring a first image, wherein the first image has a first resolution; processing a preset first fuzzy core to obtain a second fuzzy core, wherein the type or the fuzzy degree of the second fuzzy core is different from that of the first fuzzy core; performing a convolution operation on the first image by using the first blurring kernel or the second blurring kernel to obtain a second image corresponding to the first image, wherein the second image has a second resolution, the first resolution is higher than the second resolution, the first image and the second image form an image sample pair, circularly performing the convolution operation to obtain a plurality of image sample pairs, and forming an image sample set by using the plurality of image sample pairs. In the generation method, the generation device and the computer-readable storage medium of the image sample set, a large number of image sample pairs can be obtained by performing the cyclic convolution operation on the first image, and the obtaining cost of the image sample pairs is reduced.
Description
Technical Field
The present application relates to the field of deblurring, and more particularly, to a method and an apparatus for generating an image sample set, and a non-volatile computer-readable storage medium.
Background
At present, in the field of deblurring, with the popularization of deep learning algorithms, people begin to use the deep learning algorithms to solve the problem of image blurring. The key point for the successful training of the current deep learning model is on one hand the effective model structure and on the other hand the requirement of a large amount of training sample data. When an effective strong supervision training mode is adopted to train the fuzzy model, paired images (a clear image and a corresponding fuzzy image) are required to be used as a training sample pair, and the two images are required to be aligned pixel by pixel on the content details, so that the acquisition cost of the sample pair is increased. When a large number of image samples meeting requirements do not exist for training the fuzzy model, the fuzzy model is over-fitted, and the trained fuzzy model is difficult to apply.
Disclosure of Invention
The embodiment of the application provides a method and a device for generating an image sample set and a nonvolatile computer readable storage medium.
The method for generating the image sample set according to the embodiment of the application comprises the following steps: acquiring a first image, the first image having a first resolution; processing a preset first fuzzy core to obtain a second fuzzy core, wherein the second fuzzy core is different from the first fuzzy core in type or fuzzy degree; and performing a convolution operation on the first image by using the first blur kernel or the second blur kernel to obtain a second image corresponding to the first image, wherein the second image has a second resolution, the first resolution is higher than the second resolution, the first image and the second image form an image sample pair, the convolution operation is performed in a circulating mode to obtain a plurality of image sample pairs, and the image sample pairs are collected to form an image sample set.
The generation device of the embodiment of the present application includes one or more processors configured to: acquiring a first image, the first image having a first resolution; processing a preset first fuzzy core to obtain a second fuzzy core, wherein the second fuzzy core is different from the first fuzzy core in type or fuzzy degree; and performing a convolution operation on the first image by using the first blur kernel or the second blur kernel to obtain a second image corresponding to the first image, wherein the second image has a second resolution, the first resolution is higher than the second resolution, the first image and the second image form an image sample pair, the convolution operation is performed in a circulating mode to obtain a plurality of image sample pairs, and the image sample pairs are collected to form an image sample set.
The non-transitory computer-readable storage medium of the embodiment of the present application includes a computer program that, when executed by a processor, causes the processor to execute a processing method of: acquiring a first image, the first image having a first resolution; processing a preset first fuzzy core to obtain a second fuzzy core, wherein the second fuzzy core is different from the first fuzzy core in type or fuzzy degree; and performing a convolution operation on the first image by using the first blur kernel or the second blur kernel to obtain a second image corresponding to the first image, wherein the second image has a second resolution, the first resolution is higher than the second resolution, the first image and the second image form an image sample pair, the convolution operation is performed in a circulating mode to obtain a plurality of image sample pairs, and the image sample pairs are collected to form an image sample set.
In the method and the device for generating the image sample set and the non-volatile computer-readable storage medium according to the embodiments of the present application, a convolution operation is performed on a first image by using a first blur kernel or a second blur kernel to generate a second image corresponding to the first image, the first image and the second image form an image sample pair, and a cyclic convolution operation is performed on the first image to obtain a plurality of image sample pairs based on the first image, so that the acquisition cost of the image sample pairs is reduced, and the number of the image sample sets can be increased.
Additional aspects and advantages of embodiments of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of embodiments of the present application.
Drawings
The above and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a flow chart of a method of generating a sample set of images according to some embodiments of the present application;
FIG. 2 is a schematic diagram of the structure of a generating device according to some embodiments of the present application;
FIG. 3 is a schematic diagram of a first blur kernel according to some embodiments of the present application;
FIG. 4 is a schematic illustration of a pair of image samples according to some embodiments of the present application;
FIG. 5 is a flow chart of a method of generating a sample set of images according to some embodiments of the present application;
FIG. 6 is a diagram illustrating a first blur kernel undergoing a scaling transformation to obtain a second blur kernel according to some embodiments of the present application;
FIG. 7 is a diagram illustrating a first blur kernel undergoing a rotation transformation to obtain a second blur kernel according to some embodiments of the present application;
FIG. 8 is a flow chart of a method of generating a sample set of images according to some embodiments of the present application;
FIG. 9 is a schematic diagram of a connection between a non-volatile computer readable storage medium and a processor according to some embodiments of the present application.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below by referring to the drawings are exemplary only for the purpose of explaining the embodiments of the present application, and are not to be construed as limiting the embodiments of the present application.
Referring to fig. 1 to 4, an embodiment of the present application provides a method for generating an image sample set, where the method includes:
01: acquiring a first image P1, the first image P1 having a first resolution;
03: processing a preset first fuzzy core H1 to obtain a second fuzzy core H2, wherein the second fuzzy core H2 is different from the first fuzzy core H1 in type or fuzzy degree; and
05: convolving the first image P1 with the first blur kernel H1 or the second blur kernel H2 to obtain a second image P2 corresponding to the first image P1, the second image P2 having a second resolution, the first resolution being greater than the second resolution, the first image P1 and the second image P2 forming pairs of image samples, circularly performing the convolution operation to obtain a plurality of pairs of image samples, and forming the plurality of sets of pairs of image samples into a set of image samples.
Referring to fig. 2, the present application further provides a generating apparatus 100, where the generating apparatus 100 includes one or more processors 10, and the generating method of the embodiment of the present application can be applied to the generating apparatus 100 of the embodiment of the present application, where one or more processors 10 are configured to execute the method in 01, 03, and 05, that is, one or more processors 10 are configured to: acquiring a first image P1, the first image P1 having a first resolution; processing a preset first fuzzy core H1 to obtain a second fuzzy core H2, wherein the second fuzzy core H2 is different from the first fuzzy core H1 in type or fuzzy degree; and performing a convolution operation on the first image P1 by using the first blurring kernel H1 or the second blurring kernel H2 to obtain a second image P2 corresponding to the first image P1, wherein the second image P2 has a second resolution, the first resolution is higher than the second resolution, the first image P1 and the second image P2 form an image sample pair, the convolution operation is performed in a loop to obtain a plurality of image sample pairs, and the plurality of image sample pairs are collected to form an image sample set.
The image sample set generated herein can be applied to deblurring processing in the image field or other processing that needs to be applied to a pair of sharp and blurred images. The generating apparatus 100 may be any device that needs to use the image sample set, such as a mobile phone, a watch, a notebook computer, a tablet computer, a desktop computer, an intelligent eye, an intelligent helmet, an unmanned aerial vehicle, and the like, without limitation. Referring to fig. 2, the present application only illustrates that the generating device 100 is a mobile phone, and the situation when the generating device 100 is another type of device is similar to that when it is a mobile phone, and is not described in detail. The processor 10 may be integrated in the generating device 100.
At present, the more mature method for making the motion blur removed data set is as follows: the method comprises the steps of shooting a video of a moving picture through a high frame rate (240fps) mode of a camera, accumulating and averaging n continuous frames in the video, correcting gamma to obtain a motion blurred image, and selecting a middle frame in the n frames as a clear frame corresponding to the middle frame to form an image sample pair. The current virtual focus blurred image sample pair is prepared in the following way: the method comprises the steps that a shot object is a static scene, the focusing state of a camera is adjusted manually, one shot is taken when the scene is in focus, and the other shot is taken when the scene is in virtual focus, so that a virtual focus fuzzy image sample pair is obtained.
In the method for generating an image sample set according to the embodiment of the present application, a single first image P1 (sharp image) is acquired, and a convolution operation is performed on the first image P1 by using a first blurring kernel H1 or a second blurring kernel H2 to generate a second image P2 (blurred image) corresponding to the first image P1, wherein the first image P1 and the second image P2 form an image sample pair, and a cyclic convolution operation is performed on the first image P1 to acquire a plurality of image sample pairs based on the first image P1, so as to increase the number of the image sample sets and reduce the acquisition cost of the image sample pairs.
Referring to fig. 3 and 4, in the embodiment of the present application, the predetermined first blur kernel H1 may be designed according to an actual application scenario. The blur kernel is actually a matrix, and the blur kernel is converted into an image by visualizing the matrix, so that the acquisition of the first blur kernel H1 can be obtained by designing the matrix and visualizing the matrix. In particular, different kinds of first blur kernel H1 may be generated by the blur kernel generator, e.g., fig. 3 including 1-15 different kinds of first blur kernels H1. When the first image P1 is convolved with different kinds of first blur kernels H1, different kinds of second images P2 can be obtained, for example, the first blur kernel H1 numbered 1 in fig. 3 convolves the first image P1 to obtain the first kind of second image P2 (numbered 1), the first blur kernel H1 numbered 2 in fig. 3 convolves the first image P1 to obtain the second kind of second image P2 (numbered 2), the first blur kernel H1 numbered 3 in fig. 3 convolves the first image P1 to obtain the third kind of second image P2 (numbered 3), and so on, fifteen kinds of second images P2 can be obtained, numbered from 1 to 15. The first image P1 respectively forms image sample pairs with a plurality of different kinds of second images P2, for example, the first image P1 forms a first image sample pair with the second image P2 numbered 1, the first image P1 forms a second image sample pair with the second image P2 numbered 2, the first image P1 forms a third image sample pair with the second image P2 numbered 3, and so on, fifteen image sample pairs can be obtained. Thus, a greater number of pairs of image samples can be acquired by a single first image P1 to increase the number of image sample sets and reduce the acquisition cost of the image sample pairs. The number of the first blur kernel H1 is not limited to 15 shown in fig. 3, and may be one or more, and may be considered as a set number.
In some embodiments, when the convolution operation is performed on the first image P1 by using the first blur kernel H1 or the second blur kernel H2, the pixel values of each point in the first image P1 form a matrix, and since the blur kernel is actually a matrix, the matrix formed by the second image P2 is obtained by performing a mathematical operation on the two matrices. For example, the first image P1 forms a matrix of
The matrix formed by the first fuzzy core H1 or the second fuzzy core H2 is
The matrix of the second image P2 obtained after the convolution operation is
Wherein, C (1, 1) — 10 × (-1) +10 × 0+10 × 1 ═ 10, that is, matrix a is sequentially moved from left to right, from top to bottom row by row, and divided into matrices of the same size as matrix B, and then matrix-multiplied by matrix B and added to obtain corresponding element values in matrix C.
Referring to fig. 5, in some embodiments, method 03: processing the preset first fuzzy core H1 to obtain a second fuzzy core H2, including:
031: performing one or more of translation, scaling, rotation, or affine transformation on the first blur kernel H1 to obtain an intermediate blur kernel; and
033: and calculating the point values of the second fuzzy core H2 according to the point values of the intermediate fuzzy core and the point values of the first fuzzy core H1 to obtain a second fuzzy core H2.
Referring to fig. 2, one or more processors 10 are also configured to execute 031, 033, i.e., one or more processors 10 are configured to: performing one or more of translation, scaling, rotation, or affine transformation on the first blur kernel H1 to obtain an intermediate blur kernel; and calculating the point values of the second fuzzy core H2 according to the point values of the intermediate fuzzy core and the point values of the first fuzzy core H1 to obtain a second fuzzy core H2.
In particular, the one or more processors 10 may implement any one or more of translation, scaling, rotation, or affine transformation on a preset first blur kernel H1 to obtain more kinds of intermediate blur kernels with different degrees of blur, and computing different kinds of intermediate blur kernels with different degrees of blur may obtain a second blur kernel H2 with different kinds of blur or different degrees of blur. Therefore, the types of the blur kernels are increased, the data volume of the image sample set is increased more easily, and various types of blur (such as motion blur, virtual focus blur and lens blur) can be simulated.
In some embodiments, the one or more processors 10 perform any one of translation, scaling, rotation, or emulation on a preset first blur kernel H1 to obtain an intermediate blur kernel, and obtain point values of the second blur kernel H2 according to the point values of the intermediate blur kernel and the point values of the first blur kernel H1, so as to obtain a second blur kernel H2 corresponding to the first blur kernel H1.
In one embodiment, the one or more processors 10 perform a panning transformation on a preset first blur kernel H1 to obtain an intermediate blur kernel corresponding to the first blur kernel H1. Specifically, the first blurring kernel H1 includes a plurality of pixels, and the first blurring kernel H1 performs a translation transformation such that each pixel of the first blurring kernel H1 is translated from an initial position to an end position by a given offset, and finally, a gray value of each pixel of the intermediate blurring kernel is obtained through an interpolation algorithm. For example, the coordinate of a certain pixel in the first blurring kernel H1 in space is (x)0，y0) The first blur kernel H1 is translated in the horizontal direction by xtIs translated in the vertical directiontThe pixel coordinate of the pixel to be checked by the intermediate blur kernel obtained after the translation is (x)0+xt，y0+yt). The pixel points in the first blur kernel H1 that are moved out of the image display area can be increased by increasing the size of the intermediate blur kernel (e.g., the image width of the intermediate blur kernel is increased by T)xIncrease in height Ty) To ensure that the intermediate blur kernel can contain all the pixel points in the first blur kernel H1. And finally, carrying out gray level interpolation processing on the intermediate fuzzy core to ensure that pixel points of the intermediate fuzzy core obtained after translation can correspond to pixel points in the first fuzzy core H1. The interpolation algorithm comprises a nearest neighbor interpolation method, a bilinear interpolation method and a bicubic interpolation method. After the intermediate blur kernel is obtained, the numerical values of the points of the intermediate blur kernel are divided by the accumulated value of the numerical values of the points of the first blur kernel H1 to obtain a second blur kernel H2 corresponding to the intermediate blur kernel, so that the numerical accumulation of the points of the obtained second blur kernel H2 is equal to 1, and the brightness of the second blur kernel H2 obtained by transforming the first blur kernel H1 is consistent with the brightness of the first blur kernel H1.
Referring to fig. 6, in one example, one or more processors 10 perform a scaling transform on a preset first blur core H1 to obtain an intermediate blur core corresponding to the first blur core H1. Specifically, the first blurring kernel H1 includes a plurality of pixels, and the first blurring kernel H1 performs a scaling transform including a magnification transform or a reduction transformThat is, the first blur kernel H1 is scaled up or down by a certain scale, and finally, the gray value of each pixel of the intermediate blur kernel is obtained through an interpolation algorithm. For example, the coordinate of a certain pixel in the first blurring kernel H1 in space is (x)0，y0) Scaling s in the horizontal direction by scaling the first blur kernel H1 centered at an arbitrary point (x, y)xMultiple, scaling s in the vertical directionyMultiplying, then the scaled coordinate of the point pixel is (x)0+sx(x-x0)，y0+sy(y-y0). Similarly, for pixels in the first blur kernel H1 that are moved out of the image display area, the size of the intermediate blur kernel may be increased (e.g., the image width of the intermediate blur kernel is increased by T)xIncrease in height Ty) To ensure that the intermediate blur kernel can contain all the pixel points in the first blur kernel H1. And finally, carrying out gray level interpolation processing on the intermediate fuzzy core to ensure that all pixel points of the intermediate fuzzy core obtained after zooming can correspond to the pixel points in the first fuzzy core H1. The interpolation algorithm comprises a nearest neighbor interpolation method, a bilinear interpolation method and a bicubic interpolation method. After the intermediate blur kernel is obtained, the numerical values of each point of the intermediate blur kernel are divided by the accumulated value of the numerical values of each point of the first blur kernel H1 to obtain a second blur kernel H2 corresponding to the intermediate blur kernel, so that the numerical accumulation of each point of the obtained second blur kernel H2 is equal to 1, and the brightness of the second blur kernel H2 obtained by transforming the first blur kernel H1 is consistent with the brightness of the first blur kernel H1.
Referring to fig. 7, in one embodiment, the one or more processors 10 perform a rotation transformation on a preset first blur kernel H1 to obtain an intermediate blur kernel corresponding to the first blur kernel H1. Specifically, the first blur kernel H1 includes a plurality of pixels, and the rotation transformation performed by the first blur kernel H1 is that each pixel point in the first blur kernel H1 rotates around a certain point as a center by a certain angle, and finally, the gray value of each pixel of the intermediate blur kernel is obtained through an interpolation algorithm. For example, the coordinate of a certain pixel in the first blurring kernel H1 in space is (x)0，y0) The corresponding point of the first blur kernel H1 after being rotated counterclockwise by an angle θ about a point is (x, y), where(x, y) can be calculated by the following matrix expression.
Similarly, for pixels in the first blur kernel H1 that are moved out of the image display area, the size of the intermediate blur kernel may be increased (e.g., the image width of the intermediate blur kernel is increased by T)xIncrease in height Ty) The interpolation method includes a nearest neighbor interpolation method, a bilinear interpolation method and a bicubic interpolation method. The certified intermediate blur kernel can contain all the pixel points in the first blur kernel H1. And finally, carrying out gray level interpolation processing on the intermediate fuzzy core to ensure that all pixel points of the intermediate fuzzy core obtained after zooming can correspond to the pixel points in the first fuzzy core H1. After the intermediate blur kernel is reached, dividing the numerical values of each point of the intermediate blur kernel by the accumulated value of the numerical values of each point of the first blur kernel H1 to obtain a second blur kernel H2 corresponding to the intermediate blur kernel, so as to ensure that the numerical accumulation of each point of the obtained second blur kernel H2 is equal to 1, and further ensure that the brightness of the second blur kernel H2 obtained by transforming the first blur kernel H1 is consistent with the brightness of the first blur kernel H1.
Referring to fig. 2, in one embodiment, the one or more processors 10 perform affine transformation on a preset first blur kernel H1 to obtain an intermediate blur kernel corresponding to the first blur kernel H1. Specifically, the first blur kernel H1 includes a plurality of pixels, and the affine transformation performed by the first blur kernel H1 is that each pixel point in the first blur kernel H1 is subjected to linear (e.g., translation, scaling, rotation, miscut, etc.) transformation once and added with a translation, and finally, the gray value of each pixel of the intermediate blur kernel is obtained through an interpolation algorithm. For example, the coordinate of a certain pixel in the first blurring kernel H1 in space is (x)0，y0) Affine matrices are 2-by-3 matrices, e.g. matrices
Wherein the elements of the third column play a translational role, the diagonal of the elements of the first two columns play a zooming role, and the anti-diagonal plays a roleWith a rotating or shearing action. A certain pixel (x) in the first blur kernel H10，y0) And (3) obtaining an intermediate fuzzy core corresponding point after affine matrix A transformation, wherein the (x, y) can be calculated by the following matrix expression.
Similarly, for pixels in the first blur kernel H1 that are moved out of the image display area, the size of the intermediate blur kernel may be increased (e.g., the image width of the intermediate blur kernel is increased by T)xIncrease in height Ty) To ensure that the intermediate blur kernel can contain all the pixel points in the first blur kernel H1. And finally, carrying out gray level interpolation processing on the intermediate fuzzy core to ensure that all pixel points of the intermediate fuzzy core obtained after zooming can correspond to the pixel points in the first fuzzy core H1. The interpolation algorithm comprises a nearest neighbor interpolation method, a bilinear interpolation method and a bicubic interpolation method. After the intermediate blur kernel is obtained, the numerical values of each point of the intermediate blur kernel are divided by the accumulated value of the numerical values of each point of the first blur kernel H1 to obtain a second blur kernel H2 corresponding to the intermediate blur kernel, so that the numerical accumulation of each point of the obtained second blur kernel H2 is equal to 1, and the brightness of the second blur kernel H2 obtained by transforming the first blur kernel H1 is consistent with the brightness of the first blur kernel H1.
Referring to fig. 2, in some embodiments, the one or more processors 10 perform a plurality of operations of translating, scaling, rotating or emulating the preset first blur kernel H1 to obtain the intermediate blur kernel. For example, the one or more processors 10 sequentially perform panning and scaling transformations on a preset first blur kernel H1 to obtain intermediate blur kernels; for another example, the one or more processors 10 sequentially perform panning and retranslating transforms on the preset first blur kernel H1 to obtain intermediate blur kernels. That is, when the one or more processors 10 may perform any combination of translation, scaling, rotation, and affine transformation on the preset first blur kernel H1 to obtain intermediate blur kernels with more blur types or blur degrees, so as to obtain more second blur kernels H2 corresponding to the first blur kernel H1. Various transformation principles are consistent with the above description, and are not described in detail herein.
Referring to fig. 2 and 4, in the embodiment of the present application, the types of the first blur kernel H1 include multiple types, and the one or more processors 10 may randomly select a predetermined first blur kernel H1 to perform one or more transformations of translation, scaling, rotation, or affine transformation, so as to obtain a second blur kernel H2 with a greater variety or blur degree different from that of the first blur kernel H1. When performing a convolution operation on the first image P1, the one or more processors 10 randomly select the first blurring kernel H1 or the second blurring kernel H2 to perform blurring processing on the first image P1, so that the second image P2 with different blurring types or blurring degrees can be obtained only by performing a single first image P1, thereby obtaining multiple pairs of image sample pairs, and performing a circular convolution operation on the first image P1 can further increase the number of the image sample pairs, so that the number of the image sample sets is richer. Meanwhile, one or more processors 10 perform convolution operations by acquiring different first images P1 and then using different first blurring kernels H1 or second blurring kernels H2 for different first images P1, so that scenes of the image sample set are richer.
Referring to FIG. 2, in some embodiments, one or more processors 10 perform multiple transformations on each first obfuscated core H1, in the same or different manners.
When the one or more processors 10 perform multiple transformations on each first blur core H1, the manner of the multiple transformations may be the same. For example, the one or more processors 10 may perform two, three, or more times of translation or scaling or rotation or affine transformation on each first blur kernel H1, and may obtain the second blur kernels H2 having different blur types or blur degrees by transforming the first blur kernel H1 a plurality of times to obtain intermediate blur kernels, thereby increasing the types of blur kernels for the convolution operation of the first image P1 to increase the types of image sample pairs.
When the one or more processors 10 perform multiple transformations on each first blur core H1, the manner of the multiple transformations may be different. For example, when the number of transformations is two, the one or more processors 10 perform a translation on the first blur kernel H1, then may perform one or more of scaling, rotation, and emulation on the transformed first blur kernel H1 to obtain an intermediate blur kernel, and finally perform a calculation on the intermediate blur kernel to obtain a second blur kernel H2 with an image brightness consistent with the first blur kernel H1. For another example, when the number of times of transformation is three or more, the one or more processors 10 may perform translation on the first blur kernel H1, perform any one of scaling, rotation, and affine transformation on the transformed first blur kernel H1, and finally perform one or more of translation, scaling, rotation, and reflection modeling on the twice transformed first blur kernel H1 to obtain an intermediate blur kernel, and finally perform calculation on the intermediate blur kernel to obtain the second blur kernel H2 whose image brightness is consistent with the first blur kernel H1. Of course, when one or more processors 10 perform multiple transformations on each first obfuscation kernel H1, the way of the multiple transformations may be partially different and partially the same.
Referring to fig. 8, in some embodiments, the generating method may further include:
02: processing the first image P1 to obtain at least one first sub-image, the first sub-image including a subject to be blurred;
05: performing a convolution operation on the first image P1 by using the first blurring kernel H1 or the second blurring kernel H2 to obtain a second image P2 corresponding to the first image P1, including:
051: performing convolution operation on the first sub-image by using the first blurring kernel H1 or the second blurring kernel H2 to obtain a second sub-image; and
053: the second sub-image and the first image P1 are fused to obtain a second image P2.
Referring to fig. 2, one or more processors 10 are also configured to perform the methods of 02, 051, 053, i.e., one or more processors 10 are further configured to: processing the first image P1 to obtain at least one first sub-image, the first sub-image including a subject to be blurred; performing convolution operation on the first sub-image by using the first blurring kernel H1 or the second blurring kernel H2 to obtain a second sub-image; and fusing the second sub-image and the first image P1 to obtain a second image P2.
Referring to FIG. 4, in one embodiment, to simulate a non-uniform blur of the full map of the first image P1, i.e., the full map corresponds to a plurality of different types of blur kernels. In one example, the first image P1 is subjected to image segmentation to obtain at least one first sub-image, where the number of the first sub-images may include one, two, three or more, and the specific number of the first sub-images is related to the number of the subject to be blurred contained in the first image P1. For example, the first image P1 includes three subjects to be blurred, that is, when the first image P1 is blurred, three different types of blur kernels need to be used for performing convolution operation, so that the second image P2 is a blurred image including three different types of blur kernels, and the image generation apparatus 100 (shown in fig. 2) can generate various types of blur generated by various types of targets in more real scenes, so that the blur model trained through deep learning does not have a condition of overfitting, and the trained blur types have high practicability.
In particular, segmenting the first image P1 may obtain at least one first sub-image by a threshold segmentation method. The threshold segmentation method is obtained by the following transformation formula:
where T is the threshold, g is the first sub-image, f is the first image P1, g (i, j) is 1 for the image of the object and g (i, j) is 0 for the image element of the background. The determination of the threshold T may be segmented by a global threshold, an adaptive threshold, an optimal threshold. For example, when the background and foreground portions of the first image P1 are significantly contrasted, a global threshold may be used for segmentation. For another example, when the contrast of the object and the background of the first image P1 is different everywhere in the first image P1, segmentation may be performed with different thresholds according to the local features of the first image P1. Preferably, the first image P1 may be segmented using adaptive thresholds, i.e. different thresholds are selected for different regions of the first image P1 or the threshold at each point is dynamically selected according to a certain neighborhood range for image segmentation.
After the first image P1 is divided into a plurality of first sub-images, the first sub-images are convolved by the one or more processors 10 (shown in fig. 2) by using different types of first blur kernel H1 or second blur kernel H2 to obtain second sub-images corresponding to the first sub-images, and finally the second sub-images and the divided first image P1 are subjected to a fitting process to obtain a second image P2 corresponding to the first image P1. The specific convolution operation is consistent with the above-mentioned convolution operation principle, and is not described herein again.
In some embodiments, the first sub-image comprises a plurality of different first sub-images, different first sub-images being convolved with different first blurring kernel H1 or second blurring kernel H2; the subject to be blurred in the first sub-images includes a plurality of categories, and the first sub-images of the subject to be blurred with the same category are convolved by using the same first blurring kernel H1 or the second blurring kernel H2.
For example, the first image P1 is divided into a plurality of different first sub-images, which may be convolved with different first blurring kernels H1 or second blurring kernels H2, so that the resulting second image P2 (blurred image) more closely resembles the type of blur in a real scene, e.g. moving classes of objects in the street, from which scene the first image P1 (sharp image) was taken, due to the different motion trajectories of different objects in the scene, the first image P1 (sharp image) needs to be segmented into a plurality of different first sub-images, and at the same time, convolving the first sub-image with a different first blur kernel H1 or second blur kernel H2 to obtain a second image P2 (blurred image) that corresponds more closely to the first image P1 (sharp image) in the scene, such that pairs of image samples in the set of image samples have pairs of motion blurred samples in the real scene.
For another example, the first image P1 is divided into a plurality of different first sub-images, where the subject to be blurred in the plurality of first sub-images includes a plurality of categories (e.g., categories including human, animal, plant, etc.), where both the subject to be blurred in the two divided first sub-images are stationary plants, the two first sub-images may be convolved by using the same kind of first blur kernel H1 or second blur kernel H2, and the person or other animal may be convolved by using another kind of first blur kernel H1 or second blur kernel H2, so that the second image P2 (blurred image) is closer to a blurred image of a scene image (including human, animal, plant) actually captured, so that the image samples in the image sample set have motion blurred sample pairs in a real scene.
Referring to fig. 9, the present application provides a non-volatile computer-readable storage medium 200 comprising 201. The computer program 201, when executed by the one or more processors 10, causes the one or more processors 10 to perform the generation methods in 01, 02, 03, 031, 033, 05, 051, 053.
For example, the computer program 201, when executed by the one or more processors 10, causes the processors 10 to perform the generation method of:
01: acquiring a first image P1, the first image P1 having a first resolution;
03: processing a preset first fuzzy core H1 to obtain a second fuzzy core H2, wherein the second fuzzy core H2 is different from the first fuzzy core H1 in type or fuzzy degree; and
05: convolving the first image P1 with the first blur kernel H1 or the second blur kernel H2 to obtain a second image P2 corresponding to the first image P1, the second image P2 having a second resolution, the first resolution being greater than the second resolution, the first image P1 and the second image P2 forming pairs of image samples, circularly performing the convolution operation to obtain a plurality of pairs of image samples, and forming the plurality of sets of pairs of image samples into a set of image samples.
As another example, the computer program 201, when executed by the one or more processors 10, causes the processors 10 to perform the generation method as follows:
01: acquiring a first image P1, the first image P1 having a first resolution;
031: performing one or more of translation, scaling, rotation, or affine transformation on the first blur kernel H1 to obtain an intermediate blur kernel;
033: calculating the numerical values of all points of a second fuzzy core H2 according to the numerical values of all points of the intermediate fuzzy core and the numerical values of all points of the first fuzzy core H1 to obtain a second fuzzy core H2;
05: convolving the first image P1 with the first blur kernel H1 or the second blur kernel H2 to obtain a second image P2 corresponding to the first image P1, the second image P2 having a second resolution, the first resolution being greater than the second resolution, the first image P1 and the second image P2 forming pairs of image samples, circularly performing the convolution operation to obtain a plurality of pairs of image samples, and forming the plurality of sets of pairs of image samples into a set of image samples.
As another example, the computer program 201, when executed by the one or more processors 10, causes the processors 10 to perform the generation method as follows:
01: acquiring a first image P1, the first image P1 having a first resolution;
02: processing the first image P1 to obtain at least one first sub-image, the first sub-image including a subject to be blurred;
03: processing a preset first fuzzy core H1 to obtain a second fuzzy core H2, wherein the second fuzzy core H2 is different from the first fuzzy core H1 in type or fuzzy degree;
051: performing convolution operation on the first sub-image by using the first blurring kernel H1 or the second blurring kernel H2 to obtain a second sub-image; and
053: the second sub-image and the first image P1 are fused to obtain a second image P2.
Also for example, the computer program 201, when executed by the one or more processors 10, causes the processors 10 to perform the generation method of:
01: acquiring a first image P1, the first image P1 having a first resolution;
02: processing the first image P1 to obtain at least one first sub-image, the first sub-image including a subject to be blurred;
031: performing one or more of translation, scaling, rotation, or affine transformation on the first blur kernel H1 to obtain an intermediate blur kernel;
033: calculating the numerical values of all points of a second fuzzy core H2 according to the numerical values of all points of the intermediate fuzzy core and the numerical values of all points of the first fuzzy core H1 to obtain a second fuzzy core H2;
051: performing convolution operation on the first sub-image by using the first blurring kernel H1 or the second blurring kernel H2 to obtain a second sub-image; and
053: the second sub-image and the first image P1 are fused to obtain a second image P2.
In the description herein, references to the description of the terms "certain embodiments," "one example," "exemplary," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
Although embodiments of the present application have been shown and described above, it is to be understood that the above embodiments are exemplary and not to be construed as limiting the present application, and that changes, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.
Claims (11)
1. A method for generating a sample set of images, comprising:
acquiring a first image, the first image having a first resolution;
processing a preset first fuzzy core to obtain a second fuzzy core, wherein the second fuzzy core is different from the first fuzzy core in type or fuzzy degree; and
performing a convolution operation on the first image by using the first blur kernel or the second blur kernel to obtain a second image corresponding to the first image, wherein the second image has a second resolution, the first resolution is greater than the second resolution, the first image and the second image form an image sample pair, performing the convolution operation in a loop to obtain a plurality of image sample pairs, and forming an image sample set by using a plurality of image sample pairs.
2. The generation method according to claim 1, wherein the processing a preset first blur kernel to obtain a second blur kernel comprises:
performing one or more of translation, scaling, rotation, or affine transformation on the first blur kernel to obtain an intermediate blur kernel; and
and calculating the numerical values of all points of the second fuzzy core according to the numerical values of all points of the middle fuzzy core and the numerical values of all points of the first fuzzy core so as to obtain the second fuzzy core.
3. The generation method according to claim 2, characterized in that a plurality of transformations, in the same or different way, is performed on each of the first blur kernels.
4. The generation method according to claim 1, characterized in that the generation method further comprises:
processing the first image to obtain at least one first sub-image, wherein the first sub-image comprises a subject to be blurred;
performing a convolution operation on the first image by using the first blur kernel or the second blur kernel to obtain a second image corresponding to the first image, including:
performing convolution operation on the first sub-image by using the first fuzzy core or the second fuzzy core to obtain a second sub-image; and
and fusing the second sub-image and the first image to acquire a second image.
5. The generation method according to claim 4, wherein the first sub-image includes a plurality of sub-images, and different first sub-images are convolved with different first blurring kernels or second blurring kernels; or
The to-be-blurred objects in the plurality of first sub-images comprise a plurality of categories, and the first sub-images of the to-be-blurred objects with the same category are subjected to convolution operation by adopting the same first blurring kernel or the same second blurring kernel.
6. A generation apparatus for a sample set of images, comprising one or more processors configured to:
acquiring a first image, the first image having a first resolution;
processing a preset first fuzzy core to obtain a second fuzzy core, wherein the second fuzzy core is different from the first fuzzy core in type or fuzzy degree; and
performing a convolution operation on the first image by using the first blur kernel or the second blur kernel to obtain a second image corresponding to the first image, wherein the second image has a second resolution, the first resolution is greater than the second resolution, the first image and the second image form an image sample pair, performing the convolution operation in a loop to obtain a plurality of image sample pairs, and forming an image sample set by using a plurality of image sample pairs.
7. The generation apparatus of claim 6, wherein the one or more processors are further configured to:
performing one or more of translation, scaling, rotation, or affine transformation on the first blur kernel to obtain an intermediate blur kernel; and
and calculating the numerical values of all points of the second fuzzy core according to the numerical values of all points of the middle fuzzy core and the numerical values of all points of the first fuzzy core so as to obtain the second fuzzy core.
8. The generation apparatus of claim 7, wherein the one or more processors are further configured to: and executing a plurality of transformations on each first fuzzy core, wherein the plurality of transformations are the same or different in mode.
9. The generation apparatus of claim 6, wherein the one or more processors are further configured to:
processing the first image to obtain at least one first sub-image, wherein the first sub-image comprises a subject to be blurred;
performing convolution operation on the first sub-image by using the first fuzzy core or the second fuzzy core to obtain a second sub-image; and
and fusing the second sub-image and the first image to acquire a second image.
10. The generation apparatus according to claim 9, wherein the first sub-image includes a plurality,
different first sub-images adopt different first blurring kernels or second blurring kernels to perform convolution operation; or
And performing convolution operation on the first sub-images of the subjects to be blurred with the same category by using the same first blurring kernel or the second blurring kernel.
11. A non-transitory computer-readable storage medium containing a computer program, wherein the computer program, when executed by a processor, causes the processor to perform the generation method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110172289.5A CN112950496A (en) | 2021-02-08 | 2021-02-08 | Image sample set generation method, image sample set generation device and non-volatile computer-readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110172289.5A CN112950496A (en) | 2021-02-08 | 2021-02-08 | Image sample set generation method, image sample set generation device and non-volatile computer-readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112950496A true CN112950496A (en) | 2021-06-11 |
Family
ID=76244173
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110172289.5A Pending CN112950496A (en) | 2021-02-08 | 2021-02-08 | Image sample set generation method, image sample set generation device and non-volatile computer-readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112950496A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170013248A1 (en) * | 2015-07-07 | 2017-01-12 | Canon Kabushiki Kaisha | Image generating apparatus, image generating method, and image generating program |
| CN109727201A (en) * | 2017-10-30 | 2019-05-07 | 富士通株式会社 | Information processing equipment, image processing method and storage medium |
| CN110189251A (en) * | 2019-06-04 | 2019-08-30 | 厦门美图之家科技有限公司 | A kind of blurred picture generation method and device |
-
2021
- 2021-02-08 CN CN202110172289.5A patent/CN112950496A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170013248A1 (en) * | 2015-07-07 | 2017-01-12 | Canon Kabushiki Kaisha | Image generating apparatus, image generating method, and image generating program |
| CN109727201A (en) * | 2017-10-30 | 2019-05-07 | 富士通株式会社 | Information processing equipment, image processing method and storage medium |
| CN110189251A (en) * | 2019-06-04 | 2019-08-30 | 厦门美图之家科技有限公司 | A kind of blurred picture generation method and device |
Non-Patent Citations (3)
| Title |
|---|
| 毛勇: ""模糊车牌图像的盲去模糊算法研究"", 《中国优秀硕士学位论文全文数据库（信息科技辑）》, pages 31 * |
| 胡绍海 等: ""基于区域检测分割的运动模糊图像复原"", 《北京交通大学学报》, vol. 43, no. 05, pages 20 - 26 * |
| 马少勇: ""基于生成式对抗网络的运动模糊图像复原研究"", 《中国优秀硕士学位论文全文数据库（信息科技辑）》, pages 6 - 9 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhang et al. | Deep image deblurring: A survey | |
| Bao et al. | Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement | |
| US12008797B2 (en) | Image segmentation method and image processing apparatus | |
| Li et al. | Efficient and interpretable deep blind image deblurring via algorithm unrolling | |
| US20220222776A1 (en) | Multi-Stage Multi-Reference Bootstrapping for Video Super-Resolution | |
| CN111275626B (en) | Video deblurring method, device and equipment based on ambiguity | |
| Wang et al. | Deeplens: Shallow depth of field from a single image | |
| Yang et al. | Real-world video super-resolution: A benchmark dataset and a decomposition based learning scheme | |
| Dai et al. | Sparse representation-based multiple frame video super-resolution | |
| CN112837245A (en) | Dynamic scene deblurring method based on multi-mode fusion | |
| KR101671391B1 (en) | Method for deblurring video using modeling blurred video with layers, recording medium and device for performing the method | |
| Brehm et al. | High-resolution dual-stage multi-level feature aggregation for single image and video deblurring | |
| Li et al. | A maximum a posteriori estimation framework for robust high dynamic range video synthesis | |
| Su et al. | Super-resolution without dense flow | |
| Dutta | Depth-aware blending of smoothed images for bokeh effect generation | |
| CN112995678B (en) | Video motion compensation method and device and computer equipment | |
| Guan et al. | Srdgan: learning the noise prior for super resolution with dual generative adversarial networks | |
| Conde et al. | Lens-to-lens bokeh effect transformation. NTIRE 2023 challenge report | |
| Wang et al. | Joint framework for single image reconstruction and super-resolution with an event camera | |
| Yang et al. | SRDN: A unified super-resolution and motion deblurring network for space image restoration | |
| Borsoi et al. | A new adaptive video super-resolution algorithm with improved robustness to innovations | |
| Tang et al. | Structure-embedded ghosting artifact suppression network for high dynamic range image reconstruction | |
| Sharif et al. | DarkDeblur: Learning single-shot image deblurring in low-light condition | |
| CN112801890B (en) | Video processing method, device and equipment | |
| CN106846250A (en) | A kind of super resolution ratio reconstruction method based on multi-scale filtering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210611 |
|
| WD01 | Invention patent application deemed withdrawn after publication |