CN107945134B - Image processing method and device - Google Patents

Image processing method and device Download PDF

Info

Publication number
CN107945134B
CN107945134B CN201711238518.9A CN201711238518A CN107945134B CN 107945134 B CN107945134 B CN 107945134B CN 201711238518 A CN201711238518 A CN 201711238518A CN 107945134 B CN107945134 B CN 107945134B
Authority
CN
China
Prior art keywords
image
training
gradient map
neural network
processed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201711238518.9A
Other languages
Chinese (zh)
Other versions
CN107945134A (en
Inventor
杨松
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Xiaomi Mobile Software Co Ltd
Original Assignee
Beijing Xiaomi Mobile Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Xiaomi Mobile Software Co Ltd filed Critical Beijing Xiaomi Mobile Software Co Ltd
Priority to CN201711238518.9A priority Critical patent/CN107945134B/en
Publication of CN107945134A publication Critical patent/CN107945134A/en
Application granted granted Critical
Publication of CN107945134B publication Critical patent/CN107945134B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The present disclosure relates to an image processing method and apparatus, including: according to an image to be processed, obtaining a first gradient map of the image to be processed in an X direction and a second gradient map of the image to be processed in a Y direction, inputting the first gradient map into a first neural network to obtain a processed third gradient map, inputting the second gradient map into a second neural network to obtain a processed fourth gradient map, and performing minimization processing according to the image to be processed, the third gradient map and the fourth gradient map to obtain a result image. According to the method, the minimization processing method, the first neural network and the second neural network are adopted to process the image to be processed to obtain the result image, so that the image processing efficiency is favorably improved, the flaws of the face in the image to be processed can be effectively removed, and the effect of beautifying the skin of the face in the image to be processed is achieved.

Description

Image processing method and device
Technical Field
The present disclosure relates to convolutional neural networks, and more particularly, to an image processing method and apparatus.
Background
CNN (convolutional neural network) is an efficient recognition method that has been developed in recent years and has attracted much attention. CNN is a feedforward neural network whose artificial neurons can respond to a portion of the coverage of surrounding cells. The CNN has excellent performance for processing large-scale images, and along with the popularization of the photographing function of the mobile terminal, the CNN is widely applied to face image beauty algorithms.
In the related art, a face grinding algorithm is the core of a beauty algorithm and is used for removing flaws on the face and beautifying the skin. Most of the current face peeling algorithms are based on edge-preserving filtering methods, such as bilateral filtering and guided filtering. However, these methods are slow and especially the mobile end has limited computational power. If various optimization is carried out on the face buffing algorithm of bilateral filtering to improve the calculation speed, the processing effect is weakened.
Disclosure of Invention
To overcome the problems in the related art, the present disclosure provides an image processing method and apparatus.
According to a first aspect of embodiments of the present disclosure, there is provided an image processing method, including:
obtaining a first gradient map of the image to be processed in the X direction and a second gradient map of the image to be processed in the Y direction according to the image to be processed;
inputting the first gradient map into a first neural network to obtain a processed third gradient map, wherein the first neural network is obtained by fitting the obtained filter;
inputting the second gradient map into a second neural network to obtain a processed fourth gradient map, wherein the second neural network is obtained by fitting the obtained filter;
and performing minimization processing according to the image to be processed, the third gradient map and the fourth gradient map to obtain a result image.
For the above method, in a possible implementation manner, performing minimization processing according to the image to be processed, the third gradient map, and the fourth gradient map to obtain a result image, includes:
the resulting image is determined according to equation 1:
Figure BDA0001489465210000011
wherein R represents the result image, RxA gradient map, R, representing the resulting image in the X directionyRepresenting a gradient map of the resulting image in the X direction, β being a first constant greater than zero, I representing the image to be processed, l'xRepresents the third gradient map, l'yRepresenting the fourth gradient map.
For the above method, in a possible implementation manner, the method further includes:
acquiring a filter and a training image set;
training according to the filter and the training image set to obtain the first neural network;
and training according to the filter and the training image set to obtain the second neural network.
For the above method, in a possible implementation manner, the training according to the filter and the training image set to obtain the first neural network includes:
determining a fifth gradient map of each training image in the training image set in the X direction;
determining a first sample map corresponding to each training image according to each fifth gradient map and the filter;
training by adopting a random gradient descent method according to the formula 2 to obtain a first neural network;
Figure BDA0001489465210000021
wherein, IiRepresenting the I-th training image of the set of training images, the set of training images being { I }1,I2,I3，…，INI is more than or equal to 0 and less than or equal to N, and i and N are positive integers;
F(Ii)xis represented byiA corresponding first image of the sample is shown,
Figure BDA0001489465210000022
is represented byiThe corresponding fifth gradient map;
CNN1(x) A first neural network is represented that is,
Figure BDA0001489465210000023
indicating that the first neural network corresponds to
Figure BDA0001489465210000024
The first output result of (1);
λ represents a second constant greater than zero, represents a third constant greater than zero,
Figure BDA0001489465210000025
representing a loss function for the first output result.
For the above method, in one possible implementation, training the second neural network according to the filter and the training image set includes:
determining a sixth gradient map of each training image in the training image set in the Y direction;
determining a second sample graph corresponding to each training image according to each sixth gradient graph and the filter;
training by adopting a random gradient descent method according to the formula 3 to obtain a second neural network;
Figure BDA0001489465210000026
wherein, IiRepresenting the I-th training image of the set of training images, the set of training images being { I }1,I2,I3，…，INI is more than or equal to 0 and less than or equal to N, and i and N are positive integers;
F(Ii)ya second sample plot corresponding to Ii is shown,
Figure BDA0001489465210000031
represents the sixth gradient map for Ii;
CNN2(x) Representing a second neural network that is,
Figure BDA0001489465210000032
representing a second neural network corresponding to
Figure BDA0001489465210000033
A second output result of (1);
λ represents a second constant greater than zero, represents a third constant greater than zero,
Figure BDA0001489465210000034
representing a loss function for the second output result.
For the above method, in a possible implementation manner, the method further includes:
and changing each convolution kernel in the first neural network and/or the second neural network into a discrete convolution kernel.
According to a second aspect of the embodiments of the present disclosure, there is provided an image processing apparatus including:
the first determining module is used for obtaining a first gradient map of the image to be processed in the X direction and a second gradient map of the image to be processed in the Y direction according to the image to be processed;
the second determination module is used for inputting the first gradient map into a first neural network to obtain a processed third gradient map;
the third determining module is used for inputting the second gradient map into a second neural network to obtain a processed fourth gradient map;
and the fourth determining module is used for performing minimization processing according to the image to be processed, the third gradient map and the fourth gradient map to obtain a result image.
For the apparatus, in a possible implementation manner, the fourth determining module includes:
a fourth determination submodule for determining a resulting image according to equation 1:
Figure BDA0001489465210000035
wherein R represents the result image, RxA gradient map, R, representing the resulting image in the X directionyRepresenting a gradient map of the resulting image in the X direction, β being a first constant greater than zero, I representing the image to be processed, l'xRepresents the third gradient map, l'yRepresenting the fourth gradient map.
For the above apparatus, in a possible implementation manner, the method further includes:
the acquisition module is used for acquiring a filter and a training image set;
the first training module is used for obtaining the first neural network according to the filter and the training image set;
and the second training module is used for training according to the filter and the training image set to obtain the second neural network.
For the apparatus described above, in one possible implementation manner, the first training module includes:
the first training submodule is used for determining a fifth gradient map of each training image in the training image set in the X direction;
the second training submodule is used for determining a first sample map corresponding to each training image according to each fifth gradient map and the filter;
the third training submodule is used for training by adopting a random gradient descent method according to the formula 2 to obtain a first neural network;
Figure BDA0001489465210000041
wherein, IiRepresenting the I-th training image of the set of training images, the set of training images being { I }1,I2,I3，…，INI is more than or equal to 0 and less than or equal to N, and i and N are positive integers;
F(Ii)xa first sample graph corresponding to Ii is shown,
Figure BDA0001489465210000043
represents the fifth gradient map for Ii;
CNN1(x) A first neural network is represented that is,
Figure BDA0001489465210000044
indicating that the first neural network corresponds to
Figure BDA0001489465210000045
The first output result of (1);
λ represents a second constant greater than zero, represents a third constant greater than zero,
Figure BDA0001489465210000046
representing a loss function for the first output result.
For the apparatus described above, in one possible implementation manner, the second training module includes:
a fourth training submodule, configured to determine a sixth gradient map of each training image in the training image set in the Y direction;
a fifth training submodule, configured to determine, according to each of the sixth gradient maps and the filter, a second sample map corresponding to each of the training images;
the sixth training submodule is used for training by adopting a random gradient descent method according to the formula 3 to obtain a second neural network;
Figure BDA0001489465210000042
wherein, IiRepresenting the I-th training image of the set of training images, the set of training images being { I }1,I2,I3，…，INI is more than or equal to 0 and less than or equal to N, and i and N are positive integers;
F(Ii)ya second sample plot corresponding to Ii is shown,
Figure BDA0001489465210000047
represents the sixth gradient map for Ii;
CNN2(x) Representing a second neural network that is,
Figure BDA0001489465210000048
representing a second neural network corresponding to
Figure BDA0001489465210000049
A second output result of (1);
λ represents a second constant greater than zero, represents a third constant greater than zero,
Figure BDA00014894652100000410
representing a loss function for the second output result.
For the above apparatus, in a possible implementation manner, the method further includes:
and the discrete module is used for changing each convolution kernel in the first neural network and/or the second neural network into a discrete convolution kernel.
According to a third aspect of the present disclosure, there is provided an image processing apparatus comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to:
the above method is performed.
According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the above-described method.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects: according to the method, the minimization processing method, the first neural network and the second neural network are adopted to process the image to be processed to obtain the result image, so that the image processing efficiency is favorably improved, the flaws of the face in the image to be processed can be effectively removed, and the effect of beautifying the skin of the face in the image to be processed is achieved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
FIG. 1 is a flow diagram illustrating an image processing method according to an exemplary embodiment.
Fig. 2a is a schematic diagram of an image to be processed in an application example of the present disclosure.
Fig. 2b is a schematic diagram of a result image in an application example of the present disclosure.
FIG. 3 is a flow chart illustrating an image processing method according to one example of an exemplary embodiment.
Fig. 4 is a flowchart illustrating an image processing method according to an example of another exemplary embodiment.
Fig. 5 is a schematic diagram of step 300 in an application example of the present disclosure.
Fig. 6 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.
Fig. 7 is a block diagram of an image processing apparatus according to an example of an exemplary embodiment.
Fig. 8 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
FIG. 1 is a flow diagram illustrating an image processing method according to an exemplary embodiment. The image processing method can be applied to mobile terminals, such as smart phones, tablet computers, notebook computers and the like. As shown in fig. 1, the method includes steps 100 to 103.
In step 100, a first gradient map of the image to be processed in the X direction and a second gradient map of the image to be processed in the Y direction are obtained according to the image to be processed.
In step 101, the first gradient map is input to a first neural network, and a processed third gradient map is obtained.
In step 102, the second gradient map is input to a second neural network to obtain a processed fourth gradient map.
In step 103, a minimization process is performed according to the image to be processed, the third gradient map and the fourth gradient map, so as to obtain a result image.
In one possible implementation, the image to be processed may include a grayscale image or a color image.
Generally, a digital image is an array of discrete pixel points, where the values of each pixel point may constitute a two-dimensional discrete function. The gradient of the image can be represented as a derivation process for the two-dimensional discrete function. It should be noted that, a person skilled in the art may select a suitable image gradient algorithm (e.g., first order difference, second order difference, etc.) to calculate the gradient map of the image to be processed according to needs, and is not limited herein.
As an example of this embodiment, performing minimization processing according to the image to be processed, the third gradient map, and the fourth gradient map to obtain a result image includes: the resulting image is determined according to equation 1.
Figure BDA0001489465210000061
Wherein R represents the result image, RxA gradient map, R, representing the resulting image in the X directionyRepresenting the gradient map of the resulting image in the Y direction, β being a first constant greater than zero (also referred to as a balance factor between different energy terms), I representing the image to be processed, l'xRepresents the third gradient map, l'yRepresenting the fourth gradient map.
In this example, equation 1 finds a result image having the smallest difference from the image to be processed itself and its X, Y-directional gradient by a minimization method, thereby removing a flaw in the image.
According to the method, the minimization processing method, the first neural network and the second neural network are adopted to process the image to be processed to obtain the result image, so that the image processing efficiency is favorably improved, the flaws of the face in the image to be processed can be effectively removed, and the effect of beautifying the skin of the face in the image to be processed is achieved.
In an application example, as shown in fig. 2a, taking an image to be processed as a human face image as an example, the image processing method may include:
(1) for an input face image I, a gradient image I in the x direction and the y direction is obtainedxAnd Iy。
(2) Map of gradients IxInputting into the first CNN network to obtain a processed gradient map Ix′。
(3) Map of gradients IyInputting into a second CNN network to obtain a processed gradient map Iy′。
(4) Gradient plot I in x-and y-directions after treatment according to equation 1x' and Iy' solve the result graph R, see formula 1 above. As shown in fig. 2b, the result map is flawless.
In contrast, the time required for image processing in this example is 20 milliseconds. While the time required to process the image using the bilateral filter of the related art is 9 seconds. Therefore, the method and the device are beneficial to improving the image processing efficiency, can effectively remove the flaws of the face in the image to be processed, and achieve the effect of beautifying the skin of the face in the image to be processed.
FIG. 3 is a flow chart illustrating an image processing method according to one example of an exemplary embodiment. As shown in fig. 3, the difference between fig. 3 and fig. 1 is that step 201 to step 203 are further included before step 100.
In step 201, a filter and a set of training images are acquired.
In step 202, the first neural network is obtained by training according to the filter and the training image set.
In step 203, the second neural network is obtained by training according to the filter and the training image set.
The present disclosure, when solving the problem of improving the efficiency of the original algorithm, does not optimize the original filter (e.g., bilateral filter, edge preserving filter), but fits the original filter. In particular, the present disclosure fits the original filter by using a neural network. The neural network (namely the first neural network and the second neural network) obtained by training of a large amount of data can have the image processing effect consistent with that of the original filter, and has higher calculation efficiency when processing the image compared with the original filter. Therefore, the image processing efficiency can be further improved, the flaws of the human face in the image to be processed can be effectively removed, and the effect of beautifying the skin of the human face in the image to be processed is achieved.
At present, CNNs have many optimized operation frameworks, such as "offset" (Convolutional structure for Fast Feature Embedding), tensierflow (second generation artificial intelligence learning system developed by google), ncnn (open source project for deep learning in Tencent excel laboratories), and the like, and those skilled in the art can select an appropriate neural network to fit an original filter, which is not limited herein.
As an example of this embodiment, training the first neural network according to the filter and the training image set to obtain the first neural network includes: determining a fifth gradient map of each training image in the training image set in the X direction. And determining a first sample map corresponding to each training image according to each fifth gradient map and the filter. And training by adopting a random gradient descent method according to the formula 2 to obtain a first neural network.
Figure BDA0001489465210000071
Wherein, IiRepresenting the I-th training image of the set of training images, the set of training images being { I }1,I2,I3，…，INI is more than or equal to 0 and less than or equal to N, and i and N are positive integers. F (I)i)xIs represented byiA corresponding first image of the sample is shown,
Figure BDA0001489465210000072
is represented byiThe corresponding fifth gradient map. CNN1(x) A first neural network is represented that is,
Figure BDA0001489465210000073
indicating that the first neural network corresponds to
Figure BDA0001489465210000083
The first output result of (1). λ represents a second constant greater than zero, represents a third constant (a very small number) greater than zero,
Figure BDA0001489465210000084
representing for said firstAnd outputting a loss function of the result.
This example obtains the first neural network having the smallest difference from the filter calculation result by the stochastic gradient descent method according to equation 2, thereby enabling the first neural network to maintain the processing effect of the filter. Meanwhile, the first neural network obtained through the neural network frame fitting filter has higher operation efficiency.
As an example of this embodiment, training the second neural network according to the filter and the training image set includes: and determining a sixth gradient map of each training image in the training image set in the Y direction. And determining a second sample image corresponding to each training image according to each sixth gradient image and the filter. And training by adopting a random gradient descent method according to the formula 3 to obtain a second neural network.
Figure BDA0001489465210000081
Wherein, IiRepresenting the I-th training image of the set of training images, the set of training images being { I }1,I2,I3，…，INI is more than or equal to 0 and less than or equal to N, and i and N are positive integers. F (I)i)yA second sample plot corresponding to Ii is shown,
Figure BDA0001489465210000085
representing said sixth gradient map for Ii. CNN2(x) Representing a second neural network that is,
Figure BDA0001489465210000086
representing a second neural network corresponding to
Figure BDA0001489465210000087
And (4) a second output result. λ represents a second constant greater than zero, represents a third constant greater than zero,
Figure BDA0001489465210000088
representing a loss function for the second output result.
The present example obtains the second neural network having the smallest difference from the filter calculation result according to equation 3 by the stochastic gradient descent method, thereby enabling the second neural network to maintain the processing effect of the filter. Meanwhile, the second neural network obtained through the neural network fitting filter has higher operation efficiency.
The method adopts a large amount of training data and a set loss function, and finishes the training of a first CNN network (a second neural network) and a second CNN network (a second neural network) by a random gradient descent method. In addition, the convolution layer in the CNN network is improved, a common compact convolution kernel is changed into a discrete convolution kernel, the receptive field of convolution characteristics is effectively enlarged, and the quality of results is improved. See fig. 5 below for a modification of the convolution kernel.
Fig. 4 is a flowchart illustrating an image processing method according to another exemplary embodiment. As shown in fig. 4, fig. 4 is different from fig. 3 in that a step 300 is further included between step 203 and step 100.
In step 300, each convolution kernel in the first neural network and/or the second neural network is changed to a discrete convolution kernel.
As an example of this embodiment, in a first neural network, a first convolution kernel is applied
Figure BDA0001489465210000082
Becomes the first discrete convolution kernel
Figure BDA0001489465210000091
Wherein, the ajkRepresents the weight corresponding to the jth row and kth column in the first convolution kernel, j ∈ {1,2,3}, k ∈ {1,2,3 }.
As another example of this embodiment, in the second neural network, a second convolution kernel is applied
Figure BDA0001489465210000092
Becomes the second discrete convolution kernel
Figure BDA0001489465210000093
Wherein, theblmRepresenting the weights corresponding to the mth column of the ith row in the second convolution kernel, l ∈ {1,2,3}, m ∈ {1,2,3 }.
Typically, the convolution operation is represented as processing each pixel in the image with a convolution kernel. For example, in one application example, as shown in fig. 5, a first pixel to be processed is represented by a dark gray square, and a second pixel is represented by a light gray square. And carrying out weighted average on each second pixel around the first pixel and the first pixel to obtain a convolution calculation result of the first pixel.
In the related art, pixels are processed using a 3 × 3 original convolution kernel, see the left side in fig. 5. The original convolution kernel performs weighted average on the pixel value (e.g., gray value) of the first pixel and the pixel values (9 pixel values in total) of 8 second pixels around the first pixel to obtain the processed pixel value of the first pixel.
The present disclosure processes pixels with a 5 x 5 discrete convolution kernel, see right side of fig. 5. The discrete convolution kernel performs weighted average on the pixel value (for example, a gray value) of the first pixel and the pixel values (9 pixel values in total) of 8 second pixels around the first pixel to obtain a processed pixel value of the first pixel. The present disclosure changes the original convolution kernel to a discrete form (i.e., a discrete convolution kernel). The 8 second pixels within the window of the discrete convolution kernel are further away from the first pixel and more discrete than the original convolution kernel. In other words, processing the first pixel using the discrete convolution kernel can select a wider range of second pixels to perform a weighted average calculation with the first pixel. Therefore, by adopting the discrete convolution kernel disclosed by the invention, the receptive field of the result value can be effectively increased, and the accuracy is improved.
It should be noted that, those skilled in the art can select a discrete convolution kernel (for example, a 7 × 7 convolution kernel) with an appropriate size as needed, and the present invention is not limited thereto.
Fig. 6 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment. Referring to fig. 6, the apparatus includes:
a first determining module 41, configured to obtain, according to an image to be processed, a first gradient map of the image to be processed in an X direction and a second gradient map of the image to be processed in a Y direction;
a second determining module 42, configured to input the first gradient map into the first neural network, so as to obtain a processed third gradient map;
a third determining module 43, configured to input the second gradient map into a second neural network, so as to obtain a processed fourth gradient map;
a fourth determining module 44, configured to perform minimization processing according to the image to be processed, the third gradient map, and the fourth gradient map, so as to obtain a result image.
According to the method, the minimization processing method, the first neural network and the second neural network are adopted to process the image to be processed to obtain the result image, so that the image processing efficiency is favorably improved, the flaws of the face in the image to be processed can be effectively removed, and the effect of beautifying the skin of the face in the image to be processed is achieved.
Fig. 7 is a block diagram of an image processing apparatus according to an example of an exemplary embodiment. Only the parts relevant to the present embodiment are illustrated in fig. 7. Components in fig. 7 that are numbered the same as those in fig. 6 have the same functions, and detailed descriptions of these components are omitted for the sake of brevity. As shown in figure 7 of the drawings,
for the above apparatus, in a possible implementation manner, the fourth determining module 44 includes:
a fourth determination submodule 441 configured to determine a result image according to equation 1:
Figure BDA0001489465210000101
wherein R represents the result image, RxA gradient map, R, representing the resulting image in the X directionyRepresenting a gradient map of the resulting image in the X direction, β being a first constant greater than zero, I representing the image to be processed, l'xRepresents the third gradient map, l'yRepresenting the fourth gradient map.
For the above apparatus, in a possible implementation manner, the method further includes:
an obtaining module 45, configured to obtain a filter and a training image set;
a first training module 46, configured to train to obtain the first neural network according to the filter and the training image set;
and a second training module 47, configured to train according to the filter and the training image set to obtain the second neural network.
For the above apparatus, in one possible implementation, the first training module 46 includes:
a first training submodule 461, configured to determine a fifth gradient map of each training image in the set of training images in the X direction;
a second training submodule 462, configured to determine a first sample map corresponding to each training image according to each fifth gradient map and the filter;
the third training submodule 463, which is used for training by adopting a stochastic gradient descent method according to the formula 2 to obtain a first neural network;
Figure BDA0001489465210000111
wherein, IiRepresenting the I-th training image of the set of training images, the set of training images being { I }1,I2,I3，…，INI is more than or equal to 0 and less than or equal to N, and i and N are positive integers;
F(Ii)xa first sample graph corresponding to Ii is shown,
Figure BDA0001489465210000113
represents the fifth gradient map for Ii;
CNN1(x) A first neural network is represented that is,
Figure BDA0001489465210000114
indicating that the first neural network corresponds to
Figure BDA0001489465210000115
The first output result of (1);
λ represents greater than zeroA second constant, representing a third constant greater than zero,
Figure BDA0001489465210000116
representing a loss function for the first output result.
For the above apparatus, in a possible implementation manner, the second training module 47 includes:
a fourth training submodule 471, configured to determine a sixth gradient map of each training image in the training image set in the Y direction;
a fifth training sub-module 472, configured to determine, according to each of the sixth gradient maps and the filter, a second sample map corresponding to each of the training images;
a sixth training submodule 473, configured to train to obtain a second neural network by using a random gradient descent method according to equation 3;
Figure BDA0001489465210000112
wherein, IiRepresenting the I-th training image of the set of training images, the set of training images being { I }1,I2,I3，…，INI is more than or equal to 0 and less than or equal to N, and i and N are positive integers;
F(Ii)ya second sample plot corresponding to Ii is shown,
Figure BDA0001489465210000117
represents the sixth gradient map for Ii;
CNN2(x) Representing a second neural network that is,
Figure BDA0001489465210000118
representing a second neural network corresponding to
Figure BDA0001489465210000119
A second output result of (1);
λ represents a second constant greater than zero, represents a third constant greater than zero,
Figure BDA00014894652100001110
representing a loss function for the second output result.
For the above apparatus, in a possible implementation manner, the method further includes:
a discrete module 48, configured to change each convolution kernel in the first neural network and/or the second neural network into a discrete convolution kernel.
According to the method, the minimization processing method, the first neural network and the second neural network are adopted to process the image to be processed to obtain the result image, so that the image processing efficiency is favorably improved, the flaws of the face in the image to be processed can be effectively removed, and the effect of beautifying the skin of the face in the image to be processed is achieved.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Fig. 8 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, and the like.
Referring to fig. 8, the apparatus 800 may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and communication component 816.
The processing component 802 generally controls overall operation of the device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.
The memory 804 is configured to store various types of data to support operations at the apparatus 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
Power components 806 provide power to the various components of device 800. The power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the apparatus 800.
The multimedia component 808 includes a screen that provides an output interface between the device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the device 800 is in an operating mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the apparatus 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.
The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.
The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the device 800. For example, the sensor assembly 814 may detect the open/closed status of the device 800, the relative positioning of components, such as a display and keypad of the device 800, the sensor assembly 814 may also detect a change in the position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration/deceleration of the device 800, and a change in the temperature of the device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 816 is configured to facilitate communications between the apparatus 800 and other devices in a wired or wireless manner. The device 800 may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the apparatus 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.
In an exemplary embodiment, a non-transitory computer-readable storage medium comprising instructions, such as the memory 804 comprising instructions, executable by the processor 820 of the device 800 to perform the above-described method is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (12)

1. An image processing method, comprising:
obtaining a first gradient map of the image to be processed in the X direction and a second gradient map of the image to be processed in the Y direction according to the image to be processed;
inputting the first gradient map into a first neural network to obtain a processed third gradient map;
inputting the second gradient map into a second neural network to obtain a processed fourth gradient map;
performing minimization processing according to the image to be processed, the third gradient map and the fourth gradient map to obtain a result image;
further comprising:
acquiring a filter and a training image set;
training according to the filter and the training image set to obtain the first neural network;
and training according to the filter and the training image set to obtain the second neural network.
2. The method according to claim 1, wherein performing minimization processing according to the image to be processed, the third gradient map and the fourth gradient map to obtain a result image comprises:
the resulting image is determined according to equation 1:
Figure FDA0002613034360000011
wherein R represents the result image, RxA gradient map, R, representing the resulting image in the X directionyRepresenting a gradient map of the resulting image in the X direction, β being a first constant greater than zero, I representing the image to be processed, l'xRepresents the third gradient map, l'yRepresenting the fourth gradient map.
3. The method of claim 1, wherein training the first neural network based on the filter and the set of training images comprises:
determining a fifth gradient map of each training image in the training image set in the X direction;
determining a first sample map corresponding to each training image according to each fifth gradient map and the filter;
training by adopting a random gradient descent method according to the formula 2 to obtain a first neural network;
Figure FDA0002613034360000021
wherein, IiRepresenting the I-th training image of the set of training images, the set of training images being { I }1,I2,I3，…，INI is more than or equal to 0 and less than or equal to N, and i and N are positive integers;
F(Ii)xis represented byiA corresponding first image of the sample is shown,
Figure FDA0002613034360000022
is represented byiThe corresponding fifth gradient map;
CNN1(x) A first neural network is represented that is,
Figure FDA0002613034360000023
indicating that the first neural network corresponds to
Figure FDA0002613034360000024
The first output result of (1);
λ represents a second constant greater than zero, represents a third constant greater than zero,
Figure FDA0002613034360000025
representing a loss function for the first output result.
4. The method of claim 1, wherein training the second neural network based on the filter and the set of training images comprises:
determining a sixth gradient map of each training image in the training image set in the Y direction;
determining a second sample graph corresponding to each training image according to each sixth gradient graph and the filter;
training by adopting a random gradient descent method according to the formula 3 to obtain a second neural network;
Figure FDA0002613034360000026
wherein, IiRepresenting the I-th training image of the set of training images, the set of training images being { I }1,I2,I3，…，INI is more than or equal to 0 and less than or equal to N, and i and N are positive integers;
F(Ii)yis represented byiA corresponding second sample of the graph is shown,
Figure FDA0002613034360000027
is represented byiThe corresponding sixth gradient map;
CNN2(x) Representing a second neural network that is,
Figure FDA0002613034360000028
representing a second neural network corresponding to
Figure FDA0002613034360000029
A second output result of (1);
λ represents a second constant greater than zero, represents a third constant greater than zero,
Figure FDA0002613034360000031
representing a loss function for the second output result.
5. The method of any one of claims 1 to 4, further comprising:
and changing each convolution kernel in the first neural network and/or the second neural network into a discrete convolution kernel.
6. An image processing apparatus characterized by comprising:
the first determining module is used for obtaining a first gradient map of the image to be processed in the X direction and a second gradient map of the image to be processed in the Y direction according to the image to be processed;
the second determination module is used for inputting the first gradient map into a first neural network to obtain a processed third gradient map;
the third determining module is used for inputting the second gradient map into a second neural network to obtain a processed fourth gradient map;
the fourth determining module is used for performing minimization processing according to the image to be processed, the third gradient map and the fourth gradient map to obtain a result image;
further comprising:
the acquisition module is used for acquiring a filter and a training image set;
the first training module is used for obtaining the first neural network according to the filter and the training image set;
and the second training module is used for training according to the filter and the training image set to obtain the second neural network.
7. The apparatus of claim 6, wherein the fourth determining module comprises:
a fourth determination submodule for determining a resulting image according to equation 1:
Figure FDA0002613034360000032
wherein R represents the result image, RxA gradient map, R, representing the resulting image in the X directionyRepresenting a gradient map of the resulting image in the X direction, β being a first constant greater than zero, I representing the image to be processed, l'xRepresents the third gradient map, l'yRepresenting the fourth gradient map.
8. The apparatus of claim 6, wherein the first training module comprises:
the first training submodule is used for determining a fifth gradient map of each training image in the training image set in the X direction;
the second training submodule is used for determining a first sample map corresponding to each training image according to each fifth gradient map and the filter;
the third training submodule is used for training by adopting a random gradient descent method according to the formula 2 to obtain a first neural network;
Figure FDA0002613034360000041
wherein, IiRepresenting the I-th training image of the set of training images, the set of training images being { I }1,I2,I3，…，INI is more than or equal to 0 and less than or equal to N, and i and N are positive integers;
F(Ii)xis represented byiA corresponding first image of the sample is shown,
Figure FDA0002613034360000042
is represented byiThe corresponding fifth gradient map;
CNN1(x) A first neural network is represented that is,
Figure FDA0002613034360000043
indicating that the first neural network corresponds to
Figure FDA0002613034360000044
The first output result of (1);
λ represents a second constant greater than zero, represents a third constant greater than zero,
Figure FDA0002613034360000045
representing a loss function for the first output result.
9. The apparatus of claim 6, wherein the second training module comprises:
a fourth training submodule, configured to determine a sixth gradient map of each training image in the training image set in the Y direction;
a fifth training submodule, configured to determine, according to each of the sixth gradient maps and the filter, a second sample map corresponding to each of the training images;
the sixth training submodule is used for training by adopting a random gradient descent method according to the formula 3 to obtain a second neural network;
Figure FDA0002613034360000051
wherein, IiRepresenting the I-th training image of the set of training images, the set of training images being { I }1,I2,I3，…，INI is more than or equal to 0 and less than or equal to N, and i and N are positive integers;
F(Ii)yis represented byiA corresponding second sample of the graph is shown,
Figure FDA0002613034360000052
is represented byiThe corresponding sixth gradient map;
CNN2(x) Representing a second neural network that is,
Figure FDA0002613034360000053
representing a second neural network corresponding to
Figure FDA0002613034360000054
A second output result of (1);
λ represents a second constant greater than zero, represents a third constant greater than zero,
Figure FDA0002613034360000055
representing a loss function for the second output result.
10. The apparatus of any one of claims 6 to 9, further comprising:
and the discrete module is used for changing each convolution kernel in the first neural network and/or the second neural network into a discrete convolution kernel.
11. An image processing apparatus characterized by comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to: the steps of the method of any one of claims 1 to 5 are performed.
12. A non-transitory computer readable storage medium having instructions which, when executed by a processor, enable the processor to perform the steps of the method according to any one of claims 1 to 5.
CN201711238518.9A 2017-11-30 2017-11-30 Image processing method and device Active CN107945134B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711238518.9A CN107945134B (en) 2017-11-30 2017-11-30 Image processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711238518.9A CN107945134B (en) 2017-11-30 2017-11-30 Image processing method and device

Publications (2)

Publication Number Publication Date
CN107945134A CN107945134A (en) 2018-04-20
CN107945134B true CN107945134B (en) 2020-10-09

Family

ID=61947007

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711238518.9A Active CN107945134B (en) 2017-11-30 2017-11-30 Image processing method and device

Country Status (1)

Country Link
CN (1) CN107945134B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104850836A (en) * 2015-05-15 2015-08-19 浙江大学 Automatic insect image identification method based on depth convolutional neural network
CN105787878A (en) * 2016-02-25 2016-07-20 杭州格像科技有限公司 Beauty processing method and device
CN106228556A (en) * 2016-07-22 2016-12-14 北京小米移动软件有限公司 Image quality analysis method and device
CN106600553A (en) * 2016-12-15 2017-04-26 华中科技大学 DEM super-resolution method based on convolutional neural network
CN106651765A (en) * 2016-12-30 2017-05-10 深圳市唯特视科技有限公司 Method for automatically generating thumbnail by use of deep neutral network
WO2017136802A1 (en) * 2016-02-05 2017-08-10 Nec Laboratories America, Inc. Accelerating deep neural network training with inconsistent stochastic gradient descent

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104850836A (en) * 2015-05-15 2015-08-19 浙江大学 Automatic insect image identification method based on depth convolutional neural network
WO2017136802A1 (en) * 2016-02-05 2017-08-10 Nec Laboratories America, Inc. Accelerating deep neural network training with inconsistent stochastic gradient descent
CN105787878A (en) * 2016-02-25 2016-07-20 杭州格像科技有限公司 Beauty processing method and device
CN106228556A (en) * 2016-07-22 2016-12-14 北京小米移动软件有限公司 Image quality analysis method and device
CN106600553A (en) * 2016-12-15 2017-04-26 华中科技大学 DEM super-resolution method based on convolutional neural network
CN106651765A (en) * 2016-12-30 2017-05-10 深圳市唯特视科技有限公司 Method for automatically generating thumbnail by use of deep neutral network

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Deep Convolutional Neural Network for Decompressed Video Enhancement;R. Lin, Y. Zhang, H. Wang, X. Wang and Q. Dai;《2016 Data Compression Conference (DCC)》;20161219;第50-51页 *
基于RGBD图像和卷积神经网络的快速道路检测;曲磊,王康如,陈利利,李嘉茂,张晓林;《光学学报》;20171031;第37卷(第10期);第124-132页 *
美图如何Show 解读整容式APP的背后;技术宅;《电脑爱好者》;20170101(第1期);第50-51页 *

Also Published As

Publication number Publication date
CN107945134A (en) 2018-04-20

Similar Documents

Publication Publication Date Title
US11532180B2 (en) Image processing method and device and storage medium
CN110084775B (en) Image processing method and device, electronic equipment and storage medium
CN108121952B (en) Face key point positioning method, device, equipment and storage medium
CN108256555B (en) Image content identification method and device and terminal
CN107798669B (en) Image defogging method and device and computer readable storage medium
CN108154465B (en) Image processing method and device
CN111462268A (en) Image reconstruction method and device, electronic equipment and storage medium
CN108668080B (en) Method and device for prompting degree of dirt of lens and electronic equipment
CN106228556B (en) image quality analysis method and device
CN106845398B (en) Face key point positioning method and device
CN107967459B (en) Convolution processing method, convolution processing device and storage medium
CN107133354B (en) Method and device for acquiring image description information
CN111340733B (en) Image processing method and device, electronic equipment and storage medium
CN110580688B (en) Image processing method and device, electronic equipment and storage medium
CN107948510B (en) Focal length adjusting method and device and storage medium
CN112330570B (en) Image processing method, device, electronic equipment and storage medium
CN110458218B (en) Image classification method and device and classification network training method and device
CN111243011A (en) Key point detection method and device, electronic equipment and storage medium
CN109977860B (en) Image processing method and device, electronic equipment and storage medium
CN109784327B (en) Boundary box determining method and device, electronic equipment and storage medium
CN112634160A (en) Photographing method and device, terminal and storage medium
CN108154093B (en) Face information identification method and device, electronic equipment and machine-readable storage medium
CN110619325A (en) Text recognition method and device
CN107730443B (en) Image processing method and device and user equipment
CN111507131B (en) Living body detection method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant