CN111275646A - Edge-preserving image smoothing method based on deep learning knowledge distillation technology - Google Patents
Edge-preserving image smoothing method based on deep learning knowledge distillation technology Download PDFInfo
- Publication number
- CN111275646A CN111275646A CN202010066280.1A CN202010066280A CN111275646A CN 111275646 A CN111275646 A CN 111275646A CN 202010066280 A CN202010066280 A CN 202010066280A CN 111275646 A CN111275646 A CN 111275646A
- Authority
- CN
- China
- Prior art keywords
- image
- edge
- network
- structural
- teacher
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003706 image smoothing Methods 0.000 title claims abstract description 38
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000005516 engineering process Methods 0.000 title claims abstract description 20
- 238000013135 deep learning Methods 0.000 title claims abstract description 18
- 238000013140 knowledge distillation Methods 0.000 title claims abstract description 18
- 238000003708 edge detection Methods 0.000 claims abstract description 8
- 238000013528 artificial neural network Methods 0.000 claims description 13
- 238000012549 training Methods 0.000 claims description 9
- 230000006870 function Effects 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 4
- 238000013508 migration Methods 0.000 claims description 3
- 230000005012 migration Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 claims description 3
- 230000014759 maintenance of location Effects 0.000 claims description 2
- 238000012545 processing Methods 0.000 abstract description 6
- 230000000007 visual effect Effects 0.000 abstract description 2
- 230000000694 effects Effects 0.000 abstract 1
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000004821 distillation Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000003796 beauty Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
An edge-preserving image smoothing method based on a deep learning knowledge distillation technology belongs to the technical field of image processing. The method is based on a novel deep learning knowledge distillation technology, and the edge detection capability of an edge detection teacher network is distilled into an image smoothing student network, so that the student network has the image smoothing capability of edge protection. The task of the student network is to perform image smoothing, and the task of the teacher network is to extract information of structural edges. The teacher network is used for distilling and transferring the structural edge knowledge to the student network, so that the student network has the capability of keeping the structural edge of the teacher network while performing an image smoothing task. The method can keep the smooth structural edge of the image on the premise of not additionally introducing a depth network for keeping the edge information, overcomes the visual influence of image quality loss such as poor image edge information and blurring in the traditional image smoothness enhancement method, and greatly improves the image smoothing effect.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to an edge-preserving image smoothing method based on a novel deep learning knowledge distillation technology.
Background
With the rapid development of multimedia technology and the wide demand of the industry in recent years, the application scenarios of image processing technology are expanding. The existing image can be divided into fusion of a global structure and a local texture, and the texture of the image can influence the automatic extraction of image structure information to a certain extent. In addition, due to the requirement of people on the photographing technology, the face needs to be beautified by skin grinding when the face is photographed, so that the face looks more attractive. When performing style migration, the structure edge needs to be preserved and the specific texture information needs to be ignored. Based on these requirements, we need edge-preserving smoothing processing on the image after the image is captured, so that the image can be better used for subsequent image beautification, analysis and processing. Image smoothing techniques are a type of computer vision task that preserves global structural information in an image and removes local texture information in the image for certain specific needs. The main purpose of image smoothing is to make the smoothed image more convenient for human or machine analysis and processing by enhancing the visual effect of the image structure relative to the image texture. The image smoothing technology has been widely applied to various fields such as mobile phone photographing technology and social media, and plays an important role in our life. For example, beauty cameras typically smooth human faces using image smoothing techniques.
However, there are some difficulties to be solved in the task of image smoothing. Such as how well image smoothing techniques preserve information of structural edges. Human eyes can instinctively extract meaningful structural information from complex texture information of an image directly. However, it is difficult for a computer program to automatically distinguish between structure information and texture information in an image, and the existing image smoothing technology may smooth out the edge portion of the structure in order to remove the texture information. Edge preserving image smoothing techniques can automatically preserve structural edge information while removing texture, a very challenging task that is very important for computer vision applications.
The article of distinguishing the knowledge in a neural network (NIPS 2014 Deep Learning network), which was published by Hinton et al in 2015, firstly proposes a framework of Deep Learning knowledge distillation, and introduces concepts of 'student network' and 'teacher network', so as to intensively migrate knowledge of a plurality of trained teacher networks into one student network. Knowledge distillation is used here to generalize the "hard label" of the classification dataset to the "soft label" so that the degree of similarity between images of different labels can be learned. After the teacher network is trained, the teacher network provides additional discriminant and similarity knowledge hidden outside the labels, and then the knowledge is migrated to the student network through a knowledge distillation technology. In recent years, with the improvement and improvement of computing devices, neural networks have been developed rapidly. Because of its powerful representation capability, neural networks have been widely used in the research fields of computer vision, natural language processing, etc. One of the major research hotspots on neural networks today is the deployment of neural networks in portable embedded devices. Due to the nature of the neural network, the larger the parameter quantity of the neural network is, the stronger the model performance is. However, the more the network model parameters are, the more memory resources the network needs to occupy, and the operation is very time-consuming, which is contrary to the requirements of the industry that the resources are few and the operation time is short. And knowledge of distillation techniques can help to solve this problem. Knowledge distillation techniques utilize one or more networks of teachers with large parameters to improve the performance of a network of students with small parameters. It is likely that the student network will not ultimately perform as well as the teacher's network, but will be more powerful than a student network trained separately from the teacher's network.
However, the conventional deep learning knowledge distillation method cannot be directly used in the task of edge-preserving image smoothing. The reason for this is that in our task, teacher networks mainly extract structural edge information, and student networks mainly remove local texture in images. The student network is distilled by directly using the trained teacher network, so that only the student network with image edge extraction and detection can be obtained, and the student network with image smoothing capability cannot be obtained. Therefore, a novel distillation technology based on deep learning knowledge is provided and applied to the task of edge-preserving image smoothing. The method can complete the task of image smoothing and has the capability of retaining structural edges.
Disclosure of Invention
The technical problem to be solved by the invention is to reserve the structural edge information of an image while removing the image texture in an image smoothing task. The invention aims to provide a novel deep learning knowledge distillation technology, which is used for carrying out knowledge migration on the structural edge extraction capability of a teacher network to a student network with a smooth image, so that the student network can have the structural edge retention capability of the teacher network while carrying out image smoothing.
The realization process of the invention is as follows:
optionally, selecting an existing deep learning model or retraining a deep learning model as the teacher network includes:
training a deep neural network to carry out an edge detection task;
optionally, the teacher network is used to assist in training the image smoothing student network, and the iteration includes:
inputting the input image I into a student network to obtain a smooth image Is;
respectively inputting the input image I and the smoothed image Is into a teacher network to obtain a structural edge label Ie of the input image and structural edge information Ise of the smoothed image;
obtaining a smooth image label Igt from the input image I by using an existing commonly-used image smoothing algorithm (such as L0);
respectively calculating the distance loss between the smoothed image Is and the smoothed image label Igt and the distance loss between the structural edge information Ise of the smoothed image and the structural edge label Ie of the input image by using a loss function;
carrying out weighted summation on the two losses to obtain a distance loss sum;
and (4) carrying out gradient return on the student network by using the sum of the losses, and updating the parameters of the student network.
The student network is trained using a plurality of iterations as described above.
Compared with the prior art, the invention has the beneficial effects that: the student network is updated by calculating the additional structural edge information calculation loss through the teacher network, so that the student network can learn the capability of image smoothing and can also have the capability of keeping the structural edge of the teacher network. Therefore, the negative influence that the structural edge cannot be well reserved by image smoothing in the conventional image smoothing method can be compensated.
Drawings
FIG. 1 is a flow chart of a novel deep learning knowledge distillation technique-based edge preserving image smoothing method.
FIG. 2 is a schematic diagram of a model of an edge-preserving image smoothing method based on a novel deep learning knowledge distillation technology.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the examples of the present invention, and it is obvious that the described examples are only a part of the embodiments of the present invention, and not all of the embodiments. Variations, modifications, substitutions and alterations of the embodiments of the invention are within the scope of the invention, without departing from the principles of the invention.
The embodiment of the invention provides an image enhancement deep learning method based on a knowledge distillation technology, the flow is shown in figure 1, and the method comprises the following steps:
and S1, selecting a teacher network to calculate the structural edge graph.
Optionally, the structural edge information is an edge distribution map describing image content structure information in the image;
optionally, a trained deep neural network capable of extracting structural edge information is selected as a teacher network, and a deep neural network model with a similar function can be retrained, wherein a Holistic edge detection model or a classic VGG model can be adopted, and the teacher network is used for assisting in training an image smoothing student network.
And S2, calculating a smooth image by the student network.
Optionally, the student network structure uses a classical deep neural network, such as VGG or residual neural network;
specifically, an image to be smoothed Is input into a deep neural network model (such as a commonly-used VGGNet, which has 5 modules consisting of convolution layers, a correction linear unit and a maximum pooling operation, a module consisting of 3 full-link layers and a correction linear unit, and the last 1 softmax layer), the designed neural network structure Is a structure of a 'predicted image by image' (such as the VGGNet introduced earlier), and an output image which Is the same as the input image I in size and Is subjected to image smoothing Is output by the model.
And S3, calculating the structural edge information of the structural edge information label and the smoothed image by the teacher network.
Specifically, the input image I and the smoothed image Is are respectively input to an edge detection teacher network, and the teacher network predicts a structural edge label Ie of the input image and structural edge information Ise of the smoothed image. And in the edge detection teacher network, the structural edge label Ie of the input image is used as an edge image of the input image, and the structural edge information Ise of the smoothed image is an edge image detected by the edge detection teacher network on the smoothed image.
And S4, calculating the loss of the student network to update the student network parameters.
Optionally, a common loss function (such as L2 norm, L1 norm, cross entropy) Is used to calculate the loss | | | Is-Igt | | | _ F } {2} between the smoothed image Is and the smoothed image label Igt, and the loss | | | | Ise-Ie | | _ F } {2} between the structural edge information Ise of the smoothed image and the structural edge label Ie of the input image, respectively; the smoothed image label Igt in the present invention is obtained by using the conventional image smoothing algorithm (e.g. L0) to input image I.
Specifically, the two losses are weighted and summed to obtain a loss sum, the loss sum is used for carrying out conventional gradient return on the student network (the conventional gradient return can be automatically realized through an SGD (Adam algorithm) in a deep learning framework PyTorch or Tensorflow), and the student network parameters are updated.
And S5, training the student network by multiple iterations.
Specifically, the learning student network parameters are continuously updated by using the updating mode to train the student network, and a model schematic diagram is shown in fig. 2. The teacher network parameters are fixed in the training process, namely the teacher network parameters are not updated during the period of training the student network.
Claims (4)
1. An edge-preserving image smoothing method based on deep learning knowledge distillation technology is characterized by comprising the following steps: the method carries out knowledge migration on the structural edge extraction capability of the teacher network to the student network with smooth images, so that the student network has the structural edge retention capability of the teacher network while carrying out image smoothing, and comprises the following steps:
s1, training a deep neural network with a reserved structural edge as a teacher network;
s2, inputting the input image into a student network, wherein the network outputs a smooth image which has the same size as the input image and is subjected to image enhancement;
s3, respectively inputting the input image and the smoothed image into a teacher network, and calculating structural edge information by the teacher network, wherein the edge information of the input image is used as a label, and the edge information of the smoothed image is used as predicted edge information;
s4, calculating loss information of the labels and the predicted edge information in the S3 as the loss of the student network, and updating parameters of the student network by using a reverse iteration strategy;
and S5, continuously updating the learning student network parameters by using the updating mode, and training the model parameters of the student network.
2. The edge-preserving image smoothing method based on the deep learning knowledge distillation technology according to claim 1, characterized in that: respectively inputting the input image and the smooth image into a teacher network, extracting structural edge labels of the input image and structural edge information of the smooth image, taking the structural edge labels of the input image extracted by the edge detection teacher network as an edge image of the input image, and taking the structural edge information of the smooth image as an edge image of the smoothed image.
3. The edge-preserving image smoothing method based on the deep learning knowledge distillation technology according to claim 1, characterized in that: the total loss function of the student network is formed by weighted summation of two parts of loss, wherein one part is to calculate the loss between the smoothed image and the smoothed image label by using a common loss function, and the other part is to calculate the loss between the structural edge information of the smoothed image and the structural edge label of the input image by using the common loss function.
4. The edge-preserving image smoothing method based on the deep learning knowledge distillation technology according to claim 1 or 2, characterized in that: the teacher network is trained in advance, and teacher network parameters are not updated in the process of training the student network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010066280.1A CN111275646B (en) | 2020-01-20 | 2020-01-20 | Edge-preserving image smoothing method based on deep learning knowledge distillation technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010066280.1A CN111275646B (en) | 2020-01-20 | 2020-01-20 | Edge-preserving image smoothing method based on deep learning knowledge distillation technology |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111275646A true CN111275646A (en) | 2020-06-12 |
| CN111275646B CN111275646B (en) | 2022-04-26 |
Family
ID=71003289
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010066280.1A Active CN111275646B (en) | 2020-01-20 | 2020-01-20 | Edge-preserving image smoothing method based on deep learning knowledge distillation technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111275646B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112257864A (en) * | 2020-10-22 | 2021-01-22 | 福州大学 | Lifetime learning method for solving catastrophic forgetting problem |
| CN115631178A (en) * | 2022-11-03 | 2023-01-20 | 昆山润石智能科技有限公司 | Automatic wafer defect detection method, system, equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107464227A (en) * | 2017-08-24 | 2017-12-12 | 深圳市唯特视科技有限公司 | A kind of method that reflection and smoothed image are removed based on deep neural network |
| CN107633486A (en) * | 2017-08-14 | 2018-01-26 | 成都大学 | Structure Magnetic Resonance Image Denoising based on three-dimensional full convolutional neural networks |
| CN108764462A (en) * | 2018-05-29 | 2018-11-06 | 成都视观天下科技有限公司 | A kind of convolutional neural networks optimization method of knowledge based distillation |
| CN108830813A (en) * | 2018-06-12 | 2018-11-16 | 福建帝视信息科技有限公司 | A kind of image super-resolution Enhancement Method of knowledge based distillation |
| CN109087303A (en) * | 2018-08-15 | 2018-12-25 | 中山大学 | The frame of semantic segmentation modelling effect is promoted based on transfer learning |
| CN109919855A (en) * | 2019-01-17 | 2019-06-21 | 深圳禾苗通信科技有限公司 | A kind of quick guarantor side denoising method based on multiframe image |
| CN110689495A (en) * | 2019-09-25 | 2020-01-14 | 安徽信息工程学院 | Image restoration method for deep learning |
-
2020
- 2020-01-20 CN CN202010066280.1A patent/CN111275646B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107633486A (en) * | 2017-08-14 | 2018-01-26 | 成都大学 | Structure Magnetic Resonance Image Denoising based on three-dimensional full convolutional neural networks |
| CN107464227A (en) * | 2017-08-24 | 2017-12-12 | 深圳市唯特视科技有限公司 | A kind of method that reflection and smoothed image are removed based on deep neural network |
| CN108764462A (en) * | 2018-05-29 | 2018-11-06 | 成都视观天下科技有限公司 | A kind of convolutional neural networks optimization method of knowledge based distillation |
| CN108830813A (en) * | 2018-06-12 | 2018-11-16 | 福建帝视信息科技有限公司 | A kind of image super-resolution Enhancement Method of knowledge based distillation |
| CN109087303A (en) * | 2018-08-15 | 2018-12-25 | 中山大学 | The frame of semantic segmentation modelling effect is promoted based on transfer learning |
| CN109919855A (en) * | 2019-01-17 | 2019-06-21 | 深圳禾苗通信科技有限公司 | A kind of quick guarantor side denoising method based on multiframe image |
| CN110689495A (en) * | 2019-09-25 | 2020-01-14 | 安徽信息工程学院 | Image restoration method for deep learning |
Non-Patent Citations (5)
| Title |
|---|
| GEOFFREY HINTON 等: "Distilling the Knowledge in a Neural Network", 《ARXIV:1503.02531V1》 * |
| HUI XU 等: "DeepObfuscation：Securing the Structure of Convolutional Neural Networks via Knowledge Distillation", 《ARXIV：1806.10313V1》 * |
| SEYED-IMAN MIRZADEH 等: "Improved Knowledge Distillation via Teacher Assistant", 《ARXIV:1902.03393》 * |
| 何洪亮: "基于卷积神经网络的交通标志检测研究", <中国优秀硕士学位论文全文数据库 (信息科技辑)> * |
| 程肯: "深度学习的轻量化算法研究", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112257864A (en) * | 2020-10-22 | 2021-01-22 | 福州大学 | Lifetime learning method for solving catastrophic forgetting problem |
| CN115631178A (en) * | 2022-11-03 | 2023-01-20 | 昆山润石智能科技有限公司 | Automatic wafer defect detection method, system, equipment and storage medium |
| CN115631178B (en) * | 2022-11-03 | 2023-11-10 | 昆山润石智能科技有限公司 | Automatic wafer defect detection method, system, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111275646B (en) | 2022-04-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107491726B (en) | Real-time expression recognition method based on multichannel parallel convolutional neural network | |
| CN106547880B (en) | Multi-dimensional geographic scene identification method fusing geographic area knowledge | |
| Gray et al. | Predicting facial beauty without landmarks | |
| CN110033007B (en) | Pedestrian clothing attribute identification method based on depth attitude estimation and multi-feature fusion | |
| CN111160533A (en) | Neural network acceleration method based on cross-resolution knowledge distillation | |
| CN111783521B (en) | Pedestrian re-identification method based on low-rank prior guidance and based on domain invariant information separation | |
| CN109359527B (en) | Hair region extraction method and system based on neural network | |
| CN105184772A (en) | Adaptive color image segmentation method based on super pixels | |
| CN108595558B (en) | Image annotation method based on data equalization strategy and multi-feature fusion | |
| CN110796018A (en) | Hand motion recognition method based on depth image and color image | |
| CN111275646B (en) | Edge-preserving image smoothing method based on deep learning knowledge distillation technology | |
| CN111274994A (en) | Cartoon face detection method and device, electronic equipment and computer readable medium | |
| CN115830652B (en) | Deep palm print recognition device and method | |
| CN111832573A (en) | Image emotion classification method based on class activation mapping and visual saliency | |
| CN117033609B (en) | Text visual question-answering method, device, computer equipment and storage medium | |
| CN110598537A (en) | Video significance detection method based on deep convolutional network | |
| CN113420827A (en) | Semantic segmentation network training and image semantic segmentation method, device and equipment | |
| CN110135435B (en) | Saliency detection method and device based on breadth learning system | |
| CN114333062B (en) | Pedestrian re-recognition model training method based on heterogeneous dual networks and feature consistency | |
| CN111126155A (en) | Pedestrian re-identification method for generating confrontation network based on semantic constraint | |
| CN113076905A (en) | Emotion recognition method based on context interaction relationship | |
| CN113537187A (en) | Text recognition method and device, electronic equipment and readable storage medium | |
| CN114743251B (en) | Drama character facial expression recognition method based on shared integrated convolutional neural network | |
| CN116167015A (en) | Dimension emotion analysis method based on joint cross attention mechanism | |
| CN116524207A (en) | Weak supervision RGBD image significance detection method based on edge detection assistance |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |