CN112288829A - Compression method and device for image restoration convolutional neural network - Google Patents
Compression method and device for image restoration convolutional neural network Download PDFInfo
- Publication number
- CN112288829A CN112288829A CN202011207721.1A CN202011207721A CN112288829A CN 112288829 A CN112288829 A CN 112288829A CN 202011207721 A CN202011207721 A CN 202011207721A CN 112288829 A CN112288829 A CN 112288829A
- Authority
- CN
- China
- Prior art keywords
- neural network
- convolutional neural
- convolution
- convolutional
- side window
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 78
- 230000006835 compression Effects 0.000 title claims abstract description 42
- 238000007906 compression Methods 0.000 title claims abstract description 42
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000000354 decomposition reaction Methods 0.000 claims abstract description 28
- 238000011084 recovery Methods 0.000 claims abstract description 6
- 230000010339 dilation Effects 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 7
- 230000003247 decreasing effect Effects 0.000 claims description 4
- 239000011159 matrix material Substances 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 14
- 230000000694 effects Effects 0.000 description 3
- 230000006837 decompression Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
- G06T9/002—Image coding using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a compression method and a device for an image restoration convolutional neural network, wherein the method comprises the following steps: acquiring an image to be compressed, restoring a convolutional neural network, and replacing a square convolutional kernel with a side window kernel of a non-square convolutional kernel to obtain the convolutional neural network after the side window kernel is replaced; decomposing the convolution layer of the convolutional neural network after the side window kernel replacement into a channel direction space convolution and a linear projection convolution to obtain the convolutional neural network after the convolution layer decomposition; and performing performance compensation on the convolutional neural network after the convolutional layer decomposition based on a symmetric expansion convolution and attention mechanism to obtain a compressed image recovery convolutional neural network. The device comprises a memory and a processor for executing the compression method aiming at the image restoration convolutional neural network, and the invention, as the compression method and the device aiming at the image restoration convolutional neural network, can be widely applied to the field of network compression.
Description
Technical Field
The invention relates to the field of network compression, in particular to a compression method and a compression device for an image restoration convolutional neural network.
Background
Since the 3-layer convolutional neural network SRCNN designed for the task of image super-resolution and the 4-layer convolutional neural network ARCNN designed for the task of image decompression artifact were proposed, CNNs began to exhibit their effectiveness and dominate the image restoration field, however, deeper networks have more parameters and computation costs, which hinders their application in situations with limited computational and memory resources, in order to find a balance point between model size, computational efficiency and performance, various model compression solutions have been proposed, and most existing model compression methods are directed to high-level visual tasks, such as image classification, target detection, etc., and thus may not maintain good performance when directly applied to low-level visual tasks.
Disclosure of Invention
In order to solve the above technical problems, an object of the present invention is to provide a compression method and apparatus for an image restoration convolutional neural network, which improve the network output performance while improving the network computation efficiency.
The first technical scheme adopted by the invention is as follows: a compression method for an image restoration convolutional neural network, comprising the steps of:
acquiring an image to be compressed, restoring a convolutional neural network, and replacing a square convolutional kernel with a side window kernel of a non-square convolutional kernel to obtain the convolutional neural network after the side window kernel is replaced;
decomposing the convolution layer of the convolutional neural network after the side window kernel replacement into a channel direction space convolution and a linear projection convolution to obtain the convolutional neural network after the convolution layer decomposition;
and performing performance compensation on the convolutional neural network after the convolutional layer decomposition based on a symmetric expansion convolution and attention mechanism to obtain a compressed image recovery convolutional neural network.
Further, the side window core comprises an upper side window core, a lower side window core, a left side window core and a right side window core.
Further, the convolution neural network after the side window kernel replacement uses a compression ratio calculation formula of the side window kernel convolution as follows:
the CR1Representing the compression ratio of a convolution using a side window kernel, NinExpressed as the number of input channels, NoutExpressed as the number of output channelsK × k is expressed as the original convolution kernel size, said
Expressed as the side window kernel size.
Further, the step of performing performance compensation on the convolutional neural network after convolutional layer decomposition based on the symmetric expansion convolution and attention mechanism to obtain a compressed image restoration convolutional neural network specifically includes:
acquiring a convolutional neural network after convolutional layer decomposition, and setting different expansion rates for convolutional layers in the convolutional neural network after convolutional layer decomposition according to a preset rule;
and adding an attention block at the tail end of the convolutional neural network after convolutional layer decomposition to obtain a compressed image recovery convolutional neural network.
Further, the preset rule specifically includes that the expansion rate of the expansion convolution is in a symmetrical mode, the sequence of the expansion rate of the expansion convolution is that the expansion rate is increased first and then decreased, and convolution layers with the expansion rate of 1 are inserted into every other section in the expansion convolution.
Further, the attention mechanism in the attention block comprises a weighting step and a scaling step.
Further, the weighting step adopts a convolutional layer learning attention diagram with the kernel size being a preset value, and the scaling step is to perform matrix dot multiplication between the input feature diagram and the attention diagram to generate an output feature diagram.
Further, a compression ratio calculation formula of the convolutional neural network after convolutional layer decomposition is as follows:
the CR2Expressed as the compression ratio of the convolutional neural network after convolutional layer decomposition, NinExpressed as the number of input channels, NoutExpressed as the number of output channels, said k × k being expressed as the original convolution kernel size, said
Expressed as the side window kernel size.
The second technical scheme adopted by the invention is as follows: a compression apparatus for image restoration convolutional neural network, comprising:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, cause the at least one processor to implement a compression method for an image restoration convolutional neural network as described above.
The method and the device have the beneficial effects that: the invention reduces network parameters by replacing the non-square convolution kernel of the side window kernel on the convolution kernel layer surface, decomposes the standard convolution into channel space convolution and linear projection convolution on the convolution layer surface, and modifies the network by adopting symmetrical expansion convolution and attention blocks on the convolution block layer surface so as to achieve the purpose of improving the network performance.
Drawings
FIG. 1 is a flow chart of the steps of a compression method for an image restoration convolutional neural network according to the present invention;
FIG. 2 illustrates four side window kernel types according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating the convolution operation steps after replacing the side window kernel according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating the convolution operation after convolutional layer decomposition according to an embodiment of the present invention;
FIG. 5 is a schematic illustration of an attention mechanism of an attention block in accordance with an embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the figures and the specific embodiments. The step numbers in the following embodiments are provided only for convenience of illustration, the order between the steps is not limited at all, and the execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.
The invention provides a general compression method and a general compression device of a convolutional neural network for an image restoration task, which can be used for tasks such as image denoising, image super-resolution and image decompression artifact. The method is divided into three levels from micro level to macro level, and the convolutional neural network is compressed at the convolutional kernel level, the convolutional level and the convolutional block level respectively.
As shown in FIG. 1, the invention provides a general compression method of a convolutional neural network for an image restoration task, which comprises the following steps:
s101, acquiring an image to be compressed, restoring a convolutional neural network, and replacing a square convolutional kernel with a side window kernel of a non-square convolutional kernel to obtain the convolutional neural network after the side window kernel is replaced;
in particular, model compression at the convolutional kernel level is achieved by the proposed side window kernel. Unlike the common square convolution kernel, the side window kernel is non-square.
S102, decomposing the convolution layer of the convolutional neural network after side window kernel replacement into a channel direction space convolution and a linear projection convolution to obtain the convolutional neural network after convolutional layer decomposition;
specifically, to further compress the network, at the convolution level, SWConv is decomposed into two steps: the channel direction spatial convolution and the linear projection convolution, and the decomposed SWConv is called SWConv-F.
As shown in fig. 4, in SWConv-F, the standard convolution is decomposed into a channel space convolution and a linear projection convolution. In channel space convolution, we will input NinEach feature map is divided into NinAnd each group is subjected to convolution calculation by using one type of side window kernel and obtains corresponding output. To obtain equal information from each type of side window kernel, one type of side window kernel is applied to one quarter of the group. Thus, in channel space convolution, there is NinSide window core and NinAn output channel. Thereafter, we apply a 1 × 1 convolution kernel with the previous channel space convolved NinConvolving the characteristic diagrams to obtain the final NoutAnd outputting the characteristic diagram. SWConv-F is typically used as an intermediate layer in a network.
S103, performing performance compensation on the convolutional neural network after the convolutional layer decomposition based on a symmetric expansion convolution and attention mechanism to obtain a compressed image restoration convolutional neural network.
Specifically, after the network is compressed through the side window kernel and the convolutional layer decomposition, the performance of the compressed network is reduced to some extent due to a certain loss of pixel information. To maintain restoration performance similar to the original network, we employ a symmetric dilation convolution and attention mechanism.
Further as a preferred embodiment of the method, the side window core includes an upper side window core, a lower side window core, a left side window core and a right side window core.
Specifically, referring to FIG. 2, SWD represents the upper window kernel, SWD represents the lower window kernel, SWL represents the left window kernel, SWR represents the right window kernel, w2,2、w1,2、w2,2And w2,1The color deepened region is represented as the center of the convolution kernel.
Further as a preferred embodiment of the method, the convolutional neural network after side window kernel replacement uses a compression ratio calculation formula of side window kernel convolution as follows:
the CR1Representing the compression ratio of a convolution using a side window kernel, NinExpressed as the number of input channels, NoutExpressed as the number of output channels, said k × k being expressed as the original convolution kernel size, said
Expressed as the side window kernel size.
Specifically, the convolution using the side window kernel is shown in FIG. 3 and referred to as SWConv. Each side window kernel performs convolution operation on all input feature maps and outputs the convolution operation respectively
A characteristic diagram. And then connecting the four groups of feature maps. To obtain NoutAnd outputting the characteristic diagram.
In addition, SWConv is not suitable for the last layer in the network because the output channel of the last layer is often 3 or 1, and cannot be divided by 4, and if only one side window kernel is used to convolve the last layer, more performance is lost due to the deviation of the side window.
Further as a preferred embodiment of the method, the step of performing performance compensation on the convolutional neural network after convolutional layer decomposition based on a symmetric expansion convolution and attention mechanism to obtain a compressed image restoration convolutional neural network specifically includes:
acquiring a convolutional neural network after convolutional layer decomposition, and setting different expansion rates for convolutional layers in the convolutional neural network after convolutional layer decomposition according to a preset rule;
and adding an attention block at the tail end of the convolutional neural network after convolutional layer decomposition to obtain a compressed image recovery convolutional neural network.
Specifically, the dilation convolution is designed to enlarge the receptive field of the convolutional neural network. Therefore, it can be employed to compensate for the negative effects of the side window kernel on the network's receptive field.
Further, as a preferred embodiment of the method, the preset rule specifically includes that the expansion rates of the expansion convolutions are in a symmetric manner, the sequence of the expansion rates of the expansion convolutions is that the expansion rates of the expansion convolutions are increased first and then decreased, and convolution layers with expansion rates of 1 are inserted every other segment in the expansion convolutions.
Specifically, the expansion rates of the expansion convolutions are arranged in a symmetrical manner in an order of increasing and then decreasing, so that the gridding phenomenon can be effectively avoided. In these symmetric extended convolutions, convolution layers with an expansion rate of 1 are inserted every other segment. The proposed symmetric dilation convolution does not bring extra parameters and computational load.
Further as a preferred embodiment of the method, the attention mechanism in the attention block comprises a weighting step and a scaling step.
Further as a preferred embodiment of the method, the weighting step uses a convolutional layer learning attention diagram with a kernel size of a preset value, and the scaling step is to perform matrix dot multiplication between the input feature diagram and the attention diagram to generate an output feature diagram.
Specifically, as shown in fig. 5, the attention block is added at the end of the network to further improve the performance, the preset value is c × 1 × 1 × c, where c represents the number of channels of the restored image, and the attention block uses information across the channel dimensions, so that it can be considered as a compensation for the inter-channel information flow blockage caused by the previous channel spatial convolution.
In addition, the number of parameters and the calculation amount introduced by the 1 × 1 convolution and the matrix dot product are small and can be almost ignored.
Further as a preferred embodiment of the method, a compression ratio calculation formula of the convolutional neural network after convolutional layer decomposition is as follows:
the CR2Expressed as the compression ratio of the convolutional neural network after convolutional layer decomposition, NinExpressed as the number of input channels, NoutExpressed as the number of output channels, said k × k being expressed as the original convolution kernel size, said
Expressed as the side window kernel size.
A compression apparatus for an image restoration convolutional neural network:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, cause the at least one processor to implement a compression method for an image restoration convolutional neural network as described above.
The contents in the above method embodiments are all applicable to the present apparatus embodiment, the functions specifically implemented by the present apparatus embodiment are the same as those in the above method embodiments, and the advantageous effects achieved by the present apparatus embodiment are also the same as those achieved by the above method embodiments.
A storage medium having stored therein instructions executable by a processor, the storage medium comprising: the processor-executable instructions, when executed by the processor, are for implementing a compression method for an image restoration convolutional neural network as described above.
The contents in the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those in the above method embodiments, and the advantageous effects achieved by the present storage medium embodiment are also the same as those achieved by the above method embodiments.
While the preferred embodiments of the present invention have been illustrated and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (9)
1. A compression method for an image restoration convolutional neural network is characterized by comprising the following steps:
acquiring an image to be compressed, restoring a convolutional neural network, and replacing a square convolutional kernel with a side window kernel of a non-square convolutional kernel to obtain the convolutional neural network after the side window kernel is replaced;
decomposing the convolution layer of the convolutional neural network after the side window kernel replacement into a channel direction space convolution and a linear projection convolution to obtain the convolutional neural network after the convolution layer decomposition;
and performing performance compensation on the convolutional neural network after the convolutional layer decomposition based on a symmetric expansion convolution and attention mechanism to obtain a compressed image recovery convolutional neural network.
2. The compression method for an image restoration convolutional neural network as claimed in claim 1, wherein said side window kernels comprise an upper side window kernel, a lower side window kernel, a left side window kernel and a right side window kernel.
3. The compression method for the image restoration convolutional neural network as claimed in claim 2, wherein the convolutional neural network after the side window kernel replacement uses a compression ratio calculation formula of the side window kernel convolution as follows:
the CR1Representing the compression ratio of a convolution using a side window kernel, NinExpressed as the number of input channels, NoutExpressed as the number of output channels, said k × k being expressed as the original convolution kernel size, said
Expressed as the side window kernel size.
4. The method as claimed in claim 3, wherein the step of performing performance compensation on the convolutional neural network after convolutional layer decomposition based on a symmetric dilation convolution and attention mechanism to obtain the compressed image restoration convolutional neural network specifically comprises:
acquiring a convolutional neural network after convolutional layer decomposition, and setting different expansion rates for convolutional layers in the convolutional neural network after convolutional layer decomposition according to a preset rule;
and adding an attention block at the tail end of the convolutional neural network after convolutional layer decomposition to obtain a compressed image recovery convolutional neural network.
5. The method as claimed in claim 4, wherein the predetermined rule includes that the dilation convolution has symmetric dilation rates and the sequence is first increasing and then decreasing, and convolution layers with dilation rate of 1 are inserted at intervals in the dilation convolution.
6. The compression method for the image restoration convolutional neural network as claimed in claim 5, wherein the attention mechanism in the attention block comprises a weighting step and a scaling step.
7. The method as claimed in claim 6, wherein the weighting step uses convolutional layer learning attention map with kernel size being a predetermined value, and the scaling step is to perform matrix dot multiplication between the input feature map and the attention map to generate the output feature map.
8. The compression method for the convolutional neural network for image restoration according to claim 7, wherein the compression ratio of the convolutional neural network after convolutional layer decomposition is calculated by the following formula:
the CR2Expressed as the compression ratio of the convolutional neural network after convolutional layer decomposition, NinExpressed as the number of input channels, NoutExpressed as the number of output channels, said k × k being expressed as the original convolution kernel size, said
Expressed as the side window kernel size.
9. A compression apparatus for restoring a convolutional neural network for an image, comprising:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, cause the at least one processor to implement a compression method for image restoration convolutional neural network as claimed in any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011207721.1A CN112288829A (en) | 2020-11-03 | 2020-11-03 | Compression method and device for image restoration convolutional neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011207721.1A CN112288829A (en) | 2020-11-03 | 2020-11-03 | Compression method and device for image restoration convolutional neural network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112288829A true CN112288829A (en) | 2021-01-29 |
Family
ID=74350491
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011207721.1A Pending CN112288829A (en) | 2020-11-03 | 2020-11-03 | Compression method and device for image restoration convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112288829A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116310543A (en) * | 2023-03-14 | 2023-06-23 | 自然资源部第一海洋研究所 | GF-1WFV satellite red tide deep learning detection model, construction method and equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108229646A (en) * | 2017-08-08 | 2018-06-29 | 北京市商汤科技开发有限公司 | neural network model compression method, device, storage medium and electronic equipment |
| CN109886391A (en) * | 2019-01-30 | 2019-06-14 | 东南大学 | A kind of neural network compression method based on the positive and negative diagonal convolution in space |
| CN110796250A (en) * | 2019-10-11 | 2020-02-14 | 浪潮电子信息产业股份有限公司 | Convolution processing method and system applied to convolutional neural network and related components |
| CN111325155A (en) * | 2020-02-21 | 2020-06-23 | 重庆邮电大学 | Video motion recognition method based on residual difference type 3D CNN and multi-mode feature fusion strategy |
| US20200302265A1 (en) * | 2019-03-20 | 2020-09-24 | Huawei Technologies Co., Ltd. | Convolutional Neural Network-Based Image Processing Method And Image Processing Apparatus |
-
2020
- 2020-11-03 CN CN202011207721.1A patent/CN112288829A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108229646A (en) * | 2017-08-08 | 2018-06-29 | 北京市商汤科技开发有限公司 | neural network model compression method, device, storage medium and electronic equipment |
| CN109886391A (en) * | 2019-01-30 | 2019-06-14 | 东南大学 | A kind of neural network compression method based on the positive and negative diagonal convolution in space |
| US20200302265A1 (en) * | 2019-03-20 | 2020-09-24 | Huawei Technologies Co., Ltd. | Convolutional Neural Network-Based Image Processing Method And Image Processing Apparatus |
| CN110796250A (en) * | 2019-10-11 | 2020-02-14 | 浪潮电子信息产业股份有限公司 | Convolution processing method and system applied to convolutional neural network and related components |
| CN111325155A (en) * | 2020-02-21 | 2020-06-23 | 重庆邮电大学 | Video motion recognition method based on residual difference type 3D CNN and multi-mode feature fusion strategy |
Non-Patent Citations (1)
| Title |
|---|
| 李志义;黄子风;许晓绵;: "基于表示学习的跨模态检索模型与特征抽取研究综述", 情报学报, no. 04 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116310543A (en) * | 2023-03-14 | 2023-06-23 | 自然资源部第一海洋研究所 | GF-1WFV satellite red tide deep learning detection model, construction method and equipment |
| CN116310543B (en) * | 2023-03-14 | 2023-09-22 | 自然资源部第一海洋研究所 | GF-1WFV satellite red tide deep learning detection model, construction method and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109389556B (en) | Multi-scale cavity convolutional neural network super-resolution reconstruction method and device | |
| CN111046962B (en) | Sparse attention-based feature visualization method and system for convolutional neural network model | |
| KR102428176B1 (en) | Correction Method And Device for Distorted Document Image | |
| CN109087273B (en) | Image restoration method, storage medium and system based on enhanced neural network | |
| CN107301668B (en) | A kind of picture compression method based on sparse matrix, convolutional neural networks | |
| CN107944545B (en) | Computing method and computing device applied to neural network | |
| CN113421187B (en) | Super-resolution reconstruction method, system, storage medium and equipment | |
| CN113538281B (en) | Image denoising method, image denoising device, computer equipment and storage medium | |
| CN113298716B (en) | Image super-resolution reconstruction method based on convolutional neural network | |
| Xia et al. | Basic binary convolution unit for binarized image restoration network | |
| CN112288829A (en) | Compression method and device for image restoration convolutional neural network | |
| CN109993701B (en) | Depth map super-resolution reconstruction method based on pyramid structure | |
| CN113538258B (en) | Mask-based image deblurring model and method | |
| CN111667401B (en) | Multi-level gradient image style migration method and system | |
| KR20200022386A (en) | Information processing device and information processing method | |
| CN116434039A (en) | Target detection method based on multiscale split attention mechanism | |
| CN111179171A (en) | Image super-resolution reconstruction method based on residual module and attention mechanism | |
| CN111091515A (en) | Image restoration method and device, and computer-readable storage medium | |
| CN111210439B (en) | Semantic segmentation method and device by suppressing uninteresting information and storage device | |
| CN114529713A (en) | Underwater image enhancement method based on deep learning | |
| CN112734655A (en) | Low-light image enhancement method for enhancing CRM (customer relationship management) based on convolutional neural network image | |
| US20230196526A1 (en) | Dynamic convolutions to refine images with variational degradation | |
| US20220044370A1 (en) | Image processing methods | |
| CN114723997B (en) | Composite convolution operation method based on graphic algebra, storage medium and electronic equipment | |
| CN113538244B (en) | Lightweight super-resolution reconstruction method based on self-adaptive weight learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |