CN114548884B - Package identification method and system based on pruning lightweight model - Google Patents
Package identification method and system based on pruning lightweight model Download PDFInfo
- Publication number
- CN114548884B CN114548884B CN202210447343.7A CN202210447343A CN114548884B CN 114548884 B CN114548884 B CN 114548884B CN 202210447343 A CN202210447343 A CN 202210447343A CN 114548884 B CN114548884 B CN 114548884B
- Authority
- CN
- China
- Prior art keywords
- matrix
- pruning
- entropy
- characteristic
- von neumann
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013138 pruning Methods 0.000 title claims abstract description 107
- 238000000034 method Methods 0.000 title claims abstract description 45
- 239000011159 matrix material Substances 0.000 claims abstract description 165
- 238000010586 diagram Methods 0.000 claims abstract description 100
- 238000003062 neural network model Methods 0.000 claims abstract description 54
- 238000012549 training Methods 0.000 claims abstract description 37
- 230000008859 change Effects 0.000 claims abstract description 29
- 239000013598 vector Substances 0.000 claims description 35
- 238000004364 calculation method Methods 0.000 claims description 8
- 238000012217 deletion Methods 0.000 claims description 5
- 230000037430 deletion Effects 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 3
- 230000006872 improvement Effects 0.000 description 8
- 238000013528 artificial neural network Methods 0.000 description 4
- 230000001133 acceleration Effects 0.000 description 3
- 230000006835 compression Effects 0.000 description 3
- 238000007906 compression Methods 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000000137 annealing Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000013585 weight reducing agent Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/083—Shipping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biophysics (AREA)
- Economics (AREA)
- Development Economics (AREA)
- Entrepreneurship & Innovation (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a package identification method and system based on a pruning lightweight model, belongs to the technical field of image identification, and solves the problems that an existing neural network model is high in operation complexity, large in time and memory consumption and difficult to deploy on terminal equipment. Inputting a training picture into a neural network model to be pruned, and extracting a characteristic diagram matrix of each convolution layer; converting the characteristic diagram matrix into a weighted undirected graph, constructing an improved Laplace matrix, and calculating von Neumann diagram entropy as an original value; sequentially deleting single vertexes in the weighted undirected graph to obtain a new weighted undirected graph, and calculating a change value of von Neumann graph entropy of the new weighted undirected graph relative to an original value; calculating the importance of the channel according to the change value of the entropy of the von Neumann diagram, and pruning the channel to obtain a pruning lightweight model; and deploying the pruning lightweight model to a package recognition terminal device, and recognizing the pictures collected in real time. High pruning rate of the model is achieved, and real-time package recognition efficiency is improved.
Description
Technical Field
The invention relates to the technical field of image recognition, in particular to a package recognition method and system based on a pruning lightweight model.
Background
In recent years, express services in China generally have a rapid growth trend and have great development potential. In order to meet the sorting requirement brought by the express business volume which is rapidly increased, the express industry has high requirements on the express sorting capacity of a logistics transfer center. Modern logistics uses logistics parcel automatic sorting systems to sort parcels according to different regions, wherein recognition and detection of parcels are important components of express sorting.
In the current image recognition method, the traditional image processing method has the problems of difficult artificial design characteristics and the like in a complex parcel sorting scene, and compared with the target recognition algorithm based on deep learning, the method has obvious technical advantages.
With the continuous development of deep learning theory, the deep convolutional neural network is pushed to a deeper network layer number and a wider network architecture. On one hand, the complex model can effectively improve the learning ability of the neural network, thereby obtaining better performance indexes; on the other hand, this results in a dramatic increase in the number of network parameters and floating-point arithmetic, which means higher power consumption, larger memory footprint, and longer training inference time. For a long time, the deep convolutional neural network has strict requirements on use scenes, is usually deployed at a server end, and cannot meet actual requirements. In a logistics sorting scene, a deep learning model is required for real-time package identification, and has the characteristics of few parameters, high operation speed and the like, the inference time of a complex model and the communication delay between a terminal and a server cannot meet the requirement of quickly processing a package image, and the popularization and application of deep learning in the field are greatly limited.
Disclosure of Invention
In view of the foregoing analysis, embodiments of the present invention are directed to providing a package identification method and system based on a pruning lightweight model, so as to solve the problems of the existing neural network model that the operation complexity is high, the time and memory consumption are large, and the deployment on a terminal device is difficult.
On one hand, the embodiment of the invention provides a package identification method based on a pruning lightweight model, which comprises the following steps:
inputting the training picture into a pre-trained neural network model to be pruned, and extracting a characteristic diagram matrix of each convolution layer;
converting the characteristic diagram matrix of each convolution layer into a weighted undirected graph, constructing an improved Laplace matrix according to the amplitude matrix of the characteristic diagram matrix, and calculating von Neumann diagram entropy as an original value of each convolution layer; sequentially deleting single vertexes in the weighted undirected graphs of the convolution layers to obtain new weighted undirected graphs, and calculating the change value of von Neumann graph entropy of each new weighted undirected graph relative to the original value;
calculating the importance of the channel of each convolutional layer according to the variation value of the von Neumann diagram entropy in each convolutional layer, pruning the channel of each convolutional layer according to the pruning rate of each convolutional layer and the importance of the channel, and training the pruned neural network model to obtain a pruning lightweight model;
and deploying the pruning lightweight model to package identification terminal equipment, and identifying package information in the real-time collected pictures.
Based on the further improvement of the method, the characteristic diagram matrix of each convolution layer is converted into a weighted undirected graph, and the method comprises the following steps:
deforming each three-dimensional characteristic diagram in the characteristic diagram matrix of each convolution layer to obtain a characteristic row vector matrix, wherein each row is a characteristic row vector corresponding to each channel and is respectively used as the vertex of a weighted undirected graph of each convolution layer;
calculating the cosine distance of any two characteristic row vectors in the characteristic row vector matrix of each convolution layer, and taking the cosine distance as the edge weight between two corresponding vertexes in the weighted undirected graph of each convolution layer;
and acquiring an adjacency matrix and a degree matrix according to the weighted undirected graph.
Based on the further improvement of the method, the adjacent matrix is a real symmetric matrix, the off-diagonal elements of the adjacent matrix are corresponding to the edge weight between two vertexes, and the diagonal elements are 0; the degree matrix is a diagonal matrix, the diagonal elements of each row of which are the sum of all elements of the corresponding row in the adjacency matrix.
Based on the further improvement of the method, the improved Laplace matrix is constructed according to the amplitude matrix of the characteristic diagram matrix, and the method comprises the following steps:
obtaining a magnitude matrix according to the characteristic diagram matrix, wherein the magnitude matrix is a diagonal matrix, and each diagonal element is the sum of squares of all elements in corresponding channels in the characteristic diagram matrix;
and multiplying the degree matrix by the amplitude matrix, and then subtracting the adjacent matrix to obtain the improved Laplace matrix.
Based on a further improvement of the above method, the von Neumann entropy is calculated by the following formula:
wherein,H i is shown asiVon Neumann entropy of each convolutional layer,L d representing the improved Laplace matrix; trace (·) represents the trace of the matrix, i.e., the sum of all eigenvalues of the matrix;λ k representing an improved Laplace matrixL d To middlekThe value of the characteristic is used as the characteristic value,λ k is not less than 0, and
，n i is shown asiNumber of channels per convolutional layer.
Based on further improvement of the method, sequentially deleting single vertexes in the weighted undirected graph of each convolution layer to obtain a new weighted undirected graph, and calculating a change value of von Neumann diagram entropy of each new weighted undirected graph relative to an original value, wherein the method comprises the following steps:
sequentially deleting the characteristic row vectors corresponding to a single vertex aiming at the weighted undirected graph of each convolution layer, reconverting the characteristic row vector matrix obtained after deletion each time into a new weighted undirected graph, constructing a new improved Laplace matrix, and calculating a new von Neumann graph entropy;
and calculating the absolute value of the difference value between each new von Neumann diagram entropy and the original value of the corresponding convolution layer as the change value of the von Neumann diagram entropy of the channel corresponding to the deleted vertex.
Based on a further improvement of the above method, calculating the importance of the channel of each convolutional layer according to the variation value of the von Neumann map entropy in each convolutional layer, comprising:
and calculating the average value of the von Neumann diagram entropy change values of each channel according to the von Neumann diagram entropy change values of the channels of the convolutional layers obtained by each training picture, and taking the average value as the importance of each channel.
Based on the further improvement of the method, the channels of the convolutional layers are pruned according to the pruning rate of the convolutional layers and the importance of the channels, and the channels are pruned from small to large according to the pruning rate of the convolutional layers and the importance of the channels of the convolutional layers.
Based on the further improvement of the method, the neural network model to be pruned is a neural network model formed by convolution layers with CONV-BN-RELU structures.
In another aspect, an embodiment of the present invention provides a package identification system based on a pruning lightweight model, including:
the characteristic diagram matrix extraction module is used for inputting the training picture into the pre-trained neural network model to be pruned and extracting the characteristic diagram matrix of each convolution layer;
the von Neumann diagram entropy calculation module is used for converting the characteristic diagram matrix of each convolution layer into a weighted undirected graph, constructing an improved Laplace matrix according to the amplitude matrix of the characteristic diagram matrix, and calculating the von Neumann diagram entropy as an original value of each convolution layer; sequentially deleting single vertexes in the weighted undirected graphs of the convolution layers to obtain new weighted undirected graphs, and calculating the change value of von Neumann graph entropy of each new weighted undirected graph relative to the original value;
the channel pruning module is used for calculating the importance of the channel of each convolutional layer according to the variation value of the von Neumann diagram entropy in each convolutional layer, pruning the channel of each convolutional layer according to the pruning rate of each convolutional layer and the importance of the channel, training the neural network model after pruning and obtaining a pruning lightweight model;
and the package identification module is used for deploying the pruning lightweight model to package identification terminal equipment and identifying package information in the real-time collected pictures.
Compared with the prior art, the invention can realize at least one of the following beneficial effects:
1. applying theories such as an undirected graph and von Neumann diagram entropy to neural network pruning, converting a characteristic diagram matrix of a neural network model into a weighted undirected graph, simultaneously considering the amplitude and the correlation of the characteristic diagram, calculating a von Neumann diagram entropy change value after deleting a single channel, and taking the entropy change value as a pruning basis to ensure the performance of a lightweight model after pruning and realize obvious model compression and acceleration;
2. the von Neumann diagram entropy of the characteristic diagram matrix has good stability, has small correlation with the number of input training pictures, and can obtain accurate characteristic diagram channel importance only by a small number of training pictures, so that the technical scheme has high pruning efficiency, low memory occupation and stable pruning result;
3. the channel pruning method is suitable for a neural network model formed by a convolution layer with a CONV-BN-RELU structure, parameter quantity and operation quantity of the neural network model are reduced as far as possible while performance is maximized, light weight of the neural network model is achieved, the compressed light deep neural network model is deployed on edge equipment with limited resources, the use range is expanded, and application value maximization is achieved.
In the invention, the technical schemes can be combined with each other to realize more preferable combination schemes. Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.
Drawings
The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the invention, wherein like reference numerals are used to designate like parts throughout the drawings;
fig. 1 is a flowchart of a package identification method based on a pruning lightweight model according to an embodiment of the present invention;
fig. 2 is a pruning example diagram of a package identification method based on a pruning lightweight model in an embodiment of the present invention;
fig. 3 is a schematic block diagram of a package identification system based on a pruning lightweight model according to a second embodiment of the present invention.
Detailed Description
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate preferred embodiments of the invention and together with the description, serve to explain the principles of the invention and not to limit the scope of the invention.
In a logistics sorting scene, a deep learning model is required for real-time package identification, and has the characteristics of few parameters, fast operation and the like, and the inference time of a complex model and the communication delay between a terminal and a server cannot meet the requirement for fast processing a package image. Aiming at the problems of large parameter quantity and floating point calculation quantity of a current deep neural network model, large time/memory consumption, difficult deployment and the like, the invention provides a package identification method and a system based on a pruning lightweight model, which apply theories such as an undirected graph and von Neumann diagram entropy to neural network pruning, convert a characteristic diagram matrix into a weighted undirected graph, calculate the change value of the von Neumann diagram entropy after each channel is pruned, and take the change value as an index for measuring the importance of the channel.
Example one
The invention discloses a package identification method of a pruning lightweight model. As shown in fig. 1, the method comprises the steps of:
s101: inputting the training picture into a pre-trained neural network model to be pruned, and extracting a characteristic diagram matrix of each convolution layer;
in the present embodiment, the specific network structure of the neural network model to be pruned is not limited, and any neural network model may be used as long as it is composed of convolutional layers having a CONV-BN-ReLU structure. The CONV-BN-RELU structure is widely applied to various mainstream convolutional neural network models, so that the pruning scheme in the method can be conveniently applied to the mainstream neural network models in the fields of classification, identification and the like, and the lightweight of the neural network model is realized.
Illustratively, the neural network model to be pruned may be a neural network model of the YOLO and ResNet series.
It should be noted that, by continuously taking pictures through the camera on the parcel sorting conveyor belt, the pictures are collected and marked, and preprocessing such as scaling is performed on the image data, so as to construct a parcel picture data set for pre-training the neural network model to be pruned. In the step, the neural network model to be pruned which is pre-trained is pruned to be light, and the required training pictures are randomly extracted from the wrapping picture data set. When more training pictures are used, more accurate feature map channel importance can be obtained theoretically, but the pruning time is increased, the pruning efficiency is reduced, and the pruning performance is not improved obviously; when less training pictures are used, more accurate feature map channel importance cannot be obtained, and the pruning effect is poor. Therefore, in order to take both the pruning efficiency and the pruning performance into consideration, 600-1000 pictures are used for pruning. Preferably, 640 training pictures are used, divided into 5 batches, and 128 pictures are input for each batch.
Specifically, training pictures are input into a pre-trained neural network model to be pruned in batches, the extracted feature map matrix of each convolution layer is a four-dimensional matrix, the feature map matrix comprises three-dimensional feature maps generated by each picture in each batch, and the three-dimensional feature maps can be represented by the following formula:
wherein,
is shown asiA matrix of signatures output by each convolutional layer,
representing a feature map matrix
To (1)jA feature map;
representing the number of training pictures input into the neural network model to be pruned in one batch;n i is shown asiThe number of channels of each characteristic diagram in the characteristic diagram matrix is output by each convolution layer;h i andw i respectively representing the height and width of the characteristic diagram;
denotes the firstiThe first convolution layer outputs the characteristic diagram matrixjCharacteristic diagram
To (1) akA channel map.
For example, 55 convolutional layers exist in the ResNet56 neural network model, and feature map matrices generated by the convolutional layers are extracted when 128 training pictures are input in each batch, so that 275 feature map matrices can be extracted in total from 5 batches, and each feature map matrix comprises 128 three-dimensional feature maps.
S102: converting the characteristic diagram matrix of each convolution layer into a weighted undirected graph, constructing an improved Laplace matrix according to the amplitude matrix of the characteristic diagram matrix, and calculating von Neumann diagram entropy as an original value of each convolution layer; sequentially deleting single vertexes in the weighted undirected graph of each convolution layer to obtain a new weighted undirected graph, and calculating the variation value of the von Neumann diagram entropy of each new weighted undirected graph relative to the original value of the corresponding convolution layer;
the converting of the characteristic map matrix of each convolutional layer into a weighted undirected graph includes:
firstly, deforming each three-dimensional characteristic diagram in the characteristic diagram matrix of each convolution layer to obtain a characteristic row vector matrix, wherein each row is a characteristic row vector corresponding to each channel and is respectively used as a vertex of a weighted undirected graph of each convolution layer;
specifically, for each three-dimensional feature map in the extracted feature map matrix (n i ,h i ,w i ) Is modified to maintain the number of channelsn i Under each channel, (a) without changeh i ,w i ) All elements of the matrix are drawn into a row, resulting inn i A characteristic row vector of (n i ,h i w i ) Corresponding to in an undirected graph with weightsn i A vertex, wherein,h i andw i respectively representing the height and width of the feature map,h i w i representing the number of elements in each feature row vector after deformation. The characteristic row vector matrix obtained after deformation is expressed by the following formula:
wherein,
is shown asiA convolution layer ofjA feature row vector matrix after the feature map is deformed,
in (1)
In the representation and characteristic diagramkThe characteristic line vector corresponding to each channel is taken as the secondiWeighted undirected graph of convolutional layersGOne vertex of (a).
Calculating the cosine distance of any two characteristic row vectors in the characteristic row vector matrix of each convolution layer to serve as the edge weight between two corresponding vertexes in the weighted undirected graph of each convolution layer;
specifically, as shown in equation (4), calculation is performed
Any two characteristic row vectors
And
cosine distance of (d):
wherein,
and
respectively representing characteristic row vectors
And
to middlemAnd (4) each element. Computing
Cosine distances between every two line vectors in the vector space are used as weighted undirected graphs obtained by conversionGThe weight of the edge between the respective two vertices.
And thirdly, acquiring an adjacency matrix and a degree matrix according to the weighted undirected graph.
It should be noted that, the adjacency matrix of the weighted undirected graph is a real symmetric matrix, whose off-diagonal elements are corresponding to the edge weights between two vertices, and the diagonal elements are 0, which is expressed as follows:
wherein,Wrepresenting undirected graph with weightsGOf off-diagonal elements dis p,q Is a vertexpAnd a vertexqThe weight of the edge in between.
The degree matrix of the weighted undirected graph is a diagonal matrix, the diagonal elements of each row of which are the sum of all elements of the corresponding row in the adjacency matrix, and is represented as follows:
wherein,Srepresenting undirected graph with weightsGDegree matrix of diagonal elementss k Is a contiguous matrixWFirst, thekThe sum of all elements in the column.
Through the three steps, the characteristic diagram matrix of each convolutional layer is converted into a weighted undirected graph for describing the correlation of the channels of each convolutional layer.
In order to obtain a more accurate pruning standard, the channel correlation is described by a weighted undirected graph, the individual importance of the channel is described by the amplitude of a characteristic graph, and finally, the channel correlation and the individual importance of the channel are fused to construct an improved laplacian matrix.
Specifically, the magnitude of the feature map is represented by a magnitude matrix of the feature map, which is used to measure the influence of each channel on the neural network model, the magnitude matrix is a diagonal matrix, wherein each diagonal element is the sum of squares of all elements in the corresponding channel in the feature map matrix, and the formula is as follows:
wherein,Dmagnitude matrix representing characteristic diagram, its diagonal elementsd k Is a feature map matrix ofkThe sum of the squares of all the elements in each channel.
After multiplying the degree matrix by the amplitude matrix, subtracting the adjacency matrix to obtain an improved Laplace matrix, wherein the formula is as follows:
wherein,L d the improved Laplace matrix is a real symmetrical semi-positive definite matrix, and the corresponding elements of the matrix are multiplied.
From the modified laplace matrix, von neumann map entropy is calculated by:
wherein,H i is shown asiVon Neumann entropy of each convolutional layer, trace (.) represents the trace of the matrix, i.e. the sum of all eigenvalues of the matrix;λ k representing the modified Laplace matrixL d To middlekThe value of the characteristic is used as the characteristic value,λ k is not less than 0, and
，n i is shown asiNumber of channels per convolutional layer.
It should be noted that the von neumann diagram entropy has good stability, has small correlation with the number of input training pictures, and can obtain the importance of accurate feature diagram channels only by a small number of training pictures. And then, calculating a von Neumann diagram entropy change value after deleting a single channel, and taking the von Neumann diagram entropy as a pruning basis.
Specifically, the method for calculating the change value of the von Neumann diagram entropy of each new weighted undirected graph relative to the original value of the corresponding convolutional layer includes the following steps:
sequentially deleting the characteristic row vectors corresponding to a single vertex aiming at the weighted undirected graph of each convolution layer, reconverting the characteristic row vector matrix obtained after deletion each time into a new weighted undirected graph, constructing a new improved Laplace matrix, and calculating a new von Neumann graph entropy;
it should be noted that, from the original figure, the image isGWith deletion of a single vertex, i.e. from the feature row vector matrix
Deleting corresponding characteristic row vectors, and reconstructing the weighted undirected graph according to formulas (4) to (9)G ’ Obtaining a new improved Laplace matrix according to equation (10)
And obtaining new von Neumann diagram entropy according to equation (11)H ’ 。
Calculating the absolute value of the difference between each new von Neumann diagram entropy and the original value of the corresponding convolution layer as the variation value of the von Neumann diagram entropy of the channel corresponding to the deleted vertex, and the formula is as follows:
wherein abs (.) represents the absolute value calculation.
S103: calculating the importance of the channel of each convolutional layer according to the variation value of the von Neumann diagram entropy in each convolutional layer, pruning the channel of each convolutional layer according to the pruning rate of each convolutional layer, and training the pruned neural network model to obtain a pruning lightweight model;
the von neumann diagram entropy is helpful for measuring information difference and distance between diagrams, and after a certain channel is deleted, if the entropy change of a new von neumann diagram is obvious, the importance of the channel is high, and the channel needs to be reserved in the pruning process so as to ensure the performance of a lightweight model after pruning. Therefore, the goal of model channel pruning is to subtract unimportant convolution channels from the neural network model on the basis of maximally preserving the performance and generalization of the original neural network model. This goal can be defined as shown in the following equation:
wherein,Accrepresenting the performance of the neural network model after pruning;
is shown askThe mask of each channel is 0 or 1 when
When 0 is taken, it means that the channel is cut out, when
When 1 is taken, the channel is reserved; CI (b k ) Is shown askA channelb k The importance of (c);n i representing the number of channels in the convolutional layer;
representing the number of channels remaining in the convolutional layer after pruning.
The importance of the channel of each convolutional layer is approximated according to the change value of the von Neumann diagram entropy in each convolutional layer by the following formula:
wherein, DeltaH k,j Indicates the input isjThe convolution layer is the first when taking picturekA channelb k The obtained von Neumann map entropy change value;
representing the total number of pictures entered.
As can be seen from equations (13) and (14), in order to maximize the performance of the pruned model, the channel with the greatest importance can be equivalently reserved. Therefore, pruning the channels of each convolutional layer according to the pruning rate of each convolutional layer is to prune the channels from small to large according to the importance of the channels of each convolutional layer according to the pruning rate of each convolutional layer.
Taking the ResNet56 neural network model as an example, where the neural network model includes 55 convolutional layers, all channels of each of the 55 convolutional layers are sorted from small to large according to their importance.
Fig. 2 is an example of a convolutional layer, and illustrates a process of obtaining a pruning lightweight model through channel pruning in this embodiment. In fig. 2, the convolutional layer has 4 channels, and after 1 picture is input into the convolutional layer, 1 weighted undirected graph containing 4 vertexes is constructed, the von neumann diagram entropy calculated according to the 1 weighted undirected graph is used as the original value of the convolutional layer, and then the single vertex is sequentially deleted, so that 4 new weighted undirected graphs containing 3 vertexes are constructed, new von neumann diagram entropy is respectively obtained, and the new von neumann diagram entropy is compared with the original value, so that the von neumann diagram entropy change values of the 4 channels are obtained. Because the number of the change values is 1 picture, the size of the change values is equivalent to the size of the importance, when the pruning rate is 0.5, the channel 4 and the channel 1 with smaller change values are pruned according to the sequence from small to large, and the channel 2 and the channel 3 are reserved.
It should be noted that, for the neural network model, the channel importance of the same convolutional layer output feature map is comparable, and the influence of the feature map channels of different convolutional layers on the model is different, so that the channel pruning rate, the channel importance ranking and the pruning operation of the given neural network model are all performed hierarchically in different convolutional layers of the model. The neural network model channel pruning rate can be set according to actual needs so as to realize balance between model performance and model calculated quantity. According to experiments, the shallow convolution channel of the model is often more important than the deep convolution channel, so that the pruning rate of the shallow convolution channel is generally smaller than that of the deep convolution channel. In addition, for the residual error network, it is also necessary to consider that the pruning rate of the residual error output channel matches with that of the trunk network output channel, so as to ensure that the two output characteristics can realize addition operation.
In this embodiment, the fine tuning training of the pruned neural network model includes: setting fine-tuning training hyper-parameters including an initial learning rate, a cosine annealing attenuation strategy with a preheating mechanism, batch size, iteration times, random gradient reduction with momentum and weight attenuation and a layering pruning rate, and performing fine-tuning training on the lightweight model after pruning on a GPU to obtain a trained lightweight pruning model.
S104: and deploying the pruning lightweight model to a package recognition terminal device, and recognizing package information in the real-time collected pictures.
Specifically, the number and position information of the parcels in the picture are identified based on the picture collected in real time, and the index result of parcel identification is counted, wherein the index result comprises the following steps: model size, recognition speed and recognition accuracy.
Illustratively, in a CIFAR-10 dataset, pruning operation is carried out on a ResNet56 neural network model, so that the reduction of 42.8% of parameter quantity and 47.4% of floating point calculation quantity can be realized, and the precision of the lightweight model after pruning is improved by 1.02%; in the ImageNet data set, the ResNet50 neural network model is pruned, 40.8% of parameter quantity and 44.8% of floating point calculation quantity can be reduced, and the precision of the pruned lightweight model is improved by 0.25%. Compared with the neural network model without pruning, the method of the embodiment realizes remarkable model compression and acceleration, and can obtain better model precision than other similar methods under the same or even higher pruning rate.
Compared with the prior art, the package identification method based on the pruning lightweight model provided by the embodiment applies the theories of undirected graphs and von neumann diagram entropies to neural network pruning, converts the characteristic diagram matrix of the neural network model into weighted undirected graphs, considers the amplitude and the correlation of the characteristic diagram, calculates the entropy change value of the von neumann diagram after deleting a single channel, and uses the entropy change value as a pruning basis to ensure the performance of the lightweight model after pruning and realize remarkable model compression and acceleration; the von Neumann diagram entropy of the characteristic diagram matrix has good stability, has small correlation with the number of input training pictures, and can obtain accurate characteristic diagram channel importance only by a small number of training pictures, so that the technical scheme has high pruning efficiency, low memory occupation and stable pruning result; the channel pruning method in the embodiment is suitable for a neural network model formed by convolutional layers with CONV-BN-ReLU structures, parameter quantity and operation quantity of the neural network model are reduced as much as possible while performance is maximized, weight reduction of the neural network model is achieved, the compressed weight-reduced deep neural network model is deployed on edge equipment with limited resources, the use range is expanded, and application value maximization is achieved.
Example two
The invention further discloses a package identification system based on the pruning lightweight model, so that the package identification method in the first embodiment is realized. The specific implementation manner of each module refers to the corresponding description in the first embodiment. As shown in fig. 3, the system includes:
the characteristic diagram matrix extraction module S201 is used for inputting the training picture into the pre-trained neural network model to be pruned and extracting the characteristic diagram matrix of each convolution layer;
the von Neumann map entropy calculation module S202 is used for converting the characteristic map matrix of each convolution layer into a weighted undirected graph, constructing an improved Laplace matrix according to the amplitude matrix of the characteristic map matrix, and calculating the von Neumann map entropy as an original value of each convolution layer; sequentially deleting single vertexes in the weighted undirected graphs of the convolution layers to obtain new weighted undirected graphs, and calculating the change value of von Neumann graph entropy of each new weighted undirected graph relative to the original value;
the channel pruning module S203 is used for calculating the importance of the channel of each convolutional layer according to the variation value of the von Neumann diagram entropy in each convolutional layer, pruning the channel of each convolutional layer according to the pruning rate of each convolutional layer and the importance of the channel, training a neural network model after pruning and obtaining a pruning light weight model;
and the package identification module S204 is used for deploying the pruning lightweight model to the package identification terminal equipment and identifying package information in the real-time acquired pictures.
Since the parts of the package identification system based on the pruning lightweight model related to the identification method in this embodiment can be referred to each other, and are described repeatedly herein, further description is omitted here. All systems adopted by the method in the embodiment of the invention are all covered in the protection scope of the invention. Since the principle of the embodiment of the system is the same as that of the embodiment of the method, the system also has the corresponding technical effect of the embodiment of the method.
Those skilled in the art will appreciate that all or part of the flow of the method implementing the above embodiments may be implemented by a computer program, which is stored in a computer readable storage medium, to instruct related hardware. The computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention.
Claims (6)
1. A parcel identification method based on a pruning lightweight model is characterized by comprising the following steps:
inputting the training picture into a pre-trained neural network model to be pruned, and extracting a characteristic diagram matrix of each convolution layer; the training pictures are obtained by randomly extracting the training pictures from the wrapped picture data set;
converting the characteristic diagram matrix of each convolution layer into a weighted undirected graph, constructing an improved Laplace matrix according to the amplitude matrix of the characteristic diagram matrix, and calculating von Neumann diagram entropy as an original value of each convolution layer, wherein the method comprises the following steps:
deforming each three-dimensional characteristic diagram in the characteristic diagram matrix of each convolution layer to obtain a characteristic row vector matrix, wherein each row is a characteristic row vector corresponding to each channel and is respectively used as the vertex of a weighted undirected graph of each convolution layer;
calculating the cosine distance of any two characteristic row vectors in the characteristic row vector matrix of each convolution layer, and taking the cosine distance as the edge weight between two corresponding vertexes in the weighted undirected graph of each convolution layer;
acquiring an adjacency matrix and a degree matrix according to the weighted undirected graph;
obtaining a magnitude matrix according to the feature map matrix, wherein the magnitude matrix is a diagonal matrix, and each diagonal element is the sum of squares of all elements in corresponding channels in the feature map matrix;
multiplying the degree matrix by the amplitude matrix, and then subtracting the adjacent matrix to obtain an improved Laplace matrix;
the von Neumann map entropy is calculated by the following formula:
wherein,H i is shown asiVon Neumann entropy of each convolutional layer,L d representing the improved Laplace matrix; trace (·) represents the trace of the matrix, i.e., the sum of all eigenvalues of the matrix;λ k representing the modified Laplace matrixL d To middlekThe value of the characteristic is used as the characteristic value,λ k is not less than 0, and
，n i is shown asiThe number of channels of each convolutional layer;
deleting single vertexes in the weighted undirected graph of each convolution layer in sequence to obtain a new weighted undirected graph, and calculating the change value of von Neumann diagram entropy of each new weighted undirected graph relative to the original value, wherein the change value comprises the following steps:
sequentially deleting the characteristic row vectors corresponding to a single vertex aiming at the weighted undirected graph of each convolution layer, reconverting the characteristic row vector matrix obtained after deletion each time into a new weighted undirected graph, constructing a new improved Laplace matrix, and calculating a new von Neumann graph entropy;
calculating the absolute value of the difference value between each new von Neumann diagram entropy and the original value of the corresponding convolution layer to be used as the change value of the von Neumann diagram entropy of the channel corresponding to the deleted vertex;
calculating the importance of the channel of each convolutional layer according to the variation value of the von Neumann diagram entropy in each convolutional layer, pruning the channel of each convolutional layer according to the pruning rate of each convolutional layer and the importance of the channel, and training the pruned neural network model to obtain a pruning lightweight model;
and deploying the pruning lightweight model to a package recognition terminal device, and recognizing package information in the real-time collected pictures.
2. The package identification method based on the pruning lightweight model according to claim 1, wherein the adjacency matrix is a real symmetric matrix, the off-diagonal elements of the adjacency matrix are edge weights between two corresponding vertices, and the diagonal elements are 0; the degree matrix is a diagonal matrix with each row of diagonal elements being the sum of all elements of the corresponding row in the adjacency matrix.
3. The package recognition method based on the pruning lightweight model according to claim 1, wherein the calculating the importance of the channel of each convolution layer according to the variation value of the von Neumann map entropy in each convolution layer comprises:
and calculating the average value of the von Neumann diagram entropy change values of each channel according to the von Neumann diagram entropy change values of the channels of the convolutional layers obtained by each training picture, wherein the average value is used as the importance of each channel.
4. The package identification method based on the pruning lightweight model according to claim 3, wherein the pruning of the channels of each convolutional layer according to the pruning rate of each convolutional layer and the importance of the channels is performed from small to large according to the pruning rate of each convolutional layer and the importance of the channels of each convolutional layer.
5. The package identification method based on the pruning lightweight model according to claim 1 or 4, wherein the neural network model to be pruned is a neural network model composed of convolutional layers having a CONV-BN-ReLU structure.
6. A parcel recognition system based on a pruning lightweight model is characterized by comprising:
the characteristic diagram matrix extraction module is used for inputting the training picture into the pre-trained neural network model to be pruned and extracting the characteristic diagram matrix of each convolution layer; the training pictures are obtained by randomly extracting the training pictures from the package picture data set;
the von Neumann map entropy calculation module is used for converting the characteristic map matrix of each convolution layer into a weighted undirected graph, constructing an improved Laplace matrix according to the amplitude matrix of the characteristic map matrix, and calculating the von Neumann map entropy as an original value of each convolution layer, and comprises the following steps:
deforming each three-dimensional characteristic diagram in the characteristic diagram matrix of each convolution layer to obtain a characteristic row vector matrix, wherein each row is a characteristic row vector corresponding to each channel and is respectively used as the vertex of a weighted undirected graph of each convolution layer;
calculating the cosine distance of any two characteristic row vectors in the characteristic row vector matrix of each convolution layer, and taking the cosine distance as the edge weight between two corresponding vertexes in the weighted undirected graph of each convolution layer;
acquiring an adjacency matrix and a degree matrix according to the weighted undirected graph;
obtaining a magnitude matrix according to the feature map matrix, wherein the magnitude matrix is a diagonal matrix, and each diagonal element is the sum of squares of all elements in a corresponding channel in the feature map matrix;
multiplying the degree matrix by the amplitude matrix, and then subtracting the adjacent matrix to obtain an improved Laplace matrix;
the von Neumann map entropy is calculated by the following formula:
wherein,H i is shown asiVon Neumann entropy of each convolutional layer,L d representing the improved Laplace matrix; trace (·) represents the trace of the matrix, i.e., the sum of all eigenvalues of the matrix;λ k representing an improved Laplace matrixL d To middlekThe value of the characteristic is used as the characteristic value,λ k not less than 0, and
，n i is shown asiThe number of channels of each convolutional layer;
sequentially deleting single vertexes in the weighted undirected graph of each convolution layer to obtain a new weighted undirected graph, and calculating the variation value of the von Neumann graph entropy of each new weighted undirected graph relative to the original value, wherein the variation value comprises the following steps:
sequentially deleting the characteristic row vectors corresponding to a single vertex aiming at the weighted undirected graph of each convolution layer, reconverting the characteristic row vector matrix obtained after deletion each time into a new weighted undirected graph, constructing a new improved Laplace matrix, and calculating a new Von Neumann graph entropy;
calculating the absolute value of the difference value between each new von Neumann diagram entropy and the original value of the corresponding convolution layer, and taking the absolute value as the variation value of the von Neumann diagram entropy of the channel corresponding to the deleted vertex;
the channel pruning module is used for calculating the importance of the channel of each convolutional layer according to the variation value of the von Neumann diagram entropy in each convolutional layer, pruning the channel of each convolutional layer according to the pruning rate of each convolutional layer and the importance of the channel, training the neural network model after pruning and obtaining a pruning lightweight model;
and the package identification module is used for deploying the pruning lightweight model to package identification terminal equipment and identifying package information in the real-time acquired pictures.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210447343.7A CN114548884B (en) | 2022-04-27 | 2022-04-27 | Package identification method and system based on pruning lightweight model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210447343.7A CN114548884B (en) | 2022-04-27 | 2022-04-27 | Package identification method and system based on pruning lightweight model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114548884A CN114548884A (en) | 2022-05-27 |
| CN114548884B true CN114548884B (en) | 2022-07-12 |
Family
ID=81667083
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210447343.7A Active CN114548884B (en) | 2022-04-27 | 2022-04-27 | Package identification method and system based on pruning lightweight model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114548884B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105139430A (en) * | 2015-08-27 | 2015-12-09 | 哈尔滨工程大学 | Medical image clustering method based on entropy |
| WO2016083620A1 (en) * | 2014-11-28 | 2016-06-02 | Imec Taiwan Co. | Autofocus system and method in digital holography |
| CN109886397A (en) * | 2019-03-21 | 2019-06-14 | 西安交通大学 | A kind of neural network structure beta pruning compression optimization method for convolutional layer |
| CN112508225A (en) * | 2020-10-27 | 2021-03-16 | 宁波工程学院 | Multi-detail traffic cell partitioning method and system based on spectral clustering algorithm |
| CN113807466A (en) * | 2021-10-09 | 2021-12-17 | 中山大学 | Logistics package autonomous detection method based on deep learning |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210303536A1 (en) * | 2020-03-19 | 2021-09-30 | NEC Laboratories Europe GmbH | Methods and systems for graph approximation |
-
2022
- 2022-04-27 CN CN202210447343.7A patent/CN114548884B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016083620A1 (en) * | 2014-11-28 | 2016-06-02 | Imec Taiwan Co. | Autofocus system and method in digital holography |
| CN105139430A (en) * | 2015-08-27 | 2015-12-09 | 哈尔滨工程大学 | Medical image clustering method based on entropy |
| CN109886397A (en) * | 2019-03-21 | 2019-06-14 | 西安交通大学 | A kind of neural network structure beta pruning compression optimization method for convolutional layer |
| CN112508225A (en) * | 2020-10-27 | 2021-03-16 | 宁波工程学院 | Multi-detail traffic cell partitioning method and system based on spectral clustering algorithm |
| CN113807466A (en) * | 2021-10-09 | 2021-12-17 | 中山大学 | Logistics package autonomous detection method based on deep learning |
Non-Patent Citations (3)
| Title |
|---|
| Exploring the heterogeneity for node importance by von Neumann;Feng X, et al;《Physica A: Statistical Mechanics and its Applications》;20191231;全文 * |
| 两类图熵的极值特性及在复杂网络中的应用研究;薛玉龙;《中国优秀博硕士学位论文全文数据库(硕士)基础科学辑》;20201215(第12期);全文 * |
| 复杂网络结构特性及其鲁棒性研究;董璊;《中国优秀博硕士学位论文全文数据库(硕士)基础科学辑》;20190915(第9期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114548884A (en) | 2022-05-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Juefei-Xu et al. | Local binary convolutional neural networks | |
| CN108108751B (en) | Scene recognition method based on convolution multi-feature and deep random forest | |
| CN111460968A (en) | Video-based unmanned aerial vehicle identification and tracking method and device | |
| CN109063666A (en) | The lightweight face identification method and system of convolution are separated based on depth | |
| CN112257794A (en) | YOLO-based lightweight target detection method | |
| CN112639828A (en) | Data processing method, method and equipment for training neural network model | |
| CN111723915B (en) | Target detection method based on deep convolutional neural network | |
| CN109840560A (en) | Based on the image classification method for incorporating cluster in capsule network | |
| CN107291825A (en) | With the search method and system of money commodity in a kind of video | |
| CN113705580B (en) | Hyperspectral image classification method based on deep migration learning | |
| CN103927531A (en) | Human face recognition method based on local binary value and PSO BP neural network | |
| CN112446888B (en) | Image segmentation model processing method and processing device | |
| CN109871749B (en) | Pedestrian re-identification method and device based on deep hash and computer system | |
| CN111242180A (en) | Image identification method and system based on lightweight convolutional neural network | |
| RU2674326C2 (en) | Method of formation of neural network architecture for classification of object taken in cloud of points, method of its application for teaching neural network and searching semantically alike clouds of points | |
| CN107679572A (en) | A kind of image discriminating method, storage device and mobile terminal | |
| CN110222718A (en) | The method and device of image procossing | |
| CN111310821A (en) | Multi-view feature fusion method, system, computer device and storage medium | |
| CN113269224A (en) | Scene image classification method, system and storage medium | |
| Kowalczuk et al. | Classification of objects in the LIDAR point clouds using Deep Neural Networks based on the PointNet model | |
| CN112200123A (en) | Hyperspectral open set classification method combining dense connection network and sample distribution | |
| CN116977723A (en) | Hyperspectral image classification method based on space-spectrum hybrid self-attention mechanism | |
| Liu et al. | Image retrieval using CNN and low-level feature fusion for crime scene investigation image database | |
| Lyu et al. | A GPU‐free real‐time object detection method for apron surveillance video based on quantized MobileNet‐SSD | |
| CN108985346B (en) | Existing exploration image retrieval method fusing low-level image features and CNN features |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |