CN115170917A - Image processing method, electronic device, and storage medium - Google Patents
Image processing method, electronic device, and storage medium Download PDFInfo
- Publication number
- CN115170917A CN115170917A CN202210701707.XA CN202210701707A CN115170917A CN 115170917 A CN115170917 A CN 115170917A CN 202210701707 A CN202210701707 A CN 202210701707A CN 115170917 A CN115170917 A CN 115170917A
- Authority
- CN
- China
- Prior art keywords
- image processing
- processing model
- calculation
- target
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 31
- 238000012545 processing Methods 0.000 claims abstract description 261
- 238000004364 calculation method Methods 0.000 claims abstract description 190
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000013138 pruning Methods 0.000 claims description 39
- 238000004590 computer program Methods 0.000 claims description 17
- 238000004806 packaging method and process Methods 0.000 claims description 13
- 239000010410 layer Substances 0.000 description 31
- 230000008569 process Effects 0.000 description 13
- 239000011159 matrix material Substances 0.000 description 11
- 238000010586 diagram Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 238000012549 training Methods 0.000 description 6
- 238000013135 deep learning Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000006835 compression Effects 0.000 description 4
- 238000007906 compression Methods 0.000 description 4
- 238000010606 normalization Methods 0.000 description 4
- 239000003550 marker Substances 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 239000011229 interlayer Substances 0.000 description 1
- 238000012886 linear function Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000017105 transposition Effects 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The application relates to the technical field of computers, and provides an image processing method, an electronic device and a storage medium, wherein the method comprises the following steps: inputting an image to be processed into a target image processing model, and acquiring target image information output by the target image processing model, wherein the target image processing model is established based on a target mask tensor, and an image channel in the target image information corresponds to a channel with a component of 1 in the target mask tensor; the target mask tensor is determined based on the steps of: acquiring a first calculation graph corresponding to the first image processing model; determining a second computation graph based on the first computation graph; and performing forward calculation and backward calculation of the first image processing model according to the second calculation map to determine a target mask tensor. According to the method, a target image processing model for processing the image to be processed removes redundant channels, reduces the parameter quantity, improves the calculation speed of the model on the premise of not influencing the image processing precision, and reduces the hardware requirement of model deployment.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to an image processing method, an electronic device, and a storage medium.
Background
Since 2012 a deep learning algorithm has drawn attention in an image classification task, an image processing model constructed based on a deep convolutional neural network gradually replaces traditional statistical learning to become a mainstream frame and method of computer vision, and is widely applied to aspects including face recognition, assistant driving and the like.
However, the high-precision image processing model is generally complex in design and huge in parameter quantity, and occupies a high amount of storage space and computational resource consumption when deployed, so that it is difficult to directly deploy and apply the high-precision image processing model at a medium-low computational power device end.
Disclosure of Invention
The present application is directed to solving at least one of the problems in the prior art. Therefore, the application provides an image processing method which reduces the requirement of the image processing task on the calculation capacity of the equipment.
An image processing method according to an embodiment of a first aspect of the present application includes:
inputting an image to be processed into a target image processing model, and obtaining target image information output by the target image processing model, wherein the target image processing model is established based on a target mask tensor, and an image channel in the target image information corresponds to a channel with a component of 1 in the target mask tensor;
the target mask tensor is determined based on the steps of:
acquiring a first calculation graph corresponding to a first image processing model, wherein the first image processing model is a model obtained by structured pruning, the first calculation graph comprises a plurality of first nodes, and the plurality of first nodes are in one-to-one correspondence with a plurality of operators in the first image processing model;
determining a second computational graph based on the first computational graph, wherein the second computational graph comprises a plurality of second nodes, and the second nodes are obtained by packaging at least one first node;
and performing forward calculation and backward calculation of the first image processing model according to the second calculation graph to determine the target mask tensor.
According to the image processing method, the representation map reconstruction is carried out on the first image processing model subjected to model pruning to obtain a first calculation map, operator repackaging is carried out on the basis of the first calculation map to obtain a second calculation map, then the target mask tensor is determined, redundant channels are removed, the parameter number is reduced, the target image processing model is automatically generated, a compact model capable of being arranged at a medium-low computing power equipment end is automatically generated from the sparse model, the image to be processed is processed by the compact model, and the model compression is achieved without affecting the model precision.
According to an embodiment of the application, the determining the target mask tensor by performing the forward calculation and the backward calculation of the first image processing model according to the second computation graph comprises:
performing forward calculation and corresponding backward calculation on the first image processing model according to the second calculation graph, and updating a mark mask corresponding to each operator according to output results of the forward calculation and the corresponding backward calculation;
and circularly executing the forward calculation and the corresponding reverse calculation until the mark mask corresponding to each operator is not updated, and obtaining the target mask tensor based on the mark mask corresponding to each operator.
According to an embodiment of the present application, the performing a forward calculation and a corresponding backward calculation on the first image processing model according to the second calculation graph, and updating the mark mask corresponding to each of the operators according to an output result of the forward calculation and the corresponding backward calculation includes:
and determining that the output result of forward calculation of any operator in the first image processing model is 0 or the solving gradient of reverse calculation is 0, and updating the mark mask corresponding to any operator to be 0.
According to an embodiment of the application, the determining the target mask tensor by performing the forward computation and the backward computation of the first image processing model according to the second computation graph comprises:
and performing forward calculation on the first image processing model according to the second calculation graph, performing reverse calculation on the first image processing model based on an output result of the forward calculation on the first image processing model, and determining the target mask tensor.
According to an embodiment of the present application, the obtaining a first computation graph corresponding to a first image processing model includes:
traversing the plurality of operators of the first image processing model, and determining incidence relations of the plurality of operators;
determining the first computational graph based on the plurality of operators and the incidence relation.
According to an embodiment of the application, the determining a second computation graph based on the first computation graph includes:
and packaging the first nodes with the same scope in the first computational graph as the second nodes, and determining the second computational graph.
According to an embodiment of the application, before said performing forward and backward computations of said first image processing model according to said second computational graph, said method further comprises:
and randomly initializing weights corresponding to operators with weights in the first image processing model to be non-0 values.
An image processing apparatus according to an embodiment of a second aspect of the present application includes:
the processing module is used for inputting an image to be processed into a target image processing model and obtaining target image information output by the target image processing model, the target image processing model is established based on a target mask tensor, and an image channel in the target image information corresponds to a channel with a component of 1 in the target mask tensor;
the target mask tensor is determined based on the steps of:
acquiring a first computational graph corresponding to a first image processing model, wherein the first image processing model is a model obtained by structured pruning, the first computational graph comprises a plurality of first nodes, and the first nodes are in one-to-one correspondence with a plurality of operators in the first image processing model;
determining a second computational graph based on the first computational graph, wherein the second computational graph comprises a plurality of second nodes, and the second nodes are obtained by packaging at least one first node;
and performing forward calculation and backward calculation of the first image processing model according to the second calculation graph to determine the target mask tensor.
An electronic device according to an embodiment of the third aspect of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the image processing method as described in any one of the above when executing the computer program.
A non-transitory computer-readable storage medium according to an embodiment of the fourth aspect of the present application, having stored thereon a computer program which, when executed by a processor, implements the image processing method as any of the above.
A computer program product according to an embodiment of the fifth aspect of the present application comprises a computer program which, when executed by a processor, implements the image processing method as described in any of the above.
One or more technical solutions in the embodiments of the present application have at least one of the following technical effects:
the method comprises the steps of reconstructing a representation diagram of a first image processing model subjected to model pruning to obtain a first calculation diagram, performing operator repackaging based on the first calculation diagram to obtain a second calculation diagram, further determining a target mask tensor, and directly generating a compact model to perform image processing on an image to be processed through an automatic search reconstruction mode.
Furthermore, the mark masks corresponding to all operators in the first image processing model are updated according to whether the output result of the forward calculation is 0 and whether the solving gradient of the reverse calculation is 0, when the mark masks corresponding to all operators in the first image processing model are not updated, the forward calculation and the reverse calculation are stopped, the target mask tensor is determined, the operators and the parameters which need to be reserved when the compact model is established are accurately judged, and the precision of the target image processing model is ensured.
Furthermore, the second nodes in the second calculation graph correspond to the layer structure of the first image processing model one by one, and each operator in the first image processing model can determine the attributes of input and output, a domain name, a parent node, a child node and the like in the second calculation graph.
Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flowchart of an image processing method provided in an embodiment of the present application;
FIG. 2 is a schematic diagram illustrating a deployment process of a target image processing model according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in further detail with reference to the drawings and examples. The following examples are intended to illustrate the present application but are not intended to limit the scope of the present application.
In the description of the embodiments of the present application, it should be noted that the terms "first", "second", and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of an embodiment of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Moreover, various embodiments or examples and features of various embodiments or examples described in this specification can be combined and combined by one skilled in the art without being mutually inconsistent.
The image processing method according to the embodiment of the present application is described below with reference to fig. 1 and fig. 2, and a compact model is automatically generated by using a pruned sparse model to process an image to be processed, so that model compression is realized and the model processing accuracy is ensured.
As shown in fig. 1, the image processing method according to the embodiment of the present invention includes step 110, where an execution subject of the method is a controller of a device terminal, or a cloud, or an edge server.
And 110, inputting the image to be processed into the target image processing model to obtain target image information output by the target image processing model.
The target image processing model is established based on a target mask tensor, and an image channel in target image information corresponds to a channel with a component of 1 in the target mask tensor.
In this embodiment, the target image processing model may perform operations such as image transformation, image enhancement, image restoration, image compression and encoding, image reconstruction, and image segmentation and feature extraction on the image to be processed.
For example, the image to be processed is input to the target image processing model, the target image processing model performs image enhancement operation on the image to be processed, and the output target image information is the image obtained by adding the image to be processed.
For another example, the image to be processed is input to the target image processing model, the target image processing model performs image segmentation and feature extraction operations on the image to be processed, classification processing is further performed, and the output target image information is the type of the object in the image to be processed.
The target image information corresponding to the image to be processed output by the target image processing model comprises information such as image width, image height and image channel.
In the embodiment, a channel with a component of 1 in the target mask tensor is extracted and stored in the initialization operator, and a target image processing model is established, wherein an image channel in the target image information corresponds to the channel with the component of 1 in the target mask tensor.
The target mask tensor is determined based on the steps of:
acquiring a first calculation graph corresponding to a first image processing model, wherein the first image processing model is a model obtained by structured pruning, the first calculation graph comprises a plurality of first nodes, and the plurality of first nodes correspond to a plurality of operators in the first image processing model one to one;
determining a second computational graph based on the first computational graph, wherein the second computational graph comprises a plurality of second nodes, and the second nodes are obtained by packaging at least one first node;
and performing forward calculation and backward calculation of the first image processing model according to the second calculation map to determine a target mask tensor.
It should be noted that, the target image information obtained by the first image processing model performing image processing on the image to be processed through the multiple operators is consistent with the target image information obtained by the target image processing model performing image processing on the image to be processed, the first image processing model is a sparse model obtained by structured pruning, and the target image processing model is a compact model obtained by further removing redundant parameters.
It is understood that the target mask tensor is a tensor that stores the mark masks corresponding to a plurality of operators in the first image processing model, and the number of channels of the operators is equal to the number of components of the target mask tensor.
In this embodiment, after the target mask tensor is determined, only channels with component equal to 1 in the target mask tensor need to be extracted and stored into the initialized new operator, so that redundant channels with component value of 0 can be removed, and a compact target image processing model is established.
The following describes a process of determining a target mask tensor according to an embodiment of the present application:
the method comprises the following steps of firstly, obtaining a first calculation graph corresponding to a first image processing model.
The first image processing model is an image processing model obtained by structured pruning, the structured pruning is to prune by taking a convolution kernel as a unit, an auxiliary layer mask layer is inserted, the mask layer only sets the weight of the corresponding channel to 0, but the 0 weights are not actually deleted from the model, and the obtained first image processing model is a sparse model.
The operators in the first image processing model characterize the operations performed by the functions, which are operations performed on the images input to the first image processing model, such as convolution, feature fusion, and the like.
The first computational graph is a mathematical image that characterizes the relationship between nodes, with node tokens handling numeric, symbolic, or token-type inputs and outputs.
And determining a plurality of first nodes in one-to-one correspondence according to a plurality of operators in the first image processing model, and obtaining a first calculation graph according to the plurality of first nodes.
It should be noted that, in this embodiment, the plurality of operators in the first image processing model refer to all the operators in the first image processing model that characterize the operation performed by the function, and accordingly, the plurality of first nodes of the first computation graph characterize all the processing numbers, symbols, or flag-type inputs and outputs in the first image processing model.
For example, there are operator a, operator B, operator C, operator D, operator E, operator F, operator G, and operator H in the first image processing model.
The obtained first calculation graph corresponding to the first image processing model comprises a first node a, a first node b, a first node c, a first node d, a first node e, a first node f, a first node g and a first node h.
The first node a corresponds to the operator a, the first node B corresponds to the operator B, and so on.
In actual implementation, the first image processing model can be developed by using a deep learning training framework such as paddlepaddlefold, tensrflow, caffe, thano, MXNet, torch, pyTorch, and the like, to obtain a corresponding first computation graph.
Taking the first image processing model as a ResNet18 as an example, the first computational graph developed by the PyTorch deep learning training frame includes 372 first nodes corresponding to 372 operators of several types, and the 372 first nodes can be divided into constant nodes for storing input and output, network module nodes and function nodes, where the network module nodes can correspond to operations such as convolution and activation functions, and the function nodes can correspond to operations such as splicing and transposition.
And secondly, determining a second calculation map based on the first calculation map.
And repackaging the plurality of first nodes in the first computation graph to obtain a plurality of second nodes and further obtain a second computation graph.
In this embodiment, the second node is encapsulated for the at least one first node.
In actual execution, encapsulation and fusion can be performed according to the operator characteristics corresponding to each first node in the first computational graph.
For example, the first node is encapsulated according to the layer structure corresponding to the operator corresponding to the first node in the first image processing model, so as to obtain the second node.
One or more first nodes corresponding to operators belonging to a certain convolution layer in the first image processing model are packaged into a second node, and the second node represents the processing number, symbol or mark type input and output of the convolution layer.
In this embodiment, according to the layer structure feature of the first image processing model, the first node in the first computational graph is repackaged to obtain a plurality of second nodes, and the second computational graph is determined according to the plurality of second nodes.
It can be understood that, according to the layer structure feature of the first image processing model, the second node obtained by encapsulating the first node corresponds to the layer structure of the first image processing model, the number of layers of the first image processing model is large, the number of second nodes in the second calculation graph is large, and the connection relationship of the plurality of second nodes in the second calculation graph represents the interlayer data input and output relationship of the first image processing model.
Taking the first image processing model as ResNet18, the first computational graph developed by the PyTorch deep learning training framework includes 372 first nodes corresponding to 372 operators of several categories as an example.
Conv2d is split into 4 prim:: listConstruct function node and 1 aten:: constraint module node in the first computation graph, and the weights, inputs and outputs of the convolutional layer are stored in constant nodes in the first computation graph.
The input of Batch Normalization layer Batch Normalization is split into 4 constant nodes of weight, bias, running _ mean and running _ var in the first computational graph.
And packaging a plurality of first nodes such as function nodes, module nodes and constant nodes of the convolutional layer torch.nn.Conv2d into a second node, packaging 4 first nodes of Batch Normalization layer Batch Normalization into a second node, and determining a second calculation graph.
The packaged second nodes correspond to the layer structures of the first image processing model one by one, and the number of the second nodes in the second computational graph of ResNet18 obtained after the first nodes are packaged is reduced to 69.
And thirdly, performing forward calculation and backward calculation of the first image processing model according to the second calculation graph, and determining a target mask tensor of the first image processing model.
The forward calculation of the model can be composed of a plurality of operation operations, and the forward calculation of the model can be completed by using a calculation graph corresponding to an operator of the model by a frame.
The reverse calculation is a calculation process of calculating the gradient of each layer in the model by taking the error between the result of the forward calculation and the expected result as input and reversely propagating the error according to an automatic differentiation mechanism.
In this embodiment, the forward calculation and the backward calculation of the first image processing model may be performed according to the second computation graph, and the mark masks of the input and the output are created for all operators in the first image processing model, so as to obtain the target mask tensor of the first image processing model.
It is understood that, when the residual structure occurs in the first image processing model, the mark mask determined by the trunk network of the residual structure may be shared with the branch network of the residual structure, so that the channel dimensions of the trunk network and the branch network after pruning may be consistent.
It should be noted that, the target mask tensor determined according to the mark masks corresponding to the multiple operators in the first image processing model is a multiple linear function, where multiple linearity means that the tensor is linear for each parameter, and the components of the tensor are values of the tensor acting on a corresponding set of basis vectors.
It can be understood that the target mask tensor includes mark masks corresponding to a plurality of operators in the first image processing model, the mark mask corresponding to each operator represents the importance of the parameter of the operator, and whether the parameter is redundant or not and whether the operator needs to be reserved or not is determined according to the mark mask corresponding to each operator.
In this embodiment, compared with the first image processing model, the target image processing model is a compact model with a smaller parameter and a more simplified structure, and the target image processing model has smaller computational resource requirements and memory requirements, and can meet wider application requirements compared with a sparse model that only performs structured pruning.
In actual implementation, after the compact model of the target image processing model is established according to the target mask tensor, the target image processing model can be transplanted on hardware equipment, and the process of establishing the target image processing model is equivalent to the process of pruning the sparse model of the first image processing model.
For example, a convolutional neural network model for performing a computer vision recognition task is subjected to structured pruning to obtain a first image processing model a, and by the image processing method, a compact model of a target image processing model B is obtained, and the target image processing model B can be arranged on an end-side device where an image acquisition device with low or medium computational power is located.
In the embodiment of the application, the first image processing model subjected to model pruning is subjected to representation graph reconstruction to obtain a first calculation graph, operator repackaging is carried out on the basis of the first calculation graph to obtain a second calculation graph, a target mask tensor is further determined, the compact model is directly generated in an automatic search reconstruction mode, the target image processing model can be directly deployed and applied at the end of equipment with medium and low computation power, professional knowledge is not needed, and the time for deploying and landing the labor cost of the sparse model are reduced.
According to the application method of the image processing model, the target image processing model is a compact model generated by an automatic search reconstruction mode based on the first image processing model of the model pruning, the target image processing model for processing the image to be processed removes redundant channels, reduces parameters, improves the calculation speed of the model on the premise of not influencing the model precision, and reduces the hardware requirement of hardware equipment deployed by the target image processing model.
In some embodiments, the third step of determining the target mask tensor in the image processing method may include:
performing forward calculation and corresponding reverse calculation on the first image processing model according to the second calculation graph, and updating a mark mask corresponding to each operator in the plurality of operators according to output results of the forward calculation and the corresponding reverse calculation;
and circularly executing the forward calculation and the corresponding reverse calculation until the mark mask corresponding to each operator in the plurality of operators of the first image processing model is determined to be not updated, and obtaining a target mask tensor based on the mark mask corresponding to each operator.
In this embodiment, the forward calculation and the backward calculation of the first image processing model are performed according to the second computation graph, the input and output marker masks are created for all the operators in the first image processing model, and the marker mask corresponding to each operator is updated through multiple times of forward calculation and multiple times of backward calculation.
And when the mark masks corresponding to all the operators in the first image processing model are not updated, determining the mark mask corresponding to each operator in all the operators, and generating a target mask operation of the first image processing model according to the determined mark mask corresponding to each operator.
In this embodiment, performing a plurality of forward calculations and a plurality of backward calculations on the first image processing model according to the second computation graph, and updating the marker mask corresponding to each of the plurality of operators according to the output results of the forward calculations and the corresponding backward calculations may include:
and determining that the output result of forward calculation of any operator in the first image processing model is 0 or the solving gradient of reverse calculation is 0, and updating the mark mask corresponding to any operator to be 0.
In this embodiment, in the process of performing multiple forward calculation operations and reverse calculation operations, when performing forward calculation, it is checked whether an output result of forward calculation of an operator is 0, if so, a flag mask corresponding to the operator is updated to be 0, when performing reverse calculation, it is checked whether a solving gradient of reverse calculation of the operator is 0, and if so, a flag mask corresponding to the operator is updated to be 0.
For example, a forward calculation operation is performed on a first operator in the first image processing model according to the second calculation graph, and when it is determined that an output result of the forward calculation is 0, the flag mask corresponding to the first operator is updated to 0.
It can be understood that, the mark mask corresponding to the first operator is updated to 0, which indicates that the first operator in the first image processing model belongs to the non-effective position, and the weight and the mark mask are set to 0, and in the process of establishing the target image processing model, it is not necessary to perform an operation on the component of the target mask tensor corresponding to the mark mask of the first operator.
For another example, when it is determined that the solution gradient of the reverse calculation of the second operator is 0, the mark mask corresponding to the second operator is updated to 0, which indicates that the second operator in the first image processing model belongs to an inactive position, and the weight and the mark mask are set to 0, and in the process of establishing the target image processing model, it is not necessary to perform an operation on the component of the target mask tensor corresponding to the mark mask of the second operator.
In this embodiment, the mark masks corresponding to all the operators in the first image processing model are updated according to whether the output result of the forward calculation is 0 and whether the solving gradient of the reverse calculation is 0, and when the mark masks corresponding to all the operators in the first image processing model are not updated, the forward calculation and the reverse calculation are stopped, and the target mask tensor of the first image processing model is determined.
A specific embodiment is described below.
And performing structural pruning to obtain a first image processing model, obtaining an output mask of a BN layer of the first image processing model in the training process of the structural pruning, and after performing representation map reconstruction and operator repackaging to obtain a second calculation map, distributing corresponding mark masks to all output channels and model weights of the first image processing model according to the second calculation map.
Because the output channels of the BN layer and the convolutional layer are in one-to-one correspondence, when forward calculation is executed, the output channel mask code of the convolutional layer can be obtained through reasoning according to the output mask code of the BN layer in the structured pruning process.
In the third step of determining the target mask tensor, before the forward calculation and the backward calculation of the first image processing model are performed according to the second calculation graph, the weight corresponding to the operator with the weight in the first image processing model is randomly initialized to a value other than 0.
It should be noted that, before performing the forward calculation and the backward calculation, all operators with weights in the first image processing model need to be screened out, and the weights are initialized randomly to be non-0 values, so as to avoid removing the operator which is not removed but has a weight value of 0 in the subsequent process of establishing the target image processing model, and prevent interfering with the establishment of the subsequent compact model.
For example, the first image processing model includes a layer structure of Conv1- > BN1- > Conv2- > BN2, and the second computation graph characterizes a layer structure relationship of Conv1- > BN1- > Conv2- > BN 2.
The pruning of the Conv1 convolutional layer is described below.
Assuming that the input image to be processed is an RGB picture, the output shapes of Conv1 and BN1 are (B, C1, H, W), so that the weight shape of Conv1 is (3, C1, K1), the output shapes of Conv2 and BN2 are (B, C2, H, W), and the weight shape of Conv2 is (C1, C2, K2).
In the sparse training of the previous structured pruning, among the C1 channels that have obtained BN1, the mask of D channels is marked as 0, that is, the D dimensions need to be pruned.
When forward calculation is performed, positions needing pruning correspond to output channels of Conv1, namely D channels of Conv1 are marked as 0, the rest are marked as 1, after pruning, the output dimension shape of Conv1 is changed into (B, C1-D, H, W), H represents image height, W represents image width, and C1-D represents image channels.
Next, the weights of Conv1 need to be pruned, and the solution gradient of the weight tensor is calculated by performing back propagation according to the second calculation graph, where the solution gradient at a certain operator position is constantly equal to 0, which means that the weight at the certain operator position is not updated, so that the mask of the operator is marked as 0, and the weight shape of Conv1 after pruning is also changed to (3, C1-D, K1).
In some embodiments, the third step of determining the target mask tensor in the image processing method may include:
and performing forward calculation on the first image processing model according to the second calculation diagram, and performing reverse calculation on the first image processing model based on an output result of the forward calculation of the first image processing model to determine a target mask tensor.
In this embodiment, forward calculation is performed on the first image processing model according to the second calculation map, and backward calculation is performed on the first image processing model according to an output result of the forward calculation.
And performing reverse calculation on the first image processing model, and calculating the gradient of each layer in the first image processing model by taking the error between the output result of the forward calculation and the expected result as input, namely calculating the solving gradient of the output in the reverse calculation.
In some embodiments, the first step of determining the target mask tensor in the image processing method may include:
traversing a plurality of operators of the first image processing model, and determining the incidence relation of the operators;
a first computational graph is determined based on the plurality of operators and the associations.
In the embodiment, the first image processing model obtained by structured pruning is traversed, a plurality of operators of the first image processing model are structurally ordered, the incidence relation among the operators is determined, and the upper-level operator, the lower-level operator and corresponding input and output information are determined for each operator.
According to the multiple operators of the first image processing model, multiple first nodes corresponding to the multiple operators are determined, and according to incidence relations among the multiple operators, connection relations of the multiple first nodes in the first calculation graph are determined.
In actual execution, according to the upper-level operator, the lower-level operator and corresponding input and output information corresponding to each operator in the first image processing model, the parent node, the child node and the corresponding input and output information corresponding to the first node are determined.
In some embodiments, the second step of determining the target mask tensor in the image processing method may include:
and packaging the first nodes with the same scope in the first calculation graph as the second nodes, and determining the second calculation graph.
It will be appreciated that the plurality of first nodes in the first computational graph correspond to a plurality of operators characterizing the operations performed by the function in the first image processing model in which each operator has its own corresponding scope.
The first image processing model is provided with a plurality of layers, the split first nodes of each layer share the same scope, the first nodes of the same scope can be repackaged into new second nodes by searching the first nodes of the same scope, and the first computation graph is packaged to obtain a second computation graph.
In this embodiment, the scope corresponding to the first node is re-encapsulated module by module according to the layer structure to obtain a new second computation graph, the second nodes in the second computation graph correspond to the layer structure of the first image processing model one to one, and each operator in the first image processing model can determine the attributes of input and output, a domain name, a parent node, a child node, and the like in the second computation graph.
The following describes the process of compressing the sparse model obtained by unstructured pruning.
And obtaining a second image processing model through unstructured pruning, setting the redundant parameter of the second image processing model to be 0, and establishing a target sparse matrix based on the second image processing model with the redundant parameter set to be 0, wherein the target sparse matrix is used for storing non-0 elements in the second image processing model.
And mining redundant parameters in the second image processing model, setting the mined redundant parameters to be 0, and setting the 0-set second image processing model to comprise 0 elements and non-0 elements.
In the embodiment, 0 element in the second image processing model is removed in the form of a sparse matrix, a target sparse matrix is established to store non-0 elements in the second image processing model, the storage space of a memory occupied by the second image processing model is saved, and the image to be processed is processed through the model of the target sparse matrix.
For example, the numerical values of all non-0 elements in the two-dimensional matrix of the second image processing model and the line number start and stop points are stored by two one-dimensional sparse matrices, respectively.
In the embodiment, 0 element in the second image processing model is removed in the form of a sparse matrix, so that the storage space of the memory occupied by the second image processing model is saved, the time and labor cost for the second image processing model to land on the ground are reduced, and the model compression is realized without affecting the model precision.
One specific embodiment is described below.
As shown in fig. 2, step 200, model sparsification, a sparsifying operation of pruning is performed on the initial neural network model with a large number of redundant parameters, and pruning includes structured pruning and unstructured pruning.
For structured pruning:
and step 210, structured pruning to obtain a first image processing model.
And step 211, reconstructing the representation, and developing the first image processing model by using a deep learning training frame to obtain a corresponding first calculation graph.
And step 212, repackaging the operators, namely repackaging the corresponding nodes of the operators according to the layer structure of the first image processing model to obtain a second computation graph.
And step 213, creating and updating the mask, and performing forward calculation and reverse calculation of the first image processing model according to the second calculation graph to determine a target mask tensor of the first image processing model.
And 214, removing redundant channels, extracting channels with the component equal to 1 in the target mask tensor, storing the channels into the initialized new operator, removing the redundant channels with the component value of 0, and generating a compact target image processing model.
And step 230, hardware deployment is carried out, and the compact target image processing model obtained in the step 214 is deployed into hardware equipment for application.
For unstructured pruning:
and step 220, unstructured pruning to obtain a second image processing model.
And step 221, setting the weight to 0, mining the redundant parameters in the second image processing model, and setting the mined redundant parameters to 0.
And step 230, hardware deployment is carried out, and the target sparse matrix obtained in step 222 is deployed to hardware equipment for application.
According to the embodiment of the application, the pruned sparse model is further compressed, the parameter quantity of the model is reduced, the obtained compact model and the sparse matrix can be directly deployed on low-computation-force end-side equipment, and the time cost and the labor cost for deploying the neural network model to the ground are reduced.
The following describes an image processing apparatus provided in an embodiment of the present application, and the image processing apparatus described below and the image processing method described above may be referred to correspondingly.
As shown in fig. 3, an image processing apparatus according to an embodiment of the present application includes:
the processing module 310 is configured to input an image to be processed to a target image processing model, and obtain target image information output by the target image processing model, where the target image processing model is established based on a target mask tensor, and an image channel in the target image information corresponds to a channel whose component is 1 in the target mask tensor;
the target mask tensor is determined based on the steps of:
acquiring a first calculation graph corresponding to a first image processing model, wherein the first image processing model is a model obtained by structured pruning, the first calculation graph comprises a plurality of first nodes, and the plurality of first nodes correspond to a plurality of operators in the first image processing model one to one;
determining a second computational graph based on the first computational graph, wherein the second computational graph comprises a plurality of second nodes, and the second nodes are obtained by packaging at least one first node;
and performing forward calculation and backward calculation of the first image processing model according to the second calculation map to determine a target mask tensor.
According to the image processing device provided by the embodiment of the application, the target image processing model is a compact model generated by an automatic search reconstruction mode based on the first image processing model of the model pruning, the target image processing model for processing the image to be processed removes redundant channels, the parameter quantity is reduced, the calculation speed of the model is improved on the premise of not influencing the model precision, and the hardware requirement of hardware equipment deployed by the target image processing model is reduced.
In some embodiments, the processing module 310 is configured to perform a forward calculation and a corresponding backward calculation on the first image processing model according to the second computation graph, and update the tag mask corresponding to each operator according to an output result of the forward calculation and the corresponding backward calculation;
and circularly executing the forward calculation and the corresponding reverse calculation until the mark mask corresponding to each operator is not updated, and obtaining a target mask tensor based on the mark mask corresponding to each operator.
In some embodiments, the processing module 310 is configured to determine that an output result of forward calculation of any operator in the first image processing model is 0 or a solution gradient of backward calculation is 0, and update a flag mask corresponding to any operator to be 0.
In some embodiments, the processing module 310 is configured to perform a forward calculation on the first image processing model according to the second calculation map, and perform a reverse calculation on the first image processing model based on an output of the forward calculation of the first image processing model, to determine the target mask tensor.
In some embodiments, the processing module 310 is configured to traverse a plurality of operators of the first image processing model, determine an association relationship of the plurality of operators;
based on the plurality of operators and the association relationships, a first computational graph is determined.
In some embodiments, the processing module 310 is configured to package first nodes with the same scope in the first computational graph as second nodes, and determine the second computational graph.
In some embodiments, the processing module 310 is configured to randomly initialize weights corresponding to weighted operators in the first image processing model to a value other than 0 before performing the forward and backward calculations of the first image processing model according to the second computation graph.
Fig. 4 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 4: a processor (processor) 410, a communication Interface 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication Interface 420 and the memory 430 are communicated with each other via the communication bus 440. The processor 410 may invoke logic instructions in the memory 430 to perform an image processing method comprising: inputting an image to be processed into a target image processing model, and obtaining target image information output by the target image processing model, wherein the target image processing model is established based on a target mask tensor, and an image channel in the target image information corresponds to a channel with a component of 1 in the target mask tensor;
the target mask tensor is determined based on the steps of:
acquiring a first calculation graph corresponding to a first image processing model, wherein the first image processing model is a model obtained by structured pruning, the first calculation graph comprises a plurality of first nodes, and the plurality of first nodes correspond to a plurality of operators in the first image processing model one to one;
determining a second computational graph based on the first computational graph, wherein the second computational graph comprises a plurality of second nodes, and the second nodes are obtained by packaging at least one first node;
and performing forward calculation and backward calculation of the first image processing model according to the second calculation map to determine a target mask tensor.
In addition, the logic instructions in the memory 430 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Further, the present application also provides a computer program product, the computer program product comprising a computer program, the computer program being storable on a non-transitory computer-readable storage medium, the computer program, when being executed by a processor, being capable of executing the image processing method provided by the above-mentioned method embodiments, the method comprising: inputting an image to be processed into a target image processing model, and obtaining target image information output by the target image processing model, wherein the target image processing model is established based on a target mask tensor, and an image channel in the target image information corresponds to a channel with a component of 1 in the target mask tensor;
the target mask tensor is determined based on the steps of:
acquiring a first calculation graph corresponding to a first image processing model, wherein the first image processing model is a model obtained by structured pruning, the first calculation graph comprises a plurality of first nodes, and the plurality of first nodes correspond to a plurality of operators in the first image processing model one to one;
determining a second computational graph based on the first computational graph, wherein the second computational graph comprises a plurality of second nodes, and the second nodes are obtained by packaging at least one first node;
and performing forward calculation and backward calculation of the first image processing model according to the second calculation graph to determine a target mask tensor.
In another aspect, embodiments of the present application further provide a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented by a processor to perform the image processing method provided in each of the above embodiments, where the method includes: inputting an image to be processed into a target image processing model, and acquiring target image information output by the target image processing model, wherein the target image processing model is established based on a target mask tensor, and an image channel in the target image information corresponds to a channel with a component of 1 in the target mask tensor;
the target mask tensor is determined based on the steps of:
acquiring a first calculation graph corresponding to a first image processing model, wherein the first image processing model is a model obtained by structured pruning, the first calculation graph comprises a plurality of first nodes, and the plurality of first nodes are in one-to-one correspondence with a plurality of operators in the first image processing model;
determining a second computational graph based on the first computational graph, wherein the second computational graph comprises a plurality of second nodes, and the second nodes are obtained by packaging at least one first node;
and performing forward calculation and backward calculation of the first image processing model according to the second calculation map to determine a target mask tensor.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment may be implemented by software plus a necessary general hardware platform, and may also be implemented by hardware. Based on the understanding, the above technical solutions substantially or otherwise contributing to the prior art may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the various embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
The above embodiments are merely illustrative of the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that various combinations, modifications or equivalents may be made to the technical solutions of the present application without departing from the spirit and scope of the technical solutions of the present application, and the technical solutions of the present application should be covered by the claims of the present application.
Claims (10)
1. An image processing method, characterized by comprising:
inputting an image to be processed into a target image processing model, and obtaining target image information output by the target image processing model, wherein the target image processing model is established based on a target mask tensor, and an image channel in the target image information corresponds to a channel with a component of 1 in the target mask tensor;
the target mask tensor is determined based on the steps of:
acquiring a first computational graph corresponding to a first image processing model, wherein the first image processing model is a model obtained by structured pruning, the first computational graph comprises a plurality of first nodes, and the first nodes are in one-to-one correspondence with a plurality of operators in the first image processing model;
determining a second computational graph based on the first computational graph, wherein the second computational graph comprises a plurality of second nodes, and the second nodes are obtained by packaging at least one first node;
and performing forward calculation and backward calculation of the first image processing model according to the second calculation graph to determine the target mask tensor.
2. The image processing method of claim 1, wherein the performing forward and backward calculations of the first image processing model from the second computational graph to determine the target mask tensor comprises:
performing forward calculation and corresponding backward calculation on the first image processing model according to the second calculation graph, and updating a mark mask corresponding to each operator according to output results of the forward calculation and the corresponding backward calculation;
and circularly executing the forward calculation and the corresponding reverse calculation until the mark mask corresponding to each operator is not updated, and obtaining the target mask tensor based on the mark mask corresponding to each operator.
3. The method according to claim 2, wherein said performing a forward calculation and a corresponding backward calculation on the first image processing model according to the second calculation map, and updating the label mask corresponding to each of the operators according to an output result of the forward calculation and the corresponding backward calculation comprises:
and determining that the output result of forward calculation of any operator in the first image processing model is 0 or the solving gradient of reverse calculation is 0, and updating the mark mask corresponding to any operator to be 0.
4. The image processing method of claim 1, wherein the performing forward and backward calculations of the first image processing model from the second computational graph to determine the target mask tensor comprises:
and performing forward calculation on the first image processing model according to the second calculation graph, performing reverse calculation on the first image processing model based on an output result of the forward calculation on the first image processing model, and determining the target mask tensor.
5. The image processing method according to claim 1, wherein the obtaining a first computation graph corresponding to the first image processing model includes:
traversing the plurality of operators of the first image processing model, and determining incidence relations of the plurality of operators;
determining the first computational graph based on the plurality of operators and the incidence relation.
6. The image processing method according to any one of claims 1 to 5, wherein determining a second computation graph based on the first computation graph includes:
and packaging the first nodes with the same scope in the first computational graph as the second nodes, and determining the second computational graph.
7. The image processing method according to any of claims 1-5, wherein before said performing forward and backward computations of the first image processing model according to the second computation graph, the method further comprises:
and randomly initializing weights corresponding to operators with weights in the first image processing model to be non-0 values.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the image processing method according to any of claims 1 to 7 when executing the program.
9. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the image processing method according to any one of claims 1 to 7.
10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, carries out the image processing method according to any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210701707.XA CN115170917B (en) | 2022-06-20 | 2022-06-20 | Image processing method, electronic device and storage medium |
| PCT/CN2023/080053 WO2023246177A1 (en) | 2022-06-20 | 2023-03-07 | Image processing method, and electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210701707.XA CN115170917B (en) | 2022-06-20 | 2022-06-20 | Image processing method, electronic device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115170917A true CN115170917A (en) | 2022-10-11 |
| CN115170917B CN115170917B (en) | 2023-11-07 |
Family
ID=83486568
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210701707.XA Active CN115170917B (en) | 2022-06-20 | 2022-06-20 | Image processing method, electronic device and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN115170917B (en) |
| WO (1) | WO2023246177A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023246177A1 (en) * | 2022-06-20 | 2023-12-28 | 美的集团(上海)有限公司 | Image processing method, and electronic device and storage medium |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109993302A (en) * | 2019-03-06 | 2019-07-09 | 华南理工大学 | The convolutional neural networks channel of knowledge based migration is compressed from selection and accelerated method |
| CN110188795A (en) * | 2019-04-24 | 2019-08-30 | 华为技术有限公司 | Image classification method, data processing method and device |
| CN111652366A (en) * | 2020-05-09 | 2020-09-11 | 哈尔滨工业大学 | Combined neural network model compression method based on channel pruning and quantitative training |
| CN112001431A (en) * | 2020-08-11 | 2020-11-27 | 天津大学 | Efficient image classification method based on comb convolution |
| CN112183725A (en) * | 2020-09-27 | 2021-01-05 | 安徽寒武纪信息科技有限公司 | Method of providing neural network, computing device, and computer-readable storage medium |
| CN112508190A (en) * | 2020-12-10 | 2021-03-16 | 上海燧原科技有限公司 | Method, device and equipment for processing structured sparse parameters and storage medium |
| CN112927173A (en) * | 2021-04-12 | 2021-06-08 | 平安科技（深圳）有限公司 | Model compression method and device, computing equipment and storage medium |
| CN112990458A (en) * | 2021-04-14 | 2021-06-18 | 北京灵汐科技有限公司 | Compression method and device of convolutional neural network model |
| CN113157875A (en) * | 2021-03-12 | 2021-07-23 | 北京智通云联科技有限公司 | Knowledge graph question-answering system, method and device |
| CN113469340A (en) * | 2021-07-06 | 2021-10-01 | 华为技术有限公司 | Model processing method, federal learning method and related equipment |
| CN113673697A (en) * | 2021-08-24 | 2021-11-19 | 平安科技（深圳）有限公司 | Model pruning method and device based on adjacent convolution and storage medium |
| US20210397965A1 (en) * | 2020-06-22 | 2021-12-23 | Nokia Technologies Oy | Graph Diffusion for Structured Pruning of Neural Networks |
| CN113936047A (en) * | 2021-10-14 | 2022-01-14 | 重庆大学 | Dense depth map generation method and system |
| CN113947185A (en) * | 2021-09-30 | 2022-01-18 | 北京达佳互联信息技术有限公司 | Task processing network generation method, task processing device, electronic equipment and storage medium |
| CN114037074A (en) * | 2021-11-09 | 2022-02-11 | 北京百度网讯科技有限公司 | Model pruning method and device, electronic equipment and storage medium |
| CN114341891A (en) * | 2019-09-05 | 2022-04-12 | 华为技术有限公司 | Neural network pruning |
| CN114492753A (en) * | 2022-01-26 | 2022-05-13 | 南京大学 | Sparse accelerator applied to on-chip training |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021061172A1 (en) * | 2019-09-27 | 2021-04-01 | Neuralmagic Inc. | System and method of executing neural networks |
| CN113298263B (en) * | 2020-05-13 | 2022-09-13 | 阿里巴巴集团控股有限公司 | Calculation graph processing method and device, model running method and device, equipment, server and terminal |
| WO2021259039A1 (en) * | 2020-06-22 | 2021-12-30 | 深圳鲲云信息科技有限公司 | Neural network model customization method, system and device, and storage medium |
| CN114444681A (en) * | 2020-11-04 | 2022-05-06 | 安徽寒武纪信息科技有限公司 | Neural network sparsing device, method and corresponding product |
| CN115170917B (en) * | 2022-06-20 | 2023-11-07 | 美的集团(上海)有限公司 | Image processing method, electronic device and storage medium |
-
2022
- 2022-06-20 CN CN202210701707.XA patent/CN115170917B/en active Active
-
2023
- 2023-03-07 WO PCT/CN2023/080053 patent/WO2023246177A1/en unknown
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109993302A (en) * | 2019-03-06 | 2019-07-09 | 华南理工大学 | The convolutional neural networks channel of knowledge based migration is compressed from selection and accelerated method |
| CN110188795A (en) * | 2019-04-24 | 2019-08-30 | 华为技术有限公司 | Image classification method, data processing method and device |
| CN114341891A (en) * | 2019-09-05 | 2022-04-12 | 华为技术有限公司 | Neural network pruning |
| CN111652366A (en) * | 2020-05-09 | 2020-09-11 | 哈尔滨工业大学 | Combined neural network model compression method based on channel pruning and quantitative training |
| US20210397965A1 (en) * | 2020-06-22 | 2021-12-23 | Nokia Technologies Oy | Graph Diffusion for Structured Pruning of Neural Networks |
| CN112001431A (en) * | 2020-08-11 | 2020-11-27 | 天津大学 | Efficient image classification method based on comb convolution |
| CN112183725A (en) * | 2020-09-27 | 2021-01-05 | 安徽寒武纪信息科技有限公司 | Method of providing neural network, computing device, and computer-readable storage medium |
| CN112508190A (en) * | 2020-12-10 | 2021-03-16 | 上海燧原科技有限公司 | Method, device and equipment for processing structured sparse parameters and storage medium |
| CN113157875A (en) * | 2021-03-12 | 2021-07-23 | 北京智通云联科技有限公司 | Knowledge graph question-answering system, method and device |
| CN112927173A (en) * | 2021-04-12 | 2021-06-08 | 平安科技（深圳）有限公司 | Model compression method and device, computing equipment and storage medium |
| CN112990458A (en) * | 2021-04-14 | 2021-06-18 | 北京灵汐科技有限公司 | Compression method and device of convolutional neural network model |
| CN113469340A (en) * | 2021-07-06 | 2021-10-01 | 华为技术有限公司 | Model processing method, federal learning method and related equipment |
| CN113673697A (en) * | 2021-08-24 | 2021-11-19 | 平安科技（深圳）有限公司 | Model pruning method and device based on adjacent convolution and storage medium |
| CN113947185A (en) * | 2021-09-30 | 2022-01-18 | 北京达佳互联信息技术有限公司 | Task processing network generation method, task processing device, electronic equipment and storage medium |
| CN113936047A (en) * | 2021-10-14 | 2022-01-14 | 重庆大学 | Dense depth map generation method and system |
| CN114037074A (en) * | 2021-11-09 | 2022-02-11 | 北京百度网讯科技有限公司 | Model pruning method and device, electronic equipment and storage medium |
| CN114492753A (en) * | 2022-01-26 | 2022-05-13 | 南京大学 | Sparse accelerator applied to on-chip training |
Non-Patent Citations (2)
| Title |
|---|
| HUIYUAN CHEN ET AL.: "Structured Graph Convolutional Networks with Stochastic Masks for Recommender Systems", 《PROCEEDINGS OF THE 44TH INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL》, pages 614 - 623 * |
| 孙依然: "面向深度神经网络的模型压缩与加速研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》, no. 01, pages 3 - 5 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023246177A1 (en) * | 2022-06-20 | 2023-12-28 | 美的集团(上海)有限公司 | Image processing method, and electronic device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023246177A1 (en) | 2023-12-28 |
| CN115170917B (en) | 2023-11-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108154237B (en) | Data processing system and method | |
| CN111615702B (en) | Method, device and equipment for extracting structured data from image | |
| US20100275186A1 (en) | Segmentation for static analysis | |
| EP3938950B1 (en) | Spatially sparse convolutional neural networks for inking applications | |
| US20140279602A1 (en) | Method and system for generating a global representation of a product definition | |
| CN112101364B (en) | Semantic segmentation method based on parameter importance increment learning | |
| CN114124567A (en) | Cloud service processing method based on big data vulnerability mining and artificial intelligence system | |
| CN115170917A (en) | Image processing method, electronic device, and storage medium | |
| CN112766466A (en) | Neural network architecture searching method and device and electronic equipment | |
| CN109117940A (en) | To accelerated method, apparatus and system before a kind of convolutional neural networks | |
| CN111709415B (en) | Target detection method, device, computer equipment and storage medium | |
| CN115344805A (en) | Material auditing method, computing equipment and storage medium | |
| CN114157507A (en) | Cloud service vulnerability analysis method and artificial intelligence system adopting big data analysis | |
| US20220101120A1 (en) | Interpretable visualization system for graph neural network | |
| CN112329551A (en) | Unmanned aerial vehicle autonomous landing method and model training method | |
| CN112070227A (en) | Neural network training method, device and equipment | |
| US20200249968A1 (en) | Method for analyzing a simulation of the execution of a quantum circuit | |
| CN116306951A (en) | Quantum computing method and device, electronic equipment and storage medium | |
| CN112990046B (en) | Differential information acquisition method, related device and computer program product | |
| CN114861934A (en) | Model quantization method, device and equipment for machine learning model | |
| CN113128614A (en) | Convolution method based on image gradient, neural network based on directional convolution and classification method | |
| CN114124654A (en) | Alarm merging method and device, computing equipment and computer storage medium | |
| CN109741264A (en) | The excessive representation method of image, device, electronic equipment and readable storage medium storing program for executing | |
| CN114819171B (en) | Quantum convolution operation realization method and device | |
| CN115906651B (en) | Update method and device of binary neural network and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |