CN118333100A - Network clipping method and device for low-contribution neurons - Google Patents
Network clipping method and device for low-contribution neurons Download PDFInfo
- Publication number
- CN118333100A CN118333100A CN202410748530.8A CN202410748530A CN118333100A CN 118333100 A CN118333100 A CN 118333100A CN 202410748530 A CN202410748530 A CN 202410748530A CN 118333100 A CN118333100 A CN 118333100A
- Authority
- CN
- China
- Prior art keywords
- basic unit
- sequence
- data
- basic
- network model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 210000002569 neuron Anatomy 0.000 title claims abstract description 60
- 238000003062 neural network model Methods 0.000 claims abstract description 83
- 238000012549 training Methods 0.000 claims description 44
- 239000013598 vector Substances 0.000 claims description 42
- 238000005457 optimization Methods 0.000 claims description 31
- 230000006870 function Effects 0.000 claims description 21
- 238000010276 construction Methods 0.000 claims description 10
- 230000009466 transformation Effects 0.000 claims description 9
- 238000000605 extraction Methods 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 8
- 230000008859 change Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 230000009467 reduction Effects 0.000 claims description 3
- 230000011218 segmentation Effects 0.000 claims description 3
- 230000001131 transforming effect Effects 0.000 claims description 3
- 238000010899 nucleation Methods 0.000 claims 1
- 230000008569 process Effects 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 11
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000009977 dual effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000013138 pruning Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 210000004205 output neuron Anatomy 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a network clipping method and device for neurons with low contribution force, which solve the problems of reduced performance, damaged effectiveness and unnecessary complexity of the model caused by clipping of the existing neural network model. The basic unit with lower selected times in the neural network model is updated, so that the influence of the operation on the neural network model is minimized, meanwhile, the interpretability of the neural network model is increased, and a process black box in the process of updating the neural network model is avoided.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a network clipping method and device for neurons with low contribution force.
Background
The main idea of clipping existing neural networks is to reduce the complexity of the model by reducing the number of connections or neurons in the network, thereby reducing the use of computing resources and improving the speed and efficiency of the model. Clipping can be divided into two main modes, weight clipping and structure clipping.
Weight clipping (Weight Pruning), which refers to reducing the number of connections by setting the weight to zero, thereby reducing the number of parameters in the model.
Structure clipping (Structured Pruning), which refers to maintaining the integrity of certain structures (e.g., convolution kernels, neuron groups) during clipping to avoid disrupting the overall structure of the network.
Clipping can significantly reduce the number of parameters and storage requirements of the neural network, thereby saving memory and computing resources. The clipped models typically have faster reasoning speeds because they require fewer parameters to calculate. Improper clipping may result in reduced model performance such as reduced accuracy or insignificant speed improvement. The structure clipping requires a deeper understanding of the model structure, and certain expertise is required to determine the structure to be preserved and to do the clipping. While some network architectures may be less suitable for clipping, it may be possible to destroy their effectiveness or introduce unnecessary complexity after clipping.
Disclosure of Invention
The invention provides a network clipping method and device for neurons with low contribution force, which are used for solving the problems that the performance of a model is reduced, the effectiveness of the model is damaged and unnecessary complexity is introduced due to the clipping of the existing neural network model.
In a first aspect, the present invention provides a network clipping method for neurons with low contribution, which specifically includes the following steps:
Step S1, acquiring multi-mode original data, and constructing an input sequence A= { a 1,a2,...,am } according to the multi-mode original data;
s2, constructing a chain generation type neural network model according to the input sequence A;
S3, training and optimizing the chained generation type neural network model;
step S4, counting the selected times of each basic unit after the training optimization of the chained generation type neural network model is completed;
Each basic unit comprises an object, wherein the object comprises a local network formed by a neuron or a plurality of neurons, and the input quantity and the output quantity of the object are equal; or alternatively
Each basic unit comprises a chain structure formed by linearly connecting a plurality of objects, wherein each object is connected together according to a specific sequence to form a continuous chain; or alternatively
Each basic unit comprises a multi-path structure formed by nesting a plurality of chain structures, and the chain structures are formed by linearly connecting a plurality of basic units to form a continuous and linear network structure;
and S5, updating the basic units with low selected times in the chained generation type neural network model.
Wherein a 1,a2,...,am is m elements in the sequence A, and m is a natural number.
Preferably, in step S1, the multimodal data includes text data, audio data, image data or video data.
Preferably, in step S1, the input sequence includes m elements, where each element represents a basic unit in the text, including a word, a character, a subword, a symbol or other predefined text unit;
the length of the input sequence is the number and the size of elements contained in the input sequence, each element corresponds to an independent basic unit, and the basic units corresponding to different elements are the same or different.
Preferably, in step S1, the input sequence a= { a 1,a2,...,am } is constructed according to the multi-mode raw data, and specifically includes the following steps:
S101, acquiring multi-mode original data;
step S102, dividing the multi-mode original data according to equal interval time to form divided data;
Step S103, an input sequence is constructed according to the segmented data, and an input sequence a= { a 1,a2,...,am }.
Preferably, in step S2, a chain generation type neural network model is constructed according to the input sequence a, and specifically includes the following steps:
Step S201, extracting features of each data segment in the input sequence A to form a sequence B= { B 1,b2,...,bm };
step S202, performing linear or nonlinear transformation on the sequence B to form a sequence X= { X 1,x2,...,xm };
Step S203, each piece of data in the sequence X is respectively input into a basic unit selection network, and a plurality of basic units respectively corresponding to each piece of data in the sequence X are mapped to be used as candidates;
Step S204, for each piece of data in the sequence X, selecting a basic unit with the highest weight corresponding to the data from a plurality of candidate basic units mapped by the data, thereby obtaining m basic units corresponding to the input sequence A;
And step S205, constructing a chain generation type neural network model according to the obtained m basic units.
Wherein B 1,b2,...,bm is m elements in the sequence B, and m is a natural number.
Wherein X 1,x2,...,xm is m elements in the sequence X, and m is a natural number.
Preferably, in step S2, the chain-generated neural network model includes one or more (in the present disclosure, the term "plurality" means at least 2) of the multi-path (in the present disclosure, the term "multi-path" means at least 2 paths) structures.
Preferably, in step S201, the features of the input sequence are extracted, including by means of Embedding, so that the sequence ID of the input sequence is converted into a computable sequence feature.
Preferably, in step S202, the sequence a is transformed linearly or nonlinearly, including transforming the linear or nonlinear sequence feature through the Sequential network, to form a transformed sequence feature B.
Preferably, in step S203, the input and output of the chain structure are determined by the basic units contained therein, and different input and output requirements are allowed to be adapted by adding a conversion layer;
the multi-path structure is formed by nesting a plurality of chain structures, and each chain structure serves as a sub-path and allows information to be processed in parallel on a plurality of paths;
wherein the multi-path structure allows for combining and configuring by different chain structures.
Preferably, in step S203, inputting X into the basic unit selection network, mapping to obtain a plurality of basic units corresponding to each piece of data in the sequence X, including two ways;
Wherein,
The first mode specifically comprises the steps of inputting each piece of data in a sequence X into a basic unit selection network respectively to obtain a plurality of basic units corresponding to the piece of data;
The second mode specifically includes that each piece of data in the sequence X is sequentially input into a basic unit selection network according to the form of X 1,x1+x2,...,x1+x2+...,xm, and a plurality of basic units are obtained.
Preferably, in step S3, a plurality of input sequences for training are input, and training optimization is performed on the chain generation type neural network model, which specifically includes the following steps:
Step S301, constructing a loss function of the chained generation type neural network model according to the service type of the chained generation type network model application;
and step S302, inputting a plurality of input sequences for training, and completing training optimization of the chained generation type neural network model according to the loss function.
Preferably, in step S301, the loss function is used to guide the parameters of the chain generation type neural network model to perform optimization adjustment.
Preferably, in step S5, the updating of the basic unit includes one or more of the following three methods;
the first method specifically comprises the following steps S511-S513:
step S511, obtaining basic units with the selected times smaller than a threshold E;
step S512, calculating the probability of each basic unit with the selected times greater than or equal to a threshold E through a probability model;
step S513, replacing the basic unit with the highest probability;
The second method specifically comprises the following steps S521-S523:
Step S521, performing dimension reduction on all basic units, and converting the basic units into m one-dimensional vectors;
step S522, calculating Euclidean distance between any two vectors in the m vectors;
Step S523, merging basic units with Euclidean distance between two vectors smaller than a threshold F;
The third method specifically comprises the following steps S531-S535:
Step S531, obtaining a plurality of input sequences;
step S532, sequentially passing a plurality of input sequences through all the basic units to obtain the output of each input sequence through each basic unit;
step S533, converting all the outputs of each basic unit into one-dimensional vectors;
Step S534, calculating Euclidean distances among the output vectors of all the basic units;
Step S535, merging the two basic units corresponding to the euclidean distance between the output vectors smaller than the threshold G.
In a second aspect, the present invention further provides a network clipping device for neurons with low contribution, which specifically includes the following modules:
The multi-mode original data acquisition module is used for acquiring multi-mode original data and constructing an input sequence A= { a 1,a2,...,am };
The chain generation type neural network model building module is used for building a chain generation type neural network model according to the input sequence A; the chain generation type neural network model training optimization module is used for inputting a plurality of input sequences for training and carrying out training optimization on the chain generation type neural network model;
The basic unit selected times counting module is used for counting the selected times of each basic unit after the training optimization of the chained generation type neural network model is completed;
And the basic unit updating module is used for updating the basic units with low selected times in the chained generation type neural network model.
Preferably, the multi-mode raw data acquisition module comprises the following sub-modules:
The multi-mode original data segmentation sub-module is used for segmenting the acquired multi-mode original data at equal intervals to form segmented multi-mode original data;
An input sequence construction submodule, configured to construct an input sequence according to the segmented multi-mode raw data, to form a sequence a= { a 1,a2,...,am };
preferably, the chain generation type neural network model building module comprises the following sub-modules:
The sequence feature extraction submodule is used for carrying out feature extraction on each data segment in the input sequence A to form a sequence B= { B 1,b2,...,bm };
The characteristic transformation submodule is used for carrying out linear or nonlinear transformation on the sequence B to form a sequence X= { X 1,x2,...,xm };
the basic unit mapping submodule is used for respectively inputting each piece of data in the sequence X into the basic unit selection network, mapping to obtain a plurality of basic units respectively corresponding to each piece of data in the sequence X, and taking the basic units as candidates;
An optimal basic unit selecting sub-module, configured to, for each piece of data in the sequence X, select, from a plurality of candidate basic units mapped by the data, a basic unit with the highest weight corresponding to the data, thereby obtaining m basic units corresponding to the input sequence a;
And the network model construction submodule is used for constructing a chain generation type neural network model according to the obtained m basic units.
Preferably, the chain generation type neural network model training optimization module comprises the following sub-modules:
The loss function construction submodule is used for constructing a loss function of the chained generation type neural network model according to the service type applied by the chained generation type network model;
And the model optimization training sub-module is used for inputting a plurality of input sequences for training and completing training optimization of the chained generation type neural network model according to the loss function.
Preferably, the basic unit updating module specifically comprises the following sub-modules: the basic unit updating first sub-module, the basic unit updating second sub-module and the basic unit updating third sub-module can realize the updating function of the basic unit, and the structures of the sub-modules are different.
Preferably, the updating of the first sub-module by the basic unit specifically includes the following Sun Mokuai:
The basic unit updating first grandchild module is used for acquiring the basic unit with the selected times smaller than a threshold E;
The basic unit updates a second Sun Mokuai, which is used for calculating the probability of each basic unit with the selected times being greater than or equal to the threshold E through a probability model;
The third grandchild module is used for replacing the removed basic unit with the highest probability;
the basic unit updates the first grandchild module, and the threshold E changes according to the change of the actual service scene.
Preferably, the basic unit updates the second sub-module, specifically including the following Sun Mokuai:
the basic unit updates a fourth grandchild module, which is used for reducing the dimension of all the basic units and converting the dimension into m one-dimensional vectors;
the basic unit updates a fifth grand module for calculating the Euclidean distance between any two vectors in the m vectors;
a basic unit update sixth Sun Mokuai, configured to combine basic units whose euclidean distance between two vectors is less than a threshold F;
and in the sixth grandchild module, the basic units are updated, and the two basic units are combined in a mean value calculation mode.
Preferably, the basic unit updates the third sub-module, specifically including the following Sun Mokuai:
A basic unit update seventh Sun Mokuai for acquiring a plurality of input sequences;
the basic unit updating eighth grandson module is used for inputting a plurality of input sequences to sequentially pass through all the basic units to obtain the output of each input sequence passing through each basic unit;
A base unit update ninth Sun Mokuai for converting all outputs of each base unit into one-dimensional vectors;
The basic unit updating tenth sun module is used for calculating Euclidean distances among output vectors of all the basic units;
The basic unit update eleventh Sun Mokuai is configured to combine two basic units corresponding to the output vectors with a euclidean distance smaller than the threshold G;
In the eleventh Sun Mokuai of the basic unit update, the threshold G is changed according to the change of the actual service scenario; the two basic units are combined by means of calculating the mean value.
In a third aspect, the present application also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor implements a network clipping method of a low-contribution neuron according to any of the first aspects of the application.
In a fourth aspect, the present application also provides an electronic device, including: a memory storing a computer program: and the processor is in communication connection with the memory and executes the network clipping method of the low-contribution neurons according to any one of the first aspect of the application when the computer program is called.
Compared with the prior art, the invention has the following obvious prominent substantive features and obvious advantages:
The invention provides a network clipping method and device for neurons with low contribution force, which solve the problems of reduced performance, damaged effectiveness and unnecessary complexity of the model caused by clipping of the existing neural network model. The basic unit with lower selected times in the neural network model is updated, so that the influence of the operation on the neural network model is minimized, meanwhile, the interpretability of the neural network model is increased, and a process black box in the process of updating the neural network model is avoided.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention. In the drawings:
FIG. 1 is a flow chart of a method of network clipping of low-contribution neurons in accordance with a preferred embodiment of the invention;
FIG. 2 is a flowchart illustrating an example of a basic unit selection network based on an input sequence X for a low-contribution neuron network clipping method according to the preferred embodiment of the present invention;
FIG. 3 is a flowchart illustrating an exemplary process for selecting a network to screen for base units based on an input sequence X accumulation feature in a network clipping method for low-contribution neurons according to the preferred embodiment of the invention;
FIG. 4 is a schematic diagram of a basic unit structure in a network clipping method for low-contribution neurons according to a preferred embodiment of the invention;
FIG. 5 is a schematic diagram of yet another basic unit structure in a network clipping method for low-contribution neurons according to a preferred embodiment of the invention;
FIG. 6 is a schematic diagram of yet another basic unit structure in a network clipping method for low-contribution neurons according to a preferred embodiment of the invention;
FIG. 7 is a schematic diagram of a chain-generated neural network model in a network clipping method for low-contribution neurons according to a preferred embodiment of the invention;
FIG. 8 is a flowchart of a first basic unit updating method in a network clipping method for low-contribution neurons in accordance with a preferred embodiment of the invention;
FIG. 9 is a flow chart of a second basic unit update method in a network clipping method for low-contribution neurons in accordance with a preferred embodiment of the invention;
FIG. 10 is a flow chart of a third basic unit update method in a network clipping method for low-contribution neurons in accordance with a preferred embodiment of the invention;
FIG. 11 is a flowchart showing the steps for constructing an input sequence from acquired multi-modal raw data in a network clipping method for low-contribution neurons according to a preferred embodiment of the invention;
FIG. 12 is a flowchart showing the steps involved in constructing a chain-generated neural network model in a network clipping method for low-contribution neurons in accordance with a preferred embodiment of the present invention;
FIG. 13 is a flowchart of the steps for training and optimizing the chain-generated neural network model in a network clipping method for low-contribution neurons in accordance with the preferred embodiment of the invention;
FIG. 14 is a schematic diagram of a network clipping device for low-contribution neurons in accordance with a preferred embodiment of the invention;
FIG. 15 is a schematic diagram of a multi-modal raw data acquisition module in a low-contribution neuron network clipping device according to a preferred embodiment of the present invention;
FIG. 16 is a schematic diagram of a chain-generated neural network model building block in a low-contribution neuron network clipping device according to the preferred embodiment of the present invention;
FIG. 17 is a schematic diagram of a training optimization module of a chain-generated neural network model in a network clipping device for low-contribution neurons according to a preferred embodiment of the invention; FIG. 18 is a schematic diagram of a basic unit update first sub-module in a low-contribution neuron network clipping device according to the preferred embodiment of the present invention;
FIG. 19 is a schematic diagram of a basic unit update second sub-module in a low-contribution neuron network clipping device according to the preferred embodiment of the present invention;
FIG. 20 is a schematic diagram of a basic unit update third sub-module in a low-contribution neuron network clipping device according to the preferred embodiment of the present invention.
Detailed Description
The invention provides a network clipping method and device for neurons with low contribution force, which are used for making the purposes, technical schemes and effects of the invention clearer and more definite, and the invention is further described in detail below by referring to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
It is noted that the terms "first," "second," and the like in the description and claims of the present invention and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order, and it is to be understood that the data so used may be interchanged where appropriate. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Examples
As shown in fig. 1 to 13, the network clipping method for neurons with low contribution according to the embodiment specifically includes the following steps:
And S1, acquiring multi-mode original data, and constructing an input sequence A= { a 1,a2,...,am } according to the multi-mode original data.
In a specific implementation of the present invention, in step S1, the multimodal data includes text data, audio data, image data or video data.
In a specific implementation of the present invention, in step S1, the input sequence includes m elements, where each element represents a basic unit in the text, including a word, a character, a subword, a symbol, or other predefined text unit;
the length of the input sequence is the number and the size of elements contained in the input sequence, each element corresponds to an independent basic unit, and the basic units corresponding to different elements are the same or different.
In a specific implementation of the present invention, in step S1, an input sequence a= { a 1,a2,...,am } is constructed according to the multi-mode raw data, and specifically includes the following steps:
step S101, collecting multi-mode original data.
Step S102, dividing the multi-mode original data according to the equal interval time to form divided data.
Step S103, an input sequence is constructed according to the segmented data, and an input sequence a= { a 1,a2,...,am }.
And S2, constructing a chain generation type neural network model according to the input sequence A.
In the specific implementation of the present invention, in step S2, a chain generation type neural network model is constructed according to an input sequence a, and specifically includes the following steps:
step S201, performing feature extraction on each data segment in the input sequence to form a sequence b= { B 1,b2,...,bm }.
Step S202, performing linear or nonlinear transformation on the sequence B to form a sequence x= { X 1,x2,...,xm }.
Step S203, each piece of data in the sequence X is respectively input into the basic unit selection network, and a plurality of basic units respectively corresponding to each piece of data in the sequence X are mapped and obtained as candidates.
Step S204, for each piece of data in the sequence X, selecting a basic unit with the highest weight corresponding to the data from a plurality of candidate basic units mapped by the data, thereby obtaining m basic units corresponding to the input sequence A.
And step S205, constructing a chain generation type neural network model according to the obtained m basic units.
In a specific implementation of the present invention, in step S2, the chain-generated neural network model includes one or more of the multi-path structures.
In a specific implementation of the present invention, in step S201, the features of the input sequence are extracted, including by means of Embedding, so that the sequence ID of the input sequence is converted into a computable sequence feature.
In the implementation of the present invention, in step S202, the sequence a is transformed linearly or nonlinearly, including transforming the sequence feature linearly or nonlinearly through the Sequential network, to form a transformed sequence feature B.
In the specific implementation of the present invention, in step S203, the input and output of the chain structure are determined by the basic units contained therein, and different input and output requirements are allowed to be adapted by adding a conversion layer; the multi-path structure is formed by nesting a plurality of chain structures, each chain structure serving as a sub-path, allowing information to be processed in parallel on a plurality of paths.
Wherein the multi-path structure allows for combining and configuring by different chain structures.
In the specific implementation of the present invention, in step S203, the X is input into the basic unit selection network, and mapped to a plurality of basic units corresponding to each piece of data in the sequence X, including two ways.
Wherein,
The first mode specifically includes inputting each piece of data in the sequence X into a basic unit selection network, respectively, and obtaining a plurality of basic units corresponding to the piece of data.
The second mode specifically includes that each piece of data in the sequence X is sequentially input into a basic unit selection network according to the form of X 1,x1+x2,...,x1+x2+...,xm, and a plurality of basic units are obtained.
And step S3, training and optimizing the chained generation type neural network model.
In a specific implementation of the present invention, in step S3, a plurality of input sequences for training are input, and training optimization is performed on the chain generation type neural network model, which specifically includes the following steps:
And step S301, constructing a loss function of the chained generation type neural network model according to the service type applied by the chained generation type network model.
And step S302, inputting a plurality of input sequences for training, and completing training optimization of the chained generation type neural network model according to the loss function.
In a specific implementation of the present invention, in step S301, the loss function is used to guide the parameters of the chain generation type neural network model to perform optimization adjustment.
Step S4, counting the selected times of each basic unit after the training optimization of the chained generation type neural network model is completed;
Each basic unit comprises an object, wherein the object comprises a local network formed by a neuron or a plurality of neurons, and the input quantity and the output quantity of the object are equal; or alternatively
Each basic unit comprises a chain structure formed by linearly connecting a plurality of objects, wherein each object is connected together according to a specific sequence to form a continuous chain; or alternatively
Each basic unit comprises a multi-path structure formed by nesting a plurality of chain structures, and the chain structures are formed by linearly connecting a plurality of basic units to form a continuous and linear network structure.
As shown in fig. 4-7, fig. 4 is a single-input, single-output basic unit, which is composed of one neuron. Fig. 5 shows a single-input and single-output basic unit, the basic unit is divided into three layers of network structures, the first layer and the third layer are respectively composed of a neuron, the middle layer is composed of three neurons, after the basic unit is input, the input is transmitted to the three neurons of the middle layer through three paths, and finally the three neurons respectively transmit the output to the output neurons of the third layer, so that a final output result is obtained. Fig. 6 is a dual input, dual output basic unit. FIG. 7 is a chain-generated neural network model of multiple dual-input, dual-output base units.
And S5, updating the basic units with low selected times in the chained generation type neural network model.
In a specific implementation of the present invention, in step S5, the updating of the basic unit includes one or more of the following three methods;
the first method specifically comprises the following steps S511-S513:
Step S511, obtaining the basic unit with the selected times smaller than the threshold E.
Step S512, calculating the probability of each basic unit with the selected times greater than or equal to the threshold E through a probability model.
Step S513, replacing the basic unit with the highest probability;
The second method specifically comprises the following steps S521-S523:
And step S521, performing dimension reduction on all basic units, and converting the basic units into m one-dimensional vectors.
Step S522, calculating Euclidean distance between any two vectors in the m vectors.
Step S523, merging the basic units with the Euclidean distance between the two vectors smaller than the threshold value F.
The third method specifically comprises the following steps S531-S535:
step S531, a plurality of input sequences are acquired.
Step S532, a plurality of input sequences sequentially pass through all the basic units, and output of each input sequence passing through each basic unit is obtained.
Step S533, converting all the outputs of each basic unit into a one-dimensional vector.
Step S534, calculates euclidean distances between output vectors of all the basic units.
Step S535, merging the two basic units corresponding to the euclidean distance between the output vectors smaller than the threshold G.
Examples
As shown in fig. 14 to 20, the network clipping device for neurons with low contribution according to the present embodiment specifically includes the following modules:
The multi-mode original data acquisition module is used for acquiring multi-mode original data and constructing an input sequence A= { a 1,a2,...,am }.
In a specific implementation of the present invention, the multi-mode raw data acquisition module includes the following sub-modules:
the multi-mode original data segmentation sub-module is used for segmenting the acquired multi-mode original data at equal intervals to form segmented multi-mode original data.
And the input sequence construction submodule is used for constructing an input sequence according to the segmented multi-mode original data to form a sequence A= { a 1,a2,...,am }.
And the chain generation type neural network model construction module is used for constructing a chain generation type neural network model according to the input sequence A.
In a specific implementation of the present invention, the chain generation type neural network model building module includes the following sub-modules:
And the sequence feature extraction submodule is used for carrying out feature extraction on each data segment in the input sequence A to form a sequence B= { B 1,b2,...,bm }.
And the characteristic transformation submodule is used for carrying out linear or nonlinear transformation on the sequence B to form a sequence X= { X 1,x2,...,xm }.
The basic unit mapping sub-module is used for respectively inputting each piece of data in the sequence X into the basic unit selection network, and mapping to obtain a plurality of basic units respectively corresponding to each piece of data in the sequence X as candidates.
And the optimal basic unit selection submodule is used for selecting a basic unit with the highest weight corresponding to each piece of data in the sequence X from a plurality of candidate basic units mapped by the optimal basic unit selection submodule, so that m basic units corresponding to the input sequence A are obtained.
And the network model construction submodule is used for constructing a chain generation type neural network model according to the obtained m basic units.
And the chain generation type neural network model training optimization module is used for inputting a plurality of input sequences for training and carrying out training optimization on the chain generation type neural network model.
In a specific implementation of the present invention, the chain generation type neural network model training optimization module includes the following sub-modules:
And the loss function construction submodule is used for constructing the loss function of the chained generation type neural network model according to the service type applied by the chained generation type network model.
And the model optimization training sub-module is used for inputting a plurality of input sequences for training and completing training optimization of the chained generation type neural network model according to the loss function.
And the basic unit selected times counting module is used for counting the selected times of each basic unit after the training optimization of the chained generation type neural network model is completed.
And the basic unit updating module is used for updating the basic units with low selected times in the chained generation type neural network model.
In a specific implementation of the present invention, the basic unit updating module specifically includes the following sub-modules: the basic unit updating first sub-module, the basic unit updating second sub-module and the basic unit updating third sub-module can realize the updating function of the basic unit, and the structures of the sub-modules are different.
The basic unit updates the first sub-module, which specifically includes the following Sun Mokuai:
The basic unit updates the first grandchild module, and is used for obtaining the basic unit with the selected times smaller than the threshold E.
The basic unit updates the second Sun Mokuai, which is used for calculating the probability of each basic unit with the selected times greater than or equal to the threshold E through a probability model.
And the base unit updating third grandchild module is used for replacing the base unit with the highest obtained probability with the removed base unit.
The basic unit updates the first grandchild module, and the threshold E changes according to the change of the actual service scene.
The basic unit updates the second sub-module, which specifically includes the following Sun Mokuai:
the basic unit updates the fourth grandchild module, which is used for reducing the dimension of all the basic units and converting the dimension into m one-dimensional vectors.
The basic unit updates a fifth grandchild module for calculating the Euclidean distance between any two vectors of the m vectors.
The basic unit updates a sixth Sun Mokuai for merging basic units whose euclidean distance between two vectors is smaller than the threshold F.
And in the sixth grandchild module, the basic units are updated, and the two basic units are combined in a mean value calculation mode.
The basic unit updates the third sub-module, which specifically includes the following Sun Mokuai:
the base unit updates a seventh Sun Mokuai for obtaining a plurality of input sequences.
The basic unit updating eighth grandson module is used for inputting a plurality of input sequences to sequentially pass through all the basic units to obtain the output of each input sequence passing through each basic unit.
The base unit updates a ninth Sun Mokuai for converting all outputs of each base unit into one-dimensional vectors.
The basic unit updates a tenth sun module for calculating Euclidean distances between output vectors of all the basic units.
The basic unit update eleventh Sun Mokuai is configured to combine two basic units corresponding to the output vector with a euclidean distance smaller than the threshold G.
In the eleventh Sun Mokuai of the basic unit update, the threshold G is changed according to the change of the actual service scenario; the two basic units are combined by means of calculating the mean value.
The above description of the specific embodiments of the present invention has been given by way of example only, and the present invention is not limited to the above described specific embodiments. Any equivalent modifications and substitutions for the present invention will occur to those skilled in the art, and are also within the scope of the present invention. Accordingly, equivalent changes and modifications are intended to be included within the scope of the present invention without departing from the spirit and scope thereof.
Claims (20)
1. The network clipping method of the low-contribution neurons is characterized by comprising the following steps of:
Step S1, acquiring multi-mode original data, and constructing an input sequence A= { a 1,a2,...,am } according to the multi-mode original data;
s2, constructing a chain generation type neural network model according to the input sequence A;
S3, training and optimizing the chained generation type neural network model;
step S4, counting the selected times of each basic unit after the training optimization of the chained generation type neural network model is completed;
Each basic unit comprises an object, wherein the object comprises a local network formed by a neuron or a plurality of neurons, and the input quantity and the output quantity of the object are equal; or alternatively
Each basic unit comprises a chain structure formed by linearly connecting a plurality of objects, wherein each object is connected together according to a specific sequence to form a continuous chain; or alternatively
Each basic unit comprises a multi-path structure formed by nesting a plurality of chain structures, and the chain structures are formed by linearly connecting a plurality of basic units to form a continuous and linear network structure;
And S5, updating the basic units with low selected times in the chain generation type neural network model, and reducing the number of the basic units.
2. The method of claim 1, wherein in step S1, the multi-modal raw data includes text data, audio data, image data, or video data.
3. A method of clipping a network of low-contribution neurons according to claim 1, wherein in step S1, the input sequence comprises m elements, wherein each element represents a basic unit in text, including a word, character, subword, symbol or other predefined text unit;
the length of the input sequence is the number and the size of elements contained in the input sequence, each element corresponds to an independent basic unit, and the basic units corresponding to different elements are the same or different.
4. The method for clipping a network of neurons with low contribution according to claim 1, wherein in step S1, an input sequence a= { a 1,a2,...,am } is constructed according to the multi-modal raw data, specifically comprising the following steps:
S101, acquiring multi-mode original data;
step S102, dividing the multi-mode original data according to equal interval time to form divided data;
Step S103, an input sequence is constructed according to the segmented data, and an input sequence a= { a 1,a2,...,am }.
5. The method for clipping a network of neurons with low contribution according to claim 1, wherein in step S2, a chain generation type neural network model is constructed according to an input sequence a, and specifically comprises the following steps:
Step S201, for input sequence Feature extraction is carried out on each data segment in the sequence to form a sequence B= { B 1,b2,...,bm };
step S202, performing linear or nonlinear transformation on the sequence B to form a sequence X= { X 1,x2,...,xm };
Step S203, each piece of data in the sequence X is respectively input into a basic unit selection network, and a plurality of basic units respectively corresponding to each piece of data in the sequence X are mapped to be used as candidates;
Step S204, for each piece of data in the sequence X, selecting a basic unit with the highest weight corresponding to the data from a plurality of candidate basic units mapped by the data, thereby obtaining m basic units corresponding to the input sequence A;
And step S205, constructing a chain generation type neural network model according to the obtained m basic units.
6. The method of claim 5, wherein in step S2, the chain-generated neural network model includes one or more of the multi-path structures.
7. The method according to claim 5, wherein in step S203, the input and output of the chain structure are determined by the basic units contained therein, and the conversion layer is added to accommodate different input and output requirements;
the multi-path structure is formed by nesting a plurality of chain structures, and each chain structure serves as a sub-path and allows information to be processed in parallel on a plurality of paths;
wherein the multi-path structure allows for combining and configuring by different chain structures.
8. The method according to claim 5, wherein in step S201, the feature of the input sequence is extracted, including converting the sequence ID of the input sequence into a computable sequence feature by Embedding.
9. The method of claim 5, wherein in step S202, the sequence a is transformed linearly or nonlinearly, including transforming the sequence features linearly or nonlinearly via a Sequential network, to form transformed sequence features B.
10. The method of network clipping for neurons with low contribution according to claim 5, wherein in step S203, inputting X into the base unit selection network, mapping to obtain a plurality of base units corresponding to each data in the sequence X, respectively, includes two ways;
Wherein,
The first mode specifically comprises the steps of inputting each piece of data in a sequence X into a basic unit selection network respectively to obtain a plurality of basic units corresponding to the piece of data;
The second mode specifically includes that each piece of data in the sequence X is sequentially input into a basic unit selection network according to the form of X 1,x1+x2,...,x1+x2+...,xm, and a plurality of basic units are obtained.
11. The method for clipping a network of neurons with low contribution according to claim 1, wherein in step S3, a plurality of input sequences for training are input, and the training optimization is performed on the chain generation type neural network model, specifically comprising the following steps:
Step S301, constructing a loss function of the chained generation type neural network model according to the service type of the chained generation type network model application;
and step S302, inputting a plurality of input sequences for training, and completing training optimization of the chained generation type neural network model according to the loss function.
12. The method according to claim 11, wherein in step S301, the loss function is used to guide the optimization adjustment of the parameters of the chain-generated neural network model.
13. A method of clipping a network of low-contribution neurons according to claim 1, wherein in step S5, the updating of the base unit comprises one or more of the following three methods;
the first method specifically comprises the following steps S511-S513:
step S511, obtaining basic units with the selected times smaller than a threshold E;
step S512, calculating the probability of each basic unit with the selected times greater than or equal to a threshold E through a probability model;
step S513, replacing the basic unit with the highest probability;
The second method specifically comprises the following steps S521-S523:
Step S521, performing dimension reduction on all basic units, and converting the basic units into m one-dimensional vectors;
step S522, calculating Euclidean distance between any two vectors in the m vectors;
Step S523, merging basic units with Euclidean distance between two vectors smaller than a threshold F;
The third method specifically comprises the following steps S531-S535:
Step S531, obtaining a plurality of input sequences;
step S532, sequentially passing a plurality of input sequences through all the basic units to obtain the output of each input sequence through each basic unit;
step S533, converting all the outputs of each basic unit into one-dimensional vectors;
Step S534, calculating Euclidean distances among the output vectors of all the basic units;
Step S535, merging the two basic units corresponding to the euclidean distance between the output vectors smaller than the threshold G.
14. The network clipping device for the low-contribution neurons is characterized by comprising the following modules:
The multi-mode original data acquisition module is used for acquiring multi-mode original data and constructing an input sequence A= { a 1,a2,...,am };
The chain generation type neural network model building module is used for building a chain generation type neural network model according to the input sequence A;
the chain generation type neural network model training optimization module is used for inputting a plurality of input sequences for training and carrying out training optimization on the chain generation type neural network model;
The basic unit selected times counting module is used for counting the selected times of each basic unit after the training optimization of the chained generation type neural network model is completed;
And the basic unit updating module is used for updating the basic units with low selected times in the chained generation type neural network model.
15. The network clipping device of low-contribution neurons of claim 14, wherein the multi-modal raw data acquisition module comprises the following sub-modules:
The multi-mode original data segmentation sub-module is used for segmenting the acquired multi-mode original data at equal intervals to form segmented multi-mode original data;
And the input sequence construction submodule is used for constructing an input sequence according to the segmented multi-mode original data to form a sequence A= { a 1,a2,...,am }.
16. The network clipping device of low-contribution neurons of claim 14, wherein the chain-generation neural network model building module comprises the following sub-modules:
The sequence feature extraction submodule is used for carrying out feature extraction on each data segment in the input sequence A to form a sequence B= { B 1,b2,...,bm };
The characteristic transformation submodule is used for carrying out linear or nonlinear transformation on the sequence B to form a sequence X= { X 1,x2,...,xm };
the basic unit mapping submodule is used for respectively inputting each piece of data in the sequence X into the basic unit selection network, mapping to obtain a plurality of basic units respectively corresponding to each piece of data in the sequence X, and taking the basic units as candidates;
An optimal basic unit selecting sub-module, configured to, for each piece of data in the sequence X, select, from a plurality of candidate basic units mapped by the data, a basic unit with the highest weight corresponding to the data, thereby obtaining m basic units corresponding to the input sequence a;
And the network model construction submodule is used for constructing a chain generation type neural network model according to the obtained m basic units.
17. The network clipping device of low-contribution neurons of claim 14, wherein the chain-generation neural network model training optimization module comprises the following sub-modules:
The loss function construction submodule is used for constructing a loss function of the chained generation type neural network model according to the service type applied by the chained generation type network model;
And the model optimization training sub-module is used for inputting a plurality of input sequences for training and completing training optimization of the chained generation type neural network model according to the loss function.
18. The network clipping device of low-contribution neurons according to claim 14, wherein the basic unit updating module comprises the following sub-modules: the basic unit updating first sub-module, the basic unit updating second sub-module and the basic unit updating third sub-module can realize the updating function of the basic unit, and the structures of the sub-modules are different;
the basic unit updates the first sub-module, which specifically includes the following Sun Mokuai:
The basic unit updating first grandchild module is used for acquiring the basic unit with the selected times smaller than a threshold E;
The basic unit updates a second Sun Mokuai, which is used for calculating the probability of each basic unit with the selected times being greater than or equal to the threshold E through a probability model;
The third grandchild module is used for replacing the removed basic unit with the highest probability;
The basic unit updates the first grandchild module, and the threshold E is changed according to the change of the actual service scene;
The basic unit updates the second sub-module, which specifically includes the following Sun Mokuai:
the basic unit updates a fourth grandchild module, which is used for reducing the dimension of all the basic units and converting the dimension into m one-dimensional vectors;
the basic unit updates a fifth grand module for calculating the Euclidean distance between any two vectors in the m vectors;
a basic unit update sixth Sun Mokuai, configured to combine basic units whose euclidean distance between two vectors is less than a threshold F;
the basic unit updates the sixth grandchild module, and combines the two basic units in a mean value calculation mode;
The basic unit updates the third sub-module, which specifically includes the following Sun Mokuai:
A basic unit update seventh Sun Mokuai for acquiring a plurality of input sequences;
the basic unit updating eighth grandson module is used for inputting a plurality of input sequences to sequentially pass through all the basic units to obtain the output of each input sequence passing through each basic unit;
A base unit update ninth Sun Mokuai for converting all outputs of each base unit into one-dimensional vectors;
The basic unit updating tenth sun module is used for calculating Euclidean distances among output vectors of all the basic units;
The basic unit update eleventh Sun Mokuai is configured to combine two basic units corresponding to the output vectors with a euclidean distance smaller than the threshold G;
In the eleventh Sun Mokuai of the basic unit update, the threshold G is changed according to the change of the actual service scenario; the two basic units are combined by means of calculating the mean value.
19. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a processor, implements a network clipping method of seeding low-contribution neurons according to any of the claims 1-13.
20. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing a network clipping method for low-contribution neurons according to any of claims 1-13 when the computer program is executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410748530.8A CN118333100B (en) | 2024-06-12 | 2024-06-12 | Network clipping method and device for low-contribution neurons |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410748530.8A CN118333100B (en) | 2024-06-12 | 2024-06-12 | Network clipping method and device for low-contribution neurons |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118333100A true CN118333100A (en) | 2024-07-12 |
| CN118333100B CN118333100B (en) | 2024-08-09 |
Family
ID=91772850
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410748530.8A Active CN118333100B (en) | 2024-06-12 | 2024-06-12 | Network clipping method and device for low-contribution neurons |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118333100B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0694856A1 (en) * | 1994-07-28 | 1996-01-31 | International Business Machines Corporation | Daisy chain circuit for serial connection of neuron circuits |
| CN113160045A (en) * | 2020-01-23 | 2021-07-23 | 百度在线网络技术（北京）有限公司 | Model training method, super-resolution device, electronic device and medium |
| CN116614394A (en) * | 2023-05-16 | 2023-08-18 | 西南交通大学 | Service function chain placement method based on multi-target deep reinforcement learning |
| CN117610689A (en) * | 2023-12-05 | 2024-02-27 | 积至(海南)信息技术有限公司 | Training method of dynamic neural network model based on information entropy integrated learning |
| CN117808072A (en) * | 2024-02-28 | 2024-04-02 | 苏州元脑智能科技有限公司 | Model pruning method, image processing method, device, equipment and medium |
-
2024
- 2024-06-12 CN CN202410748530.8A patent/CN118333100B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0694856A1 (en) * | 1994-07-28 | 1996-01-31 | International Business Machines Corporation | Daisy chain circuit for serial connection of neuron circuits |
| CN113160045A (en) * | 2020-01-23 | 2021-07-23 | 百度在线网络技术（北京）有限公司 | Model training method, super-resolution device, electronic device and medium |
| CN116614394A (en) * | 2023-05-16 | 2023-08-18 | 西南交通大学 | Service function chain placement method based on multi-target deep reinforcement learning |
| CN117610689A (en) * | 2023-12-05 | 2024-02-27 | 积至(海南)信息技术有限公司 | Training method of dynamic neural network model based on information entropy integrated learning |
| CN117808072A (en) * | 2024-02-28 | 2024-04-02 | 苏州元脑智能科技有限公司 | Model pruning method, image processing method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118333100B (en) | 2024-08-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109948029B (en) | Neural network self-adaptive depth Hash image searching method | |
| CN107609694B (en) | Structure optimization method for offshore wind power cluster power transmission system and storage medium | |
| CN111738169A (en) | Handwriting formula recognition method based on end-to-end network model | |
| CN112163601A (en) | Image classification method, system, computer device and storage medium | |
| CN113537365B (en) | Information entropy dynamic weighting-based multi-task learning self-adaptive balancing method | |
| CN112418291B (en) | Distillation method, device, equipment and storage medium applied to BERT model | |
| CN110796233A (en) | Self-adaptive compression method of deep residual convolution neural network based on transfer learning | |
| CN114897155A (en) | Integrated model data-free compression method for satellite | |
| CN116229170A (en) | Task migration-based federal unsupervised image classification model training method, classification method and equipment | |
| CN110009048B (en) | Method and equipment for constructing neural network model | |
| CN113111971A (en) | Intelligent processing method and device for classification model, electronic equipment and medium | |
| CN114676629A (en) | Multi-means composite light weight processing method for modulation type recognition model | |
| CN118333100B (en) | Network clipping method and device for low-contribution neurons | |
| KR102305981B1 (en) | Method for Training to Compress Neural Network and Method for Using Compressed Neural Network | |
| WO2024001653A1 (en) | Feature extraction method and apparatus, storage medium, and electronic device | |
| CN116797850A (en) | Class increment image classification method based on knowledge distillation and consistency regularization | |
| CN116861021A (en) | Cross-modal retrieval model construction method based on denoising and momentum distillation | |
| CN116188785A (en) | Polar mask old man contour segmentation method using weak labels | |
| CN116152263A (en) | CM-MLP network-based medical image segmentation method | |
| CN115062690A (en) | Bearing fault diagnosis method based on domain adaptive network | |
| CN114648005A (en) | Multi-fragment machine reading understanding method and device for multitask joint learning | |
| CN113377907A (en) | End-to-end task type dialogue system based on memory mask self-attention network | |
| CN114299517A (en) | Image processing method, apparatus, device, storage medium, and computer program product | |
| CN118333102B (en) | Chain generation type neural network model construction method and model optimization method thereof | |
| CN113554145A (en) | Method, electronic device and computer program product for determining output of neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |