KR102336295B1 - Convolutional neural network system using adaptive pruning and weight sharing and operation method thererof - Google Patents

Abstract

The present invention relates to a neural network system capable of reducing the number of parameters and reducing the amount of computation, and an operating method thereof. The method of operating a convolutional neural network system of the present invention includes the steps of: performing learning on weights between neural network nodes using input data, removing a weight having a size smaller than a threshold value from among the weights, and then receiving the input data an adaptive parameter removal step of performing learning to use; and an adaptive weight sharing step of mapping weights surviving in the adaptive parameter removal step to a plurality of representative values.

Description

Convolutional neural network system using adaptive pruning and weight sharing and method of operation thereof

The present invention relates to a neural network system, and more particularly, to a convolutional neural network system capable of adaptively reducing the number of parameters and an operating method thereof.

Recently, as a technology for image recognition, a convolutional neural network (CNN), which is one of deep neural network techniques, is being actively used. The neural network structure shows excellent performance in various object recognition fields such as object recognition and handwriting recognition. In particular, convolutional neural networks (CNNs) provide very effective performance for object recognition.

The number of parameters used in a convolutional neural network is very large, and since the number of connections between nodes is very large, the size of memory required for operation must be large. In addition, since the convolutional neural network substantially requires a high bandwidth memory, it is not easy to implement in an embedded system or a mobile system. In addition, the convolutional neural network has a disadvantage in that the size of the internal operator increases because it requires a high amount of computation for fast processing.

Therefore, in order to reduce the computational complexity of the neural network algorithm and shorten the recognition time, there is an urgent need for a method of reducing the number of parameters used in the neural network system.

An object of the present invention is to provide a convolutional neural network system capable of reducing the number of parameters used in a convolutional neural network, and an operating method thereof.

A method of operating a convolutional neural network system according to an embodiment of the present invention includes: performing learning on weights between neural network nodes using input data; after removing a weight having a size smaller than a threshold value from among the weights and an adaptive parameter removal step of performing learning using the input data, and an adaptive weight sharing step of mapping weights surviving in the adaptive parameter removal step to a plurality of representative values.

A convolutional neural network system according to an embodiment of the present invention includes an input buffer for buffering input data, an operation unit for learning parameters between a plurality of neural network nodes using the input data, and storing and updating the learning result of the operation unit. an output buffer that transmits the parameters between the plurality of neural network nodes to the operation unit, and updates the parameters according to the learning result, and a weight having a size smaller than a threshold among the weights between the neural network nodes. and a control unit that controls the parameter buffer to remove the values, and maps survival weights among the weights to at least one representative value.

According to embodiments of the present invention, the neural network system of the present invention may provide an output having the same recognition rate without attenuating the recognition accuracy by using a small number of parameters. In addition, the neural network system using the adaptive parameter removal technique and the adaptive representative weight sharing technique of the present invention enables object recognition using a deep learning network in a mobile terminal because parameters can be used in a compressed size of hundreds of times or more. In addition, the neural network system of the present invention is very advantageous in terms of energy consumed per recognition, so that the power required to drive the convolutional neural network system can be dramatically reduced.

1 is a block diagram showing a convolutional neural network system according to an embodiment of the present invention.
2A and 2B are diagrams showing a computational procedure and number of parameters of an exemplary convolutional neural network.
3 is a flowchart schematically illustrating a method of operating a convolutional neural network for reducing parameters according to an embodiment of the present invention.
FIG. 4 is a diagram schematically illustrating a method of reducing parameters of the neural network of the present invention described in FIG. 3 .
5 is a flowchart briefly illustrating an adaptive parameter removal technique according to an embodiment of the present invention.
FIG. 6 is a diagram showing the probability distribution of parameters in stages when the adaptive parameter removal technique of FIG. 5 is applied.
7 is a diagram illustrating an adaptive weight sharing technique according to an embodiment of the present invention.
8 is a table exemplarily showing the effects of the present invention.

In general, a convolution operation is an operation for detecting a correlation between two functions. The term 'Convolutional Neural Network (CNN)' refers to the process of performing a convolution operation with a kernel indicating a specific feature, and repeating the result of the operation to determine the pattern of the image or system can be called.

Hereinafter, embodiments of the present invention will be described clearly and in detail to the extent that those skilled in the art can easily practice the present invention.

1 is a block diagram showing a convolutional neural network system according to an embodiment of the present invention. Referring to FIG. 1 , the convolutional neural network system 100 may generate an output value by processing an input image provided from the external memory 120 .

The input image may be a still image or a moving image provided through an image sensor. The input image may be an image transmitted through a wired/wireless communication means. The input image may represent a two-dimensional array of digitized image data. The input images may be sample images provided for training of the convolutional neural network system 100 . The output value is a result value of the convolutional neural network system 100 processing the input image. The output values are judgment result values for an image input in a learning operation or an estimation operation of the convolutional neural network system 100 . The output value may be a pattern or identification information detected by the convolutional neural network system 100 as being included in the input image.

The convolutional neural network system 100 may include an input buffer 110 , an operation unit 130 , a parameter buffer 150 , an output buffer 170 , and a control unit 190 .

The input buffer 110 is loaded with data values of the input image. The size of the input buffer 110 may vary according to the size of a kernel for a convolution operation. For example, when the size of the kernel is K×K, the input buffer 110 contains input data of sufficient size for sequentially performing a convolution operation (or kerneling) with the kernel by the operation unit 130 . will have to be loaded. The loading of input data into the input buffer 110 may be controlled by the control unit 190 .

The operation unit 130 performs a convolution operation or a pooling operation using the input buffer 110 , the parameter buffer 150 , and the output buffer 170 . The operation unit 130 may, for example, perform kerneling in which multiplication and addition of an input image with a kernel are repeatedly processed. The operation unit 130 may include parallel processing cores for processing a plurality of kerneling or pooling operations in parallel.

The kernel may be provided from, for example, the input buffer 110 or the parameter buffer 150 . The process of multiplying all data of the overlapping positions of the kernel and the input image and summing the results will be referred to as kernelling hereinafter. Each of the kernels may be regarded as a specific feature identifier (Feature Identifier). This kerneling will be performed on the kernels corresponding to the input image and various feature identifiers. A procedure in which such kerneling is performed by all kernels may be performed in a convolution layer, and as a result, feature maps corresponding to a plurality of channels may be generated.

The calculation unit 130 may process the feature maps generated by the convolutional layer as down-sampling. Since the size of the feature maps generated by the convolution operation is relatively large, the operation unit 130 may reduce the size of the feature maps by performing pooling. The result value of each kerneling or pooling operation is stored in the output buffer 170 and may be updated whenever the number of convolution loops increases and whenever a pooling operation occurs.

The parameter buffer 150 provides parameters necessary for kerneling, bias addition, activation (Relu), pooling, etc. performed by the operation unit 130 . In addition, parameters learned in the learning step may be stored in the parameter buffer 150 .

The output buffer 170 is loaded with a result value of kernel ringing or pooling executed by the operation unit 130 . The result value loaded into the output buffer 170 is updated according to the execution result of each convolution loop by the plurality of kernels.

The control unit 190 may control the operation unit 130 to perform a convolution operation, a pooling operation, an activation operation, and the like according to an embodiment of the present invention. The control unit 190 may perform a convolution operation using an input image or a feature map and a kernel. The control unit 190 may control the operation unit 130 to perform an adaptive parameter removal operation for removing low-weight parameters from a learning operation or an actual runtime operation. In addition, the control unit 190 may map a parameter having the same or similar value for each layer as a representative parameter from among the weights surviving through the adaptive parameter removal operation. When the same or similar parameters for each layer are shared as representative parameters, the bandwidth requirement for data exchange with the external memory 120 can be significantly reduced.

In the above, the configuration of the convolutional neural network system 100 of the present invention has been exemplarily described. Through the above-described adaptive parameter removal operation and adaptive parameter sharing operation, the number of parameters to be managed by the convolutional neural network system 100 may be remarkably reduced. As the number of parameters is reduced, a memory size or bandwidth of a memory channel required to configure the convolutional neural network system 100 may be reduced. In addition, reduction in memory size or channel bandwidth may improve the hardware implementation possibility of a convolutional neural network in a mobile device.

2A and 2B are diagrams showing a computational procedure and number of parameters of an exemplary convolutional neural network. 2A is a diagram showing layers of a convolutional neural network for processing an input image. FIG. 2B is a table showing the number of parameters used for each layer shown in FIG. 2A .

Referring to FIG. 2A , a very large number of parameters must be input, newly created, and updated in a convolution operation, a pooling operation, and an activation operation performed in an operation such as learning or object recognition. The input image 210 is processed by a first convolutional layer conv1 and a first pooling layer for downsampling the result (pool1). When the input image 210 is provided, a first convolutional layer conv1 that performs a convolution operation with the kernel 215 is first applied. That is, data of the input image 210 overlapping the kernel 215 is multiplied by data defined in the kernel 215 . Then, all the multiplied values are summed to generate one feature value, and will constitute one point of the first feature map 220 . This kerneling operation will be repeatedly performed while the kernel 215 is sequentially shifted.

A kerneling operation on one input image 210 is performed on a plurality of kernels. In addition, according to the application of the first convolutional layer conv1 , the first feature map 220 in the form of an array corresponding to each of the plurality of channels may be generated. For example, if 4 kernels are used, the first feature map 220 composed of 4 arrays or channels may be generated. However, when the input image 210 is a three-dimensional image, the number of feature maps may rapidly increase, and the depth, which is the number of iterations of the convolution loop, may also rapidly increase.

Subsequently, when the execution of the first convolutional layer conv1 is completed, down-sampling is performed to reduce the size of the first feature map 220 . The data of the first feature map 220 may have a size that becomes a burden for processing according to the number of kernels or the size of the input image 210 . Accordingly, down-sampling (or sub-sampling) for reducing the size of the first feature map 220 is performed in the first pooling layer pool1 in a range that does not significantly affect the operation result. A typical calculation method of downsampling is pooling. While sliding the filter for downsampling in a predetermined stride on the first feature map 220 , a maximum value or an average value in the corresponding region may be selected. The case of selecting the maximum value is called max pooling, and the method of outputting the average value is called average pooling. The first feature map 220 is generated as a second feature map 230 with a reduced size by the pooling layer (pool1).

The convolutional layer on which the convolution operation is performed and the pooling layer on which the downsampling operation is performed may be repeated as needed. That is, as shown, the second convolution layer (conv2) and the second pooling layer (pool2) may be performed. The third feature map 240 may be generated through the second convolutional layer conv2, respectively, and the fourth feature map 250 may be generated by the second pooling layer pool2. In addition, the fourth feature map 250 is generated as fully connected layers 260 and 270 and an output layer 280 through the processing of the fully connected network operations ip1 and ip2 and the activation layer Relu, respectively. The kernel is not used for fully connected network operation (ip1, ip2) and activation layer (Relu). Of course, although not shown, a bias addition or activation operation may be added between the convolutional layer and the pooling layer.

2A and 2B , when an input image having a size of 28×28 pixels is input, a convolution operation using a kernel 215 having a size of 5×5 is performed in the first convolution layer conv1 . Since the convolution operation is performed without padding at the edge of the input image, the first feature map 220 having a size of 24×24 of 20 output channels is output. The number of output channels 20 is the number of channels determined by the number of kernels 215 used in the first convolutional layer conv1. And the bias is a value added between the respective channels, and corresponds to the number of channels.

Under the above conditions, the number of parameters (or weights) used in the first convolutional layer (conv1) is the number of output channels (20), the number of input channels (1), and the size of the kernel (5×5). The multiplied value is 500. In addition, the number of connections in the first convolutional layer conv1 is generated by multiplying the size (24×24) of the output first feature map by the number of parameters (500+20) (299,520). In the first pooling layer (pool1), the width and height of a channel are adjusted while maintaining the number of channels in the spatial domain. The pooling operation has an effect of approximating the characteristic data of the image in the spatial domain.

Each operation of the second convolutional layer (conv2) and the second pooling layer (pool2) performs the same operation as compared to the first convolutional layer (conv1) and the first pooling layer (pool1), only different in the number of channels and the kernel size. do. The number of parameters (or weights) used in the second convolutional layer (conv2) is a value (25000) multiplied by the number of output channels (50), the number of input channels (20), and the size of the kernel (5×5) becomes this And the number of connections in the second convolutional layer conv2 is generated as a value (1,603,200) multiplied by the size (8×8) of the output third feature map 240 and the number of parameters (25000+50).

The fully connected layers (ip1, ip2) perform Fully Connected Networks (FCN) operations. In full network operation, the kernel is not used. All input nodes and all output nodes maintain all connections. Accordingly, the number of parameters in the first fully connected layer ip1 is quite large (400,500).

In the present invention, as shown in FIGS. 2A and 2B , it can be seen that when the number of parameters used is reduced, the number of connections is naturally reduced, thereby reducing the amount of computation. In addition, the parameter can be divided into a weight and a bias. Since the number of biases is relatively small, a method of reducing the weight can be used to provide a high compression effect.

3 is a flowchart schematically illustrating a method of operating a convolutional neural network for reducing parameters according to an embodiment of the present invention. Referring to FIG. 3 , the operation method of the present invention can provide a high parameter reduction effect by applying an operation of removing a low-weight parameter and a weight sharing technique.

In step S110, the original neural network learning for the input is performed. That is, a neural network is trained using inputs in a state where all nodes exist. Then, we can obtain the learned parameters for all connections between nodes. Examining the distribution of the learned parameters in this state, it has a shape similar to that of a normal distribution.

In step S120, an adaptive parameter removal technique is applied to the parameters of the learned neural network. The adaptive parameter removal technique mainly goes through three steps. In the first step, an initial threshold is calculated for every layer of the neural network. In the second step that follows, parameters are removed little by little while repeatedly learning from the initial threshold calculated in the first step. If parameters are continuously removed through iterative learning, parameters lower than the threshold do not occur at any moment. In this case, proceed to step 3. In the third step, the threshold is adjusted upward to further increase the neural network compression efficiency. If the above-described second and third steps are repeatedly used, parameters having low weights are naturally removed and learned. Therefore, only the parameters that are ultimately necessary for the neural network exist.

In step S130, an adaptive representative weight sharing technique is applied to the parameters from which the low weight has been removed. The adaptive representative weight sharing technique is a method of mapping and sharing the same or similar parameters for each layer to a parameter having a single representative value in the neural network. The parameter sharing technique will be described in detail in FIG. 7 .

In step S140, the neural network of parameters processed by the adaptive representative weight sharing technique is retrained. By re-learning, the representative weights of the neural network can be fine-tuned to have high accuracy.

The learning process of a neural network using the adaptive parameter removal technique and the adaptive representative weight sharing technique according to an embodiment of the present invention has been described above. By using the adaptive parameter removal technique and the adaptive representative weight sharing technique, the number of parameters required for the neural network can be drastically reduced. In addition, as the number of parameters decreases, the complexity and amount of computation of neural network operations are reduced. Accordingly, the size and bandwidth of memory required for neural network operation can be significantly reduced.

FIG. 4 is a diagram schematically illustrating a method of reducing parameters of the neural network of the present invention described in FIG. 3 . Referring to FIG. 4 , the convolutional neural network system 100 (refer to FIG. 1 ) of the present invention may configure a neural network with a remarkably reduced number of parameters by performing an adaptive parameter removal technique and an adaptive representative weight sharing technique.

(a) shows the original neural network before weight removal or sharing is applied. That is, the neural network learning should proceed in a state where all nodes of the neural network exist. In the original neural network, the connection relationship of each of the 12 nodes N1 to N16 is illustrated by way of example. There are connections between the nodes N1 to N4 and the nodes N5 to N8, each of which can be represented by four weights for each node. Similarly, the neural network may be configured to have connections between the nodes N5 to N8 and the nodes N9 to N12, respectively, which can be represented by four weights. Weights between nodes of this original neural network will be generated as learned parameters. It is known that the distribution of the learned parameters in this state has a shape similar to that of a normal distribution.

An adaptive parameter removal technique for removing low-weight parameters for the original neural network of (a) will be applied. This procedure is indicated in the identification number ①. The adaptive parameter removal technique used in the present invention as a low weight removal technique is as follows. That is, an initial threshold value is generated for each layer. Then, iterative learning is performed starting from the generated initial threshold. Before learning, a weight higher than the initial threshold for each layer will be lower than the initial threshold after learning. In this case, the parameter learned with a weight lower than the initial threshold is removed. Once removed, the connection between nodes is not restored, and learning is repeated. The reduced neural network of (b) will be generated through iterative learning with this initial threshold applied.

As the learning iteration proceeds, a moment occurs when the weights of connections between nodes no longer fall below the initial threshold. In this case, if learning is repeated by applying an elevated threshold higher than the initial threshold in order to increase the compression efficiency of the neural network, only nodes and weights that are absolutely necessary in the neural network survive. When the changed threshold is determined, pruning will be performed again to further reduce the neural network. In this process, weights with low values are removed while repeating ② re-learning and ③ japrune loop.

After the parameters of the low weight using the threshold are removed, the adaptive weight sharing technique using the representative value is applied. That is, a weight sharing method is applied to nodes and weights that have survived by applying the parameter removal method. Convolutional neural networks have the characteristic of weight sharing. Using this characteristic, parameters similar to or identical to representative values can be grouped and managed for nodes and weights reduced by the adaptive parameter removal technique. As an example of sharing weights, if the weights between the nodes N1 and N5 and the weights of the nodes N1 and N6 are similar, these connections may be mapped to a weight of one representative value. In addition, if the weights between the nodes N6 and N9 and the weights of the nodes N9 and N9 are similar, these connections may be mapped to a weight of one representative value.

The form of a neural network to which the above-described adaptive weight sharing technique is applied is shown in (d). And if the neural network generated by the adaptive weight sharing technique is processed through the ⑤ re-learning process, the final neural network of (e) can be constructed. Consequently, after the adaptive parameter removal technique is applied, the nodes N7 and N10 and weights related to the nodes N7 and N10 may all be removed.

5 is a flowchart briefly illustrating an adaptive parameter removal technique according to an embodiment of the present invention. Referring to FIG. 5 , the parameters of the low weight of the neural network may be removed through repetition of a learning process accompanied by setting and adjusting a threshold value. And only essential parameters that determine the characteristics of the neural network can survive by the adaptive parameter removal technique. By initial learning, the weights for all nodes are provided to the original neural network.

In step S210, an initial threshold is calculated for each layer of the original neural network. In a neural network, each layer has different importance. For example, it is very important because the earliest feature point is extracted from the first convolutional layer of the input image. In addition, the last convolutional layer is important because probabilities for feature values are computed. Removal of parameters for relatively important layers should be prudent. Therefore, before removing the parameters of important layers, it is necessary to investigate the sensitivities of these layers.

In the neural network, each layer needs to investigate the sensitivity while adjusting the threshold value to remove the weight while the remaining layers are maintained. For example, in order to calculate the initial threshold value of the first convolutional layer, while maintaining the connection between nodes of the remaining layers, the threshold value is gradually increased to calculate the accuracy according to the removal rate. When the threshold is raised little by little, there is a section where the accuracy suddenly drops significantly, and the threshold at this time is determined as the initial threshold of the first layer. For the remaining layers, the initial threshold can be obtained by calculating the thresholds little by little in various ways.

In step S220, iterative learning is performed starting with the initial threshold determined in step S210. First, as for the weights of each layer of the neural network, as learning progresses, a weight higher than the threshold may come down below the threshold. In addition, among the connections of the original neural network, there will be ones with weights lower than the determined initial threshold. Parameters with weights lower than this initial threshold are removed in this step. Here, the connection between nodes that have been removed once is not restored, and learning proceeds in a state where it is removed as it is. If parameters are continuously removed through iterative learning, at some point, parameters lower than the threshold will not occur.

In step S230 , it is detected whether there is a weight lower than an initial threshold among the weights between nodes generated as a result of the learning progress. If, as a result of the learning progress, a weight lower than the initial threshold is no longer detected (No direction), the procedure moves to step S240. On the other hand, if it is detected that there is still a weight lower than the initial threshold (yes direction), the procedure returns to S220, and additional learning and weight removal procedures will be performed.

In step S240 , when the weight lower than the initial threshold no longer exists, the threshold is adjusted upward in order to increase compression efficiency. When the threshold is raised, the standard deviation of the parameter is calculated for each layer. Then, the calculated standard deviation of the parameters for each layer is multiplied by a certain ratio and reflected when the threshold is raised. When the threshold value is changed, re-learning by applying the increased threshold value is performed.

In step S250 , it will be detected whether a weight equal to or less than the raised threshold value exists for each layer among the parameters of the learned neural network. If, among the parameters of the neural network, a weight lower than the raised threshold is no longer detected (No direction), the overall procedure of the adaptive parameter removal technique is terminated. On the other hand, if it is detected that there is still a weight lower than the raised threshold (yes direction), the procedure returns to S240, and additional learning and weight removal procedures will be performed.

As described above, if learning and parameter removal are repeated while varying the threshold, parameters having low weights are naturally removed, and only parameters necessary for the neural network ultimately survive. When the adaptive parameter removal technique is used, the original neural network is already learned parameters. Accordingly, the adjustment of the threshold value may be performed in units of small step values. In the process of obtaining these step values, the distribution of the parameters of the corresponding layer is very important.

FIG. 6 is a diagram showing the probability distribution of parameters in stages when the adaptive parameter removal technique of FIG. 5 is applied. Referring to FIG. 6 , as for the weights of the neural network, only parameters existing above a threshold will survive by the adaptive parameter removal technique of the present invention.

(a) shows the probability distribution of the original weights and the initial threshold. The original weight means the weights of the original neural network in which all nodes exist after the learning process. These weights generated by learning form a normal distribution centered on the mean (0). In addition, it can be seen that weights having a value lower than the initial threshold occupies a majority among all parameters.

(b) shows the probability distribution of parameters when weights having values lower than the initial threshold are removed. That is, by pruning using the initial threshold, parameters included in the positive and negative sections of the initial threshold are removed based on the average.

(c) shows the distribution of weights generated as a result of performing additional learning after removing parameters with weights lower than the initial threshold. By re-learning, the sharp distribution at the initial threshold is changed to a smooth distribution. However, as a result of re-learning, parameters having a weight lower than the initial threshold exist. It will be well understood that these parameters may be removed through iteration of the pruning and re-learning loop.

(d) shows the process of deleting low-weighted parameters using the adjusted threshold. That is, if an elevated threshold of a level higher than the initial threshold is calculated and the re-learning step of ④ is performed, the final weight distribution of (e) is obtained.

Among all parameters (especially weights), there are low-level weights that do not significantly affect neural network computation. In the probability distribution, such low-weight parameters occupy a relatively large number, and these low-weight parameters act as burdens for convolution, activation, and pooling operations. According to the adaptive parameter removal technique of the present invention, these low-weight parameters can be removed at a level that hardly affects the performance of the neural network.

7 is a diagram illustrating an adaptive weight sharing technique according to an embodiment of the present invention. Referring to FIG. 7 , similar parameters for each layer in the neural network are mapped and shared as parameters having a single representative value according to the adaptive representative weight sharing technique.

When parameters with low weights are removed according to the adaptive parameter removal technique, the weights distributed in each layer have a bimodal distribution. The beekeeping distribution is divided into a negative region and a positive region. At this time, the representative value is determined by evenly distributing from the lowest value to the highest value for each area. These representative values are called centroids. The weights of the beekeeping distribution and the distribution of an exemplary centroid in each of these regions are shown in the graph below the figure.

The centroid value for each area is determined according to the total number of representative values set. If N centroid values are used, N/2 centroid values will be used in the negative band, and N/2 centroid values will be used in the positive band. Each of the initial centroid values is arranged linearly to have an equal difference value. In the graph shown, four centroids (-3.5, -2.5, -1.5, -0.5) are allocated in the negative band, and four centroids (0.5, 1.5, 2.5, 3.5) can be allocated in the positive band. have.

According to the above-described centroid setting, the weights of the exemplary real weight set 320 are approximated based on the centroid value. That is, the real weights may be approximated by the centroid value 345 , and may be mapped to the centroid index 340 after being approximated. Through approximation to such a representative value, the real weight set 320 may be converted into an index map 360 . The linearly arranged initial centroid values are refined through a re-learning process.

When the centroid value is used in the deep learning recognition engine, a mapping table between the centroid index 340 and the centroid value may be used. If the centroid mapping table of the corresponding layer is read before the recognition operation is performed and stored in the recognition hardware engine, only the index value can be read from the memory and processed thereafter.

In the above, a method using a centroid as an example of an adaptive representative weight sharing method has been described. However, it will be understood that the adaptive representative weight sharing method of the present invention is not limited thereto, and representative values may be mapped and shared in various ways.

8 is a table exemplarily showing the effects of the present invention. Referring to FIG. 8 , the result of reducing the parameters of the neural network (eg, LeNet network) shown in FIG. 2 without loss of accuracy is shown.

By using the adaptive parameter removal technique, it was confirmed that the weights among parameters can be reduced from a total of 430,500 to 12,261. This means that 97.15% of the total weight is not used, and even if only 2.85% is used, it is possible to recognize handwritten digits without affecting the accuracy. In addition, when the adaptive representative weight sharing technique is used, it can be seen that there is no problem in number recognition even if only 8 centroid parameters are used for each layer in the LeNet neural network. In practice, the total number of weights used is only 32. Accordingly, when the 12,261 parameters are stored in the memory, they may be stored as index values indicating representative weights rather than weight values.

When the above-described adaptive parameter removal technique and adaptive representative weight sharing technique are used, the size of a memory for driving a neural network can be remarkably reduced. In addition, the memory bandwidth requirement according to the reduction of parameters stored in the memory is also greatly reduced. In addition, the power required for convolution operation, pooling operation, activation operation, etc. due to the decrease in the number of parameters is expected to be drastically reduced.

As described above, the present invention proposes a parameter compression method for reducing the amount of computation, which is the biggest problem when implementing a neural network in hardware. When the adaptive parameter removal technique and the adaptive representative weight sharing technique of the present invention are used, an output having the same recognition rate can be implemented using a relatively small number of parameters without the effect of attenuating the recognition accuracy. In addition, according to the compression method of the present invention, the size of the parameter can be compressed several hundred times or more from the scale of several megabytes (MByte) to the scale of several megabytes (MByte). Accordingly, object recognition using a deep learning network is possible even in a mobile terminal. This feature is also very advantageous in terms of energy management.

The contents described above are specific examples for carrying out the present invention. The present invention will include not only the above-described embodiments, but also simple design changes or easily changeable embodiments. In addition, the present invention will include techniques that can be easily modified and implemented in the future using the above-described embodiments.

Claims (16)

A method of operating a convolutional neural network system, comprising:
performing learning on weights between neural network nodes using input data;
an adaptive parameter removal step of performing learning using the input data after removing a weight having a size smaller than a threshold value from among the weights; and
an adaptive weight sharing step of mapping the weights surviving in the adaptive parameter removal step to a plurality of representative values;
The adaptive parameter removal step includes:
determining an initial threshold value for each layer of the neural network;
performing weight removal and learning using the initial threshold; and
and performing weight removal and learning using an upward threshold having a value greater than the initial threshold. The method of claim 1,
In the step of performing the learning, the learning is performed in a state including all nodes of the neural network, and learned weights for connections between all the nodes are generated. delete The method of claim 1,
In the determining of the initial threshold value, the initial threshold value of each of the layers is applied by sequentially varying the adjusted threshold value while maintaining the connection of the other layers, and setting a threshold value lower than the reference accuracy. An operating method of determining the initial threshold value of each of the layers. The method of claim 1,
The step of weight removal and learning using the initial threshold includes:
removing weights having a size smaller than the initial threshold from among the weights of each of the layers; and
and performing learning on survival weights from which weights having a size smaller than the initial threshold have been removed. 6. The method of claim 5,
The steps of removing weights having a size smaller than the initial threshold and performing learning on the survival weights constitute an iterative loop, wherein the iterative loop is performed when weights having a size smaller than the initial threshold are removed. Repeated operation method until . The method of claim 1,
The step of weight removal and learning using an upward threshold having a value greater than the initial threshold includes:
removing weights smaller than the upward threshold from among the survival weights; and
and performing learning on weights equal to or greater than the upward threshold. 8. The method of claim 7,
The steps of removing the weights having a size smaller than the upward threshold and learning the weights having a size equal to or greater than the upward threshold constitute an iterative loop, wherein the iterative loop is smaller than the upward threshold. A method of operation that is repeated until smaller weights are removed. The method of claim 1,
In the adaptive weight sharing step, a plurality of representative values are determined as centroid values of the surviving weights. 10. The method of claim 9,
The centroid value is refined through re-learning of the surviving weight. In a convolutional neural network system:
an input buffer for buffering input data;
an arithmetic unit for learning parameters between a plurality of neural network nodes using the input data;
an output buffer for storing and updating the learning result of the operation unit;
a parameter buffer that transmits the parameters between the plurality of neural network nodes to the operation unit and updates the parameters according to a result of the learning; and
a control unit that controls the parameter buffer to remove weights having a size smaller than a threshold among the weights between the neural network nodes, and maps survival weights from among the weights to at least one representative value,
The control unit performs learning on weights between neural network nodes using the input data, removes a weight having a size smaller than a threshold value from among the weights, performs re-learning, and sets the survival weights. mapping to a plurality of representative values,
The control unit generates first survival weights by executing a first iterative loop that performs learning by removing weights smaller than a first threshold from among the weights, and uses a second threshold greater than the first threshold to generate the survival weights by executing a second iterative loop that removes weights smaller than a second threshold among the first survival weights. delete 12. The method of claim 11,
The control unit determines the threshold value for each layer of the neural network. delete 12. The method of claim 11,
The representative value is a neural network system corresponding to a centroid. 12. The method of claim 11,
The control unit stores a centroid index to which the representative value is mapped and a mapping table of the centroid, and exchanges the centroid index with an external memory as the survival weight.

Images