KR102163498B1 - Apparatus and method for pruning-retraining of neural network - Google Patents

Abstract

It relates to a pruning-retraining method of a neural network, and the pruning-retraining method of a neural network includes the steps of: (a) performing pruning on nodes in the neural network; And (b) recovering at least some of the pruned inter-node weights in the pruned neural network, and performing retraining on the recovered inter-node weights.

Description

Neural network pruning-retraining device and method {APPARATUS AND METHOD FOR PRUNING-RETRAINING OF NEURAL NETWORK}

The present application relates to a neural network pruning-retraining apparatus and method. In particular, in retraining the pruned neural network, the present invention does not retrain the weights between all nodes in the neural network through retraining of the entire neural network, but retrains the weights between nodes that partially recover the weights between the pruned nodes. It relates to an apparatus and method for pruning-retraining of a neural network to be trained.

Pruning in a neural network refers to the process of deleting connections between neurons in a neural network. In general, pruning can save a lot of money because it can reduce unnecessary redundant connections in the network.

Pruning determines the importance of the connections of each neuron in the trained neural network, deletes the connections in unimportant order, and recovers the accuracy of the initially trained neural network through a retraining process. Here, the criterion for deleting the connection of neurons varies depending on the method, and generally, a method of making the weight value of a neuron (nerve) 0 is used.

1 and 2 are conventional documents [Han, Song, et al. "DSD: regularizing deep neural networks with dense-sparse-dense training flow." arXiv preprint arXiv:1607.04381 3.6 (ICLR 2017)] is a diagram for explaining a training technique using pruning, DSD (Dense-Sparse-Dense). In particular, if the DSD technique is expressed as an algorithm, it may be as shown in FIG. 2.

1 and 2, the DSD technique disclosed in the conventional literature can be largely divided into an Initial Dense Phase, a Sparse Phase, and a Final Dense Phase.

Briefly, the Initial Dense Phase represents a neural network training process using a normal optimizer. Sparse Phase refers to a process of masking all weights of a value smaller than a threshold to make it zero. The Final Dense Phase is a retraining process for Fine Tuning, and is conducted to increase the accuracy that has fallen from pruning.

In contrast to the original training process, the neural network to which DSD training has been applied can derive a more robust solution to a given problem as insignificant connections in the neural network are deleted. In addition, since the neural network is optimized for the used data set, advantages such as an increase in inference accuracy and prevention of overfitting can be obtained.

However, in DSD training, after pruning is performed on the neural network, all weight values between nodes in the neural network are retrained as the entire pruned neural network is retrained. The neural network to which DSD training is applied has a limitation in reducing the parameter volume (parameter size), which requires a large memory space, a slow inference speed (time), and a high power (POWER) requirement. have.

The present application is to solve the problems of the prior art described above, and a pruning-retraining apparatus and method of a neural network enabling the use of an adaptive neural network in consideration of power, accuracy, speed, memory space, parameter volume, etc. It aims to provide.

The present application is to solve the problems of the prior art described above, and an object of the present invention is to provide an apparatus and method for pruning-retraining a neural network that enables a neural network to be selectively used according to system performance or required conditions.

However, the technical problem to be achieved by the embodiments of the present application is not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the pruning-retraining method of a neural network according to an embodiment of the present application includes the steps of: (a) performing pruning on nodes in the neural network; And (b) recovering at least some of the pruned inter-node weights in the pruned neural network, and performing retraining on the recovered inter-node weights.

In addition, the step (b) may further recover at least some of the weights between nodes among the remaining weights excluding the recovered weights between the nodes among the pruned node weights.

In addition, step (b) may be repeatedly performed.

In addition, the pruning-retraining method of a neural network according to an embodiment of the present application may further include (c) generating a multi-phase neural network in consideration of the restored weights between nodes.

Further, the multi-phase neural network may include a first layer generated to include the pruned inter-node weights in the neural network and a second layer generated to include the recovered inter-node weights.

In addition, the second layer includes a plurality of sub-layers, any one of the plurality of sub-layers is a sub-layer generated to include the recovered weights between at least some of the nodes, and the plurality of sub-layers The remaining sub-layers excluding any one of the sub-layers may be sub-layers that are additionally generated to include the recovered inter-node weights each time the step (b) is repeatedly performed when step (b) is repeatedly performed. have.

Also, the plurality of sub-layers may be generated by applying a sparse matrix format.

In addition, in step (a), the neural network may be a trained convolution neural network.

Meanwhile, the apparatus for pruning-retraining a neural network according to an embodiment of the present disclosure includes: a pruning unit that performs pruning on nodes in a neural network; And a recovery retraining unit recovering at least a part of the pruned inter-node weights in the pruned neural network and performing retraining on the restored inter-node weights.

In addition, the recovery retraining unit may additionally recover at least some of the weights between nodes among remaining node weights excluding the restored at least some of the weights between nodes among the pruned node weights.

In addition, the recovery retraining unit may repeatedly perform a process of recovering weights between nodes and retraining restored weights between nodes.

In addition, the apparatus for pruning-retraining a neural network according to an embodiment of the present application may further include a generator configured to generate a multi-phase neural network in consideration of the restored weights between nodes.

Further, the multi-phase neural network may include a first layer generated to include the pruned inter-node weights in the neural network and a second layer generated to include the recovered inter-node weights.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present application. In addition to the above-described exemplary embodiments, additional embodiments may exist in the drawings and detailed description of the invention.

According to the above-described problem solving means of the present application, by providing a pruning-retraining apparatus and method of a neural network, it is possible to use an adaptive neural network in consideration of power, accuracy, speed, memory space, parameter volume, etc. I can.

According to the above-described problem solving means of the present application, by providing an apparatus and method for pruning-retraining a neural network, it is possible to selectively use a neural network according to system performance or required conditions.

However, the effect obtainable in the present application is not limited to the effects as described above, and other effects may exist.

1 and 2 are conventional documents [Han, Song, et al. "DSD: regularizing deep neural networks with dense-sparse-dense training flow." arXiv preprint arXiv:1607.04381 3.6 (ICLR 2017)] is a diagram for explaining a training technique using pruning, DSD (Dense-Sparse-Dense).
3 is a block diagram showing a schematic configuration of a pruning-retraining apparatus of a neural network according to an embodiment of the present application.
4 is a view for explaining a pruning-retraining process of a neural network by a pruning-retraining apparatus of a neural network according to an embodiment of the present application.
5 is a diagram illustrating a multi-phase neural network generated by a pruning-retraining apparatus of a neural network according to an embodiment of the present application.
6 is a diagram showing an example of a method of storing a compressed sparse column (CSC) matrix among sparse matrix types.
7 is a flowchart illustrating an operation of a method for pruning-retraining a neural network according to an embodiment of the present application.

Hereinafter, exemplary embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present application. However, the present application may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts not related to the description are omitted in order to clearly describe the present application, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the present specification, when a part is said to be "connected" with another part, it is not only "directly connected", but also "electrically connected" or "indirectly connected" with another element interposed therebetween. "Including the case.

Throughout this specification, when a member is positioned "on", "upper", "upper", "under", "lower", and "lower" of another member, this means that a member is located on another member. It includes not only the case where they are in contact but also the case where another member exists between the two members.

Throughout the specification of the present application, when a certain part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

3 is a block diagram showing a schematic configuration of a pruning-retraining apparatus 100 for a neural network according to an embodiment of the present application. 4 is a diagram illustrating a pruning-retraining process of a neural network by the apparatus 100 for pruning-retraining a neural network according to an embodiment of the present application. The circle shape shown in FIG. 4 schematically shows weights between nodes in a neural network as an example.

Hereinafter, the apparatus 100 for pruning-retraining a neural network according to an embodiment of the present application will be referred to as the apparatus 100 for convenience of description.

3 and 4, the apparatus 100 may include a pruning unit 110, a recovery retraining unit 120, and a generation unit 130.

The pruning unit 110 may perform pruning on a node in the neural network 10 (S2). At this time, the apparatus 100 may prepare the neural network 10 for pruning (S1) before performing pruning on the node in the neural network 10 (S2).

Here, the neural network 10 may be a trained neural network. That is, the neural network 10 may be a trained neural network in which training is performed on an initial (original, existing) neural network. The neural network 10 considered herein may be, for example, a neural network model trained on a specific image data set. Accordingly, in the present application, the neural network 10 may be expressed differently as a trained neural network model, a trained model, or the like.

In addition, the neural network 10 considered herein may be a trained convolution neural network (CNN) as an example.

A convolutional neural network (CNN) is one of the types of artificial neural networks (deep learning, neural networks), and is mainly specialized in image-related data input, and shows excellent performance in classification and segmentation processing. A convolutional neural network outputs a feature map through a convolution operation by applying a filter included in a convolutional layer to an input image. Since important feature information of an image is compressed and stored in this feature map, when a convolutional neural network is constructed with a deep neural network (deep neural network), it is effective in processing images and moving pictures with a lot of data to be processed.

In the present application, it is illustrated that a convolutional neural network (CNN) trained as the neural network 10 is considered, but is not limited thereto, and the neural network 10 considered in the present application includes a conventional recurrent neural network (RNN), etc. Various neural networks already known or developed in the future (including trained neural networks, untrained neural networks, etc.) can be applied.

The pruning unit 110 may perform pruning (S2, Initial Pruning) on the neural network 10 prepared in step S1. Accordingly, the pruned neural network 11 may include a weight between nodes 2 pruned by pruning and a weight 3 between nodes that are not pruned.

When the pruning unit 110 performs pruning on the neural network 10, the weights between nodes in which some of the weights between nodes in the neural network 10 (all weights between all nodes in the neural network, 1) are pruned. It can be pruned (removed, pruned) as (2).

The pruning unit 110 may perform pruning on nodes in the neural network 10 so that weights of a preset ratio among the weights 1 between nodes in the neural network 10 are pruned.

Here, the preset ratio may be set by user input as an example.

The preset ratio may be any one of ratios between 70% and 90% of the weights 1 between nodes in the neural network 10. However, the present invention is not limited thereto and may be set in various ratios.

In addition, the pruning unit 110 may perform pruning in consideration of an absolute value of the weight 1 between nodes in the neural network 10. As a specific example, when the pruning unit 110 sequentially lists the absolute values of the weights 1 between nodes in the neural network 10 (for example, in ascending order or descending order), the upper (or lower) Weights between nodes belonging to the ratio can be pruned.

In the present application, the pruning unit 110 is exemplarily illustrated as performing pruning at a preset ratio (%), but is not limited thereto, and pruning may be performed in units such as a preset number (number). .

When pruning is performed on the nodes in the neural network 10 by the pruning unit 110, the recovery retraining unit 120 after that is at least a part of the weights 2 between the pruned nodes in the pruned neural network 11 ( That is, at least some of the weights between nodes) may be restored, and retraining on the recovered at least some of the weights between nodes may be performed (Weight restoring & Retraining).

At this time, the recovery retraining unit 120 is at least a part of the pruned node weights 2 (that is, at least some of the weights between nodes) in consideration of a preset ratio and an absolute value of the weight, as in the previous pruning method. Can recover. For example, when the recovery retraining unit 120 sequentially lists the absolute values of the pruned node weights (2) (eg, in ascending or descending order), a preset ratio from the upper (or lower) Weights between nodes belonging to at least some of the weights between nodes may be recovered.

In addition, the recovery retraining unit 120 may additionally recover at least some of the weights between nodes among remaining node weights excluding at least some of the restored weights between nodes among the pruned node weights.

The recovery retraining unit 120 may repeatedly perform a process of recovering the weights between nodes and retraining the restored weights between nodes. The detailed description is as follows.

After pruning for the nodes in the neural network 10 is performed (S2) by the pruning unit 110, the recovery retraining unit 120 is among the weights 2 among pruned nodes in the pruned neural network 11 At least a portion (ie, at least a portion of the weights between nodes 4) may be recovered, and retraining on the recovered at least some of the weights between nodes 4 may be performed (S3). In this case, a process of recovering at least some of the weights between nodes among the pruned node weights and retraining the restored at least some of the weights between nodes may be referred to as a recovery retraining process. According to this, step S3 may be referred to differently as a primary recovery retraining process.

Thereafter, when the first recovery retraining process (S3) is performed and then the recovery retraining process is repeatedly performed (ie, the recovery retraining process is repeatedly performed once based on the S3 process), the pruning unit 110 Add at least some of the weights (5) among the weights (2') of the remaining nodes except for the weights (4) of the nodes recovered in the first recovery retraining process from the weights (2) of the pruned nodes. It can be recovered by, and then, retraining for at least some of the weights 5 between nodes that have been additionally recovered can be performed (S4). This step S4 may be referred to differently as the second recovery retraining process (S4) as a result of one repetition (S4) of the first recovery retraining process (S3).

Thereafter, when the second recovery retraining process (S4) is performed and then the recovery retraining process is repeatedly performed (ie, the recovery retraining process is repeatedly performed twice based on the S3 process), the pruning unit 110 The rest of the pruned inter-node weights (2), excluding at least some inter-node weights (4) recovered in the first recovery retraining process and at least some inter-node weights (5) recovered during the second recovery retraining process. It is possible to additionally recover at least some of the inter-node weights (6) among the inter-node weights (2''), and then perform retraining (S5) on at least some of the additionally recovered inter-node weights (6). I can. This step S5 may be referred to differently as a third recovery retraining process (S5) as a result of performing two iterations (S4, S5) of the first recovery retraining process (S2).

As such, the pruning unit 110 may repeatedly perform the recovery retraining process. At this time, each time the recovery retraining process is repeatedly performed, the pruning unit 110 selects the weights between nodes recovered from the previous recovery retraining process (at least one or more previous recovery retraining processes) among the weights 2 among the pruned nodes. At least some of the weights between nodes of the remaining nodes are additionally recovered, and retraining can be performed only on the weights between nodes that have been additionally recovered.

According to this, in conventional DSD training, pruning is performed on a neural network, and then all weight values between nodes in the neural network are retrained by retraining the entire pruned neural network. In other words, in the conventional DSD training, in the process of performing retraining after restoring, retraining was performed on the weights between all nodes in the neural network. On the other hand, according to the recovery retraining unit 120 of the present apparatus 100, the present application is used when performing Pruning-Restoring training (in other words, in retraining the pruned neural network). As described above, instead of retraining the weights between all nodes (that is, all weights between nodes) in the pruned neural network, retraining may be performed only on the weights between nodes that have partially recovered among the weights between the pruned nodes.

The generation unit 130 may generate a multi-phase network 20 in consideration of the recovered weights between nodes retrained by the pruning unit 120. A description of the multi-phase neural network 20 may be more easily understood with reference to FIG. 5.

5 is a diagram illustrating a multi-phase neural network 20 generated by the pruning-retraining apparatus 100 of a neural network according to an embodiment of the present application.

Referring to FIG. 5, the multi-phase neural network 20 includes a first layer 21 generated to include the weights 2 between nodes pruned in the neural network 10 and the recovery material by the recovery retraining unit 120. It may include a second layer 22 generated to include the weights between nodes recovered through the training process.

Here, the first layer 21 may mean a layer generated to include the pruned inter-node weight 2 in the neural network 10 as 0. In other words, when pruning for the neural network 10 is performed (S2), the first layer 21 is generated to include the weights 3 between nodes that are not pruned in the pruned neural network 11 It can mean a layer. That is, the first layer 21 may mean a layer generated to include the weights 3 between nodes that have not been pruned corresponding to step S2 in FIG. 4 by way of example.

The second layer 22 may include a plurality of sub-layers 22a, 22b, 22c, ...).

Any one of the plurality of sub-layers 22a, 22b, 22c, ...) is at least recovered retrained in the primary recovery retraining process (ie, step S3) by the recovery retraining unit 120 It may be a layer (sub-layer) generated to include some of the weights 4 between nodes.

Here, one of the plurality of sub-layers 22a, 22b, 22c, ...) may be expressed differently as a first sub-layer 22a. One of the sub-layers 22a is a layer that is generated to include at least some of the recovered weights 4 between nodes that are retrained in the first recovery retraining process in response to step S3 in FIG. 4 ( It may mean a sub-layer.

In addition, among the plurality of sub-layers 22a, 22b, 22c, …), the remaining sub-layers 22b, 22c,… excluding any one of the sub-layers 22a are a recovery retraining process by the recovery retraining unit 120 When this is repeatedly performed, it may be a sub-layer that is additionally generated to include the recovered inter-node weights each time the recovery retraining process is repeatedly performed.

The remaining sub-layers 22b, 22c, ... may include a second sub-layer 22b, a third sub-layer 22c, and the like.

At this time, the second sub-layer 22b exemplarily corresponds to step S4 in FIG. 4, in the second recovery retraining process (that is, when performing one repetition of the first recovery retraining process) It may mean a layer (sub-layer) generated to include at least some of the weights 5 between nodes. In addition, the third sub-layer 22c exemplarily corresponds to step S5 in FIG. 4, in the third recovery retraining process (that is, when performing two repetitions of the first recovery retraining process) It may mean a layer (sub-layer) generated to include at least some of the weights 6 between nodes.

As such, the sub-layers included in the second layer 22 are additionally recovered and retrained during each repetition of the recovery retraining process by the pruning unit 110 (in other words, It may be additionally generated to include the retrained and additionally recovered inter-node weights).

In other words, the number of sub-layers included in the second layer 22 is increased to a number proportional to the number of repetitions of the recovery retraining process each time the recovery retraining process is repeated after the first sub-layer 22a is created. I can. That is, the sub-layer included in the second layer 22 may be additionally generated in proportion to the number of repetitions of the recovery retraining process. In other words, the remaining sub-layers 22b, 22c,… except for the first sub-layer 22a among the second layers 22 may be generated in a number proportional to the number of repetitions of the recovery retraining process.

In this case, the plurality of sub-layers 22a, 22b, 22c, ... included in the second layer 22 may be generated by applying a sparse matrix format.

The sparse matrix refers to a matrix whose internal values are mostly 0. Examples of the sparse matrix type (type) include Compressed Sparse Row (CSR), Compressed Sparse Column (CSC), and Coordinate List (COO). For example, FIG. 6 shows an example of a method of storing a compressed sparse column (CSC) matrix among sparse matrix types.

Since the sparse matrix stores only non-zero values in the matrix through a kind of processing, it has the advantage that the required storage space (required storage space) is smaller than the dense matrix, and the operation speed between matrices is fast. .

Therefore, when generating the multi-phase neural network 20, the generation unit 130 may generate the plurality of sub-layers 22a, 22b, 22c,… included in the second layer 22 by applying a sparse matrix format. I can.

The generation unit 130 may generate a multi-phase neural network in a plurality of types. Here, the plurality of types of the multi-phase neural network generated by the generation unit 130 may be determined according to the number of layers included in the generated multi-phase neural network.

Referring to FIG. 5, for example, four types may be included in a plurality of types of a multi-phase neural network generated by the generation unit 130.

Here, the first type of multi-phase neural network may mean a neural network including the first layer 21. The second type of multi-phase neural network may mean a neural network including a first layer 21 and a first sub-layer 22a in the second layer 22. The third type of multi-phase neural network may mean a neural network including a first sub-layer 22a and a second sub-layer 22b in the first layer 21 and the second layer 22. The fourth type of multi-phase neural network is a neural network including a first sub-layer 22a, a second sub-layer 22b, and a third sub-layer 22c in the first layer 21 and the second layer 22. It can mean.

As such, the generation unit 130 may generate a plurality of types of multi-phase neural networks according to the number of sub-layers included in the second layer 22.

Further, although not shown in the drawing, the apparatus 100 may include a multi-phase neural network providing unit (not shown).

The multi-phase neural network providing unit (not shown) may provide the multi-phase neural network generated by the generation unit 130. In particular, the multi-phase neural network provider (not shown) may provide any one of a plurality of types of multi-phase neural networks. In this case, the multi-phase neural network provider (not shown) may selectively provide any one of a plurality of types of multi-phase neural networks in consideration of the input system performance and/or a required condition (required condition).

Here, the type of system performance and/or requirements may include, but is not limited to, a memory space, an inference speed, an accuracy, and a power (power). As another example, in particular, the type of performance of the system may include a battery embedded system type, a connected embedded system type, a server system type, and the like.

In other words, when generating the multi-phase neural network 20, the generation unit 130 fixes the value of the weight 3 between nodes that are not pruned in the pruned neural network 11, and then the recovery retraining process Multiple types of multi-phase neural networks can be generated by increasing the weight of the recovered nodes retrained through execution in proportion to the number of times the recovery retraining process is performed step by step.

As an example, the weight remaining after the initial pruning (S2) for the neural network 10 is performed is referred to as A, and the weights between nodes recovered and retrained in stages are B, C, and respectively, each time the recovery retraining process is performed. Suppose it is D. In this case, the generation unit 130 includes a neural network including A, a neural network including A+B, a neural network including A+B+C, and A+B+C+D as multiple types of multi-phase neural networks. You can create neural networks.

Here, as described above, A denotes the weight (3) between unpruned nodes remaining in the neural network pruned in step S2, and B denotes the weight (4) between nodes recovered and retrained in step S3. Here, C may mean a weight 5 between nodes recovered and retrained in step S4, and D may mean a weight 6 between nodes recovered and retrained in step S5.

Therefore, a neural network including A is a first type of multi-phase neural network, a neural network including A+B is a second type of multi-phase neural network, and a neural network including A+B+C is a third type of multi-phase neural network, The neural network including A+B+C+D may mean a fourth type of multi-phase neural network.

Thereafter, the multi-phase neural network providing unit (not shown) provides multiple types of multi-phase neural networks (e.g., first to fourth types of multi-phase neural networks) according to the performance and/or requirements (required conditions) of the input system. Neural network) can be optionally provided.

At this time, referring to FIG. 5, as the number of layers in the generated multi-phase neural network increases (in other words, as the number of retrained recovered inter-node weights in the generated multi-phase neural network increases, or the first type of multi-phase As we go from neural networks to multi-phase neural networks of the 4th type), neural networks can become more dense. In this way, as the neural network becomes denser, the required memory space becomes larger (Large), the accuracy (accuracy) and power (power) consumption become higher (high), while the inference speed (especially forward inference Speed) can be slow. That is, there may be a trade-off relationship between performance-accuracy, and the device 100 can trade off power, speed, and accuracy in the neural network 10 (for example, a convolutional neural network). off), pruning-recovery training (retraining) techniques can be provided.

As the ratio of the weights between nodes pruned in the neural network 10 increases, the number of weights having a value of 0 increases in the neural network 10, which can be expressed as an increase in the sparsity rate of the neural network 10. Therefore, in order to solve the problem that the inference speed decreases and the required memory space increases as the neural network becomes denser, the generation unit 130 in the apparatus 100 generates the multi-phase neural network 20 when generating the multi-phase neural network 20. Layers included within (especially, a plurality of sub-layers included in the second layer) may be generated by applying a sparse matrix format.

That is, as the sparse matrix format is applied to the multi-phase neural network 20 generated by the present apparatus 100, the inference speed (inference time, forward inference speed) is faster compared to the prior art, and the memory space can be saved (can be reduced). ) have. In other words, the apparatus 100 configures filters of layers (especially, a plurality of sub-layers included in the second layer) in the generated multi-phase neural network 20 by applying a sparse matrix format. In the case of a neural network to which 20) is applied, due to the characteristics of a sparse matrix, the forward inference speed (time) is faster and memory space can be saved compared to a neural network to which the basic matrix format is applied. Here, the intra-layer filter may mean a set of parameters existing in the layer.

The present apparatus 100 can perform pruning-recovery training in consideration of trade-offs in power, speed, and accuracy in the neural network 10 (for example, a convolutional neural network). In other words, the present application may provide a pruning-recovery training technique in consideration of the accuracy-speed/power tradeoff. This technique proposed herein may be referred to differently as a multi-phase pruning technique.

Meanwhile, a multi-phase neural network provider (not shown) provides multiple types of multi-phase neural networks (e.g., first to fourth types of multi-phase neural networks) according to the input system performance and/or requirements (required conditions). In relation to selectively providing any one of neural networks), for example, it is as follows.

As an example, the multi-phase neural network provider (not shown) is a first type of multi-phase neural network when a condition in which the accuracy is any one of 0% to 24%, for example, is input as a performance and/or requirement condition of the input system. You can choose to provide it. In addition, the multi-phase neural network provider (not shown) provides a second type of multi-phase neural network when a condition in which the accuracy is one of 25% to 49%, for example, is input as a performance and/or requirement condition of the input system. You can choose to provide it. In addition, the multi-phase neural network providing unit (not shown) provides a third type of multi-phase neural network when a condition in which accuracy is one of 50% to 74%, for example, is input as a performance and/or requirement condition of the input system. You can choose to provide it. In addition, the multi-phase neural network provider (not shown) provides a fourth type of multi-phase neural network when a condition in which accuracy is 75% to 100%, for example, is input as a performance and/or requirement condition of the input system. You can choose to provide it.

That is, in the neural network 10 that has undergone a pruning process for a model trained on a specific image data set, the apparatus 100 is a weight between nodes of a preset ratio among the pruned weights 2 Can recover. Thereafter, the apparatus 100 may perform a retraining process only for the restored weights between nodes. The apparatus 100 may recover the accuracy of the neural network degraded by pruning by performing retraining on the restored weights between nodes.

In addition, in the present apparatus 100, when pruning is performed on the neural network 10, the value of the unpruned weight (ie, live weight, 3) does not change during the recovery retraining process. It is possible to construct a neural network such that each separately added recovered weight is added (for each iteration).

In other words, in the present apparatus 100, when pruning is performed, the value of the unpruned weight (i.e., live weight, 3) is fixed, and each time the recovery retraining process is performed (every repetition) is added in the corresponding process. A plurality of types of neural networks (multi-phase neural networks) can be created by separately (stepwise) adding layers containing recovered weights (retrained restored weights).

In other words, in order to generate the multi-phase neural network 20, the apparatus 100 performs the recovery retraining process multiple times (repeatedly), and the weights recovered in each process are multi-stepped as shown in FIG. It can be created as a neural network of (i.e., multiple types of multi-phase neural networks). At this time, each time the restored weight is added in each process (every time a layer containing the restored weight is added) (that is, as the number of layers in the multi-phase neural network increases), the sparse ratio decreases, so a large memory in the operation Space is required, and forward inference speed (time) takes a long time. Therefore, the present apparatus 100 constructs (generates) a multi-phase neural network into a plurality of types as shown in FIG. can do. Accordingly, the device 100 adapts multiple types of multi-phase neural networks generated by the device 100 to meet the performance and/or requirements of the system (for example, requirements such as memory or speed). It can be provided to be used adaptively.

The device 100 constructs (generates) a plurality of types of multi-phase neural networks through repeated recovery retraining processes, and based on this, a neural network customized to the system performance and/or requirements such as power and accuracy is considered. Can be provided by adaptively selecting.

Hereinafter, based on the details described above, the operation flow of the present application will be briefly described.

7 is a flowchart illustrating an operation of a method for pruning-retraining a neural network according to an embodiment of the present application.

The pruning-retraining method of the neural network shown in FIG. 7 may be performed by the apparatus 100 described above. Therefore, even if omitted below, the description of the apparatus 100 may be equally applied to the description of the pruning-retraining method of a neural network.

Referring to FIG. 7, in step S110, pruning may be performed on a node in the neural network.

At this time, the neural network considered in step S110 may be a trained convolution neural network, for example.

Next, in step S120, at least some of the pruned inter-node weights in the neural network pruned in step S110 may be recovered, and retraining may be performed on the recovered inter-node weights.

In this case, in step S120, at least some of the weights between nodes among the remaining node weights excluding at least some of the restored weights between nodes among the pruned node weights may be additionally restored.

This step S120 may be repeatedly performed.

In addition, although not shown in the drawing, the pruning-retraining method of a neural network according to an embodiment of the present application includes, after step S12, generating a multi-phase neural network in consideration of the weights between the recovered nodes. I can.

In this case, the multi-phase neural network is a first layer created to include the weights between nodes pruned in the neural network and the weights between nodes recovered in step S120 (weights between nodes recovered and retrained in step S120, differently expressed and retrained) It may include a second layer generated to include the restored weight between nodes.

Also, the second layer may include a plurality of sub-layers. Here, any one of the plurality of sub-layers may be a sub-layer generated to include the recovered at least some inter-node weights. In addition, the remaining sub-layers excluding any one of the plurality of sub-layers may be sub-layers that are additionally generated to include the recovered inter-node weights each time step S120 is repeatedly performed when step S120 is repeatedly performed. have.

Also, a plurality of sub-layers in the second layer included in the multi-phase neural network may be generated by applying a sparse matrix format.

In the above description, steps S110 and S120 may be further divided into additional steps or may be combined into fewer steps, according to an exemplary embodiment of the present disclosure. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

The pruning-retraining method of a neural network according to an embodiment of the present disclosure may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The above-described hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

In addition, the aforementioned method of pruning-retraining a neural network may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The foregoing description of the present application is for illustrative purposes only, and those of ordinary skill in the art to which the present application pertains will be able to understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present application is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present application.

100: neural network pruning-retraining device
110: pruning part
120: recovery retraining department
130: generation unit

Claims (13)

As a pruning-retraining method of a neural network in which each step is performed by a computer-implemented pruning-retraining device of a neural network,
(a) performing pruning for nodes in the neural network; And
(b) recovering at least some of the pruned inter-node weights in the pruned neural network and performing retraining on the restored inter-node weights; And
(c) generating a multi-phase neural network in consideration of the weights between the retrained and recovered nodes,
Including,
The multi-phase neural network includes a first layer generated to include the pruned inter-node weights in the neural network and a second layer generated to include the recovered inter-node weights. Training method. The method of claim 1,
The step (b),
The pruning-retraining method of a neural network, wherein at least some of the weights between nodes among the remaining node weights excluding the recovered at least some of the weights between nodes of the pruned nodes are additionally restored. The method of claim 2,
The step (b) is repeatedly performed, pruning of a neural network-retraining method. delete delete The method of claim 1,
The second layer includes a plurality of sub-layers,
Any one sub-layer among the plurality of sub-layers is a sub-layer generated to include at least some of the restored weights between nodes,
When the step (b) is repeatedly performed, the remaining sub-layers among the plurality of sub-layers are additionally generated to include the restored weights between nodes each time the step (b) is repeatedly performed. The pruning-retraining method of a neural network that is a sub-layer. The method of claim 6,
The plurality of sub-layers is generated by applying a sparse matrix format, pruning-retraining method of a neural network. The method of claim 1,
In the step (a), the neural network is a trained convolution neural network, pruning-retraining a neural network. As a neural network pruning-retraining device,
A pruning unit that performs pruning on nodes in the neural network;
A recovery retraining unit for recovering at least some of the pruned inter-node weights in the pruned neural network and performing retraining on the recovered inter-node weights; And
A generator for generating a multi-phase neural network in consideration of the retrained and restored weights between nodes,
Including,
The multi-phase neural network includes a first layer generated to include the pruned inter-node weights in the neural network and a second layer generated to include the recovered inter-node weights. Training device. The method of claim 9,
The recovery retraining unit,
The pruning-retraining apparatus of a neural network to further recover at least some of the weights between nodes of the remaining node weights excluding the recovered at least some of the weights between nodes among the pruned node weights. The method of claim 10,
The recovery retraining unit is to repeatedly perform a process of recovering weights between nodes and retraining restored weights between nodes, pruning-retraining apparatus of a neural network. delete delete

Images