TWI823488B - Method for implementing edge-optimized incremental learning for deep neural network and computer system - Google Patents

Abstract

A method for implementing edge-optimized incremental learning for deep neural network and a computer system are provided. The method is performed in a computer system that operates a deep neural network. In the method, a deep neural network model that connects with fully-connected layers and an output layer is prepared. In the beginning, the connections between the fully-connected layers and the deep neural network model are disconnected for a purpose of making the fully-connected layers not to participate in an incremental learning process and updating training parameters. Next, the fully-connected layers are cloned as new fully-connected layers, in which one of the new fully-connected layers is connected with a new output layer. The fully-connected layers and the output layer are connected with the deep neural network model. In the incremental learning process, through the deep neural network model, the newly-added incremental dataset are trained for updating the model.

Description

Edge optimization incremental learning method and computer system for deep neural networks

The description discloses an incremental learning technology applied to deep neural networks, specifically an edge-optimized incremental learning method and computer system for deep neural networks that uses a shared learning mechanism to reduce model training complexity and resource requirements.

Deep convolutional neural networks have been widely and successfully used in many application fields, especially in the field of computer vision. Deep neural networks (DNN) have automatically achieved high-accuracy face recognition and object detection and classification. .

There are three main steps to deploying a deep neural network model, first developing the deep neural network model, then training the developed model given a data set, and deploying the trained deep neural network model for a specific application. Deep neural network models face the challenge of high computational load in the training phase because a large number of deep neural network model parameters require a long training time. Therefore, in most cases, the training of deep neural network models is performed on the graphics processing unit. (GPU) and Tensor Processing Unit (TPU).

For deployed and trained deep neural network models, problems arise when the data changes and the model needs to be retrained, which is a major issue for large neural networks because they require a long training time. This problem is solved by using transfer learning technology. In the described transfer learning, the trained neural network is used as a fixed feature extractor model and the fully connected layer or the fully connected layer and a few convolutional layers at the end of the deep neural network model are retrained for the new data set. , and these training processes will reduce the training time of the deep neural network model several times. However, when such models are deployed in an application, their output categories cannot be changed, rendering these models unable to learn about new data. And because the categories are fixed, in many applications, data arrives in increments over time. .

In a specific application, Xi Zhi proposed a sorting robot that uses computer vision technology to sort different objects. The sorting robot is initially trained to sort a fixed number of types of objects. If the user wants the robot to When more categories can be added to the sorting category, the entire model needs to be retrained using the old data set (old category) and the new data set (to be added), which leads to the problem that the deep neural network model needs to have incremental learning capabilities.

Incremental learning algorithms should have three properties to achieve high performance. First, when new data arrives at any time, the incremental learning algorithm should be able to train the model at any time without starting from scratch, but after it has been trained. second, a trained model can classify categories with high accuracy; third, when new categories are added to the model, the memory footprint and computational requirements should be low or grow slowly.

Incremental learning technology aims to improve the ability of deep neural network (DNN) models to add new categories in pre-trained models. The model can incrementally learn new categories through availability. Incremental learning is different from conventional deep neural network model training. , For the training of conventional deep neural network models, the entire set of training data must be fully involved in the training of the deep neural network model. Figure 1 shows a schematic diagram of the operation of the current incremental learning model, which shows a trained incremental learning model 100. In the incremental learning of the incremental learning model 100, the data used for model training are at different times. Participate in training. As shown in the figure, data of different new categories (classes) are introduced at different times, schematically showing that there are category one 11, category two 12 and category three 13.

However, despite good progress in the field of computer imaging in recent years, the deep neural network algorithms have limitations in the incremental learning process. Most deep neural network algorithms can perform training and inference on a fixed number of categories. , and all training data are required to be available when executing the training phase of the deep neural network model. If only new data is used to train the deep neural network model, this will lead to a decrease in classification accuracy, which is called catastrophic forgetting.

Existing incremental learning techniques attempt to reduce the impact of catastrophic forgetting by using pre-trained data while adding new types of data to the model or designing complex model architectures, which results in high design complexity and memory storage space requirements, making Incremental learning cannot be implemented on edge devices with limited memory storage space and computing resources.

Therefore, in view of the fact that deep neural networks consume a large amount of memory space and computing resources during the incremental learning process, it is necessary to propose new learning technologies that can implement incremental learning on devices with limited storage space and computing resources. Quantitative learning method.

In order to implement incremental learning methods on devices with limited storage space and computing resources, the disclosure proposes an edge-optimized incremental learning method for deep neural networks, and this method can be run on a computer system.

In the proposed computer system for running a deep neural network, the computer system includes a processor and a memory with specific computing power and storage resources. The edge optimization incremental learning method process executed includes preparing a deep neural network model. , including one or more fully connected layers and an output layer, first disconnect the original connection between the one or more fully connected layers and the deep neural network model, so that the one or more fully connected layers do not participate in incremental learning training parameter updates , and then copy one or more fully connected layers into one or more new fully connected layers, and connect one of the new fully connected layers to a new output layer. Then connect one or more new fully connected layers and new output layers to the deep neural network model. In an incremental learning training process, the deep neural network model participates in training with the newly added incremental data set, and Update deep neural network models.

Further, the size of the new output layer is equal to the number of categories to be incrementally learned in a given time.

Moreover, during the incremental learning and training process, only the newly added incremental data sets participate in the training, while the data sets that have been trained by the original deep neural network model do not participate in the new training. For different applications and different quantities, categories can be gradually added to the deep neural network model during each incremental learning training process.

Further, after incremental learning is completed and the deep neural network model is updated, in a model inference stage, one or more fully connected layers that were previously disconnected from the deep neural network model are reconnected to the updated deep neural network. model, so that one or more fully connected layers of the original learning type and one or more new fully connected layers of the incremental learning type share the updated deep neural network model.

In order to further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and illustration and are not used to limit the present invention.

The following is a specific example to illustrate the implementation of the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the concept of the present invention. In addition, the drawings of the present invention are only simple schematic illustrations and are not depictions based on actual dimensions, as is stated in advance. The following embodiments will further describe the relevant technical content of the present invention in detail, but the disclosed content is not intended to limit the scope of the present invention.

It should be understood that although terms such as “first”, “second” and “third” may be used herein to describe various components or signals, these components or signals should not be limited by these terms. These terms are mainly used to distinguish one component from another component, or one signal from another signal. In addition, the term "or" used in this article shall include any one or combination of more of the associated listed items depending on the actual situation.

Because incremental learning methods require a large amount of computing power during the training phase, it is difficult to train such algorithms on edge devices due to limitations in computing power, memory storage space, and energy consumption. The disclosure proposes a method for deep neural The edge of the network optimizes the incremental learning method and computer system, reducing the complexity of the overall incremental learning algorithm through a simplified training algorithm, so that the incremental learning algorithm can perform learning in some parameters of any given deep neural network model . In order to train some parameters of the deep neural network model, an incremental learning algorithm is designed using a transfer learning concept.

Transfer learning is a method that only retrains the fully connected layer or a few two-dimensional convolutional layers and fully connected layers of the model. Compared with retraining the entire deep neural network model, transfer learning makes the model require less memory during training. body storage space. The concept of transfer learning can be referred to the schematic diagram shown in Figure 2.

The source dataset 21 shown in Figure 2 is existing knowledge or a learned dataset, which forms a neural network model through the neural network learning algorithm 210. Then a target data set (target dataset) to be learned and trained 22 is proposed. When performing the new neural network learning algorithm 220, transfer learning can be used to transfer the trained neural network model parameters to the new neural network. The learning algorithm 220 makes it possible to obtain a new neural network model without training a new model from scratch.

With reference to the concept of transfer learning, the edge optimization incremental learning method for deep neural networks proposed in the disclosure is based on the concept of cloning and sharing to implement incremental learning of learning with sharing (LwS). Method, the cloning refers to copying the fully connected layer in the deep neural network model that has been trained on some data, so that when the application performs the fully connected layer in the edge optimization incremental learning method for deep neural network ( The cloning and sharing of fully-connected layers can gradually add new categories to the neural network model while retaining the knowledge of the original categories. There is no need to store data from previously pre-trained categories, and the knowledge only focuses on implementation. Compared with high-accuracy incremental learning methods, the proposed edge-optimized incremental learning method can achieve high-speed model training and inference and low energy consumption, and can maintain traditional algorithms or higher classification accuracy.

According to embodiments of the edge-optimized incremental learning method, in deep neural networks, classifiers and feature extraction architectures can be used to implement multiple different types of deep neural network models, and refer to the above-mentioned transfer learning of shared convolutional layers and only training all The connection layer method is used to cope with the need for repeated training as new types of data increase over time. Furthermore, one of the main features of the edge optimization incremental learning method for deep neural networks is to maintain the classification ability of the original categories without the need for previous training data. When the incremental learning algorithm uses When the deep neural network model learns new categories, it does not use the old data that the deep neural network model has been trained on.

The advantage of transfer learning applied by the edge optimization incremental learning method is that when training a new kind of data set (such as the target data set 22 shown in Figure 2), only the fully connected layer or part of the deep neural network model needs to be retrained, and the edge The optimized incremental learning method is based on the incremental learning training method of shared attributes in transfer learning to implement the shared learning method. The architecture can be referred to the embodiment of the neural network architecture of the edge-optimized incremental learning method in the training phase shown in Figure 3. Schematic diagram.

According to the schematic diagram of an embodiment of the neural network architecture shown in Figure 3, please refer to the flow chart of the training phase in the edge optimization incremental learning method shown in Figure 4. Prepare the basic deep neural network model 30, that is, the feature extraction layer, which includes one or more fully connected layers, schematically represented in the figure as fully connected layers FC1 and FC2, and the output layer 301. Initially, the deep neural network All layers of model 30 are frozen (step S401), especially one or more fully connected layers connected to the model, so that they do not participate in incremental learning training parameter updates. The legend shows that the fully connected layers FC1, FC2 and depth are disconnected. The original connection of the neural network model 30 (step S403).

Next, one or more fully connected layers of the original deep neural network model 30 are copied to form one or more new fully connected layers. In the figure, the fully connected layers FC1 and FC2 are copied into new fully connected layers FC1' and FC2. ' (step S405), one of them is connected to the new output layer. This example shows that the copied new fully connected layer FC2' is connected to the new output layer 303. The size of the new output layer 303 is equal to the number of categories to be incrementally learned within a given time. For example, adding two categories will result in a new fully connected layer FC2' connected to a new output layer 303 of both nodes.

Afterwards, the copied new fully connected layers FC1' and FC2', together with the newly inserted new output layer 303, are connected to the frozen layer (ie, feature extraction layer) of the deep neural network model 30 (step S407). It is worth mentioning that in the above steps, the original fully connected layers FC1 and FC2 are copied instead of instantiating the fully connected layers with random values. This is because of the type of incremental addition and the basic model (i.e., deep neural network model 30) types have similar characteristics. Compared with using randomly started fully connected layers for training, replicating the fully connected layers helps the deep neural network model 30 quickly achieve high classification accuracy.

Then, the copied new fully connected layers FC1' and FC2' and the new output layer 303 perform incremental learning training (step S409), and continuously update the deep neural network model 30 to achieve a deep neural network model 30 adapted to new types. (Step S411). During the incremental learning training process, only new incremental data sets to be added participate in the training, and the data sets that have been trained by the original deep neural network model 30 do not participate in new training. It is worth mentioning that for different applications, different numbers of categories can be gradually added to the deep neural network model 30 during each incremental learning training process, and when adding new categories to the deep neural network model 30, The fully connected layers FC1 and FC2 are copied from the initial deep neural network model 30, and a variable number of types can be added to the deep neural network model 30 in increments.

Figure 5 then shows a schematic diagram of an embodiment of the neural network architecture of the edge optimization incremental learning method in the inference stage, while referring to the flow chart of an embodiment of the inference stage of the edge optimization incremental learning method shown in Figure 6 .

After the above training phase is completed, the deep neural network model 50 is updated through incremental learning, and then in the model inference phase, one or more fully connected layers that were previously disconnected from the deep neural network model, as shown in the above embodiment The fully connected layers FC1 and FC2 will be connected to the updated deep neural network model 50 again (step S601), so that the original learning type fully connected layers FC1 and FC2 and the incremental learning type new fully connected layers FC1' and FC2 'The updated deep neural network model 50 can be shared. The updated deep neural network model 50 obtained in the incremental learning process will be able to perform classification of the originally learned categories and the newly added categories (step S603).

Since the deep neural network model in the neural network is a feature extraction layer, it will be frozen during the incremental learning process and will not participate in the update of the incremental learning training parameters. Therefore, after being disconnected during the incremental learning training, even after experiencing Through incremental learning, the finally updated deep neural network model 50 can still maintain the ability to classify the originally learned categories. If you want to add more categories later, you can still use the same incremental learning method mentioned above. After the incremental learning training phase is completed, the same model inference method is also used. Further, as more types are added to the deep neural network model, through the edge optimization incremental learning method for deep neural networks proposed in the disclosure, the above process is repeated to continuously update the deep neural network model, so that It adapts to new and old types of applications.

The edge-optimized incremental learning method for deep neural networks is executed in a computer system. Refer to Figure 7 , which shows a computer system 70 running a deep neural network. Preferably, it can be a computer system with low computing power and Edge devices with small memory storage space include processors 71 and memory 72 that can use specific computing power and storage resources. For example, the Nvidia Jetson TX-2 motherboard has an ARM A57 processor, 4GB RAM, and 32GB Storage space, as can be seen from the above embodiment process, the concept of transfer learning can simplify the training process of the deep neural network model 700 in the computer system, and reduce the computing power and memory requirements of the incremental learning method in the training phase, update The resulting deep neural network model 700 can adapt to both old type data 701 and new type data 702 at the same time.

After completing incremental learning and updating the deep neural network model, the final results can be successfully verified on some famous deep neural network models, such as MobileNetv3, FBNet, MNasNet, etc., and also with well-known data sets (such as Cifar-100 , Caltech-101), compared with other state-of-the-art incremental learning methods, such as LwS, LwM, PNS, LwF and ICaRL, etc., all have been verified.

Table 1 shows the accuracy degradation rate (%) at each incremental learning step of the deep neural network model implemented by the edge-optimized incremental learning method for deep neural networks, on the Cifar-100 dataset, respectively. The incremental category size of each batch (B) is set to 5, 10, 20 and 50 categories, and the accuracy analysis and comparison of incremental learning is performed in 4 different scenarios. It shows the usability of shared learning (LwS) applied by the method proposed in the disclosure on different deep neural network models compared with other incremental learning methods on the Caltech-101 data set with accuracy of 10 categories per increment. .

Table I: method B=5 B=10 B=20 B=50 LwM[14] - 6.84 8.5 9.8 LwF[12] 3.9 7 12.25 20.8 iCaRL[10] 3.33 5.43 10.15 14.5 DMC[25] 3.6 5.25 11.25 15 EWC[30] 4.1 7.26 14.12 twenty three MAS[31] 4.0 7.77 14 twenty two PODNET[26] 2.65 4.67 8.12 16.9 LwS(MNV3L) 3.4 5.8 8.72 19.72 LwS(FBNet) 3.1 5.3 9.7 16.3 LwS(MNasNet) 3.2 5.3 9.21 17.03

In terms of model storage space, the shared learning (LwS) technology used in the edge optimization incremental learning method for deep neural networks proposed in the disclosure is reduced compared with the state-of-the-art technologies DMC and LwM, because the incremental The training methods are different. LwM requires 2 times the storage space for model training during the training process; DMC requires the entire model to be retrained during training, which also requires 2 times the storage space. Table 2 shows the comparison of different deep neural network models with incremental training of model parameters.

Table II: Increased number of types ResNet34 [13] MNV3-L/ MNasNet FBNet The difference between ResNet34[13] and MNV3-L/MNasNet(%) The difference between ResNet34[13] and FBNet(%) 10 8518998 789770 1150218 90.7 86.49 20 8524128 1579540 2300436 81.5 73.0 30 8529258 2369310 3450654 72.2 59.4

Table 3 shows a comparison of the minimum memory space requirements required by other advanced technologies during the training process, including which layers in the model need to participate in incremental learning training. Which assumes the size of each parameter is a 32-bit floating point number.

Table 3: Minimum memory requirements for training compared to base deep neural network models Which layers need to participate in incremental training? A: All models B: Fully connected layers and convolutional layers C: Only fully connected layers LwM[14], LwF[12], DMC[25], LSIL[22], GEM[27], PODNET[26], DCM[28], PNN[24], iCaRL[10] 2x A ST[29] 2x B PNS[13] 1.3-1.6x B PackNet[43] 1.05-1.6x A, B (depending on the task) MAS[31], EWC[30], DAN[21], FTDNN[32] 1.05-1.6x B, C (depending on the task) wxya 1.05-1.1x C

Table 4 shows that as 2, 5, and 10 categories are added in each increment for incremental learning, the shared learning (LwS) technology used in the edge optimization incremental learning method of deep neural networks and other advanced technologies are targeted at training. Comparative analysis of time (training time) and energy consumption (power consumption). The training time and energy consumption of all methods are standardized by shared learning (LwS).

Table 4: B=2 B=5 B=10 method training time Energy consumption training time Energy consumption training time Energy consumption PNS[13] 1.4x 1.6x 1.2x 1.4x 1.2x 1.5x iCaRL[10] 3.1x 2.4x 5.6x 4.4x 10.8x 9.0x EWC[30] 24.4x 29.0x 24.1x 29.0x 22.8x 28.9x PNDNET[26] 6.4x 7.3x 4.2x 6.6x 4.3x 7.1x LwF[12] 1.3x 1.2x 1.2x 1.2x 1.3x 1.3x wxya 1x 1x 1x 1x 1x 1x

In summary, according to the implementation of the edge-optimized incremental learning method and computer system for deep neural networks proposed in the disclosure, a shared learning method is adopted when using deep neural network models to implement new incremental learning technologies. (learning with sharing) mechanism, the technical benefits that can be achieved include reducing model training complexity, reducing training time, model computing energy consumption and memory storage space requirements, while at the same time in the incremental learning process (incremental learning process) Achieve high accuracy.

The contents disclosed above are only preferred and feasible embodiments of the present invention, and do not limit the scope of the patent application of the present invention. Therefore, all equivalent technical changes made by using the description and drawings of the present invention are included in the application of the present invention. within the scope of the patent.

100: Incremental learning model

11: Type one

12:Type two

13: Type three

21: Source data set

210: Neural network learning algorithm

22: Target data set

220: New Neural Network Learning Algorithm

30:Deep neural network model

FC1, FC2: fully connected layer

301:Output layer

303: New output layer

FC1’, FC2’: new fully connected layer

50:Updated deep neural network model

701:Old type data

702: New type of data

70:Computer system

71: Processor

72:Memory

Steps S401 to S411: The process of the training phase in the edge optimization incremental learning method

Steps S601 to S603: Process of the inference phase in the edge optimization incremental learning method

Figure 1 shows the operation diagram of the current incremental learning model; Figure 2 shows a conceptual diagram of transfer learning; Figure 3 shows a schematic diagram of an embodiment of the neural network architecture of the edge optimization incremental learning method in the training stage; Figure 4 shows an embodiment flow chart of the training phase in the edge optimization incremental learning method; Figure 5 shows a schematic diagram of an embodiment of the neural network architecture of the edge optimization incremental learning method in the inference stage; Figure 6 shows an embodiment flow chart of the inference phase in the edge optimization incremental learning method; and Figure 7 shows a schematic diagram of an embodiment of a computer system running the edge optimization incremental learning method.

21: Source data set

210: Neural network learning algorithm

22: Target data set

220: New Neural Network Learning Algorithm

Claims (8)

An edge optimization incremental learning method for deep neural networks, including: preparing a deep neural network model, originally connecting one or more fully connected layers and an output layer; disconnecting the one or more fully connected layers and The original connection of the deep neural network model is such that the one or more fully connected layers do not participate in the incremental learning training parameter update; the one or more fully connected layers are copied into one or more new fully connected layers, one of which The new fully connected layer is connected to a new output layer, where the size of the new output layer is equal to the number of categories to be incrementally learned in a given time; the one or more new fully connected layers and the new output layer are connected to the a deep neural network model; and during an incremental learning training process, through the deep neural network model, participate in training with a newly added incremental data set, and update the deep neural network model. The edge-optimized incremental learning method for deep neural networks as described in claim 1, wherein copying the original one or more fully connected layers helps the deep neural network model quickly achieve high classification accuracy. . The edge-optimized incremental learning method for deep neural networks as described in claim 1, wherein during the incremental learning training process, only newly added incremental data sets participate in the training, and the original deep neural network The data set for which the road model has completed training will not participate in new training. The edge-optimized incremental learning method for deep neural networks as described in claim 3, wherein different applications and different numbers of categories can be gradually added to the deep neural network during each incremental learning training process. in the model. The edge optimization incremental learning method for deep neural networks as described in any one of claims 1 to 4, wherein after completing incremental learning and updating the deep neural network model, in a model inference stage, The one or more fully connected layers that were previously disconnected from the deep neural network model are reconnected to the updated deep neural network model, so that the one or more fully connected layers of the original learning type and the incremental learning type are The one or more new fully connected layers share the updated The deep neural network model. A computer system running a deep neural network, wherein the computer system includes a processor and a memory with specific computing power and storage resources, and an edge-optimized incremental learning method executed therein includes: preparing a deep neural network model , originally connecting one or more fully connected layers and an output layer; disconnecting the original connection between the one or more fully connected layers and the deep neural network model, so that the one or more fully connected layers do not participate in incremental learning Training parameter update; copy the one or more fully connected layers into one or more new fully connected layers, one of the new fully connected layers is connected to a new output layer, wherein the size of the new output layer is equal to a given time The number of categories to be incrementally learned within; the one or more new fully connected layers and the new output layer are connected to the deep neural network model; and during an incremental learning training process, through the deep neural network model, Participate in training with the newly added incremental data set and update the deep neural network model. The computer system for running a deep neural network as described in claim 6, wherein during the incremental learning and training process, only the newly added incremental data set participates in the training, and the original deep neural network model has completed training The data set does not participate in new training. Different applications and different numbers of categories can be gradually added to the deep neural network model during each incremental learning training process. The computer system for running a deep neural network as described in claim 6 or 7, wherein after completing incremental learning and updating the deep neural network model, in a model inference stage, the previous connection with the deep neural network model is disconnected. The one or more fully connected layers of the open connection are connected again to the updated deep neural network model, so that the one or more fully connected layers of the original learning type and the one or more new fully connected layers of the incremental learning type The layers share the updated deep neural network model.

Images