KR101947780B1 - Method and system for downsizing neural network - Google Patents

Abstract

A neural network downsizing method is disclosed. The method for downsizing a neural network according to an embodiment of the present invention includes: training a large neural network by using first training data for downsizing a neural network, the method comprising: generating second training data for knowledge transfer to a small neural network; And training the small neural network by using the second training data.

Description

[0001] The present invention relates to a method and system for downsizing a neural network,

The present invention relates to a method for downsizing a neural network, in particular for training a large neural network in an optimized manner for downsizing of a neural network and downsizing a neural network for performing a particular test by training a small neural network using training results of a large neural network It is about the method.

A neural network is a network of neurons in a human brain. Neurons as mathematical models are interconnected to form a network, and may be referred to as artificial neural networks, distinct from biological neural networks.

As the application range of artificial intelligence increases, neural network, which is one of the algorithms for artificial intelligence implementation, is also applied to many technical fields.

Neural networks can perform highly complex tasks by performing highly complex operations. However, the implementation of the neural network itself requires a complicated structure and algorithm, and thus requires more memory and computing power than a certain amount. As the use of neural networks becomes more prevalent, a method of reducing the size of a neural network for personalized data processing is proposed.

It is important to minimize the performance degradation while reducing the size of the neural network. A training method for improving the performance of a neural network with reduced size is required.

According to another aspect of the present invention, there is provided a method of downsizing a neural network, comprising: training a large neural network by using first training data for downsizing a neural network, Wherein each of the k hidden layers includes M_k nodes and the first training data is a data set for training a neural network to provide an answer to a particular task, The first label data indicating a correct answer to the first input data; Generating second training data for knowledge transfer to a small neural network; And training the small neural network by using the second training data, wherein the small neural network comprises k hidden layers, each k hidden layer comprising N_k nodes less than the M_k, .

The method of downsizing a neural network according to an exemplary embodiment of the present invention may include training the large neural network by using the first training data by selecting N_k nodes among the M_k nodes of each of the plurality of hidden layers Processing the input data by outputting first output data as a result of processing the input data; And comparing the first output data with the label data to adjust the parameters of the large neural network.

The method of downsizing a neural network according to an embodiment of the present invention is characterized in that the second training data is a data set for training a neural network to provide an answer to a specific task, The second input data of the second training data corresponds to the first input data of the first training data, and the second input data of the second training data corresponds to the second input data of the second training data, The second label data corresponds to the first output data output from the large neural network.

In the neural network downsizing method according to an embodiment of the present invention, the N_k nodes used in the plurality of hidden layers are randomly selected for each of the hidden layer and / or the data pair.

In the neural network downsizing method according to an embodiment of the present invention, training the small neural network by using the second training data may include processing the second input data and outputting second output data; And adjusting the parameter of the small neural network by comparing the second output data with the second label data.

In the neural network downsizing method according to an embodiment of the present invention, the first label data is one hot encoded data, and includes a plurality of binary values indicating whether an answer is correct, And a plurality of probability values indicating probability of correct answers.

According to another aspect of the present invention, there is provided a computing system for downsizing a neural network, comprising: a memory for storing data; And a processor coupled to the memory for performing neural network operations and neural network downsizing, wherein the processor trains a large neural network by using first training data for neural network downsizing, and transmits knowledge to a small neural network And training the small neural network by using the second training data, the large neural network comprising k hidden layers, each of the k hidden layers including M_k nodes, The first training data is a data set for training a neural network so as to provide an answer to a specific task, the first training data including first input data for the task and first label data representing a correct answer to the first input data, The small neural network includes k hidden layers And each of k hidden layers includes N_k nodes smaller than M_k.

According to the present invention, the system can reduce the size of the neural network by reducing the number of nodes in the hidden layer. And the system can train the small neural network to train the small neural network quickly and efficiently by transmitting the pre - trained knowledge in the large neural network. In the course of training a large neural network, the present invention can train a neural network of a target size in an optimized manner, so that the training knowledge of a large neural network can be effectively applied to a small neural network. Therefore, the present invention can minimize performance degradation due to size reduction while reducing neural network size for task execution. And the system of the present invention can train the reduced size neural network efficiently and with good performance.

In the following specification, the effects of the present invention can be further explained together with the constitution.

FIG. 1 shows a neural network model and operation of each node according to an embodiment of the present invention.
2 shows a neural network according to an embodiment of the present invention.
FIG. 3 shows a learning process of a neural network according to an embodiment of the present invention.
4 shows a teacher model and a student model for transferring a neural network model according to an embodiment of the present invention.
5 shows a method of generating training data using a teacher model according to an embodiment of the present invention.
FIG. 6 shows a method of training a target neural network using new training data according to an embodiment of the present invention.
FIG. 7 shows a method of training a neural network model according to another embodiment of the present invention.
8 shows a neural network downsizing method according to an embodiment of the present invention.
9 shows a learning method of a large neural network according to an embodiment of the present invention.
10 shows a method for generating a new data set for training a small neural network according to an embodiment of the present invention.
11 shows learning of a small neural network according to an embodiment of the present invention.
12 illustrates a neural network downsizing method according to an embodiment of the present invention.
13 shows a neural network downsizing system according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following detailed description with reference to the attached drawings is for the purpose of illustrating preferred embodiments of the present invention rather than illustrating only embodiments that may be implemented according to embodiments of the present invention. The following detailed description includes details in order to provide a thorough understanding of the present invention, but the present invention does not require all of these details. The present invention is not limited to the embodiments described below. Multiple embodiments or all of the embodiments may be used together, and specific embodiments may be used as a combination.

Most of the terms used in the present invention are selected from common ones widely used in the field, but some terms are arbitrarily selected by the applicant and the meaning will be described in detail in the following description as necessary. Accordingly, the invention should be understood based on the intended meaning of the term rather than the mere name or meaning of the term.

The present invention is directed to a Deep Neural Network, which may be abbreviated herein as a neural network.

A neural network is a network of neurons in a human brain. Neurons as mathematical models are interconnected to form a network, and may be referred to as artificial neural networks, distinct from biological neural networks.

In the present invention, a computing system may implement a neural network and perform data operations. A computing system may refer to a computing processing portion of any device capable of data processing, such as a computer, a cell phone, a smart hub, a notebook, an IoT device, and the like. In this specification, a computing system may be abbreviated as a system. The computing system may perform the invention by performing data, such as an application stored on a storage device.

FIG. 1 shows a neural network model and operation of each node according to an embodiment of the present invention.

1 (a) shows a neural network according to an embodiment of the present invention. The neural network includes neurons, and each circle in Fig. 1 (a) represents neurons. A set of plural neurons is called a layer. Neurons can be referred to as nodes below. The layer receiving the input data is referred to as an input layer, and the layer outputting the result of operation of the neural network is referred to as an output layer.

Figure 1 (b) shows the behavior of neurons in neural networks. The neuron applies weights (w_0, w_1) to the inputs (i_0, i_1) and performs an active function (f (x)) operation to provide an output. The active function may be a function that increases rapidly when a stimulus is given above a certain level, such as a real neuron. The use of linear functions can reduce analytical capabilities, and nonlinear functions can be used. As an example, a function such as sigmoid, tenh, relu, etc. may be used as the activation function. The activation function converts the sum of the input signals into an output signal without using the sum. That is, the activation function determines whether or not to activate the weighted inputs.

The neural network learns to perform specific tasks. That is, the system can construct a neural network and train a neural network to perform specific tasks. For example, a task can be performed to find out whether a person is a cat by viewing a photograph. The system repeatedly performs input data operation (processing), comparison of output data and correct data, and weight adjustment operation by using correct answers data (dog or cat) for a plurality of photographs and a plurality of photographs. The neural network thus trained is ready to perform the task. In this specification, data including inputs and correct answers for learning / training of a neural network may be referred to as a data set or training data.

2 shows a neural network according to an embodiment of the present invention.

FIG. 2 (a) shows a neural network according to an embodiment of the present invention, and FIG. 1 (b) shows a data set for neural network modeling.

In FIG. 1, the neural network includes an input layer, an output layer, and hidden layers (layer 1 to layer 3). The input layer can forward incoming signal / data to the next layer. The output of nodes in the output layer may correspond to the final result of the neural network. At least one layer between an input layer and an output layer is referred to as a hidden layer. In particular, a neural network having plural hidden layers may be referred to as a deep neural network. In the present invention, a depth neural network will be described as an embodiment.

The neural network can be modeled by learning or training. Once the neural network is modeled, it can be seen that the system is ready to perform the desired task. That is, the modeled neural network can provide an inferred output to the input.

The neural network can be modeled to perform the target task. A data set may be prepared to train the neural network to achieve the target task, and such data set may be referred to as training data. The data set includes a plurality of data pairs (Data Pair 1, Data Pair 2, Data Pair 3...), And the data pair includes input data and label data. The label data may represent one correct answer as one hot encoded data. As an example, the label data may be raw hot encoded data and may include a plurality of binary values.

The training data may include a plurality of data pairs. However, training data may include input data and label data depending on the classification of data. The input data and the label data may be matched with each other. That is, the correct answer to the n-th input data may correspond to the n-th label data.

The hidden layer of the neural network contains nodes, and the nodes have weights. The weights of the nodes can be determined / adjusted through learning in the training phase. The neural network can adjust the weight of each node so that the difference between the label data and the output data is small. At least one weight for the nodes may be referred to as a parameter of the neural network. That is, the system / neural network can perform parameter adjustment of the neural network for each input data, output data and label data, and such parameter adjustment is repeatedly performed as many times as the number of input data, output data and label data included in the training data .

FIG. 3 shows a learning process of a neural network according to an embodiment of the present invention.

In the embodiment of FIG. 3, the system trains a neural network model by entering a set of data into a neural network model. That is, (1,2,0,9,5,4,7,3) is input to the neural network, and the neural network outputs (0.1, 0.2, 0.6, 0.1). Then, the system / neural network compares the label data (0, 0, 1, 0) for the output with the input data. The system / neural network can adjust the parameters of each layer to reduce the difference between the output data (0.1, 0.2, 0.6, 0.1) and the label data (0, 0, 1, 0).

The neural network computes the input training data, compares the output with the label, and performs the process of parameter adjustment repeatedly. Once learning has been completed for all training data, this neural network can be seen as modeled for a particular task.

However, the size of the neural network that can be implemented may vary depending on the computing environment implementing the neural network. The greater the number of hidden layers and the more nodes the hidden layer contains, the more generally the neural network can be modeled. A better modeled neural network is more likely to provide the correct answer for a particular task. However, many hidden layers and a large number of nodes may require strong computing power and storage space. Therefore, it may be necessary to transfer the learned neural network model to a small neural network depending on the environment. In this specification, the original neural network model learned for the transfer of the neural network model is referred to as a large neural network model or a teacher neural network model, and the neural network model is referred to as a small neural network model or a student neural network model can do. The neural network model may be abbreviated as a neural network or model.

Hereinafter, a method of transferring a learned model from one neural network to another neural network will be described.

FIGS. 4 to 6 illustrate a method of transferring a neural network model according to an embodiment of the present invention.

4 shows a teacher model and a student model for transferring a neural network model according to an embodiment of the present invention.

In FIG. 4, model A represents a model trained using training data as described in FIG. 2, and model B represents a model that has not yet been learned. Although FIG. 4 shows two models with the same number of layers, model A and model B may contain different numbers of layers or different numbers of nodes.

5 shows a method of generating training data using a teacher model according to an embodiment of the present invention.

As described above, the neural network model is learned by training data, i.e., data pairs including input data and label data. 5, when the input data is (1, 2, 0, 9, 5, 4, 7, 3), the output data of the model A can be (0.1, 0.2, 0.6, 0.1). In this case, the system can generate new training data by considering the output data (0.1, 0.2, 0.6, 0.1) as the label for the input data (1,2,0,9,5,4,7,3). By repeating the process shown in Fig. 5, the system can generate new training data having a plurality of data pairs by considering a plurality of output data as label data for a plurality of input data.

FIG. 6 shows a method of training a target neural network using new training data according to an embodiment of the present invention.

In Fig. 6, the system trains the target model by using the new training data generated in Fig. That is, when learning the target model, the output data generated by the original neural network model performing the same task is used as the label data.

In Fig. 6, the target neural network model is learned using output data (0.1, 0.2, 0.6, 0.1) output as a result of learning in the original neural network model. That is, since the output data corresponding to the learning result of the original neural network model is used as the label, the target neural network model can perform optimized learning using more information.

The label data of the new training data generated as shown in FIG. 6 may include soft decision values or soft values. The soft decision value or the soft value may be a probability value indicating that each value of the output data is correct. That is, the new training data may include a plurality of soft values indicating a probability of being correct.

In FIG. 6, for the first data pair, the output data is the probability that the first value is 1 (0.3), the second value is 1 (0.2), the third value is 1 (0.4) (0.1). The system can adjust the parameters of the neural network by comparing this output data with label data (0.1, 0.2, 0.6, 0.1).

This training data includes knowledge learned by the training of the model A. [ That is, the result (knowledge) of the training of the model A is used as the training data by the model B. Thus, Model B can be trained more efficiently and better than when using label values. Especially, learning data of common area among the data of tasks and transferring the knowledge to the target neural network can further reduce the training burden of the target neural network.

The output data of the learned original model has information on how to classify each class and how to distinguish each class by a certain criterion. By using this knowledge transfer method, the target model can learn faster and more accurately.

For example, the task of the neural network model may be the dog / cat classification of the photograph. For a general learning method, the neural network model learns all dog pictures using the label data [dog, cat] = [1,0]. And the output data is [0.7, 0.3], but as the result of the task, you can output the correct answer as dog. The knowledge transfer method uses the output data of the original neural network including the soft value as the label data so that the target network can be trained faster and more accurately. As an embodiment, it is possible to improve the performance of the target neural network by learning a larger original neural network to accumulate knowledge and using this knowledge to train a smaller target neural network.

The nodes can learn more about the target model with data that has less information but more information, which can further improve performance.

FIG. 7 shows a method of training a neural network model according to another embodiment of the present invention.

Figure 7 shows a method for optimizing a neural network model. That is, in addition to the neural network learning method described above, additional operations are added to the layers. In the embodiment of Fig. 7, at least one node is excluded from the calculation for each layer in neural network learning. FIG. 7 shows a neural network having nine nodes of each hidden layer.

 In the example of FIG. 7, the first, third, fourth, and ninth nodes are excluded from the operation in the layer 2. In Layer 3, nodes 3, 5, 6, and 7 are excluded from operation, and in Layer 4, nodes 2, 4, 7, and 8 are excluded from operations. The nodes excluded from the computation can be determined stochastically. The nodes excluded from the computation may be determined randomly. Nodes that are excluded from the operation can be ignored in the process of continuing the learning and modifying the parameters / weights. However, all neighbors may be used when neural networks operate after learning.

By excluding nodes with a certain probability in the learning process, it is similar to learning by using different models of similar type each time learning one data. That is, it may be an operation similar to learning a plurality of different models sharing some nodes. Therefore, using all the nodes in the neural network operation is equivalent to combining the results of a plurality of trained models, so that the neural network's correct inference performance can be greatly improved. According to this embodiment, it is possible to prevent the result from being deflected based on the training set as compared with learning only with one model, and therefore the performance of the neural network can be further improved.

8 shows a neural network downsizing method according to an embodiment of the present invention.

Recently, neural networks have been used in various electronic devices such as mobile devices and Internet on Things (IoT) devices. The process of training and inferring the neural network requires more computation as the size of the neural network model increases. Therefore, the training and speculation operation of the neural network can be performed in the external server. However, in this case, the personal information needs to be transmitted to the external server, which increases the risk of leakage of personal information and may incur server operating / communication costs. Therefore, in order to utilize neural network in individual devices, neural network model downsizing method which reduces the size of model while minimizing the performance of neural network is needed.

For this, a knowledge transfer method for transferring the neural network model of FIGS. 3 to 5 can be used. However, the above knowledge transfer method can improve the performance by learning the target neural network with the training data having more information, but it is difficult to avoid performance deterioration due to the difference of the network size. That is, if the size difference between the large neural network and the small neural network is large, it is difficult for the small neural network to express the knowledge transferred by the large neural network. Or because of the difference in size, knowledge transmitted in a large neural network may impair the learning performance of a small neural network. Therefore, there is a need to reduce the size of the neural network and minimize the performance degradation.

As shown in FIG. 8, the present invention proposes a method of training a large neural network so that a small neural network can generate a data set that can be learned. That is, the present invention describes a method for generating a data set that a small neural network can learn, and training a small neural network downsized with the generated data set. In the following, a large neural network is described as an embodiment in which the number of hidden layer nodes is nine and a small neural network has three hidden layer nodes, but the spirit of the present invention is not limited to such an embodiment.

9 shows a learning method of a large neural network according to an embodiment of the present invention.

The present invention proposes a method of reducing the number of nodes according to the number of nodes of a small neural network to be downsized when training a large neural network. In Figure 9, when learning a data set, the number of nodes in the hidden layer of a large neural network is reduced to three. Instead of using three predetermined nodes for the data set at this time, three different nodes may be used for each operation. That is, nodes are randomly selected at the time of each data operation for neural network training, and nodes can be selected randomly for each hidden layer.

In FIG. 9, a large neural network performs an operation using only three of nine nodes in the operation of the hidden layer. The three nodes used in the hidden layer operation are not fixed. Any three nodes randomly selected at any probability can be used for data operations at each hidden layer. In FIG. 8, when the first data (1, 2, 0, 9, 5, 4, 7, 3) is input, the first hidden layer performs operations using the first, fourth and eighth nodes, 5, and 9 nodes to output the output data of (0.1, 0.2, 0.6, 0.1). The system can adjust the parameters / weights of the neural network by comparing the output data with label data (0, 0, 1, 0).

When the second data (2, 6, 2, 8, 7, 4, 7, 2) is input, each hidden layer outputs output data using three randomly determined nodes that are the same as or different from the first data . The label data and the output data for the second data are compared with each other to adjust the weight or parameter for each node of each hidden layer of the neural network. The large neural network terminates learning by repeating this process for the entire data set.

As an example, the number of nodes of each hidden layer may be different. In this case, the present invention can randomly select the number of nodes of the hidden layer of the small neural network for the hidden layer nodes of the large neural network. That is, when the number of nodes of the second hidden layer of the small neural network is 3, the large neural network performs training using three nodes of the second hidden layer.

10 shows a method for generating a new data set for training a small neural network according to an embodiment of the present invention.

The large neural network generates a new data set with label data for the learning result for the data set, i.e., the output data. The data set thus generated may be referred to as a training data / data set for downsizing. The data set for downsizing receives the original data set (or raw data set) of a large neural network, and has data output as learning as label data. That is, the large neural network becomes a data set having label data for each input data as output data for learning and outputting the original data set.

9, the large neural network computes the input data (1, 2, 0, 9, 5, 4, 7, 3) by using three nodes per hidden layer, 0.1). By replacing the output data thus output with the label data, a data set for training a small neural network is generated. In Figures 9 and 10, if the data set includes a pair of x input data and label data, the data set for downsizing includes a pair of x input data and label data. The x label data of the data set for downsizing is output data which is a learning result of a large neural network, and is data including soft values.

For the generation of new training data, the above description with reference to Figs. 4 to 6 applies.

11 shows learning of a small neural network according to an embodiment of the present invention.

In Fig. 11, a small neural network having three nodes of each hidden layer starts learning. The small neural network starts learning using training data for downsizing generated in the procedure of FIGS.

11, the first input data (1, 2, 0, 9, 5, 4, 7, 3) is input to the smaller neural network, and the smaller neural network inputs the output data of (0.2, 0.1, 0.5, 0.2) . And the small neural network adjusts the parameter / weight using the training label data (0.2, 0.1, 0.5, 0.2). For the data set for training, the neural network performs the learning, and when the learning is finished, the task is ready to perform.

12 illustrates a neural network downsizing method according to an embodiment of the present invention.

The system trains a large neural network to downsize the neural network (S12010).

The system uses the first training data to train a large neural network. A large neural network contains k hidden layers, and each of the k hidden layers may contain M_k nodes. The first training data is a set of data that trains the neural network so that the neural network can provide an answer to a particular task. The first training data may include first input data and first label data.

The system may generate second training data for knowledge transfer to the small neural network (S12020).

The second training data is a set of data that trains a neural network to provide an answer to a particular task. The second training data may include second input data for the task and second label data indicating the correct answer for the second input data.

The system may train a small neural network by using the second training data (S12030).

A small neural network contains k hidden layers, and each of the k hidden layers may contain N_k nodes smaller than M_k. The system can construct or determine the structure of a small neural network before large neural network tracing. In other words, the system can deliver optimized training results to a small neural network by learning the number of nodes in a large layer according to the number of nodes per layer of the target small neural network.

The step of training the large neural network may include processing the first input data by selectively using N_k of the M_k nodes of each of the plurality of hidden layers, outputting the first output data processed input data, And comparing the first output data with the label data to adjust the parameters of the neural network. Such processing, output, and adjustment steps may be repeatedly performed for each input data and each label data of the training data. N_k nodes may be randomly selected / determined for each hidden layer and / or data pair. M_k is a natural number, and N_k is a natural number less than or equal to M_k.

k, M_k, and N_k may all be natural numbers. 8 to 11, k = 2, M_k = 9, N_k = 3. That is, although M_1 = M_2 = 9 and N_1 = N_2 = 3 are described, M_1 and M_2 may not be the same, and N_1 and N_2 may not be the same. The present invention satisfies the necessary condition when N_1 is less than or equal to M_1 and N_2 is less than or equal to M_2.

The second training data generation step is performed by exchanging the label data of the first training data with the first output data. That is, for the second training data, the second input data corresponds to the first input data of the first training data, and the second label data corresponds to the first output data output from the larger neural network.

Training the small neural network includes processing the second input data to output the second output data and comparing the second output data to the second label data to adjust the parameters of the small neural network.

The steps of training a large neural network for downsizing a neural network (S12010) and generating training data for knowledge transfer to a small neural network (S12020) are illustrated in FIGS. 1 to 10, , The above description can be applied to each step.

The first label data may be raw hot encoded data, and may include a plurality of binary values indicating whether or not an answer is correct. The second label data may include a plurality of probability values indicating a probability of correct answer.

According to the present invention, the system can downsize the neural network with minimal performance degradation. Also, instead of learning a completely new neural network, the amount of additional learning can be reduced by using neural networks learned for common data. In other words, the transfer of the neural network model can significantly reduce the learning amount of neural network modeling for personalization. Depending on the embodiment, the system may perform neural network downsizing according to the method described above for the common data, and may additionally train a small neural network for private information only.

13 shows a neural network downsizing system according to an embodiment of the present invention.

The neural network downsizing system 13000 is a computing system and may be included in any electronic device.

The memory 13010 is connected to the processor 13020 and stores various information for driving the processor 13020. [ Memory 13010 may be internal to processor 13020 or external to processor 13020 and may be coupled to processor 13020 by known means. Memory 13010 is collectively referred to as volatile and non-volatile memory. In the present invention, memory 13010 may store data such as neural networks, training information for neural networks, applications for neural network implementation and training.

The processor 13020 may be connected to the memory 13010 to perform the neural network, the training method of the neural network, and the downsizing method of the neural network according to the present invention. At least one of the modules, data, programs, or software that implement the operation of system 13000 according to various embodiments of the invention described above may be stored in memory 13010 and executed by processor 13020. [ The processing unit 12030 may implement the method of the present invention by driving application / software to perform the method of the present invention.

The communication unit 13030 can perform wired communication or wireless communication with an external device of the system. The communication unit 13030 may not be provided depending on the configuration of the device. Also, the communication unit 13030 may be composed of a plurality of communication chipsets. The communication unit 12030 includes a communication module and can perform communication based on various communication protocols such as 3G, 4G (LTE), 5G, WIFI, Bluetooth, and NFC.

The system 13000 or the processor 13020 of the system may perform the neural network downsizing method of the present invention described above with reference to FIGS.

The embodiments described above are those in which the elements and features of the present invention are combined in a predetermined form. Each component or feature shall be considered optional unless otherwise expressly stated. Each component or feature may be implemented in a form that is not combined with other components or features. It is also possible to construct embodiments of the present invention by combining some of the elements and / or features. The order of the operations described in the embodiments of the present invention may be changed. Some configurations or features of certain embodiments may be included in other embodiments, or may be replaced with corresponding configurations or features of other embodiments. It is clear that the claims that are not expressly cited in the claims may be combined to form an embodiment or be included in a new claim by an amendment after the application.

Embodiments in accordance with the present invention may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, an embodiment of the present invention may include one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs) field programmable gate arrays, processors, controllers, microcontrollers, microprocessors, and the like.

In the case of an implementation by firmware or software, an embodiment of the present invention may be implemented in the form of a module, a procedure, a function, or the like which performs the functions or operations described above. The software code can be stored in memory and driven by the processor. The memory is located inside or outside the processor and can exchange data with the processor by various means already known.

It will be apparent to those skilled in the art that the present invention may be embodied in other specific forms without departing from the essential characteristics thereof. Accordingly, the foregoing detailed description is to be considered in all respects illustrative and not restrictive. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention.

13000: Neural network downsizing system
13010: Memory
13020: Processor
13030: Communication unit

Claims (12)

A method for downsizing a neural network of a computing system,
Training a large neural network by using first training data for neural network downsizing and outputting first output data, wherein the large neural network includes k hidden layers, each of the k hidden layers including M_k nodes Wherein the first training data is a data set for training a neural network so as to provide an answer to a specific task, the first training data including first input data for the task and first label data representing a correct answer to the first input data Include;
Generating second training data for knowledge transfer to a small neural network wherein the second training data comprises a plurality of training data including second input data for the task and second label data representing a correct answer for the input data, Wherein the second input data of the second training data corresponds to the first input data of the first training data and the second label data of the second training data corresponds to the first output data ; And
Training the small neural network by using the second training data, the small neural network comprising k hidden layers, each k hidden layer comprising N_k nodes less than the M_k,
Wherein the first label data is one hot encoded data and includes a plurality of binary values indicating whether an answer is correct and the second label data includes a plurality of probability values indicating a probability of a correct answer,
Wherein k, M_k and N_k are natural numbers. The method according to claim 1,
Wherein training the large neural network by using the first training data and outputting the first output data comprises:
Processing the input data by selectively using the N_k nodes among the M_k nodes of each of the k hidden layers,
Outputting first output data that is a processing result of the input data; And
And comparing the first output data with the label data to adjust the parameters of the large neural network. delete 3. The method of claim 2,
Wherein the N_k nodes used in the k hidden layers are randomly selected for each of k hidden layers and / or data pairs of the first input data and the first label data. The method according to claim 1,
Wherein training the small neural network by using the second training data comprises:
Processing the second input data to output second output data; And
And comparing the second output data with the second label data to adjust the parameters of the small neural network. delete 1. A computing system for performing neural network downsizing,
A memory for storing data; And
And a processor coupled to the memory to perform neural network operation and neural network downsizing,
The processor comprising:
Training a large neural network by using first training data for neural network downsizing to output first output data, generating second training data for knowledge transfer to a small neural network, and transmitting the small neural network to the second training data , ≪ / RTI >
Wherein the large neural network comprises k hidden layers and each of the k hidden layers comprises M_k nodes and the first training data is a data set for training the neural network to provide an answer to a particular task, The first input data for the task and the first label data indicating the correct answer for the first input data,
The small neural network comprising k hidden layers, each k hidden layer comprising N_k nodes less than the M_k,
Wherein the second training data includes a plurality of data including second input data for the task and second label data representing a correct answer to the input data, The second training data corresponds to the first input data of the first training data, the second label data of the second training data corresponds to the first output data,
Wherein the first label data is one hot encoded data and includes a plurality of binary values indicating whether an answer is correct and the second label data includes a plurality of probability values indicating a probability of a correct answer,
Wherein k, M_k, and N_k are natural numbers. 8. The method of claim 7,
The training of the large neural network using the first training data and the output of the first output data,
Processing the input data by selectively using the N_k nodes among the M_k nodes of each of the k hidden layers,
Outputting first output data which is a processing result of the input data,
And comparing the first output data with the label data to adjust parameters of the large neural network. delete 9. The method of claim 8,
Wherein the N_k nodes used in the k hidden layers are randomly selected for each of the hidden layer and / or data pairs of the first input data and the first label data. 8. The method of claim 7,
Wherein the training of the small neural network using the second training data comprises:
Processing the second input data to output second output data, and
And comparing the second output data with the second label data to adjust parameters of the small neural network. delete

Images