JP2020091855A - Training model, method of generating model, and program - Google Patents

Abstract

To enable efficient graph-based computation.SOLUTION: A training device is provided, comprising a graph generation unit configured to generate a graph based on error back propagation paths; an ID allocation unit configured to allocate in identifier to each node in the graph based on the error back propagation paths; and a back propagation unit configured to perform error back propagation based on the graph and the identifiers.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a training device, a model generation method, and a program.

In machine learning, a neural network model represents a transition of data from an input layer to an output layer as a graph, and forward propagation and back propagation are performed and training is performed based on the connection of the graph. As a network construction, for example, there is a Define-by-Run form in which the network is defined during execution of training. In the Define-by-Run format, as shown in Non-Patent Document 1, the shape of the graph changes during training, so a graph showing the processing of data is formed during forward propagation, and the graph is used to reverse Propagate. As a result, a considerable area for storing the graph is required, and the memory area is pressed, which may make it difficult to efficiently train the network.

S. Tokui, et.al., "Chainer: a Next-Generation Open Source Framework for Deep Learning," Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS), 2015

In the present embodiment, a graph calculation device, a graph calculation method, and a program for efficiently performing calculation using a graph are provided.

According to one embodiment, the training device generates a graph based on a path of error backpropagation, a graph generation unit, and in the graph, assigns an identifier to each node based on the path of error backpropagation, An ID assigning unit and a backpropagation unit that performs error backpropagation based on the graph and the identifier.

The block diagram which shows the function of the training apparatus which concerns on one Embodiment. The figure which shows an example of the graph generation which concerns on one Embodiment. The flowchart which shows the process of the training apparatus which concerns on one Embodiment. The figure which shows an example of the graph generation which concerns on one Embodiment. The figure which shows an example of the graph generation which concerns on one Embodiment. The figure which shows an example of the graph generation which concerns on one Embodiment. The figure which shows an example of the graph generation which concerns on one Embodiment. The figure which shows an example of the graph generation which concerns on one Embodiment. The figure which shows the example of hardware implementation of the training apparatus which concerns on one Embodiment.

Hereinafter, embodiments will be described with reference to the drawings.

FIG. 1 is a block diagram showing functions of the training device according to the embodiment. The training device 1 includes an input unit 10, a storage unit 12, a graph generation unit 14, a forward propagation unit 16, an ID assignment unit 18, a back propagation unit 20, and an output unit 22, and inputs data. On the other hand, a trained model that outputs output data that has been subjected to predetermined processing is trained.

The input unit 10 receives data input. The input data is, for example, training data, and is necessary data such as data input to the network to be trained and teacher data (label) data used for calculating the loss.

The storage unit 12 stores data necessary for training in the training device 1 or a training result. The storage unit 12 stores, for example, a configuration of a network to be trained, parameters updated in the training, and the like. Further, the data input from the input unit 10 may be temporarily stored. Further, final parameters after the training is finished may be stored. Although the training device 1 is provided with the storage unit 12, a part or all of the storage unit 12 is provided outside the training device 1 so that the training device 1 can transmit and receive data via a communication line or the like. You may.

The graph generation unit 14 generates an operation graph at the timing when data is input to the network. The forward propagation unit 16 calculates the input data based on the description of the network formation definition stored in the storage unit 12. As another example, a network definition description part that describes a definition forming a network is provided, and even if a graph is generated while performing forward propagation based on the definition of the network described in the network definition description part. Good. As described above, the graph is not generated in advance based on the predetermined network definition and the processing such as forward propagation and back propagation is performed, but the timing of forward propagation is based on the description of the network definition as described above. The graph is generated in, and the subsequent processing is performed based on the generated graph. Different graphs may be formed depending on, for example, the form of input/output variables.

More specifically, the graph generation unit 14 and the forward propagation unit 16 do not perform separate operations, but the forward propagation unit 16 performs the forward propagation process based on the structure of the network to be trained. At the same time, the graph generation unit 14 generates a graph. That is, the training device 1 performs forward propagation of the network on a certain data or data group based on the respective data, and also generates a graph for back propagation. In this way, instead of generating an operation graph of a predetermined network and then performing the process after the forward propagation, the graph is generated each time according to the operation of the input data along with the forward propagation process. , And subsequent processing (for example, back propagation) is executed.

The ID assigning unit 18 assigns an identifier (hereinafter, referred to as a backpropagation ID) indicating a backpropagation route to the graph generated by the graph generating unit 14. When backtracking a different computation path for each variable included in the data, a unique backpropagation ID for each computation path is given to a node included in the computation path, for example, a variable node and a computation node. For example, when batches have the same type of variable, a graph may be generated for each batch and a back-propagation ID may be assigned to each variable. The back propagation ID is uniquely given to each graph.

The back-propagation unit 20 calculates a loss by comparing the result output by the forward-propagation unit 16 that forward-propagates the network and the teacher data (label), and executes the error back-propagation process based on the loss. This back-propagation process is executed for each back-propagation ID assigned by the ID assigning unit 18. The back propagation unit 20 may delete the graph for which back propagation has ended. To delete, the graph itself may be discarded, or the graph data may be substantially discarded by making the memory area in which the graph is saved, etc. overwritable (for example, releasing the memory). .. Further, this graph deletion is not executed by the back propagation unit 20, but is provided with a separate graph deletion unit (not shown), and the graph deletion unit deletes the graph based on the operation of the back propagation unit 20. Good.

When performing a batch operation or the like, the same type of variable may exist. In such a case, if there is a condition that the calculation paths do not differ for these variables of the same type or that differentiation is not performed twice or more, a variable node and a calculation node are generated as a variable group, and the same variables are generated. The calculation may be performed using the back-propagation ID of.

After the operation of the backpropagation unit 20 is completed, if necessary, the above-described processing is performed by the graph generation unit 14, the forwardpropagation unit 16, the ID assignment unit 18, and the backpropagation unit 20 in order to further train the network. Repeated until the end condition is met.

The output unit 22 outputs the learned model after the training is completed. The output of the learned model may be the output of the entire model, or the data such as the shape and parameters related to the model may be output, and the data capable of constructing the same learned model externally may be output. It may be one. Further, the trained model may be stored and stored in the storage unit 12 instead of being output to the outside via the output unit 22, and in this case, the trained device 1 after the training is used as the trained model. It may be made to function as an estimation device or the like.

FIG. 2 is a diagram showing an example of graph generation according to this embodiment. For example, a state in which the variables A and B are input and the variable C is output (C=F(A, B)) is shown. The edges (broken lines) connecting the nodes in each graph are directed from the output to the input, but the back propagation is illustrated in an easy-to-understand manner, and during forward propagation, transitions occur between the nodes in the reverse direction. Although it is shown as a directed graph, it may not necessarily be a directed graph as long as the order of back propagation can be determined. For example, an instruction may be given to follow a node so that an operation is performed from output to input. Specifically, a sequence of operation nodes from output to input may be stored and executed in order.

When the variables A and B are input, the function F is processed, and the variable C is output. These processes are executed by the forward propagation unit 16. In parallel with this forward propagation processing, the graph generation unit 14 generates a graph. For example, the graph generation unit 14 determines a back-propagation route at the timing when the process of the function F is defined by the forward propagation unit 16, and when there are a plurality of routes, generates a graph corresponding to each route. As a result, for example, when there is a back propagation path for each of the variables A and B, a graph including the variable node A, the function operation node F1, and the variable node C1, and the variable node B and the function operation node F2. And a graph including the variable node C2.

The ID assigning unit 18 assigns a backpropagation ID to the graph for each route that is backpropagated. In FIG. 2, in the graph to which the variable node A, the operation node F1, and the variable node C1 belong, back propagation ID1 is given to each node. On the other hand, in the graph to which the variable node B, the operation node F2, and the variable node C2 belong, the back propagation ID2 is given to each node. Note that the backpropagation ID may be added after the graph is generated or in parallel with the graph generation as described above, and the graph may be generated based on the backpropagation ID. Also in this case, the ID assigning unit 18 may assign the back-propagation ID to each node forming the graph when the graph is generated.

Although only one operation node is shown for each graph, this is for simplicity of explanation, and in practice, a graph in which a plurality of operation nodes are connected may be formed. Similarly, in the following description, only one operation node is shown for the graph, but there may be two or more operation nodes, and there is an operation node for each operation that requires back propagation in forward propagation. The back propagation is generated and the back propagation is calculated according to each calculation node. Also, regarding the input/output variable node connected to the operation node, it is not necessarily input/output to/from the first operation node (for example, input layer) or the last operation node (for example, output layer), and there are a plurality of operations. Input/output may be performed with respect to a node in the middle of the nodes. This relationship is equivalent to the input/output of variables with respect to the network to be trained, especially the backpropagating path.

When there are a plurality of operation nodes, the same applies to the connection between the operation nodes, and when a function branches to another function due to a back propagation path, different back propagation IDs are assigned. Multiple graphs may be generated. In this way, a graph and a node belonging to the graph are generated for each backpropagation route based on the variables and functions, and the backpropagation ID is uniquely assigned to the node in each graph.

The back propagation unit 20 performs error back propagation processing for each of the given back propagation IDs and updates the network. For example, by tracing the graph of back propagation ID1, back propagation is executed from the variable node C1 for the parameter in the function F1, and the parameter is updated. When all the operations are completed for the assigned ID, for example, when back propagation is performed up to the variable node A, the information of the node and the edge to which the back propagation ID1 is assigned is discarded, so that the graph is discarded. Alternatively, the information of the graph may be discarded. As yet another example, the back propagation unit 20 may discard the information for each node at the timing when it is no longer needed. By discarding each node, it becomes possible to release the used resources at an earlier timing.

At this stage, the graph to which the back propagation ID2 is assigned still exists. Therefore, the back-propagation unit 20 updates the network by executing the error back-propagation for the node to which the back-propagation ID2 is assigned in the same manner as above.

Similar to general machine learning, another variable may be input after this to further update the network. In this case, a graph is similarly generated with respect to the input variables and forward-propagated, a back-propagation ID is given, and back-propagation is performed, while unnecessary graphs are automatically discarded. For training, a general machine learning method can be used. Of course, it is also possible to perform processing in parallel using a mini batch or the like.

FIG. 3 is a flowchart showing the processing according to this embodiment.

First, the input of data is accepted via the input unit 10 (S100). The data may be input for each individual data, or a predetermined number or a predetermined size of data may be received at once. Further, the input is not limited to being explicitly input from the outside, and the input unit 10 may acquire the data stored in the external storage or the like to receive the input, or the storage unit 12 may receive the input. It may be obtained from the data stored in.

Next, the forward propagation part 16 forward-propagates the input data (S102). By performing the forward propagation, the output when the input data is input to the model to be learned is acquired.

Next, the graph generator 14 generates a graph (S104). The generation of the graph is performed for each variable (that is, the received input data) based on the network configuration, or based on the path that is backpropagated for each function. That is, when there are a plurality of backpropagating paths, a plurality of graphs are generated. This route can be obtained from the network definition description.

Next, the ID assigning unit 18 assigns the back propagation ID to each node of the generated graph based on the variable node held by the input variable (S106). When the route to be followed in the back propagation in the model to be trained differs depending on the input variable, a different back propagation ID is given to each route required in the back propagation due to the input variables, parameters, output variables and the like. In the flowchart, the processes of S104 and S106 are shown separately, but in reality, the ID assigning unit 18 generates the graph of S104 based on the backpropagation ID of the variable node, and Is given. That is, the processes of S102 to S106 may be executed in cooperation with each other instead of being executed individually.

As another example, the processes of S102 to S106 may be sequentially executed. For example, in order of S102, S104, and S106, forward propagation may be performed to generate a graph and IDs may be assigned. After S102, the processes of S104 and S106 may be executed together, that is, forward propagation may be performed. After completion, the graph creation and the ID assignment may be executed together. Alternatively, S102 may be performed after S104, that is, forward propagation may be performed after a graph for back propagation is generated. As described above, the processes of S102 to S106 are applied to any implementation as long as the result of forward propagation can be acquired, and a graph can be generated for each route of back propagation and a back propagation ID can be given. be able to.

Next, the back propagation unit 20 executes the error back propagation processing based on the given back propagation ID (S108). Further, the graph data, for example, the node after the operation is completed and the back propagation ID required in the subsequent operation is not added, is discarded after the error back propagation based on the back propagation ID is completed ( S110). As another example, a node that is not reused may be discarded while performing error back propagation. When there is a reference relationship between the graphs, the backpropagation unit 20 may determine the order of backpropagation and perform backpropagation processing in that order.

Next, the back-propagation unit 20 branches the processing depending on whether there is a back-propagated graph, that is, a graph that has not been discarded (S112). If there is a graph that has not been discarded (S112: NO), backpropagation has not yet been completed, so error backpropagation is performed on the graph that has not been discarded (S108 to S110).

For example, after the back propagation and the discarding of the graph are completed for the graph of back propagation ID1, the process is subsequently executed for the graph of back propagation ID2. The branch in this flowchart is described for convenience only, and may not be dynamically formed, and means that back propagation and graph discard are performed for all generated graphs.

Further, as indicated by the broken line, another forward propagation process may be executed after the back propagation is completed using a certain graph. In this case, after the processing of a certain graph, the forward propagation processing for another back propagation ID may be executed to generate a graph and back propagation may be performed (S102 to S110). In this way, other forward propagation processing and graph generation processing may be performed after the back propagation processing.

Furthermore, for example, the forward propagation of the back propagation IDs 1 and 2 and the graph generation are executed, the back propagation of the back propagation ID1 and the graph discard are executed, the back propagation of the back propagation ID3 and the graph generation are performed, and then the back propagation ID2 is performed. It may be a complicated process such as back propagation of, and graph discard. As described above, the order of this process is uniquely determined and executed based on the description of the network definition, for example, when the forward propagation process based on the input variable is defined and the forward propagation is started. May be done. As another example, there may be a branch of a further forward propagation process at the timing of the start of the subsequent forward propagation process. In this way, the processes of S102 to S112 can be appropriately executed with respect to any process executed in the general Define-by-Run format.

That is, the description of the processing from S102 to S110 in the flowchart is merely arranged in order for convenience, and forward propagation and graph generation, back propagation, and graph discard are performed for each back propagation path. It is sufficient that the processes of 1) are executed in order, and it is not required that the processes having the same back propagation ID be continuously executed. As described above, the processing of the back propagation ID1 is continuously performed in order, while the processing of the back propagation ID2 and the processing of the back propagation ID3 are not performed consecutively in order for each ID. Good. Even in more complicated cases, it is possible to appropriately change the order of processing.

On the other hand, when all the graphs have been discarded (S112: YES), it is determined whether or not the training is completed (S114). The end of the training is the same as general learning, the training of a predetermined number of epochs was performed, the loss value was smaller than the predetermined value, the accuracy was larger than the predetermined value, etc. Judgment is made according to the termination condition of.

When the training is not completed (S114: NO), a new graph is created for the new input variable (S102), and the training process is repeated (S104 to S112).

On the other hand, when the training is completed (S114: YES), the result is output, stored, etc. (S116), and the training process is ended.

As described above, according to the present embodiment, it is not necessary to retain the information of the generated graph until back propagation is performed for all variables and functions. Therefore, while performing Define-by-Run processing, memory It is possible to improve the efficiency of using the. More specifically, since multiple graphs can be used, it is possible to more clearly and finely control which part of the graph is retained and which part is not, for the purpose of calculating the error backpropagation, which improves memory efficiency. Can be planned.

For example, in the error back propagation, when the graph has a plurality of paths, such as performing differentiation twice or more with respect to the operation node, it becomes possible to perform the training more efficiently. This can also be used when the error backpropagation performed in the loss function calculates the gradient with respect to the input, but the error backpropagation with respect to the loss function does not calculate the gradient with respect to the input. By generating and discarding the graph in this way, for example, when the error backpropagation is completed for a part of the variables, the unnecessary graph is discarded and the memory area used for the graph is released. It becomes possible.

In FIG. 2, an example of graph generation has been described, but a further different example will be described below. In any case, the overall processing flow is the same as the above-mentioned flowchart.

FIG. 4 is a diagram showing graph generation when the variable A has two nodes in the input node. For example, in the calculation of B=F(A), when the variable A has two nodes A1 and A2, operation nodes corresponding to the respective nodes A1 and A2 are generated in the back propagation as shown in FIG. It In the following figures, the solid line indicates the edge of the graph, and the alternate long and short dash line indicates node generation from variables and the like.

A graph having nodes A1, F1, and B1 and a back propagation ID1 assigned to each node for the variable A, and a graph having nodes A2, F2, and B2 having a back propagation ID2 assigned to each node And are generated respectively. More specifically, in forward propagation, the variable F is output by the operation F. When there are two backpropagation routes, the graph of the backpropagation ID1 including the nodes A1, F1, and B1 and the nodes A2, F2, and B2 are included based on the backpropagation routes as shown in FIG. And a graph of back propagation ID2.

FIG. 5 is a diagram showing graph generation when two variables are input and one variable is output. For example, it is a case of an operation that requires two arguments such as C=F(A,B).

In this case, when A and B have variable nodes having different backpropagation IDs, operation nodes corresponding to the respective backpropagation IDs are generated, and for output variable C, variable nodes C1 and C2 corresponding to all of them are generated. Is generated. For example, in the operation node F1, the error backpropagation process is executed from the variable node C1 which is an output for the variable A, while in the operation node F2, the error backpropagation process is executed from the variable node C2 which is an output for the variable B. Is executed.

In this way, graphs to which different backpropagation IDs are given for outputs from different variables and the same or different operations are generated.

In all the above-mentioned examples, it has been described that the variables have different nodes, but the operation node is connected to each variable from only one path. However, the present invention is not limited to this. There may be connections to different input variable nodes in the node.

FIG. 6 shows graph generation in the case where a plurality of input nodes and output nodes are connected to one operation node as described above. As an example, a case where the input node and the output node are referred to in the operation of B=F(A) will be described. In the following figures, broken lines indicate reference relationships. For example, the graph generation unit 14 sets the reference relationship between the nodes based on the back propagation path.

The backpropagation ID1 is given to the variable nodes A1 and B1 and the operation node F1, and the backpropagation ID2 is given to the variable nodes A2 and B2 and the operation node F2. However, since the variables of the variable nodes A1 and B1 are used in the calculation of the calculation node F2, the calculation node F2 is further provided with a reference relationship (broken line arrow in the figure) to the variable nodes A1 and B1.

In such a case, the back propagation unit 20 executes an operation from the graph of the back propagation ID having the referenced node. This will be described using the example of FIG. There are two graphs having backpropagation IDs 1 and 2, and the operation node F2 to which the backpropagation ID2 is assigned has a reference to the variable nodes A1 and B1 to which the backpropagation ID1 is assigned.

In this case, the back propagation unit 20 starts the calculation from the graph related to back propagation ID2. First, the variable node B2 calculates the loss, and the calculation node F2 calculates the gradient based on the backpropagated loss. For example, variables A1 and B1 are used to calculate this gradient. Similar to the above, only one operation node is shown, but a plurality of operation nodes may be used. Then, the variables A1 and B1 may be used to calculate the gradient for a certain operation node, the variables A1 and B1 may be used to calculate the gradient for different operation nodes, respectively, or a combination thereof. May be. The same applies when the number of input/output variables is 3 or more.

When the backpropagation unit 20 calculates the gradient of the operation node F2, the error backpropagation process is executed up to the variable node A2, and when the error backpropagation process of the graph relating to the backpropagation ID2 ends, the backpropagation unit 20 performs the backpropagation process. Discard the graph that has each node to which ID2 is assigned. Then, the error backpropagation process for the backpropagation ID1 is executed.

In this way, by having a reference relationship, even when there is a mutual relationship between a variable node or an operation node among multiple graphs, the graphs are created and discarded to save memory. It is possible to improve the use efficiency of. The order of performing the back propagation processing (calculation) is determined, for example, by the back propagation unit 20 extracting the reference relationship in each graph and based on the extraction result.

When the differential operation depends on another backpropagation ID variable node, the differential calculation itself is stored using the operation node and the variable node in the same manner as the normal backpropagation calculation, and based on the reference relationship. And calculate.

As shown in FIG. 7, variable nodes may have a reference relationship. As another example, each graph is provided with a variable node that indicates a reference relationship, and a reference relationship from a node in another graph or to a node in another graph is stored, and at the timing of performing back propagation. By confirming the reference to the variable node, backpropagation may be performed in an order in which the graph can be efficiently discarded. For example, the back-propagation is executed from a graph having a reference to another graph by confirming a reference relation to a node having another back-propagation ID.

As another example, a reference graph showing a reference relationship may be further provided, and the operation may be executed from the graph having the backpropagation ID corresponding to the terminal node of the reference graph. By doing so, it is possible to easily apply even when there are a plurality of graphs having complicated reference relationships.

The reference relationship is provided for a graph that refers to a node of another graph, but conversely, a relationship having a node referred to by another graph may be stored together with the graph. In this case, it is possible to discard the information from the graph for which the computation is completed by executing the computation of the back propagation from the node that appropriately refers to the same as described above.

FIG. 8 shows an example of graph generation in a more complicated case. It is assumed that the relationship between input/output and calculation is B=F(A) as in the above. Further, it is assumed that the differential F′ of the operation F is a calculation using the input/output variables A and B. In FIG. 8, dotted lines indicate discarded nodes and edges.

First, at the time of graph generation, the calculation node F2 refers to the variable nodes A1 and B1. ΔB represents a gradient for the variable B obtained as a result of the error back propagation, and ΔA represents a gradient for the variable A obtained as a result of the error back propagation for the operation F. FIG. 8 shows the case where there is a node to which the back propagation ID1 is assigned for ΔB.

The back propagation processing is executed from the graph to which the back propagation ID2 is assigned. When the differential F2' of F2 is calculated, the back propagation ID1 is given to the operation node of F2'. Then, reference is made from the operation node F2' to the variable nodes A1 and B1. Then, the variable node B2, the operation node F2, and the variable node A2 for which the back propagation is completed are discarded.

In this way, when there is a node that depends on another graph, by generating a node that transmits the input of the variable to which the backpropagation ID is assigned, the graph is discarded for each backpropagation route as described above. It becomes possible to do. After the error backpropagation of the referenced node is completed, the variable node and the operation node to which the backpropagation ID is assigned are discarded, and in particular, of the resources held by the operation node, other backpropagation IDs Resources are released for those that are not shared with compute nodes. On the other hand, the error backpropagation with respect to the calculation graph of the other backpropagation ID can be correctly calculated by, for example, holding the calculation node F2'.

In this embodiment, a simple case has been described with some examples, but a graph is generated, a backpropagation ID having uniqueness is given to a node for each generated graph, and different backpropagation is performed if necessary. By providing the reference relationship between the nodes to which the IDs are assigned, it is possible to release the resources in order from the calculation graph in which the back propagation is completed. This can be similarly applied to a more complicated graph than that shown above.

In this way, when backpropagating a network, a graph is not generated for the entire network, but a graph is generated based on a backpropagating path, so that resource utilization efficiency can be improved. As a result, it is possible to improve the efficiency of the memory even in the Define-by-Run method that performs complicated calculation.

In the training device 1 according to the above-described embodiment, each function may be a circuit including an analog circuit, a digital circuit, or an analog/digital mixed circuit. Moreover, a control circuit for controlling each function may be provided. Each circuit may be mounted by using an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

In all the above descriptions, at least a part of the training apparatus 1 may be configured by hardware, or may be configured by software, and a CPU (Central Processing Unit) or the like may be implemented by information processing of the software. .. In the case of being configured by software, the training apparatus 1 and a program that realizes at least a part of the functions thereof may be stored in a storage medium such as a flexible disk or a CD-ROM, read by a computer, and executed. Good. The storage medium is not limited to a removable medium such as a magnetic disk or an optical disk, and may be a fixed storage medium such as a hard disk device or a memory. That is, information processing by software may be specifically implemented by using hardware resources. Further, the processing by software may be implemented in a circuit such as FPGA and executed by hardware. The job may be executed by using an accelerator such as GPU (Graphics Processing Unit).

For example, the computer can be the device of the above-described embodiment by the computer reading the dedicated software stored in the computer-readable storage medium. The type of storage medium is not particularly limited. Further, the computer can be the device of the above embodiment by installing the dedicated software downloaded via the communication network by the computer. In this way, information processing by software is specifically implemented using hardware resources.

FIG. 9 is a block diagram showing an example of the hardware configuration according to the embodiment of the present invention. The training device 1 includes a processor 71, a main storage device 72, an auxiliary storage device 73, a network interface 74, and a device interface 75, and these can be realized as a computer device 7 connected via a bus 76. ..

Note that the computer device 7 of FIG. 9 includes one each of the constituent elements, but may include a plurality of the same constituent elements. Further, although one computer device 7 is shown, software may be installed in a plurality of computer devices, and each of the plurality of computer devices may execute a part of processing of different software.

The processor 71 is an electronic circuit (Processing circuit, Processing circuitry) including a control device and an arithmetic unit of a computer. The processor 71 performs arithmetic processing based on data and programs input from each device of the internal configuration of the computer device 7, and outputs the arithmetic result and control signal to each device. Specifically, the processor 71 controls each constituent element of the computer device 7 by executing an OS (Operating System) of the computer device 7, an application, or the like. The processor 71 is not particularly limited as long as it can perform the above processing. The training device 1 and each component thereof are implemented by the processor 71. Here, the processing circuit may refer to one or a plurality of electric circuits arranged on one chip, or one or a plurality of electric circuits arranged on two or more chips or devices. Good.

The main storage device 72 is a storage device that stores instructions executed by the processor 71 and various data. The information stored in the main storage device 72 is directly read by the processor 71. The auxiliary storage device 73 is a storage device other than the main storage device 72. Note that these storage devices mean arbitrary electronic components capable of storing electronic information, and may be a memory or a storage. The memory includes a volatile memory and a non-volatile memory, but either may be used. A memory for storing various data in the training device 1, for example, the storage unit 12 may be realized by the main storage device 72 or the auxiliary storage device 73. For example, at least a part of each storage unit described above may be mounted in the main storage device 72 or the auxiliary storage device 73. As another example, when an accelerator is provided, at least a part of each storage unit described above may be implemented in the memory provided in the accelerator.

The network interface 74 is an interface for connecting to the communication network 8 wirelessly or by wire. The network interface 74 may be one that conforms to the existing communication standard. The network interface 74 may exchange information with the external device 9A that is communicatively connected via the communication network 8.

The external device 9A includes, for example, a camera, a motion capture device, an output destination device, an external sensor, an input source device, and the like. Further, the external device 9A may be a device having a function of a part of the constituent elements of the training device 1. Then, the computer device 7 may send and receive a part of the processing result of the training device 1 via the communication network 8 like a cloud service.

The device interface 75 is an interface such as a USB (Universal Serial Bus) that is directly connected to the external device 9B. The external device 9B may be an external storage medium or a storage device. Each storage unit may be realized by the external device 9B.

The external device 9B may be an output device. The output device may be, for example, a display device for displaying an image, a device for outputting sound, or the like. For example, there are an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a PDP (Plasma Display Panel), a speaker, and the like, but the invention is not limited thereto.

The external device 9B may be an input device. The input device includes devices such as a keyboard, a mouse, and a touch panel, and gives information input by these devices to the computer device 7. The signal from the input device is output to the processor 71.

Aspects of the invention are not limited to the individual embodiments described above. Various additions, changes and partial deletions are possible without departing from the conceptual idea and gist of the present invention derived from the contents defined in the claims and the equivalents thereof. For example, in all the embodiments described above, the numerical values used for the description are shown as an example, and the numerical values are not limited to these.

1: Training device, 10: Input unit, 12: Storage unit, 14: Graph generation unit, 16: Forward propagation unit, 18: ID assignment unit, 20: Back propagation unit, 22: Output unit

Claims (10)

One or more memories,
One or more processors,
Equipped with
The one or more processors are
Generate a graph based on the path of error backpropagation,
In the graph, an identifier is given to each node based on the path of the error back propagation,
Performing backpropagation based on the graph and the identifier,
Training equipment. The one or more processors are
Generates a node that is a path of error back propagation corresponding to the input variable, the operation in forward propagation, and the output variable,
The training device according to claim 1. The one or more processors are
When there are a plurality of different backpropagation paths, the plurality of graphs showing the respective paths are generated.
The training device according to claim 1 or 2. The one or more processors are
For each of the graphs having the same error backpropagation path, the identifier is uniquely assigned to each node,
The training device according to any one of claims 1 to 3. The one or more processors are
The different identifiers are given to nodes belonging to the graphs having different paths for the back propagation of errors,
The training device according to claim 4. The one or more processors are
The error backpropagation is executed for the nodes to which the same identifier is added, and when the error backpropagation is completed for the identifier, the data of the graph to which the identifier is added is discarded.
The training device according to any one of claims 1 to 5. The one or more processors are
When there is a reference relationship between the nodes having different identifiers, a node having the reference relationship is generated,
Determining the order of the graphs for backpropagation based on the reference relationship,
The training device according to any one of claims 1 to 6. One or more processors generate a graph based on the path of error backpropagation,
The one or more processors assign an identifier to each node in the graph based on the error backpropagation path;
The one or more processors perform error backpropagation based on the graph and the identifier,
How to generate the model. The one or more processors store the generated model in one or more memories,
The method for generating a model according to claim 8. When executed by one or more processors,
Generate a graph based on the path of error backpropagation,
In the graph, an identifier is assigned to each node based on the path of error back propagation,
Performing backpropagation based on the graph and the identifier,
program.

Images