WO2021240636A1 - Distributed deep learning system - Google Patents

Abstract

This distributed deep learning system comprises a client terminal (1) and a cloud server (2) connected to the client terminal (1) via a network. An input layer group (200) and an output layer group (202) of a model are built on the client terminal (1), and an intermediate layer group (201) of the model is built on the cloud server (2).

Description

Distributed deep learning system

The present invention relates to a distributed deep learning system that executes deep learning in a distributed and coordinated manner on a plurality of nodes.

For deep learning, various applications have been proposed due to its high performance and wide range of application, and its performance surpasses that of previous technologies. It is known that both computational and data resources are required for deep learning to surpass existing methods.

[Computational resources]
Deep learning requires a large number of matrix operations to be performed. In the latest deep learning model, it is difficult to complete the process in a realistic time by a conventional personal computer, and a dedicated arithmetic unit called an accelerator is required. In recent years, it is common to perform deep learning / inference using a computer equipped with a plurality of these accelerators. However, this computer cannot be easily introduced by a general user because the cost of purchasing the computer is high and the power consumption is extremely high.

[About data resources]
In deep learning, supervised learning is known as one that can achieve high accuracy. Supervised learning is a method in which learning data with a label indicating a correct answer is given to a computer for learning. In supervised learning, it is difficult to achieve high accuracy when the number of learning data is insufficient, and it is considered that tens of thousands or more of data are required to train difficult processes these days.

However, there are two problems with data resources for learning. The first problem is that labeling data requires human resources with knowledge of the subject area. An example that requires specialized knowledge is the medical field. The second problem is that learning data and labels may contain personal information, so methods such as uploading data to a cloud server that may leak information to an unspecified number of people are not allowed. Is.

Previous research has proposed a method of separating deep learning into an edge device (client terminal) and a cloud server (see Non-Patent Document 1). This method focuses on the fact that the inference stage of deep learning can be performed with less computational resources and data resources than the learning stage, and that learning data cannot be reproduced from the weight of the trained model.

FIG. 21 shows the configuration of the distributed deep learning system disclosed in Non-Patent Document 1. The cloud server 100 has an initial model 1000. The cloud server 100 distributes the model 1000 to the client terminals 101-A, 101-B, and 101-C.
Each client terminal 101-A, 101-B, 101-C deploys the model provided by the cloud server 100 on the terminal. Of the client terminals 101-A, 101-B, and 101-C, the client terminal 101-C, which has sufficient computational resources and data resources, learns the model 1000-C in its own environment and uses the model 1000-C. Update.

The client terminal 101-C that has updated the model 1000-C returns the difference between the weights of each layer of the model distributed from the cloud server 100 and the updated model 1000-C to the cloud server 100.
The cloud server 100 averages the models sent from the client terminals 101-A, 101-B, and 101-C, updates its own model 1000, and renews the updated model 1000 to the client terminals 101-A, It will be distributed to 101-B and 101-C.

The method disclosed in Non-Patent Document 1 has the following effects.
(I) Client terminals with scarce computational resources do not have to learn.
(II) The learning results can be enjoyed even by the client terminal of the user who lacks data resources and lacks knowledge such as labeling.
(III) Since the client terminal and the cloud server send and receive only the weights of each layer of the model, the personal information contained in the learning data is protected.

However, the method disclosed in Non-Patent Document 1 has a problem that a model specialized for a client terminal cannot be created when the tendency of the learning data that can be acquired differs depending on the client terminal.

The present invention has been made to solve the above problems, and provides a distributed deep learning system capable of creating a model specialized for a client terminal without requiring computational resources of the client terminal as compared with the conventional method. The purpose is to do.

The distributed deep learning system of the present invention includes a client terminal and a cloud server connected to the client terminal via a network, and the client terminal is an output value as a result of inputting sample data into an input layer group of a model. The first calculation unit configured to calculate the above and the output value of the intermediate layer group calculated by the cloud server are input to the output layer group of the model and the output value of the model is calculated. A third calculation configured to calculate the error function of the weight of the output layer group based on the output value of the model and the label of the sample data when the model is trained. And a fourth calculation unit configured to calculate the weight error function of the input layer group based on the weight error function of the intermediate layer group calculated by the cloud server during training of the model. , The weight of the input layer group is updated based on the error function calculated by the fourth calculation unit, and the weight of the output layer group is updated based on the error function calculated by the third calculation unit. The first model update unit configured as described above, the first transmission unit configured to transmit the output value of the input layer group and the error function of the weight of the output layer group to the cloud server, and the first transmission unit. The cloud server includes a first receiving unit configured to receive an output value of the intermediate layer group calculated by the cloud server and an error function of the weight of the intermediate layer group, and the cloud server is a client terminal. A fifth calculation unit configured to calculate the output value of the result of inputting the output value of the input layer group calculated by the above into the intermediate layer group, and calculated by the client terminal at the time of training the model. Based on a sixth calculation unit configured to calculate the weight error function of the intermediate layer group based on the weight error function of the output layer group, and an error function calculated by the sixth calculation unit. The second model update unit configured to update the weights of the intermediate layer group, and the output value of the intermediate layer group and the error function of the weights of the intermediate layer group are transmitted to the client terminal. A second transmitter configured to receive an output value of the input layer group calculated by the client terminal and an error function of the weight of the output layer group. It is characterized by being prepared.

According to the present invention, by constructing the input layer group and the output layer group of the model on the client terminal and the intermediate layer group of the model on the cloud server, the computational resources of the client terminal are required more than the conventional method. Instead, it is possible to realize a distributed deep learning system that can create a model specialized for the client terminal.

FIG. 1 is a diagram showing a configuration of a distributed deep learning system according to a first embodiment of the present invention. FIG. 2 is a block diagram showing a configuration of a client terminal of the distributed deep learning system according to the first embodiment of the present invention. FIG. 3 is a block diagram showing a configuration of a cloud server of the distributed deep learning system according to the first embodiment of the present invention. FIG. 4 is a flowchart illustrating the inference operation of the client terminal of the distributed deep learning system according to the first embodiment of the present invention. FIG. 5 is a flowchart illustrating the inference operation of the cloud server of the distributed deep learning system according to the first embodiment of the present invention. FIG. 6 is a flowchart illustrating a learning operation of a client terminal of the distributed deep learning system according to the first embodiment of the present invention. FIG. 7 is a flowchart illustrating the learning operation of the cloud server of the distributed deep learning system according to the first embodiment of the present invention. FIG. 8 is a diagram showing a configuration of a distributed deep learning system according to a second embodiment of the present invention. FIG. 9 is a block diagram showing a configuration of a cloud server of the distributed deep learning system according to the second embodiment of the present invention. FIG. 10 is a diagram showing a configuration of a distributed deep learning system according to a third embodiment of the present invention. FIG. 11 is a block diagram showing a configuration of a client terminal of the distributed deep learning system according to the third embodiment of the present invention. FIG. 12 is a block diagram showing a configuration of a cloud server of the distributed deep learning system according to the third embodiment of the present invention. FIG. 13 is a flowchart illustrating a learning operation of a client terminal of a distributed deep learning system according to a third embodiment of the present invention. FIG. 14 is a diagram showing a configuration of a distributed deep learning system according to a fourth embodiment of the present invention. FIG. 15 is a block diagram showing a configuration of a client terminal of a distributed deep learning system according to a fourth embodiment of the present invention. FIG. 16 is a flowchart illustrating a learning operation of a client terminal of a distributed deep learning system according to a fourth embodiment of the present invention. FIG. 17 is a diagram showing a configuration of a distributed deep learning system according to a fifth embodiment of the present invention. FIG. 18 is a block diagram showing a configuration of a client terminal of the distributed deep learning system according to the fifth embodiment of the present invention. FIG. 19 is a flowchart illustrating a learning operation of a client terminal of a distributed deep learning system according to a fifth embodiment of the present invention. FIG. 20 is a block diagram showing a configuration example of a computer that realizes a client terminal according to the first to fifth embodiments of the present invention. FIG. 21 is a diagram showing a configuration of a conventional distributed deep learning system.

[First Example]
Hereinafter, examples of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing a configuration of a distributed deep learning system according to a first embodiment of the present invention. The distributed deep learning system includes a client terminal 1 and a cloud server 2 connected to the client terminal 1 via a network.

The model (neural network model) used in this embodiment is divided into three groups: an input layer group 200, an output layer group 202, and an intermediate layer group 201 between the input layer group 200 and the output layer group 202. .. The input group 200, the intermediate group 201, and the output group 202 are each composed of one or more layers. In this embodiment, the input layer group 200 and the output layer group 202 are mounted on the client terminal 1, and the intermediate layer group 201 is mounted on the cloud server 2.

FIG. 2 is a block diagram showing the configuration of the client terminal 1, and FIG. 3 is a block diagram showing the configuration of the cloud server 2. The client terminal 1 includes a storage unit 10, a data acquisition unit 11, a calculation unit 12 (first calculation unit), a transmission unit 13 (first transmission unit), and a reception unit 14 (first reception unit). , Calculation unit 15 (second calculation unit), calculation unit 16 (third calculation unit), calculation unit 17 (fourth calculation unit), and model update unit 18 (first model update unit). And have. The storage unit 10 stores the data of the input layer group 200 and the output layer group 202, and the input layer group 200 and the output layer group 202 are constructed. The construction of the input layer group 200 and the output layer group 202 is performed by the CPU (not shown) of the client terminal 1.

The cloud server 2 includes a storage unit 20, a reception unit 21 (second reception unit), a calculation unit 22 (fifth calculation unit), a transmission unit 23 (second transmission unit), and a calculation unit 24 ( A sixth calculation unit) and a model update unit 25 (second model update unit) are provided. The data of the intermediate group 201 is stored in the storage unit 20, and the intermediate group 201 is constructed. The construction of the intermediate layer group 201 is performed by the CPU (not shown) of the cloud server 2.

FIG. 4 is a flowchart explaining the inference operation of the client terminal 1 of the distributed deep learning system of this embodiment, and FIG. 5 is a flowchart explaining the inference operation of the cloud server 2.
The data acquisition unit 11 of the client terminal 1 acquires the sample data input by the user (step S100 in FIG. 4).
The calculation unit 12 of the client terminal 1 calculates the result of inputting the sample data acquired by the data acquisition unit 11 into the input layer group 200 (step S101 in FIG. 4).

The transmission unit 13 of the client terminal 1 receives the calculation result of the output value of the input layer group 200 from the calculation unit 12, and transmits this calculation result to the cloud server 2 (step S102 in FIG. 4).
The receiving unit 21 of the cloud server 2 receives the output value of the input layer group 200 from the client terminal 1 (step S200 in FIG. 5).

The calculation unit 22 of the cloud server 2 calculates the result of inputting the output value of the input layer group 200 into the intermediate layer group 201 (FIG. 5, step S201).
The transmission unit 23 of the cloud server 2 receives the calculation result of the output value of the intermediate layer group 201 from the calculation unit 22, and transmits this calculation result to the client terminal 1 (step S202 of FIG. 5).

The receiving unit 14 of the client terminal 1 receives the output value of the intermediate layer group 201 from the cloud server 2 (step S103 in FIG. 4).
The calculation unit 15 of the client terminal 1 calculates the result of inputting the output value of the intermediate layer group 201 into the output layer group 202 (FIG. 4, step S104).

In this way, the output value of the output layer group 202, that is, the output value of the model can be calculated. In the step of obtaining this output value, since the calculation is performed in order from the input layer group 200 of the model toward the output layer group 202, this step is called forward propagation.

FIG. 6 is a flowchart explaining the learning operation of the client terminal 1 of the distributed deep learning system of this embodiment, and FIG. 7 is a flowchart explaining the learning operation of the cloud server 2.
The data acquisition unit 11 of the client terminal 1 acquires sample data (learning data) with a label input by the user (step S300 in FIG. 6).

The operation of the client terminal 1 in steps S301 to S304 of FIG. 6 is as described in steps S101 to S104.
The operation of the cloud server 2 in steps S400 to S402 of FIG. 7 is as described in steps S200 to S202.

The calculation unit 16 of the client terminal 1 calculates the gradient of the error function for each of the layer weights in the output layer group 202 based on the output value of the model and the label attached to the sample data (step 6 of FIG. 6). S305).
The transmission unit 13 of the client terminal 1 receives the calculation result of the gradient of the error function from the calculation unit 16 and transmits this calculation result to the cloud server 2 (step S306 in FIG. 6).

The receiving unit 21 of the cloud server 2 receives the calculation result of the gradient of the error function from the client terminal 1 (step S403 in FIG. 7).
The calculation unit 24 of the cloud server 2 calculates the gradient of the error function for each of the weights of the layers in the intermediate layer group 201 based on the gradient of the error function received from the client terminal 1 (step S404 of FIG. 7).

The transmission unit 23 of the cloud server 2 receives the calculation result of the gradient of the error function from the calculation unit 24, and transmits this calculation result to the client terminal 1 (step S405 of FIG. 7).
The model update unit 25 of the cloud server 2 updates the weights of the layers in the intermediate layer group 201 based on the gradient of the error function calculated by the calculation unit 24 (step S406 of FIG. 7).

The receiving unit 14 of the client terminal 1 receives the calculation result of the gradient of the error function from the cloud server 2 (step S307 in FIG. 6).
The calculation unit 17 of the client terminal 1 calculates the gradient of the error function for each of the weights of the layers in the input layer group 200 based on the gradient of the error function received from the cloud server 2 (step S308 in FIG. 6).

The model update unit 18 of the client terminal 1 updates the weights of the layers in the input layer group 200 based on the gradient of the error function calculated by the calculation unit 17, and outputs the weight based on the gradient of the error function calculated by the calculation unit 16. The weights of the layers in the layer group 202 are updated (step S309 in FIG. 6).

In this way, the update of the entire model is completed. In the process of obtaining the error function, since the calculation is performed in order from the output layer group 202 of the model toward the input layer group 200, this process is called back propagation.

In this embodiment, in both the inference operation and the learning operation, the calculation of the intermediate layer group 201 is executed by the cloud server 2, so that the client terminal 1 does not require computational resources as compared with the existing method.
Further, in this embodiment, since the input layer group 200 and the output layer group 202 are learned on the client terminal 1, a model specialized for the client terminal 1 can be created.

Further, in this embodiment, the sample data is not sent to the cloud server 2, and the label is added to the sample data on the client terminal 1, so that the sample data and the information contained in the label can be protected. ..

[Second Example]
Next, a second embodiment of the present invention will be described. FIG. 8 is a diagram showing a configuration of a distributed deep learning system according to a second embodiment of the present invention. The distributed deep learning system of this embodiment is composed of client terminals 1a-A and 1a-B, and a cloud server 2a connected to client terminals 1a-A and 1a-B via a network.

In this embodiment, when there are a plurality of client terminals, each client terminal has a separate input layer group and output layer group. For example, when there are two client terminals as in the example of FIG. 8, two models are created. The input layer group 200a-A and the output layer group 202a-A of the first model are mounted on the client terminals 1a-A, and the intermediate layer group 201a of the first model is mounted on the cloud server 2a. The input layer group 200a-B and the output layer group 202a-B of the second model are mounted on the client terminals 1a-B, and the intermediate layer group 201a of the second model is mounted on the cloud server 2a. Thus, the first model and the second model share the intermediate group 201a.

Since the configurations of the client terminals 1a-A and 1a-B are the same as those of the client terminal 1 of the first embodiment, they will be described with reference to the reference numerals in FIG.
FIG. 9 is a block diagram showing the configuration of the cloud server 2a. The cloud server 2a includes a storage unit 20a, a reception unit 21, a calculation unit 22a, a transmission unit 23, a calculation unit 24a, and a model update unit 25a. The data of the intermediate group group 201a is stored in the storage unit 20a, and the intermediate group group 201a is constructed. The construction of the intermediate layer group 201a is performed by the CPU (not shown) of the cloud server 2a.

The inference operation flow of each of the client terminals 1a-A and 1a-B is the same as the operation of the client terminal 1 of the first embodiment, and the inference operation flow of the cloud server 2a is the first implementation. Since the operation is the same as that of the cloud server 2 of the example, the inference operation of this embodiment will be described with reference to the reference numerals of FIGS. 4 and 5.

The client terminals 1a-A and 1a-B execute the process of FIG. 4 for the acquired sample data, respectively.
The difference from the first embodiment in the inference operation of this embodiment is that when data arrives from the client terminals 1a-A and the client terminals 1a-B at the same time, the client terminals 1a-A and the client terminals 1a-B It is to share the middle layer group 201a by time division. That is, the cloud server 2a processes the data from the client terminals 1a-A and the data from the client terminals 1a-B in a time-division manner.

Specifically, when the calculation unit 22a of the cloud server 2a receives the output values of the input layer groups 200a-A and 200a-B from the client terminals 1a-A and 1a-B, for example, the input of the client terminals 1a-A. The result of inputting the output value of the layer group 200a-A into the intermediate layer group 201a is calculated (FIG. 5, step S201). The transmission unit 23 of the cloud server 2a returns the calculation result of the calculation unit 22a to the client terminals 1a-A that are the transmission sources of the output values of the input layer groups 200a-A (step S202 of FIG. 5).

Subsequently, the calculation unit 22a calculates the result of inputting the output value of the input layer group 200a-B of the client terminals 1a-B into the intermediate layer group 201a (step S201). The transmission unit 23 returns the calculation result of the calculation unit 22a to the client terminals 1a-B of the transmission source of the output value of the input layer group 200a-B (step S202).

The flow of learning operation of each of the client terminals 1a-A and 1a-B is the same as the operation of the client terminal 1 of the first embodiment, and the flow of the learning operation of the cloud server 2a is the first implementation. Since the operation is the same as that of the cloud server 2 of the example, the learning operation of this embodiment will be described with reference to the reference numerals of FIGS. 6 and 7.

The client terminals 1a-A and 1a-B execute the process of FIG. 6 for the acquired sample data with labels.
The difference from the first embodiment in the learning operation of this embodiment is that when data arrives from the client terminals 1a-A and the client terminals 1a-B at the same time, the cloud server 2a has the data from the client terminals 1a-A. And the data from the client terminals 1a-B is processed by time division. The time division processing of the cloud server 2a in steps S401 and S402 of FIG. 7 is the same as the processing described in steps S201 and S202 of this embodiment.

When the calculation unit 24a of the cloud server 2a receives the calculation result of the gradient of the error function from the client terminals 1a-A and 1a-B, for example, the intermediate layer is based on the gradient of the error function received from the client terminals 1a-A. The gradient of the error function is calculated for each of the layer weights in group 201a (FIG. 7 step S404). The transmission unit 23 of the cloud server 2a returns the calculation result of the calculation unit 24a to the client terminals 1a-A of the transmission source of the gradient of the error function (step S405 of FIG. 7).

Subsequently, the calculation unit 24a calculates the gradient of the error function for each of the weights of the layers in the intermediate layer group 201a based on the gradient of the error function received from the client terminals 1a-B (step S404). The transmission unit 23 returns the calculation result of the calculation unit 24a to the client terminals 1a-B of the transmission source of the gradient of the error function (step S405).

The model update unit 25a of the cloud server 2a calculates the calculation result of the calculation unit 24a based on the gradient of the error function received from the client terminals 1a-A and the calculation unit 24a based on the gradient of the error function received from the client terminals 1a-B. The average value with the result is calculated for each layer weight in the intermediate layer group 201a, and the weight of the layer in the intermediate layer group 201a is updated based on the calculated average value (FIG. 7, step S406).

When there are a plurality of client terminals as in this embodiment, for example, in the data acquisition of the client terminals 1a-A and the client terminals 1a-B, there is a possibility that the sample data is biased depending on the environment surrounding the client terminals. In the conventional method, the bias of the data of the client terminals 1a-A and the bias of the data of the client terminals 1a-B are averaged, so that the difference in the data between the client terminals may adversely affect the inference / learning. There is. On the other hand, in this embodiment, since the input layer group and the output layer group can be learned for each client terminal, the learning specialized for each client terminal can be carried out.

Further, in this embodiment, since the client terminals 1a-B can utilize the intermediate layer learned by the client terminals 1a-A, the client terminals 1a-B only need to acquire data, and the labeling cost of the client terminals 1a-B is sufficient. Can be suppressed.

[Third Example]
Next, a third embodiment of the present invention will be described. FIG. 10 is a diagram showing a configuration of a distributed deep learning system according to a third embodiment of the present invention. The distributed deep learning system of this embodiment is composed of client terminals 1b-A and 1b-B, and a cloud server 2b connected to client terminals 1b-A and 1b-B via a network.

In this embodiment, each of the plurality of client terminals has a separate input layer group and output layer group, and there are a plurality of sample data types (for example, image data and audio data), and each client terminal is a sample. It has an input layer group for each type of data.

For example, when there are two client terminals and two types of sample data as in the example of FIG. 10, four models are created. The input layer group 200b-A-α and the output layer group 202b-A for the sample data α (for example, image data) are mounted on the client terminal 1b-A, and the intermediate layer group 201b of the first model is mounted. Is implemented in the cloud server 2b. The input layer group 200b-A-β and the output layer group 202b-A of the second model for sample data β (for example, voice data) are mounted on the client terminal 1b-A, and the intermediate layer group 201b of the second model is implemented. Is implemented in the cloud server 2b. The first model and the second model share the intermediate group 201b and the output group 202b-A.

The input layer group 200b-B-α and the output layer group 202b-B of the third model for the data α are mounted on the client terminal 1b-B, and the intermediate layer group 201b of the third model is mounted on the cloud server 2b. Has been done. The input layer group 200b-B-β and the output layer group 202b-B of the fourth model for data β are mounted on the client terminal 1b-B, and the intermediate layer group 201b of the fourth model is mounted on the cloud server 2b. Has been done. The third model and the fourth model share the intermediate group 201b and the output group 202b-B.

FIG. 11 is a block diagram showing the configuration of the client terminals 1b-A and 1b-B, and FIG. 12 is a block diagram showing the configuration of the cloud server 2b. The client terminals 1b-A and 1b-B have a storage unit 10b, a data acquisition unit 11, a calculation unit 12b, 15b, 16b, 17b, a transmission unit 13, a reception unit 14, and a model update unit 18b, respectively. It includes a transmitting unit 19 and a receiving unit 30.

The storage unit 10b of the client terminal 1b-A stores the data of the input layer group 200b-A-α, 200b-A-β and the output layer group 202b-A, and the input layer group 200b-A-α, A 200b-A-β and an output group 202b-A are constructed. The construction of the input layer group 200b-A-α, 200b-A-β and the output layer group 202b-A is performed by the CPU (not shown) of the client terminal 1b-A.

The storage unit 10b of the client terminal 1b-B stores the data of the input layer group 200b-B-α, 200b-B-β and the output layer group 202b-B, and the input layer group 200b-B-α, A 200b-B-β and an output group 202b-B are constructed. The construction of the input layer group 200b-B-α, 200b-B-β and the output layer group 202b-B is performed by the CPU (not shown) of the client terminal 1b-B.

The cloud server 2b includes a storage unit 20b, a reception unit 21, calculation units 22b and 24b, a transmission unit 23, and a model update unit 25b. The data of the intermediate layer group 201b is stored in the storage unit 20b, and the intermediate layer group 201b is constructed. The construction of the intermediate layer group 201b is performed by the CPU (not shown) of the cloud server 2b.

The inference operation flow of each of the client terminals 1b-A and 1b-B is the same as the operation of the client terminal 1 of the first embodiment, and the inference operation flow of the cloud server 2b is the first implementation. Since the operation is the same as that of the cloud server 2 of the example, the inference operation of this embodiment will be described with reference to the reference numerals of FIGS. 4 and 5.

The client terminals 1b-A and 1b-B each execute the processing of FIG. 4 for the acquired sample data.
The calculation unit 12b of the client terminal 1b-A calculates the result of inputting the data α acquired by the data acquisition unit 11 into the input layer group 200b-A-α (step S101 in FIG. 4). The transmission unit 13 of the client terminal 1b-A receives the calculation result of the output value of the input layer group 200b-A-α from the calculation unit 12b, and transmits this calculation result to the cloud server 2b (step S102 in FIG. 4).

The calculation unit 12b of the client terminal 1b-A calculates the result of inputting the data β acquired by the data acquisition unit 11 into the input layer group 200b-A-β (step S101). The transmission unit 13 of the client terminal 1b-A transmits the calculation result of the output value of the input layer group 200b-A-β to the cloud server 2b (step S102).

The calculation unit 12b of the client terminal 1b-B calculates the result of inputting the data α acquired by the data acquisition unit 11 into the input layer group 200b-B-α (step S101). The transmission unit 13 of the client terminal 1b-B transmits the calculation result of the output value of the input layer group 200b-B-α to the cloud server 2b (step S102).

The calculation unit 12b of the client terminal 1b-B calculates the result of inputting the data β acquired by the data acquisition unit 11 into the input layer group 200b-B-β (step S101). The transmission unit 13 of the client terminal 1b-B transmits the output value of the input layer group 200b-B-β to the cloud server 2b (step S102).
The type of data can be easily identified by, for example, an identifier attached to the data.

Similar to the second embodiment, the cloud server 2b processes the data from the client terminal 1b-A and the data from the client terminal 1b-B in a time-division manner. The cloud server 2b inputs the calculation result of the output value of the input layer group 200b-A-α having the data α as the input, the calculation result of the output value of the input layer group 200b-A-β having the data β as the input, and the data α. Processing of steps S201 and S202 of FIG. 5 for the calculation result of the output value of the input layer group 200b-B-α to be input and the calculation result of the output value of the input layer group 200b-B-β to be input to the data β, respectively. Is executed in time divisions.

The calculation unit 15b of the client terminals 1b-A and 1b-B calculates the result of inputting the output value of the intermediate layer group 201b received from the cloud server 2b into the output layer groups 202b-A and 202b-B, respectively (FIG. 4). Step S104).

At this time, the data received by the client terminal 1b-A includes the output value of the intermediate layer group 201b calculated from the output value of the input layer group 200b-A-α and the output value of the input layer group 200b-A-β. Since there are two types of output values of the intermediate layer group 201b calculated from, the process of step S104 is executed for each of these two types of output values. Similarly, the data received by the client terminal 1b-B includes the output value of the intermediate layer group 201b calculated from the output value of the input layer group 200b-B-α and the output value of the input layer group 200b-B-β. Since there are two types of output values of the intermediate layer group 201b calculated from, the process of step S104 is executed for each of these two types of output values.

When the client terminal 1b-A acquires only the data α and the client terminal 1b-B acquires only the data β, the data acquisition unit 11 of the client terminal 1b-A that could not acquire the data β may acquire the data β. Complementary data (for example, zero value, average value of past data, etc.) may be generated. The data acquisition unit 11 of the client terminal 1b-B that could not acquire the data α may generate complementary data of the data α.

FIG. 13 is a flowchart illustrating the learning operation of the client terminals 1b-A and 1b-B of the distributed deep learning system of this embodiment. Since the flow of the learning operation of the cloud server 2b is the same as the operation of the cloud server 2 of the first embodiment, the reference numerals of FIG. 7 will be used for description.

The client terminals 1b-A and 1b-B each execute the processing of FIG. 13 for the acquired sample data with labels.
The processing of the client terminals 1b-A and 1b-B in steps S500 to S504 of FIG. 13 is the same as the processing of steps S100 to S104 described in this embodiment.

The calculation unit 16b and the transmission unit 13 of the client terminal 1b-A execute the same processing as in steps S305 and S306 of FIG. 6 in a time-division manner for each of the data α and the data β (steps S505 and S506 in FIG. 13). Specifically, the calculation unit 16b calculates the gradient of the error function for each of the weights of the layers in the output layer group 202b-A based on the output value of the first model and the label of the data α, and the second. The gradient of the error function is calculated for each of the layer weights in the output layer group 202b-A based on the output value of the model and the label of the data β.

The calculation unit 16b and the transmission unit 13 of the client terminal 1b-B execute the same processing as in steps S305 and S306 in a time-division manner for each of the data α and the data β (steps S505 and S506). Specifically, the calculation unit 16b calculates the gradient of the error function for each of the weights of the layers in the output layer group 202b-B based on the output value of the third model and the label of the data α, and the fourth The gradient of the error function is calculated for each of the layer weights in the output layer group 202b-A based on the output value of the model and the label of the data β.

The time division processing of the cloud server 2b in steps S401 and S402 of FIG. 7 is the same as the processing described in steps S201 and S202 of this embodiment.
The calculation unit 24b and the transmission unit 23 of the cloud server 2b perform the processing of steps S404 and S405 in FIG. 7 on the client terminal 1b-A based on the output value of the first model for the data α and the label of the data α. The gradient of the error function calculated by the client terminal 1b-A based on the output value of the second model for the data β and the label of the data β, the gradient of the error function calculated by the client terminal 1b-A, the output of the third model for the data α. The gradient of the error function calculated by the client terminal 1b-B based on the value and the label of this data α, the output value of the fourth model for the data β and the label of this data β are used by the client terminal 1b-B. Perform time divisions for each of the calculated error function gradients.

The model update unit 25b of the cloud server 2b calculates the calculation result of the calculation unit 24b based on the gradient of the error function received from the client terminals 1b-A and the calculation unit 24b based on the gradient of the error function received from the client terminals 1b-B. The average value with the result is calculated for each layer weight in the intermediate layer group 201b, and the weight of the layer in the intermediate layer group 201b is updated based on the calculated average value (FIG. 7, step S406).

However, if the client terminals 1b-A and 1b-B cannot acquire the sample data, or if the sample data is not labeled, the error function cannot be calculated.
The calculation result of the calculation unit 24b includes the calculation result based on the gradient of the error function calculated by the client terminal 1b-A using the output value of the first model and the client terminal 1b using the output value of the second model. -The calculation result based on the gradient of the error function calculated by A, the calculation result based on the gradient of the error function calculated by the client terminal 1b-B using the output value of the third model, and the output value of the fourth model. There are four types of calculation results based on the gradient of the error function calculated by the client terminal 1b-B using.

For example, it is assumed that the client terminal 1b-A could not acquire the data β, or the data β acquired by the client terminal 1b-A was not labeled. Further, it is assumed that the client terminal 1b-B could not acquire the data α, or the data α acquired by the client terminal 1b-B was not labeled. In this case, the calculation unit 16b of the client terminal 1b-A cannot calculate the gradient of the error function for the weights of the layers in the output layer group 202b-A using the output value of the second model, and the client terminal cannot calculate the gradient of the error function. The calculation unit 16b of 1b-B cannot calculate the gradient of the error function for the weights of the layers in the output layer group 202b-B using the output value of the third model.

Therefore, the calculation unit 24b of the cloud server 2b is an error function for the weight of the layer in the intermediate layer group 201b based on the result that the client terminal 1b-A should have calculated using the output value of the second model. The gradient cannot be calculated, and the gradient of the error function is calculated for the weights of the layers in the intermediate layer group 201b based on the result that the client terminal 1b-B should have calculated using the output value of the third model. It cannot be calculated.

The calculation unit 17b of the client terminal 1b-A is based on the calculation result of the cloud server 2b based on the gradient of the error function calculated by the client terminal 1b-A using the output value of the first model, and the input layer group 200b-. The gradient of the error function is calculated for each of the layer weights in A-α (FIG. 13, step S508). Further, the calculation unit 17b of the client terminal 1b-A is an input layer group based on the calculation result of the cloud server 2b based on the gradient of the error function calculated by the client terminal 1b-A using the output value of the second model. The gradient of the error function is calculated for each of the layer weights in 200b-A-β (step S508).

The calculation unit 17b of the client terminal 1b-B uses the output value of the third model to calculate the input layer group 200b-based on the calculation result of the cloud server 2b based on the gradient of the error function calculated by the client terminal 1b-B. The gradient of the error function is calculated for each of the layer weights in B-α (step S508). Further, the calculation unit 17b of the client terminal 1b-B is an input layer group based on the calculation result of the cloud server 2b based on the gradient of the error function calculated by the client terminal 1b-B using the output value of the fourth model. The gradient of the error function is calculated for each of the layer weights in 200b-B-β (step S508).

The model update unit 18b of the client terminal 1b-A is a layer in the input layer group 200b-A-α based on the gradient of the error function calculated by the calculation unit 17b for the weights of the layers in the input group group 200b-A-α. The weights of the layers in the input layer group 200b-A-β are updated based on the gradient of the error function calculated by the calculation unit 17b for the weights of the layers in the input group group 200b-A-β. Further, in the model update unit 18b of the client terminal 1b-A, the calculation unit 16b calculates the gradient of the error function based on the output value of the first model and the label of the data α, and the calculation unit 16b is the first. The average value of the output value of the model 2 and the calculation result of the gradient of the error function calculated based on the label of the data β is calculated for each layer weight in the output layer group 202b-A, and the calculated average value is calculated. The weights of the layers in the output layer group 202b-A are updated based on (FIG. 13, step S509).

The model update unit 18b of the client terminal 1b-B is a layer in the input layer group 200b-B-α based on the gradient of the error function calculated by the calculation unit 17b for the weights of the layers in the input group group 200b-B-α. The weights of the layers in the input layer group 200b-B-β are updated based on the gradient of the error function calculated by the calculation unit 17b for the weights of the layers in the input group group 200b-B-β. Further, in the model update unit 18b of the client terminal 1b-B, the calculation unit 16b calculates the gradient of the error function based on the output value of the third model and the label of the data α, and the calculation unit 16b is the first. The average value of the output value of the model 4 and the calculation result of the gradient of the error function calculated based on the label of the data β is calculated for each layer weight in the output layer group 202b-B, and the calculated average value is calculated. The weights of the layers in the output layer group 202b-B are updated based on (step S509).

However, if the sample data cannot be acquired or the sample data is not labeled, the client terminals 1b-A and 1b-B input using the calculation result of the error function in their own device. The strata cannot be updated.

For example, it is assumed that the client terminal 1b-A could not acquire the data β, or the data β acquired by the client terminal 1b-A was not labeled. Further, it is assumed that the client terminal 1b-B could not acquire the data α, or the data α acquired by the client terminal 1b-B was not labeled. In this case, the calculation unit 16b of the client terminal 1b-A cannot calculate the gradient of the error function for the weights of the layers in the output layer group 202b-A using the output value of the second model, and the client terminal cannot calculate the gradient of the error function. The calculation unit 16b of 1b-B cannot calculate the gradient of the error function for the weights of the layers in the output layer group 202b-B using the output value of the third model.

Therefore, the model update unit 18b of the client terminal 1b-A cannot use the result that the calculation unit 16b should have calculated using the output value of the second model to update the output layer group 202b-A. .. Similarly, the model update unit 18b of the client terminal 1b-B may use the result that the calculation unit 16b should have calculated using the output value of the third model to update the output layer group 202b-B. Can not.

Further, the calculation unit 17b of the client terminal 1b-A cannot calculate the gradient of the error function with respect to the weight of the layer in the input layer group 200b-A-β, and the calculation unit 17b of the client terminal 1b-B does not. The gradient of the error function cannot be calculated for the weights of the layers in the input layer group 200b-B-α. Therefore, the model update unit 18b of the client terminal 1b-A cannot use the result that the calculation unit 17b should have calculated to update the input layer group 200b-A-β. Similarly, the model update unit 18b of the client terminal 1b-B cannot use the result that the calculation unit 17b should have calculated to update the input layer group 200b-B-α. Therefore, in order to update the input layer groups 200b-A-β and 200b-B-α, it is necessary to transmit the weight from the client terminal that can acquire the labeled data α and β.

Specifically, in the transmission unit 19 of the client terminal 1b-A, the model update unit 18b of the client terminal 1b-A could not update the input layer group 200b-A-β, so that the transmission unit 19 is included in the input layer group for data β. The update result of the layer weight is requested from another client terminal (step S510 in FIG. 13).

In the transmission unit 19 of the client terminal 1b-B, since the model update unit 18b of the client terminal 1b-B could not update the input layer group 200b-B-α, the weight of the layer in the input layer group for the data α is updated. The result is requested from another client terminal (step S510).

The receiving unit 30 of the client terminal 1b-A receives the request from the client terminal 1b-B (step S511 in FIG. 13). The transmission unit 19 of the client terminal 1b-A transmits the update result of the weight of the layer in the input layer group 200b-A-α to the client terminal 1b-B in response to the request from the client terminal 1b-B (FIG. 13). Step S512).

The receiving unit 30 of the client terminal 1b-B receives the request from the client terminal 1b-A (step S511). The transmission unit 19 of the client terminal 1b-B transmits the update result of the weight of the layer in the input layer group 200b-B-β to the client terminal 1b-A in response to the request from the client terminal 1b-A (step S512). ).

The receiving unit 30 of the client terminal 1b-A receives the update result of the weight of the layer in the input layer group 200b-B-β from the client terminal 1b-B (step S513 in FIG. 13). The model update unit 18b of the client terminal 1b-A updates the weights of the layers in the input layer group 200b-A-β by using the update result of the weights of the layers in the input layer group 200b-B-β (FIG. 13). Step S514).

The receiving unit 30 of the client terminal 1b-B receives the update result of the weight of the layer in the input layer group 200b-A-α from the client terminal 1b-A (step S513). The model update unit 18b of the client terminal 1b-B updates the weights of the layers in the input layer group 200b-B-α by using the update result of the weights of the layers in the input layer group 200b-A-α (step S514). ).
Needless to say, if the client terminals 1b-A and 1b-B can acquire the labeled data α and β, the processing of steps S510 to S514 becomes unnecessary.

In deep learning, there are cases where it is desired to integrate data acquired by different methods for inference / learning, but data α can only be acquired from client terminal 1b-A, and data β can only be acquired from client terminal 1b-B. Sometimes. Even in such a case, inference can be realized by exchanging the sample data acquired by the client terminals 1b-A and 1b-B, respectively, and the personal information included in the sample data can be protected.

If a multifaceted model can be constructed as in this example, even a client terminal that can acquire only one of the sample data α and β can perform inference with a certain degree of accuracy, which is useful for initial decision making. be able to. In addition, even for a client terminal that can acquire only one of the sample data α and β, the model can be trained by sharing the calculation result with the client terminal that can acquire the sample data, and the sample data can be used. It is possible to protect the personal information contained.

[Fourth Example]
Next, a fourth embodiment of the present invention will be described. FIG. 14 is a diagram showing a configuration of a distributed deep learning system according to a fourth embodiment of the present invention. The distributed deep learning system of this embodiment is composed of client terminals 1c-A and 1c-B, client terminals 1c-A and 1c-B, and a cloud server 2c connected via a network.

Similar to the third embodiment, in this embodiment, each of the plurality of client terminals has a separate input layer group and output layer group, and there are a plurality of types of sample data, and each client terminal has a different type. It has an input layer group and an output layer group for each type of sample data.

For example, when there are two client terminals and two types of sample data as in the example of FIG. 14, four models are created. The input layer group 200c-A-α and the output layer group 202c-A-α for the sample data α (for example, image data) are mounted on the client terminal 1c-A, and the intermediate layer of the first model. The group 201c is implemented in the cloud server 2c. The input layer group 200c-A-β and the output layer group 202c-A-β of the second model for sample data β (for example, voice data) are mounted on the client terminal 1c-A, and the intermediate layer of the second model. The group 201c is implemented in the cloud server 2c.

The input layer group 200c-B-α of the third model and the output layer group 202c-B-α for the data α are mounted on the client terminal 1c-B, and the intermediate layer group 201c of the third model is the cloud server 2c. It is implemented in. The input layer group 200c-B-β and the output layer group 202c-B-β of the fourth model for data β are mounted on the client terminal 1c-B, and the intermediate layer group 201c of the fourth model is the cloud server 2c. It is implemented in. The first to fourth models share the intermediate group 201c.

FIG. 15 is a block diagram showing the configurations of client terminals 1c-A and 1c-B, and the same configurations as those in FIG. 11 are designated by the same reference numerals. The client terminals 1c-A and 1c-B have a storage unit 10c, a data acquisition unit 11, a calculation unit 12b, 15c, 16c, 17b, a transmission unit 13, a reception unit 14, and a model update unit 18c, respectively. It includes a transmitting unit 19c and a receiving unit 30c.

The storage unit 10c of the client terminal 1c-A stores data of the input layer groups 200c-A-α, 200c-A-β and the output layer groups 202c-A-α, 202c-A-β, and inputs the data. Group groups 200c-A-α, 200c-A-β and output group groups 202c-A-α, 202c-A-β are constructed. The construction of the input layer groups 200c-A-α, 200c-A-β and the output layer groups 202c-A-α, 202c-A-β is performed by the CPU (not shown) of the client terminal 1c-A.

The storage unit 10c of the client terminal 1c-B stores data of the input layer groups 200c-B-α, 200c-B-β and the output layer groups 202c-B-α, 202c-B-β, and inputs the data. Group groups 200c-B-α, 200c-B-β and output group groups 202c-B-α, 202c-B-β are constructed. The construction of the input layer groups 200c-B-α, 200c-B-β and the output layer groups 202c-B-α, 202c-B-β is performed by the CPU (not shown) of the client terminal 1c-B.
Since the configuration of the cloud server 2c is the same as that of the cloud server 2b of the third embodiment, the reference numerals in FIG. 12 will be used for description.

The client terminals 1c-A and 1c-B each execute the process shown in FIG. 4 for the acquired sample data. The processes of steps S100 to S102 are the same as the processes described in the third embodiment. The inference operation of the cloud server 2c is the same as that of the third embodiment.

The calculation unit 15c of the client terminal 1c-A receives the output value of the intermediate layer group 201c calculated from the output value of the input layer group 200c-A-α from the cloud server 2c via the reception unit 14, and this intermediate layer group. The result of inputting the output value of 201c into the output layer group 202c-A-α is calculated (FIG. 4, step S104). Further, the calculation unit 15c of the client terminal 1c-A receives the output value of the intermediate layer group 201c calculated from the output value of the input layer group 200c-A-β from the cloud server 2c, and the output value of the intermediate layer group 201c. Is input to the output layer group 202c-A-β, and the result is calculated (step S104).

The calculation unit 15c of the client terminal 1c-B receives the output value of the intermediate layer group 201c calculated from the output value of the input layer group 200c-B-α from the cloud server 2c via the reception unit 14, and the intermediate layer group The result of inputting the output value of 201c into the output layer group 202c-B-α is calculated (step S104). Further, the calculation unit 15c of the client terminal 1c-B receives the output value of the intermediate layer group 201c calculated from the output value of the input layer group 200c-B-β from the cloud server 2c, and the output value of the intermediate layer group 201c. Is input to the output layer group 202c-B-β, and the result is calculated (step S104).

FIG. 16 is a flowchart illustrating the learning operation of the client terminals 1c-A and 1c-B of the distributed deep learning system of this embodiment. Since the flow of the learning operation of the cloud server 2c is the same as the operation of the cloud server 2 of the first embodiment, the reference numerals of FIG. 7 will be used for description.

The client terminals 1c-A and 1c-B each execute the processing of FIG. 16 for the acquired sample data with labels.
The processing of the client terminals 1c-A and 1c-B in steps S600 to S604 of FIG. 16 is the same as the processing of steps S100 to S104 described in this embodiment.

The calculation unit 16c and the transmission unit 13 of the client terminal 1c-A execute the same processing as in steps S305 and S306 of FIG. 6 in a time-division manner for each of the data α and the data β (steps S605 and S606 of FIG. 16). Specifically, the calculation unit 16c calculates the gradient of the error function based on the output value of the first model and the label of the data α for each of the weights of the layers in the output layer group 202c-A-α. The gradient of the error function is calculated for each of the layer weights in the output layer group 202c-A-β based on the output value of the second model and the label of the data β.

The calculation unit 16c and the transmission unit 13 of the client terminal 1c-B execute the same processing as in steps S305 and S306 in a time-division manner for each of the data α and the data β (steps S605 and S606). Specifically, the calculation unit 16c calculates the gradient of the error function based on the output value of the third model and the label of the data α for each of the weights of the layers in the output layer group 202c-B-α. The gradient of the error function is calculated for each of the layer weights in the output layer group 202c-A-β based on the output value of the fourth model and the label of the data β.

The processing of the cloud server 2c in steps S400 to S406 of FIG. 7 is the same as the processing described in the third embodiment.
The processing of the client terminals 1c-A and 1c-B in steps S607 and S608 of FIG. 16 is the same as the processing of steps S507 and 508 described in the third embodiment.

The model update unit 18c of the client terminal 1c-A is a layer in the input layer group 200c-A-α based on the gradient of the error function calculated by the calculation unit 17b for the weights of the layers in the input group group 200c-A-α. The weights of the layers in the input layer group 200c-A-β are updated based on the gradient of the error function calculated by the calculation unit 17b for the weights of the layers in the input group group 200c-A-β. Further, the model update unit 18c of the client terminal 1c-A has the output layer group 202c-A- based on the gradient of the error function calculated by the calculation unit 16c based on the output value of the first model and the label of the data α. The layers in the output layer group 202c-A-β are updated based on the gradient of the error function calculated by the calculation unit 16c based on the output value of the second model and the label of the data β by updating the weights of the layers in α. The weight of is updated (FIG. 16 step S609).

The model update unit 18c of the client terminal 1c-B is a layer in the input layer group 200c-B-α based on the gradient of the error function calculated by the calculation unit 17b for the weights of the layers in the input group group 200c-B-α. The weights of the layers in the input layer group 200c-B-β are updated based on the gradient of the error function calculated by the calculation unit 17b for the weights of the layers in the input group group 200c-B-β. Further, the model update unit 18c of the client terminal 1c-B has an output layer group 202c-B- based on the gradient of the error function calculated by the calculation unit 16c based on the output value of the third model and the label of the data α. The layers in the output layer group 202c-B-β are updated based on the gradient of the error function calculated by the calculation unit 16c based on the output value of the fourth model and the label of the data β by updating the weights of the layers in α. The weight of is updated (step S609).

However, if the sample data cannot be acquired or the sample data is not labeled, the client terminals 1c-A and 1c-B input using the calculation result of the error function in their own device. It is not possible to update layers and output layers.

For example, it is assumed that the client terminal 1c-A could not acquire the data β, or the data β acquired by the client terminal 1c-A was not labeled. Further, it is assumed that the client terminal 1c-B could not acquire the data α, or the data α acquired by the client terminal 1c-B was not labeled. In this case, the calculation unit 16c of the client terminal 1c-A cannot calculate the gradient of the error function with respect to the weight of the layer in the output layer group 202c-A-β, and the calculation unit 16c of the client terminal 1c-B cannot calculate. , The gradient of the error function cannot be calculated for the weights of the layers in the output layer group 202c-B-α. The calculation unit 17b of the client terminal 1c-A cannot calculate the gradient of the error function for the weight of the layer in the input layer group 200c-A-β, and the calculation unit 17b of the client terminal 1c-B cannot calculate the gradient of the error function. The gradient of the error function cannot be calculated for the weights of the layers in the group 200c-B-α.

Therefore, in order to update the input layer groups 200c-A-β, 200c-B-α and the output layer groups 202c-A-β, 202c-B-α, the labeled data α, β could be acquired. It is necessary to send the weight from the client terminal.

Specifically, in the transmission unit 19c of the client terminal 1c-A, the model update unit 18c of the client terminal 1c-A could not update the input layer group 200c-A-β and the output layer group 202c-A-β. , The update result of the weight of the layer in the input layer group for data β and the update result of the weight of the layer in the output layer group for data β are requested from other client terminals (FIG. 16 step S610).

The transmission unit 19c of the client terminal 1c-B is for data α because the model update unit 18c of the client terminal 1c-B could not update the input layer group 200c-B-α and the output layer group 202c-B-α. The update result of the layer weight in the input layer group and the update result of the layer weight in the output layer group for the data α are requested from the other client terminal (step S610).

The receiving unit 30c of the client terminal 1c-A receives the request from the client terminal 1c-B (step S611 in FIG. 16). The transmission unit 19c of the client terminal 1c-A updates the weight of the layer in the input layer group 200c-A-α and the layer in the output layer group 202c-A-α in response to the request from the client terminal 1c-B. The update result of the weight of is transmitted to the client terminal 1c-B (step S612 in FIG. 16).

The receiving unit 30c of the client terminal 1c-B receives the request from the client terminal 1c-A (step S611). The transmission unit 19c of the client terminal 1c-B updates the weight of the layer in the input layer group 200c-B-β and the layer in the output layer group 202c-B-β in response to the request from the client terminal 1c-A. The update result of the weight of is transmitted to the client terminal 1c-A (step S612).

The receiving unit 30c of the client terminal 1c-A updates the weight of the layer in the input layer group 200c-B-β and the update result of the weight of the layer in the output layer group 202c-B-β from the client terminal 1c-B. And is received (FIG. 16 step S613). The model update unit 18c of the client terminal 1c-A updates the weights of the layers in the input layer group 200c-A-β by using the update result of the weights of the layers in the input layer group 200c-B-β, and the output layer. The weight of the layer in the output layer group 202c-A-β is updated using the update result of the weight of the layer in the group 202c-B-β (step S614 of FIG. 16).

The receiving unit 30c of the client terminal 1c-B updates the weight of the layer in the input layer group 200c-A-α and the update result of the weight of the layer in the output layer group 202c-A-α from the client terminal 1c-A. And is received (step S613). The model update unit 18c of the client terminal 1c-B updates the weights of the layers in the input layer group 200c-B-α by using the update result of the weights of the layers in the input layer group 200c-A-α, and the output layer. The weight of the layer in the output layer group 202c-B-α is updated using the update result of the weight of the layer in the group 202c-A-α (step S614).
Needless to say, if the client terminals 1c-A and 1c-B can acquire the labeled data α and β, the processing of steps S610 to S614 becomes unnecessary.

Normally, in order to deep-learn various data and various tasks, an expert who can label each data is required. By using this embodiment, even when each data and a labelable expert exist in a distant place, the data and the label can be processed without communication. Therefore, personal information can be protected more effectively.

[Fifth Example]
The weight communication method in the third and fourth embodiments includes a centralized type and a distributed type. A centralized configuration is shown in FIG. The distributed deep learning system of this embodiment is composed of client terminals 1d-A and 1d-B, a cloud server 2c, and a storage server 3 connected to client terminals 1d-A and 1d-B via a network. NS.

FIG. 18 is a block diagram showing the configurations of client terminals 1d-A and 1d-B, and the same configurations as those in FIG. 15 are designated by the same reference numerals. The client terminals 1d-A and 1d-B have a storage unit 10c, a data acquisition unit 11, a calculation unit 12b, 15c, 16c, 17b, a transmission unit 13, a reception unit 14, and a model update unit 18c, respectively. It includes a writing unit 31 and a reading unit 32.

The inference operation of the client terminals 1d-A and 1d-B and the inference / learning operation of the cloud server 2c are the same as those in the fourth embodiment.
FIG. 19 is a flowchart illustrating the learning operation of the client terminals 1d-A and 1d-B. The processing of steps S600 to S609 of FIG. 19 is the same as that of the fourth embodiment.

The writing unit 31 of the client terminal 1d-A has the update result of the layer weight in the input layer group 200c-A-α, 200c-A-β and the output layer group 202c-A-α, 202c-A-β. The update result of the weight of the layer is written to the storage server 3 via the network (step S615 in FIG. 19).

The writing unit 31 of the client terminal 1d-B has the update result of the layer weight in the input layer group 200c-B-α, 200c-B-β and the output layer group 202c-B-α, 202c-B-β. The update result of the weight of the layer is written to the storage server 3 via the network (step S615).

However, the client terminals 1d-A and 1d-B may write at least a part of the update result to the storage server 3 when the sample data cannot be acquired or the sample data is not labeled. Can not.

For example, it is assumed that the client terminal 1d-A could not acquire the data β, or the data β acquired by the client terminal 1d-A was not labeled. Further, it is assumed that the client terminal 1d-B could not acquire the data α, or the data α acquired by the client terminal 1d-B was not labeled. In this case, the client terminal 1d-A cannot write the update results of the input layer group 200c-A-β and the output layer group 202c-A-β to the storage server 3. The client terminal 1d-B cannot write the update results of the input layer group 200c-B-α and the output layer group 202c-B-α to the storage server 3.

The reading unit 32 of the client terminal 1d-A is for data β because the model updating unit 18c of the client terminal 1d-A could not update the input layer group 200c-A-β and the output layer group 202c-A-β. The update result of the layer weight in the input layer group and the update result of the layer weight in the output layer group for data β are read from the storage server 3 (FIG. 19, step S616).

The reading unit 32 of the client terminal 1d-B is for data α because the model updating unit 18c of the client terminal 1d-B could not update the input layer group 200c-B-α and the output layer group 202c-B-α. The update result of the layer weight in the input layer group and the update result of the layer weight in the output layer group for data α are read from the storage server 3 (step S616).

The model update unit 18c of the client terminal 1d-A updates the weights of the layers in the input layer group 200c-A-β by using the update result of the weights of the layers in the input layer group 200c-B-β, and the output layer. The weight of the layer in the output layer group 202c-A-β is updated using the update result of the weight of the layer in the group 202c-B-β (FIG. 19, step S617).

The model update unit 18c of the client terminal 1d-B updates the weights of the layers in the input layer group 200c-B-α by using the update result of the weights of the layers in the input layer group 200c-A-α, and the output layer. The weight of the layer in the output layer group 202c-B-α is updated using the update result of the weight of the layer in the group 202c-A-α (step S617).

In this way, the client terminal that could not acquire the sample data can update the input layer group and the output layer group by reading the update result of the client terminal that could acquire the sample data from the storage server 3.

The storage server 3 stores weight data for each type of input layer group (for each type of data), and also stores weight data for each type of output layer group (for each type of data).
When writing the weight update result to the storage server 3, the writing unit 31 of the client terminals 1d-A and 1d-B overwrites the weight if the same type of weight is already stored in the storage server 3. You may. Further, the writing unit 31 may calculate the average value of the accumulated weights of the same type and the newly written weights, and overwrite the accumulated weights with this average value.

The distributed configuration and operation are as described in the third and fourth embodiments. In this case, the processes of steps S510 to S514 and S610 to S614 are performed by one-to-one communication between the two client terminals.

In this embodiment, even a client terminal that cannot acquire labeled sample data can infer / learn various types of data and tasks. At that time, since the weight is exchanged between the client terminals, the personal information contained in the data can be protected.

In the case of the centralized type, the system operates even if the number of client terminals is increased or decreased, so it is robust against communication failures.
In the case of the distributed type, since the client terminals communicate with each other, the communication load is small and the delay is also small. However, as the network becomes complicated, the cost increases and the robustness against communication failure decreases.

In this embodiment, the centralized configuration is applied to the fourth embodiment, but it goes without saying that the configuration may be applied to the third embodiment. In this case, the update result is written to the storage server and read from the storage server only for the input layer group.
Further, in the second to fifth embodiments, the number of client terminals is two, but it goes without saying that there may be three or more client terminals.

Each of the client terminals described in the first to fifth embodiments can be realized by a computer provided with a CPU (Central Prodessing Unit), a storage device, and an interface, and a program for controlling these hardware resources. An example of the configuration of this computer is shown in FIG.

The computer includes a CPU 300, a storage device 301, and an interface device (I / F) 302. A network or the like is connected to the I / F 302. In such a computer, the program for realizing the present invention is stored in the storage device 301. Each CPU 300 of the client terminal executes the process described in the first to fifth embodiments according to the program stored in each storage device 301. The cloud server and the storage server can also be realized by a computer having the same configuration as in FIG.

The present invention can be applied to a distributed deep learning system that executes deep learning in a distributed and coordinated manner on a client terminal and a cloud server.

1,1a, 1b, 1c, 1d ... Client terminal, 2,2a, 2b, 2c ... Cloud server, 3 ... Storage server, 10,20 ... Storage unit, 11 ... Data acquisition unit, 12, 12b, 15, 15b, 15c, 16, 16b, 16c, 17, 17b, 22, 22a, 22b, 24, 24a, 24b ... Calculation unit, 13, 19, 19c, 23 ... Transmission unit, 14, 21, 30, 30c ... Receiver unit, 18 , 18b, 25, 25a, 25b ... Model update unit, 31 ... Writing unit, 32 ... Reading unit, 200, 200a, 200b, 200c ... Input layer group, 201, 201a, 201b, 201c ... Intermediate layer group, 202, 202a, 202b, 202c ... Output layer group.

Claims (8)

1. With the client terminal
   A cloud server connected to the client terminal via a network is provided.
   The client terminal is
   A first calculator configured to calculate the output value of the result of inputting sample data into the input layer of the model,
   A second calculation unit configured to input the output value of the intermediate layer group calculated by the cloud server into the output layer group of the model and calculate the output value of the model.
   A third calculation unit configured to calculate the error function of the weights of the output layer group based on the output value of the model and the label of the sample data when training the model.
   A fourth calculation unit configured to calculate the error function of the weight of the input group based on the error function of the weight of the intermediate group calculated by the cloud server at the time of training the model.
   Update the weights of the input layer group based on the error function calculated by the fourth calculation unit, and update the weights of the output layer group based on the error function calculated by the third calculation unit. The first model update unit configured in
   A first transmission unit configured to transmit the output value of the input layer group and the error function of the weight of the output layer group to the cloud server, and
   It comprises a first receiver configured to receive the output value of the intermediate layer group calculated by the cloud server and the error function of the weight of the intermediate layer group.
   The cloud server is
   A fifth calculation unit configured to calculate the output value of the result of inputting the output value of the input layer group calculated by the client terminal into the intermediate layer group.
   A sixth calculation unit configured to calculate the weight error function of the intermediate layer group based on the weight error function of the output group group calculated by the client terminal during training of the model.
   A second model updater configured to update the weights of the intermediate group based on the error function calculated by the sixth calculator.
   A second transmission unit configured to transmit the output value of the intermediate layer group and the error function of the weight of the intermediate layer group to the client terminal, and
   A distributed deep learning system comprising a second receiving unit configured to receive an output value of the input layer group calculated by the client terminal and an error function of the weight of the output layer group. ..
2. In the distributed deep learning system according to claim 1,
   The input layer group and the output layer group are constructed on each of the plurality of client terminals, respectively.
   The fifth calculation unit of the cloud server processes the output value of the input layer group calculated by each of the plurality of client terminals in a time division manner to calculate the output value of the intermediate layer group.
   The sixth calculation unit of the cloud server processes the error function of the weight of the output layer group calculated by each of the plurality of client terminals in a time division to calculate the error function of the weight of the intermediate layer group.
   The second model update unit of the cloud server is a distributed deep learning system characterized in that the weights of the intermediate layer group are updated based on an error function calculated for each client terminal by the sixth calculation unit.
3. In the distributed deep learning system according to claim 1,
   The input layer group and the output layer group are constructed on each of the plurality of client terminals, and the input layer group is constructed for each type of the sample data.
   The first calculation unit of each client terminal calculates the output value of the result of inputting the sample data into the input layer group for this data type.
   The second calculation unit of each client terminal inputs the output value of the intermediate layer group calculated by the cloud server for each type of sample data into the output layer group, and inputs the output value of the model for each type of sample data. Calculate to
   The third calculation unit of each client terminal calculates the error function of the weight of the output layer group for each type of sample data.
   The fourth calculation unit of each client terminal uses the error function of the weight of the input layer group as the type of sample data based on the error function of the weight of the intermediate layer group calculated for each type of sample data by the cloud server. Calculated for each
   The first model update unit of each client terminal updates the weight of the input layer group for each type of sample data based on the error function calculated for each type of sample data by the fourth calculation unit. The weight of the output layer group is updated based on the error function calculated for each type of sample data by the third calculation unit.
   The fifth calculation unit of the cloud server processes the output value of the input layer group calculated for each type of sample data by the plurality of client terminals in a time division to calculate the output value of the intermediate layer group. death,
   The sixth calculation unit of the cloud server processes the error function of the weight of the output layer group calculated for each type of sample data by the plurality of client terminals in a time division to obtain the weight of the intermediate layer group. Calculate the error function,
   The second model update unit of the cloud server is characterized in that the weight of the intermediate layer group is updated based on the error function calculated for each client terminal and each type of sample data by the sixth calculation unit. Distributed deep learning system.
4. In the distributed deep learning system according to claim 3,
   Each client terminal
   When the input layer group cannot be updated due to the inability to acquire the sample data, the other client terminal is requested to update the weight of the input layer group, and the input layer group has been updated by acquiring the sample data. In the case of, a third transmitter configured to transmit the update result of the weight of the input layer group in response to a request from another client terminal, and
   When the input layer group cannot be updated due to the inability to acquire the sample data, the update result of the weight of the input layer group is received from another client terminal, and the input layer group has been updated by acquiring the sample data. Further provided with a third receiver configured to receive requests from other client terminals in the case of
   The first model update unit of each client terminal updates the weights of the input layer groups that could not be updated based on the weights of the input layer groups received from other client terminals. system.
5. In the distributed deep learning system according to claim 3,
   Further equipped with a storage server connected to each client terminal via a network,
   Each client terminal
   A writing unit configured to write the update result of the weight of the input layer group to the storage server when the input layer group has been updated by acquiring the sample data.
   Further, it is provided with a reading unit configured to read the update result of the weight of the input layer group from the storage server when the input layer group cannot be updated due to the inability to acquire the sample data.
   The first model update unit of each client terminal is a distributed deep learning system characterized in that the weight of the input layer group that cannot be updated is updated based on the weight of the input layer group read from the storage server. ..
6. In the distributed deep learning system according to claim 1,
   The input layer group and the output layer group are constructed in each of the plurality of client terminals, and the input layer group and the output layer group are constructed for each type of the sample data.
   The first calculation unit of each client terminal calculates the output value of the result of inputting the sample data into the input layer group for this data type.
   The second calculation unit of each client terminal inputs the output value of the intermediate layer group calculated by the cloud server for each type of sample data into the output layer group for this data type, and outputs the output value of the model. Is calculated for each type of sample data,
   The third calculation unit of each client terminal calculates the error function of the weight of the output layer group for each type of sample data.
   The fourth calculation unit of each client terminal uses the error function of the weight of the input layer group as the type of sample data based on the error function of the weight of the intermediate layer group calculated for each type of sample data by the cloud server. Calculated for each
   The first model update unit of each client terminal updates the weight of the input layer group for each type of sample data based on the error function calculated for each type of sample data by the fourth calculation unit. The weight of the output layer group is updated for each type of sample data based on the error function calculated for each type of sample data by the third calculation unit.
   The fifth calculation unit of the cloud server processes the output value of the input layer group calculated for each type of sample data by the plurality of client terminals in a time division to calculate the output value of the intermediate layer group. death,
   The sixth calculation unit of the cloud server processes the error function of the weight of the output layer group calculated for each type of sample data by the plurality of client terminals in a time division to obtain the weight of the intermediate layer group. Calculate the error function,
   The second model update unit of the cloud server is characterized in that the weight of the intermediate layer group is updated based on the error function calculated for each client terminal and each type of sample data by the sixth calculation unit. Distributed deep learning system.
7. In the distributed deep learning system according to claim 6,
   Each client terminal
   When the input layer group and the output layer group cannot be updated due to the inability to acquire the sample data, the other client terminal is requested to update the weights of the input layer group and the output layer group, and the sample is obtained. When the input layer group and the output layer group have been updated by acquiring data, the update result of the weights of the input layer group and the output layer group is transmitted in response to a request from another client terminal. With the third transmitter
   When the input layer group and the output layer group cannot be updated due to the inability to acquire the sample data, the update result of the weights of the input layer group and the output layer group is received from another client terminal, and the sample is received. Further, a third receiving unit configured to receive a request from another client terminal when the input layer group and the output layer group have been updated by acquiring data is provided.
   The first model update unit of each client terminal updates the weight of the input layer group that could not be updated based on the weight of the input layer group received from the other client terminal, and the update is not possible. A distributed deep learning system characterized in that the weights of the output layer group are updated based on the weights of the output layer group received from other client terminals.
8. In the distributed deep learning system according to claim 6,
   Further equipped with a storage server connected to each client terminal via a network,
   Each client terminal
   A writing unit configured to write the update result of the weights of the input layer group and the output layer group to the storage server when the input layer group and the output layer group have been updated by acquiring the sample data. When,
   When the input layer group and the output layer group cannot be updated due to the inability to acquire the sample data, the update result of the weights of the input layer group and the output layer group is read from the storage server. Further equipped with a reading unit,
   The first model update unit of each client terminal updates the weight of the input layer group that could not be updated based on the weight of the input layer group read from the storage server, and the output that could not be updated. A distributed deep learning system characterized in that the weights of the layer groups are updated based on the weights of the output layer groups read from the storage server.

Images