JP7171477B2 - Information processing device, information processing method and information processing program - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act Submitted Paper Overview of the 2019 Spring Research Presentation Meeting of the Acoustical Society of Japan http://www. asj. gr. jp/annualmeeting/index. html

The present invention relates to an information processing device, an information processing method, and an information processing program.

Conventionally, various classification processes using models such as DNNs (Deep Neural Networks) have been implemented. A method of learning a model using the stochastic gradient descent (SGD) method is known in order to realize classification processing using such a DNN. For example, different learning data is distributed to each arithmetic unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and each arithmetic unit learns a model using the distributed learning data. A technique for repeatedly performing processing for synchronizing learning results of devices is known. Also, in order to reduce the cost of synchronizing processing of learning results in each arithmetic unit, a technique is known in which synchronization is performed after each arithmetic unit performs learning processing a plurality of times.

“Experiments on Parallel Training of Deep Neural Network using Model Averaging”, Hang Su, Haoyu Chen, Internet < https://arxiv.org/abs/1507.01239> (searched March 1, 2019)

However, the technique described above leaves room for improving the accuracy of the model.

For example, in the above-described technology, each arithmetic unit synchronizes models trained using different learning data, so the model finally obtained is the same as the model trained using all the learning data. It becomes only an approximation. Moreover, with the above-described technology, there is a possibility that the characteristics of the learning data cannot be learned appropriately if the number of times of synchronization processing is not sufficient.

The present application has been made in view of the above, and an object thereof is to improve the learning accuracy of a model using a plurality of arithmetic units.

An information processing device according to the present application includes a distribution unit that distributes different learning data to a plurality of computing devices that individually perform model learning using the distributed learning data, and a distribution unit that distributes the distributed learning data. and a synchronizing unit for synchronizing the model learned by each arithmetic unit in a mode according to the result of learning executed by each arithmetic unit.

According to one aspect of the embodiments, it is possible to improve the learning accuracy of a model using a plurality of computing devices.

FIG. 1 is a diagram illustrating an example of processing executed by an information providing device according to an embodiment. FIG. 2 is a diagram illustrating a configuration example of an information providing device according to the embodiment; 3 is a diagram illustrating an example of a functional configuration of a second calculation unit according to the embodiment; FIG. 4 is a diagram illustrating an example of a functional configuration of a second calculation unit according to the embodiment; FIG. FIG. 5 is a diagram illustrating an example of a hardware configuration;

Embodiments for implementing an information processing apparatus, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail below with reference to the drawings. The information processing apparatus, information processing method, and information processing program according to the present application are not limited to this embodiment. Further, each embodiment can be appropriately combined within a range that does not contradict the processing contents. Also, in each of the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

[1. About the information providing device]
First, an example of an information processing method executed by an information providing apparatus 10, which is an example of an information processing apparatus, will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of processing executed by an information providing device according to an embodiment. In FIG. 1 , as processing executed by the information providing apparatus 10, learning processing for learning a model and information classification using a trained model (hereinafter sometimes referred to as a “learning model”) are performed. An example of the classification process flow has been described.

An information providing device 10 shown in FIG. 1 is an information processing device, and is realized by, for example, a server device, a cloud system, or the like. Also, the data server 100 shown in FIG. 1 manages various data, and is realized by, for example, a server device, a cloud system, or the like. Also, the user terminal 200 is a terminal device used by a user who uses the result of the classification process, and is realized by, for example, a PC (Personal Computer), a server device, various smart devices, and the like.

Here, the information providing apparatus 10 acquires learning data from the data server 100 and causes the model to learn the features of the acquired learning data. When various types of measurement data are acquired from the user terminal 200, the information providing apparatus 10 executes classification processing according to the characteristics of the measurement data using the learning model, and outputs the classification results to the user terminal 200. will be provided.

In this series of processes, what kind of data should be used as learning data, what kind of characteristics of the learning data should be learned by the model, what kind of data should be used as measurement data, what kind of Arbitrary settings can be adopted as to whether to perform feature-based classification. As a specific example, the information providing apparatus 10 stores attribute information indicating user demographic attributes and psychographic attributes, history of browsed content, purchase history of transaction targets (products and services), location history, and the like. Information such as various history information shown is acquired as learning data. Here, for each piece of learning data, for example, if information indicating the type of advertisement selected by the user is registered as a label, the information providing apparatus 10, when various types of history information are input to the model, The model is trained to output information indicating corresponding labels (that is, history information and corresponding information indicating advertisements selected by the user). After acquiring various types of attribute information of the user as measurement data, the information providing apparatus 10 inputs the acquired attribute information into the learning model, thereby estimating advertisements that the user is likely to select.

Such a model is realized by a neural network, that is, a DNN, in which a plurality of nodes are connected via connection paths set with individual connection coefficients. Note that the model may be a neural network having an arbitrary structure such as an autoencoder, CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), or its extension LSTM (Long short-term memory).

Further, when the information providing apparatus 10 causes a model to learn the features of one piece of learning data, the learning is performed using the statistical gradient descent method. For example, the information providing apparatus 10 sets an arbitrary objective function according to input information and output information for a model, and uses error backpropagation or the like so that the set objective function satisfies a predetermined condition. By correcting the connection coefficients (that is, parameters) that the model has, the model learns the features that the learning data has.

For example, when the objective function indicates an error between the input information and the output information, the information providing apparatus 10 corrects the parameters so that the value of the objective function becomes smaller. On the other hand, for example, when the objective function indicates a value based on cross entropy, the information providing device 10 modifies the parameters so that the value of the objective function increases. Therefore, in the following description, the amount by which the objective function changes in the direction that satisfies a predetermined condition between before and after learning using certain learning data may be referred to as an "improvement amount." For example, if the objective function indicates the error between the input information and the output information, the value obtained by subtracting the value of the objective function after learning from the value of the objective function before learning corresponds to the "improvement amount", and the objective function is the cross entropy. When indicating the value based on the learning, the value obtained by subtracting the value of the objective function before learning from the value of the objective function after learning corresponds to the "improvement amount".

[1-1. About mini-batch learning and model averaging]
Before explaining the learning process executed by the information providing apparatus 10, an outline of the mini-batch learning will be explained. For example, when executing a learning process using mini-batch learning, the information providing apparatus 10 uses M data randomly selected from all N learning data as a mini-batch, and updates model parameters for each mini-batch. .

Here, a method of speeding up the learning by executing such mini-batch learning in parallel is conceivable. For example, a different mini-batch is distributed to each of a plurality of arithmetic units such as CPUs, GPUs, or their cores, and independent model learning processing is executed for each arithmetic unit. Then, a method of synchronizing the learning results of each arithmetic unit and repeatedly distributing new mini-batches is conceivable. For example, a new model is generated using the average values of the parameters of the model learned by each arithmetic unit, the new model is distributed to each arithmetic unit, and the learning process is executed again using different mini-batches.

When such mini-batch learning is performed, if the number of arithmetic units is K, it is expected that the time required for calculating the correction amount of each model in the time required for learning processing is reduced to 1/K. However, in such mini-batch learning, the model independently learned by each arithmetic unit is synchronized by communicating between each arithmetic unit. Therefore, assuming that the total number of learning data is N and the number of learning data included in a mini-batch is M, when synchronization of all arithmetic units is performed for each mini-batch, N/M times of synchronization processing is required as an overhead. This will add to the processing time.

In order to reduce the overhead of such synchronization processing, a model averaging method is known in which synchronization processing is performed each time mini-batch learning is performed a predetermined number of times. For example, when performing model learning by the model averaging method, the information providing apparatus 10 does not perform synchronization processing each time mini-batch learning is performed, but a predetermined number of times (for example, F times) for each arithmetic device. ), and after performing mini-batch learning, the models of each arithmetic unit are synchronized. When such processing is executed, the number of times of synchronization becomes N/(M×F), so the overhead of synchronization processing can be reduced to 1/F.

[1-2. About learning process]
However, in the model averaging method described above, there is a possibility that the accuracy of model identification may be degraded due to the reduction in synchronization processing. Also, since the model that is finally obtained by mini-batch learning is the average of the models that learned the characteristics of each different training data, input all the training data one by one for a single model and input the training data It is just an approximation of a model with modified parameters so that the objective function improves each time. For this reason, it can be said that mini-batch learning and model averaging methods have room for improving model accuracy.

Therefore, the information providing device 10 executes the following learning process. First, the information providing device 10 distributes different learning data to each of a plurality of computing devices that individually perform model learning using the distributed learning data. Then, the information providing device 10 synchronizes the model learned by each computing device in a mode according to the result of learning executed by each computing device using the distributed learning data. In other words, the information providing device 10 does not simply use a simple average of the models as a synchronization result, but adaptively synchronizes the models according to the learning results of the models by the arithmetic units.

For example, a model with a large amount of improvement can be said to be a model that has appropriately learned the features of the learning data. For this reason, among the models learned by each arithmetic unit, the parameters of the model with a large improvement amount are considered to greatly contribute to the accuracy of the final model. Therefore, the information providing apparatus 10 synchronizes the models so that the larger the improvement amount of the model, the more important it is. That is, the information providing apparatus 10 generates a model as a synchronization result by integrating each model in a state where a larger weight is applied to a model with a larger improvement amount. As a result of such processing, for example, the information providing apparatus 10 generates a synchronization result that preferentially uses a model with a greater amount of improvement, that is, a model that is estimated to increase the accuracy of the final model. can improve the accuracy of the final model.

In addition, the information providing device 10 dynamically changes the model synchronization timing in the learning process. For example, the information providing device 10 synchronizes each model so that the number of times each computing device performs model learning using newly distributed learning data is random. More specifically, the information providing device 10 generates an integer random number within the interval [1, F], and synchronizes each model after mini-batch learning is performed the number of times indicated by the generated random number. As a result of such processing, the information providing apparatus 10 can reduce the possibility that the objective function of the model that is finally generated falls into a local minimum. It is believed that the accuracy of the generated model can be improved.

[1-3. An example of the flow of processing executed by the information providing device]
An example of the flow of processing executed by the information providing apparatus 10 will be described below with reference to FIG. In the following description, an example in which model learning is performed using K GPUs #1 to #K in parallel as arithmetic units will be described. First, the information providing device 10 acquires learning data from the data server 100 (step S1). In such a case, the information providing device 10 registers learning data in the learning data database 31 . Then, the information providing device 10 dynamically changes the synchronization timing of the model trained by each GPU in the mini-batch learning (step S2).

For example, the information providing device 10 randomly extracts learning data #1-1 to #1-M from the learning data database 31. FIG. Subsequently, the information providing device 10 divides the extracted learning data into K mini-batches, and distributes each mini-batch to the GPUs #1 to #K. That is, the information providing device 10 distributes M/K pieces of learning data as a mini-batch to each of the GPUs #1 to #K. In other words, the information providing device 10 distributes different learning data to each computing device.

Here, each GPU #1 to #K performs model learning using the distributed learning data. That is, each GPU #1 to #K executes mini-batch learning using the distributed mini-batch. For example, GPU #1 generates M/K copies of a model to be learned, and inputs different learning data to each generated copy. Then, GPU #1 corrects the parameters of each model so that the objective function of each copied model is improved, and sets the model obtained by integrating the corrected parameters as the first learning result. For example, GPU #1 may use the average value of the corrected parameters of each model as the parameter of the model that is the learning result. Note that GPU #1 may learn one model using M/K pieces of learning data without copying the model. Similarly, the other GPUs #2 to #K also execute mini-batch learning using learning data individually distributed to each of the GPUs #1 to #K.

Here, instead of synchronizing the learning results of each GPU #1 to #K each time mini-batch learning is performed, the information providing apparatus 10 synchronizes each GPU #1 each time mini-batch learning is performed a random number of times. ~ Synchronize the learning results of #K. For example, the information providing device 10 generates a random number within a predetermined range, and, for example, when the generated random number is "3", mini-batch learning is performed three times.

For example, the information providing device 10 divides the learning data #1-1 to #1-M into K mini-batches #1-1 to #1-K, and divides the mini-batches #1-1 to #1-K into It is distributed to individual GPUs #1 to #K to execute the first mini-batch learning. Subsequently, the information providing device 10 extracts new learning data #2-1 to #2-M from the learning data database 31 without synchronizing the models of the GPUs #1 to #K, and extracts the extracted learning data. #2-1 to #2-M are divided into K mini-batches #2-1 to #2-K. Then, the information providing apparatus 10 distributes the mini-batches #2-1 to #2-K to individual GPUs #1 to #K to execute the second mini-batch learning. Similarly, the information providing apparatus 10 generates new mini-batches #3-1 to #3-K, distributes them to individual GPUs #1 to #K, and causes them to perform the third mini-batch learning. Then, when mini-batch learning is executed three times, the information providing apparatus 10 synchronizes the models of the GPUs #1 to #K.

Subsequently, the information providing apparatus 10 generates a new random number, and, for example, if the generated random number is "1", causes each of the GPUs #1 to #K to perform mini-batch learning for the fourth time. After the fourth mini-batch learning, the information providing device 10 synchronizes the models of the GPUs #1 to #K. In this way, the information providing apparatus 10 executes model synchronization so that the number of times of mini-batch learning executed by each GPU #1 to #K is random. That is, the information providing device 10 randomly changes the synchronization timing of the models of the GPUs #1 to #K.

That is, the information providing device 10 corrects the parameter values of the model using the learning data, thereby distributing different learning data to a plurality of computing devices that cause the model to learn the features of the learning data. Then, the information providing device 10 synchronizes the parameter values of each model at random timing and distributes new learning data to each computing device, thereby generating a synchronized model using the newly distributed learning data. The process of learning is repeatedly executed.

Here, when synchronizing the models of the GPUs #1 to #K, the information providing apparatus 10 applies a weight based on the improvement amount of the objective function of each model, and synchronizes the parameters of each model with the synthesized model. (step S3). For example, when model synchronization is performed after the third mini-batch learning, the information providing device 10 calculates the improvement amount of the objective function of each model. For example, the information providing apparatus 10 outputs the value of the objective function of model #1-2 generated by GPU #1 in the second mini-batch learning and the value of the objective function of model #1-3 generated in the third mini-batch learning. and the improvement amount #1 is calculated. Similarly, the information providing apparatus 10 outputs the values of the objective functions of the models #2-2 to #K-2 generated by the GPUs #2 to #K in the second mini-batch learning and the models generated in the third mini-batch learning. The improvement amounts #2 to #K are calculated from the objective function values #2-3 to #K-3.

Then, the information providing apparatus 10 sets weights #1 to #K based on the values of the improvement amounts #1 to #K. For example, the information providing apparatus 10 may set the weights #1 to #K by dividing the values of the improvement amounts #1 to #K by the sum of the improvement amounts #1 to #K. In addition, if the information providing apparatus 10 sets the weights #1 to #K so that the model with a larger improvement amount value has a higher weight, the weights #1 to #K calculated by an arbitrary method can be used. You may set #K.

Subsequently, the information providing apparatus 10 integrates the models #1-3 to #K-3 using the weights #1 to #K. For example, the information providing apparatus 10 calculates a value obtained by multiplying the parameters of the models #1-3 by the weight #1. Similarly, the information providing device 10 calculates values obtained by multiplying the parameters of the models #2-3 to #K-3 by individual weights #2 to #K. Then, the information providing apparatus 10 sums the calculated values to generate the parameters of the model #X-4 that integrates the models #1-3 to #K-3.

After that, the information providing device 10 distributes the model #X-4 to each of the GPUs #1 to #K as a model for the fourth learning. As a result, each of the GPUs #1 to #K executes mini-batch learning using different mini-batches for the model #X-4.

In addition, the information providing device 10 repeatedly executes learning using new learning data until a predetermined condition is satisfied. Then, when a predetermined condition is satisfied, the information providing device 10 generates a final learning model by integrating each model (step S4). For example, the information providing device 10 repeatedly executes the learning process until mini-batch learning is performed using all the learning data registered in the learning data database 31 . When the mini-batch learning using all the learning data is completed, the information providing device 10 synchronizes the models of the GPUs #1 to #K. For example, the information providing apparatus 10 may average the parameters of each model, or may perform integration in consideration of a weight according to the amount of improvement in the value of the objective function of each model.

Then, the information providing device 10 executes classification processing using the generated learning model. For example, the information providing device 10 acquires measurement data from the user terminal 200 (step S5). In such a case, the information providing device 10 inputs the measured data to the learning model, and provides the user terminal 200 with the classification result based on the information output by the learning model (step S6). Note that the information providing apparatus 10 may perform information distribution according to the classification result, such as content distribution according to the classification result, instead of providing the classification result itself. Further, the information providing apparatus 10 may provide the classification result to a service providing server that provides various services to the user terminal 200 instead of the user terminal 200 . In such a case, the service providing server will provide the user terminal 200 with a service with content corresponding to the classification result.

[1-4. About synchronization timing]
In the above description, the information providing apparatus 10 synchronizes the models for which mini-batch learning has been performed by each arithmetic unit (that is, GPU) at random timing. However, embodiments are not so limited. If the information providing apparatus 10 dynamically changes the model synchronization timing, the model synchronization timing may be determined based on any index.

For example, the information providing device 10 may synchronize each model when the result of learning executed by each arithmetic device satisfies a predetermined condition. For example, when the information providing device 10 satisfies a predetermined condition, the amount of improvement between the value of the objective function before learning and the value of the objective function after learning of the model trained by at least one of the computing devices satisfies a predetermined condition. may synchronize each model.

As a more specific example, the information providing device 10 acquires the improvement amount of the objective function of each model, and counts the number of models whose improvement amount exceeds a predetermined threshold. Then, when the counted number exceeds a predetermined threshold, the information providing apparatus 10 may synchronize each model. For example, the information providing apparatus 10 may synchronize each model when there is even one model whose improvement amount exceeds a predetermined threshold. For example, the information providing apparatus 10 may set the weight of a model whose improvement amount exceeds a predetermined threshold to a value greater than the weight of other models, and integrate each model.

Further, the information providing apparatus 10 may set the synchronization timing according to the accumulated amount of improvement. For example, the information providing device 10 calculates the cumulative improvement amount of each model each time mini-batch learning is performed, and performs synchronization when the number of models whose cumulative total exceeds a predetermined threshold exceeds a predetermined threshold. good too. Further, the information providing apparatus 10 may perform synchronization when the cumulative amount of improvement of all models exceeds a predetermined threshold. Further, the information providing apparatus 10 may randomly change the improvement amount threshold each time synchronization is performed.

Note that the information providing apparatus 10 may perform synchronization so that the number of models whose improvement amount exceeds a predetermined threshold does not exceed a predetermined threshold. For example, the information providing apparatus 10 estimates whether the number of models whose improvement amount exceeds a predetermined threshold exceeds a predetermined threshold when the next mini-batch learning is performed based on the history and accumulation of the improvement amount. and if estimated to exceed, synchronization may occur.

[1-5. Synchronization method]
In the above description, the information providing apparatus 10 considers the weight according to the amount of improvement between the objective function value before learning and the objective function value after learning of each model, and weights the parameters of each model. The sum was calculated, and the weighted sum of the calculated parameters was used as the parameter of the model after synchronization. However, embodiments are not so limited.

For example, the information providing device 10 may perform arbitrary processing as long as the models are synchronized in a manner corresponding to the value of the objective function of the model trained by each computing device. For example, the information providing apparatus 10 extracts only models whose objective function values exceed a predetermined threshold value (or models whose objective function values are less than a predetermined threshold value), and averages or weights the parameters of the extracted models. The sum may be the synchronization result.

Further, the information providing apparatus 10 may take into consideration the weight according to the value of the objective function of each model, and take a model obtained by integrating each model as the synchronization result. That is, the information providing device 10 may consider the weight according to the value of the objective function itself instead of the improvement amount. For example, the information providing device 10 may set a larger value of weight as the value of the objective function of each model is lower (or higher).

Further, the information providing device 10 may specify a model with the smallest (or largest) value of the objective function, and distribute the specified model to each computing device as a synchronization result. Further, the information providing device 10 may specify a model with the largest improvement in the objective function, and distribute the specified model to each computing device as a synchronization result.

Further, the information providing apparatus 10 may synchronize each model using a genetic algorithm that selects a model according to the value of the objective function. For example, the information providing apparatus 10 sets the value of the objective function of each model or the amount of improvement of the objective function as the fitness of each model, and copies, crossovers, or mutations (hereinafter referred to as , may be described as “operation”) to generate the next-generation model. For example, the information providing device 10 may select two models, randomly intersect the parameters of the selected models, or randomly change the parameters of the selected models. By executing such processing, the information providing device 10 generates new K models to be subjected to the n+1th mini-batch learning from the K models that have undergone the n-th mini-batch learning, The generated K models may be distributed to each arithmetic unit. In addition, the information providing device 10 may perform model synchronization in a manner based on various arbitrary genetic algorithms.

In addition to the objective function, the information providing device 10 may synchronize the models in any manner as long as the synchronization manner of each model is adaptively changed according to the learning result in each arithmetic unit. . For example, the information providing device 10 may calculate the weighted sum of the parameters of each model in consideration of the weight according to the time required for mini-batch learning by each arithmetic device, and the genetic algorithm with probability according to time. You may select a model to be subjected to various operations in .

[1-6. Synchronization target]
Note that the information providing device 10 does not have to synchronize the models trained by all the arithmetic devices. For example, the information providing device 10 may synchronize models trained by some of the plurality of computing devices. For example, the information providing apparatus 10 may divide each arithmetic unit into a predetermined number of groups, such as GPUs #1 to #10 and GPUs #11 to #20, and perform synchronization for each group. For example, if there is a model whose objective function improvement amount exceeds a predetermined threshold among the models trained by all the computing devices, the information providing device 10 selects the computing device that learned the model. Model synchronization may occur only within the containing group. It should be noted that when such processing is performed, the computing devices that are not subject to synchronization will continue learning the model that is the result of the previous mini-batch learning.

Further, the information providing apparatus 10 may synchronize models trained by a plurality of computing devices whose communication delay is within a predetermined range. For example, the information providing device 10 may synchronize a model that has been trained by a predetermined number of arithmetic devices that are physically located nearby.

Further, the information providing device 10 may synchronize a model that has been learned by some of the arithmetic devices that are randomly selected from among the plurality of arithmetic devices. For example, the information providing device 10 may synchronize only models trained by a predetermined number of arithmetic devices randomly selected from among the arithmetic devices. In addition, the information providing device 10 may synchronize only models whose objective function value or improvement amount exceeds a predetermined threshold and models that have been learned by a predetermined number of randomly selected arithmetic units. .

Further, the information providing device 10 has a plurality of computing devices, the number of computing devices corresponding to the number of dimensions of information that can be calculated by each computing device and the number of learning data distributed to all the computing devices. Synchronization of the model trained by is also possible. That is, the information providing apparatus 10 is presumed to be able to learn efficiently according to the performance of each arithmetic unit, the number of each arithmetic unit, the number of all learning data, the number of learning data to be mini-batches, and the like. Any number of models may be synchronized.

[1-7. About arithmetic unit]
In the example described above, the processing using a plurality of GPUs as arithmetic devices has been described, but the embodiment is not limited to this. For example, the information providing device 10 may apply the learning process described above to a plurality of CPUs. The learning process described above may be applied to the connected system. Further, the information providing apparatus 10 may perform the above-described learning process by regarding a plurality of cores included in one CPU or GPU as an arithmetic device. In addition, the information providing device 10 may regard GPUs and GPU cores arranged on one or more graphic cards as arithmetic devices. Further, the information providing device 10 may regard a plurality of CPUs and GPUs as one arithmetic device, and regard one or more cores included in these CPUs and GPUs as one arithmetic device. good too.

Further, the information providing device 10 may have the arithmetic device described above in its own housing, or may have it in a different housing. For example, the information providing device 10 may execute the above-described learning process using an arithmetic device in a server device connected via various networks.

That is, the information providing device 10 regards a device capable of individually executing model learning as an arithmetic device, and regards any device as an arithmetic device as long as it executes the learning process described above. good. Note that each arithmetic unit does not need to have an independent storage device. For example, each arithmetic unit or a part of arithmetic units may share a storage device such as a memory or a register. Also, each computing device may be, for example, a so-called virtual machine.

[1-8. About the executing body]
Note that the learning process described above may be executed by any execution subject. For example, the information providing device 10 may have a control device that controls each arithmetic device separately from each arithmetic device. In such cases, such controllers may perform training data distribution and synchronization. Further, determination of synchronization timing and model synchronization processing may be realized by cooperative operation of each arithmetic unit.

[1-9. Regarding the relationship between synchronization timing and synchronization method]
Further, the information providing apparatus 10 may execute the dynamic change of the synchronization timing and the synchronization of the model according to the learning result independently or in association with each other. For example, when the information providing apparatus 10 dynamically changes the synchronization timing, the model synchronization may be realized by calculating a simple average. Further, when synchronizing the models in a manner corresponding to the learning result, the information providing apparatus 10 does not need to dynamically change the synchronization timing.

Further, for example, the information providing apparatus 10 may change the synchronization mode each time the synchronization process is executed. For example, each time the information providing apparatus 10 executes synchronization processing, a method of synchronizing the models by simple averaging, a method of adopting a weight according to the amount of improvement, a method of determining the model with the largest amount of improvement as the result of synchronization, etc. Alternatively, one of a plurality of methods may be selected at random or with a probability according to the learning result, and model synchronization may be performed using the selected method. Further, the information providing apparatus 10 may perform synchronization at synchronization timing according to the synchronization method adopted last time. For example, when synchronizing models by simple averaging, the information providing apparatus 10 performs the next synchronization when the improvement amount of any model exceeds a predetermined threshold, and adopts a weight according to the improvement amount. If the method synchronizes the model, the number of mini-batch trainings until the next synchronization may be randomly selected.

[2. Example of functional configuration]
An example of the functional configuration of the information providing apparatus 10 that implements the learning process described above will be described below. FIG. 2 is a diagram illustrating a configuration example of an information providing device according to the embodiment; As shown in FIG. 2 , the information providing device 10 has a communication section 20 , a storage section 30 , a first calculation section 40 and a second calculation section 50 .

The communication unit 20 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 20 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the data server 100 and the user terminal 200, for example.

The storage unit 30 is implemented by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 30 also stores a learning data database 31 and a model database 32 .

Learning data is registered in the learning data database 31 . For example, various learning data acquired from the data server 100 are registered in the learning data database 31 . In the model database 32, data of a learning model that has undergone learning through the above-described learning process is registered.

The first computing unit 40 is a controller, and various programs stored in a storage device inside the information providing device 10 are executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). It is realized by being executed using a RAM or the like as a work area. Also, the first calculation unit 40 is a controller, and may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 2 , the first computing section 40 has a learning control section 41 and an information providing section 42 . The learning control unit 41 executes the learning process described above by controlling the second calculation unit 50 . For example, the learning control unit 41 acquires learning data from the data server 100 and registers the acquired learning data in the learning data database 31 . Further, the learning control unit 41 provides the learning data registered in the learning data database 31 to the second calculation unit 50, and executes the learning process described above, thereby obtaining a learning model. The learning control unit 41 then registers the learning model in the model database 32 .

The information providing unit 42 provides the classification result of the measurement data using the learning model. For example, when acquiring measurement data from the user terminal 200, the information providing unit 42 reads a learning model from the model database 32, and inputs the measurement data into the read learning model. Then, the information providing unit 42 outputs information according to the classification result output by the learning model to the user terminal 200 or the like.

The second calculation unit 50 is an information processing unit having a plurality of calculation devices, and is realized by, for example, a graphic card in which a plurality of GPUs or GPU cores are arranged, or a plurality of graphic cards. For example, the second calculation unit 50 has a calculation unit 51 and a calculation control unit 52 .

Here, FIG. 3 is a diagram showing an example of the functional configuration of the second calculation unit according to the embodiment. As shown in FIG. 3, the calculation unit 51 has a plurality of calculation devices. Note that each arithmetic unit is, for example, a GPU or a core of a GPU, and performs model learning using a mini-batch of distributed learning data. That is, each computing device holds its own model, and corrects the parameter values of the model using the distributed learning data, thereby allowing the model to learn the features of the learning data.

Further, the arithmetic control unit 52 has a distribution unit 521 and a synchronization unit 522 . The distributing unit 521 and the synchronizing unit 522 are realized by the second calculating unit 50 executing various programs stored in the storage device inside the information providing apparatus 10 using the RAM or the like as a work area. . Note that the calculation control unit 52 may be implemented by, for example, any one of the calculation devices included in the calculation unit 51 .

Here, the distributing unit 521 distributes different learning data to each of the plurality of arithmetic units that individually perform model learning using the distributed learning data. For example, when the computing unit 51 has K computing devices, the distributing unit 521 generates mini-batches by dividing M data randomly selected from learning data into K mini-batches. Then, the distribution unit 521 distributes the generated mini-batch to different computing devices, thereby causing each computing device to perform mini-batch learning. Further, when each computing device executes mini-batch learning, the distributing unit 521 generates K mini-batches from the new M learning data, and distributes the generated mini-batches to each computing device again.

The synchronization unit 522 synchronizes the model learned by each computing device in a mode according to the result of the learning executed by each computing device using the distributed learning data. More specifically, the synchronization unit 522 synchronizes the parameter values of each model and distributes the synchronized parameter values to each arithmetic unit. That is, the synchronizing unit 522 distributes the model resulting from synchronization to each arithmetic unit and continues the mini-batch learning.

Also, the synchronization unit 522 dynamically changes synchronization opportunities for models trained by a plurality of arithmetic units. For example, the synchronizing unit 522 synchronizes each model so that the number of times each arithmetic unit learns the model using newly distributed learning data is random. More specifically, the synchronizing unit 522 performs mini-batch learning for a randomly selected number of times, and when each computing device performs mini-batch learning, in a manner according to the value of the objective function of the model trained by each computing device. , synchronize each model. Then, the synchronization unit 522 again executes the mini-batch learning of each model for the randomly selected number of times.

For example, when each arithmetic unit executes mini-batch learning, the synchronization unit 522 identifies the value of the objective function of the model that each arithmetic unit has individually learned, and assigns a weight according to the value of the identified objective function. Considering this, a model that integrates each model is taken as a synchronization result. For example, the synchronization unit 522 considers the weight according to the amount of improvement between the objective function value before learning and the objective function value after learning of each model, and the model that integrates each model as a synchronization result. good too. Then, the synchronization unit 522 distributes the model resulting from synchronization to each arithmetic unit, and causes the mini-batch learning to be executed again.

Note that the synchronizing unit 522 may distribute the model with the largest improvement amount between the objective function value before learning and the objective function value after learning among the models as the synchronization result to each arithmetic device. . Also, the synchronization unit 522 may synchronize each model using a genetic algorithm that selects a model according to the value of the objective function.

Further, the synchronization unit 522 may synchronize each model when the result of learning executed by each arithmetic unit satisfies a predetermined condition. For example, the synchronization unit 522 acquires the value of the objective function of the model of each arithmetic unit and the amount of improvement each time mini-batch learning is performed, and synchronizes each model if the amount of improvement of at least one of the models satisfies a predetermined condition. You may let

Further, the synchronization unit 522 may synchronize the models trained by some of the plurality of arithmetic devices. For example, the synchronizing unit 522 may synchronize models trained by a plurality of computing devices whose communication delay is within a predetermined range. Further, the synchronization unit 522 may synchronize a model that has been trained by some of the arithmetic devices that are randomly selected from among the plurality of arithmetic devices.

In addition, the synchronization unit 522 uses the number of arithmetic units corresponding to the number of dimensions of information that can be calculated by each arithmetic unit and the number of learning data distributed to all the arithmetic units among the plurality of arithmetic units. Synchronization of trained models may be performed. That is, the synchronization unit 522 may dynamically change the number of arithmetic units to be synchronized according to the performance of each arithmetic unit, the number of learning data, and the like.

In addition, the synchronization unit 522 determines that the learning end condition is satisfied when all the learning data is distributed or when the improvement amount of each model does not continue to change, and the learning model that integrates each model to generate The synchronization unit 522 then outputs the learning model to the first calculation unit 40 .

[3. Regarding the flow of processing executed by the information providing device]
Next, an example of the flow of processing executed by the information providing apparatus 10 will be described with reference to FIG. FIG. 4 is a flowchart illustrating an example of the flow of processing executed by the information providing device according to the embodiment;

For example, the information providing device 10 randomly extracts M pieces of data from undistributed learning data (step S101). Subsequently, the information providing apparatus 10 divides the extracted M data into K groups, and distributes the data of each group to different arithmetic units (step S102). Then, the information providing device 10 determines whether or not a predetermined learning end condition is satisfied (step S103).

Here, when the information providing apparatus 10 determines that the learning end condition is not satisfied (step S103: No), it determines whether the objective function of each model satisfies a predetermined synchronization condition (step S104), If it is determined that the condition is satisfied (step S104: Yes), a model is generated by synchronizing each model with a weight according to the improvement value of the objective function of each model (step S105). Then, the information providing device 10 distributes a new model to each arithmetic unit (step S106), and executes step S101 again. Further, when the information providing apparatus 10 determines that the objective function of each model does not satisfy the predetermined synchronization condition (step S104: No), it also executes step S101 again.

Then, when the information providing apparatus 10 determines that the learning end condition is satisfied (step S103: Yes), it generates a learning model in which the models of the arithmetic units are synchronized (step S107), and ends the process.

[4. Modification]
An example of processing by the information providing apparatus 10 has been described above. However, embodiments are not so limited. Variations of processing executed by the information providing apparatus 10 will be described below.

[4-1. Device configuration〕
Each database 31, 32 registered in the storage unit 30 may be held in an external storage server, or may be stored in various storage devices individually held by the first calculation unit 40 and the second calculation unit 50. may be retained. Further, the information providing apparatus 10 does not need to have the second calculation unit 50 inside the housing, and may have it inside an external housing, for example. Further, the information providing device 10 may have a plurality of second calculation units 50 and perform learning processing using the calculation devices of the second calculation units 50 in an integrated manner.

[4-2. others〕
Further, among the processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, and conversely, the processes described as being performed manually can be performed manually. can also be performed automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

Moreover, each of the embodiments described above can be appropriately combined within a range that does not contradict the processing contents.

[4-3. program〕
Also, the information providing apparatus 10 according to the above-described embodiment is implemented by a computer 1000 configured as shown in FIG. 5, for example. FIG. 5 is a diagram illustrating an example of a hardware configuration; The computer 1000 is connected to an output device 1010, an input device 1020, a first arithmetic device 1030, a second arithmetic device 1031, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080. are connected by a bus 1090 .

The first arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The primary storage device 1040 is a memory device, such as a RAM, that temporarily stores data used for various calculations by the first arithmetic device 1030 . The secondary storage device 1050 is a storage device in which data used for various calculations by the first arithmetic device 1030 and various databases are registered. It is implemented by a memory or the like.

The second arithmetic unit 1031 has an arithmetic unit for learning the model described above, that is, a plurality of cores. For example, the second arithmetic unit 1031 is realized by a graphic card or the like in which a GPU is installed.

The output IF 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various types of information such as a monitor and a printer. It is realized by a connector conforming to a standard such as HDMI (registered trademark) (High Definition Multimedia Interface). Also, the input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, scanner, etc., and is realized by, for example, USB.

Note that the input device 1020 includes, for example, optical recording media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), PDs (Phase change rewritable discs), magneto-optical recording media such as MOs (Magneto-Optical discs), and tapes. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. Also, the input device 1020 may be an external storage medium such as a USB memory.

The network IF 1080 receives data from other equipment via the network N and sends the data to the first arithmetic unit 1030, and also transmits data generated by the first arithmetic unit 1030 to other equipment via the network N.

The first arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070 . For example, the first arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 10, the first arithmetic device 1030 of the computer 1000 performs the functions of the first arithmetic unit 40 by executing the program or data loaded on the primary storage device 1040. The second arithmetic device 1031 operates as the second arithmetic unit 50 by executing the program or data loaded on the primary storage device 1040 . The first arithmetic unit 1030 and the second arithmetic unit 1031 of the computer 1000 read and execute these programs or data from the primary storage device 1040. As another example, these programs are read from other devices via the network N. may be obtained.

[5. effect〕
As described above, the information providing device 10 distributes different learning data to a plurality of computing devices that individually perform model learning using the distributed learning data. Then, the information providing device 10 synchronizes the model learned by each computing device in a mode according to the result of learning executed by each computing device using the distributed learning data. As a result of such processing, the information providing apparatus 10 realizes synchronization that places importance on the model that has undergone more appropriate learning, so that the learning accuracy of the model using a plurality of arithmetic units can be improved.

Further, the information providing device 10 synchronizes each model in a mode according to the value of the objective function of the model trained by each arithmetic device. For example, the information providing device 10 considers the weight according to the value of the objective function of each model, and takes a model obtained by integrating each model as a synchronization result. Further, for example, the information providing apparatus 10 considers the weight according to the amount of improvement between the value of the objective function before learning of each model and the value of the objective function after learning, and creates a model that integrates each model. Synchronize result. Note that, for example, the information providing apparatus 10 may take the model with the largest improvement amount between the value of the objective function before learning and the value of the objective function after learning among the models as the synchronization result. Further, the information providing apparatus 10 may synchronize each model using a genetic algorithm that selects a model according to the value of the objective function. As a result of such processing, the information providing device 10 can perform synchronization with an emphasis on models that are considered to contribute to improvement in accuracy among the models that are individually learned by each computing device. can be improved.

Further, the information providing apparatus 10 corrects the parameter values of the model using the learning data, thereby distributing different learning data to a plurality of computing devices that cause the model to learn the features of the learning data. , to synchronize the values of the parameters of each model. In addition, the information providing device 10 newly distributes different learning data to a plurality of computing devices that perform model learning synchronized by the synchronization unit using the newly distributed learning data. Therefore, the information providing device 10 can improve the learning accuracy of various neural networks.

In addition, the information providing apparatus 10 newly distributes different learning data to each arithmetic unit each time the model is learned by each arithmetic unit, and synchronizes the models trained by the plural arithmetic units. Change opportunities dynamically.

For example, the information providing device 10 synchronizes each model so that the number of times each computing device performs model learning using newly distributed learning data is random. Further, the information providing device 10 synchronizes each model when the result of learning executed by each arithmetic device satisfies a predetermined condition. In addition, the information providing device 10, when the improvement amount between the value of the objective function before learning and the value of the objective function after learning of the model trained by at least one of the arithmetic devices satisfies a predetermined condition. synchronizes each model. Therefore, the information providing apparatus 10 can improve the learning accuracy of the model while preventing an increase in overhead due to synchronization processing.

In addition, the information providing device 10 synchronizes models that have been trained by some of the plurality of computing devices. For example, the information providing apparatus 10 synchronizes a model learned by a plurality of arithmetic devices whose communication delay is within a predetermined range among the plurality of arithmetic devices. Further, for example, the information providing apparatus 10 synchronizes a model that has been learned by some of the arithmetic devices that are randomly selected from among the plurality of arithmetic devices. Therefore, the information providing apparatus 10 can further reduce overhead in synchronization processing.

Further, the information providing device 10 has a plurality of computing devices, the number of computing devices corresponding to the number of dimensions of information that can be calculated by each computing device and the number of learning data distributed to all the computing devices. Synchronize the model trained by Therefore, the information providing device 10 can realize more efficient model learning.

As described above, some of the embodiments of the present application have been described in detail based on the drawings. It is possible to carry out the invention in other forms with modifications.

Also, the "section, module, unit" described above can be read as "means" or "circuit". For example, the detection unit can be read as detection means or a detection circuit.

10 information providing device 20 communication section 30 storage section 31 learning data database 32 model database 40 first calculation section 41 learning control section 42 information provision section 50 second calculation section 51 calculation section 52 calculation control section 521 distribution section 522 synchronization section 100 data Server 200 User terminal

Claims (18)

a distributing unit that newly distributes different learning data each time the model is learned by each arithmetic unit to a plurality of arithmetic units that individually learn a model using the distributed learning data;
a synchronizing unit for synchronizing the model learned by each computing unit when one of the learned models executed by each computing unit using the distributed learning data satisfies a predetermined condition; ,
The information processing apparatus, wherein the synchronization unit synchronizes the models using a genetic algorithm that selects the model according to the value of the objective function of the model trained by each arithmetic unit. 2. The information processing apparatus according to claim 1, wherein the synchronization unit considers a weight according to the value of the objective function of each model, and sets a model obtained by integrating each model as a synchronization result. The synchronization unit considers a weight corresponding to an improvement amount between the objective function value of each model before learning and the objective function value after learning, and sets a model obtained by integrating each model as a synchronization result. 3. The information processing apparatus according to claim 1 or 2, characterized by: 4. The synchronizing unit, among the models, sets the model having the largest improvement between the value of the objective function before learning and the value of the objective function after learning as the synchronizing result. The information processing apparatus according to any one of The distribution unit corrects the parameter values of the model using the learning data, thereby distributing different learning data to a plurality of computing devices that cause the model to learn the features of the learning data. ,
The information processing apparatus according to any one of claims 1 to 4, wherein the synchronization unit synchronizes values of parameters of each model. 3. The distributing unit newly distributes different learning data to a plurality of arithmetic units that perform model learning synchronized by the synchronization unit using the newly distributed learning data. 6. The information processing device according to any one of 1 to 5. a distributing unit that newly distributes different learning data each time the model is learned by each arithmetic unit to a plurality of arithmetic units that individually learn a model using the distributed learning data;
a synchronizing unit for synchronizing the model learned by each computing unit when one of the learned models executed by each computing unit using the distributed learning data satisfies a predetermined condition; ,
The information processing apparatus, wherein the synchronization unit synchronizes each model when a result of learning executed by each arithmetic unit satisfies a predetermined condition. The synchronizing unit, when the improvement amount between the value of the objective function before learning and the value of the objective function after learning of the model trained by at least one of the arithmetic devices satisfies a predetermined condition, The information processing apparatus according to claim 7, wherein the models are synchronized. a distributing unit that newly distributes different learning data each time the model is learned by each arithmetic unit to a plurality of arithmetic units that individually learn a model using the distributed learning data;
a synchronizing unit for synchronizing the model learned by each computing unit when one of the learned models executed by each computing unit using the distributed learning data satisfies a predetermined condition; ,
The synchronizing unit synchronizes the models learned by some of the plurality of arithmetic devices, the models learned by the plurality of arithmetic devices having communication delays within a predetermined range. An information processing device characterized by: a distributing unit that newly distributes different learning data each time the model is learned by each arithmetic unit to a plurality of arithmetic units that individually learn a model using the distributed learning data;
a synchronizing unit for synchronizing the model learned by each computing unit when one of the learned models executed by each computing unit using the distributed learning data satisfies a predetermined condition; ,
The synchronizing unit is a model learned by some of the plurality of arithmetic units, and has a number of dimensions of information that can be calculated by each arithmetic unit and learning distributed to all the arithmetic units. An information processing device characterized by synchronizing a model trained by a number of arithmetic units corresponding to the number of data. An information processing method executed by an information processing device,
a distributing step of distributing different learning data each time the model is learned by each computing device to a plurality of computing devices that individually learn a model using the distributed learning data;
a synchronization step of synchronizing the model learned by each computing device when one of the learned models executed by each computing device using the distributed learning data satisfies a predetermined condition,
The information processing method, wherein the synchronizing step synchronizes each model using a genetic algorithm that selects a model according to the value of the objective function of the model trained by each arithmetic unit. a distribution procedure for distributing different learning data each time the model is learned by each computing device to a plurality of computing devices that individually learn a model using the distributed learning data;
a synchronization procedure for synchronizing the model learned by each computing unit when any of the learned models executed by each computing unit using the distributed learning data satisfies a predetermined condition; An information processing program for execution,
The information processing program, wherein the synchronization procedure synchronizes each model using a genetic algorithm that selects a model according to the value of the objective function of the model trained by each arithmetic unit. An information processing method executed by an information processing device,
a distribution step of newly distributing different learning data each time the model is learned by each computing device to a plurality of computing devices that individually learn a model using the distributed learning data;
a synchronization step of synchronizing the model learned by each computing device when one of the learned models executed by each computing device using the distributed learning data satisfies a predetermined condition,
The information processing method, wherein the synchronizing step synchronizes each model when a result of learning executed by each arithmetic unit satisfies a predetermined condition. a distribution procedure for newly distributing different learning data each time the model is learned by each arithmetic unit to a plurality of arithmetic units that individually learn a model using the distributed learning data;
a synchronization procedure for synchronizing the model learned by each computing unit when any of the learned models executed by each computing unit using the distributed learning data satisfies a predetermined condition; An information processing program for execution,
The information processing program, wherein the synchronization procedure synchronizes each model when a result of learning executed by each arithmetic unit satisfies a predetermined condition. An information processing method executed by an information processing device,
a distribution step of newly distributing different learning data each time the model is learned by each computing device to a plurality of computing devices that individually learn a model using the distributed learning data;
a synchronization step of synchronizing the model learned by each computing device when one of the learned models executed by each computing device using the distributed learning data satisfies a predetermined condition,
The synchronizing step synchronizes the models learned by a part of the plurality of arithmetic devices, and the models learned by the plurality of arithmetic devices whose communication delay is within a predetermined range. An information processing method characterized by: a distribution procedure for newly distributing different learning data each time the model is learned by each arithmetic unit to a plurality of arithmetic units that individually learn a model using the distributed learning data;
a synchronization procedure for synchronizing the model learned by each computing unit when any of the learned models executed by each computing unit using the distributed learning data satisfies a predetermined condition; An information processing program for execution,
The synchronization procedure synchronizes the models learned by some of the plurality of arithmetic units, and the models learned by the plurality of arithmetic units having communication delays within a predetermined range. An information processing program characterized by: An information processing method executed by an information processing device,
a distribution step of newly distributing different learning data each time the model is learned by each computing device to a plurality of computing devices that individually learn a model using the distributed learning data;
a synchronization step of synchronizing the model learned by each computing device when one of the learned models executed by each computing device using the distributed learning data satisfies a predetermined condition,
In the synchronization step, a model learned by a part of the plurality of arithmetic units, the number of dimensions of information that can be calculated by each arithmetic unit, and the learning distributed to all the arithmetic units. An information processing method characterized by synchronizing a model trained by a number of arithmetic units corresponding to the number of data. a distribution procedure for newly distributing different learning data each time the model is learned by each arithmetic unit to a plurality of arithmetic units that individually learn a model using the distributed learning data;
a synchronization procedure for synchronizing the model learned by each computing unit when any of the learned models executed by each computing unit using the distributed learning data satisfies a predetermined condition; An information processing program for execution,
The synchronization procedure is a model learned by a part of the plurality of arithmetic units, the number of dimensions of information that can be calculated by each arithmetic unit, and the learning distributed to all the arithmetic units. An information processing program characterized by synchronizing a model trained by a number of arithmetic units corresponding to the number of data.

Images