JP7159884B2 - Information processing device and information processing method - Google Patents

Description

The present disclosure relates to an information processing device and an information processing method.

In recent years, attention has been paid to neural networks, which are mathematical models that imitate the mechanisms of the nervous system. Also, many techniques for speeding up learning by neural networks have been proposed. For example, Non-Patent Document 1 discloses a technique for changing the batch size during learning.

Samuel L. Smith, 3 others, "Don't Decay the Learning Rate, Increase the Batch Size", November 1, 2017, [Online], [searched September 7, 2018], Internet <https: //arxiv.org/pdf/1711.00489.pdf>

However, the technology described in Non-Patent Document 1 depends on a specific learning method, and is difficult to apply to learning that does not employ that method.

According to the present disclosure, an acquisition unit that acquires a gap value from an ideal state related to learning by a neural network, and based on the gap value from the ideal state, dynamically change the batch size value in the neural network. An information processing device is provided, comprising: an instruction unit for instructing.

In addition, according to the present disclosure, the processor acquires a gap value from an ideal state related to learning by the neural network, and based on the gap value from the ideal state, changes the batch size value in the neural network. A method of processing information is provided, comprising:

FIG. 10 is a diagram showing an example of loss transition when Step Learning rate decay is applied; FIG. 5 is a diagram for explaining an overview of batch size change according to an embodiment of the present disclosure; FIG. 2 is a block diagram showing an example of the functional configuration of the information processing device according to the same embodiment; FIG. FIG. 10 is a diagram showing verification results when batch size change based on the gradient of loss according to the embodiment is applied to ImageNet/ResNet-50; FIG. 10 is a diagram showing verification results when batch size change based on training values according to the same embodiment is applied to ImageNet/ResNet-50; FIG. 10 is a diagram showing verification results when batch size change based on loss according to the same embodiment is applied to learning using MNIST; It is a figure which shows the verification result at the time of applying the batch size change based on the loss which concerns on the same embodiment to learning using cifar10. It is a figure which shows the verification result at the time of applying the batch size change based on the loss which concerns on the same embodiment to learning using cifar10. It is a figure which shows an example of the training script which implement|achieves the change of the batch size based on the 1 time differential value of the loss, and a loss inclination calculation module which concerns on the same embodiment. FIG. 10 is a diagram showing a verification result when increasing the batch size for each epoch according to the same embodiment is applied to learning using MNIST; FIG. 10 is a diagram showing a verification result when batch size change based on loss and epoch according to the same embodiment is applied to learning using cifar10; FIG. 10 is a diagram showing an example of a training script for increasing or decreasing the batch size based on losses and epochs according to the embodiment; FIG. 5 is a diagram for explaining recreating of a GPU medium model by a batch size changing unit according to the embodiment; It is a figure for demonstrating increase/decrease control of the number of calculation loops by the batch size change part which concerns on the same embodiment. FIG. 5 is a diagram for explaining increase/decrease control of GPUs in use by a batch size changing unit according to the embodiment; It is a flowchart which shows the flow of control by the batch size change part which concerns on the same embodiment. 1 is a diagram illustrating a hardware configuration example according to an embodiment of the present disclosure; FIG.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

Note that the description will be given in the following order.
1. Embodiment 1.1. Overview 1.2. Functional Configuration Example of Information Processing Apparatus 10 1.3. Verification result 1.4. 2. Realization method of batch size increase/decrease. Hardware configuration example 3 . summary

<1. embodiment>
<<1.1. Overview＞＞
First, an outline of an embodiment of the present disclosure will be described. As described above, in recent years, many techniques for speeding up learning by neural networks have been proposed. In general, the time required for DNN (Deep Neural Network) learning is proportional to the number of parameter updates. Therefore, reducing the number of updates can be an effective means for speeding up learning.

The number of parameter updates can be reduced, for example, by increasing the batch size. Also, in the second half of learning, it is known that learning converges even if the batch size is increased. By setting a large batch size, it is possible to reduce the number of parameter updates, which is expected to have the effect of speeding up learning.

However, the method of changing the batch size described in Non-Patent Document 1 is a technology applicable only to a specific learning method. Here, the above specific learning method refers to a method called Step Learning rate decay.

FIG. 1 is a diagram showing an example of transition of loss when Step Learning rate decay is applied. Step Learning rate decay is a method of decreasing the loss stepwise by decreasing the learning rate stepwise, as shown in FIG. Referring to the example shown in FIG. 1, it can be seen that the loss drops significantly near epochs 30 and 60 and the graph has a staircase shape.

According to the technique described in Non-Patent Document 1, it is possible to change the batch size at timings such as epochs 30 and 60 when the loss drops significantly, but the loss transition shows a stepped shape as described above. It cannot be applied to learning methods that do not exist.

The technical idea according to the present disclosure was conceived by paying attention to the above points, and makes it possible to effectively speed up learning by DNN regardless of the learning method. For this reason, the information processing apparatus 10 according to an embodiment of the present disclosure includes a learning unit 120 that performs learning using a neural network. One of the features is to dynamically change the batch size value during learning based on the value.

Here, the gap value from the ideal state may be an index that quantitatively expresses the difference between the expected output and the actual output. The gap value from the ideal state according to this embodiment includes, for example, loss. Also, the gap value from the ideal state according to the present embodiment may include training errors and validation errors.

Examples of the training error and variation error used as the gap value include, for example, the mean square error (MSE) and the mean absolute error (MAE), which may be used as a loss, and the image Top-k-error (especially top-1-error, top-5-error, etc.) used in classification, mAP (mean Average Precision) used in object detection, and the like.

Here, with reference to FIG. 2, an outline of batch size change according to the present embodiment will be described. FIG. 2 shows a graph representing changes in loss over the course of epochs. In each graph shown in FIG. 2 and subsequent figures, the solid line indicates the loss transition without changing the batch size (Reference), and the dashed line indicates the loss transition (Approach) to which the batch size change according to the present embodiment is applied. ing.

The learning unit 120 according to the present embodiment may increase the value of the batch size during learning, for example, when convergence of learning is estimated based on loss.

A smaller loss value indicates that the DNN is approaching a solution, that is, learning is converging (learning is stable). Therefore, the learning unit 120 according to the present embodiment may increase the batch size value during learning based on the n-th differential value of the loss.

For example, the learning unit 120 according to the present embodiment can increase the value of the batch size when the first derivative of the loss, that is, the slope is below a predetermined threshold. In the example shown in FIG. 2, the learning unit 120 increases the batch size value from 32K to 64K at the timing T1 (epoch 30) when the slope of the loss settles down.

Further, for example, the learning unit 120 according to the present embodiment can also increase the value of the batch size when the 0th differential value of loss, that is, the loss value itself is below a predetermined threshold. Here, if the above threshold is 0.3, the learning unit 120 may increase the batch size value at timing T2 (epoch 60) when the loss value falls below 0.3. Note that the learning unit 120 may increase the value of the batch size based on the n-times differentiation value of n>2 or more.

Here, when Approach and Reference are compared, it can be seen that learning does not diverge and performance is maintained even when the batch size changing method according to the present embodiment is applied. That is, according to the batch change method realized by the information processing apparatus 10 according to the present embodiment, it is possible to achieve both securing of learning performance and reduction of the number of parameter updates, that is, shortening of the learning time.

Further, according to the batch change method according to the present embodiment, even in a learning method in which the change in loss does not show a staircase shape as shown in FIG. 2, it is possible to increase the batch size and shorten the learning time. It becomes possible. As described above, according to the information processing apparatus 10 according to the present embodiment, it is possible to effectively speed up the learning by DNN regardless of the learning method.

<<1.2. Functional Configuration Example of Information Processing Device 10>>
Next, a functional configuration example of the information processing apparatus 10 according to this embodiment will be described. FIG. 3 is a block diagram showing a functional configuration example of the information processing apparatus 10 according to this embodiment. Referring to FIG. 3 , the information processing apparatus 10 according to this embodiment includes an input/output control section 110 , a learning section 120 , a differential calculation section 130 and a batch size changing section 140 .

(Input/output control unit 110)
The input/output control unit 110 according to this embodiment controls a user interface related to DNN learning. For example, the input/output control unit 110 according to this embodiment hands over various data input via an input device to the learning unit 120 . Also, for example, the input/output control unit 110 delivers the value output by the learning unit 120 to the output device.

(Learning unit 120)
The learning unit 120 according to this embodiment performs learning using DNN. As described above, the learning unit 120 according to the present embodiment is characterized by dynamically changing the batch size value during learning based on the gap value between the ideal state related to learning output by the DNN. be one. The gap value from the ideal state according to this embodiment includes, for example, loss, training error, validation error, and the like.

(Differential calculator 130)
The differential calculation unit 130 according to the present embodiment performs n-time differential processing on the loss input from the learning unit 120 to calculate an n-time differential value, and outputs the n-time differential value to the learning unit 120 .

(Batch size changing unit 140)
The batch size changing unit 140 according to the present embodiment has a function of controlling increase/decrease of the batch size based on the batch size value set by the learning unit 120 . Details of the functions of the batch size changing unit 140 according to this embodiment will be described separately later.

The functional configuration example of the information processing apparatus 10 according to the present embodiment has been described above. Note that the above configuration described using FIG. 3 is merely an example, and the functional configuration of the information processing apparatus 10 according to the present embodiment is not limited to the example. The functional configuration of the information processing apparatus 10 according to this embodiment can be flexibly modified according to specifications and operations.

<<1.3. Verification result >>
Next, verification results of the batch size changing method realized by the information processing apparatus 10 according to the present embodiment will be described.

First, the verification results when ImageNet is used as the data set and ResNet-50 is used as the DNN will be described. FIG. 4 is a diagram showing verification results when batch size change based on the gradient of loss according to the present embodiment is applied to ImageNet/ResNet-50.

Here, learning was performed with the batch size in Reference fixed at 32K. On the other hand, in Approach, the batch size was increased from 32K to 68K at timing T3 (epoch 30) when the first derivative of the loss, that is, the slope fell below the threshold, and learning was continued.

Comparing Reference and Approach, it can be seen that even when the batch size is increased by the batch change method according to the present embodiment, loss convergence is not significantly affected.

FIG. 5 is a diagram showing verification results when applying batch size change based on training values according to the present embodiment to ImageNet/ResNet-50.

Here, the batch size was increased from 2K to 20K at timing T4 (epoch 30) when the 0th differential value of the training error fell below the threshold value of 1.8, and learning was continued.

Referring to FIG. 5, it can be seen that learning converges without any effect even when the batch size is increased based on the 0th derivative of the training error.

Next, the verification results when using MNIST for the data set will be described. FIG. 6 is a diagram showing a verification result when the loss-based batch size change according to the present embodiment is applied to learning using MNIST.

Here, the batch size in Reference was fixed at 128, and learning was performed. On the other hand, in Approach, the batch size is increased from 128 to 3072 at timing T5 (epoch 1) when the 1st derivative of the loss falls below the threshold and the 0th derivative of the loss falls below the threshold of 0.03. continued.

As a result of the above control, the number of parameter updates can be reduced from 2000 to 560, and the learning time can be greatly shortened.

Next, the verification results when cifar10 is used for the data set will be described. 7 and 8 are diagrams showing verification results when applying loss-based batch resizing according to the present embodiment to learning using cifar10.

In the verification according to FIG. 7, the batch size in Reference was fixed at 64 and learning was performed. On the other hand, in Approach, the batch size is increased from 64 to 1024 at timing T6 (Epoch 5) when the 1st derivative of the loss falls below the threshold and the 0th derivative of the loss falls below the threshold of 0.35. continued.

As a result of the above control, the number of parameter updates can be reduced from 20000 to 5000, and the learning time can be greatly shortened.

Moreover, in the verification according to FIG. 8, learning was performed with the batch size in Reference fixed at 64, as in the verification according to FIG. On the other hand, in Approach, the batch size was increased from 64 to 1024 at timing T7 (epoch 8) when the 0th differential value of the loss fell below the threshold value of 0.35, and learning was continued.

As a result of the above control, the number of parameter updates can be reduced from 20000 to 7250, and the learning time can be greatly shortened.

The verification results of the batch size changing method according to the present embodiment have been described above. The verification results described above show that when the batch size changing method according to the present embodiment is applied, the number of parameter updates can be reduced by about 1/3 to 1/4 with almost no impact on performance. As described above, according to the information processing apparatus 10 according to the present embodiment, it is possible to effectively speed up the learning by DNN regardless of the learning method.

It should be noted that changing the batch size based on the first derivative of the loss can be realized, for example, by a training script TS1 and a loss slope calculation module CM as shown in FIG. In addition, in FIG. 9, codes are shown in a pseudo manner.

In the case of the example shown in FIG. 9, the training script TS1 first executes the process of calling the loss slope acquisition API, that is, the loss slope calculation module CM, and acquires the current loss_grad value as a return value.

Next, a process of comparing the obtained value of loss_grad with a threshold is performed, and if the value of loss_grad is below the threshold, a process of increasing the batch size is performed.

In the training script TS1, the above processes are repeatedly executed until learning converges.

In addition, when called by the training script TS1, the loss gradient calculation module CM saves the retained value of loss to loss_prev, and calculates loss_grad by obtaining the difference between the newly obtained loss and loss_prev. At this time, the loss slope calculation module CM may take a moving average of losses and remove noise, as shown in the figure.

In the above description, the case of increasing the batch size has been described as a main example. It is also possible to decrease the value.

For example, in the case of the example shown in FIG. 5, the learning is still unstable in the period D1, so the learning unit 120 maintains the small batch size value given as the initial value. On the other hand, when learning stabilizes in period D2, the learning unit 120 may increase the value of the batch size.

However, in step learning rate decay, since the loss generally drops significantly immediately after the learning rate is lowered, it is assumed that the learning diverges more easily than before the learning rate is lowered. Therefore, the learning unit 120 according to the present embodiment can converge learning by decreasing the value of the batch size, which is once increased in the period D2, in the period D3. At this time, the learning unit 120 may set a batch size value between the period D1 and the period D2, for example.

Next, changing the batch size based on the epoch according to this embodiment will be described. In learning by DNN, if the learning rate does not decay, there is a strong tendency for learning to become easier as learning progresses, that is, as epochs overlap. Therefore, the learning unit 120 according to this embodiment can increase the value of the batch size as epochs elapse. For example, the learning unit 120 according to this embodiment may increase the value of the batch size for each epoch.

FIG. 10 is a diagram showing a verification result when increasing the batch size for each epoch according to this embodiment is applied to learning using MNIST. Here, the initial batch size is set to 128, the batch size is set to 256 in epoch 1 (timing T8), the batch size is set to 512 in epoch 2 (timing T9), and the batch size is set to epoch 3 (timing T10). 1024 were controlled to double each.

As a result of the above control, the number of parameter updates was reduced from 2000 to 938. According to the verification results, it can be seen that even if the batch size is increased for each epoch, the learning time can be significantly reduced without significantly affecting loss convergence.

In addition, the learning unit 120 according to the present embodiment, as a result of increasing the batch size value based on the loss and epoch, when learning divergence is estimated, the network model before divergence, that is, in the immediately preceding epoch It is also possible to attempt convergence of learning by reloading.

FIG. 11 is a diagram showing a verification result when batch size change based on loss and epoch according to this embodiment is applied to learning using cifar10.

Here, 64 was set as the initial value of the batch size, and the threshold value of the 0th differential value of the loss was set to 0.35. In the case of the example shown in FIG. 11, batch size increase processing was started based on the fact that the loss fell below the threshold value of 0.35 at epoch 8 (timing T11), and thereafter the batch size was increased for each epoch.

After that, when the batch size was increased to 4K at epoch 14 (timing T12), divergence of learning was estimated. Therefore, the learning unit 120 according to the present embodiment stops increasing the batch size value at the start of epoch 15, reloads the model at the start of epoch 14, and fixes the batch size value to 2K. and continued learning.

In this way, the learning unit 120 according to the present embodiment may set a smaller batch size value than the value set in the past epoch when reloading the network model in the past epoch.

According to the above function of the learning unit 120 according to the present embodiment, the batch size value can be automatically increased or decreased based on the loss or epoch, and the number of parameter updates can be effectively reduced while avoiding learning divergence. can be reduced to

It should be noted that the increase and decrease of the batch size based on loss and epoch as described above can be realized by, for example, a training script TS2 as shown in FIG. Note that FIG. 12 shows codes in a pseudo manner.

In the example shown in FIG. 12, the training script TS2 first compares loss_grad obtained as a return value from calling the loss gradient calculation module CM shown in FIG. 9 with a threshold. Now, if loss_grad is below the threshold, the training script TS2 starts auto-increasing the batch size.

The training script TS2 then determines whether the loss is greater than the previous epoch by more than a threshold. Now, if an increase in loss is observed, the training script TS2 stops automatically increasing the batch size.

Also, at this time, the training script TS2 reloads the network model of the DNN in the previous epoch.

<<1.4. Realization method of batch size increase/decrease >>
Subsequently, a method for realizing batch size increase/decrease according to the present embodiment will be described in detail. The batch size changing unit 140 according to the present embodiment acquires the value of the batch size set by the learning unit 120, and controls the GPU (Graphics Processing Unit) based on the value to increase or decrease the batch size. .

For example, the batch size changing unit 140 according to this embodiment may control the increase or decrease of the batch size by recreating the model in the GPU.

FIG. 13 is a diagram for explaining recreating the GPU medium model by the batch size changing unit 140 according to the present embodiment. In this case, first, the learning unit 120 inputs the value of the loss to the differential calculation unit 130, and obtains the n-th partial value of the value. In addition, the learning unit 120 determines the value of the batch size after change based on the acquired n-times partial value, and inputs the value of the batch size to the batch size changing unit 140 .

Next, the batch size changing unit 140 according to the present embodiment instructs the GPU currently used for learning to recreate the model based on the input batch size value.

In FIG. 13, among GPU_0 and GPU_1, GPU_0 is currently used for learning, and when the batch size of the model in GPU_0 is 32, the batch size changing unit 140 instructs GPU_0 to recreate the model. and the batch size of the model is changed to 64.

According to the above control by the batch size changing unit 140 according to the present embodiment, the batch size can be increased globally without being affected by the number of GPUs that the information processing apparatus 10 has, and the parallel computing capability of the GPUs can be utilized. The effect leading to further speedup is also described in .

Further, for example, the batch size changing unit 140 according to the present embodiment may control increase/decrease of the batch size by increasing/decreasing the number of loops of calculation related to learning. Techniques such as the above are also referred to as accum-grad.

FIG. 14 is a diagram for explaining increase/decrease control of the number of calculation loops by the batch size changing unit 140 according to this embodiment. In this case, the batch size changing unit 140 according to the present embodiment instructs the GPU currently used for learning to change the number of calculation loops based on the batch size value input from the learning unit 120 .

In FIG. 14, among GPU_0 and GPU_1, GPU_0 is currently used for learning, and when the batch size of the model in GPU_0 is 32, the batch size changing unit 140 performs accum-grad to GPU_0 twice. An example of a case where the instruction is given and learning with a batch size of 32 is performed twice is shown.

According to the above control by the batch size changing unit 140 according to the present embodiment, the batch size can be increased without being restricted by the number of GPUs or the memory capacity, and the number of synchronization processes is reduced. The speed of learning can be increased by the number of times of processing.

Also, for example, the batch size changing unit 140 according to the present embodiment may control increase/decrease of the batch size by increasing/decreasing the number of GPUs used for learning.

FIG. 15 is a diagram for explaining increase/decrease control of used GPUs by the batch size changing unit 140 according to the present embodiment. In this case, the batch size changing unit 140 according to the present embodiment instructs GPUs not currently used for learning to operate based on the batch size value input from the learning unit 120 .

Note that FIG. 15 shows an example in which the batch size changing unit 140 instructs GPU_1 to operate when only GPU_0 of GPU_0 and GPU_1 is currently used for learning.

According to the above control by the batch size changing unit 140 according to the present embodiment, learning can be speeded up by increasing computational resources.

The batch size change control method by the batch size change unit 140 according to the present embodiment has been described above. Note that the batch size change unit 140 according to the present embodiment selects a batch size change control method based on the priority shown in FIG. can do.

FIG. 16 is a flow chart showing the flow of control by the batch size changer 140 according to this embodiment.

Referring to FIG. 16, the batch size changing unit 140 first determines whether or not there is an additionally usable GPU (S1101).

Here, if there is an additionally usable GPU (S1101: Yes), the batch size changing unit 140 controls the batch size increase by allocating the usable GPU to learning (S1102).

Subsequently, the batch size changing unit 140 determines whether the target batch size has been achieved by the processing in step S1102 (S1103).

Here, if the target batch size has been achieved (S1103: Yes), the batch size changing unit 140 ends the processing related to changing the batch size.

On the other hand, if the desired batch size has not been achieved (S1103: No), or if there is no additionally usable GPU (S1101: No), the batch size changing unit 140 stores It is determined whether or not there is free space (S1104).

Here, if there is free space in the memory of the GPU currently in use (S1104: Yes), the batch size changing unit 140 controls the increase in batch size by recreating the model in the GPU currently in use. (S1105).

Subsequently, the batch size changing unit 140 determines whether the target batch size has been achieved by the processing in step S1105 (S1106).

Here, if the target batch size has been achieved (S1106: Yes), the batch size changing unit 140 ends the processing related to changing the batch size.

On the other hand, if the target batch size has not been achieved (S1106: No), or if there is no free space in the memory of the GPU currently in use (S1104: No), the batch size changing unit 140 performs calculations related to learning. By increasing the number of loops, the batch size increase is controlled (S1107), and the process related to batch size change ends.

<2. Hardware configuration example>
Next, a hardware configuration example of the information processing device 10 according to an embodiment of the present disclosure will be described. FIG. 17 is a block diagram showing a hardware configuration example of the information processing device 10 according to an embodiment of the present disclosure. Referring to FIG. 17, the information processing apparatus 10 includes, for example, a processor 871, a ROM 872, a RAM 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, an input device 878, and an output device 879. , a storage 880 , a drive 881 , a connection port 882 and a communication device 883 . Note that the hardware configuration shown here is an example, and some of the components may be omitted. Moreover, it may further include components other than the components shown here.

(processor 871)
The processor 871 functions as, for example, an arithmetic processing device or a control device, and controls the overall operation of each component or a part thereof based on various programs recorded in the ROM 872, RAM 873, storage 880, or removable recording medium 901. . The processor 871 includes, for example, GPU and CPU. Note that the information processing device 10 according to an embodiment of the present disclosure includes at least two GPUs.

(ROM872, RAM873)
The ROM 872 is means for storing programs to be read into the processor 871, data used for calculation, and the like. The RAM 873 temporarily or permanently stores, for example, programs to be read into the processor 871 and various parameters that change appropriately when the programs are executed.

(Host Bus 874, Bridge 875, External Bus 876, Interface 877)
The processor 871, ROM 872, and RAM 873 are interconnected via, for example, a host bus 874 capable of high-speed data transmission. On the other hand, the host bus 874 is connected, for example, via a bridge 875 to an external bus 876 with a relatively low data transmission speed. External bus 876 is also connected to various components via interface 877 .

(input device 878)
For the input device 878, for example, a mouse, keyboard, touch panel, button, switch, lever, or the like is used. Furthermore, as the input device 878, a remote controller (hereinafter referred to as a remote controller) capable of transmitting control signals using infrared rays or other radio waves may be used. The input device 878 also includes a voice input device such as a microphone.

(output device 879)
The output device 879 is, for example, a display device such as a CRT (Cathode Ray Tube), LCD, or organic EL, an audio output device such as a speaker, headphones, a printer, a mobile phone, a facsimile, or the like, and outputs the acquired information to the user. It is a device capable of visually or audibly notifying Output devices 879 according to the present disclosure also include various vibration devices capable of outputting tactile stimuli.

(storage 880)
Storage 880 is a device for storing various data. As the storage 880, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like is used.

(Drive 881)
The drive 881 is, for example, a device that reads information recorded on a removable recording medium 901 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, or writes information to the removable recording medium 901 .

(Removable recording medium 901)
The removable recording medium 901 is, for example, DVD media, Blu-ray (registered trademark) media, HD DVD media, various semiconductor storage media, and the like. Of course, the removable recording medium 901 may be, for example, an IC card equipped with a contactless IC chip, an electronic device, or the like.

(Connection port 882)
The connection port 882 is, for example, a USB (Universal Serial Bus) port, an IEEE1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or a port for connecting an external connection device 902 such as an optical audio terminal. be.

(External connection device 902)
The external connection device 902 is, for example, a printer, a portable music player, a digital camera, a digital video camera, an IC recorder, or the like.

(Communication device 883)
The communication device 883 is a communication device for connecting to a network. subscriber line) or a modem for various communications.

<3. Summary>
As described above, the information processing apparatus 10 according to an embodiment of the present disclosure includes the learning unit 120 that performs learning using a neural network. One of the features is to dynamically change the batch size value during learning based on the gap value of . According to such a configuration, it is possible to effectively speed up learning by DNN regardless of the learning method.

Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is obvious that those who have ordinary knowledge in the technical field of the present disclosure can conceive of various modifications or modifications within the scope of the technical idea described in the claims. is naturally within the technical scope of the present disclosure.

Also, the effects described herein are merely illustrative or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification, in addition to or instead of the above effects.

It is also possible to create a program for causing hardware such as a CPU, ROM, and RAM built into a computer to exhibit functions equivalent to those of the configuration of the information processing apparatus 10. A possible non-transitory recording medium may also be provided.

Further, each step related to the processing of the information processing apparatus 10 of this specification does not necessarily have to be processed in chronological order according to the order described in the flowchart. For example, each step related to the processing of the information processing device 10 may be processed in an order different from the order described in the flowchart, or may be processed in parallel.

Note that the following configuration also belongs to the technical scope of the present disclosure.
(1)
an acquisition unit that acquires a gap value between an ideal state and an ideal state related to learning by a neural network;
an instruction unit that instructs dynamic change of the batch size value in the neural network based on the gap value from the ideal state;
comprising
Information processing equipment.
(2)
The gap value from the ideal state includes at least a loss-related value,
The acquisition unit acquires a value related to the loss,
The instruction unit instructs a dynamic change of the batch size value in the neural network based on the loss-related value.
The information processing device according to (1) above.
(3)
The value related to the loss includes the n-th differential value of the loss (n is an integer of 0 or more, the same applies hereinafter),
The information processing device according to (2) above.
(4)
The n-th differential value of the loss is the differential value of the loss in the time direction,
The information processing device according to (3) above.
(5)
The acquisition unit acquires the value related to the loss via an API,
The instruction unit automatically instructs a change in the batch size value based on the loss-related value.
The information processing apparatus according to any one of (2) to (4) above.
(6)
The instruction unit instructs to increase the value of the batch size when convergence of learning is estimated from the value related to the loss,
The information processing apparatus according to any one of (2) to (5).
(7)
The instruction unit instructs to increase the batch size value based on the value of the n-th differential value of the loss.
The information processing apparatus according to (3) or (4).
(8)
The instruction unit instructs to increase the batch size value based on at least one of the loss value or the loss slope being below a threshold.
The information processing device according to (7) above.
(9)
The instruction unit instructs a decrease in the batch size value when learning divergence is estimated based on the loss-related value.
The information processing apparatus according to any one of (2) to (8).
(10)
The instruction unit instructs reloading of the network model in a past epoch when learning divergence is estimated based on the loss-related value.
The information processing device according to (9) above.
(11)
When the network model is reloaded in the past epoch, the instruction unit causes the batch size value to be set smaller than the value set in the past epoch.
The information processing device according to (10) above.
(12)
The instruction unit instructs to increase or decrease the batch size by recreating the model in the GPU.
The information processing apparatus according to any one of (1) to (11) above.
(13)
The instruction unit instructs to increase or decrease the batch size by increasing or decreasing the number of loops for calculation related to learning,
The information processing apparatus according to any one of (1) to (12) above.
(14)
The instruction unit instructs an increase or decrease in batch size due to an increase or decrease in the number of GPUs used for learning,
The information processing apparatus according to any one of (1) to (13) above.
(15)
If there is an additional available GPU, the instruction unit instructs to increase the batch size by allocating the GPU to learning.
The information processing apparatus according to any one of (1) to (14) above.
(16)
If there is no additionally usable GPU and there is free space in the memory of the GPU currently in use, the instruction unit instructs to increase the batch size by recreating the model in the GPU currently in use. ,
The information processing apparatus according to any one of (1) to (15).
(17)
The instruction unit instructs to increase the batch size by increasing the number of loops for calculation related to learning when there is no free space in the memory of the GPU currently in use.
The information processing apparatus according to any one of (1) to (16) above.
(18)
The gap value with the ideal state includes at least one of training error or validation error,
The information processing device according to (1) above.
(19)
A processor acquiring a gap value between an ideal state and an ideal state related to learning by a neural network;
Directing a dynamic change of a batch size value in the neural network based on the gap value from the ideal state;
including,
Information processing methods.

10 Information Processing Device 110 Input/Output Control Unit 120 Learning Unit 130 Differential Calculation Unit 140 Batch Size Changing Unit

Claims (19)

an acquisition unit that acquires a gap value between an ideal state and an ideal state related to learning by a neural network;
an instruction unit that instructs dynamic change of the batch size value in the neural network based on the gap value from the ideal state;
comprising
Information processing equipment. The gap value from the ideal state includes at least a loss-related value,
The acquisition unit acquires a value related to the loss,
The instruction unit instructs a dynamic change of the batch size value in the neural network based on the loss-related value.
The information processing device according to claim 1 . The value related to the loss includes the n-th differential value of the loss (n is an integer of 0 or more, the same applies hereinafter),
The information processing apparatus according to claim 2. The n-th differential value of the loss is the differential value of the loss in the time direction,
The information processing apparatus according to claim 3. The acquisition unit acquires a value related to the loss via an API,
The instruction unit automatically instructs a change in the batch size value based on the loss-related value.
The information processing apparatus according to claim 2. The instruction unit instructs to increase the value of the batch size when convergence of learning is estimated from the value related to the loss,
The information processing apparatus according to claim 2. The instruction unit instructs to increase the batch size value based on the value of the n-th differential value of the loss.
The information processing apparatus according to claim 3. The instruction unit instructs to increase the batch size value based on at least one of the loss value or the loss slope being below a threshold.
The information processing apparatus according to claim 7. The instruction unit instructs a decrease in the batch size value when learning divergence is estimated based on the loss-related value.
The information processing apparatus according to claim 2. The instruction unit instructs reloading of the network model in a past epoch when learning divergence is estimated based on the loss-related value.
The information processing apparatus according to claim 9 . When the network model is reloaded in the past epoch, the instruction unit causes the batch size value to be set smaller than the value set in the past epoch.
The information processing apparatus according to claim 10. The instruction unit instructs to increase or decrease the batch size by recreating the model in the GPU.
The information processing device according to claim 1 . The instruction unit instructs to increase or decrease the batch size by increasing or decreasing the number of loops for calculation related to learning,
The information processing device according to claim 1 . The instruction unit instructs an increase or decrease in batch size due to an increase or decrease in the number of GPUs used for learning,
The information processing device according to claim 1 . If there is an additional available GPU, the instruction unit instructs to increase the batch size by allocating the GPU to learning.
The information processing device according to claim 1 . If there is no additionally usable GPU and there is free space in the memory of the GPU currently in use, the instruction unit instructs to increase the batch size by recreating the model in the GPU currently in use. ,
The information processing device according to claim 1 . The instruction unit instructs to increase the batch size by increasing the number of loops for calculation related to learning when there is no free space in the memory of the GPU currently in use.
The information processing device according to claim 1 . The gap value from the ideal state includes at least one of training error or validation error,
The information processing device according to claim 1 . A processor acquiring a gap value between an ideal state and an ideal state related to learning by a neural network;
Directing dynamic change of a batch size value in the neural network based on the gap value from the ideal state;
including,
Information processing methods.

Images