JP2023028232A - Learning device and learning method - Google Patents

Abstract

To provide a technique capable of appropriately improving the performance of a student model in the deep layer learning using the knowledge distillation.SOLUTION: Disclosed is an exemplary learning device which trains a student model using a teacher model and includes: an evaluation part for determining the performance difference between the teacher model and the student model; and a change part for changing the teacher model based on the performance difference or changing a weight coefficient at the time of calculating a loss of the student model.SELECTED DRAWING: Figure 3

Description

The present invention relates to a learning device and a learning method.

Conventionally, a technique called knowledge distillation is known as one of the learning techniques of deep learning (see Patent Document 1, for example). In knowledge distillation, a large teacher model with a large number of parameters is used to train a student model with a small number of parameters. By knowledge distillation, it is possible to reduce model performance degradation while reducing model size.

JP 2020-71883 A

In the learning method using knowledge distillation, the performance of the student model is saturated in the latter half of the learning when the performance of the student model approaches that of the teacher model, and the performance improvement effect of knowledge distillation may not be sufficiently obtained. Also, if the performance of the teacher model used is too high, it may not be possible to sufficiently improve the performance of the student model.

Also, in the learning method using knowledge distillation, due to the difference in feature space between the teacher model and the student model at the end of learning, for example, bringing the feature map of the student model closer to the feature map of the teacher model is not possible. It is feared that it will adversely affect the learning of children. There is concern that this adverse effect may deteriorate the performance of the student model.

In view of the above points, it is an object of the present invention to provide a technique capable of appropriately improving the performance of a student model in deep learning using knowledge distillation.

An exemplary learning device according to the present invention is a learning device that trains a student model using a teacher model, and includes an evaluation unit that obtains a performance difference between the teacher model and the student model, and based on the performance difference, and a changing unit that changes the teacher model or changes a weighting factor when calculating the loss of the student model.

An exemplary learning method of the present invention is a learning method that uses a teacher model to train a student model, and changes the teacher model based on a performance difference between the teacher model and the student model.

An exemplary learning method of the present invention is a learning method for training a student model using a teacher model, wherein a loss of the student model is obtained using a loss for learning data and a loss for the teacher model, The student model is trained by changing, at a predetermined timing, a weighting factor that adjusts the balance between the loss for the learning data and the loss for the teacher model.

According to the exemplary invention, it is possible to appropriately improve the performance of a student model in deep learning using knowledge distillation.

Block diagram showing the hardware configuration of the learning device FIG. 3 is a block diagram showing the functional configuration of a processor included in the learning device; Schematic diagram for explaining the configuration for changing the teacher model Schematic diagram showing how the performance of a student model changes when the learning device of the first embodiment is used to train the student model. 3 is a flow chart showing the flow of student model training in the learning device according to the first embodiment. Flowchart showing the flow of student model training in the learning device according to the second embodiment A diagram showing an example where the weighting factor λ is decayed every multiple epochs A diagram showing an example in which the weighting factor λ is attenuated every epoch

Exemplary embodiments of the invention are described in detail below with reference to the drawings.

<1. First Embodiment>
FIG. 1 is a block diagram showing the hardware configuration of a learning device 10 according to the first embodiment of the invention. The learning device 10 is a learning device for machine learning, and uses knowledge distillation as a learning method. Hereinafter, knowledge distillation may be simply referred to as "distillation". The learning device 10 is a learning device that trains a student model using a teacher model. The learning device 10 executes a learning method of training a student model using a teacher model. Specifically, the learning device 10 makes the student model learn using the learning data and the teacher model. In this embodiment, the learning device 10 trains student models using a plurality of teacher models.

In this embodiment, the learning data is learning data with a correct answer label. A teacher model is a large-scale deep neural network with a large number of parameters, and is a trained model. The student model is a small-scale deep neural network with fewer parameters than the teacher model. A student model is a model that is to be learned (trained) from now on.

As shown in FIG. 1 , the learning device 10 includes a processor 11 and a storage section 12 . The storage unit 12 stores teacher models and student models. The teacher model contains learned parameters. The student model contains parameters that are updated by learning. In this embodiment, there are a plurality of teacher models, and the storage unit 12 stores a plurality of teacher models.

The processor 11 exhibits a function of inferring learning data using the teacher model stored in the storage unit 12 and inferring learning data using the student model. The processor 11 functions to update the parameters of the student model so that the error between the inference result of the teacher model and the training result of the student model is reduced. Note that a feature map of the intermediate layer may be used instead of the inference result.

FIG. 2 is a block diagram showing the functional configuration of the processor 11 included in the learning device 10 according to the first embodiment of the invention. The functions of the processor 11 are exhibited by executing arithmetic processing according to a program stored in the storage unit 12 . As shown in FIG. 2, the processor 11 of the present embodiment has functions such as an acquisition unit 110, a first inference unit 111, a second inference unit 112, a learning unit 113, an evaluation unit 114, and a change unit. 115. In other words, learning device 10 includes acquisition unit 110 , first inference unit 111 , second inference unit 112 , learning unit 113 , evaluation unit 114 , and change unit 115 . Acquisition unit 110 acquires learning data.

The first inference unit 111 uses a trained teacher model to infer input learning data. The function of the first inference unit 111 is realized by reading the teacher model stored in the storage unit 12 by the processor 11 . Through inference, an inference result, which is the final result of inference, and a feature map, which is an intermediate output, are obtained. In this embodiment, the learning data is image data, and the first inference unit 111 performs object detection on the input image data. The inference result of the first inference unit 111 includes identification of the type and position of the object. The types of objects are, for example, pedestrians, bicycles, automobiles, and traffic lights. A bounding box is used to specify the position.

The second inference unit 112 uses the student model, which is the target of training by the teacher model, to infer the input learning data. The function of the second inference unit 112 is realized by reading the student model stored in the storage unit 12 by the processor 11 . Through inference, an inference result, which is the final result of inference, and a feature map, which is an intermediate output, are obtained. Note that the parameters of the student model are successively updated by learning. In this embodiment, the learning data is image data as described above, and the second inference unit 112 performs object detection on the input image data. As with the first inference unit 111, the inference result of the second inference unit 112 includes identification of the type and position of the object.

The learning unit 113 obtains the loss (distillation loss Ldstl) of the student model with respect to the teacher model. The loss for the teacher model (distillation loss Ldstl) is the error in the inference result or the error in the intermediate layer feature map (intermediate output) between the teacher model and the student model. This makes it possible to easily obtain the loss function using conventional techniques.

In this embodiment, the distillation loss Ldstl is the error between the feature map of the teacher model and the feature map of the student model. The distillation loss Ldstl may be, for example, the L2 loss represented by Equation (1) below. However, the distillation loss Ldstl may be other than the L2 loss, and may be, for example, the KL divergence representing the loss between the output distributions of the teacher model and the student model.

式（１）において、ｎはサンプル（データ）の総数を示す。ｆ_ｘｉ ^ｓは、サンプルｘｉの生徒モデルの出力を示す。ｆ_ｘｉ ^ｔは、サンプルｘｉの教師モデルの出力を示す。

In Equation (1), n indicates the total number of samples (data). f _xi ^s denotes the output of the student model for sample xi. f _xi ^t denotes the output of the teacher model for sample xi.

Also, the learning unit 113 obtains the loss for the learning data of the student model. Specifically, the learning unit 113 obtains the learning task-specific error (loss) of the student model with respect to the correct label of the learning data. In this embodiment, the learning task is object detection, and the learning task-specific losses Luq include, for example, the classification loss Lcls and the bounding box regression loss Lbbox. The learning task-specific loss Luq of this embodiment is obtained, for example, by the following equation (2).

Note that the following formula (2) is based on the assumption that YOLO, which is an example of an object detection algorithm, is used. In YOLO, the input image is divided into S×S grids, and for each grid, the center coordinates (x, y) and the width and height scales (w, h) of B rectangular regions with a certain aspect ratio , and the probability (C) that an object exists within the rectangular region. Furthermore, when an object exists in each rectangular area, the posterior probability p(c) indicating which class the object belongs to is also predicted. YOLO integrates predictions of rectangular regions, object existence probabilities, and class posterior probabilities into a single loss function.

添え字がない変数は予測値、添え字「truth」が付与された変数が正解を示す。λcoord、λnoobjは係数である。式（２）において、第１項および第２項は、矩形領域の中心座標と大きさに関する損失関数を示す。式（２）において、第３項および第４項は、存在確率に関する損失関数を示す。式（２）において、第５項はクラスの事後確率に関する損失関数を示す。

Variables without subscripts indicate predicted values, and variables with the subscript "truth" indicate correct answers. λcoord and λnoobj are coefficients. In Equation (2), the first and second terms represent loss functions related to the center coordinates and size of the rectangular area. In Equation (2), the third and fourth terms represent loss functions related to existence probability. In equation (2), the fifth term indicates the loss function for the posterior probability of the class.

The learning unit 113 minimizes the loss L calculated using the loss of the student model to the teacher model (distillation loss Ldstl) and the loss of the student model to the learning data (learning task specific loss Luq). learn the model. Specifically, the learning unit 113 updates the parameters of the student model using backpropagation.

Note that the loss L calculated using the loss for the teacher model and the loss for the learning data may be calculated, for example, by the following equation (3). In Equation (3), λ is a weighting factor. The weighting factor λ is a factor that adjusts the balance between the loss to the training data and the loss to the teacher model. In this embodiment, the weighting factor λ is a constant.
L = Luq + λ·Ldstl (3)

The evaluation unit 114 obtains a performance difference (performance difference) between the teacher model and the student model. As described above, in this embodiment, the performance is object detection performance. According to this embodiment, it is possible to generate a high-performance trained model capable of performing object detection processing at high speed using knowledge distillation.

Specifically, the evaluation unit 114 calculates the performance difference using mAP (mean average precision), which is an evaluation index for the object detection task. That is, the evaluation unit 114 obtains the mAP of the student model and calculates the difference from the mAP of the teacher model. Note that the mAP of the teacher model may be calculated each time or may be stored in the storage unit 12 in advance. If the learning task is not object detection, a performance evaluation index different from mAP may be used. For example, if the learning task is image segmentation, mean intersection over union (mIoU) may be used as a performance metric.

The changing unit 115 changes the teacher model based on the performance difference between the teacher model and the student model. FIG. 3 is a schematic diagram for explaining the configuration for changing the teacher model. As shown in FIG. 3, in this embodiment, the number of teacher models prepared in advance is not one but a plurality. In FIG. 3, N>1. A plurality of teacher models have different performances. The teacher model used to train the student model is sequentially changed according to the performance of the teacher model and the student model.

With such a configuration, when the performance of the student model approaches the performance of the teacher model, it is possible to train the student model by changing to a teacher model with higher performance. For this reason, the performance of the student model can be improved compared to the case where the student model is trained with only one teacher model. In addition, the performance of the teacher model can be gradually changed according to the performance of the student model, and the student model can be appropriately trained.

In FIG. 3, as the teacher model number increases, the performance of the teacher model increases. That is, in the example shown in FIG. 3, when the performance difference between the teacher model and the student model becomes smaller than a predetermined threshold, the changing unit 115 changes the teacher model used for training the student model to the current teacher model. Change to a teacher model with higher performance than the model. The predetermined threshold is an arbitrary value determined by trial and error, for example.

FIG. 4 is a schematic diagram showing how the performance of a student model changes when training is performed using the learning device 10 according to the first embodiment of the present invention. The horizontal axis in FIG. 4 is the iteration, and more specifically, the number of times the parameters of the student model have been updated. The vertical axis in FIG. 4 represents performance, for example, the detection rate of the object to be detected.

As shown in FIG. 4, when the performance of the student model approaches the performance of the teacher model 1 by training using the teacher model 1, which is the first teacher model, the performance of the teacher model from the teacher model 1 is higher than that of the teacher model 1. Training is performed with a change to the high supervised model 2. Then, when the performance of the student model approaches the teacher model 2, the teacher model is changed from the teacher model 2 to the teacher model 3 having higher performance than the teacher model 2, and training is performed. Such repetition is repeated by the number of teacher models prepared in advance (N).

Thus, in this configuration, the training of the student model is started using the teacher model having the performance corresponding to the performance of the student model, and the performance of the teacher model used for training is gradually increased in accordance with the improvement of the performance of the student model. It is configured to be raised. For this reason, the performance of the student model can be appropriately improved compared to training the student model using a high-performance teacher model from the beginning. Moreover, according to this configuration, it is possible to train the student model efficiently.

FIG. 5 is a flow chart showing the flow of training a student model in the learning device 10 according to the first embodiment of the present invention. Note that N (N>1) trained teacher models are created before the flow chart shown in FIG. 5 is started. A known method may be used as a learning method for learning the teacher model.

In step S1, a variable "i" corresponding to the teacher model number used for learning is set to zero. The N teacher models are assigned numbers 0, 1, . . . When the process of step S1 is completed, the process proceeds to the next step S2.

In step S2, a variable "epoch" representing the number of epochs is set to zero. Note that the maximum number of epochs is set to M. The M epoch is, for example, 100 epochs. When the process of step S2 is completed, the process proceeds to the next step S3.

In step S3, learning of the student model is performed using the i-th teacher model. The learning is learning using the knowledge distillation described above. Here, learning for one epoch is performed. Specifically, the learning data set is divided into a plurality of subsets according to the batch size, and learning using knowledge distillation is performed for each subset (every one iteration). Inference by the i-th teacher model and inference by the student model are performed for each subset. Then, for each subset, the parameters of the student model are updated using the i-th teacher model and the loss function (see equation (3)) obtained by inference using the student model. Completion of learning for all subsets means completion of learning for one epoch. When the process of step S3 is completed, the process proceeds to the next step S4.

In step S4, the performance of the student model trained in step S3 is evaluated. In this embodiment, mAP, which is an evaluation index of the object detection task, is obtained for the student model after learning in step S3. After obtaining the mAP, the process proceeds to the next step S5.

In step S5, it is determined whether or not the performance difference between the teacher model and the student model is less than a threshold. Specifically, it is determined whether or not the difference between the mAP of the teacher model and the mAP of the student model is less than a threshold. The threshold is a constant value stored in the storage unit 12 in advance. If the performance difference is greater than or equal to the threshold (No in step S5), the process proceeds to next step S6. If the performance difference is less than the threshold (Yes in step S5), the process proceeds to step S8. That is, the processing of steps S6 and S7 is skipped.

In step S6, 1 is added to the variable "epoch". If the variable "epoch" was zero at this time, the variable "epoch" becomes one. That is, one epoch of learning (knowledge distillation) using the i-th teacher model is completed. Also, if the variable “epoch” was 50, the variable “epoch” becomes 51. That is, 51 epochs of learning using the i-th teacher model have been completed. When the process of step S6 is completed, the process proceeds to the next step S7.

In step S7, it is checked whether the variable "epoch" is smaller than the maximum number M of epochs. If the variable "epoch" is smaller than the maximum number of epochs M (Yes in step S7), the process returns to step S3. That is, learning using the i-th teacher model is repeated. If the variable "epoch" has reached the maximum number of epochs M (No in step S7), the process proceeds to the next step S8.

In step S8, 1 is added to the variable "i". That is, the teacher model used for learning is set to the i-th teacher model to the (i+1)-th teacher model. For example, if i=0, i=1, and teacher model 0 is changed to teacher model 1. FIG. When the process of step S8 is completed, the process proceeds to the next step S9.

In step S9, it is checked whether the variable "i" is smaller than N. That is, it is checked whether or not all teacher models prepared in advance have been used for training the student models. If the variable "i" is smaller than N (Yes in step S9), the process returns to step S2 because not all teacher models have been used for training student models. This initiates training of the student model by knowledge distillation using a new teacher model with higher performance than the teacher model used in the previous learning. Note that the threshold in the process of step S5 may also be changed in accordance with the change of the teacher model. If the variable "i" has reached N (No in step S9), training (learning) of the student model shown in FIG. 5 is completed, and a new learned (trained) model is completed.

The trained student model obtained by the process of FIG. 5 can acquire performance close to that of the teacher model with the highest performance among the teacher models prepared in advance. In addition, the trained model has fewer parameters than the teacher model, and can be processed quickly.

<2. Second Embodiment>
Next, the learning device of the second embodiment will be described. Similarly to the first embodiment, the learning device of the second embodiment also trains the student model by knowledge distillation using the teacher model. The hardware configuration of the learning device of the second embodiment is the same as the hardware configuration of the learning device 10 of the first embodiment shown in FIG. Also, the functional configuration of the processor included in the learning device of the second embodiment is the same as the functional configuration of the processor 11 included in the learning device 10 of the first embodiment shown in FIG. However, the details of the functions of the learning unit 113 and the changing unit 115 included in the processor are different in the second embodiment from those in the first embodiment. The following description will focus on these different parts. The description of the contents that overlap with the first embodiment will be omitted unless it is particularly necessary.

As in the first embodiment, the learning unit 113 learns the student model so as to minimize the loss L shown in Equation (3). However, in the first embodiment, the weighting factor λ shown in equation (3) is constant, but in the second embodiment the weighting factor λ is not constant. This point is different from the first embodiment.

The changing unit 115 changes the weighting factor when calculating the loss of the student model based on the performance difference between the teacher model and the student model. By adopting such a configuration, for example, when the performance of the student model approaches the performance of the teacher model, the weighting factor λ can be changed, and the student model can be learned in a state in which it is less susceptible to the influence of the teacher model.

Specifically, the loss L in the student model is calculated by equation (3) as described above. That is, in the present embodiment, the weighting factor is a factor λ for adjusting the balance between the loss Luq for the training data and the loss Ldstl for the teacher model. By reducing the value of λ when the performance of the student model approaches the performance of the teacher model, learning can be made less susceptible to the teacher model. This can reduce the possibility that the teacher model will adversely affect the learning of the student model.

FIG. 6 is a flow chart showing the flow of student model training in the learning device according to the second embodiment of the present invention. Before starting the flow chart shown in FIG. 6, a trained teacher model which has been trained by a known method is created. In this embodiment, unlike the first embodiment, the number of teacher models is one.

In step S11, a variable "epoch" representing the number of epochs is set to zero. Note that the number of epochs to be learned is set to M. The number of epochs M is, for example, 100 epochs. When the process of step S11 is completed, the process proceeds to the next step S12.

In step S12, the student model is learned using the teacher model. The learning is learning using the knowledge distillation described above. Here, learning for one epoch is performed. Specifically, the learning data set is divided into a plurality of subsets according to the batch size, and learning using knowledge distillation is performed for each subset (every one iteration). Inference by the teacher model and inference by the student model are performed for each subset. Then, for each subset, the parameter of the student model is updated using a loss function (see Equation (3)) obtained by inference using the teacher model and the student model. Completion of learning for all subsets means completion of learning for one epoch. When the process of step S12 is completed, the process proceeds to the next step S13.

In step S13, the performance of the student model trained in step S12 is evaluated. In this embodiment, mAP, which is an evaluation index of the object detection task, is obtained for the student model after learning in step S12. After obtaining the mAP, the process proceeds to the next step S14.

In step S14, it is determined whether or not the performance difference between the teacher model and the student model is less than a threshold. Specifically, it is determined whether or not the difference between the mAP of the teacher model and the mAP of the student model is less than a threshold. The threshold is a constant value stored in the storage unit 12 in advance. If the performance difference is less than the threshold (Yes in step S14), the process proceeds to next step S15. If the performance difference is greater than or equal to the threshold (No in step S14), the process proceeds to step S16. That is, the process of step S15 is skipped.

In step S15, the weighting factor λ included in equation (3) is changed. Specifically, the value of the weighting factor λ is reduced. For example, the current value of λ is multiplied by a constant smaller than 1 (eg, 0.5, etc.). That is, the weight of the term represented by λ×Ldstl in Equation (3) is reduced. As a result, the weight of the loss Ldstl with respect to the teacher model in the loss L of the student model is reduced, and the student model can be learned while reducing the influence of the teacher model. When the process of step S15 is completed, the process proceeds to the next step S16.

Note that when the weighting factor λ is changed in step S15, the threshold in step S14 may be changed in accordance with the change. Also, the threshold may be kept constant even after the weighting factor λ is changed. In this case, after the weighting factor λ is changed for the first time, the weighting factor λ is gradually decreased for each epoch. Further, after the weighting factor λ is changed once, the weighting factor λ may not be changed thereafter.

In step S16, 1 is added to the variable "epoch". If the variable "epoch" was zero at this time, the variable "epoch" becomes one. That is, one epoch of learning (knowledge distillation) using the teacher model is completed. Also, if the variable “epoch” was 50, the variable “epoch” becomes 51. That is, 51 epochs of learning using the teacher model have been completed. When the process of step S16 is completed, the process proceeds to the next step S17.

In step S17, it is checked whether or not the variable "epoch" is smaller than the number M of epochs to be learned. If the variable "epoch" is smaller than the number of epochs M (Yes in step S17), the process returns to step S12 and learning is repeated. When the variable "epoch" has reached the number of epochs M (No in step S17), the training (learning) of the student model shown in FIG. 6 is completed, and a new trained model is completed.

The trained student model obtained by the process of FIG. 6 is less likely to be adversely affected by the teacher model, and the performance can be appropriately improved by using knowledge distillation. In addition, the trained model has fewer parameters than the teacher model, and can be processed quickly.

As can be seen from the above, also in the learning method of the second embodiment, the loss L of the student model is obtained using the loss Luq of the learning data and the loss Ldstl of the teacher model. Further, in the learning method of the second embodiment, the weighting coefficient λ for adjusting the balance between the loss Luq for the learning data and the loss Ldstl for the teacher model is changed at a predetermined timing to train the student model. With such a configuration, it is possible to train the student model while adjusting the influence of the teacher model during the training of the student model.

As described above, in this embodiment, the predetermined timing is when the performance difference between the teacher model and the student model becomes smaller than a predetermined threshold. As a result, when the performance of the student model approaches the performance of the teacher model, the weighting factor λ can be changed, and the student model can be learned in a state in which it is less susceptible to the influence of the teacher model.

However, as another example, the predetermined timing may be every predetermined number of epochs. With this configuration, the weighting factor λ can be attenuated every predetermined number of epochs, and the influence of the teacher model can be gradually reduced in accordance with the progress of learning using knowledge distillation.

The predetermined number of epochs may be, for example, multiple epochs or one epoch. FIG. 7 is a diagram showing an example in which the weighting factor λ is attenuated every multiple epochs. In FIG. 7, the horizontal axis is the number of epochs and the vertical axis is the value of λ. In the example shown in FIG. 7, the value of λ with respect to the number of epochs can be expressed by Equation (4) below.
λ = λ_int * (0.9^(epoch mod 50)) (4)
λ_int is the initial value of λ. epoch indicates the current epoch number. In the example shown in equation (4), the weighting factor λ is multiplied by 0.9 every 50 epochs. 0.9 times is an example, and other numbers less than 1 may be used.

FIG. 8 is a diagram showing an example in which the weighting factor λ is attenuated for each epoch. In FIG. 8, the horizontal axis is the number of epochs and the vertical axis is the value of λ. In the example shown in FIG. 8, the value of λ with respect to the number of epochs can be expressed by Equation (5) below.
λ = λ_int*(cos(π*epoch/M) + 1)/2 (5)
λ_int is the initial value of λ. epoch indicates the current epoch number. M is the number of epochs to be learned. In the example shown in FIG. 8, λ is attenuated continuously (every epoch) by so-called Cosine Annesling.

<3. Notes, etc.>
Various modifications can be made to the various technical features disclosed in this specification without departing from the gist of the technical creation in addition to the above-described embodiments. That is, the above embodiments should be considered as examples in all respects and not restrictive, and the technical scope of the present invention is not defined by the description of the above embodiments, but by the scope of claims. All changes that come within the meaning and range of equivalency of the claims are to be understood. In addition, the multiple embodiments and modifications shown in this specification may be implemented in appropriate combinations within a possible range.

10... Learning device 114... Evaluation unit 115... Change unit

Claims (9)

A learning device for training a student model using a teacher model,
an evaluation unit that obtains a performance difference between the teacher model and the student model;
a changing unit that changes the teacher model based on the performance difference, or changes a weighting factor when calculating the loss of the student model;
A learning device comprising: wherein, when the performance difference becomes smaller than a predetermined threshold, the changing unit changes the teacher model used for training the student model to a teacher model having higher performance than the teacher model in use. Item 1. The learning device according to item 1. 2. The learning device according to claim 1, wherein said weighting coefficient is a coefficient for adjusting a balance between a loss for learning data and a loss for said teacher model. 4. The learning device according to claim 3, wherein the loss for said teacher model is an error in an inference result or an error in an intermediate layer feature map between said teacher model and said student model. The learning device according to any one of claims 1 to 4, wherein the performance is object detection performance. A learning method for training a student model using a teacher model, comprising:
A learning method, wherein the teacher model is changed based on a performance difference between the teacher model and the student model. A learning method for training a student model using a teacher model, comprising:
Obtaining the loss of the student model using the loss for the learning data and the loss for the teacher model,
A learning method, wherein a weighting factor for adjusting a balance between a loss for the learning data and a loss for the teacher model is changed at a predetermined timing to train the student model. 8. The learning method according to claim 7, wherein said predetermined timing is when a performance difference between said teacher model and said student model becomes smaller than a predetermined threshold. 8. The learning method according to claim 7, wherein said predetermined timing is every predetermined number of epochs.

Images