KR20240050709A - Method and apparatus for self-knowledge distillation using cross entropy - Google Patents

Abstract

The self-knowledge distillation method using cross-entropy according to the present invention includes the process of identifying a target distribution considering one-hot encoded label information; A process of grouping and normalizing each of the target distribution and output distribution in which the label information is considered into at least two or more classes; and identifying a loss between a first group of target distributions in which the label information is considered and a second group of output distributions corresponding to the first group. may include.

Description

Self-knowledge distillation method and device using cross entropy {METHOD AND APPARATUS FOR SELF-KNOWLEDGE DISTILLATION USING CROSS ENTROPY}

The present invention relates to a self-knowledge distillation method and device using cross-entropy, and more specifically, to a self-knowledge distillation method and device that considers a low probability distribution in the output distribution according to self-knowledge distillation.

The content described in this section simply provides background information about the present invention and does not constitute prior art.

The core of deep learning, which has developed AI technology, is to effectively perform complex calculations by learning high-level abstract knowledge from data using deep neural networks. Data-based AI models acquire the knowledge necessary for tasks from data. In order to perform new tasks, data for learning new tasks must be secured, and the demand for such data may increase. When deep learning is applied to large-scale data, knowledge can be learned more effectively, so deep learning performance can be improved by using more data. However, securing large-scale data for learning may require a lot of cost and time. Accordingly, the feature vector of a model trained through a large-scale dataset (or may be referred to as a pre-learning model or pre-trained model) can be used to recalibrate to fit a new model. A knowledge distillation technique was proposed. Through knowledge distillation technology, it is possible to learn a deep learning model that can be applied to a new model even when only a relatively small amount of data is available.

Conventionally, in knowledge distillation technology, many methods have been proposed on where and how to obtain the data needed for model learning, but not much research has been done on how to effectively transfer the acquired knowledge. In addition, methods such as Kullback-Leibler divergence (KLD) are used to transfer knowledge between a model trained through a large-scale dataset and a new model that receives knowledge distilled from the model. The performance of the learning model is limited because distributions with low similarity are not considered in the loss function applied during the model's data processing process.

Japanese Patent Publication JP 2022-516452 A (Data processing method and device, electronic device and storage medium, February 28, 2022)

As mentioned above, it is a self-knowledge distillation method using cross-entropy to consider even distributions with low similarity for learning, and the target distribution and output distribution are aligned on the same basis and a loss function is applied for each certain section. The purpose is to provide a self-knowledge distillation method and device using cross-entropy that can improve deep learning performance.

To achieve the above object, the self-knowledge distillation method using cross-entropy according to the present invention includes the process of identifying a target distribution considering one-hot encoded label information; A process of grouping and normalizing each of the target distribution and output distribution in which the label information is considered into at least two or more classes; and identifying a loss between a first group of target distributions in which the label information is considered and a second group of output distributions corresponding to the first group. may include.

Depending on the embodiment, the process of grouping and normalizing the at least two classes may include arranging a target distribution and an output distribution in which the label information is linearly combined by the same standard; may include.

Depending on the embodiment, the process of grouping and normalizing into at least two or more classes includes identifying a representative value by taking a softmax function for each of the target distribution and output distribution considering the label information grouped into at least two or more classes. ; may include.

Depending on the embodiment, the process of identifying the loss may include taking a softmax function for each of the target distribution and the output distribution considering the label information grouped into the at least two classes and applying the loss function to the representative value; Includes, and the loss function may include a Kullback-Leibler divergence (KLD) function.

In addition, the self-knowledge distillation device using cross-entropy according to the present invention includes a first classifier that identifies a target distribution based on previously input data; a second classifier to which a softmax function different from the pre-learning classifier is applied; And identifying a target distribution in which one-hot encoded label information is considered, grouping and normalizing each of the target distribution and output distribution in which the label information is considered into at least two classes, and a control unit that identifies a loss between a first group of target distributions and a second group of output distributions corresponding to the first group; may include.

Depending on the embodiment, the control unit may align the target distribution and the output distribution in which the label information is linearly combined based on the same standard.

Depending on the embodiment, the control unit may identify a representative value by taking a softmax function for each of the target distribution and output distribution considering the label information grouped into the at least two classes.

According to an embodiment, the control unit takes a softmax function to each of the target distribution and the output distribution considering the label information grouped into the at least two classes and applies a loss function to the representative value, and the loss function is coolback- May include Kullback-Leibler divergence (KLD) function.

The self-knowledge distillation method and device using cross-entropy provided as an embodiment of the present invention utilize a self-knowledge distillation method that does not require a separate prior model and are relatively simple in terms of memory, number of parameters, and structure. .

In addition, according to an embodiment of the present invention, it is possible to evenly learn the plurality of class information contained in the input data, and as a result, the relationship between classes is clarified and the variance between multiple data within the same class is lowered to provide more accurate information. It can have a decision boundary.

Meanwhile, the effects of the present invention are not limited to the effects mentioned above, and various effects may be included within the range apparent to those skilled in the art from the contents described below.

Figure 1 is a diagram for explaining a self-knowledge distillation method using cross entropy according to an embodiment of the present invention.
Figure 2 is a diagram for explaining a self-knowledge distillation device using cross entropy according to an embodiment of the present invention.
Figure 3 is a flowchart illustrating a self-knowledge distillation method using cross entropy according to an embodiment of the present invention.
Figures 4a to 4d are diagrams for explaining the calculation results of the self-knowledge distillation method and device using cross-entropy according to an embodiment of the present invention.

Hereinafter, the terms used in the present specification will be briefly explained, and the structure and operation of a preferred embodiment of the present invention will be described in detail as specific details for carrying out the present invention. The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to provide a general understanding of the technical field to which the present invention pertains. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other elements in addition to the mentioned elements. Like reference numerals refer to like elements throughout the specification, and “and/or” includes each and every combination of one or more of the referenced elements. Although “first”, “second”, etc. are used to describe various components, these components are of course not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may also be a second component within the technical spirit of the present invention.

The word “exemplary” is used herein to mean “used as an example or illustration.” Any embodiment described herein as “exemplary” should not necessarily be construed as preferred or as having an advantage over other embodiments.

Additionally, the term “unit” used in the specification refers to a hardware element such as software, FPGA, or ASIC, and the “unit” performs certain roles. However, “wealth” is not limited to software or hardware. The “copy” may be configured to reside on an addressable storage medium and may be configured to run on one or more processors. Thus, as an example, “part” refers to elements such as software elements, object-oriented software elements, class elements, and task elements, processes, functions, properties, procedures, subroutines, and programs. Includes segments of code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functionality provided within elements and “parts” may be combined into smaller numbers of elements and “parts” or may be further separated into additional elements and “parts”.

Additionally, all “units” in this specification may be controlled by at least one processor, and at least one processor may perform the operations performed by the “units” of the present disclosure.

Embodiments of the present specification may be described in terms of a function or a block that performs a function. Blocks that may be referred to as 'units' or 'modules' of the present disclosure include logic gates, integrated circuits, microprocessors, microcontrollers, memories, passive electronic components, active electronic components, optical components, and hardwired circuits. It is physically implemented by analog or digital circuits, etc., and can optionally be driven by firmware and software.

Embodiments herein may be implemented using at least one software program running on at least one hardware device and may perform network management functions to control elements.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless clearly specifically defined.

Conventionally, various knowledge distillation methods have been proposed depending on where and how data helpful for model learning are obtained, and these methods typically use loss functions such as cross entropy and Kullback-Leibler divergence (KLD). , it cannot overcome the problems caused by this loss function. Specifically, much of the loss arises from class information that is closely related to the weight as the final result of deep learning, and information in a distribution with low similarity (hereinafter, may be referred to as a tail distribution) does not significantly contribute to the loss function. It may not be sufficiently considered during the optimization process.

Accordingly, the self-knowledge distillation method and device using cross-entropy according to an embodiment of the present invention allow the deep learning model to evenly utilize all class information contained in the data during the learning process to produce the final result of deep learning during the learning process. The performance of the learning model can be improved by evenly utilizing not only the weight and class information closely related to it, but also class information in the tail distribution, which is far from the weight as the final result of deep learning.

The self-knowledge distillation method and device using cross-entropy according to an embodiment of the present invention can utilize a typical classification model among deep learning models. The self-knowledge distillation method and device using cross-entropy including a typical classification model calculates a logit value when input data passes through a deep learning network, and a softmax function is applied to the calculated logit value. can be taken and transformed into a probability distribution whose sum is 1, and one-hot encoding can be used to derive the output by assigning 1 to the class with the highest input value and 0 to the remaining classes. At this time, in one embodiment of the present invention, the method of giving a predicted value consisting of only 1 and 0 is referred to as a hard target, and in this case, except for the class with the highest probability value, the probability values of other classes are equally 0. can be ignored. For example, if a cat image has a shape closer to a dog than to a panda, it may have probability distributions of cat (0.6), dog (0.3), and panda (0.1). This probability distribution can be viewed as a properly trained deep learning model, and the appropriately trained deep learning model can be referred to as a teacher model, and the student model is the feature extractor or classifier of the teacher model. All or part of it can be utilized through soft targets. The soft target can use the output of the softmax function as the final output of the model, and obtain probability information for all categories (or classes) to reduce information loss.

The self-knowledge distillation method and device using cross-entropy according to an embodiment of the present invention uses logit to alleviate the phenomenon of focusing on 1 and 0, which are characteristic of the softmax function, in the process of defining a soft target. A hyperparameter (t, temperature) that plays a scaling role can be applied. When the hyperparameter (t) is 1, it is the same as the softmax result, and the larger the hyperparameter (t), the softer the probability distribution can be.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings.

In Figure 1, a schematic process for the self-knowledge distillation method using cross entropy is explained, and in Figures 2 and 3, specific configurations and detailed processes according to each configuration are explained. Next, the self-knowledge distillation method using cross entropy and the effects of the device will be described in FIGS. 4A to 4D.

Figure 1 is a diagram for explaining a self-knowledge distillation method using cross entropy according to an embodiment of the present invention.

Referring to Figure 1, the self-knowledge distillation method using cross-entropy can evenly learn the class information contained in the data, can consider relationships between classes, and can generate multiple classes corresponding to at least one class classified into the same class. By lowering the variance between data, the decision boundary of the applied model can be made clear. Forming a decision boundary in a classification process based on deep learning can be an important criterion for securing the performance of deep learning.

The self-knowledge distillation method using cross-entropy according to the embodiment may include a normal self-knowledge distillation method, and may include a method in which knowledge is learned while being transferred within one model, and both the teacher model and the student model may be the same. It could be a model. One model according to the self-knowledge distillation method using cross-entropy according to the embodiment may include a pre-learning classifier that acts as a teacher model and a fine-tuning classifier that acts as a student model. Depending on the embodiment, the pre-learning classifier (or may be referred to as the deepest classifier) may include at least one resNet block and a softmax function block, and the fine-tuning classifier (or shallowest classifier) may include at least one resNet block and a softmax function block. The classifier (may be referred to as a shallow classifier) may include at least one fully connected layer (FC layer) and a softmax function block.

The decision boundary in the self-knowledge distillation method using cross entropy according to the embodiment may be composed of a combination of the activation boundaries of at least one hidden layer, and the decision boundary of each hidden layer. Considering , the performance of knowledge distillation of a previously trained model can ultimately be improved.

Depending on the embodiment, the self-knowledge distillation method utilizing cross-entropy may be performed by a previously learned classifier (or may be referred to as a pre-trained classifier, pre-trained model, or pre-trained network) and a fine-tuned classifier.

Depending on the embodiment, the pre-learning classifier and the fine-tuning classifier may include a feature extractor capable of extracting features of the input data and a classifier that calculates the output value of the feature extractor with probability, but the pre-learning classifier and the fine-tuning classifier The same output value can be received from separately configured feature extractors.

Depending on the embodiment, receiving the same output value from a feature extractor that is configured separately from the pre-learning classifier and the fine-tuning classifier may be understood as using the same input data, and by being based on the same input, the pre-learning classifier and the fine-tuning classifier Knowledge can be understood as being distilled or transmitted between people. This is because the fine tuning classifier can use the label value (label information) of the pre-learning classifier and the soft target previously derived by the pre-learning classifier.

Depending on the embodiment, the feature extractor may be composed of at least one convolution layer and a pooling layer, and may be learned in advance based on input data already provided in advance, and the output value of the feature extractor may be bottleneck. It may be referred to as a bottleneck or a feature vector.

Depending on the embodiment, each of the pre-learning classifier and the fine-tuning classifier may calculate the output value of the feature extractor as a probability value based on each softmax function and classify it into at least two categories (or classes).

Depending on the embodiment, the pre-learning classifier and the fine-tuning classifier may be connected by a fully connected layer and may be composed of at least one dense layer (or may be referred to as a dense layer), and at least one dense layer A dropout layer and a batch normalization layer are located in between to reduce overfitting.

Depending on the embodiment, the fine tuning classifier adjusts the weight of the softmax function applied to the pre-learning classifier, analyzes the type and total number of data, and based on this, retrains only part of the weight of the pre-learning classifier or all weights. You can learn again from the beginning.

Depending on the embodiment, the pre-learning classifier and the fine-tuning classifier have various types such as VGG16, ResNet50, MobilNet, and InceptionV3, and at least one of them may be applied depending on the size of the input data or the type of input data. However, the embodiment of the present invention It is not limited to the above types.

Depending on the embodiment, the pre-learning classifier and fine-tuning classifier may include flatten or GlobalAveragePooling2D, which perform the role of converting into a one-dimensional vector to build a neural network.

Depending on the embodiment, cross-entropy and Kullback-Leibler divergence (KLD) may be applied for overall optimization between the pre-learning classifier and the fine-tuning classifier.

Depending on the embodiment, the self-knowledge distillation method using cross-entropy can sort or normalize the probability value of the probability distribution of each pre-learning classifier and fine-tuning classifier, and identify representative values by grouping categories into certain intervals. can do.

FIG. 2 is a diagram illustrating a self-knowledge distillation device using cross-entropy according to an embodiment of the present invention, and FIG. 3 is a flowchart illustrating a self-knowledge distillation method using cross-entropy according to an embodiment of the present invention. am.

Referring to Figures 2 and 3, the self-knowledge distillation device 200 using cross entropy includes an input unit 210 that receives input data, a feature extractor 220 that extracts features of the input data and identifies them as an output vector, A pre-learning classifier 230 that outputs a target distribution based on previously learned data, distills or transfers knowledge from the pre-learning classifier 230, and generates an output distribution based on a softmax function different from the pre-learning classifier 230. It may include a fine-tuning classifier 240 that outputs output, and a control unit 250 that applies a loss function based on the difference between the target distribution and the output distribution.

According to an embodiment, the self-knowledge distillation device 200 utilizing cross-entropy obtains a target distribution from a self-knowledge distillation method based on previous input data, performs overall optimization through KL-Divergence loss, and probabilities. After sorting the distribution, the loss can be calculated separately by splitting it at regular intervals.

Depending on the embodiment, the self-knowledge distillation device 200 using cross-entropy may evenly consider at least one category (or class) information contained in the input data when trained through the self-knowledge distillation method.

Depending on the embodiment, the self-knowledge distillation device 200 utilizing cross-entropy may utilize a loss function in the form of a plug-in that further improves performance in addition to the self-knowledge distillation method, and may utilize a loss function in the form of a plug-in that further improves performance The overall technology of the distillation method can be utilized.

Depending on the embodiment, the control unit 250 may identify a target distribution learned from an existing self-knowledge distillation method by the pre-learning classifier 230 (S310).

Depending on the embodiment, the control unit 250 may identify a probability distribution to be used for learning through a linear combination of one-hot encoded label information and the target distribution (S320).

For example, if the target distributions for dogs, cats, and pandas for each category are 0.3, 0.6, and 0.1, and the output distributions are 0.5, 0.3, and 0.2, the highest probability value in the target distribution is the cat class, and the label information in the input data It can be a dog or a panda, and if label information is not used, the output distribution follows the incorrect target distribution, so the actual correct answer class can have the highest probability value through a linear combination of the label information and the target distribution. . For example, if the actual label information is a dog, a weight of about 0.1 is given to the target distribution, and a weight of 0.9 is given to the label information according to the sort order (cat, dog, panda), resulting in 0.1 * (0.6, 0.3, 0.1) + It can be identified as 0.9 * (0, 1, 0) = (0.06, 0.93, 0.01). If label information was not used, cat would have the highest value (0.6, 0.3, 0.1).

Depending on the embodiment, the control unit 250 may identify and learn the difference (loss) between the target distribution and the output distribution in which label information is linearly combined through KL-Divergence loss optimization.

Depending on the embodiment, the control unit 250 divides or groups at least two probability distributions included in each of the target distribution and the output distribution in which label information is linearly combined into categories based on the same standard, and then corresponds to the grouped target distribution and output distribution. The cross-entropy loss between each group can be identified (S330). The control unit 250 may sort the target distribution and the output distribution in which the label information is linearly combined by the same standard, and may sort the probability distribution of the first identified target distribution in descending order, and the category (or class) of the sorted target distribution. ) can be sorted in the order of the output distribution. For example, if the target distribution of dog, cat, and panda by category is 0.3, 0.6, 0.1, and the output distribution is 0.5, 0.3, 0.2, first, the target distribution is cat (0.6), dog (0.3), and panda ( 0.2), and the output distribution can be listed in the same order as the target distribution: cat (0.3), dog (0.5), and panda (0.2).

Depending on the embodiment, the control unit 250 may divide each of the target distribution and the output distribution, in which the label information is linearly combined, by a certain size (window size) and at a certain interval (stride), and the target distribution and the output distribution are divided and correspond to each other. The cross-entropy loss between groups can be calculated.

Depending on the embodiment, the control unit 250 may normalize the target distribution and the output distribution, and identify representative values by grouping categories into certain sections. The control unit 250 may group M adjacent classes from a total of N classes for the sorted target distribution and output distribution, starting from the first class. The control unit 250 can apply the softmax function to values grouped into two N classes. When the softmax function is taken, the total sum can be 1, and the difference can be identified by comparing the values taken from the softmax function. The control unit 250 can identify the difference by skipping s classes sideways and comparing the last class.

For example, for the cat, dog, and panda classes, if N=2 classes are grouped for a total of M=3 classes, (cat, dog) can be grouped at first, and the target distribution (0.6, 0.3), output If you take the softmax function for the distribution (0.3, 0.5), you get (0.5744, 0.4256) and (0.4502, 0.5498), and you can compare the two to identify the difference. When s = 1, if you go one space to the side, the same softmax function is taken for the (dog, panda) class, and then you can identify the difference.

Depending on the embodiment, the control unit 250 may learn the model by considering the difference (loss) between the target distribution and the output distribution in which label information is linearly combined through KL-Divergence loss optimization and the cross-entropy loss identified in S330. (S340). The control unit 250 can identify the loss function after normalization and perform the process of combining each block by adding the loss values calculated by applying the KL divergence function. In other words, by using the KL divergence function Losses can be identified. For example, taking the preceding examples of cat, dog, and panda, applying (0.5744, 0.4256), (0.4502, 0.5498) to the KL divergence function calculates 0.5477 * log(0.4502) + 0.4256 * log(0.5498). A value can be identified as loss. The control unit 250 is a process of combining each block. After grouping N classes together, the control unit 250 calculates the loss through the KL divergence function and adds the value calculated through the KL divergence function next to the next s space to calculate the loss. can be identified.

Figures 4a to 4d are diagrams for explaining the calculation results of the self-knowledge distillation method and device using cross-entropy according to an embodiment of the present invention.

Referring to Figure 4a, in the self-knowledge distillation method using cross-entropy according to an embodiment and the model trained in the device, the average of each data for each class for the CIFAR-100 dataset is calculated, and the Pearson Correlation Coefficient is calculated. One result is shown. For convenience of observation, child classes with the same parent class can be adjacent, and the expressiveness of the data can be increased by lowering the correlation for classes with different parents and increasing the correlation for classes with the same parent class. there is.

Referring to Figure 4b, t-SNE results are shown for the CIFAR-100 dataset in a model trained in the self-knowledge distillation method and device using cross-entropy according to one embodiment. For the convenience of observation, for a total of 10 classes, for fairness of the experiment, the hyperparameters used in t-SNE can all be set to be the same.

Referring to FIGS. 4C and 4D, FIG. 4C shows the results of the CIFAR-100 experiment, and FIG. 4D shows the results of the Fine-grained Image Classification experiment - CUB-200, Stanford Cars, and FGVC-Aircraft. According to one embodiment, a self-knowledge distillation method using cross-entropy and a model trained in the device can be used to create a model with a small model size but high performance, and can be used by users in mobile devices or edge computing environments. Experience can be increased.

Various embodiments of the present invention include one or more instructions stored in a storage medium (e.g., memory) readable by a machine (e.g., a vehicle generated data recorder or computer). It can be implemented as software. For example, the processor of the device may call at least one instruction among one or more instructions stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term is used when data is stored semi-permanently in a storage medium. There is no distinction between temporary storage and temporary storage. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored.

According to one embodiment, methods according to various embodiments disclosed in this specification may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices. It can be distributed (e.g. downloaded or uploaded) directly between devices (e.g. smartphones) or online. In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) is stored on a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server. It can be temporarily stored or created temporarily. Above, embodiments of the present invention have been described with reference to the attached drawings, but those skilled in the art will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

Claims (8)

A process of identifying a target distribution considering one-hot encoded label information;
A process of grouping and normalizing each of the target distribution and output distribution in which the label information is considered into at least two or more classes; and
identifying a loss between a first group of target distributions in which the label information is considered and a second group of output distributions corresponding to the first group;
Including,
Self-knowledge distillation method using cross-entropy. According to paragraph 1,
The process of grouping and normalizing into at least two or more classes is,
Including a process of aligning the target distribution and the output distribution in which the label information is linearly combined by the same standard,
Self-knowledge distillation method using cross-entropy. According to paragraph 1,
The process of grouping and normalizing into at least two or more classes is,
A process of identifying a representative value by taking a softmax function for each of the target distribution and output distribution in which the label information grouped into the at least two classes is considered, comprising:
Self-knowledge distillation method using cross-entropy. According to paragraph 3,
The process of identifying the loss is:
A process of applying a loss function to a representative value by taking a softmax function to each of the target distribution and the output distribution in which the label information grouped into the at least two classes is considered,
The loss function includes the Kullback-Leibler divergence (KLD) function,
Self-knowledge distillation method using cross-entropy. a first classifier that identifies a target distribution based on previously input data;
a second classifier to which a softmax function different from the pre-learning classifier is applied; and
Identify the target distribution considering one-hot encoded label information,
Grouping and normalizing each of the target distribution and output distribution in which the label information is considered into at least two or more classes,
a control unit that identifies loss between a first group of target distributions considering the label information and a second group of output distributions corresponding to the first group;
Including,
Self-knowledge distillation device using cross-entropy. According to clause 5,
The control unit,
Sorting the target distribution and output distribution in which the label information is linearly combined by the same standard,
Self-knowledge distillation device using cross-entropy. According to clause 5,
The control unit,
Identifying a representative value by taking a softmax function for each of the target distribution and output distribution in which the label information grouped into the at least two classes is considered,
Self-knowledge distillation device using cross-entropy. In clause 7,
The control unit,
A softmax function is taken from each of the target distribution and output distribution in which the label information grouped into at least two classes is considered, and a loss function is applied to the representative value,
The loss function includes the Kullback-Leibler divergence (KLD) function,
Self-knowledge distillation device using cross-entropy.