KR102615517B1 - Technique for class and property classification of input data - Google Patents

Abstract

A device according to some embodiments of the present disclosure includes: a communication unit that receives an artificial intelligence model learned or retrained from a learning device; A memory unit that stores input data and the artificial intelligence model; a first prediction unit that classifies the input data into one of a plurality of classes based on a first value output by inputting the input data into the artificial intelligence model; and a second prediction unit that inputs the input data into the artificial intelligence model to obtain a second value related to classification of the nature of the input data.

Description

Class and property classification technique of input data {TECHNIQUE FOR CLASS AND PROPERTY CLASSIFICATION OF INPUT DATA}

The present disclosure relates to a class and property classification technique for input data. Specifically, the input data is classified into one of a plurality of classes based on the first value and the second value output by inputting the input data into an artificial intelligence model. It relates to an apparatus and method for classifying and classifying the properties of input data.

The content described in this section simply provides background information for this embodiment and does not constitute prior art.

With the advent of deep learning, machine learning, or artificial intelligence (AI) technology, the technology goes beyond simple repetitive tasks and is applied to fields that require human judgment, replacing human processing.

Typically, the technology receives input data that has already been labeled by humans, and establishes the relationship between input data and labeling by learning it from a specific deep learning model architecture. For example, assume a situation where a device that performs deep learning, machine learning, or artificial intelligence replaces the task of classifying images with circles and images with squares. At this time, the device receives input images classified as circles or squares, learns about a specific architecture, and creates an artificial intelligence model (Trained Model). Afterwards, when an image with either a circle or a square is input, an artificial intelligence model is used to classify the input image into a specific class (circle or square) by distinguishing whether the image has a circle or a square. .

In this way, the artificial intelligence model once learned and created is stored in the device and predicts (inference) which class the input (real-time) data should be classified into.

However, problems appear when the class of the input data is different from the class at the time the artificial intelligence model was created, or the boundaries become ambiguous. According to the above example, the input data at the time the device created the artificial intelligence model had only two classes: circle and square. However, as time passes, data of existing classes may be slightly modified in form, or data may exist at the boundary between classes, making the distinction ambiguous or requiring human labeling, or a class that is completely different from existing classes may occur. of data may occur. For example, input data about a circle may be slightly transformed, and input data having an oval shape may occur. Alternatively, pentagon-shaped data, which is data that may be a boundary of conventional classes (circles and squares), may be generated, and data of a class (for example, star-shaped) that is completely unrelated to existing classes may be generated.

When input data of the above-mentioned class is input, the already created artificial intelligence model can classify only the class that it has learned, even if the input data of the above-mentioned class is input. Accordingly, data of completely different classes are classified into a different class from the correct answer, causing a problem in which the correct answer rate decreases. On the other hand, in the case of data that is slightly modified in form from data of an existing class or exists at the boundary between classes, even if the device classifies the data into one class, it is unclear whether the correct answer is correct, and it is unclear whether the data is accurately recognized. There is a problem with not being able to guarantee the confidence of the classification, whether it was classified or not accurately recognized.

Because of this, there is a demand for a way to recognize and improve the problem of lower reliability of classification of the device and lower correct answer rate that can be caused by lower reliability.

In addition, in order to compensate for the decrease in the correct response rate of the learning model, methods of re-learning various types of data are being used, but learning vague data is likely to bring up the above-mentioned problems again, so there is a demand for complementary measures. .

Japanese Patent Publication No. 2018-045516 (2018.03.22.)

The present disclosure aims to solve the above-described problems and other problems. The technical task to be achieved by some embodiments of the present disclosure aims to perform class classification and property classification on input data using an artificial intelligence model.

The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art of the present disclosure from the description below. There will be.

A device according to some embodiments of the present disclosure includes: a communication unit that receives an artificial intelligence model learned or retrained from a learning device; A memory unit that stores input data and the artificial intelligence model; a first prediction unit that classifies the input data into one of a plurality of classes based on a first value output by inputting the input data into the artificial intelligence model; and a second prediction unit that inputs the input data into the artificial intelligence model to obtain a second value related to classification of the nature of the input data.

According to some embodiments of the present disclosure, the artificial intelligence model includes a first sub-model that outputs the first value when the input data is input; and a second sub-model that shares a layer related to feature extraction for the input data with the first sub-model and outputs a second value when the input data is input.

According to some embodiments of the present disclosure, the learning device may perform learning or re-learning of the artificial intelligence model by giving weight to the first sub-model rather than the second sub-model.

According to some embodiments of the present disclosure, the learning device performs learning or re-learning of the artificial intelligence model so that the first sub-model is learned or re-learned before the second sub-model. When learning or re-learning a model, learning or re-learning the second sub-model may be performed while fixing at least one parameter included in the layer.

According to some embodiments of the present disclosure, the learning device includes a first step of learning or re-learning the artificial intelligence model, a second step of learning the first sub-model, and the layer. The artificial intelligence model may be trained or re-trained based on the third step of learning the second sub-model while fixing at least one parameter.

According to some embodiments of the present disclosure, the property is such that when the first prediction unit performs prediction to classify the input data into a specific class, it can be confident that the input data is classified into the specific class regardless of whether the answer is correct. It may be related to reliability, which refers to the degree to which something exists.

In the method of receiving input data, classifying classes, and classifying properties of the input data according to some embodiments of the present disclosure, the method includes: receiving a learned or re-trained artificial intelligence model from a learning device through a communication unit of the device; steps; Storing input data and the artificial intelligence model in a memory unit of the device; Inputting the input data into the artificial intelligence model through a first prediction unit of the device and classifying the input data into one of a plurality of classes based on a first value output; and inputting the input data into the artificial intelligence model through a second prediction unit of the device to obtain a second value related to classification of the nature of the input data.

The technical solutions obtainable from this disclosure are not limited to the solutions mentioned above, and other solutions not mentioned above will be clearly apparent to those skilled in the art from the description below. It will be understandable.

The effect of the technique for classifying the class and nature of input data according to the present disclosure will be described as follows.

According to some embodiments of the present disclosure, since a value that can classify the class and nature of input data can be calculated using one artificial intelligence model, the amount of calculation can be reduced compared to the case of using two artificial intelligence models.

According to some embodiments of the present disclosure, compared to the case of learning two tasks with separate artificial intelligence models, when two tasks are learned with one artificial intelligence model, the two tasks interfere with each other's learning, resulting in lower accuracy. The problem can be solved with an effective trade-off method that takes into account the roles of the two tasks.

The effects that can be obtained through the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1A to 1D are diagrams illustrating the configuration of an artificial intelligence model health management system according to some embodiments of the present disclosure.
Figure 2 is a diagram illustrating the configuration of a learning device according to some embodiments of the present disclosure.
Figure 3 is a diagram illustrating a method of clustering according to class by a learning device according to some embodiments of the present disclosure.
FIG. 4 is a diagram illustrating classes clustered in various ways according to class by a learning device according to some embodiments of the present disclosure.
Figure 5 is a diagram showing the configuration of a prediction device according to the first embodiment of the present disclosure.
Figure 6 is a diagram showing the configuration of an operating device according to a first embodiment of the present disclosure.
FIG. 7 is a diagram illustrating the relationship between each class classified by the operating device and the calculated second value in the latent space according to some embodiments of the present disclosure.
FIG. 8 is a diagram illustrating the arrangement of groups 1, 2, and 3 classified by an operating device according to some embodiments of the present disclosure in a latent space.
Figure 9 is a timing chart showing a method by which an operating device determines the health of an artificial intelligence model according to the first embodiment of the present disclosure.
FIG. 10 is a timing chart illustrating a method by which an operating device selects data to learn or re-learn according to the first embodiment of the present disclosure.
FIG. 11 is a timing chart illustrating a method in which an operating device according to the first embodiment of the present disclosure selects and outputs a sample along with a predicted value of input data.
FIG. 12 is a diagram illustrating the configuration of a prediction device according to a second embodiment of the present disclosure.
Figure 13 is a diagram showing the configuration of an operating device according to a second embodiment of the present disclosure.
Figure 14 is a timing chart showing a method by which an operating device determines the health of an artificial intelligence model according to a second embodiment of the present disclosure.
FIG. 15 is a timing chart illustrating a method by which an operating device selects data to learn or re-learn according to a second embodiment of the present disclosure.
FIG. 16 is a timing chart illustrating a method in which an operating device according to a second embodiment of the present disclosure selects and outputs a sample along with a predicted value of input data.
Figure 17 is a timing chart showing how a learning device learns a model according to some embodiments of the present disclosure.
Figure 18 is a timing chart showing how the artificial intelligence model health management system determines the reliability of the artificial intelligence model according to some embodiments of the present disclosure.
19 and 20 are diagrams illustrating screens on which a display unit outputs a sample selected by a labeling guide unit according to some embodiments of the present disclosure.
Figure 21 is a diagram for explaining an example of an artificial intelligence model according to some embodiments of the present disclosure.

Hereinafter, various embodiment(s) of the device and method according to the present disclosure will be described in detail with reference to the drawings. However, identical or similar components are given the same reference numbers regardless of the reference numerals, and duplicate descriptions thereof are omitted. I decided to do it.

The purpose and effects of the present disclosure, and technical configurations for achieving them, will become clear by referring to the embodiments described in detail below along with the accompanying drawings. In describing one or more embodiments of the present disclosure, if it is determined that a detailed description of related known technology may obscure the gist of at least one embodiment of the present disclosure, the detailed description will be omitted.

The terms in this disclosure are terms defined in consideration of the functions in this disclosure, and may vary depending on the intention or custom of the user or operator. In addition, the attached drawings are only for easy understanding of one or more embodiments of the present disclosure, and the technical idea of the present disclosure is not limited by the attached drawings, and all changes included in the spirit and technical scope of the present disclosure are not limited. , should be understood to include equivalents or substitutes.

The suffixes “module” and “part” for components used in the following description are given or used interchangeably only considering the ease of preparation of the present disclosure, and do not have distinct meanings or roles in themselves.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. Accordingly, the first component mentioned below may also be the second component within the technical spirit of the present disclosure.

Singular expressions include plural expressions unless the context clearly dictates otherwise. That is, unless otherwise specified or clear from context to indicate a singular form, the singular in this disclosure and claims should generally be construed to mean “one or more.”

In the present disclosure, terms such as “comprising,” “includes,” or “have” are intended to indicate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the present disclosure. It should be understood that this does not exclude in advance the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In the present disclosure, the term “or” should be understood not as “or” in the exclusive sense but as “or” in the connotative sense. That is, unless otherwise specified or clear from context, “X utilizes A or B” is intended to mean one of the natural implicit substitutions. That is, either X uses A; X uses B; Or, if X uses both A and B, “X uses A or B” can apply to either of these cases. Additionally, the term “and/or” as used in this disclosure should be understood to refer to and include all possible combinations of one or more of the related listed items.

As used in this disclosure, the terms “information” and “data” may be used interchangeably.

Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure may be used with meanings that can be commonly understood by those skilled in the art. Additionally, terms defined in commonly used dictionaries are not overly interpreted unless specifically defined.

However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. Only some embodiments of the present disclosure are provided to fully inform those skilled in the art of the present disclosure of the scope of the present disclosure, and the present disclosure is only defined by the scope of the claims. Therefore, the definition should be made based on the content throughout this disclosure.

1A to 1D are diagrams illustrating the configuration of an artificial intelligence model health management system according to some embodiments of the present disclosure.

1A to 1D, the artificial intelligence model health management system 100 according to some embodiments of the present disclosure includes a learning device 110, a prediction device 120, and an operating device 130. Hereinafter, the description is limited to data being transmitted and received directly from a specific device (e.g., learning device 110) to another device (e.g., prediction device 120), but is not necessarily limited to this. Data can be transmitted and received from a specific device to another device through a separate medium such as a database or USB. However, hereinafter, for convenience, it is explained that data, etc. are directly transmitted and received, and the present disclosure is not limited to this.

The instrument/equipment 145 is placed in a certain space 140 and can output data requiring human judgment or inspection. For example, the space 140 may be implemented as a factory, and the equipment/equipment 145 may be a manufacturing facility that produces specific materials or parts. In this way, one or more instruments/equipment 145 may be disposed within the space 140 to output data requiring human judgment or inspection. Data previously produced from the instrument/equipment 145 is provided to the learning device 110, and the learning device 110 can generate an artificial intelligence model by training an artificial intelligence model using the previously produced data. there is. Meanwhile, data output in real time from the instrument/facility 145 may be provided to the prediction device 120, and the prediction device 120 uses an artificial intelligence model to classify the input data into a specific class and determine the properties of the input data. The second value used to classify can be calculated.

The manager, operator, or person involved in the data output from the apparatus/equipment 145 of the space 140 (hereinafter abbreviated as 'related person') is the generated artificial intelligence model (artificial intelligence model generated by the learning device 110). ), the predicted values (first value and second value) predicted by the prediction device 120 (Inference) can be confirmed. In addition, the person concerned can check the health index of the artificial intelligence model output by the operating device 130 to indirectly check whether the artificial intelligence model is properly classifying the class for the data input in real time. Here, the health of the artificial intelligence model refers to whether the class of input data has been properly classified based on the second value included in the predicted value output from the artificial intelligence model, and refers to the accuracy of the predicted value in the artificial intelligence model. It can mean (correct answer rate). The person concerned may check the health index of the artificial intelligence model directly from the prediction device 120 or the operation device 130, or use the person's own terminal (not shown) to check the health index of the artificial intelligence model (Health Index). You can also check by communicating with 130).

In addition, a person who can directly judge or inspect the results based on the input data (for example, may be a labeler, hereinafter referred to as 'labeler') uses the operating device (130) to learn or re-learn. ) can perform labeling or relabeling on the selected data. As the learning device 110 performs learning or re-learning on data that has been labeled or re-labeled by the labeler, the artificial intelligence model can perform predictions on the input data with higher accuracy.

The learning device 110 is a dataset for learning from the equipment/facility 145 or the terminal (not shown) of the person involved in the space 140, a deep learning model architecture (hereinafter abbreviated as 'architecture'), and a cost function. Receives input for cost function and optimization method and creates an artificial intelligence model.

The learning device 110 receives input data and result values labeled with the input data as a dataset for learning, and together with it receives an architecture, a cost function, and an optimization method. Here, the cost function is defined as follows.

The first cost function may be a function proportional to the distance between points in the latent space of two input data of the same class.

According to some embodiments of the present disclosure, the second sub-model (related to the output of the second value) included in the artificial intelligence model may be learned using the first cost function.

The second cost function may be a function proportional to the difference between the first value and the result value (correct answer value). The cost value refers to a numerical value derived by the cost function. According to some embodiments of the present disclosure, the second cost function may be used when training the first sub-model (related to outputting the first value) included in the artificial intelligence model.

Meanwhile, in the present disclosure, the predicted value may include a first value used when determining which of the plurality of classes to classify the input data into, and a second value related to the nature of the input data. Here, the first value is derived as a probability value for each class as to which class the input data corresponds to, and the prediction device 120 uses the first value to calculate a result value for which class the input data corresponds to. You can. Meanwhile, the second value is a value related to the nature of the input data and may mean a coordinate value where the first value is located in the latent space. However, the present disclosure is not limited to this and various types of values may be the second value. .

For example, if there are three classes and the input data is input data that should be classified as class 1, the first value may be (0.7, 0.2, 0.1) (the number can be varied depending on reliability), and the result value may correspond to (1, 0, 0). The cost value of the second cost function may mean a numerical value proportional to the difference between the first value (or result value) and the correct answer value. Meanwhile, the properties of each input data can be identified by placing each input data point in the virtual space at the second value coordinate of the input data. Here, the input data point may mean input data expressed on the latent space.

Meanwhile, the first cost function may be a function proportional to the distance between points in the latent space of two input data of the same class.

Conversely, the first cost function may be a function inversely proportional to the distance between points in the latent space of two input data of different classes. Here, the expression inverse proportion is used, but it is not limited to this, and the first cost function may be set so that input data points corresponding to the same class become closer to each other, while input data points corresponding to different classes move away from each other.

According to some embodiments of the present disclosure, the first cost function may include both types of first cost functions.

The optimization method refers to a method of minimizing the cost value determined by the cost function. The learning device 110 receives the above-described data and can perform learning on the artificial intelligence model by continuously calculating and selecting parameters within the architecture in a way that minimizes the cost value according to an optimization method. Accordingly, the learning device 110 can generate an architecture with parameters selected to minimize the cost value as an artificial intelligence model.

The learning device 110 may generate an artificial intelligence model by calculating parameters that minimize the cost value according to an optimization method.

The learning device 110 may receive data selected for (re)learning from the operating device 130 to create an artificial intelligence model or retrain a previously created artificial intelligence model. It is generally known that the more (labeled) input data is input for learning, the more accurate the prediction of an artificial intelligence model will be. However, the quantity of input data and the accuracy of the artificial intelligence model's prediction are not necessarily proportional. Labeled input data may contain mislabeled data (for example, data that was accidentally mislabeled). If these data are included in the input data, the prediction accuracy of the artificial intelligence model may decrease. In addition, when the amount of data to learn is quite large (for example, tens of thousands to hundreds of thousands or more), even if the data itself has a high error rate, the artificial intelligence model can recognize errors based on data learned from other data. You can. However, when the amount of data to learn is relatively insufficient (less than a few thousand items), if there are errors in the data itself, these errors can have a fatal impact on the prediction. In addition, when the amount of data is considerable and the proportion of relatively low-importance data is much higher than the relatively high-importance data, learning of the relatively high-importance data is not smooth, resulting in lowering the accuracy of the prediction of the artificial intelligence model. It can be loaded. As in the example mentioned in the technology behind the invention, in a model for classifying circles and squares, the importance of input data that is clearly a circle shape and input data that is clearly a square shape is more important than that of the boundary between the shape of a circle and a square shape. The importance of input data that is difficult to classify as one type may be high. If a significant amount of such data is learned, predictions can be made relatively accurately even if input data is on the borderline. However, as described above, if the proportion of relatively low-importance data is much higher than that of relatively high-importance data, there is a high possibility that learning of the relatively high-importance data will not occur smoothly. Therefore, simply having a large number of datasets to learn from does not guarantee the accuracy of the prediction of an artificial intelligence model. Recognizing this, the learning device 110 can secure or maintain the prediction accuracy of the artificial intelligence model by learning or relearning the input data selected by the operating device 130 and labeled by the labeler. The specific configuration of the artificial intelligence model will be described later with reference to FIG. 21.

The prediction device 120 receives the artificial intelligence model generated from the learning device 110, calculates a first value for classifying the class of input data generated in real time from the instrument/equipment 145, and A second value related to the property can be calculated.

The first value is a value indicating which class the input data corresponds to, and may be the probability of corresponding to each class. The result may be drawn that the input data corresponds to a specific class with an overwhelming probability, but the result may be derived that the input data corresponds to a specific class with a slight (probability) difference. The prediction device 120 may perform prediction using an artificial intelligence model with a probability value (first value) regarding which class the input data will correspond to.

The second value is a coordinate value related to the properties of the input data and may be a value indicating where the input data point is located in the latent space. The nature of the input data may be related to reliability, which refers to the degree to which one can be confident that the input data is classified into a specific class, regardless of whether the correct answer is correct, when performing a prediction that classifies the input data into a specific class.

In the present disclosure, when the operating device 130 receives the second value, the operating device 130 may classify the nature of the input data, which will be described in detail later.

The prediction device 120 receives and stores the artificial intelligence model generated by the learning device 110 through the above-described process, and can receive data generated in real time from the instrument/equipment 145. The prediction device 120 uses received real-time data as input data, classifies the class of the input data using an artificial intelligence model, and calculates a second value related to the properties of the input data.

The prediction device 120 stores the input data and the corresponding first and second values for a preset period and transmits the data to the operating device 130.

The operating device 130 may receive input data and predicted values (first and second values) thereof from the prediction device 120 and calculate the reliability of the first value based on the received values.

The operating device 130 receives input data and predicted values (first and second values) thereof from the prediction device 120, calculates the confidence of the first value, and predicts the prediction device 120 based on the confidence. You can determine the health of the artificial intelligence model stored in or select data to learn or relearn.

Reliability may refer to the degree to which one can be confident that the input data is classified into a specific class when making predictions to classify input data into a specific class.

Confidence is the distance between each class (more specifically, the centroid of each class) and its input data point (located at the second value coordinate) in the latent space; more specifically, the ratio of the distance between each class and its input data point. It can be calculated as: Here, the distance may mean Euclidian distance, but is not necessarily limited thereto and may mean Mahalanobis distance, etc.

For example, an input data point with high reliability may have a short distance to the center of a specific class in the latent space, while the distance to the center of the remaining classes may be long (only the distance to the class in which it is classified is short). .

As another example, an input data point with low reliability has a distance from the center of all classes that is greater than a preset standard value, or a state in which the distance to the center of two or more classes is similar (not only to the class in which it is classified, but also to other classes). The distance can be short or long.

If input data points with low reliability increase, as described above, it may mean that data of a new class or data at the boundary of two or more classes are appearing or increasing. Accordingly, the operating device 130 may determine that the health of the artificial intelligence model is lowered and the correct answer rate is decreasing or is likely to decrease.

In the present disclosure, since the second value is a coordinate value indicating where the input data point is located in the latent space, the operating device 130 uses the second value to confirm the position of the input data point in the latent space, Using this, reliability can be calculated. However, the present disclosure is not limited to this.

The operating device 130 may directly output and notify the relevant person that the health level of the artificial intelligence model has decreased, or may notify the relevant person's terminal (not shown) that the artificial intelligence model needs re-learning.

The operating device 130 may select data to learn or re-learn based on the calculated reliability. The operating device 130 may calculate the reliability of the first value for all input data and classify the properties of the input data having the corresponding first value.

Input data can be classified according to its properties as follows.

Data that can be fully classified only into a specific class because the input data point is located within the radius of a specific class in the latent space (hereinafter referred to as 'first group'), and the input data point is located outside the radius of all classes in the latent space, but , data located relatively adjacent to two or more classes (hereinafter referred to as 'second group') (at a distance within a preset standard value), and data whose input data points are separated from all classes in the latent space by more than a preset standard value (hereinafter referred to as 'second group'). Hereinafter, it may be classified into 'Group 3'). Here, since the second value is a coordinate value indicating where the input data point is placed in the latent space, the position where the input data point is placed in the latent space can be confirmed using the second value.

In order to classify the properties of input data, distance information between the input data point and a specific class in the latent space may be required.

The operating device 130 may calculate the reliability of the first value for each input data using the second value, and classify which group the input data belongs to based on the reliability.

The operating device 130 can classify each input data according to its properties and select a preset number of data for each property. As described above, a factor that reduces prediction accuracy within real-time input data may be input data belonging to the second group or input data corresponding to a new class among the third group. Since these input data must be mainly studied in learning, the operating device 130 may select a preset number of data for each property so that the second and third groups of data can be included in a certain ratio. For example, the operating device 130 may select data in the first group, second group, and third group at a ratio of 4:3:3 or 6:2:2, respectively.

The operating device 130 transmits the selected data to the learning device 110, causing the learning device 110 to learn the artificial intelligence model only with the selected data when learning the artificial intelligence model for the first time, or using the selected data. This can cause the artificial intelligence model to retrain.

As mentioned above, the correct answer rate of the artificial intelligence model does not increase if there is a lot of data to learn, so the operating device 130 selects the data to increase the correct answer rate of the artificial intelligence model to be created for the first time or input data (generated in real time) You can increase the correct answer rate by relearning according to changes in .

The operating device 130 may directly output input data classified into the second or third group so that the labeler can smoothly (re)label. At this time, when outputting the input data, the operating device 130 may output, as an example, some of the data (with high reliability) classified into each class that had a small probability difference in the prediction process of the input data. , some of the data that has been previously classified with the same properties as the input data can be output as an example.

For example, when input data with low reliability is the second group, the operating device 130 may also output some of the data classified into each class that had a slight probability difference during the prediction process of the input data. More specifically, when labeling the first input data classified into the second group, second input data that is close to the first input data point among the input data classified into the first group may be output together. . In this case, the labeler can help determine which class the first input data should be labeled with. Meanwhile, in the present disclosure, since the second value is a coordinate value indicating where the corresponding input data point is placed in the latent space, the second value is used to extract the second input data that is close to the corresponding input data point. It can be.

As another example, when the input data with low reliability is the second group, the operating device 130 may output some of the various data classified into the second group as an example. In particular, as the operating device 130 outputs data classified by the same nature, the labeler can refer to what class the data previously classified into the second group was classified into, and thus input data in the same manner as the existing one. It can help you classify based on criteria.

The learning device 110, the prediction device 120, and the operating device 130 may each be arranged as shown in any one of FIGS. 1A to 1D.

As shown in FIG. 1A, both the learning device 110 and the prediction device 120 may be placed in the space 140 along with the instrument/equipment 145. The learning device 110 can generate an artificial intelligence model by directly receiving data to be learned from the instrument/equipment 145, and the prediction device 120 uses data generated in real time from the instrument/equipment 145 to create an artificial intelligence model. It can be classified using a model.

The operating device 130 communicates with each component 110, 120, and 145 and can transmit and receive necessary data. The operating device 130 may receive input data from the prediction device 120 and the first and second values calculated by inputting the input data into the artificial intelligence model, and the operating device 130 may receive the health of the artificial intelligence model. The data and selected data can be notified to the learning device 110 or the relevant person.

The person concerned can check the first and second values calculated using the prediction device 120 in real time within the space 140, and the person or labeler can use the operation device 130 regardless of location to model the artificial intelligence model. You can judge your health.

The learning device 110 can receive data through wired communication with the instruments/equipment 145, and the learning device 110 is implemented in a form such as edge computing and can directly receive data from the instrument/equipment 145. You can.

As shown in FIG. 1B, only the prediction device 120 is placed in the space 140 to perform prediction, and the learning device 110 and the operating device 130 are located in separate places. ) Alternatively, necessary data can be transmitted and received with the prediction device 120.

In this case, encryption may be performed on transmitted and received data, especially data transmitted from the device/equipment 145 to the learning device 110. Data output or previously output and accumulated from the device/equipment 145 may be an important asset in a specific industry and may necessarily require security. Accordingly, the device/equipment 145 can encrypt and transmit the data to be transmitted, and the learning device 110 can receive and decrypt it to create an artificial intelligence model.

As shown in FIG. 1C, all devices 110 to 130 on the artificial intelligence model health management system 100 can be arranged and operated in the space 140.

As shown in FIG. 1D, the prediction device 120 and the operation device 130 are placed in the space 140, and the learning device 110 may be placed in a separate location.

However, it is not necessarily limited to this, and as long as each component 110 to 130 can perform the above-described operation, it may be arranged in any way in relation to the space 140.

A schematic operation of each component 110 to 130 in the artificial intelligence model health management system 100 is shown in FIG. 18.

Figure 18 is a timing chart showing how the artificial intelligence model health management system determines the reliability of the artificial intelligence model according to some embodiments of the present disclosure.

The learning device 110 may generate an artificial intelligence model and transmit it to the prediction device 120 (S2410).

The prediction device 120 may store the received artificial intelligence model (S2415).

Input data may be received in real time from the instrument/equipment 145 by the prediction device 120 (S2420).

The prediction device 120 may calculate a first value related to class classification of the received input data and a second value related to property classification of the input data using an artificial intelligence model (S2425).

The prediction device 120 may transmit the received input data and their first and second values to the operating device 130 (S2430).

The operating device 130 calculates the reliability of the received first value (S2435). Here, the reliability of the first value can be calculated using the second value.

The operating device 130 may determine the health of the artificial intelligence model from the calculated reliability or select input data for re-learning (S2440).

The operating device 130 may transmit selected input data for re-learning to the learning device 110 (S2445).

The learning device 110 may perform re-learning with selected data for the artificial intelligence model (S2450).

Figure 2 is a diagram illustrating the configuration of a learning device according to some embodiments of the present disclosure.

Referring to FIG. 2, the learning device 110 according to some embodiments of the present disclosure may include a communication unit 210, a preprocessing unit 220, a training unit 230, and a memory unit 240. The above-described components are not essential for configuring the learning device 110, so the learning device 110 described in this disclosure may have more or less components than the components listed above.

The communication unit 210 receives labeled input data, cost functions, optimization methods, and architecture from the apparatus/equipment 145 or a stakeholder terminal (not shown), and selects data for learning or re-learning from the operating device 130. can receive. The communication unit 210 receives labeled input data, a cost function, and an optimization method from the apparatus/equipment 145 or a stakeholder terminal (not shown), and in some cases, may receive an architecture.

The communication unit 210 may additionally receive labeled input data for testing from the instrument/equipment 145 or a stakeholder terminal (not shown).

The preprocessing unit 220 may perform preprocessing so that the training unit 230 can train the received labeled input data or selected data. Labeled input data or selected data is data that the instrument/equipment 145 outputs for its own purposes, and may be different from the data format for learning by the training unit 230. The pre-processing unit 220 may transform the format of the data so that the received, labeled input data or data can be transferred to the training unit 230 and used for learning without difficulty.

The training unit 230 can train a model using labeled input data, a cost function, an optimization method, and an architecture. The training unit 230 may select the model architecture on its own or may receive the architecture from an external source. Here, an artificial neural network may be used as the architecture, and various architectures may be selected depending on the type of input data.

For example, if the input data is an image, the architecture may be implemented with a convolutional neural network (CNN), etc. For input data other than images, the architecture can be implemented as a multi-layer perceptron. Alternatively, if the input data includes time series elements, the architecture can be implemented as a Recurrent Neural Network (RNN). However, the above-mentioned example is only an example, and depending on the type of input data, the architecture may be implemented in various types.

The training unit 230 may conduct learning based on the selected or received architecture, labeled input data, cost function, and optimization method. The training unit 230 randomly selects each parameter in the architecture and calculates a cost function by applying the labeling result and input data to the architecture for which the parameter has been selected. Afterwards, the training unit 230 converts the parameters according to the optimization method and selects parameters to reduce the cost value. An example of an optimization method is gradient descent, which is a method of adjusting parameters in a direction where the slope of the cost function decreases. Through this process, the training unit 230 can select the parameter with the smallest cost value when the labeled input data is applied to the selected architecture and learn the model. An artificial intelligence model is created according to this learning process of the training unit 230.

The artificial intelligence model generated by the training unit 230 has the characteristic of minimizing the cost value. As mentioned above, the cost function is proportional to the difference between the first value and the correct value, or at the same time, proportional to the distance between points in the latent space of two input data of the same class and/or the latent space of two input data of different classes. It may correspond to a value inversely proportional to the distance between the above points. Accordingly, when input data is input, the generated artificial intelligence model can predict the correct answer value of the input data as close as possible (minimizing the difference in probability for each class). In addition, the artificial intelligence model makes predictions as described above according to the definition of the cost function, and at the same time arranges the same classes to cluster as closely as possible in the latent space. The form in which the predicted values are arranged in the latent space by the training unit 230 is shown in FIGS. 3 and 4.

FIG. 3 is a diagram illustrating a method of clustering according to classes by a learning device according to some embodiments of the present disclosure, and FIG. 4 is a diagram illustrating classes clustered in various ways according to classes by a learning device according to some embodiments of the present disclosure. This is a drawing illustrating.

As described above, since the data placed on the latent space has a matrix or a vector value derived therefrom, the number of dimensions increases according to its components. For example, if the vector has n components, the latent space has n dimensions. The drawings shown in FIGS. 3 and 4 show a two-dimensional projection of this n-dimensional latent space.

If the cost function is a first cost function or a second cost function that is defined as a number proportional to the distance between the predicted value and other data of the same class, the gap between data (predicted values) between each class gradually narrows as learning progresses. As shown in the drawing on the far right, each class is clustered together. The form of clustering can be determined depending on how the cost function is defined.

Figure 4(a) is a diagram showing the same classes arranged in clusters according to distance, and Figure 4(b) or Figure 4(c) is a diagram showing the same classes arranged clustered at the same angle.

As can be seen with reference to FIGS. 3 and 4, the artificial intelligence model generated by the training unit 230 can not only classify classes but also cluster the classified classes.

Referring again to FIG. 2, the training unit 230 may test an artificial intelligence model created using labeled input data for testing. Since labeled input data is received for testing purposes, the training unit 230 can predict the class of the input data based on the input data and compare the predicted value and the labeling value (result value). The training unit 230 can check the correct answer rate of the prediction of the artificial intelligence model based on the comparison results.

In creating an artificial intelligence model (process of learning the model) according to the above-described process, the training unit 230 creates an artificial intelligence model using (labeled) input data selected for learning from the operating device 130. can be created. The input data selected by the operating device 130 may have minimal noise and may include data from the first to third groups in an appropriate ratio. Accordingly, the artificial intelligence model generated by the training unit 230 may have relatively high accuracy compared to a conventional artificial intelligence model that is learned by indiscriminately inputting numerous input data.

Additionally, the training unit 230 may retrain a previously created artificial intelligence model using input data selected for relearning from the operating device 130. When the input data of the second or third group increases, a situation occurs in which the artificial intelligence model created by learning with the input data of the first group has difficulty predicting completely. Accordingly, the training unit 230 receives selected (labeled) input data for re-learning from the operating device 130, and uses the selected input data for the architecture, cost function, and optimization method of the generated artificial intelligence model. Retrain the artificial intelligence model by learning again. Accordingly, the prediction accuracy (correct answer rate) of the artificial intelligence model can be improved.

The memory unit 240 stores (labeled) input data received from the instrument/equipment 145 and the artificial intelligence model generated by the training unit 230. Furthermore, the memory unit 240 may store selected (labeled) input data for learning or re-learning received from the operating device 130 and the artificial intelligence model re-trained by the training unit 230. The memory unit 240 not only stores the (labeled) input data and the artificial intelligence model learned therefrom, but also stores the (labeled) input data selected for re-learning and the artificial intelligence model re-learned from it, so that it can be stored later. , The selected (labeled) input data for re-learning and the re-trained artificial intelligence model can be distinguished and distributed (transmitted or sold, etc.).

The method by which the learning device 120 learns the model is shown in FIG. 17.

Figure 17 is a timing chart showing how a learning device learns a model according to some embodiments of the present disclosure.

The preprocessor 220 may receive labeled input data, architecture, cost function, and optimization method from the device/equipment 145 through the communication unit 210 (S2310).

The preprocessor 220 may perform preprocessing of the received (labeled) input data (S2320).

The training unit 230 may train the model by calculating and selecting parameters within the architecture with the lowest cost value according to the optimization method (S2330).

The training unit 230 may receive labeled input data and output data for testing from the instrument/equipment 145 (S2340).

The training unit 230 may generate predicted values (first value and second value) by inputting test input data to the artificial intelligence model (S2350).

The training unit 230 may determine the accuracy of the artificial intelligence model by comparing the predicted value (first value) of the input data for testing and the output data (S2360).

The memory unit 240 may store the generated artificial intelligence model (S2370).

Through the above-described process, an artificial intelligence model is created, and tests can be performed on the created artificial intelligence model.

Figure 5 is a diagram showing the configuration of a prediction device according to the first embodiment of the present disclosure.

Referring to FIG. 5, the prediction device 120 according to the first embodiment of the present disclosure may include a communication unit 510, a prediction unit 520, and a memory unit 530. The above-described components are not essential for configuring the prediction device 120, so the prediction device 120 described in this disclosure may have more or less components than the components listed above.

The communication unit 510 receives an artificial intelligence model from the learning device 110 and input data from the instrument/equipment 145 in real time. Additionally, the communication unit 510 may transmit the first and second values, which are outputs of the artificial intelligence model, to the operating device 130 along with the received input data.

The prediction unit 520 calculates a first value for classifying the class of input data input in real time using the received artificial intelligence model, and then classifies the input data into one of a plurality of classes using the first value. Classes can be classified.

When predicting input data using an artificial intelligence model, the prediction unit 520 does not output which class the first value is, but derives a probability corresponding to each class. The result may be derived that the first value corresponds to a specific class with overwhelming probability, but the result may be derived that the first value corresponds to a specific class with a slight (probability) difference. The prediction unit 520 may use an artificial intelligence model to make predictions for class classification using a probability value for which class the input data will correspond to.

Meanwhile, the prediction unit 520 may calculate a second value related to property classification of the input data using an artificial intelligence model. The second value may mean a coordinate value indicating the location of the corresponding input data point in the latent space, but the present disclosure is not limited thereto.

The received artificial intelligence model corresponds to a model learned to minimize the cost values of the first cost function and the second cost function. Accordingly, when the corresponding input data point is placed at the second value position predicted by the artificial intelligence model prediction unit 520, it is clustered and arranged in the latent space for each class.

The memory unit 530 stores the received artificial intelligence model and input data, and stores the predicted value predicted by the prediction unit 520.

Figure 6 is a diagram showing the configuration of an operating device according to a first embodiment of the present disclosure.

Referring to FIG. 6, the operating device 130 according to the first embodiment of the present disclosure includes a communication unit 610, a reliability calculation unit 620, a health level determination unit 630, a data selection unit 640, and a labeling guide unit. It may include 650 and a memory unit 660. Furthermore, the operating device 130 may further include a display unit 670. The above-described components are not essential for configuring the operating device 130, so the operating device 130 described in this disclosure may have more or less components than the components listed above.

The communication unit 610 receives input data, the first value, and the second value from the prediction device 120, and transmits the health determination result or selected data to the learning device 110 or an employee terminal (not shown). . The communication unit 610 may receive input data, a first value, and a second value for a preset period from the prediction device 120, and may continuously receive data for a preset period.

The reliability calculation unit 620 may determine the reliability of each first value using the second value received from the prediction device 120. As described above, the artificial intelligence model used by the prediction device 120 for prediction corresponds to a model learned to minimize the cost value of the first cost function. Accordingly, the input data point may be placed on the latent space, and the second value may mean a coordinate value indicating the location of the input data point on the latent space. Accordingly, the reliability calculation unit 620 may calculate the reliability of the first value using the second value. As mentioned above, confidence is the distance between each class (more specifically, the centroid of each class) and the corresponding input data point (located at the second value coordinate) in the latent space; more specifically, the distance between each class and the corresponding input data point It can be calculated as the ratio of the distance between them. Reliability can be calculated as follows with reference to FIGS. 7 and 8.

FIG. 7 is a diagram showing the relationship between each class classified by an operating device according to some embodiments of the present disclosure and the latent space of the input data point, and FIG. 8 is a diagram showing the relationship between the classes classified by the operating device according to some embodiments of the present disclosure. This is a diagram showing how each class is arranged in the latent space.

Referring to FIG. 7, the corresponding input data point (l) is placed in the latent space from one input data by the artificial intelligence model. In the latent space, the input data point (l) can be expressed as a second value that is a coordinate value in the latent space. There may be n classes (Class 1 to Class n) in the latent space, and for each class, data classified into each class may exist clustered with a certain radius (r1 to rn). Each class has a centroid (c1 to cn). There is a center (C) of the entire class, and a radius (rC) equal to the preset standard value exists.

Here, the data placed on the latent space has a matrix or a vector value derived therefrom. Accordingly, each center can be calculated as the average value of the corresponding data. For example, the centroid (c1 to cn) of each class may be the average value of data included in each class, and the centroid (C) of all classes may be the average value of the centroids (c1 to cn) of each class.

The latent space is defined as follows, and depending on the reliability calculation method, the reliability calculation unit 620 can calculate the distance (d1 to dn) between the predicted value (l) and each class, more specifically, the center (c) of each class. there is.

Referring to FIG. 8, the reliability calculation unit 620 can classify the properties of input data based on the calculated reliability.

According to the reliability operation, if the distance between the center (c1, c2, cn) of a specific class and the input data point (l) is less than the radius (r1, r2, rn) of the specific class, the corresponding first value is the first group ( 810) and has high reliability. Conversely, if the distance between the input data point (l) and the center (C) of the entire class is greater than the radius (rC) (of the entire class), the corresponding first value corresponds to the third group 830, and (is it the correct answer? (regardless of whether or not) has low reliability.

If neither of the above two conditions applies, or the distance between the center (ci) of a specific class and the input data point (l) is greater than the radius (r) of all classes, but the distance to the centers of two or more classes is relatively If it is smaller than the distance to the center of the class, the first value corresponds to the second group 820 and has low reliability. The reliability of the first value corresponding to the second group 820 is not close to the center of any one class, and the closer it is to the boundary between classes, the lower the reliability becomes.

Referring again to FIG. 5, the reliability calculation unit 620 calculates the reliability of each first value predicted by the prediction device 120 by receiving input data in the same manner as described above.

The health level determination unit 630 determines the health level of the artificial intelligence model using the second value received from the prediction device 120. The health determination unit 630 uses the reliability calculated by the reliability calculation unit 620 to determine the health of the artificial intelligence model in the following manner.

As one method, the health determination unit 630 calculates the average of the number or reliability of the first values classified into the second group 820 and observes changes in the number or reliability. Since the health determination unit 630 can obtain the above-described data from the currently received second value and can obtain the above-mentioned data from the previously received second value, the health determination unit 630 determines the second group ( 820), a change in the number or reliability of the classified first values can be observed. The fact that the average reliability is lower than before means that there is more data classified into the second group (820), and at the same time, it means that there is more data that is ambiguous to classify into classes. It can be seen that the increase in borderline data classified into the second group 820 is the time when relabeling and relearning of the data should be performed. Accordingly, the health level determination unit 630 may determine that the health level of the artificial intelligence model has decreased (below a preset standard value) in the above-described situation.

Alternatively, the health determination unit 630 calculates the average of the number or reliability of the first values classified into the third group 830 and observes changes in the number or reliability. A large number of data classified into the third group 830 may mean an increase in noise, but may also mean that new classes of data are increasing. Additionally, if the average reliability suddenly increases or suddenly decreases compared to before, it may mean that data of a new class is increasing. Accordingly, it can be seen that the increase in data classified into the third group 820 is also a time when relabeling and relearning of the data should be performed. Accordingly, the health level determination unit 630 may determine that the health level of the artificial intelligence model has decreased (below a preset standard value) in the above-described situation.

Alternatively, the health determination unit 630 observes changes in diameter (ri) for each class. An increase in the diameter of a specific class means that the number of data classified into a specific class has increased. This means that even data that was previously classified or could be classified into the second group 820 in the latent space has been classified into a specific class. Conversely, the diameter of a particular class may suddenly decrease. This also results in a situation where data that has conventionally been classified or could be classified into a specific class is classified into the second group 820. Accordingly, the health level determination unit 630 may determine that the health level of the artificial intelligence model has decreased (below a preset standard value) in the above-described situation.

In another method, the health determination unit 630 observes the change in coordinates of the center (ci) of each class. Compared to before, the sudden change in the coordinates of the center (ci) of each class may mean that a rapid change has occurred in the operating environment of the device/equipment 145 within the space 140. Such rapid changes in the environment can affect the accuracy of artificial intelligence model predictions. Accordingly, the health determination unit 630 may determine that the health of the artificial intelligence model has been lowered (below a preset standard value) even when the coordinates of some or all of the class centers (ci) change.

In this way, the health determination unit 630 performs various judgments using reliability and determines the health of the artificial intelligence model used by the prediction device 120 for prediction. When the health of the artificial intelligence model falls below a preset standard, the health judgment unit 630 may determine that it is time to retrain the artificial intelligence model. Accordingly, the health level determination unit 630 causes the display unit 670 to display that the health level has decreased and that it is time for re-learning. In addition, the health determination unit 630 may transmit this to the learning device 110 or a stakeholder terminal (not shown) through the communication unit 610.

When the health level determination unit 630 determines that retraining of the artificial intelligence model is necessary (health level has decreased), the data selection unit 640 selects data to be used for retraining. The data selection unit 640 selects data to be used for re-learning from the input data received from the prediction device 120 and input during a preset period. As described above, the reliability calculation unit 620 calculates the reliability and then classifies the properties of the received input data into each group (first to third groups) according to the reliability of each predicted value. The data selection unit 640 selects data by each property (first group 810 to third group 830) at a preset ratio from the received input data.

At this time, the data selected by the data selection unit 640 may be selected based on different criteria for each group. Within the first group, the data selection unit 640 may select a certain number of data in order of high or low reliability. Within the second group, the data selection unit 640 may select a certain number of data in descending order of reliability (data arranged more on the border). Within the third group, the data selection unit 640 may select a certain number of data in the order of their distance from the center (C) of all classes. The data selection unit 640 can further increase the correct answer rate of the artificial intelligence model to be retrained by selecting data for each characteristic based on the above-mentioned criteria.

The labeling guide unit 650 selects data that can help the labeler label the data classified into the second or third group as samples. The display unit 670 outputs data classified into the second or third group, allowing the labeler to easily relabel the data. At this time, the labeling guide unit 650 selects the display unit 670 to output data that can be helpful in (re)labeling the data as samples along with the corresponding data. When the display unit 670 outputs data classified into the second or third group, the sample selected by the labeling guide unit 650 is also output to assist the labeler's (re)labeling work. Examples of selection by the labeling guide unit 650 are shown in FIGS. 19 and 20.

19 and 20 are diagrams illustrating screens on which a display unit outputs a sample selected by a labeling guide unit according to some embodiments of the present disclosure.

Referring to FIG. 19, the labeling guide unit 650 may select data of various classes to which input data may correspond and output them together as samples. For example, input data may be predicted by the prediction device 120 with an overwhelming probability for one class, but may also be predicted with a slight difference in probability for multiple classes. In the latter case, the labeling guide unit 650 may select random data that has been previously classified into classes with small probability differences as samples. As shown in FIG. 19 , there may be a situation where the input data 2510 to the prediction device 120 is predicted to be one of class 1, class 2, class 3, and class 4 with a small probability difference. In this case, the labeling guide unit 650 predicts class 1 with an overwhelming probability, or one of the data 2520 labeled as class 1, one of the data 2530 described above for class 2, or one of the data 2530 for class 3. Select one of the above-described data 2540 and for class 4 one of the above-described data 2550. The display unit 670 outputs each selected data, allowing the labeler to more easily label the input data 2510 by referring to the output data.

Additionally, referring to FIG. 20, the labeling guide unit 650 may select various data having the same properties as the input data as samples and output them together. As described above, data classified into the second or third group within the predicted value are on the boundary of each class or correspond to data of a completely different class. In displaying such data, the labeling guide unit 650 may select other data classified into the same nature (group) as samples and output them together. For example, when the input data 2510 shown in FIG. 20 is classified into the second group, the labeling guide unit may select an arbitrary number of other data 2610 to 2640 selected as the second group as samples. there is. The display unit 670 outputs each selected data, allowing the labeler to check the data previously selected as the second group and refer to what class they were labeled as.

Referring again to FIG. 6, the memory unit 660 stores the input data and predicted values (first and second values) received from the prediction device 120, and selects the data selected by the data selection unit 640. Save. In particular, the data selected by the data selection unit 640 is effective data for retraining the artificial intelligence model and can be an excellent asset in relation to the artificial intelligence model. Accordingly, the memory unit 660 may separately store the data selected by the data selection unit 640.

The display unit 670 outputs input data determined to be the second or third group by the reliability calculation unit 620. By outputting the above-described input data, the display unit 670 can allow the labeler to check and re-label them (in order for the artificial intelligence model to retrain them). When outputting the corresponding data, the display unit 670 may output the data selected by the labeling guide unit 650 together to assist the labeler in re-labeling.

Figure 9 is a timing chart showing a method by which an operating device determines the health of an artificial intelligence model according to the first embodiment of the present disclosure.

The prediction device 120 can receive data in real time from the instrument/equipment 145 and perform prediction (S910).

Specifically, the prediction unit 520 calculates a first value by inputting the received input data to the artificial intelligence model, and performs prediction to classify the input data into one of a plurality of classes based on the first value. It can be done. Additionally, the prediction unit 520 may input input data into the artificial intelligence model and obtain a second value related to property classification of the input data. Here, the nature of the input data may be related to reliability, which refers to the degree to which one can be confident that the input data is classified into a specific class, regardless of whether or not the correct answer is correct, when performing a prediction that classifies the input data into a specific class. That is, the properties of the input data may be determined based on whether the reliability of the first value for the input data is high.

The prediction device 120 may transmit input data input during a preset period and prediction values (first value and second value) for each input data to the operation device 130 (S920).

The operating device 130 may calculate the reliability of the received first value using the second value (S930).

The operating device 130 may determine the health of the artificial intelligence model of the prediction device 120 based on the calculated reliability (S940).

Specifically, the health level determination unit 630 may determine whether the health level satisfies preset conditions. The preset conditions include whether there has been a change in the number or reliability of predicted values classified into the second group 820, whether there has been a change in the number or reliability of predicted values classified into the third group 830, and the diameter for each class. It may be whether a change occurred in (ri) or whether a change occurred in the coordinates of the center (ci) of each class.

The operating device 130 may output the determined health level (S950).

Specifically, if the health level does not satisfy the preset conditions, the operating device 130 may directly output the determined health level using the display unit 670, and in some cases, the operating device 130 may use a learning device ( 110) or it can be transmitted to the relevant person’s terminal (not shown).

FIG. 10 is a timing chart illustrating a method by which an operating device selects data to learn or re-learn according to the first embodiment of the present disclosure.

The prediction device 120 can receive data in real time from the instrument/equipment 145 and perform prediction (S1010).

Specifically, the prediction unit 520 calculates a first value by inputting the received input data to the artificial intelligence model, and performs prediction to classify the input data into one of a plurality of classes based on the first value. It can be done. Additionally, the prediction unit 520 may input input data into the artificial intelligence model and obtain a second value related to property classification of the input data. Here, the nature of the input data may be related to reliability, which refers to the degree to which one can be confident that the input data is classified into a specific class, regardless of whether or not the correct answer is correct, when performing a prediction that classifies the input data into a specific class. That is, the properties of the input data may be determined based on whether the reliability of the first value for the input data is high.

The prediction device 120 may transmit the input data and each predicted value (first value and second value) input during a preset period to the operation device (S1020).

The operating device 130 may calculate the reliability of the received first value using the second value (S1030).

Based on the calculated reliability, the operating device 130 selects data in which the input data point is completely classified into a specific class in the latent space, data in which the input data point is located on the boundary of two or more classes in the latent space, and data in which the input data point is located on the boundary of two or more classes in the latent space. The nature of the data is classified into data that is separated from all classes by more than a preset standard (S1040). Here, since the second value is a coordinate value indicating where the input data point is placed in the latent space, the position where the input data point is placed in the latent space can be confirmed using the second value.

The operating device 130 classifies the first value associated with each input data into one of the first group 810 to the third group 830 based on the calculated reliability.

The operating device 130 may select a preset number of data for each property (S1050). The operating device 130 may select a preset number of data for each property at a certain ratio.

The operating device 130 may transmit the selected data to the learning device 110 (S1060).

The learning device 110 may retrain the artificial intelligence model using data selected by the operating device 130 (S1070).

FIG. 11 is a timing chart illustrating a method in which an operating device according to the first embodiment of the present disclosure selects and outputs a sample along with a predicted value of input data.

The prediction device 120 can receive data in real time from the instrument/equipment 145 and perform prediction (S1110).

Specifically, the prediction unit 520 calculates a first value by inputting the received input data to the artificial intelligence model, and performs prediction to classify the input data into one of a plurality of classes based on the first value. It can be done. Additionally, the prediction unit 520 may input input data into the artificial intelligence model and obtain a second value related to property classification of the input data. Here, the nature of the input data may be related to reliability, which refers to the degree to which one can be confident that the input data is classified into a specific class, regardless of whether or not the correct answer is correct, when performing a prediction that classifies the input data into a specific class. That is, the properties of the input data may be determined based on whether the reliability of the first value for the input data is high.

The prediction device 120 may transmit the input data input during a preset period and the first and second values for each input data to the operating device 130 (S1120).

The operating device 130 may calculate the reliability of the received first value using the second value (S1130).

Based on the calculated reliability, the operating device 130 selects data in which the input data point is completely classified into a specific class in the latent space, data in which the input data point is located on the boundary of two or more classes in the latent space, and data in which the input data point is located on the boundary of two or more classes in the latent space. The nature of the data is classified into data that is separated from all classes by more than a preset standard (S1140). Here, since the second value is a coordinate value indicating where the input data point is placed in the latent space, the position where the input data point is placed in the latent space can be confirmed using the second value.

The operating device 130 may select data of various classes to which the input data may correspond or various data having the same characteristics as the input data as samples for guiding (S1150). The operating device 130 may select data of various classes to which the input data may correspond as samples for the labeling guide, and select various data having the same properties as the input data as samples for the labeling guide. You can do this, and both can be selected as samples.

The operating device 130 may output each input data and selected samples together (S1160).

FIG. 12 is a diagram illustrating the configuration of a prediction device according to a second embodiment of the present disclosure.

Referring to FIG. 12, the prediction device 120 according to the second embodiment of the present disclosure may include a communication unit 1510, a first prediction unit 1520, a second prediction unit 1530, and a memory unit 1540. You can. The above-described components are not essential for configuring the prediction device 120, so the prediction device 120 described in this disclosure may have more or less components than the components listed above.

The communication unit 1510 performs the same operation as the communication unit 510. In the present disclosure, the communication unit 1510 may receive a learned or re-trained artificial intelligence model from the learning device 110. However, the present disclosure is not limited to this.

The memory unit 1540 can store input data and artificial intelligence models. Here, the input data may refer to data input in real time from the instrument/equipment 145, and the artificial intelligence model may be generated by the learning device 110 and received through the communication unit 1510. However, the present disclosure is not limited to this.

According to some embodiments of the present disclosure, the artificial intelligence model shares a first sub-model that calculates a first value related to class classification of input data and a layer related to feature extraction for input data with the first sub-model, and a second sub-model. It may include a second sub-model that outputs a value. A detailed description of this will be provided later with reference to FIG. 21.

In the present disclosure, the prediction device 120 may include a first prediction unit 1520 and a second prediction unit 1530, or may include one prediction unit 520 as shown in FIG. 5 .

The first prediction unit 1520 may use the received artificial intelligence model to calculate a first value for classifying the class of input data input in real time. Additionally, the first prediction unit 1520 may perform prediction to classify input data into one of a plurality of classes based on the first value.

The second prediction unit 1530 may use the received artificial intelligence model to calculate a second value related to classification of the properties of input data input in real time.

The memory unit 1540 may store the received artificial intelligence model and input data, as well as the first and second values predicted by each prediction unit 1520 and 1530, respectively.

In this way, the prediction device 120 can have the following advantages by calculating the first value and the second value through a plurality of prediction units. Prediction of the first value for class classification in real time can be performed by the first prediction unit 1520 with relatively high accuracy, thereby ensuring prediction accuracy. In addition, in order to determine the reliability of the operating device 130 or select data, the second value is predicted by the second prediction unit 1530, so that the operating device 130 confirms the location of the corresponding input data point in the latent space. Allows the reliability of the first value to be calculated.

Figure 13 is a diagram showing the configuration of an operating device according to a second embodiment of the present disclosure.

Referring to FIG. 13, the operating device 130 according to the second embodiment of the present disclosure includes a communication unit 1610, a reliability calculation unit 1620, a health determination unit 1630, a data selection unit 1640, and a labeling guide unit. It includes (1650) and a memory unit (1660). Furthermore, the operating device 130 may further include a display unit 1670. The above-described components are not essential for configuring the operating device 130, so the operating device 130 described in this disclosure may have more or less components than the components listed above.

The communication unit 1610 receives input data and predicted values (first and second values) from the first prediction unit 1510 and the second prediction unit 1520 of the prediction device 120, and provides a health determination result or selection One data can be transmitted to the learning device 110 or a stakeholder terminal (not shown). Since the input data is the same for both prediction units 1510 and 1520, the communication unit 1610 can receive each predicted value (first value and second value) of both prediction units 1510 and 1520 along with the input data. there is.

The reliability calculation unit 1620 also performs the same operation as the reliability calculation unit 620. However, since the reliability calculation unit 1620 needs distance information in the latent space to perform the reliability calculation, etc., the reliability calculation unit 1620 uses the first prediction unit 1520 to perform the reliability calculation. The second value predicted by the second prediction unit 1530, rather than the 1 value, can be used.

The health determination unit 1630 also performs the same operation as the health determination unit 630.

The data selection unit 1640 also performs the same operation as the data selection unit 640. However, the data selection unit 1640 may also use the second value predicted by the second prediction unit 1530 rather than the first value predicted by the first prediction unit 1520 to select data.

The labeling guide unit 1650 also performs the same operation as the labeling guide unit 650. However, the labeling guide unit 1650 may use the first value predicted by the first prediction unit 1520 with relatively high accuracy when selecting data of various classes to which the input data may correspond as samples.

On the other hand, the labeling guide unit 1650 uses the second value predicted by the second prediction unit 1530 when selecting various data having the same properties as the input data as samples.

The memory unit 1660 and the display unit 1670 also perform the same operations as the memory unit 660 and the display unit 670, respectively.

Figure 14 is a timing chart showing a method by which an operating device determines the health of an artificial intelligence model according to a second embodiment of the present disclosure.

The prediction device 120 receives data in real time from the instrument/equipment 145, and the first prediction unit 1520 and the second prediction unit 1530 each perform prediction (S1710).

The prediction device 120 may transmit input data input during a preset period and prediction values (first and second values) from the first prediction unit 1520 and the second prediction unit 1530 (S1720).

The operating device 130 may calculate the reliability of the first value received from the first prediction unit 1520 using the second value from the second prediction unit 1530 (S1730).

The operating device 130 determines the health of the artificial intelligence model of the prediction device 120 based on the calculated reliability (S1740).

Specifically, the health level determination unit 1630 may determine whether the health level satisfies preset conditions.

The operating device 130 may output the determined health level (S1750). Specifically, if the health level does not satisfy preset conditions, the display unit 1670 may output the determined health level. The operating device 130 may directly output the determined health level using the display unit 670, and in some cases, the operating device 130 may transmit it to the learning device 110 or a stakeholder terminal (not shown).

FIG. 15 is a timing chart illustrating a method by which an operating device selects data to learn or re-learn according to a second embodiment of the present disclosure.

The prediction device 120 receives data in real time from the instrument/equipment 145, and the first prediction unit 1520 and the second prediction unit 1530 can each perform prediction (S1810).

The prediction device 120 may transmit the input data input during a preset period, the first value from the first prediction unit 1520, and the second value from the second prediction unit 1530 (S1820).

The operating device 130 may calculate the reliability of the received first value using the predicted value from the second prediction unit 1530 (S1830).

Based on the calculated reliability, the operating device 130 selects data in which the input data point is completely classified into a specific class in the latent space, data in which the input data point is located on the boundary of two or more classes in the latent space, and data in which the input data point is located on the boundary of two or more classes in the latent space. The nature of the data is classified into data that is separated from all classes by more than a preset standard (S1840). Here, since the second value is a coordinate value indicating where the input data point is placed in the latent space, the position where the input data point is placed in the latent space can be confirmed using the second value.

The operating device 130 selects a preset number of data for each property (S1850). The operating device 130 may select a preset number of data for each property at a certain ratio.

The operating device 130 transmits the selected data to the learning device 110 (S1860).

The learning device 110 re-trains the artificial intelligence model using data selected by the operating device 130 (S1870).

FIG. 16 is a timing chart illustrating a method in which an operating device according to a second embodiment of the present disclosure selects and outputs a sample along with a predicted value of input data.

The prediction device 120 receives data in real time from the instrument/equipment 145, and the first prediction unit 1520 and the second prediction unit 1530 respectively make predictions (S1910).

The prediction device 120 transmits the input data and the prediction value from the second prediction unit 1530 for a preset period (S1920).

The operating device 130 calculates the reliability of the predicted value received from the first prediction unit 1520 (S1930).

Based on the calculated reliability, the operating device 130 selects data in which the input data point is completely classified into a specific class in the latent space, data in which the input data point is located on the boundary of two or more classes in the latent space, and data in which the input data point is located on the boundary of two or more classes in the latent space. The nature of the data is classified into data that is separated from all classes by more than a preset standard (S1940). Here, since the second value is a coordinate value indicating where the input data point is placed in the latent space, the position where the input data point is placed in the latent space can be confirmed using the second value.

The operating device 130 selects data of various classes to which the input data may correspond or various data having the same properties as the input data as samples for guiding (S1950).

The operating device 130 outputs each input data and selected samples together (S1960).

In FIGS. 9 to 11 and 14 to 18, each process is described as being sequentially executed, but this is merely an illustrative explanation of the technical idea of some embodiments of the present disclosure. In other words, a person skilled in the art to which some embodiments of the present disclosure pertain may change the order depicted in each drawing or perform one or more of the processes without departing from the essential characteristics of the several embodiments of the present disclosure. Since various modifications and variations can be applied by executing in parallel, FIGS. 9 to 11 and 14 to 18 are not limited to the time series order.

Meanwhile, the processes shown in FIGS. 9 to 11 and 14 to 18 can be implemented as computer-readable codes on a computer-readable recording medium. Computer-readable recording media include all types of recording devices that store data that can be read by a computer system. That is, computer-readable recording media include storage media such as magnetic storage media (eg, ROM, floppy disk, hard disk, etc.) and optical read media (eg, CD-ROM, DVD, etc.). Additionally, computer-readable recording media can be distributed across networked computer systems so that computer-readable code can be stored and executed in a distributed manner.

Figure 21 is a diagram for explaining an example of an artificial intelligence model according to some embodiments of the present disclosure. In Figure 21, the first layer, the second layer, and the layer related to feature extraction may be composed of several layers.

Referring to FIG. 21, the artificial intelligence model 700 of the present disclosure includes a first sub-model 710 that outputs a first value 751 when input data 740 is input, and a first sub-model 710 that outputs a first value 751 when input data 740 is input. It may include a second sub-model 720 that outputs 2 values 752. However, the present disclosure is not limited to this, and the artificial intelligence model may include more or fewer sub-models than the first sub-model 710 and the second sub-model 720 described above.

The first value 751 may be expressed as a probability value for each class indicating which class the input data 740 corresponds to. Additionally, the first prediction unit 1520 may classify the input data 740 into one of a plurality of classes based on the first value.

For example, if there are three classes and the input data 740 is input data that must be classified into the first class, the first value may be (0.7, 0.2, 0.1) (the value can be varied depending on reliability). there is. In this case, the first prediction unit 1520 may classify the input data 740 into a first class among a plurality of classes based on the first value.

However, the present disclosure is not limited to the above-described example, and the first value 751 may be a value of various types that can be used to classify the input data 740 into one of a plurality of classes.

The second value 752 may be a value related to classification of the nature of the input data 740. Here, the nature of the input data 740 is such that when the first prediction unit 1520 performs a prediction to classify the input data 740 into a specific class, the input data 740 is classified into a specific class regardless of whether the answer is correct or not. It may be related to reliability, which refers to the degree to which one can be certain. Since the reliability may be determined based on the second value 752, the second value 752 may be a value related to classification of the nature of the input data 740.

Specifically, reliability can be calculated as the distance between each class and the corresponding input data point in the latent space, and more specifically, the ratio of the distance between each class and the corresponding input data point. Here, the distance may mean Euclidian distance, but is not necessarily limited thereto and may mean Mahalanobis, etc.

Since the second value may mean a coordinate value in latent space, the second value may be used when calculating the reliability of the first value.

For example, an input data point with high reliability may have a short distance to the center of a specific class in the latent space, while the distance to the center of the remaining classes may be long (only the distance to the class in which it is classified is short). .

As another example, low-reliability input data points have distances from the centers of all classes that are greater than a preset standard or distances from the centers of two or more classes that are similar (not only to the class in which they are classified, but also to other classes). The distance can be short or long.

In the present disclosure, the second value calculated by the prediction device 120 may be transmitted to the operating device 130 along with the first value and input data, and the operating device 130 may determine the first value based on the second value. By calculating the reliability, the properties of the input data having the corresponding first value can be classified.

Input data can be classified according to its properties as follows.

Data that can be fully classified only into a specific class because the input data point is located within the radius of a specific class in the latent space (hereinafter referred to as 'first group'), and the input data point is located outside the radius of all classes in the latent space, but , data located relatively adjacent to two or more classes (hereinafter referred to as 'second group') (at a distance within a preset standard value), and data whose input data points are separated from all classes in the latent space by more than a preset standard value (hereinafter referred to as 'second group'). Hereinafter, it may be classified into 'Group 3').

In the present disclosure, the first sub-model 710 may be composed of a set of a plurality of layers to perform the task of calculating a first value indicating which of the plurality of classes it corresponds to, and the second sub-model 720 may be composed of a set of a plurality of layers to perform the task of calculating values related to classification of the properties of the input data 740.

The first sub-model 710 and the second sub-model 720 may share a layer 730 related to feature extraction. Therefore, after the input data 740 is input to the artificial intelligence model 700, the output value output from the layer 730 related to feature extraction is the first layer 731 and the first sub-model 710. 2 may be input to each of the second layers 732 included in the sub model 720.

The layer 730 related to feature extraction may be composed of at least one layer and may be composed of a set of a convolutional layer and a pooling layer. However, the present disclosure is not limited to this, and various types of layers may be used as the layer 730 related to feature extraction.

The first layer 731 included in the first sub-model 710 is a layer that performs the task of calculating a first value indicating which of the plurality of classes the input data corresponds to, and may include at least one layer. there is.

The second layer 732 included in the second sub-model 720 is a layer that performs the task of calculating a second value related to classification of the properties of the input data 740 and may include at least one layer.

As described above, one artificial intelligence model performs a plurality of tasks (a task for calculating a first value and a task for calculating a second value) including the first sub-model 710 and the second sub-model 720. In this case, the number of parameters can be reduced compared to when performing multiple tasks using multiple artificial intelligence models. Therefore, according to the present disclosure, it can be efficient when operating in real time because the amount of calculation is reduced.

In addition, since the first sub-model 710 and the second sub-model 720 share the layer 730 for extracting features of input data, there are tasks related to each other, so the overall performance of learning the artificial intelligence model 700 can be improved.

According to some embodiments of the present disclosure, the artificial intelligence model 700 includes label data labeled in the training data (the first value and the second value labeled in the learning data) and the predicted value output from the artificial intelligence model (the first value). and a second value) can be learned to update the parameters of the artificial intelligence model by back propagating the difference (error) between them.

According to some embodiments of the present disclosure, the artificial intelligence model 700 may have a deep neural network structure.

A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to the input layer and output layer. Deep neural networks allow you to identify latent structures in data.

According to some embodiments of the present disclosure, when learning or retraining the artificial intelligence model 700, the learning device 110 trains the artificial intelligence model by giving weight to the first sub-model rather than the second sub-model. Alternatively, relearning may be performed.

For example, when the learning device 110 trains or retrains the artificial intelligence model 700 composed of a first sub-model and a second sub-model sharing a layer, the linearity of the first cost function and the second cost function A cost function composed of combinations can be used. In this case, if the linear combination constant is set larger in the second cost function, the accuracy of the first value can be relatively increased.

In the present disclosure, the accuracy of the first value 751, which is the output value of the first sub-model 710, may be more important than the accuracy of the second value 752, which is the output value of the second sub-model 720. Therefore, the learning device 110 uses a learning method such that the accuracy of the first value 751, which is the output value of the first sub-model 710, is prioritized over the accuracy of the second value 752, which is the output value of the second sub-model 720. can be adjusted appropriately.

According to some other embodiments of the present disclosure, the learning device 110 trains or retrains the artificial intelligence model 710 so that the first sub model 710 is trained or retrained before the second sub model. It can be done. In this case, when learning or re-learning the second sub-model 720, the learning device 110 fixes at least one parameter included in the layer 730 related to feature extraction for the input data 740. In this state, learning or re-learning for the second sub model 720 may be performed.

That is, at least one parameter included in the layer 730 related to feature extraction for the input data 740 is confirmed when training or retraining the first sub-model 710, and is used in the second sub-model 720. When performing learning or re-learning, only the remaining parameters included in the second layer 732 may be changed. Here, the second layer 732 may not mean one layer but may mean composed of one or more layers.

According to some other embodiments of the present disclosure, the learning device 110 is a first artificial intelligence model that trains or re-trains all of the first sub-model 710 and the second sub-model 720. Step, a second step of learning the first sub-model 710, and learning of the second sub-model 720 while fixing at least one parameter included in the layer 730 related to feature extraction. Based on the third step, learning or re-learning of the artificial intelligence model 700 may be performed.

Specifically, the learning device 110 learns or re-learns all of the first sub-model 710 and the second sub-model 720 using a high learning rate in the first stage, which is the initial stage of learning of the artificial intelligence model 700. can be performed. In this case, efficiency can be increased by ensuring that the artificial intelligence model 700 quickly achieves a certain level of performance.

Meanwhile, the learning device 110 may learn or retrain each of the first sub-model 710 and the second sub-model 720 in the second and third steps using a lower learning rate than the first step.

For example, when the learning device 110 trains the artificial intelligence model 700 in the second step, the layer 730 related to feature extraction included in the first sub-model 710 has a learning rate lower than that of the first step. ) and learning or re-learning may be performed so that at least one parameter included in the first layer 731 is all updated. And, when training the artificial intelligence model 700 in the third step, the learning device 110 uses the second layer 732 included in the second sub-model 720 at a learning rate lower than the learning rate of the first step. Training or retraining can be performed so that at least one parameter is updated. When the artificial intelligence model 700 is learned in the third step, at least one parameter included in the layer 730 related to feature extraction may be fixed.

As described above, when the artificial intelligence model 700 is trained, the accuracy of the first value predicted by the artificial intelligence model 700 may be prioritized over the accuracy of the second value.

The above description is merely an illustrative explanation of the technical idea of the present embodiment, and those skilled in the art will be able to make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are not intended to limit the technical idea of the present embodiment, but rather to explain it, and the scope of the technical idea of the present embodiment is not limited by these examples. The scope of protection of this embodiment should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of this embodiment.

100: Artificial intelligence model health management system
110: learning device
120: prediction device
130: operating device
140: space
145: Equipment/equipment
210, 510, 610, 1510, 1610: Department of Communications
220: Preprocessing unit
230: Training Department
240, 530, 650, 1540, 1650: Memory unit
520: prediction unit
620, 1620: Reliability calculation unit
630, 1630: Health judgment unit
640, 1640: Data selection unit
660, 1660: Labeling guide unit
670, 1670: display unit
1520: First prediction unit
1530: Second prediction unit

Claims (12)

A communication unit that receives a learned or re-learned artificial intelligence model from a learning device;
A memory unit that stores input data and the artificial intelligence model;
a first prediction unit that classifies the input data into one of a plurality of classes based on a first value output by inputting the input data into the artificial intelligence model; and
a second prediction unit that inputs the input data into the artificial intelligence model to obtain a second value related to classification of the nature of the input data;
Including,
The artificial intelligence model is
a first sub-model composed of a first set of a plurality of layers to output the first value when the input data is input; and
Among the plurality of layers, a layer related to feature extraction for the input data is shared with the first sub-model, and when the input data is input, the first sub-model outputs a second value different from the first value. a second sub-model composed of a second set of layers in which the remaining layers, excluding the set and the layer related to feature extraction, are different;
Including,
The learning device is,
Training the artificial intelligence model so that the accuracy of the first value takes priority over the accuracy of the second value,
Device. delete According to paragraph 1,
The learning device is,
Performing learning or re-learning of the artificial intelligence model with weight on the first sub-model rather than the second sub-model,
Device. According to paragraph 1,
The learning device is,
Perform training or re-training on the artificial intelligence model so that the first sub-model is trained or re-trained before the second sub-model,
When learning or re-learning the second sub-model, learning or re-learning the second sub-model is performed while fixing at least one parameter included in the layer.
Device. According to paragraph 1,
The learning device is,
A first step of learning or re-learning the artificial intelligence model, a second step of learning the first sub-model, and the second sub-model with at least one parameter included in the layer fixed. Learning or re-learning the artificial intelligence model based on the third step of learning the model,
Device. According to paragraph 1,
The above properties are,
When the first prediction unit performs a prediction to classify the input data into a specific class, it is related to reliability, which refers to the degree to which the input data can be confident that the input data is classified into the specific class, regardless of whether the answer is correct or not.
Device. In the method of receiving input data, classifying classes, and classifying the properties of the input data, the method is:
Receiving a learned or re-trained artificial intelligence model from a learning device through a communication unit of the device;
Storing input data and the artificial intelligence model in a memory unit of the device;
Inputting the input data into the artificial intelligence model through a first prediction unit of the device and classifying the input data into one of a plurality of classes based on a first value output; and
Inputting the input data into the artificial intelligence model through a second prediction unit of the device to obtain a second value related to classification of the nature of the input data;
Including,
The artificial intelligence model is
a first sub-model composed of a first set of a plurality of layers to output the first value when the input data is input; and
Among the plurality of layers, a layer related to feature extraction for the input data is shared with the first sub-model, and when the input data is input, the first sub-model outputs a second value different from the first value. a second sub-model composed of a second set of layers in which the remaining layers, excluding the set and the layer related to feature extraction, are different;
Including,
The learning device is,
Training the artificial intelligence model so that the accuracy of the first value takes priority over the accuracy of the second value,
method. delete In clause 7,
The learning device is,
Performing learning or re-learning of the artificial intelligence model with weight on the first sub-model rather than the second sub-model,
method. In clause 7,
The learning device is,
Perform training or re-training on the artificial intelligence model so that the first sub-model is trained or re-trained before the second sub-model,
When learning or re-learning the second sub-model, learning or re-learning the second sub-model is performed while fixing at least one parameter included in the layer.
method. In clause 7,
The learning device is,
A first step of learning or re-learning the artificial intelligence model, a second step of learning the first sub-model, and the second sub-model with at least one parameter included in the layer fixed. Learning or re-learning the artificial intelligence model based on the third step of learning the model,
method. In clause 7,
The above properties are,
When the first prediction unit performs a prediction to classify the input data into a specific class, it is related to reliability, which refers to the degree to which the input data can be confident that the input data is classified into the specific class, regardless of whether the answer is correct or not.
method.

Images