KR20210050414A - Method for training neural network - Google Patents

Abstract

Provided is a computer program stored in a computer readable storage medium for training a neural network is provided. The computer program, when executed, allows on one or more processors to perform the following operations of: displaying a first screen including at least one first object for receiving a selection input for a project; and displaying a second screen for displaying project-related information corresponding to the selected project, wherein the second screen includes at least one of a first output area for displaying time series data obtained from a sensor, a selection area including at least one second object for receiving a selection input related to model retraining, or a second output area for displaying information corresponding to the second object.

Description

How to train a neural network {METHOD FOR TRAINING NEURAL NETWORK}

The present invention relates to a method for training a neural network, and more specifically, to a method for providing a user interface for training a neural network.

Deep learning is a set of machine learning algorithms that attempt a high level of abstraction through a combination of several nonlinear transformation techniques.

Various types of research on machine learning algorithms are being conducted, and accordingly, various deep learning techniques such as a deep neural network, a convolutional neural network, and a recurrent neural network are used in computer vision, It is applied in fields such as speech recognition and natural language processing.

These machine learning algorithms have a complex structure and can output results through complex operations. In order to process data using machine learning algorithms, a considerable understanding of machine learning algorithms must be preceded, and accordingly, users who can use machine learning algorithms are limited.

In addition, as fields using machine learning algorithms diversify, experts in other domains who do not have a significant understanding of machine learning algorithms are also rapidly increasing attempts to incorporate machine learning algorithms into their specialized fields.

Accordingly, there is a need in the art to allow users to easily access machine learning algorithms.

Korean Patent Publication No. 2016-0012537 discloses a neural network learning method and apparatus, and a data processing apparatus.

The present disclosure has been devised in response to the above-described background technology, and an object of the present disclosure is to provide a method for learning a neural network.

A computer program stored in a computer-readable storage medium according to an embodiment of the present disclosure for realizing the above-described task, wherein the computer program is executed in one or more processors, to perform the following operations for learning a neural network. The operations include: displaying a first screen including at least one first object for receiving a selection input for a project; And displaying a second screen for displaying project-related information corresponding to the selected project, wherein the second screen includes a first output area for displaying time series data acquired from the sensor, a selection related to model retraining It may include at least one of a selection area including at least one second object for receiving an input or a second output area for displaying information corresponding to the second object.

In an alternative embodiment of computer program operations to perform the following operations to provide a method for training a neural network, the project is an artificial intelligence-related project for achieving a specific goal using artificial intelligence, and the specific goal is, It may include goals to improve the performance of models to which artificial intelligence is applied.

In an alternative embodiment of computer program operations to perform the following operations for providing a method for training a neural network, the selection region is a region including an object for displaying information related to model training in a second output region As, a performance monitoring selection object for displaying the performance information of the model in the second output area, a training data set selection object for displaying training data set information related to training of the model in the second output area, and the training progress of the current model. A learning console selection object for displaying information on a second output area, a model archive selection object for displaying information on at least one model in a second output area, or anomaly information detected using a model is provided. 2 It may include at least one of the detected anomaly output selection objects for display in the output area.

In an alternative embodiment of computer program operations to perform the following operations for providing a method for training a neural network, further comprising an operation of displaying a training data set output screen in a second output area, and outputting the training data set. The screen shows a list of training data sets listing at least one training data set, an additional training data set object for the user to receive selection inputs for the training data set to be used for training the model, or training data that the user will not use for training the model. It may include at least one of the training data set removal object for receiving a selection input for the set.

In an alternative embodiment of computer program operations to perform the following operations to provide a method for training a neural network, the training data set is the first training data set used for training the model, or the retraining It includes at least one of the new second training data set to be used, and the second training data set may include at least some of the time series data acquired from the sensor in real time and a label corresponding thereto.

In an alternative embodiment of computer program operations to perform the following operations to provide a method for training a neural network, displaying a training data set selection screen allowing a user to select a second training data set is performed. It may contain more.

In an alternative embodiment of computer program operations to perform the following operations for providing a method for training a neural network, the training data set selection screen is a screen allowing a user to select a second training data set, A time variable setting area for filtering data acquired by inputting time series data into the model based on a predetermined first criterion, or a data chunk obtained by entering time series data into the model and dividing the acquired data based on a predetermined second criterion It may include at least one of a data chunk area for displaying.

In an alternative embodiment of computer program operations to perform the following operations to provide a method for training a neural network, the data chunk is input to a model by inputting a plurality of time series data divided based on a second predetermined criterion. Statistical features of each of the acquired data sets may be included.

In an alternative embodiment of computer program operations to perform the following operations to provide a method for training a neural network, the predetermined second criterion is a criterion for detecting misclassified data among data acquired using a model. As, time series data is applied to the model based on at least one of a first point at which data acquired using the model changes from a first state to a second state or a second point at which the output of the model changes from a second state to a first state. It may include a criterion for dividing the data obtained by inputting into a plurality of data chunks.

In an alternative embodiment of computer program operations to perform the following operations for providing a method for training a neural network, the data chunk is calculated to the user through a data chunk recommendation algorithm. It may include at least one of a data chunk including a data chunk, or a data chunk including at least one data chunk having statistical characteristics similar to a data chunk selected by receiving a user selection input signal.

In an alternative embodiment of computer program operations for performing the following operations for providing a method for training a neural network, the operation of displaying a model archive output screen in the second output area may be further included.

In an alternative embodiment of computer program operations to perform the following operations to provide a method for training a neural network, the model archive output screen is a screen that displays information on each of a plurality of models, which is stored in the model archive. It may include at least one of a model list output area for displaying a plurality of models so that they can be viewed at a glance, or a model information output area for displaying information on a selected model by receiving a user's selection input signal.

In an alternative embodiment of computer program operations to perform the following operations to provide a method for training a neural network, the model list is a model trained while working on a project, and a second training data set is input to the trained model. In order to recommend a model that has been retrained and retrained, a newly created model by integrating a model having similar statistical characteristics among a plurality of models included in the model archive, or a model corresponding to the newly entered data to the user, It may include at least one of the models determined based on the hit rates of each of the plurality of models.

In an alternative embodiment of computer program operations for performing the following operations for providing a method for training a neural network, an operation of displaying an anomaly output screen detected in the second output area may be further included.

In an alternative embodiment of computer program operations to perform the following operations to provide a method for training a neural network, the detected anomaly output screen contains information related to anomalous data among data acquired using a model. As a screen for display, an anomaly detection result output area for displaying a list of anomaly data acquired using a model or an anomaly information output area for displaying information on the selected anomaly data by receiving a user selection input signal It may include at least one of.

A method for training a neural network according to an embodiment of the present disclosure for realizing the above-described task, comprising: displaying a first screen including at least one first object for receiving a selection input for a project ; And displaying a second screen for displaying project-related information corresponding to the selected project; wherein the second screen includes a first output area for displaying time series data obtained from the sensor, and related to model retraining. It may include at least one of a selection area including at least one second object for receiving a selection input or a second output area for displaying information corresponding to the second object.

A computing device for providing a method for learning a neural network according to an embodiment of the present disclosure for realizing the above-described task, comprising: a processor including one or more cores; And a memory, wherein the processor displays a first screen including at least one first object for receiving a selection input for a project, and a second screen for displaying project-related information corresponding to the selected project. And the second screen includes a first output area for displaying time series data acquired from a sensor, a selection area or a second object including at least one second object for receiving a selection input related to model retraining It may include at least one of the second output areas for displaying information corresponding to.

The present invention can provide a method for training a neural network.

In order that the features of the present disclosure mentioned above may be understood in detail, in a more specific description, with reference to the following embodiments, some of the embodiments are shown in the accompanying drawings. In addition, like reference numerals in the drawings are intended to refer to the same or similar functions over several aspects. However, the accompanying drawings are merely illustrative of specific exemplary embodiments of the present disclosure, and are not considered to limit the scope of the present disclosure, and other embodiments having the same effect may be sufficiently recognized. Be careful.
1 is a diagram illustrating a block diagram of a computing device performing an operation to provide a method for training a neural network according to an embodiment of the present disclosure.
FIG. 2 is a diagram illustrating a neural network that is an object of learning in a method for training a neural network according to an embodiment of the present disclosure.
3 is a diagram illustrating a first screen by way of example, according to an embodiment of the present disclosure.
4 is a diagram illustrating a second screen by way of example, according to an embodiment of the present disclosure.
5 is a diagram illustrating an output screen of a training data set according to an embodiment of the present disclosure.
6 is a diagram illustrating an example of a screen for selecting a training data set according to an embodiment of the present disclosure.
7 is a diagram illustrating an example of a model archive output screen according to an embodiment of the present disclosure.
8 is an exemplary diagram for describing a hit model according to an embodiment of the present disclosure.
9 is an exemplary diagram for describing an integrated model according to an embodiment of the present disclosure.
10 is a diagram illustrating an example of a detected anomaly output screen according to an embodiment of the present disclosure.
11 is a diagram illustrating an output screen of a monitoring line according to an embodiment of the present disclosure.
12 is a flowchart of a method for training a neural network according to an embodiment of the present disclosure.
13 is a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is clear that these embodiments may be implemented without this specific description.

As used herein, the terms "component", "module", "system" and the like refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or execution of software. For example, a component may be, but is not limited to, a process executed on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a server and a server may be components. One or more components may reside within a processor and/or thread of execution. A component can be localized within a single computer. A component can be distributed between two or more computers. In addition, these components can execute from a variety of computer readable media having various data structures stored therein. Components can be, for example, via a signal with one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system and/or a signal through another system and a network such as the Internet. Depending on the data being transmitted), it may communicate via local and/or remote processes.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless specified otherwise or is not clear from the context, "X employs A or B" is intended to mean one of the natural inclusive substitutions. That is, X uses A; X uses B; Or when X uses both A and B, "X uses A or B" can be applied to either of these cases. In addition, the term “and/or” as used herein should be understood to refer to and include all possible combinations of one or more of the listed related items.

In addition, the terms "comprises" and/or "comprising" should be understood to mean that the corresponding features and/or components are present. However, it is to be understood that the terms "comprising" and/or "comprising" do not exclude the presence or addition of one or more other features, elements, and/or groups thereof. Further, unless otherwise specified or when the context is not clear as indicating a singular form, the singular in the specification and claims should be interpreted as meaning "one or more" in general.

In addition, the term "at least one of A or B" should be interpreted as meaning "when including only A", "when including only B", and "when combined in a configuration of A and B".

Those of skill in the art would further describe the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein, including electronic hardware, computer software, or a combination of both. It should be recognized that it can be implemented as To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or as software depends on the specific application and design limitations imposed on the overall system. Skilled technicians can implement the described functionality in various ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

Description of the presented embodiments is provided so that a person of ordinary skill in the art of the present disclosure can use or implement the present invention. Various modifications to these embodiments will be apparent to those of ordinary skill in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present invention is not limited to the embodiments presented herein. The invention is to be accorded the widest scope consistent with the principles and novel features presented herein.

1 is a diagram illustrating a block diagram of a computing device performing an operation to provide a method for training a neural network according to an embodiment of the present disclosure.

The configuration of the computing device 100 illustrated in FIG. 1 is only an example simplified. In an embodiment of the present disclosure, the computing device 100 may include other components for performing a computing environment of the computing device 100, and only some of the disclosed components may configure the computing device 100.

The computing device 100 may include a processor 110, a memory 130, and a network unit 150.

According to an embodiment of the present disclosure, the processor 110 may provide a user interface for training a neural network. The user interface may include a first screen including at least one first object for receiving a selection input for a project.

The first screen will be described in detail with reference to FIG. 3.

According to an embodiment of the present disclosure, the processor 110 may display a first screen 200 including at least one first object 230 for receiving a selection input for a project. Project 210 may include artificial intelligence-related projects to achieve specific goals using artificial intelligence. Specifically, the project 210 may include a deep learning-related project to achieve a specific goal by using deep learning. The first object 230 may include an object for selecting the project 210. The first object 230 may include a function of moving to a second screen for the selected project 210. The first object 230 may include, for example, an icon on the screen. The first screen 200 may be a screen on which at least one or more projects conducted by a user are displayed. The first screen 200 may include a screen that displays a plurality of projects conducted by a user in an easy to see at a glance. The selection input may include an input signal including information on the item selected by the user. Accordingly, when receiving a selection input, the processor 110 may perform an operation based on the selection or display a result of the selection input on the screen. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the project 210 may include an artificial intelligence-related project for achieving a specific goal using artificial intelligence. The specific goal may include a goal for maintaining a model to which artificial intelligence is applied, and may include, for example, a goal of improving the performance of a model to which artificial intelligence is applied. The specific goal may include, for example, improving the prediction accuracy of the model, shortening the training time of the model, and reducing the amount of computation or power used for model training. A specific goal may also include a goal to create a model for obtaining a forecast (eg, stock price prediction in the case of finance) in a specific domain (eg manufacturing, healthcare, legal, business, finance). The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 110 may display a second screen for displaying project-related information corresponding to the selected project. The second screen may include a screen in which a selection input of the first screen is received and the processor 110 displays project-related information corresponding to the selection input. The foregoing is only an example, and the present disclosure is not limited thereto.

The second screen will be described in detail with reference to FIG. 4.

According to an embodiment of the present disclosure, the processor 110 may display the second screen 300 for displaying project-related information corresponding to the selected project. The second screen 300 may include a screen that displays information on a specific project. The second screen 300 may include a screen for displaying information necessary for the user to improve the performance of the deep learning model. The second screen 300 may include, for example, a user interface on which information for improving the performance of the deep learning model is displayed. Accordingly, the second screen 300 may include model performance information, information on a training data set used for model training, a prediction value output by the processor 110 using the model, and the like. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the second screen includes a first output area 310 for displaying time series data acquired from a sensor, and at least one second object for receiving a selection input related to model retraining. It may include at least one of the included selection area 330 or the second output area 340 for displaying information corresponding to the second object. The first output area 310 may include an area for displaying time series data acquired from a sensor. For example, the first output area 310 may include an area for displaying time series data acquired from a sensor in real time. The time series data obtained from the sensor is, for example, time series data obtained from a sensor attached to the robot joint, time series data obtained from a material measurement sensor, temperature data, wind direction, wind speed data, ultraviolet sensor data, infrared sensor data, and light sensor Data, sound sensor data, and the like. The second output area 340 may include an area for displaying information corresponding to the second object. The processor 110 may receive a selection input for the second object and display information corresponding to the second object. For example, when receiving a selection input for a second object (eg, a training data selection icon), the processor 110 may display training data related information on the second output area 340. The selection area may include an area including at least one second object for receiving a selection input related to model retraining. The selection area 330 may be an area including an object for displaying information related to model training in the second output area. For example, the selection area 330 may include a plurality of icons on the user interface in order to allow the user to view information related to model retraining desired by the user through the second output area. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the selection area is an area including an object for displaying information related to model training in the second output area, and performance monitoring for displaying the performance information of the model in the second output area A selection object 331, a training data set selection object 333 for displaying training data set information related to model training in a second output area, a model for displaying information on at least one or more models in a second output area It may include at least one of the archive selection object 337 or the detected anomaly output selection object 339 for displaying the anomaly information sensed using the model in the second output area. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the performance monitoring selection object 331 may include an object for displaying model performance information in the second output area. For example, when receiving a selection input for the performance monitoring selection object 331, the processor 110 may display model performance information (eg, prediction accuracy of the model) in the second output area. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the training data set selection object 333 may include an object for displaying training data set information related to model training in the second output area 340. For example, when the processor 110 receives a selection input for the training data set selection object 333, the training data set information used for model training, the training data set used when retraining the trained model Information, new learning data set information newly added by the user, and the like may be displayed on the second output area 340. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the learning console selection object 335 may include an object for displaying information on a learning progress state of a current model in a second output area. For example, when the processor 110 receives a selection input for the training console selection object 335, the current progress of model training (eg, training target model information, training progress, time remaining until completion of training, A current CPU calculation amount required for learning and an amount of memory used for learning) may be displayed in the second output area 340. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the model archive selection object 337 may include an object for displaying information on at least one model in the second output area. For example, when the processor 110 receives a selection input for the model archive selection object 337, at least one model list, detailed information for each model (eg, training completion time, Learning data set information), etc. may be displayed on the second output area 340. The model archive may include a repository in which a plurality of models are stored. The model archive may contain, for example, trained models, untrained models, retrained models, and the like. The processor 110 may call a model selected by the user by receiving a user's input signal from the model archive. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the detected anomaly output selection object 339 may include an object for displaying the anomaly information detected using a model in the second output area 340. For example, when the processor 110 receives an input signal for the detected anomaly output selection object 339, the processor 110 transmits information on the anomaly data acquired using the model to the second output area It can be displayed at 340. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 110 may display a training data set output screen in the second output area. The processor 110 may receive an input signal for the training data set selection object 333 and display the training data set output screen 371 on the second output area 340. The training data set output screen will be described in detail with reference to FIG. 5.

According to an embodiment of the present disclosure, the processor 110 may display a training data set output screen in the second output area. The training data set screen may include an object capable of displaying training data set related information and editing a training data set list. The training data set may include at least one or more training data. The training data may include data in any format used for learning artificial intelligence. For example, the training data may include image data, audio data, text data, natural language data, time series data, and the like. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the training data set output screen 371 includes a training data set list listing at least one training data set, and training for a user to receive a selection input for a training data set to be used for model training. It may include at least one of a data set addition object 373 or a training data set removal object 375 for receiving a selection input for a training data set that will not be used for model training by the user. The training data set list may include a list in which at least one training data set is displayed in an ordered state. The training data set alignment may be determined based on a user's input signal. For example, the user's input signal may be a signal that gives a high priority to a data set used for model training, and puts data obtained by relabeling misclassified data as a low priority. In this case, the processor 110 may display the training data set used for training the model in the upper part of the training data set list. In addition, the training data set list may include, for example, a training data set selected by receiving a user's selection input, a training data set recommended by the processor 110, a newly added training data set, and the like. The training data set list includes, for example, training data set 1 (377) used for training the model, training data set 3 (378) including data chunk 1 relabeled misclassified data, and for assessing the performance of the model. A sample data set 379 may be included. The sample data set 379 may include a data set input to the model to measure the performance of the model when there is no training data set. The training data set addition object 373 may include an object for a user to receive a selection input for a training data set to be used for model training. For example, when receiving a selection input for the learning data set addition object 373, the processor 110 may newly display a screen for adding the learning data set. Also, the processor 110 may display a pop-up window for adding a learning data set. The training data set removal object 375 may include an object for a user to receive a selection input for a training data set that will not be used for model training. For example, when a selection input for a training data set removal object 375 is received after receiving a selection input for a specific training data set from a user, the processor 110 displays the training data set on the training data set output screen ( 371). Therefore, the removed training data set may not be displayed on the user's screen. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the training data set may include at least one or more training data. The training data may include data in any format used for learning artificial intelligence. According to an embodiment of the present disclosure, the training data set may include at least one of a first training data set used for training a model or a new second training data set used for retraining a model. The first training data set may include a training data set used for training a model. According to an embodiment of the present disclosure, the second training data set may include a new training data set to be used for retraining the model. The second training data set may include at least some of the time series data acquired from the sensor in real time and a label corresponding thereto. The label may include a value obtained by using a model that automatically performs labeling (eg, a learned model that performs auto-labeling) or a value that is determined based on a user's selection input. The label may also include a relabeled value for the input data because a value obtained for the input data using the learned model corresponds to misclassification data. Misclassified data may refer to data in which results obtained using a model are misclassified. Misclassified data may be, for example, data that is classified as abnormal although the data subset is a normal data subset. When the processor 110 obtains a result that the data subset is anomalous using the model, the data subset may be a misclassified data subset. These misclassified data subsets may be labeled by an operator or other model, and may be included in the training data set by assigning the corrected labels. The anomaly may be a part having an irregular pattern that is not normal. The anomaly may include, for example, a defective part of the product. The anomaly may also include a portion of the machine's motion (eg, motion of a robotic arm) in which there is a malfunction. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 110 may display a training data set selection screen allowing a user to select a second training data set. When a selection input for a learning data addition object is received, the processor 110 may display a learning data selection screen. Hereinafter, the learning data selection screen will be described in detail with reference to FIG. 6.

According to an embodiment of the present disclosure, the training data set selection screen 400 may include a screen allowing a user to select a second training data set. The processor 110 may display a screen so that a user can select a second training data set necessary to improve model performance. Accordingly, the user can select a training data set necessary for retraining the model through the displayed training data set selection screen. The processor 110 may receive a selection input for a training data set selected by a user. The processor 110 may train the model by inputting the selected training data set into the model. In addition, the training data set selection screen 400 uses a newly added training data set (for example, a training data set purchased from another company, a training data set related to a new machine device, data before and after a recipe change, etc.) as a user. It may include a screen to allow selection. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the training data set selection screen is a screen for allowing a user to select a second training data set, and data obtained by inputting time series data into a model are based on a predetermined first criterion It may include at least one of a time variable setting area for filtering with or a data chunk area 450 for displaying data chunks obtained by inputting time series data into the model and dividing the data obtained based on a predetermined second criterion. . The second training data set may include a new training data set to be used for retraining the model. According to an embodiment of the present disclosure, the time variable setting region may include a region for filtering data obtained by inputting time series data into a model based on a first predetermined criterion. The first predetermined criterion may include a criterion for time. The first predetermined criterion may include, for example, a period in units of year, month, day, hour, minute, and second. The processor 110 may display only data corresponding to the input period on the learning data set selection screen 400 based on the first predetermined criterion input by the user. For example, if the input predetermined first criterion is 00:00-24:00 on October 30, 2019, the processor 110 is the data corresponding to 00:00-24:00 on October 30, 2019. Only can be displayed on the learning data set selection screen 400. The time variable setting area may include a period setting area 410 for filtering data on a specific date or a time setting area 430 for filtering data on a specific time on a specific date. For example, the date of October 30, 2019 may be received through the period setting area 410, and a time of 6:00-12:00 may be received through the time setting area 430. In this case, the processor 110 may display data corresponding to 6:00-12:00 on October 30, 2019 on the training data set selection screen 400. Specifically, the data displayed on the training data set selection screen 400 may include data obtained by inputting time series data corresponding to 6:00-12:00 on October 30, 2019 into the model. Through this, the processor 110 may selectively provide data for a time period requiring re-labeling to the user in order to improve the performance of the model. Accordingly, the user can set the time zone in which the misclassified data exists and view only the data in the corresponding time zone through a screen that is selectively displayed. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the data chunk area 450 may include an area for displaying data chunks obtained by inputting time series data into a model and dividing data obtained based on a second predetermined criterion. The data chunk may include a data subset that is at least a part of the data set. The data chunk may include some of data obtained by inputting time series data into a model and divided based on a second predetermined criterion. Data chunks may be classified as representing a first state and/or a second state, respectively. The first state and the second state may mean a binary classification result when the processor 110 performs binary classification using a model. For example, the first state may be normal, and the second state may be anomaly. A method of dividing the data obtained by inputting into the model based on the second predetermined criterion by the processor 110 will be described later. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the data chunk may include statistical characteristics of each data set obtained by inputting a plurality of time series data divided based on a second predetermined criterion into a model. The statistical feature is a feature that represents a feature of a data set through a statistical technique, and may include a probability distribution, a mean, a standard deviation, and a variance of the data set. Accordingly, the data chunk may include probability distribution, average, standard deviation, variance, and the like of each data set obtained by inputting time series data into the model. For example, if the data chunks are divided at 5 minute intervals, the processor 110 inputs the time series data subset divided at 5 minute intervals into the model and calculates the average value of the data set corresponding to the corresponding data chunk. Can be determined by features. In addition, when the data chunk is divided according to the second predetermined criterion, the processor 110 inputs a subset of the time series data divided by the second predetermined criterion into the model, and calculates the median value of the data set corresponding to the data chunk. It can also be determined by features. The processor 110 may display statistical characteristics of the data chunk. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the second predetermined criterion is a criterion for detecting misclassified data among data acquired using a model, and the data acquired using the model changes from a first state to a second state. The first point that changes, the second point at which the output of the model changes from the second state to the first state, any third point present in the output of the model in the first state, or any third point present in the output of the model in the second state. It may include a criterion for dividing data obtained by inputting time series data into a model based on at least one of the four points into a plurality of data chunks. The first state and the second state may mean a binary classification result when the processor 110 performs binary classification using a model. For example, the first state may be normal, and the second state may be anomaly. The processor 110 may determine a criterion for dividing the data chunk as a first point and/or a second point. Accordingly, the processor 110 may determine a first point that changes from the first state to the second state as the start point of the data chunk. In this case, the processor 110 may determine a second point that changes from the second state to the first state as the end point of the corresponding data chunk. Conversely, the processor 110 may determine a second point that changes from the second state to the first state as the start point of the data chunk. In this case, the processor 110 may determine the first point that changes from the first state to the second state as the end point of the corresponding data chunk. Through this, the processor 110 may classify a data chunk in which the model output is normal and a data chunk in which the model output is anomaly.

According to another embodiment of the present disclosure, the second predetermined criterion is the data obtained by inputting time series data into the model based on a third point present in the output of the model in the first state, into a plurality of data chunks. It may contain criteria for dividing. In an embodiment of the present disclosure, the first state may be normal, and the second state may be anomaly. The third point may include a point for dividing the output of the normal model into at least one data chunk. The processor 110 may divide the output of the normal model into a plurality of data chunks based on an arbitrary third point present in the output of the normal model. For example, the processor 110 may determine a third point as a starting point and another third point as an ending point. Through this, the processor 110 may obtain the subdivided data chunk even for the output of the normal model. As another example, the processor 110 may determine a first point as a starting point and a third point as an ending point. The processor 110 may determine a second point as a starting point and determine another third point as an ending point. The processor 110 may determine a third point as a starting point and determine a first point and/or a second point as an end point. The foregoing is only an example, and the present disclosure is not limited thereto.

According to another embodiment of the present disclosure, the predetermined second criterion is the data obtained by inputting time series data into the model based on a fourth point present in the output of the model in the second state, and converting the acquired data into a plurality of data chunks. It may contain criteria for dividing. In an embodiment of the present disclosure, the first state may be normal, and the second state may be anomaly. The fourth point may include a point for dividing the output of the anomaly model into at least one data chunk. The processor 110 may divide the output of the anomaly model into a plurality of data chunks based on an arbitrary fourth point present in the output of the anomaly model. For example, the processor 110 may determine a fourth point as a starting point and another fourth point as an ending point. Through this, the processor 110 may obtain the subdivided data chunk for the output of the anomaly model. As another example, the processor 110 may determine a first point as a starting point and a fourth point as an ending point. The processor 110 may determine the second point as the start point and the fourth point as the end point. The processor 110 may determine the third point as the starting point and the fourth point as the ending point. The processor 110 may determine a fourth point as a starting point and determine a first point, a second point, and/or a third point as an end point. The foregoing is only an example, and the present disclosure is not limited thereto.

Accordingly, the processor 110 may display data chunks for each classification result in the data chunk area 450. The user can see the displayed data chunk and determine which data chunk is misclassified data. For example, as illustrated in FIG. 6, a data chunk 1 451, a data chunk 2 453, and a data chunk 3 455 may be displayed in the data chunk area 450. Here, the data chunk 1 451 may be misclassified data (eg, false positive-normal data, but when the output obtained by using the model is anomaly). That is, if the prediction of the model is incorrect, the processor 110 may receive an input signal from the user and relabel the misclassified data chunk. That is, if the data is normal, but the output obtained by using the model is anomaly, the processor 110 may relabel the label of the corresponding data as normal. The processor 110 may retrain the model by inputting the relabeled data chunk into the model. Through this, the processor 110 may update the model with respect to a portion that the model incorrectly predicts. Ultimately, the prediction accuracy of the model can be improved through this update process. In the case of data chunk 3 455, a result value obtained by using the model may be Normal. And the input data may also be Normal. In this case, data chunk 3 455 may be an accurately predicted and/or classified data set. Therefore, the need for data chunk 3 to be retrained by re-input to the model may decrease. In this case, the processor 110 may not perform relabeling for the data chunk 3. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the data chunk includes a data chunk including a data chunk calculated through a data chunk recommendation algorithm for a data chunk to be used for model retraining to a user, or data selected by receiving a user selection input signal. It may include at least one of data chunks including at least one data chunk having a similar statistical characteristic to the chunk. The data chunk recommendation algorithm may include an algorithm for recommending a data chunk to be used for model retraining to a user. Specifically, the data chunk recommendation algorithm may include an algorithm that recommends a data chunk necessary to improve the performance of the model. The data chunk recommendation algorithm may include, for example, an algorithm for recommending misclassified data chunks, an algorithm for outputting time information at which misclassified data occurs most often, and the like. The processor 110 may automatically detect and display misclassified data chunks. In addition, the processor 110 may display a data chunk in a time period in which the misclassified data most often occurs. Accordingly, the processor 110 may help a user to quickly find data necessary for retraining a model by displaying the data chunks calculated through the data chunk recommendation algorithm. Through this, the user selectively sees the data or data set required for retraining the model, so that a decision required for retraining the model can be quickly made. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the data chunk may include at least one data chunk having a statistical characteristic similar to the data chunk selected by receiving a user selection input signal. The processor 110 may receive a user's selection input and display a data chunk having a statistical characteristic similar to the selected data chunk (eg, having a similar average value, similar probability distribution, and similar variance). Through this, when receiving a selection input for the misclassified data chunk selected by the user, the processor 110 displays the data chunk having similar statistical characteristics to the corresponding data chunk, allowing the user to quickly view the data required for model retraining. You can do it. Accordingly, the processor 110 receives the data chunk calculated through the recommendation algorithm and/or the user's selection input, and displays the data chunk having statistical characteristics similar to the selected data chunk in the data chunk area 450, so that the user can recreate the model. It's easy to see at a glance what data sets you might need for training. That is, a data chunk having similar statistical characteristics to the misclassified data may be highly likely to be misclassified data. Accordingly, the processor 110 may selectively display only data chunks that are likely to be misclassified data by displaying data chunks having similar statistical characteristics to the data chunk. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 110 may display a model archive output screen in the second output area. The processor 110 may receive an input signal for the model archive selection object 337 and display the model archive output screen 510 on the second output area 340. The model archive output screen will be described in detail with reference to FIG. 7.

According to an embodiment of the present disclosure, the model archive output screen 510 may include a screen displaying information on each of a plurality of models. The model archive output screen 510 is a model list output area 513 for displaying a plurality of models stored in the model archive for the user to see, or model information for displaying information on a selected model by receiving a user's selection input signal It may include at least one of the output areas 515. The model archive may include a repository in which a plurality of models are stored. The model archive may contain, for example, trained models, untrained models, retrained models, and the like. The processor 110 may call a model selected by the user by receiving a user's input signal from the model archive. According to an embodiment of the present disclosure, the model list output area 513 may include an area for displaying a plurality of models stored in the model archive so that they can be viewed at a glance. The model list output area 513 may include an area for displaying a model selected based on a user's selection input so that the selected model can be viewed at a glance. For example, the model list output area 513 may include a model included in a project, a model selected by a user, a model recommended to a user, and the like. According to an embodiment of the present disclosure, the model information output area 515 may include an area for receiving a user's selection input signal and displaying information on a selected model. The model information output area 515 is an area in which, for example, a model name, a model status (eg, before training, after training, during training), a model creation date, information on a training data set that trained the model, and the like are displayed. Can include. The foregoing is only an example, and the present disclosure is not limited thereto.

The models included in the model list will be described in detail with reference to FIGS. 8 and 9.

According to an embodiment of the present disclosure, the model list includes a model trained while performing a project, a retrained model 517 by inputting a second training data set into the trained model, and a plurality of models included in the model archive. In order to recommend to the user a newly created model 525 by integrating a model having statistical characteristics or a model corresponding to newly input data, a model determined based on the hit rate of each of the plurality of models included in the model archive ( 521). The foregoing is only an example, and the present disclosure is not limited thereto. The anomaly depression in a multiple steady state environment using a plurality of models is specifically discussed in Korean Patent Application 10-2018-0080482 (2018.07.11), the entirety of which is incorporated herein by reference.

According to an embodiment of the present disclosure, a model archive 520 and a hit model 521 are shown in FIG. 8. The hit model 521 may include a model determined based on a hit rate of each of a plurality of models included in the model archive in order to recommend a model corresponding to the newly input data to the user. The hit rate may include a probability that the input data is determined to be normal or abnormal (eg, final determination) by the corresponding model. For newly input time series data, the user may not know which model outputs the best results. In addition, when comparing performance by inputting newly input time series data to all models included in the model archive, it may take a lot of time and cost. Accordingly, for newly input time series data, the processor 110 may select a model with a high probability of hitting among the models stored in the model archive 520 and display the selected model. Through this, the user can quickly acquire a model suitable for newly inputted data without going through an experiment process. The foregoing is only an example, and the present disclosure is not limited thereto.

In accordance with an embodiment of the present disclosure, a model archive 520, a model 523 having a similar distribution, and an integrated model 525 are shown in FIG. 8. The processor 110 may newly generate the integrated model 525 by integrating the models 523 having similar distributions. That is, the processor 110 may generate a model that is likely to be used for newly input data by integrating at least one or more models that are unlikely to be used for newly input data. Through this, the processor 110 may selectively calculate a hit rate for each of the models included in the group that is likely to be used, so that a model suitable for newly input data may be quickly calculated. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 110 may display the detected anomaly output screen in the second output area. The processor 110 may receive an input signal for the detected anomaly output selection object 339 and display the detected anomaly output screen 530 on the second output area 340. The detected anomaly output screen is described in detail with reference to FIG. 10.

According to an embodiment of the present disclosure, the processor 110 may display the detected anomaly output screen 530 in the second output area. The detected anomaly output screen 530 may include an anomaly detection result output area for displaying the anomaly detection result obtained using the model.

According to an embodiment of the present disclosure, the detected anomaly output screen 530 may include a screen for displaying information related to anomaly data among data acquired using a model. The detected anomaly output screen 530 displays information on the anomaly data selected by receiving an anomaly detection result output area 531 or a user selection input signal for displaying a list of anomaly data acquired using the model. It may include at least one of the anomaly information output area 533 for performing the operation. The anomaly detection result output area 531 may include an area for displaying an anomaly data list acquired using a model. For example, the processor 110 may display data classified as anomaly among data acquired using a model on the output area 531 as a result of the anomaly detection. The anomaly information output area 533 may include an area for receiving a user's selection input signal and displaying information on the selected anomaly data. For example, the processor 110 may receive a user's selection input signal and display information on the selected anomaly data, such as a start time of anomaly occurrence, an end time of anomaly end, a period at which the anomaly occurs, and the like. . The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 110 may display a model performance output screen in the second output area. The processor 110 may receive an input signal for the performance monitoring selection object 331 and display the model performance output screen 550 on the second output area 340. The model performance output screen is described in detail with reference to FIG. 11.

According to an embodiment of the present disclosure, as illustrated in FIG. 11, the processor 110 may display the model performance output screen 550 in the second output area. The model performance output screen 550 may include a screen for displaying model performance information. The model performance information may include all information related to the model performance. For example, the performance information of the model may include a measure of how accurate the model outputs prediction results for input data. In addition, the performance information of the model may include a prediction value obtained by the processor 110 using the model. The foregoing is only an example, and the present disclosure is not limited thereto.

According to the present disclosure, by providing a user interface for training a model, even a person who does not have expert knowledge in the field of artificial intelligence can help retrain the model for a portion where an abnormality has occurred. Through this, even a person without expertise in the field of artificial intelligence can maintain and repair the model or retrain the model by inputting new data into the model. Therefore, experts in various domains (eg, manufacturing, medical, legal, management, manufacturing) who use the model to improve the model can directly discover the problem and update the model by adding their expertise. Therefore, through the present disclosure, as the process of using the deep learning model is simplified and only necessary information is selectively output, the speed at which the deep learning technology is extended and applied to various technology domains can be increased.

FIG. 2 is a diagram illustrating a neural network that is an object of learning in a method for training a neural network according to an embodiment of the present disclosure.

Throughout this specification, a computational model, a neural network, a network function, and a neural network may be used with the same meaning. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. The neural network includes at least one or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more links.

In a neural network, one or more nodes connected through a link may relatively form a relationship between an input node and an output node. The concepts of an input node and an output node are relative, and any node in an output node relationship with respect to one node may have an input node relationship in a relationship with another node, and vice versa. As described above, the relationship between the input node and the output node can be created around a link. One or more output nodes may be connected to one input node through a link, and vice versa.

In the relationship between the input node and the output node connected through one link, the value of the output node may be determined based on data input to the input node. Here, a node interconnecting the input node and the output node may have parameters. The parameters may be variable and may be changed by a user or an algorithm in order for the neural network to perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node includes values input to input nodes connected to the output node and a link corresponding to each input node. The output node value may be determined based on the parameter.

As described above, in a neural network, one or more nodes are interconnected through one or more links to form an input node and an output node relationship in the neural network. The characteristics of the neural network may be determined according to the number of nodes and links in the neural network, the association relationship between the nodes and the links, and values of parameters assigned to the links. For example, when there are the same number of nodes and links, and two neural networks having different parameter values between the links exist, the two neural networks may be recognized as being different from each other.

A neural network may be configured including one or more nodes. Some of the nodes constituting the neural network may configure one layer based on distances from the initial input node. For example, a set of nodes having a distance n from an initial input node may constitute n layers. The distance from the first input node may be defined by the minimum number of links that must be passed from the first input node to the corresponding node. However, the definition of such a layer is arbitrary for explanation, and the order of the layer in the neural network may be defined in a different manner than that described above. For example, the layer of nodes may be defined by the distance from the last output node.

The first input node may mean one or more nodes to which data is directly input without going through a link in a relationship with other nodes among nodes in the neural network. Alternatively, in a neural network network, in a relationship between nodes based on a link, it may mean nodes that do not have other input nodes connected by a link. Similarly, the final output node may mean one or more nodes that do not have an output node in a relationship with other nodes among nodes in the neural network. In addition, the hidden node may refer to nodes constituting a neural network other than the first input node and the last output node. In the neural network according to an embodiment of the present disclosure, the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as the input layer proceeds to the hidden layer. I can. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes of an input layer is less than the number of nodes of an output layer, and the number of nodes decreases as the number of nodes progresses from the input layer to the hidden layer. have. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes of an input layer is greater than the number of nodes of an output layer, and the number of nodes increases as the number of nodes progresses from the input layer to the hidden layer. I can. A neural network according to another embodiment of the present disclosure may be a neural network in the form of a combination of the aforementioned neural networks.

A deep neural network (DNN) may mean a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can be used to identify potential structures in data. In other words, it is possible to grasp the potential structure of photos, texts, videos, voices, and music (e.g., what objects are in photos, what are the content and emotions of the text, what are the contents and emotions of the voice, etc.) . Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), auto encoders, Generative Adversarial Networks (GANs), and restricted Boltzmann machines (RBMs). boltzmann machine), deep belief network (DBN), Q network, U network, and Siam network. The description of the above-described deep neural network is only an example, and the present disclosure is not limited thereto.

In an embodiment of the present disclosure, the network function may include an auto encoder. The auto encoder may be a kind of artificial neural network for outputting output data similar to input data. The auto encoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between the input/output layers. The number of nodes of each layer may be reduced from the number of nodes of the input layer to an intermediate layer called a bottleneck layer (encoding), and then reduced and expanded symmetrically from the bottleneck layer to an output layer (symmetric with the input layer). The nodes of the dimensionality reduction layer and the dimensionality restoration layer may or may not be symmetrical. Auto encoders can perform nonlinear dimensionality reduction. The number of input layers and output layers may correspond to the number of sensors remaining after pre-processing of input data. In the auto-encoder structure, the number of nodes of the hidden layer included in the encoder may have a structure that decreases as it moves away from the input layer. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and the decoder) is too small, a sufficient amount of information may not be delivered, so more than a certain number (e.g., more than half of the input layer, etc.) ) May be maintained.

The neural network may be learned in at least one of supervised learning, unsupervised learning, and semi supervised learning. Learning of neural networks is to minimize output errors. In learning of a neural network, iteratively inputs training data to the neural network, calculates the output of the neural network for the training data and the error of the target, and reduces the error from the output layer of the neural network to the input layer. It is a process of updating the parameters of each node of the neural network by backpropagation in the direction. In the case of teacher learning, learning data in which the correct answer is labeled in each learning data is used (ie, labeled learning data), and in the case of non-satellite learning, the correct answer may not be labeled in each learning data. That is, for example, in the case of teacher learning related to data classification, the learning data may be data in which a category is labeled with each learning data. Labeled training data is input to a neural network, and an error may be calculated by comparing an output (category) of the neural network with a label of the training data. As another example, in the case of comparative history learning regarding data classification, an error may be calculated by comparing training data as input with a neural network output. The calculated error is backpropagated in the neural network in the reverse direction (ie, from the output layer to the input layer), and parameters of each node of each layer of the neural network may be updated according to the backpropagation. A change amount may be determined according to a learning rate in the parameter of each node to be updated. The computation of the neural network for the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of repetitions of the learning cycle of the neural network. For example, a high learning rate is used in the early stages of training of a neural network, so that the neural network quickly secures a certain level of performance to increase efficiency, and a low learning rate can be used in the later stages of training to increase accuracy.

In learning of a neural network, in general, training data may be a subset of actual data (that is, data to be processed using a trained neural network), and thus, errors in training data decrease but errors in actual data There may be increasing learning cycles. Overfitting is a phenomenon in which errors in actual data increase due to over-learning on learning data. For example, a neural network learning a cat by showing a yellow cat may not recognize that it is a cat when it sees a cat other than yellow, which may be a kind of overfitting. Overfitting can cause an increase in errors in machine learning algorithms. Various optimization methods can be used to prevent such overfitting. In order to prevent overfitting, methods such as increasing training data, regularization, and dropout in which some nodes of the network are omitted during the training process may be applied.

A computer-readable medium storing a data structure according to an embodiment of the present disclosure is disclosed.

Data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. The data structure may refer to an organization of data for solving a specific problem (eg, data search, data storage, data modification in the shortest time). Data structures may be defined as physical or logical relationships between data elements, designed to support specific data processing functions. The logical relationship between data elements may include a connection relationship between data elements that a user thinks. The physical relationship between the data elements may include an actual relationship between the data elements physically stored in a computer-readable storage medium (eg, hard disk). The data structure may specifically include a set of data, a relationship between data, and a function or instruction applicable to the data. Through an effectively designed data structure, the computing device can perform calculations while using the resources of the computing device to a minimum. Specifically, computing devices can increase the efficiency of computation, reading, insertion, deletion, comparison, exchange, and search through an effectively designed data structure.

The data structure can be classified into a linear data structure and a non-linear data structure according to the shape of the data structure. The linear data structure may be a structure in which only one data is connected after one data. The linear data structure may include a list, a stack, a queue, and a deck. The list may mean a series of data sets in which order exists internally. The list may include a linked list. The linked list may be a data structure in which data is linked in a way that each data has a pointer and is linked in a single line. In the linked list, the pointer may include connection information with the next or previous data. The linked list can be expressed as a single linked list, a double linked list, or a circular linked list. The stack can be a data listing structure that allows limited access to data. The stack may be a linear data structure that can process (eg, insert or delete) data only at one end of the data structure. Data stored in the stack may be a data structure (LIFO-Last in First Out) that comes out sooner as it enters the stack. A queue is a data arrangement structure that allows limited access to data, and unlike a stack, a queue may be a data structure (FIFO-First in First Out) that comes out later as data stored later. A deck can be a data structure that can process data at both ends of the data structure.

The nonlinear data structure may be a structure in which a plurality of data is connected behind one data. The nonlinear data structure may include a graph data structure. The graph data structure may be defined as a vertex and an edge, and the edge may include a line connecting two different vertices. The graph data structure may include a tree data structure. The tree data structure may be a data structure in which a path connecting two different vertices among a plurality of vertices included in the tree is one. That is, it may be a data structure that does not form a loop in the graph data structure.

Throughout this specification, a computational model, a neural network, a network function, and a neural network may be used with the same meaning. (Hereinafter, it will be unified and described as a neural network.) The data structure may include a neural network. In addition, the data structure including the neural network may be stored in a computer-readable medium. The data structure including the neural network may also include data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, an active function associated with each node or layer of the neural network, and a loss function for learning the neural network. have. The data structure including the neural network may include any of the components disclosed above. In other words, the data structure including the neural network is the data input to the neural network, the weight of the neural network, the hyper parameter of the neural network, the data obtained from the neural network, the active function associated with each node or layer of the neural network, the loss function for training the neural network, etc. It may be configured to include any combination of. In addition to the above-described configurations, the data structure including the neural network may include any other information that determines the characteristics of the neural network. In addition, the data structure may include all types of data used or generated in the process of calculating a neural network, and is not limited to the above-described matters. Computer-readable media may include computer-readable recording media and/or computer-readable transmission media. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. The neural network includes at least one or more nodes.

The data structure may include data input to the neural network. A data structure including data input to the neural network may be stored in a computer-readable medium. The data input to the neural network may include training data input during the neural network training process and/or input data input to the neural network on which the training has been completed. Data input to the neural network may include data that has undergone pre-processing and/or data to be pre-processed. Pre-processing may include a data processing process for inputting data into a neural network. Accordingly, the data structure may include data to be pre-processed and data generated by pre-processing. The above-described data structure is only an example, and the present disclosure is not limited thereto.

The data structure may include data input to or output from a neural network. A data structure including data input to or output from the neural network may be stored in a computer-readable medium. The data structure stored in the computer-readable medium may include data input during an inference process of a neural network, or output data output as a result of inference of a neural network. In addition, since the data structure may include data processed by a specific data processing method, it may include data before and after processing. Accordingly, the data structure may include data to be processed and data processed through a data processing method.

The data structure may include weights of the neural network. (In the present specification, weights and parameters may have the same meaning.) And the data structure including the weights of the neural network may be stored in a computer-readable medium. The neural network may include a plurality of weights. The weight may be variable and may be changed by a user or an algorithm in order for the neural network to perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node includes values input to input nodes connected to the output node and a link corresponding to each input node. The output node value may be determined based on the parameter. The above-described data structure is only an example, and the present disclosure is not limited thereto.

As an example and not a limitation, the weight may include a weight variable in a neural network training process and/or a weight on which neural network training has been completed. The variable weight in the neural network learning process may include a weight at a time point at which the learning cycle starts and/or a weight variable during the learning cycle. The weight at which the neural network training is completed may include the weight at which the learning cycle is completed. Accordingly, the data structure including the weight of the neural network may include a data structure including a weight variable during the neural network training process and/or a weight on which the neural network training has been completed. Therefore, it is assumed that the above-described weight and/or a combination of each weight are included in the data structure including the weight of the neural network. The above-described data structure is only an example, and the present disclosure is not limited thereto.

The data structure including the weights of the neural network may be serialized and then stored in a computer-readable storage medium (eg, memory, hard disk). Serialization may be a process of storing data structures on the same or different computing devices and converting them into a form that can be reconstructed and used later. The computing device serializes the data structure to transmit and receive data through a network. The data structure including the weights of the serialized neural network can be reconstructed in the same or different computing devices through deserialization. The data structure including the weights of the neural network is not limited to serialization. Furthermore, the data structure including the weight of the neural network is a data structure to increase the efficiency of operation while using the resources of the computing device to a minimum (e.g., B-Tree, Trie, m-way search tree, AVL tree, in nonlinear data structure, Red-Black Tree). The foregoing is only an example, and the present disclosure is not limited thereto.

The data structure may include a hyper-parameter of a neural network. In addition, the data structure including the hyper parameter of the neural network may be stored in a computer-readable medium. The hyper parameter may be a variable that is variable by a user. Hyper parameters include, for example, learning rate, cost function, number of iterations of learning cycles, weight initialization (e.g., setting the range of weight values to be weighted to be initialized), Hidden Unit It may include the number (eg, the number of hidden layers, the number of nodes of the hidden layer). The above-described data structure is only an example, and the present disclosure is not limited thereto.

3 is a diagram illustrating a first screen according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the computing device 100 may display a first screen 200 including at least one first object 230 for receiving a selection input for a project. Project 210 may include artificial intelligence-related projects to achieve specific goals using artificial intelligence. Specifically, the project 210 may include a deep learning-related project to achieve a specific goal by using deep learning. The first object 230 may include an object for selecting the project 210. The first object 230 may include a function of moving to a second screen for the selected project 210. The first object 230 may include, for example, an icon on the screen. The first screen 200 may be a screen on which at least one or more projects conducted by a user are displayed. The first screen 200 may include a screen that displays a plurality of projects conducted by a user in an easy to see at a glance. The selection input may include an input signal including information on the item selected by the user. Accordingly, when receiving a selection input, the computing device 100 may perform an operation based on the selection or display a result of the selection input on the screen. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the project 210 may include an artificial intelligence-related project for achieving a specific goal using artificial intelligence. Specific goals may include goals of improving the performance of models to which artificial intelligence is applied. The specific goal may include, for example, improving the prediction accuracy of the model, shortening the training time of the model, and reducing the amount of computation or power used for model training. A specific goal may also include a goal to create a model for obtaining a forecast (eg, stock price prediction in the case of finance) in a specific domain (eg manufacturing, healthcare, legal, business, finance). The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the computing device 100 may display a second screen for displaying project-related information corresponding to the selected project. The second screen may include a screen in which a selection input of the first screen is received and the computing device 100 displays project related information corresponding to the selection input. The foregoing is only an example, and the present disclosure is not limited thereto.

4 is a diagram illustrating a second screen by way of example, according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the computing device 100 may display the second screen 300 for displaying project-related information corresponding to the selected project. The second screen 300 may include a screen that displays information on a specific project. The second screen 300 may include a screen for displaying information necessary for the user to improve the performance of the deep learning model. The second screen 300 may include, for example, a user interface on which information for improving the performance of the deep learning model is displayed. Accordingly, the second screen 300 may include performance information of the model, information on a training data set used for model training, and a prediction value output by the computing device 100 using the model. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the second screen includes a first output area 310 for displaying time series data acquired from a sensor, and at least one second object for receiving a selection input related to model retraining. It may include at least one of the included selection area 330 or the second output area 340 for displaying information corresponding to the second object. The first output area 310 may include an area for displaying time series data acquired from a sensor. For example, the first output area 310 may include an area for displaying time series data acquired from a sensor in real time. The time series data obtained from the sensor is, for example, time series data obtained from a sensor attached to the robot joint, time series data obtained from a material measurement sensor, temperature data, wind direction, wind speed data, ultraviolet sensor data, infrared sensor data, and light sensor Data, sound sensor data, and the like. The second output area 340 may include an area for displaying information corresponding to the second object. The computing device 100 may receive a selection input for the second object and display information corresponding to the second object. For example, when receiving a selection input for a second object (eg, a training data selection icon), the computing device 100 may display training data related information on the second output area 340. The selection area may include an area including at least one second object for receiving a selection input related to model retraining. The selection area 330 may be an area including an object for displaying information related to model training in the second output area. For example, the selection area 330 may include a plurality of icons on the user interface in order to allow the user to view information related to model retraining desired by the user through the second output area. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the selection area is an area including an object for displaying information related to model training in the second output area, and performance monitoring selection for displaying performance information of the model in the second output area Object 331, a training data set selection object 333 for displaying training data set information related to model training in a second output area, a model archive for displaying information on at least one model in a second output area It may include at least one of the selection object 337 or the detected anomaly output selection object 339 for displaying the anomaly information sensed using the model in the second output area. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the performance monitoring selection object 331 may include an object for displaying model performance information in the second output area. For example, when receiving a selection input for the performance monitoring selection object 331, the computing device 100 may display model performance information (eg, prediction accuracy of a model) in the second output area. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the training data set selection object 333 may include an object for displaying training data set information related to model training in the second output area 340. For example, when receiving a selection input for the training data set selection object 333, the computing device 100 includes training data set information used for model training, and training data used when retraining the trained model. Set information, new training data set information newly added by the user, and the like may be displayed on the second output area 340. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the learning console selection object 335 may include an object for displaying information on a learning progress state of a current model in a second output area. For example, when the computing device 100 receives a selection input for the training console selection object 335, the current progress of model training (eg, training target model information, training progress, time remaining until completion of training) , A current CPU calculation amount required for learning) may be displayed in the second output area 340. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the model archive selection object 337 may include an object for displaying information on at least one model in the second output area. For example, when the computing device 100 receives a selection input for the model archive selection object 337, at least one model list, detailed information about each model (e.g., training completion time, used for training) The obtained training data set information) and the like may be displayed in the second output area 340. The model archive may include a repository in which a plurality of models are stored. The model archive may contain, for example, trained models, untrained models, retrained models, and the like. The computing device 100 may call a model selected by the user by receiving a user's input signal from the model archive. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the detected anomaly output selection object 339 may include an object for displaying the anomaly information detected using a model in the second output area 340. For example, when the computing device 100 receives an input signal for the detected anomaly output selection object 339, the computing device 100 may provide the second information on the anomaly data acquired by using the model. It can be displayed in the output area 340. The foregoing is only an example, and the present disclosure is not limited thereto.

5 is a diagram illustrating an output screen of a training data set according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the computing device 100 may display a training data set output screen in the second output area. The computing device 100 may receive an input signal for the training data set selection object 333 and display the training data set output screen 371 on the second output area 340.

According to an embodiment of the present disclosure, the computing device 100 may display a training data set output screen in the second output area. The training data set screen may include an object capable of displaying training data set related information and editing a training data set list. The training data set may include at least one or more training data. The training data may include data in any format used for learning artificial intelligence. For example, the training data may include image data, audio data, text data, natural language data, time series data, and the like. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the training data set output screen 371 includes a training data set list listing at least one training data set, and training for a user to receive a selection input for a training data set to be used for model training. It may include at least one of a data set addition object 373 or a training data set removal object 375 for receiving a selection input for a training data set that will not be used for model training by the user. The training data set list may include a list in which at least one training data set is displayed in an ordered state. The training data set list may include, for example, a training data set selected by receiving a user's selection input, a training data set recommended by the computing device 100, a newly added training data set, and the like. The training data set list includes, for example, training data set 1 (377) used for training the model, training data set 3 (378) including data chunk 1 relabeled misclassified data, and for assessing the performance of the model. A sample data set 379 may be included. The sample data set 379 may include a data set input to the model to measure the performance of the model when there is no training data set. The training data set addition object 373 may include an object for a user to receive a selection input for a training data set to be used for model training. For example, when receiving a selection input for the learning data set addition object 373, the computing device 100 may newly display a screen for adding the learning data set. In addition, the computing device 100 may display a pop-up window for adding a learning data set. The training data set removal object 375 may include an object for a user to receive a selection input for a training data set that will not be used for model training. For example, when a selection input for a training data set removal object 375 is received after receiving a selection input for a specific training data set from a user, the computing device 100 displays the training data set on the training data set output screen. Can be removed from (371). Therefore, the removed training data set may not be displayed on the user's screen. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the training data set may include at least one or more training data. The training data may include data in any format used for learning artificial intelligence. According to an embodiment of the present disclosure, the training data set may include at least one of a first training data set used for training a model or a new second training data set used for retraining a model. The first training data set may include a training data set used for training a model. According to an embodiment of the present disclosure, the second training data set may include a new training data set to be used for retraining the model. At least some of the time series data acquired from the sensor in real time and a label corresponding thereto may be included. The label may include a value obtained by using a model that automatically performs labeling (eg, a learned model that performs auto-labeling) or a value that is determined based on a user's selection input. The label may include a relabeled value as a value obtained by using the learned model corresponds to misclassification data. Misclassified data may refer to data in which results obtained using a model are misclassified. For the misclassified data, for example, the training data subset may be a normal data set. When the computing device 100 obtains a result that the corresponding training data subset is anomalous using the model, the training data subset may be a misclassified training data subset. The anomaly may be a part having an irregular pattern that is not normal. The anomaly may include, for example, a defective part of the product. The anomaly may also include a portion of the machine's motion (eg, motion of a robotic arm) in which there is a malfunction. The foregoing is only an example, and the present disclosure is not limited thereto.

6 is a diagram illustrating a screen for selecting a training data set according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the computing device 100 may display a learning data set selection screen allowing a user to select a second learning data set. When a selection input for the learning data addition object 373 is received, the computing device 100 may display a learning data selection screen.

According to an embodiment of the present disclosure, the training data set selection screen 400 may include a screen allowing a user to select a second training data set. The computing device 100 may display a screen so that a user can select a second training data set necessary to improve model performance. Accordingly, the user can select a training data set necessary for retraining the model through the displayed training data set selection screen. The computing device 100 may receive a selection input for a training data set selected by a user. The computing device 100 may train the model by inputting the selected training data set into the model. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the training data set selection screen is a screen for allowing a user to select a second training data set, and data obtained by inputting time series data into a model are based on a predetermined first criterion It may include at least one of a time variable setting area for filtering with or a data chunk area 450 for displaying data chunks obtained by inputting time series data into the model and dividing the data obtained based on a predetermined second criterion. . The second training data set may include a new training data set to be used for retraining the model. According to an embodiment of the present disclosure, the time variable setting region may include a region for filtering data obtained by inputting time series data into a model based on a first predetermined criterion. The first predetermined criterion may include a criterion for time. The first predetermined criterion may include, for example, a period in units of year, month, day, hour, minute, and second. The computing device 100 may display only data corresponding to the input period on the learning data set selection screen 400 based on the first predetermined criterion input by the user. For example, if the input predetermined first criterion is 00:00-24:00 on October 30, 2019, the computing device 100 corresponds to 00:00-24:00 on October 30, 2019. Only data may be displayed on the training data set selection screen 400. The time variable setting area may include a period setting area 410 for filtering data on a specific date or a time setting area 430 for filtering data on a specific time on a specific date. For example, the date of October 30, 2019 may be received through the period setting area 410, and a time of 6:00-12:00 may be received through the time setting area 430. In this case, the computing device 100 may display data corresponding to 6:00-12:00 on October 30, 2019 on the training data set selection screen 400. Specifically, the data displayed on the training data set selection screen 400 may include data obtained by inputting time series data corresponding to 6:00-12:00 on October 30, 2019 into the model. Through this, the computing device 100 may selectively provide data for a time period requiring re-labeling to the user in order to improve the performance of the model. Accordingly, the user can set the time zone in which the misclassified data exists and view only the data in the corresponding time zone through a screen that is selectively displayed. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the data chunk area 450 may include an area for displaying data chunks obtained by inputting time series data into a model and dividing data obtained based on a second predetermined criterion. The data chunk may include a data subset that is at least a part of the data set. The data chunk may include some of data obtained by inputting time series data into a model and divided based on a second predetermined criterion. Data chunks may be classified as representing a first state and/or a second state, respectively. The first state and the second state may mean a binary classification result when the computing device 100 performs binary classification using a model. For example, the first state may be normal, and the second state may be anomaly. A method of dividing the data obtained by inputting into the model based on a second predetermined criterion by the computing device 100 will be described later. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the data chunk may include statistical characteristics of each data set obtained by inputting a plurality of time series data divided based on a second predetermined criterion into a model. The statistical feature is a feature that represents a feature of a data set through a statistical technique, and may include a probability distribution, a mean, a standard deviation, and a variance of the data set. Accordingly, the data chunk may include probability distribution, average, standard deviation, variance, and the like of each data set obtained by inputting time series data into the model. For example, when the data chunks are divided at 5 minute intervals, the computing device 100 inputs the time series data subset divided at 5 minute intervals into the model and calculates the average value of the data set corresponding to the corresponding data chunk. It can be determined by statistical characteristics. In addition, when the data chunk is divided by a predetermined second criterion, the computing device 100 inputs the time-series data subset divided by the predetermined second criterion to the model and calculates the average value of the data set corresponding to the data chunk. It can be determined by statistical characteristics. The computing device 100 may display statistical characteristics of the data chunk. The foregoing is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the second predetermined criterion is a criterion for detecting misclassified data among data acquired using a model, and the data acquired using the model changes from a first state to a second state. Based on at least one of a changing first point or a second point in which the output of the model changes from the second state to the first state, the data obtained by inputting time series data to the model may be divided into a plurality of data chunks. have. The first state and the second state may mean a binary classification result when the computing device 100 performs binary classification using a model. For example, the first state may be normal, and the second state may be anomaly. The computing device 100 may determine a criterion for dividing the data chunk as a first point and/or a second point. Accordingly, the computing device 100 may determine a first point that changes from the first state to the second state as the start point of the data chunk. In this case, the computing device 100 may determine a second point that changes from the second state to the first state as the end point of the corresponding data chunk. Conversely, the computing device 100 may determine a second point that changes from the second state to the first state as a starting point of the data chunk. In this case, the computing device 100 may determine the first point that changes from the first state to the second state as the end point of the corresponding data chunk. Through this, the computing device 100 may classify a data chunk in which the model output is normal and a data chunk in which the model output is anomaly. Accordingly, the computing device 100 may display data chunks for each classification result in the data chunk area 450. The user can see the displayed data chunk and determine which data chunk is misclassified data. For example, as illustrated in FIG. 6, a data chunk 1 451, a data chunk 2 453, and a data chunk 3 455 may be displayed in the data chunk area 450. Here, the data chunk 1 451 may be misclassified data (eg, false positive-normal data, but when the output obtained by using the model is anomaly). That is, if the prediction of the model is incorrect, the computing device 100 may receive an input signal from the user and relabel the misclassified data chunk. That is, if the data is normal, but the output obtained by using the model is anomaly, the processor 110 may relabel the label of the corresponding data as normal. The computing device 100 may retrain the model by inputting the relabeled data chunk into the model. Through this, it is possible to improve the prediction accuracy of the model. In the case of data chunk 3 455, a result value obtained by using the model may be Normal. And the input data may also be Normal. In this case, data chunk 3 455 may be an accurately predicted and/or classified data set. Therefore, the need for data chunk 3 to be retrained by re-input to the model may decrease. In this case, the computing device 100 may not perform relabeling of the data chunk 3. The foregoing is only an example, and the present disclosure is not limited thereto.

7 is a diagram illustrating an example of a model archive output screen according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the computing device 100 may display the model archive output screen 510 in the second output area. The computing device 100 may receive an input signal for the model archive selection object 337 and display the model archive output screen 510 on the second output area 340.

According to an embodiment of the present disclosure, the model archive output screen 510 may include a screen displaying information on each of a plurality of models. The model archive output screen 510 is a model list output area 513 for displaying a plurality of models stored in the model archive at a glance, or model information for displaying information on a selected model by receiving a user's selection input signal. It may include at least one of the output areas 515. The model archive may include a repository in which a plurality of models are stored. The model archive may contain, for example, trained models, untrained models, retrained models, and the like. The computing device 100 may call a model selected by the user by receiving a user's input signal from the model archive. According to an embodiment of the present disclosure, the model list output area 513 may include an area for displaying a plurality of models stored in the model archive so that they can be viewed at a glance. The model list output area 513 may include an area for displaying a model selected based on a user's selection input so that the selected model can be viewed at a glance. For example, the model list output area 513 may include a model included in a project, a model selected by a user, a model recommended to a user, and the like. According to an embodiment of the present disclosure, the model information output area 515 may include an area for receiving a user's selection input signal and displaying information on a selected model. The model information output area 515 is an area in which, for example, a model name, a model status (eg, before training, after training, during training), a model creation date, information on a training data set that trained the model, and the like are displayed. Can include. The foregoing is only an example, and the present disclosure is not limited thereto.

8 is an exemplary diagram for describing a hit model according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the model list may include a model 521 determined based on a hit rate of each of a plurality of models included in the model archive.

According to an embodiment of the present disclosure, a model archive 520 and a hit model 521 are shown in FIG. 8. The hit model 521 may include a model determined based on a hit rate of each of a plurality of models included in the model archive in order to recommend a model corresponding to the newly input data to the user. The hit model 521 may refer to a model in which input data is determined to be normal or abnormal. Among the plurality of models, a model for which a final judgment on input data is made may be a hit model 521. For newly input time series data, the user may not know which model outputs the best results. In addition, when comparing performance by inputting newly input time series data to all models included in the model archive, it may take a lot of time and cost. Accordingly, for newly input time series data, the computing device 100 may select a model with a high probability of hitting a model stored in the model archive 520 and display a model with the highest probability of hitting. Through this, the user can quickly acquire a model suitable for newly inputted data without going through an experiment process. The foregoing is only an example, and the present disclosure is not limited thereto.

9 is an exemplary diagram for describing an integrated model according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the model list may include a model 525 newly generated by integrating models having similar statistical characteristics among a plurality of models included in the model archive.

In accordance with an embodiment of the present disclosure, in FIG. 9 a model archive 520, a model 523 having a similar distribution, and an integrated model 525 are shown. The computing device 100 may newly generate an integrated model 525 by integrating a model 523 having a similar distribution. That is, the computing device 100 may generate a model that is likely to be used for newly input data by integrating at least one or more models that are unlikely to be used for newly input data. Through this, the computing device 100 may selectively calculate a hit rate for each of the models included in the group that is likely to be used, so that a model suitable for newly input data may be quickly calculated. The foregoing is only an example, and the present disclosure is not limited thereto.

10 is a diagram illustrating an example of a detected anomaly output screen according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the computing device 100 may display the detected anomaly output screen in the second output area. The computing device 100 may receive an input signal for the detected anomaly output selection object 339 and display the detected anomaly output screen 530 on the second output area 340.

According to an embodiment of the present disclosure, the computing device 100 may display the detected anomaly output screen 530 in the second output area. The detected anomaly output screen 530 may include an anomaly detection result output area for displaying the anomaly detection result obtained using the model.

According to an embodiment of the present disclosure, the detected anomaly output screen 530 may include a screen for displaying information related to anomaly data among data acquired using a model. The detected anomaly output screen 530 displays information on the anomaly data selected by receiving an anomaly detection result output area 531 or a user selection input signal for displaying a list of anomaly data acquired using the model. It may include at least one of the anomaly information output area 533 for performing the operation. The anomaly detection result output area 531 may include an area for displaying an anomaly data list acquired using a model. For example, the computing device 100 may display data classified as anomaly among data acquired using a model on the output area 531 as a result of the anomaly detection. The anomaly information output area 533 may include an area for receiving a user's selection input signal and displaying information on the selected anomaly data. For example, the computing device 100 may receive a user's selection input signal and display the information on the selected anomaly data, such as a start time of anomaly occurrence, an end time of anomaly end, a period at which anomaly occurs, and the like. have. The foregoing is only an example, and the present disclosure is not limited thereto.

11 is a diagram illustrating an output screen of a monitoring line according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the computing device 100 may display a model performance output screen in the second output area. The computing device 100 may receive an input signal for the performance monitoring selection object 331 and display the model performance output screen 550 on the second output area 340.

According to an embodiment of the present disclosure, as illustrated in FIG. 11, the computing device 100 may display a model performance output screen 550 in the second output area. The model performance output screen 550 may include a screen for displaying model performance information. The model performance information may include all information related to the model performance. For example, the performance information of the model may include a measure of how accurate the model outputs prediction results for input data. In addition, the performance information of the model may include a predicted value obtained by the computing device 100 using the model. The foregoing is only an example, and the present disclosure is not limited thereto.

12 is a flowchart of a method for training a neural network according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, a first screen including at least one first object for receiving a selection input for a project may be displayed 710.

According to an embodiment of the present disclosure, a second screen 720 for displaying project-related information corresponding to the selected project may be displayed.

According to an embodiment of the present disclosure, the second screen includes a first output area for displaying time series data acquired from a sensor, and at least one second object for receiving a selection input related to model retraining. It may include at least one of a second output area for displaying information corresponding to the area or the second object.

According to an embodiment of the present disclosure, the project is an artificial intelligence-related project for achieving a specific goal by using artificial intelligence, and the specific goal may include a goal of improving the performance of a model to which the artificial intelligence is applied.

According to an embodiment of the present disclosure, the selection area is an area including an object for displaying information related to model training in the second output area, and performance monitoring selection for displaying the performance information of the model in the second output area An object, a training data set selection object for displaying training data set information related to training of the model in the second output area, a learning console selection object for displaying information on the training progress of the current model in the second output area, at least At least one of a model archive selection object for displaying information on one or more models in a second output area, or a sensed anomaly output selection object for displaying anomaly information detected using a model in a second output area Can include.

According to an embodiment of the present disclosure, further comprising an operation of displaying a training data set output screen in the second output area, wherein the training data set output screen includes: a list of training data sets in which at least one training data set is listed, and a user At least one of a training data set addition object for receiving selection input for a training data set to be used for model training, or a training data set removal object for receiving selection input for a training data set that is not to be used by the user to train the model. Can include.

According to an embodiment of the present disclosure, the training data set includes at least one of a first training data set used for training a model, or a new second training data set used for retraining a model, and the second training data The set may include at least some of the time series data acquired from the sensor in real time and a label corresponding thereto.

According to an embodiment of the present disclosure, the training data set selection screen is a screen that allows a user to select a second training data set, and data obtained by inputting time series data into a model are based on a predetermined first criterion. It may include at least one of a time variable setting area for filtering with or a data chunk area for displaying data chunks obtained by inputting time series data into the model and dividing the data based on a predetermined second criterion.

According to an embodiment of the present disclosure, the data chunk may include statistical characteristics of each data set obtained by inputting a plurality of time series data divided based on a second predetermined criterion into a model.

According to an embodiment of the present disclosure, the second predetermined criterion is a criterion for detecting misclassified data among data acquired using a model, and the data acquired using the model changes from a first state to a second state. Based on at least one of a changing first point or a second point in which the output of the model changes from the second state to the first state, the data obtained by inputting time series data to the model may be divided into a plurality of data chunks. have.

According to an embodiment of the present disclosure, the data chunk includes a data chunk including a data chunk calculated through a data chunk recommendation algorithm for a data chunk to be used for model retraining to a user, or data selected by receiving a user selection input signal. It may include at least one of data chunks including at least one data chunk having a similar statistical characteristic to the chunk.

According to an embodiment of the present disclosure, the model archive output screen is a screen for displaying information on each of a plurality of models, and a model list output area or a user for displaying a plurality of models stored in the model archive so that they can be viewed at a glance. It may include at least one of the model information output area for receiving the selection input signal of and displaying information on the selected model.

According to an embodiment of the present disclosure, the model list includes a model that is trained while a project is being progressed, a model that is retrained by inputting a second training data set to the trained model, and a statistical feature similar among a plurality of models included in the model archive In order to recommend to the user a newly created model by integrating a model having a model or a model corresponding to the newly input data, at least one of the models determined based on the hit rate of each of the plurality of models included in the model archive may be included. I can.

According to an embodiment of the present disclosure, the detected anomaly output screen is a screen for displaying information related to anomaly data among data acquired using a model, and displays a list of anomaly data acquired using a model. It may include at least one of an anomaly detection result output area for receiving a user selection input signal and an anomaly information output area for displaying information on the selected anomaly data by receiving a user selection input signal.

13 is a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

While the present disclosure has been described above as generally capable of being implemented by a computing device, one of ordinary skill in the art would appreciate the disclosure in combination with computer-executable instructions and/or other program modules that may be executed on one or more computers and/or of hardware and software. It will be appreciated that it can be implemented in combination.

Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In addition, to those skilled in the art, the method of the present disclosure is not limited to single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable household appliances, etc. (each of which is It will be appreciated that other computer system configurations may be implemented, including one that may operate in connection with one or more associated devices.

The described embodiments of the present disclosure may also be practiced in a distributed computing environment where certain tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer-readable media. Computer-readable media can be any computer-readable media, including volatile and nonvolatile media, transitory and non-transitory media, removable and non-transitory media. Includes removable media. By way of example and not limitation, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and nonvolatile media, temporary and non-transitory media, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Includes the medium. Computer-readable storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage. Devices, or any other medium that can be accessed by a computer and used to store desired information.

Computer-readable transmission media typically implement computer-readable instructions, data structures, program modules, or other data on a modulated data signal such as a carrier wave or other transport mechanism. Includes all information delivery media. The term modulated data signal refers to a signal in which one or more of the characteristics of the signal is set or changed so as to encode information in the signal. By way of example and not limitation, computer-readable transmission media include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above-described media are also intended to be included within the scope of computer-readable transmission media.

An exemplary environment 1100 is shown that implements various aspects of the present disclosure, including a computer 1102, which includes a processing device 1104, a system memory 1106, and a system bus 1108. do. System bus 1108 couples system components, including but not limited to, system memory 1106 to processing device 1104. The processing unit 1104 may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be used as processing unit 1104.

The system bus 1108 may be any of several types of bus structures that may be additionally interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. System memory 1106 includes read-only memory (ROM) 1110 and random access memory (RAM) 1112. The basic input/output system (BIOS) is stored in non-volatile memory 1110 such as ROM, EPROM, EEPROM, etc. This BIOS is a basic input/output system that helps transfer information between components within the computer 1102, such as during startup. Includes routines. RAM 1112 may also include high speed RAM such as static RAM for caching data.

The computer 1102 also includes an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA)-this internal hard disk drive 1114 can also be configured for external use within a suitable chassis (not shown). Yes—, a magnetic floppy disk drive (FDD) 1116 (for example, to read from or write to a removable diskette 1118), and an optical disk drive 1120 (for example, a CD-ROM). For reading the disk 1122 or for reading from or writing to other high-capacity optical media such as DVD). The hard disk drive 1114, the magnetic disk drive 1116, and the optical disk drive 1120 are each connected to the system bus 1108 by a hard disk drive interface 1124, a magnetic disk drive interface 1126, and an optical drive interface 1128. ) Can be connected. The interface 1124 for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer readable media provide non-volatile storage of data, data structures, computer executable instructions, and the like. In the case of computer 1102, drives and media correspond to storing any data in a suitable digital format. Although the description of the computer-readable medium above refers to a removable optical medium such as a HDD, a removable magnetic disk, and a CD or DVD, those skilled in the art may use a zip drive, a magnetic cassette, a flash memory card, a cartridge, etc. It will be appreciated that other types of computer-readable media, such as the ones, may also be used in the exemplary operating environment and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

A number of program modules, including the operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136, may be stored in the drive and RAM 1112. All or part of the operating system, applications, modules and/or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented on several commercially available operating systems or combinations of operating systems.

A user may input commands and information into the computer 1102 through one or more wired/wireless input devices, for example, a pointing device such as a keyboard 1138 and a mouse 1140. Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, and the like. These and other input devices are often connected to the processing unit 1104 through the input device interface 1142, which is connected to the system bus 1108, but the parallel port, IEEE 1394 serial port, game port, USB port, IR interface, It can be connected by other interfaces such as etc.

A monitor 1144 or other type of display device is also connected to the system bus 1108 through an interface such as a video adapter 1146. In addition to the monitor 1144, the computer generally includes other peripheral output devices (not shown) such as speakers, printers, and the like.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148, via wired and/or wireless communication. The remote computer(s) 1148 may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other common network node, and is generally connected to the computer 1102. Although it includes many or all of the components described for simplicity, only memory storage device 1150 is shown. The logical connections shown include wired/wireless connections to a local area network (LAN) 1152 and/or to a larger network, eg, a wide area network (WAN) 1154. Such LAN and WAN networking environments are common in offices and companies, and facilitate enterprise-wide computer networks such as intranets, all of which can be connected to worldwide computer networks, for example the Internet.

When used in a LAN networking environment, the computer 1102 is connected to the local network 1152 via a wired and/or wireless communication network interface or adapter 1156. Adapter 1156 may facilitate wired or wireless communication to LAN 1152, which also includes a wireless access point installed therein to communicate with wireless adapter 1156. When used in a WAN networking environment, the computer 1102 may include a modem 1158, connected to a communication computing device on the WAN 1154, or through the Internet, to establish communication over the WAN 1154. Have other means. The modem 1158, which may be an internal or external and a wired or wireless device, is connected to the system bus 1108 through a serial port interface 1142. In a networked environment, program modules described for the computer 1102 or portions thereof may be stored in the remote memory/storage device 1150. It will be appreciated that the network connections shown are exemplary and other means of establishing communication links between computers may be used.

Computer 1102 is associated with any wireless device or entity deployed and operated in wireless communication, e.g., a printer, scanner, desktop and/or portable computer, portable data assistant (PDA), communication satellite, wireless detectable tag. It operates to communicate with any device or place, and a phone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network or may simply be ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) allows you to connect to the Internet or the like without wires. Wi-Fi is a wireless technology such as a cell phone that allows such devices, for example computers, to transmit and receive data indoors and outdoors, that is, anywhere within the coverage area of a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide a secure, reliable and high-speed wireless connection. Wi-Fi can be used to connect computers to each other, to the Internet, and to a wired network (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz radio bands, for example, at a data rate of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in a product including both bands (dual band) .

Those of ordinary skill in the art of this disclosure will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols and chips that may be referenced in the above description are voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields. Or particles, or any combination thereof.

A person of ordinary skill in the art of the present disclosure includes various exemplary logical blocks, modules, processors, means, circuits and algorithm steps described in connection with the embodiments disclosed herein, electronic hardware, (convenience). For the sake of clarity, it will be appreciated that it may be implemented by various types of programs or design code or a combination of both (referred to herein as software). To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. A person of ordinary skill in the art of the present disclosure may implement the described functions in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CD, DVD, etc.), smart cards, and flash Memory devices (eg, EEPROM, cards, sticks, key drives, etc.). In addition, the various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. Based on the design priorities, it is to be understood that within the scope of the present disclosure a specific order or hierarchy of steps in processes may be rearranged. The appended method claims provide elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

A description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those of ordinary skill in the art, and general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not to be limited to the embodiments presented herein, but is to be construed in the widest scope consistent with the principles and novel features presented herein.

Claims (17)

A computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, causes the following operations to be performed for training a neural network, the operations:
Displaying a first screen including at least one first object for receiving a selection input for a project; And
Displaying a second screen for displaying project-related information corresponding to the selected project;
Including,
The second screen,
A first output area for displaying time series data acquired from a sensor, a selection area including at least one second object for receiving a selection input related to model retraining, or for displaying information corresponding to the second object Including at least one of the second output areas,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
The above project is:
As an artificial intelligence-related project to achieve specific goals using artificial intelligence,
The specific goals above are,
Including the goal of improving the performance of models to which artificial intelligence is applied,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
The selection area,
An area including an object for displaying information related to model training in a second output area,
A performance monitoring selection object for displaying model performance information in the second output area,
A training data set selection object for displaying training data set information related to training of the model in the second output area,
Learning console selection object for displaying information on the training progress of the current model in the second output area,
A model archive selection object for displaying information on at least one model in the second output area, or
Among the detected anomaly output selection objects for displaying the anomaly information detected using the model in the second output area
Containing at least one,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
Displaying a training data set output screen in the second output area;
Including more,
The training data set output screen,
A list of training data sets listing at least one training data set,
A training data set additional object for the user to receive selection input for the training data set to be used for model training, or
Training dataset removal object for the user to receive selection inputs for training datasets that will not be used to train the model.
Containing at least one of,
A computer program stored on a computer-readable storage medium.
The method of claim 4,
The training data set,
The first set of training data used to train the model, or
A new second training data set to be used for retraining the model
Including at least one of,
The second training data set,
Including at least a portion of the time series data acquired in real time from the sensor and a label corresponding thereto,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
Displaying a training data set selection screen allowing a user to select a second training data set;
It further includes,
A computer program stored on a computer-readable storage medium.
The method of claim 6,
The training data set selection screen,
As a screen allowing a user to select the second training data set,
A time variable setting area for filtering the data obtained by inputting the time series data into the model based on a predetermined first criterion, or
A data chunk area for displaying a data chunk obtained by inputting the time series data into a model and dividing the data obtained based on a second predetermined criterion
Containing at least one of,
A computer program stored on a computer-readable storage medium.
The method of claim 7,
The data chunk is
Statistical features of each data set obtained by inputting a plurality of the time series data divided based on a predetermined second criterion into a model
Containing,
A computer program stored on a computer-readable storage medium.
The method of claim 7,
The predetermined second criterion is,
As a criterion for detecting misclassified data among data acquired using the model,
A first point at which data acquired using the model changes from a first state to a second state, a second point at which the output of the model changes from the second state to the first state, and the first state of the model Data obtained by inputting the time series data to a model based on at least one of a third point present in the output or a fourth point present in the output of the model in the second state is converted into a plurality of data chunks. Standard to divide
Containing,
A computer program stored on a computer-readable storage medium.
The method of claim 7,
The data chunk is
A data chunk including a data chunk calculated through a data chunk recommendation algorithm for the data chunk to be used for model retraining to the user, or
A data chunk including at least one data chunk having statistical characteristics similar to the selected data chunk by receiving a user-selected input signal
Containing at least one of,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
Displaying a model archive output screen in the second output area;
It further includes,
A computer program stored on a computer-readable storage medium.
The method of claim 11,
The above model archive output screen is:
As a screen that displays information on each of a plurality of models,
Model list output area for displaying multiple models stored in the model archive in a glance, or
Model information output area for receiving user's selection input signal and displaying information on the selected model
Containing at least one of,
A computer program stored on a computer-readable storage medium.
The method of claim 12,
The above model list is:
The model learned while proceeding with the above project,
The retrained model by inputting the second training data set into the trained model,
A model newly created by integrating a model having similar statistical characteristics among a plurality of models included in the model archive, or
A model determined based on the hit rate of each of a plurality of models included in the model archive in order to recommend a model corresponding to the newly inputted data to the user
Containing at least one of,
A computer program stored on a computer-readable storage medium.
The method of claim 1,
Displaying an anomaly output screen detected in the second output area;
It further includes,
A computer program stored on a computer-readable storage medium.
The method of claim 14,
The detected anomaly output screen,
As a screen for displaying information related to anomaly data among data acquired using the model,
An anomaly detection result output area for displaying an anomaly data list acquired using the model, or
An anomaly information output area for receiving a user-selected input signal and displaying information on the selected anomaly data
Containing at least one of
A computer program stored on a computer-readable storage medium.
As a way to train a neural network,
Displaying a first screen including at least one first object for receiving a selection input for a project; And
Displaying a second screen for displaying project-related information corresponding to the selected project;
Including,
The second screen,
A first output area for displaying time series data acquired from a sensor, a selection area including at least one second object for receiving a selection input related to model retraining, or for displaying information corresponding to the second object Including at least one of the second output areas,
A method for training a neural network.
A computing device for providing a method for training a neural network,
A processor including one or more cores; And
Memory;
Including,
The processor,
Displaying a first screen including at least one first object for receiving a selection input for the project, and
Display a second screen for displaying project-related information corresponding to the selected project,
The second screen,
A first output area for displaying time series data acquired from a sensor, a selection area including at least one second object for receiving a selection input related to model retraining, or for displaying information corresponding to the second object Including at least one of the second output areas,
Computing device

Images