WO2020101108A1

WO2020101108A1 - Artificial-intelligence model platform and method for operating artificial-intelligence model platform

Info

Publication number: WO2020101108A1
Application number: PCT/KR2018/015476
Authority: WO
Inventors: 송중석; 권태웅; 최상수; 최윤수; 이윤수; 박진학; 신익수; 이혁로; 박학수; 박진형
Original assignee: 한국과학기술정보연구원
Priority date: 2018-11-17
Filing date: 2018-12-07
Publication date: 2020-05-22
Also published as: KR102271449B1; KR20200057903A

Abstract

The present invention relates to a technique, which implements an artificial-intelligence model platform capable of creating an artificial-intelligence model for security control and, in particular, can optimally recommend/apply feature information and normalization methods directly related to artificial-intelligence model performance, thereby enabling general users, who are not familiar with a security control technique, to create an optimal artificial-intelligence model for security control.

Description

AI model platform and how to operate AI model platform

The present invention relates to a technology for generating artificial intelligence models for security control.

More specifically, it is to provide an artificial intelligence model platform that enables an ordinary user who is not familiar with security control technology to generate an optimal AI model for security control.

Currently, the Science and Technology Cyber Safety Center provides real-time security control services for public research institutes based on the TMS.

The real-time security control service is provided as a service structure that provides analysis and response support by security control personnel based on security events detected and collected by the intrusion threat management system (TMS).

However, the number of security events detected by TMS is explosively increasing, and it is realistically difficult for security control personnel to analyze such large-capacity security events.

In addition, since the existing security control service relies on the expertise and experience of the security control personnel, analysis leveling cannot be realized, such as occurrence of work bias that concentrates analysis on specific security events or deviation of analysis results. Things are happening.

As a result, in the current situation in which the number of security events detected by TMS is exploding, it is necessary to innovate the service structure of the existing security control service itself, which relies on the analysis of security control personnel.

Accordingly, it is possible to think of a security control service structure that utilizes an artificial intelligence model that can replace the analysis of security control personnel.

In the present invention, it is intended to provide an artificial intelligence model platform capable of generating an artificial intelligence model for security control.

In particular, in the present invention, it is intended to provide an artificial intelligence model platform that enables an ordinary user who is not familiar with security control technology to generate an optimal artificial intelligence model for security control.

The object to be reached in the present invention is to provide a method (technology) for implementing an AI model platform that enables generation of an AI model for security control.

An artificial intelligence model platform according to an embodiment of the present invention includes: a data collection module for collecting security events to be used as learning / test data by specific search conditions from source security data; A feature extraction module that extracts preset feature information for the collected security event; A normalization module that performs preset normalization on the extracted feature information of the security event; A data output module that extracts learning data or test data from the security event where the specific information normalization is completed according to a given condition; And a model generation module that applies an artificial intelligence algorithm to the learning data to generate an artificial intelligence model for security control.

Specifically, by using the test data, it may further include a performance management module for testing the accuracy of the artificial intelligence model.

Specifically, a UI module that provides a user interface (UI) for setting at least one of specific search conditions of the data collection module, feature information of the feature extraction module, normalization method of the normalization module, and conditions of the data output module. It may further include.

Specifically, when the total number of collection cases exceeds the maximum number of collections that can be simultaneously performed, the data collection module sequentially stores the number of collections exceeding the maximum number of collections in a queue and sequentially proceeds. In the case of a collection case proceeding after being stored in the queue, the security event can be collected only for data prior to the occurrence point of the collection case in the source security data.

Specifically, the feature extraction module may recommend a change to the feature information to increase the accuracy of the AI model based on the result of the accuracy test of the performance management module.

Specifically, the normalization module may recommend changing the normalization method for the normalization to increase the accuracy of the artificial intelligence model.

The feature information recommendation apparatus according to an embodiment of the present invention is a model performance confirmation unit that checks model performance with respect to an AI model generated based on learning predetermined feature information among all feature information that can be set when generating an AI model. ; A combination performance checking unit configured to set a plurality of feature information combinations from the entire feature information to check the performance of the artificial intelligence model generated based on learning for each of the plurality of feature information combinations; And a recommendation unit recommending a specific feature information combination having a higher performance than the model performance confirmed by the model performance checking unit among performances of the plurality of feature information combinations.

Specifically, the combination of the plurality of feature information is a combination in which at least one of the remaining specific information excluding the preset feature information from the entire feature information is sequentially added to the preset feature information, and the specific feature information combination is: Among the plurality of feature information combinations, it may be the top N having higher performance than the model performance.

Specifically, the predetermined specific information is the entire feature information, and the combination performance checking unit is the maximum performance among the performances of the artificial intelligence model generated based on learning for each single feature information in the whole feature information. Single feature information performance comparison process to check if the maximum performance is higher than the model performance, the single feature information of the maximum performance is reset to the feature information, and the feature information is preset to the feature information from the whole feature information. The combination setting process of setting the combination of the plurality of characteristic information by sequentially adding one of the remaining specific information one by one, each of the combination of feature information having a higher performance than the model performance of the re-set feature information among the plurality of characteristic information combinations Resetting process for resetting the feature information so that the combination setting process is repeatedly performed for each re-set feature information. When there is no feature information combination having a higher performance than the model performance among the multiple feature information combinations , It is possible to perform a process of delivering the previous feature information as the combination of the specific feature information to the recommendation unit.

An apparatus for recommending a normalization method according to an embodiment of the present invention includes: an attribute confirmation unit that checks an attribute of feature information used for learning when generating an artificial intelligence model; Determining unit for determining a normalization method according to the attribute of the feature information, from among all the settable normalization method; And a recommendation unit recommending the determined normalization method.

Specifically, if the same normalization method is applied to all of the feature information fields, if the attribute of the feature information is a numeric attribute, the determination unit determines a first normalization method according to the whole number pattern of the feature information, and the feature When the attribute of the information is a category attribute, a second normalization method for expressing as a non-zero characteristic value is determined only at a designated location for each category of feature information in a vector defined as the total number of categories of the feature information, and the feature information When the attribute of is a combination of a number and a category, the second normalization scheme and the first normalization scheme may be determined.

Specifically, the first normalization method includes a standard score normalization method, a mean normalization normalization method, and a feature scaling normalization method according to a predefined priority, and the determining unit includes standard deviations for all numeric patterns of feature information and Based on whether there is an upper / lower limit of the normalization scaling range, a normalization scheme having the highest priority applicable among the first normalization schemes may be determined.

Specifically, if the normalization method is applied for each field in the entire field of the feature information, the determining unit has the highest priority applicable among the normalization method and the normalizing method of the feature scaling for the attribute attribute type field in the feature information. A high normalization method is determined, and for the field of the attribute whose number of attributes is attribute in the feature information, a normalization method having the highest priority applicable among the mean normalization normalization method and the feature scaling normalization method is determined, and the attribute attribute is the ratio attribute in the feature information. The normalization method is not determined and excluded from the normalization target for the field of, or the standard score normalization method is determined, and whether the attribute exists in the feature information, the normalization scheme is not determined and excluded from the normalization target. have.

An artificial intelligence model platform operating method according to an embodiment of the present invention includes: a data collection step of collecting security events to be used as learning / test data according to specific search conditions from source security data; A feature extraction step of extracting predetermined feature information for the collected security event; A normalization step of performing preset normalization on the extracted feature information of the security event; A data output step of extracting training data or test data according to a given condition from the security event in which the specific information normalization is completed; And a model generation step of applying an artificial intelligence algorithm to the learning data to generate an artificial intelligence model for security control.

Specifically, a performance management step of testing the accuracy of the artificial intelligence model using the test data may be further included.

Specifically, providing a user interface (UI) for setting at least one of a specific search condition of the data collection step, feature information of the feature extraction module, normalization method of the normalization module, and condition of the data output module. It may further include.

Specifically, in the data collection step, if the total number of collection cases exceeds the maximum number of simultaneous collections, the number of collection cases exceeding the maximum number of collection cases is stored in a queue and sequentially performed. In the case of a collection case proceeding after being stored in the queue, the security event can be collected only for data prior to the occurrence point of the collection case in the source security data.

Specifically, based on the result of the accuracy test of the performance management step, it may further include the step of recommending a change to the feature information to increase the accuracy of the artificial intelligence model.

Specifically, in the normalization step, it is possible to recommend changing the normalization method for the normalization to increase the accuracy of the artificial intelligence model.

A computer program according to an embodiment of the present invention is a model that checks model performance with respect to an artificial intelligence model generated based on learning preset feature information among all feature information that can be set when generating an artificial intelligence model in combination with hardware. Performance check step; A combination performance checking step of setting a combination of a plurality of feature information from the whole feature information, and confirming the performance of the artificial intelligence model generated based on learning for each of the plurality of feature information combinations; And a performance of recommending a specific feature information combination having a higher performance than the model performance confirmed by the model performance checking unit among performances of the plurality of feature information combinations.

Specifically, the predetermined specific information is the entire feature information, and the combination performance checking step is the maximum performance among the performances of the artificial intelligence model generated based on learning for each single feature information in the whole feature information. Single feature information performance comparison process to check whether it is higher, if the maximum performance is higher than the model performance, the single feature information of the maximum performance is reset to the feature information, and the predetermined feature is set in the whole feature information in the feature information The combination setting process of setting the combination of the plurality of feature information by sequentially adding one by one of the specific information except the information, each of the feature information combinations having higher performance than the model performance of the re-set feature information among the plurality of feature information combinations Resetting as the feature information, so that the combination setting process is repeatedly performed for each re-set feature information, there is no feature information combination having a higher performance than the model performance among the multiple feature information combinations. In this case, a process of delivering the previous feature information as the specific feature information combination to the recommender may be performed.

Specifically, the predetermined specific information is the entire feature information, and the combination performance checking step is the maximum performance among the performances of the artificial intelligence model generated based on learning for each single feature information in the whole feature information. Single feature information performance comparison process to check if it is higher, if the maximum performance is not higher than the model performance, combination setting process to set the combination of the plurality of feature information excluding one specific information from the feature information, the plurality of Among the feature information combinations, each of the feature information combinations having a performance higher than the model performance is reset as feature information, and a reset process is performed so that the combination setting process is repeatedly performed for each re-set feature information. If there is no feature information combination having a higher performance than the model performance, a process of delivering the immediately preceding feature information as the specific feature information combination to the recommendation unit may be performed.

A computer program according to an embodiment of the present invention comprises: an attribute checking step in combination with hardware to check the attribute of feature information used for learning when creating an artificial intelligence model; A determining step of determining a normalization method according to the attribute of the feature information from among all the settable normalization methods; And a recommendation step of recommending the determined normalization method.

Specifically, in the determining step, if the same normalization method is applied to all the feature information fields, if the attribute of the feature information is a numeric attribute, the first normalization method according to the whole number pattern of the feature information is determined, and the When the attribute of the feature information is a category attribute, a second normalization method for expressing as a non-zero characteristic value is determined only at a designated location for each category of the feature information in a vector defined as the total number of categories of the feature information, and the feature When the attribute of the information is a combination attribute of a number and a category, the second normalization method and the first normalization method may be determined.

Specifically, the first normalization method includes a standard score normalization method, a mean normalization normalization method, and a feature scaling normalization method according to a predefined priority, and the determining step includes standard deviation of the entire numeric pattern of feature information. And a normalization scheme having the highest priority applicable among the first normalization schemes based on whether there is an upper / lower limit of the normalization scaling range.

Specifically, in the determining step, if the normalization method is applied for each field in the entire feature information field, the attribute has the highest priority applicable to the normalization method of the feature normalization method and the feature scaling normalization method for the attribute type attribute field in the feature information. Determines a normalization method having a high value, determines a normalization method having the highest priority applicable among a means normalization method and a feature scaling normalization method for the field of the attribute number attribute in the feature information, and the attribute ratio in the feature information For the field of the attribute, the normalization method is not determined and excluded from the normalization target, or the standard score normalization method is determined, and whether the attribute is present in the feature information or not. Can be.

Accordingly, according to the artificial intelligence model platform and the artificial intelligence model platform operating method according to the present invention, an artificial intelligence model platform capable of generating an artificial intelligence model for security control is implemented, in particular, a feature directly related to the performance of the artificial intelligence model. By allowing information and normalization methods to be optimally recommended / applied, it is possible to implement an artificial intelligence model platform that allows an average user who is not familiar with security control technology to generate an optimal artificial intelligence model for security control.

For this reason, according to the present invention, since the optimal artificial intelligence model suitable for the purpose and requirements for security control can be flexibly and variously generated and applied, the quality improvement of the security control service can be maximized, and large scale It can be expected to have the effect of supporting the construction of an AI-based infringement response system to efficiently analyze the signs of cyber attacks and anomalies.

1 is a conceptual diagram showing an AI model platform according to an embodiment of the present invention.

2 is a configuration diagram showing the configuration of the AI model platform according to an embodiment of the present invention.

3 is a block diagram showing the configuration of a feature information recommendation device according to an embodiment of the present invention.

4 is a configuration diagram showing the configuration of a normalization method recommendation apparatus according to an embodiment of the present invention.

5 is a flowchart illustrating an artificial intelligence model platform operating method according to an embodiment of the present invention.

6 is a flowchart illustrating a method of operating a feature information recommendation device according to an embodiment of the present invention.

7 is a flowchart illustrating an operation method of a normalization method recommendation apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

Currently, the real-time security control service provided by the Science and Technology Cyber Safety Center is based on security events detected and collected by the intrusion threat management system (TMS), and provides rule-based analysis and response support by security control personnel. It has a service structure that is made.

1 conceptually shows an embodiment of the AI model platform proposed in the present invention.

As shown in FIG. 1, the AI model platform of the present invention is based on various data collected and processed in a collection function and a collection function for collecting and processing various data necessary for generating an AI model for security control. Based on the artificial intelligence function that creates an intelligent model and manages the performance and history associated with it, and the management responsible for various settings and user management related to the collection / artificial intelligence function based on the user interface (UI) provided to system administrators and general users. It can be divided into functions.

And, the artificial intelligence model platform of the present invention includes a search engine that periodically collects the newly generated source security data from the big data integrated storage storage, and loads various data from the collection function into the search engine to search the data. Can be used as storage.

In this case, various modules (eg, collection / feature extraction / normalization / output) belonging to the collection function may operate based on a search engine (data storage).

Hereinafter, the configuration of the AI model platform and the role of each configuration will be described in detail with reference to FIG. 2.

The artificial intelligence model platform 100 of the present invention includes a data collection module 110, a feature extraction module 120, a normalization module 130, a data output module 140, and a model generation module 150.

Furthermore, the AI model platform 100 of the present invention may further include a performance management module 160 and a UI module 170.

All or at least part of the configuration of the AI model platform 100 may be implemented in the form of a hardware module or a software module, or a combination of a hardware module and a software module.

Here, the software module may be understood as, for example, instructions executed by a processor that controls operations within the AI model platform 100, and these instructions are in a form mounted in a memory in the AI model platform 100. Will have.

After all, the artificial intelligence model platform 100 according to an embodiment of the present invention realizes the technology proposed in the present invention through the above-described configuration, that is, the technology capable of generating an optimal artificial intelligence model for security control. , Hereinafter, each configuration in the AI model platform 100 for realizing this will be described in more detail.

First, the UI module 170 has at least one of a specific search condition of the data collection module 110, feature information of the feature extraction module 120, normalization method of the normalization module 130, and condition of the data output module 140. Provides a UI (User Interface) for setting.

For example, the UI module 170, according to the operation of a system administrator or a general user (hereinafter referred to as a user) to create an AI model for security control in the AI model platform 100 of the present invention, data Provides a UI for setting at least one of a specific search condition of the collection module 110, feature information of the feature extraction module 120, a normalization method of the normalization module 130, and a condition of the data output module 140.

Accordingly, the UI module 170, based on the provided UI, various settings related to the collection / artificial intelligence function, specifically the specific search condition of the data collection module 110 for the artificial intelligence model to be generated later, feature extraction module The feature information of 120, the normalization method of the normalization module 130, the conditions of the data output module 140, etc. are stored / managed in the user information / setting information storage.

The data collection module 110 collects security events to be used as learning / testing data based on specific search conditions, that is, predetermined search conditions previously set by the user, from the source security data.

For example, as a specific search condition of the data collection module 110, a date (or period) to be used as learning / test data, number of cases, IP, detection pattern name, detection pattern type, and the like may be set.

Here, the detection pattern name means a representative name of security logs detected by the intrusion threat management system (TMS), and the detection pattern type means a group of detection patterns having similar detection pattern characteristics (property, type). For example, the detection pattern type can be divided into six types: worm virus damage, data corruption and leakage, waypoint abuse, homepage alteration, service rejection attack damage, and simple intrusion attempt.

Accordingly, when the specific search condition is a date (or period), the data collection module 110 may collect security events belonging to a set date (or period) from the original security data.

Alternatively, if the specific search condition is the number, the data collection module 110 may collect the security events of the number (for example, 500,000) set at the specified time from the original security data.

Alternatively, when the specific search condition is IP, the data collection module 110 may collect a security event in which the IP set from the source security data matches the source IP or destination IP.

Of course, as a specific search condition, a combination of date (or period), number of cases, IP, detection pattern name, and detection pattern type may be set.

In this case, too, the data collection module 110 may collect security events according to a combination of date (or period), number, IP, detection pattern name, detection pattern type, etc. set from the source security data.

More specifically, the data collection module 110, in collecting security events from the original security data, as described above, may limit the maximum number of simultaneous executions to reduce the load on the system.

For example, when collecting security events belonging to a set date (or period) from the source security data, the total number of security event collection cases belonging to the set date (or period) is 1000,000, and the maximum number of concurrent collections It can be assumed that 500,000 cases.

In this case, the data collection module 110 determines that the total number of collections this time exceeds the maximum number of collections that can be performed simultaneously, and stores the collections that exceed the maximum number of collections in the queue in a queue After that, you can proceed sequentially.

That is, the data collection module 110 collects / progresses the maximum number of collections of 500,000 according to the time sequence among the total number of collections of 1000,000, but queues for 500,000 collections exceeding the maximum number of collections of 500,000 After storing in (queue), it can be collected / progressed sequentially.

In this case, the data collection module 110 collects security events only for the data prior to the occurrence of the collection case from the source security data in the case of 500,000 collection cases that proceed after being stored in the queue.

That is, in the case of 500,000 collection cases that are processed after being stored in the queue among the total number of collection cases of 1000,000, there is a difference between when the collection cases occurred and the actual collection / progress time, thus preventing security event collection errors. In order to do this, the security event is collected only from the data prior to the point of occurrence of the collection in the original security data.

On the other hand, the artificial intelligence model platform 100 of the present invention was previously mentioned that it includes a search engine that periodically collects the newly generated source security data from the big data integrated storage storage.

In this case, the data collection module 110 may collect security data from source security data in a search engine (data store).

Since the big data integrated storage storage is a storage utilized not only in the AI model platform 100 of the present invention, but also in other systems, when a large amount of data (security events) is collected from the big data integrated storage storage, the big data integrated storage storage Loads can also affect other systems.

However, in the present invention (the data collection module 110), the data collection module 110 does not collect security events directly from the big data integrated storage storage, but periodically only the source security data newly generated from the big data integrated storage storage. Since security events are collected based on the collected search engine, it is possible to avoid the big data integrated storage storage load problem described above.

The feature extraction module 120 extracts pre-set feature information for the security event collected by the data collection module 110, that is, pre-set feature information by the user.

When creating an artificial intelligence model, in order to classify data (security events) with an artificial intelligence algorithm, it is necessary to find out what features the data (security events) have and make them into vectors. This process is called feature information extraction process.

The feature extraction module 120 is responsible for performing a feature information extraction process for security events collected by the data collection module 110.

Then, the feature information of each security event extracted by the feature extraction module 120 will be used for machine learning (eg, deep learning) when creating an artificial intelligence model described later.

Particularly, in the present invention, the user can set a single feature as feature information and set a composite feature.

Here, the single feature means features that can be extracted from one security event.

For example, detection time, source IP, source port, destination IP, destination port, protocol, security event name, security event type, number of attacks, attack direction, packet size, automatic analysis result, dynamic analysis result, organization number, jumbo Whether it is a payload, a payload using a word2vec conversion method, or the like may belong to a single feature.

For reference, the payload conversion method using Word2Vec is a method of converting a word into a vector, and is a method of determining a vector of a corresponding word through a relationship between adjacent words. In normal sentences, words can be distinguished on a space-by-space basis, but payload is very difficult to distinguish in semantic units and contains a lot of special characters, so pre-processing is required to apply word2vec.

In the present invention, as a pre-process for applying word2vec, the following four steps can be performed.

1) Convert the hexadecimal-encoded string to ASCII string (convert to space except ASCII code value (32 ~ 127))

2) url encoded part processing (% 25-> '%',% 26-> '&',% 2A-> '*' ...)

3) Special symbols except '@', '\', '-', ':', '%', '_', '.', '!', '/', And '`' are replaced with spaces. Replace all uppercase letters with lowercase letters

4) Apply word2vec algorithm except for one word

On the other hand, the composite feature means a feature that can be extracted by using aggregate and statistical techniques between various security events.

For example, a security event group is formed based on a period or the number of cases, and one feature (eg, a result of an operation) that can be extracted through intra-group operations (eg, aggregation, statistical technique, etc.) is a complex feature. Can belong to.

For example, it is assumed that a security event group as shown in Table 1 below is formed based on a period (8.22 to 9.3).

One feature (e.g. four) that can be extracted through operations in the security event group (e.g., source IP, destination IP, number of security events with the security event name of 100.100.100.100/111.111.111.11/AAA), is compounded The feature extraction module 120 may extract pre-set feature information (single feature and / or composite feature) with respect to the security event collected by the data collection module 110.

The normalization module 130 performs predetermined normalization on the extracted feature information of the security event.

Normalization refers to the process of consistently matching the range of values of the extracted features. If field A has a range of 50 to 100 and field B has a range of 0 to 100, the meaning is different because even the same 50 is a value measured by different scales. Therefore, it is necessary to adjust the values of different fields to a common scale to have a certain meaning and this is called normalization.

The normalization module 130 performs normalization on the extracted feature information of the security event to adjust the values of different fields to a common scale according to a preset normalization method to have a certain meaning.

At this time, the preset normalization scheme means a normalization scheme preset by the user.

In the artificial intelligence model platform 100 of the present invention, the following three normalization methods are provided to allow a user to pre-set.

Equation 1 means Feature scaling [a, b] normalization, Equation 2 means Mean normalization [-1,1] normalization, and Equation 3 means Standard score normalization.

The normalization module 130 performs normalization on the extracted feature information of the security event according to the normalization method preset by the user among the three normalization methods described above.

The data output module 140 extracts training data or test data from a security event in which the normalization of specific information is completed, based on a given condition, that is, a preset (given) condition by the user.

Specifically, the data output module 140 outputs the security event for which the specific information is normalized, to a screen or a file according to a user's desired value, order, format, learning / test data ratio, and file division method.

The output training data or test data is managed through database or file storage for each date and user so that they can be used immediately when creating an artificial intelligence model.

The model generation module 150 applies an artificial intelligence algorithm to learning data managed in the output / file storage in the data output module 140 to generate an artificial intelligence model for security control.

That is, the model generation module 150 may apply an artificial intelligence algorithm to the learning data, and generate an artificial intelligence model for security control, for example, an artificial intelligence model of a function required by a user.

For example, the model generation module 150 may generate an artificial intelligence detection model for detecting whether a security event is malicious or not according to a user request, and an artificial intelligence classification model for classifying spying / falsification of a security event. You can also create

Specifically, the model generation module 150, based on the learning data managed in the output / file storage in the data output module 140, to an artificial intelligence algorithm, such as a machine learning (eg, Deep Learning) algorithm previously selected by the user. Accordingly, an artificial intelligence model for security control can be generated.

For example, the model generation module 150 uses a learning function (Loss function) representing a deviation between a predicted result and an actual result through a model in a machine learning technique based on Backward Propagation calculation. Accordingly, an artificial intelligence model in which the deviation of the loss function is zero based on the learning data can be generated.

As described above, according to the artificial intelligence model platform 100 of the present invention, by providing a platform environment that enables to create an artificial intelligence model for security control based on the UI without any programming, the security control technology Even unfamiliar general users can create artificial intelligence models suitable for their purposes and requirements for security control.

Furthermore, in the artificial intelligence model platform 100 of the present invention, the performance management module 160 utilizes test data managed in the output / file storage in the data output module 140, of the generated artificial intelligence model. Test accuracy.

The performance management module 160 is for managing the artificial intelligence model generated by the model generation module 150, 'who' 'when' 'some data' 'some field' 'some sampling method' 'some normalization method 'Records and manages performance information on the system (file storage), such as' what model' the artificial intelligence model was created for, and how much performance (correct answer rate) the created artificial intelligence model has.

In addition, the performance management module 160 can compare conditions and performance for model generation at a glance based on such performance information management, so that it is easy to grasp the correlation between conditions and performance.

In the present invention, the accuracy (performance) test of the artificial intelligence model generated in the platform environment of the present invention is provided by providing a platform environment that allows an ordinary user who is not familiar with security control technology to generate an artificial intelligence model. It may be necessary.

Specifically, the performance management module 160 utilizes test data (security events that know the actual result of detection and detection of malicious or false positives) managed in the output / file storage in the data output module 140, Test the accuracy of the AI model created above.

For example, the performance management module 160 uses the test data to test the artificial intelligence model generated above, and the accuracy of the model (performance) ) That is, it can be output as a test result.

In order to create an artificial intelligence model, which features are used and which normalization method is applied have a great influence on model performance (accuracy).

However, it will be difficult for a general user who is unfamiliar with people, especially security control technology, to combine / set feature information that can provide optimal performance in generating an artificial intelligence model desired by the user.

Thus, in the present invention, the feature extraction module 120 recommends a change to the feature information (Feature) to increase the accuracy of the above-described artificial intelligence model, based on the accuracy test result of the performance management module 160 Can be.

It will also be difficult for people, especially ordinary users, unfamiliar with security control technology, to know and set up a normalization method that can achieve optimal performance in generating the artificial intelligence model they want.

In addition, in the present invention, the normalization module 130 may recommend changing the normalization method for normalization to increase the accuracy of the artificial intelligence model.

Hereinafter, a technique for recommending a feature change to increase the accuracy of the artificial intelligence model will be described with reference to FIG. 3, and specifically, a feature information recommendation device for realizing the technique.

3 is a block diagram of a feature information recommendation apparatus according to an embodiment of the present invention.

As shown in FIG. 3, the feature information recommendation device 200 of the present invention includes a model performance confirmation unit 210, a combination performance confirmation unit 220, and a recommendation unit 230.

All or at least a part of the configuration of the feature information recommendation device 200 may be implemented in the form of a hardware module or a software module, or a combination of a hardware module and a software module.

Here, the software module may be understood as, for example, an instruction executed by a processor that controls an operation within the feature information recommendation apparatus 200, and these instructions may include a form mounted in the memory in the feature information recommendation apparatus 200. Will have.

After all, the feature information recommendation apparatus 200 according to an embodiment of the present invention, through the above-described configuration, the technology proposed in the present invention, that is, the technology for recommending feature information (Feature) change to increase the accuracy of the artificial intelligence model In the following, each configuration in the feature information recommendation device 200 for realizing this will be described in more detail.

The model performance checking unit 210 checks model performance with respect to the artificial intelligence model generated based on learning the preset feature information among all the feature information that can be set when the artificial intelligence model is generated.

That is, the model performance checking unit 210 checks the performance (accuracy) of the artificial intelligence model generated based on learning feature information set by the user.

For detailed description, hereinafter, it will be described on the assumption that the artificial intelligence model platform learning / generated feature information (hereinafter, user set feature information) set by the user in the AI model platform 100 of the present invention.

The model performance checking unit 210 checks the model performance with respect to the artificial intelligence model generated by learning the user set feature information in the artificial intelligence model platform 100 as described above.

For example, the model performance checking unit 210, with respect to the artificial intelligence model, test data output from the artificial intelligence model platform 100 (especially the data output module 140) of the present invention (sense / false classification and malignantness) Model performance (accuracy) can be tested / verified by utilizing the security event (which knows the actual result of detection).

Accordingly, the model performance checking unit 210 targets the artificial intelligence model generated in the artificial intelligence model platform 100 (especially the data output module 140) of the present invention, and tests the artificial intelligence model using test data, The matching ratio between the predicted result value and the known actual result value can be output as the model's accuracy (performance), that is, the test result.

The combination performance checking unit 220 sets a plurality of feature information combinations from the whole feature information, and checks the performance of the artificial intelligence model generated based on learning for each of the multiple feature information combinations.

Specifically, the combination performance checking unit 220 sets a variety of feature information combinations by setting various feature information combinations in addition to user-set feature information learned at the time of creation of the AI model, from all feature information that can be set when the AI model is generated. You can check the performance of the artificial intelligence model generated based on learning.

The recommendation unit 230, among a plurality of feature information combination-specific performances confirmed by the combination performance confirmation unit 220, model performances confirmed by the model performance confirmation unit 210, that is, performances of the artificial intelligence model generated based on the user setting this time Higher performance specific feature information combinations can be recommended.

Hereinafter, specific embodiments for recommending specific feature information combinations will be described.

According to one embodiment, the combination of the plurality of feature information set by the combination performance checking unit 220 is specified in the user set feature information learned when the artificial intelligence model is generated, except for the user set feature information in the whole feature information. It may be a combination of at least one piece of information sequentially added.

Hereinafter, user-set feature information (e.g., a, b, c, d, e) learned at the time of creation of this AI model among all feature information (e.g., a, b, c ..., z (n = 26)) , f (k = 6)). In this case, it is assumed that the artificial intelligence model performance (m _k ) checked by the model performance verification unit 210 is 85%.

Accordingly, the combination performance checking unit 220 includes user-set feature information (a, b, c, d, e) out of all feature information (n) in user-set feature information (a, b, c, d, e, f). A plurality of feature information combinations may be set by sequentially adding at least one of the specific information other than, f).

For example, the combination performance checking unit 220, the user-set feature information (a, b, c, d, e, f) set by the user, the user-set feature information (a, out of all the feature information n) Among the remaining specific information except b, c, d, e, f), 1 ~ (nk) feature information can be sequentially added to set a plurality of feature information combinations as follows.

a, b, c, d, e, f, g-> m _{(k + 1)} ^1- > 82%

a, b, c, d, e, f, h-> m _{(k + 1)} ^2- > 80%

...

a, b, c, d, e, f, g, h, i-> m _{(k + 3)} ^1- > 88%

...

a, b, c, d, e, f, ..., z-> m _(n) -> 85%

And, the combination performance checking unit 220 performs the performance of the artificial intelligence model generated based on learning for each of a plurality of feature information combinations as described above, 82%, 80%, ... 88%, ... 85%. Can be confirmed.

In this case, the recommendation unit 230, among a plurality of performances for each feature information combination, the top N having higher performance than the performance (m _k = 85%) of the artificial intelligence model generated based on this user setting (for example, 4) can be selected / recommended as a combination of specific feature information.

Of course, the top N are the number that can be specified / changed by the system administrator or user.

For another example, the combination performance checking unit 220 includes user-set feature information (a) among all feature information (n) in the user-set feature information (a, b, c, d, e, f) set by the user. A plurality of characteristic information combinations can be set as follows, by sequentially adding the remaining specific information one by one except for (b, c, d, e, f).

a, b, c, d, e, f, g-> m _{(k + 1)} ^1- > 82%

a, b, c, d, e, f, h-> m _{(k + 1)} ^2- > 80%

...

a, b, c, d, e, f, z-> m _{(k + 1)} ^{ζ + 1-} > 90%

Then, the combination performance checking unit 220 may check the performance, 82%, 80%, ... 90% of the artificial intelligence model generated based on learning for each combination of feature information as described above.

In this case, the recommendation unit 230, among a plurality of performances for each feature information combination, the top N having higher performance than the performance (m _k = 85%) of the artificial intelligence model generated based on this user setting (for example, 3) can be selected / recommended as a combination of specific feature information.

Meanwhile, according to another embodiment, the predetermined specific information used when the artificial intelligence model is generated may be full feature information (k = n = 26).

In this case, the combination performance checking unit 220 generates based on learning for each single feature information in the preset feature information, that is, all feature information (eg, a, b, c ..., z (n = 26)). The performance of a single feature information comparison process may be performed to check the performance of the artificial intelligence model and to determine whether the maximum performance (Max (m ₁ )) of the performance of each single feature information is higher than the model performance (m ₂₆ ).

Combination performance checking unit 220, the maximum performance of a single feature information (e.g., c) than the model performance (m ₂₆ ) of the predetermined feature information (a, b, c ..., z (n = 26)) When Max (m ₁ )) is high, the maximum performance single feature information (c) is reset to feature information, and the remaining specific information except feature information (c) from the whole feature information (n) in the feature information (c) By adding one by one, it is possible to perform a combination setting process of setting a plurality of combinations of feature information.

In this case, the combination performance checking unit 220 may check the performance for each combination of the plurality of feature information as described above.

c, a-> m ₂ ^1- > 81%

c, b-> m ₂ ^2- > 90.5%

...

c, z-> m ₂ ^25- > 85%

The combination performance checking unit 220 resets and resets each combination of feature information having a performance higher than the model performance (m ₁ ) of the feature information (c) that is reset among a plurality of feature information combinations, and resets the feature information. For each feature information, a reset process may be performed so that the combination setting process is repeatedly performed.

That is, the combination performance checking unit 220 deletes feature information combinations having a performance equal to or lower than the model performance (m ₁ ) of the feature information (c) among a plurality of feature information combinations, and performs model performance of the feature information (c). (m ₁ ) Only the combination of feature information with higher performance is left as follows, and each of them is reset to feature information to reset the combination setting process repeatedly for each feature information reset as shown in Table 2 below. You can carry out the process.

c, l-> m ₂ ^12- > 92.5%

c, m-> m ₂ ^13- > 93%

c, n-> m ₂ ^14- > 94%

The combination performance checking unit 220 repeats the above-described combination setting process and resetting process, and among a plurality of feature information combinations, there is a feature information combination having a higher performance than the artificial intelligence model generated based on the previous feature information. If not, the process of selecting the previous feature information as a specific feature information combination and passing it to the recommender 230 is performed.

In this case, the recommendation unit 230 has higher performance than the performance of the artificial intelligence model generated by using the preset feature information from the feature information transmitted from the combination performance checking unit 220 among performances of a plurality of feature information combinations. It can be recommended as a combination of specific feature information.

On the other hand, the combination performance checking unit 220, the maximum performance (Max (m (m) of a single feature information) than the model performance (m ₂₆ ) of the predetermined feature information (a, b, c ..., z (n = 26)) _{If 1} )) is not high, a combination setting process of setting a plurality of feature information combinations by excluding one specific information from feature information (a, b, c ..., z (n = 26)) It can be done.

b, c, d ~ z-> m ₂₅ ^1- > 96%

a, c, d ~ z-> m ₂₅ ^2- > 95.6%

...

a, b, c ~ y-> m ₂₅ ^25- > 90%

The combination performance checking unit 220 resets each combination of feature information having higher performance than the model performance (m ₂₆ ) among a plurality of feature information combinations as feature information, and the combination setting process is performed for each re-set feature information. A reset process may be performed to be repeatedly performed.

That is, the combination performance checking unit 220 deletes feature information combinations having a performance lower than or equal to model performance (m ₂₆ ) among a plurality of feature information combinations, and combinations of feature information having higher performance than model performance (m ₂₆ ). It is possible to perform a reset process in which the combination setting process is repeatedly performed for each of the characteristic information that is reset by resetting each of them as characteristic information, leaving only the following as follows.

b, c, d ~ z-> m ₂₅ ^1- > 96%

a, c, d ~ z-> m ₂₅ ^2- > 95.6%

a, b, d ~ y-> m ₂₅ ^3- > 96%

As described above, according to the present invention, in the environment provided by the artificial intelligence model platform 100, recommending an optimal feature having optimal performance (accuracy) to a user generating an artificial intelligence model for security control based on UI. / By making it applicable, even an average user who is not familiar with security control technology can create an optimal AI model for security control.

Hereinafter, with reference to FIG. 4, a technique for recommending a normalization method change to increase the accuracy of an artificial intelligence model, and specifically, a normalization method recommendation apparatus for realizing the technique will be described.

4 illustrates a configuration of a normalization method recommendation apparatus according to an embodiment of the present invention.

As shown in FIG. 4, the normalization method recommendation apparatus 300 of the present invention includes an attribute confirmation unit 310, a determination unit 320, and a recommendation unit 330.

The whole or at least part of the configuration of the normalization method recommendation device 300 may be implemented in the form of a hardware module or a software module, or a combination of a hardware module and a software module.

Here, the software module may be understood as, for example, an instruction executed by a processor that controls an operation within the normalization method recommendation apparatus 300, and these instructions may include a form mounted in the memory in the normalization method recommendation apparatus 300. Will have.

In the end, the normalization method recommendation apparatus 300 according to an embodiment of the present invention realizes the technique proposed in the present invention, that is, the technique of recommending the normalization method change to increase the accuracy of the artificial intelligence model through the above-described configuration, Hereinafter, each configuration in the normalization method recommendation apparatus 300 for realizing this will be described in more detail.

The attribute checking unit 310 checks the attribute of feature information used for learning when the artificial intelligence model is generated.

Here, the feature information used for learning when the AI model is generated may be feature information that is directly set by a user based on a UI among all the feature information that can be set when the AI model is generated, or a specific feature that is recommended among all feature information. The feature information combination may be feature information applied / set.

And, the attribute of the characteristic information can be largely divided into a number attribute and a category attribute.

That is, the attribute checking unit 310 may check whether the attribute of the feature information (direct setting or recommendation application) used for learning when the artificial intelligence model is generated is a numeric attribute, a category attribute, or a number and category combination attribute. .

The determination unit 320 determines a normalization method according to the attribute of the feature information checked by the attribute confirmation unit 310 among all the settable normalization methods.

Specifically, before determining the normalization method according to the attribute of the feature information, the determination unit 320 determines whether the same normalization method is applied to all the feature information fields or the normalization method for each field in the whole feature information field. It can be distinguished first whether it is applied.

When only numeric and / or category data exists in the entire feature information field (including a single feature case), the determination unit 320 may classify that the same normalization method is applied to the entire feature information field.

In this case, the determining unit 320 determines the first normalization method according to the entire numeric pattern of the feature information when the feature information attribute is a numeric attribute, and when the feature information attribute is a category attribute, the feature information is the whole of the feature information. If a second normalization method for expressing as a non-zero characteristic value is determined only at a location designated for each category of feature information in a vector defined by the number of categories, and if the attribute of the feature information is a number and category combination attribute, the second The normalization scheme and the first normalization scheme can be determined.

Specifically, the first normalization method includes a standard score normalization method, a mean normalization normalization method, and a feature scaling normalization method according to a predefined priority (see Equations 1, 2, and 3).

The determination unit 320 classifies the attribute of the feature information as a numeric attribute when only numeric data exists in the entire feature information field, and determines the first normalization method according to the whole numeric pattern of the feature information.

At this time, the determining unit 320 determines the standard score normalization method, the mean normalization normalization method, and the feature scaling normalization method according to the priority among the first normalization methods, but the standard deviation and normalization of the entire numeric pattern of the feature information Based on the existence of the upper / lower limit of the scaling range, the normalization method having the highest priority applicable among the first normalization methods may be determined.

In addition, when only category data exists in the entire feature information field, the determination unit 320 classifies the attribute of the feature information as a category attribute, and in this case, the feature information in a vector defined as the total number of categories of the feature information. A second normalization method that expresses a non-zero characteristic value only at a location designated by each category may be determined.

In order to generate an artificial intelligence model by applying an artificial intelligence algorithm (for example, machine learning) to training data, the data must be converted into numerical data that can be understood by the machine. In the present invention, this conversion method (second normalization) Method) One Hot Encoding can be adopted.

Accordingly, when the attribute of the feature information is a category attribute, the determination unit 320 has a non-zero characteristic value (eg, 1) in a location designated for each category of feature information in a vector defined as the total number of categories of the feature information. ) To determine the second normalization method _One Hot Encoding.

The second normalization method _One Hot Encoding, briefly, assumes that feature information has a category attribute of fruit, and that apples, pears, and persimmons (expressed as a three-dimensional vector because there are three kinds of fruits) are the total number of categories.

In this case, each feature information having apple, pear, and persimmon as data may be expressed as follows according to the second normalization method _One Hot Encoding.

Apple = {1, 0, 0}

Times = {0, 1, 0}

Persimmon = {0, 0, 1}

In addition, when the number and category data are present in the entire feature information field, the determination unit 320 classifies the attribute of the feature information as a numeric and category combination attribute, and in this case, the second normalization method and the first normalization method described above. Can decide.

That is, when the attribute of the feature information is a combination of numbers and categories, the determination unit 320 first applies the second normalization method _One Hot Encoding described above to the data of the category attribute in the feature information, and then the whole of the feature information. The second normalization method and the first normalization method may be determined in order to determine the highest priority normalization method applicable among the first normalization methods based on the existence of the upper and lower limit of the standard deviation and the normalization scaling range for the numeric pattern. .

On the other hand, if the feature information is a composite feature (one feature that can be extracted using aggregation and statistical techniques between multiple security events), the feature information is applied to the normalization method for each field in the entire feature information field. Can be distinguished.

In this case, the determination unit 320 may determine a normalization method having the highest priority that can be applied among a means normalization method and a feature scaling normalization method for a field of attribute type attribute in the feature information.

In addition, the determination unit 320 may determine a normalization method having the highest priority that is applicable among a means normalization method and a feature scaling normalization method for a field of a number attribute whose attribute is in the feature information.

In addition, the determining unit 320 may determine whether to normalize the normalization method for the attribute attribute field in the attribute information and exclude it from the normalization target, or determine the standard score normalization method.

In addition, the determination unit 320 may determine that the normalization method is not determined and excluded from the normalization target for the field of the attribute presence or absence (for example, presence / absence of an operation result value) in the feature information.

The recommendation unit 330 recommends the normalization method determined by the determination unit 320.

As described above, according to the present invention, in the environment provided by the AI model platform 100, the recommendation / applying the optimal normalization method with optimal performance (accuracy) to the user generating the AI model for security control based on the UI By doing so, even an average user who is not familiar with security control technology can create an optimal artificial intelligence model for security control.

As described above, according to the present invention, an artificial intelligence model platform that enables the creation of an artificial intelligence model for security control is implemented, but in particular, feature information and normalization methods directly related to the performance of the artificial intelligence model are optimally recommended / By making it possible to apply, it is possible to implement an artificial intelligence model platform that allows an ordinary user who is not familiar with security control technology to generate an optimal AI model for security control.

Hereinafter, an artificial intelligence model platform operating method according to an embodiment of the present invention will be described with reference to FIG. 5.

The artificial intelligence model platform 100 of the present invention periodically collects the newly generated source security data from the big data integrated storage storage (S10).

The artificial intelligence model platform 100 of the present invention collects / artificial intelligence functions through a UI according to the operation of a system administrator or a general user (hereinafter referred to as a user) who wants to create an artificial intelligence model for security control. It receives various related settings and stores / manages them as setting information (S20).

Then, the artificial intelligence model platform 100 of the present invention collects security events to be used as learning / test data based on a specific search condition, that is, a specific search condition previously set by the user from the original security data (S30).

The AI model platform 100 of the present invention extracts pre-set feature information for the security event collected in step S30, that is, pre-set feature information by the user (S40).

Then, the AI model platform 100 of the present invention performs normalization preset by the user on the extracted feature information of the security event (S50).

In the artificial intelligence model platform 100 of the present invention, the above three normalization methods are provided to allow a user to pre-set.

At this time, the artificial intelligence model platform 100 of the present invention, since the normalization scheme set by the user may not be optimal, it may recommend the optimal normalization scheme to increase the accuracy of the artificial intelligence model (S50).

The detailed description of the normalization method recommendation will be specifically described in FIG. 7 to be described later.

The AI model platform 100 of the present invention extracts training data or test data from a security event in which normalization of specific information is completed, based on a given condition, that is, a predetermined (given) condition by the user (S60).

Specifically, the artificial intelligence model platform 100 of the present invention, to output a security event that has been normalized specific information, the screen or file according to the value, order, format, learning / test data ratio, file division method, etc. do.

Then, the artificial intelligence model platform 100 of the present invention applies an artificial intelligence algorithm to the learning data to generate an artificial intelligence model for security control (S70).

That is, the artificial intelligence model platform 100 of the present invention may apply an artificial intelligence algorithm to learning data to generate an artificial intelligence model for security control, for example, an artificial intelligence model of a function required by a user.

For example, the artificial intelligence model platform 100 of the present invention may generate an artificial intelligence detection model for detecting whether a security event is malicious or not according to a user's request, and to classify the spying / falsification of the security event. You can also create an artificial intelligence classification model.

Specifically, the AI model platform 100 of the present invention, based on the learning data managed in the output / file storage in step S60, AI algorithms, such as machine learning (eg, Deep Learning) algorithms previously selected by the user Accordingly, an artificial intelligence model for security control can be generated.

For example, the artificial intelligence model platform 100 of the present invention is a learning loss function (Loss) indicating a deviation between a predicted result and an actual result through a model in a machine learning technique based on computation of backward propagation. function), it is possible to generate an artificial intelligence model in which the deviation of the loss function is zero based on the learning data.

Further, the artificial intelligence model platform 100 of the present invention utilizes test data (security events that know the actual result of detection and detection of malicious or false positives) managed in the output / file storage in step S60, The accuracy of the artificial intelligence model generated above is tested (S80).

For example, the artificial intelligence model platform 100 of the present invention uses the test data to test the artificial intelligence model generated above, and model the matching ratio between the predicted result value and the known actual result value through the model. The accuracy (performance) of ie can be output as a test result.

Thus, the AI model platform 100 of the present invention, 'who', 'when', 'some data', 'some field', 'some sampling method', 'some normalization method' and 'some model' using the 'AI model' Performance information such as whether or not the generated artificial intelligence model has a certain performance (correct answer rate) can be recorded and managed in a system (file storage).

In addition, the artificial intelligence model platform 100 of the present invention, based on such performance information management, can compare conditions and performance for model generation at a glance so that it is easy to grasp the correlation between conditions and performance.

At this time, the AI model platform 100 of the present invention may recommend a change to the feature information (Feature) to increase the accuracy of the generated AI model based on the accuracy test result of step S80 ( S90, S100).

That is, the AI model platform 100 of the present invention has a combination of other feature information capable of improving the accuracy of the AI model, compared to the feature information (hereinafter, user set feature information) used for learning when the AI model is generated. If there is (S90 Yes), this is the recommended method (S100).

Hereinafter, a computer program executed in the hardware (recommended device) of the present invention will be described with reference to FIG. 6, which is referred to as an operation method of the feature information recommendation device 200 for convenience of description. I will explain.

According to the operating method of the computer program, that is, the feature information recommendation apparatus 200 of the present invention, the performance (accuracy) of the artificial intelligence model generated based on the feature information learning set by the user is checked (S110).

The operation method of the feature information recommendation apparatus 200 according to the present invention checks model performance with respect to the artificial intelligence model generated by learning user-set feature information in the artificial intelligence model platform 100 as described above (S110). .

For example, the operation method of the feature information recommendation device 200 according to the present invention, for the artificial intelligence model, the test data output from the artificial intelligence model platform (100, especially the data output module 140) of the present invention ( Model performance (accuracy) can be tested / confirmed by utilizing the security event (which knows the actual result of the detection of false positives / false positives and malicious detection).

Accordingly, the operation method of the feature information recommendation device 200 according to the present invention is based on the artificial intelligence model generated by the artificial intelligence model platform 100 (especially the data output module 140) of the present invention, and utilizes test data. By testing the artificial intelligence model, it is possible to output the accuracy (performance) of the model, that is, the test result, as the ratio of the predicted result through the model and the known actual result.

The operation method of the feature information recommendation apparatus 200 according to the present invention sets a plurality of feature information combinations from the whole feature information, and checks the performance of the AI model generated based on learning for each combination of the feature information ( S120, S130).

Specifically, the operation method of the feature information recommendation apparatus 200 according to the present invention sets a combination of various feature information in addition to user-set feature information learned at the time of creation of the AI model, from all feature information that can be set when the AI model is generated Thus, it is possible to check the performance of the artificial intelligence model generated based on learning for each combination of feature information.

Hereinafter, user-set feature information (e.g., a, b, c, d, e) learned at the time of creation of this AI model among all feature information (e.g., a, b, c ..., z (n = 26)) , f (k = 6)). In this case, it is assumed that the verified artificial intelligence model performance (m _k ) is 85%.

Accordingly, the operation method of the feature information recommendation apparatus 200 according to the present invention includes user-set feature information (a, out of all feature information (n) in user-set feature information (a, b, c, d, e, f). A plurality of feature information combinations may be set by sequentially adding at least one of the specific information other than b, c, d, e, f) (S120).

For example, the operation method of the feature information recommendation apparatus 200 according to the present invention includes the user set feature information (a, b, c, d, e, f) set by the user, among the whole feature information (n) One to (nk) feature information among the remaining specific information except for the user-set feature information (a, b, c, d, e, f) can be sequentially added to set a plurality of feature information combinations as follows.

a, b, c, d, e, f, g-> m _{(k + 1)} ^1- > 82%

a, b, c, d, e, f, h-> m _{(k + 1)} ^2- > 80%

...

a, b, c, d, e, f, g, h, i-> m _{(k + 3)} ^1- > 88%

...

a, b, c, d, e, f, ..., z-> m _(n) -> 85%

And, the operation method of the feature information recommendation apparatus 200 according to the present invention is the performance of the artificial intelligence model generated based on learning for each combination of multiple feature information as described above, 82%, 80%, ... 88% , ... 85% can be confirmed (S130).

In this case, the operation method of the feature information recommendation device 200 according to the present invention is higher than the performance (m _k = 85%) of the artificial intelligence model generated based on the current user setting among performances of a plurality of feature information combinations. The top N (for example, 4) having performance may be selected / recommended as a specific feature information combination (S140 Yes, S150).

For another example, the operating method of the feature information recommendation apparatus 200 according to the present invention includes user-specified feature information (a, b, c, d, e, f), and overall feature information (n) Among the user-specific feature information (a, b, c, d, e, f), the remaining specific information may be sequentially added one by one to set a plurality of feature information combinations as follows (S120).

a, b, c, d, e, f, g-> m _{(k + 1)} ^1- > 82%

a, b, c, d, e, f, h-> m _{(k + 1)} ^2- > 80%

...

a, b, c, d, e, f, z-> m _{(k + 1)} ^{ζ + 1-} > 90%

And, the operation method of the feature information recommendation device 200 according to the present invention is the performance of the artificial intelligence model generated based on learning for each combination of a plurality of feature information, as described above, 82%, 80%, ... 90% It can be confirmed (S130).

In this case, the operation method of the feature information recommendation apparatus 200 according to the present invention is higher than the performance (m _k = 85%) of the artificial intelligence model generated based on the current user setting among performances of a plurality of feature information combinations. The top N (for example, three) having performance may be selected / recommended as a specific feature information combination (S140 Yes, S150).

Hereinafter, a computer program performed in the hardware (recommended device) of the present invention will be described with reference to FIG. 7, that is, a computer program for recommending a normalization method, but for convenience of description, referred to as an operation method of the normalization method recommendation apparatus 300 I will explain.

According to the method of operation of the computer program, that is, the normalization method recommendation apparatus 300 of the present invention, when generating an artificial intelligence model, the property of feature information used for learning is checked (S200).

That is, the operation method of the normalization method recommendation apparatus 300 according to the present invention is whether the attribute of feature information (direct setting or recommendation application) used for learning when generating an artificial intelligence model is a numeric attribute or a category attribute or a number and It can be checked whether the category is a combination attribute (S200).

The operation method of the normalization method recommendation apparatus 300 according to the present invention determines a normalization method according to the attribute of the feature information identified in step S200 among all the settable normalization methods.

Specifically, the method of operation of the apparatus 300 for recommending a normalization method according to the present invention, prior to determining the normalization method according to the attribute of the feature information, is the same normalization method applied to all of the feature information fields or this feature In the entire information field, whether a normalization method is applied for each field may be distinguished first (S210).

In the operation method of the normalization method recommendation apparatus 300 according to the present invention, when only numeric and / or category data exists in the entire feature information field (including a single feature case), the same normalization method is applied to the entire feature information field. It can be classified as being (S210 Yes).

In this case, in the operation method of the normalization method recommendation apparatus 300 according to the present invention, when the attribute of the feature information is a numeric attribute, the first normalization method according to the entire number pattern of the feature information is determined, and the attribute of the feature information is In the case of a category attribute, a second normalization method for expressing as a non-zero characteristic value is determined only at a designated location for each category of feature information in a vector defined as the total number of categories of the feature information, and the attribute of the feature information is a number and In the case of a category combination attribute, the second normalization method and the first normalization method may be determined (S220).

The operation method of the normalization method recommendation apparatus 300 according to the present invention is to classify the attribute of the feature information as a numeric attribute when only numeric data exists in the entire feature information field, and in this case, the method according to the whole number pattern of the feature information 1 Determine the normalization method.

At this time, the operation method of the normalization method recommendation apparatus 300 according to the present invention is determined in the order of the standard score normalization method, the mean normalization normalization method, and the feature scaling normalization method according to the priority of the first normalization method. Based on the existence of the standard deviation and the upper / lower limit of the normalized scaling range for the entire numeric pattern, the highest normalized normalization method applicable among the first normalization methods may be determined.

In addition, in the operation method of the normalization method recommendation apparatus 300 according to the present invention, when only category data exists in the entire feature information field, the attribute of the feature information is classified as a category attribute, and in this case, the total number of categories of the feature information A second normalization method _One Hot Encoding that expresses a non-zero characteristic value (eg, 1) only at a designated location for each category of feature information in a defined vector can be determined.

In addition, in the operation method of the normalization method recommendation apparatus 300 according to the present invention, when the number and category data exist in the entire feature information field, the attribute of the feature information is divided into a number and category combination attribute. The second normalization method and the first normalization method may be determined.

That is, in the operation method of the normalization method recommendation apparatus 300 according to the present invention, when the attribute of the feature information is a number and category combination attribute, the second normalization method of the above-mentioned second normalization method_One Hot After Encoding is applied, the second normalization method is used to determine the highest normalization method applicable among the first normalization methods based on whether there is a standard deviation and a normalization scaling range upper / lower limit for the entire numeric pattern of the feature information. And a first normalization method.

On the other hand, in the operation method of the normalization method recommendation apparatus 300 according to the present invention, when the feature information is a composite feature (a single feature that can be extracted by using statistical and statistical methods between multiple security events), the entire feature information of this time It can be classified as being applied to the normalization method for each field in the field (S210 No).

In this case, the operation method of the normalization method recommendation apparatus 300 according to the present invention includes a normalization method having the highest priority, which is applicable among the means of the normalization method and the feature scaling normalization method for the field of attribute type attribute in the feature information. It can be determined (S230).

In addition, the operation method of the normalization method recommendation apparatus 300 according to the present invention determines the normalization method having the highest priority among the normalization method, the mean normalization normalization method, and the feature scaling normalization method for the field of the attribute whose number of attributes is the feature information. It can be (S230).

In addition, in the operation method of the normalization method recommendation apparatus 300 according to the present invention, the attribute in the attribute information determines the normalization method for the field of the ratio attribute and decides to exclude it from the normalization target or determines the standard score normalization method. It can be (S230).

In addition, in the operation method of the normalization method recommendation apparatus 300 according to the present invention, the normalization method is not determined and the normalization target is determined for the field of the attribute presence or absence (for example, presence / absence of an operation result value) in the feature information. It may be decided to exclude (S230).

The operation method of the normalization method recommendation apparatus 300 according to the present invention recommends the normalization method determined in step S220 or step S230 (S240).

The artificial intelligence model platform operating method according to an embodiment of the present invention described above may be implemented in a form of program instructions that can be executed through various computer means and may be recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

The present invention has been described in detail with reference to preferred embodiments, but the present invention is not limited to the above-described embodiments, and the technical field to which the present invention pertains without departing from the gist of the present invention claimed in the claims below. Anyone with ordinary knowledge in the technical idea of the present invention extends to the extent that various modifications or modifications are possible.

Claims

A data collection module that collects security events to be used as learning / test data according to specific search conditions from the source security data;

A feature extraction module that extracts preset feature information for the collected security event;

A normalization module that performs preset normalization on the extracted feature information of the security event;

A data output module that extracts learning data or test data from the security event where the specific information normalization is completed according to a given condition; And

An artificial intelligence model platform comprising a model generation module that generates an artificial intelligence model for security control by applying an artificial intelligence algorithm to the learning data.
According to claim 1,

The artificial intelligence model platform further comprising a performance management module for testing the accuracy of the artificial intelligence model using the test data.
According to claim 1,

Further comprising a UI module providing a user interface (UI) for setting at least one of the specific search condition of the data collection module, the feature information of the feature extraction module, the normalization method of the normalization module, and the condition of the data output module. AI model platform, characterized in that.
According to claim 1,

The data collection module,

If the total number of collections exceeds the maximum number of collections that can be performed simultaneously, the collections exceeding the maximum number of collections among the total number of collections are stored in a queue and sequentially executed.

In the case of a collection case proceeding after being stored in the queue, the artificial intelligence model platform characterized in that the security event is collected only for data prior to the occurrence point of the collection case in the source security data.
According to claim 2,

The feature extraction module,

Based on the result of the accuracy test of the performance management module, the AI model platform, characterized by recommending a change to the feature information to increase the accuracy of the AI model.
According to claim 1,

The normalization module,

AI model platform, characterized by recommending a normalization method change for the normalization to increase the accuracy of the AI model.
A model performance checking unit that checks model performance with respect to an artificial intelligence model generated based on learning predetermined feature information among all feature information that can be set when generating the artificial intelligence model;

A combination performance checking unit configured to set a plurality of feature information combinations from the entire feature information to check the performance of the artificial intelligence model generated based on learning for each of the plurality of feature information combinations; And

And a recommendation unit for recommending a specific feature information combination having a higher performance than the model performance confirmed by the model performance checking unit among performances for each of the plurality of feature information combinations.
The method of claim 7,

The combination of the plurality of feature information,

It is a combination that is sequentially added to at least one of the remaining specific information excluding the preset characteristic information from the entire characteristic information to the preset characteristic information,

The specific feature information combination,

Feature information recommendation device, characterized in that the top N having a higher performance than the model performance among the plurality of feature information combinations.
The method of claim 7,

The predetermined specific information is the entire feature information,

The combination performance confirmation unit,

A single feature information performance comparison process to determine whether the maximum performance of the AI model generated based on learning for each single feature information in the entire feature information is higher than the model performance,

When the maximum performance is higher than the model performance, the single feature information of the maximum performance is reset to the feature information, and the feature information is sequentially added to the feature information one by one of the remaining specific information excluding the preset feature information. A combination setting process for setting the combination of the plurality of feature information,

Each of the combinations of feature information having higher performance than the model performance of the re-set feature information among the plurality of feature information combinations is reset as feature information so that the combination setting process is repeatedly performed for each re-set feature information Reset process,

If there is no feature information combination having a higher performance than the model performance among the plurality of feature information combinations, feature information characterized by performing a process of delivering the immediately preceding feature information as the specific feature information combination to the recommendation unit Recommended device.
The method of claim 7,

The predetermined specific information is the entire feature information,

The combination performance confirmation unit,

A single feature information performance comparison process to determine whether the maximum performance of the AI model generated based on learning for each single feature information in the entire feature information is higher than the model performance,

If the maximum performance is not higher than the model performance, a combination setting process of setting the combination of the plurality of feature information excluding one specific information different from the feature information,

A reset process for resetting each combination of feature information having a performance higher than the model performance among the plurality of feature information combinations as feature information, so that the combination setting process is repeatedly performed for each re-set feature information,

If there is no feature information combination having a higher performance than the model performance among the plurality of feature information combinations, feature information characterized by performing a process of delivering the immediately preceding feature information as the specific feature information combination to the recommendation unit Recommended device.
An attribute confirmation unit that checks attributes of feature information used for learning when creating an artificial intelligence model;

Determining unit for determining a normalization method according to the attribute of the feature information, from among all the settable normalization method; And

And a recommendation unit for recommending the determined normalization method.
The method of claim 11,

The determining unit,

If the same normalization method is applied to all of the feature information fields,

When the attribute of the feature information is a numeric attribute, a first normalization method according to the entire number pattern of the feature information is determined,

When the attribute of the feature information is a category attribute, a second normalization method for expressing as a non-zero characteristic value is determined only at a designated location for each category of feature information in a vector defined as the total number of categories of the feature information,

A device for recommending a normalization method, wherein the second normalization method and the first normalization method are determined when the attribute of the feature information is a combination of numbers and categories.
The method of claim 12,

The first normalization method includes a standard score normalization method, a mean normalization normalization method, and a feature scaling normalization method according to a predefined priority,

The determining unit,

A normalization method recommendation apparatus characterized by determining a normalization method having the highest priority applicable among the first normalization methods based on whether there is a standard deviation and a normalization scaling range upper / lower limit for the entire numeric pattern of the feature information.
The method of claim 11,

The determining unit,

If the normalization method is applied to each field in the entire feature information field,

In the feature information, for the field of the attribute type attribute, the normalization method having the highest priority applicable to the normalization method of the normalization method or the feature scaling normalization method is determined,

In the feature information, for the field of the number attribute of attribute, the normalization method having the highest priority applicable among the means normalization method and the feature scaling normalization method is determined,

In the feature information, a normalization method is not determined for a field whose attribute is a ratio attribute and is excluded from normalization, or a standard score normalization method is determined,

A device for recommending a normalization method, characterized in that a normalization method is not determined and excluded from the normalization target for a field of an attribute in the feature information.
A data collection step of collecting security events to be used as learning / test data according to specific search conditions from the source security data;

A feature extraction step of extracting predetermined feature information for the collected security event;

A normalization step of performing preset normalization on the extracted feature information of the security event;

A data output step of extracting training data or test data according to a given condition from the security event in which the specific information normalization is completed; And

A method of operating an artificial intelligence model platform comprising applying an artificial intelligence algorithm to the learning data and model-generating an artificial intelligence model for security control.
The method of claim 15,

Using the test data, the artificial intelligence model platform operating method further comprises a performance management step of testing the accuracy of the artificial intelligence model.
The method of claim 15,

The method further includes providing a user interface (UI) for setting at least one of a specific search condition of the data collection step, feature information of the feature extraction module, normalization method of the normalization module, and condition of the data output module. Artificial intelligence model platform operating method, characterized in that.
The method of claim 15,

The data collection step,

If the total number of collections exceeds the maximum number of collections that can be performed simultaneously, the collections exceeding the maximum number of collections among the total number of collections are stored in a queue and sequentially executed.

In the case of a collection case proceeding after being stored in the queue, the artificial intelligence model platform operating method characterized in that the security event is collected only for data prior to the occurrence point of the collection case in the source security data.
The method of claim 16,

And recommending a change to the feature information to increase the accuracy of the AI model based on the accuracy test result of the performance management step.
The method of claim 15,

The normalization step,

The artificial intelligence model platform operating method characterized in that it is recommended to change the normalization method for the normalization to increase the accuracy of the artificial intelligence model.
A model performance verification step of confirming model performance with respect to the artificial intelligence model generated based on learning the predetermined feature information among all the feature information that can be set when generating the artificial intelligence model in combination with hardware;

A combination performance checking step of setting a combination of a plurality of feature information from the whole feature information, and confirming the performance of the artificial intelligence model generated based on learning for each of the plurality of feature information combinations; And

A computer program stored in a medium to perform a recommendation step of recommending a specific feature information combination having a higher performance than the model performance confirmed by the model performance checking unit among performances of the plurality of feature information combinations.
The method of claim 21,

The combination of the plurality of feature information,

It is a combination that is sequentially added to at least one of the remaining specific information excluding the preset characteristic information from the entire characteristic information to the preset characteristic information,

The specific feature information combination,

A computer program characterized by being the top N having a higher performance than the model performance among the combination of the plurality of feature information.
The method of claim 21,

The predetermined specific information is the entire feature information,

The combination performance check step,

A single feature information performance comparison process to determine whether the maximum performance of the AI model generated based on learning for each single feature information in the entire feature information is higher than the model performance,

When the maximum performance is higher than the model performance, the single feature information of the maximum performance is reset to the feature information, and the feature information is sequentially added to the feature information one by one of the remaining specific information excluding the preset feature information. A combination setting process for setting the combination of the plurality of feature information,

Each of the combinations of feature information having higher performance than the model performance of the re-set feature information among the plurality of feature information combinations is reset as feature information so that the combination setting process is repeatedly performed for each re-set feature information Reset process,

If there is no feature information combination having a higher performance than the model performance among the plurality of feature information combinations, a computer program characterized by performing a process of delivering the immediately preceding feature information as the specific feature information combination to the recommendation unit. .
The method of claim 21,

The predetermined specific information is the entire feature information,

The combination performance check step,

A single feature information performance comparison process to determine whether the maximum performance of the AI model generated based on learning for each single feature information in the entire feature information is higher than the model performance,

If the maximum performance is not higher than the model performance, a combination setting process of setting the combination of the plurality of feature information excluding one specific information different from the feature information,

A reset process for resetting each combination of feature information having a performance higher than the model performance among the plurality of feature information combinations as feature information, so that the combination setting process is repeatedly performed for each re-set feature information,

If there is no feature information combination having a higher performance than the model performance among the plurality of feature information combinations, a computer program characterized by performing a process of delivering the immediately preceding feature information as the specific feature information combination to the recommendation unit. .
In combination with hardware, an attribute checking step of checking the attribute of the feature information used for learning when creating the artificial intelligence model;

A determining step of determining a normalization method according to the attribute of the feature information from among all the settable normalization methods; And

A computer program stored in the medium to execute the recommendation step of recommending the determined normalization method.
The method of claim 25,

The determining step,

If the same normalization method is applied to all the feature information fields,

If the attribute of the feature information is a numeric attribute, a first normalization method according to the entire number pattern of the feature information is determined,

When the attribute of the feature information is a category attribute, a second normalization method for expressing as a non-zero characteristic value is determined only at a designated location for each category of feature information in a vector defined as the total number of categories of the feature information,

And the second normalization method and the first normalization method when the attribute of the characteristic information is a combination attribute of a number and a category.
The method of claim 26,

The first normalization method includes a standard score normalization method, a mean normalization normalization method, and a feature scaling normalization method according to a predefined priority,

The determining step,

A computer program characterized by determining a normalization method having the highest priority applicable among the first normalization methods, based on whether there is a standard deviation and a normalization scaling range upper / lower limit for the entire numeric pattern of the feature information.
The method of claim 25,

The determining step,

If the normalization method is applied to each field in the entire feature information field,

In the feature information, for the field of the attribute type attribute, the normalization method having the highest priority applicable to the normalization method of the normalization method or the feature scaling normalization method is determined,

In the feature information, for the field of the number attribute of attribute, the normalization method having the highest priority applicable among the means normalization method and the feature scaling normalization method is determined,

In the feature information, a normalization method is not determined for a field whose attribute is a ratio attribute and is excluded from normalization, or a standard score normalization method is determined,

A computer program characterized in that the normalization method is determined for the field of the attribute whether the attribute exists or not in the feature information and is excluded from the normalization target.