KR20230087232A - Method and system for analyzing performance of machine learning algorithm by using lid-ds dataset - Google Patents

Abstract

A method for analyzing performance of a machine learning algorithm according to the present invention is performed by an algorithm comparison system, comprising: performing preprocessing on cyber attack data; Dividing the preprocessed cyber attack data into training data and test data; and analyzing performance of machine learning algorithms using the divided training data and the test data.

Description

Method and system for analyzing machine learning algorithm performance using LID-DS data set

The present invention relates to a technique for analyzing the detection accuracy of a cyber attack, and more particularly, to a method and system for analyzing the performance of a machine learning algorithm using a LID-DS data set.

Today, as information and communication technology develops rapidly, the importance of security in IT infrastructure is gradually increasing, and at the same time, cyber attacks are becoming sophisticated and intelligently diversified like intelligent continuous attacks.

It is a very important issue to defend against increasingly sophisticated cyberattacks, but the speed of development of IDS (Intrusion Detection System) technology does not completely detect rapidly changing cyberattacks. Therefore, the data generated by the intrusion detection system is currently being used to defend against the above cyber attacks through HIDS data analysis.

Intrusion detection systems can be divided into two types: network-based NIDS and host-based HIDS. Unlike network-based intrusion detection systems, host-based intrusion detection systems have difficulties in monitoring the inside and outside of the system as a whole.

Accordingly, there is a problem in that there is a lot of lack of research, the intrusion detection system has insufficient defense measures against new cyber attacks and internal cyber attacks, and false alarms increase.

The host-based intrusion detection system method can be divided into two methods: misuse detection and anomaly detection. Since the misuse detection method detects cyberattacks based on signatures, it is effective to detect existing cyberattacks, but it is inadequate to detect new cyberattacks.

Contrary to the misuse detection method, the anomaly detection method detects all situations that are not defined as normal actions and behaviors as abnormal actions. In other words, unlike the misuse detection method, the anomaly detection method is suitable for detecting zero-data attacks, but it is difficult to apply it to machine learning because it requires a lot of data to determine normal and abnormal behavior or the data is too insufficient.

In the signature-based detection technology, the IDS detects data intrusion using a database containing attack signatures, so the detection rate is relatively excellent. However, the problem with the detection system is that it cannot detect new cyber attacks because there is no data on new cyber attacks.

Recently, host-based intrusion detection research uses machine learning technology, anomaly-based detection technology finds abnormal behavior, detects new attacks, and machine learning algorithms are used to detect anomalies.

Korean Patent Registration No. 10-2292968 (Announcement date: 2021. 08. 25.) Korean Patent Publication No. 10-2018-0085157 (Publication date: 2018. 07. 26.)

The present invention seeks to provide a method and system for analyzing the performance of a machine learning algorithm using a LID-DS data set.

According to one aspect, the present invention provides a method for analyzing the performance of a machine learning algorithm performed by an algorithm comparison system, comprising: performing preprocessing on cyber attack data; Dividing the preprocessed cyber attack data into training data and test data; and analyzing performance of machine learning algorithms using the divided training data and the test data.

The cyber attack data of the present invention is a Leipzig Intrusion Detection-Data Set (LID-DS), and the LID-DS data is classified according to a plurality of cyber attack methods and is composed of a plurality of characteristics according to each cyber attack method. Including, the step of performing the preprocessing includes deleting argument features and missing values for the LID-DS data, removing colons for event_time features, and using LabelEncoder for event_direction and event_type. steps may be included.

The step of performing the preprocessing of the present invention is the step of integrating the processes used in each cyber attack into one process, and attaching labels to each process using LabelEncoder for the labels composed of the integrated process. can include

In the step of performing the preprocessing of the present invention, the minimum value is 0, the maximum value is 1, and the remaining values are 0 and 1 for each feature of the LID-DS data using Min-Max Normalization. It may include converting to a value in between.

The dividing step of the present invention may include dividing the training data and the test data to be used for the machine learning model at a preset ratio.

In the analyzing step of the present invention, the divided training data is input to each machine learning model, each machine learning model is trained using the divided training data, and each machine learning model is learned. and deriving intrusion detection accuracy for the divided test data using each of the learned machine learning models.

The analyzing step of the present invention may include evaluating the performance of each of the trained machine learning models using precision, recall, F1 score, and error rate.

According to another aspect, the present invention is a computer readable recording medium storing a computer program, wherein the computer program, when executed by a processor, performs preprocessing on cyber attack data; Dividing the preprocessed cyber attack data into training data and test data; and analyzing performance of machine learning algorithms using the divided training data and the test data.

The present invention, according to another aspect, is a computer program stored in a computer readable recording medium, wherein the computer program, when executed by a processor, performs preprocessing on cyber attack data; Dividing the preprocessed cyber attack data into training data and test data; and analyzing performance of machine learning algorithms using the divided training data and the test data.

According to another aspect, the present invention provides a system for analyzing the performance of a machine learning algorithm, comprising: a pre-processing unit performing pre-processing on cyber attack data; a data division unit dividing the preprocessed private cyber attack data into training data and test data; and a performance analyzer configured to analyze performance of machine learning algorithms using the divided training data and the test data.

The cyber attack data of the present invention is a Leipzig Intrusion Detection-Data Set (LID-DS), and the LID-DS data is classified according to a plurality of cyber attack methods and is composed of a plurality of characteristics according to each cyber attack method. The pre-processing unit may delete an argument feature and a missing value from the LID-DS data, remove a colon from event_time feature, and use LabelEncoder from event_direction and event_type.

The pre-processing unit of the present invention may integrate processes used in each cyber attack into one process, and attach labels to each process using a LabelEncoder for labels composed of the integrated processes.

The pre-processing unit of the present invention sets the minimum value to 0, the maximum value to 1, and the remaining values to values between 0 and 1 for each characteristic of the LID-DS data using Min-Max Normalization. can be converted

The data partitioning unit of the present invention may divide the training data and the test data to be used for the machine learning model at a predetermined ratio.

The performance analysis unit of the present invention receives the divided training data as input to each machine learning model, uses the divided training data to learn the respective machine learning model, and uses the divided training data to learn each machine learning model. The divided test data may be input, and intrusion detection accuracy for the divided test data may be derived using each of the learned machine learning models.

The performance analysis unit of the present invention may evaluate the performance of each of the learned machine learning models using precision, recall, F1 score, and error rate.

According to an embodiment of the present invention, the performance of the machine learning algorithm according to the attack type can be effectively evaluated in the LID-DS data set.

1 is a table for explaining a data set according to an embodiment of the present invention.
2 is a diagram illustrating an attack simulation procedure of an LID-DS data set according to an embodiment of the present invention.
3 is a diagram for explaining the structure of a LID-DS data set performance comparison model in an embodiment of the present invention.
4 is a block diagram for explaining the configuration of an algorithm comparison system according to an embodiment of the present invention.
5 is a flowchart illustrating a method of comparing machine learning algorithms in an algorithm comparison system according to an embodiment of the present invention.
6 is a diagram for explaining a data format stored in an LID-DS according to an embodiment of the present invention.
7 is a table for explaining data types and attack types according to an embodiment of the present invention.

First, the advantages and characteristics of the present invention, and methods for achieving them will become clear with reference to the embodiments described later in detail in conjunction with the accompanying drawings. Here, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments make the disclosure of the present invention complete, and conventional techniques in the technical field to which the present invention pertains. Since it is provided as an example so that those skilled in the art can clearly understand the scope of the invention, the technical scope of the present invention should be defined by the claims.

In addition, in describing the present invention below, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to intentions or customs of users, operators, etc., of course. Therefore, the definition should be made based on the technical idea described throughout this specification.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a table for explaining a data set according to an embodiment of the present invention.

In 2018, Leipzig University published the LID-DS data set for an anomaly detection study in host-based intrusion detection systems.

Unlike previously published data, the LID-DS data set consists of various features of the latest computer system and cyber attack methods and scenarios compared to currently published data sets.

Through this LID-DS data, it is possible to solve the part that was difficult to apply to machine learning due to the lack of data in existing data sets, and to more accurately detect and block new abnormal behaviors using machine learning methods. The false alarm rate, which is a problem, can be reduced.

The LID-DS data set contains various data related to system calls, software and various cyber attacks are recorded. As an example, the LID-DS data set is composed of a cyber attack method and several scenarios, as shown in the table of FIG. 1, and a process of generating and recording normal and abnormal data can be configured through the scenarios.

2 is a diagram illustrating an attack simulation procedure of an LID-DS data set according to an embodiment of the present invention.

Figure 2 shows the process of generating data using the cyber attack scenario of Figure 1. In order to record the system call trace as a result of the LID-DS data set, the attack target defines an initial state and returns to the initial state after each attack. To revert, it can be run within Docker 10 container virtualization software.

For the record, we use the LID-DS framework to start the Docker container hosting the attack target. Initialization tasks are then executed according to the scenario and simulation of normal operation begins.

After that, it waits for a short period of time before Sysdig is activated in order not to record the starting effect of the targeted software. In the case of recording an attack action, the attack starts after a certain amount of time, and after the recording is executed for the desired time, the recording is stopped by the control script. Also, it stops and removes normal operation and simulation of used Docker containers.

The ADFA data set is not suitable for anomaly detection testing because it contains only a set of system call IDs and does not include modern cyberattack patterns.

In the embodiment, LID-DS (Leipzig Intrusion Detection-Data Set) is used to include thread information, meta data, and buffer data that are lacking in previously used data sets, and to solve problems that have not been applied to machine learning due to lack of data. use the data Comparative studies on machine learning algorithms using LID-DS data can provide directions for host-based intrusion detection systems.

Referring to FIG. 6, it is a table showing the format of the LID-DS data file itself. Unlike ADFA, LID-DS data can include system call arguments, return values, high-precision timestamps, corresponding process names, and the contents of data buffers.

3 is a diagram for explaining the structure of a LID-DS data set performance comparison model in an embodiment of the present invention.

The algorithm comparison system may perform pre-processing on the LID-DS data set. In the LID-DS data set, data may be classified according to a plurality of cyber attack methods, and each cyber attack method may be composed of a plurality of characteristics.

Referring to FIG. 7, it is a table for explaining data types and attack types. 7 is a table showing that data is classified according to 10 cyber attack methods for the LID-DS data set, and each cyber attack method is composed of 7 characteristics. To use the data in machine learning algorithms, it can be organized in CSV format except for some data with large original size.

The algorithmic comparison system can delete argument features and missing values for all data, and remove colons (:) from event_time features. Event_direction and event_type can use LabelEncoder.

There are a total of 16 process categories, and the number of processes is different for each attack method. Accordingly, the processes used in each cyber attack method can be integrated into one process. As a result, labels composed of a total of 10 processes can be used by attaching a label to each process using LabelEncoder.

At this time, the reason why OneHotEncoder is not used is that event_type has 99 categories. That is, only LabelEncoder can be used because there is a risk that the Curse of Dimensionality may occur. Normalization uses Min-Max Normalization to solve the problem that the scale of the features of the data differs greatly, setting the minimum value to 0 and the maximum value to 1 for each feature, and other values to 0 and 1. It can be converted to a value between 1 and 1.

The algorithm comparison system may divide preprocessed LID-DS data into training data and test data. Training data can be used by combining 10 cyber attack method data into one CSV file. Training data and test data to be used for machine learning can be split at the commonly used 8:2 ratio using the train_test_split module.

However, even though the model is applied to the test data after learning the training data, overfitting may be found. Overfitting is a phenomenon in which the prediction rate drops significantly because the model has been trained to overfit. In order to prevent overfitting, model evaluation may be performed by cross-validation, including data divided by different ratios.

As a result, the model performance of a data set divided by a specific ratio (eg, 7:3 ratio) may be the highest. In this embodiment, an example in which training data and test data are divided and used at a ratio of 7:3 will be described.

The algorithm comparison system may analyze the performance of machine learning algorithms using the divided training data and test data. The algorithm comparison system can analyze the performance of six machine learning algorithms including Decision Tree, Naive Bayes, MLP, Logistic Regression, LSTM, and RNN. For example, experiments may be performed to analyze the performance of machine learning algorithms. The hyperparameters of the algorithm used in this experiment are shown in Table 1 below.

[Table 1]

As an example, cyber attack data pre-processed by the above-described LID-DS method may be used. An experiment comparing the intrusion detection accuracy of the LID-DS data set can be conducted through the data of 10 process categories labeled using LabelEncoder.

Performance evaluation of the learned model can use precision, recall, and F1 score. The reason is that it is inappropriate to evaluate data only with accuracy. In addition, in order to check the error rate, which is an important problem in security, values of a false alarm rate (FAR) and an error rate (ERR) may be checked.

For example, the algorithm used in the experiment can be largely experimented with four algorithms that learn without considering the sequence of data and two algorithms that learn in consideration of the sequence. It is possible to evaluate which machine learning algorithm can best detect continuous system calls according to the type of attack, which is the most characteristic part of the LID-DS data set.

According to the embodiment, in the case of LID-DS data, cyber attack data can be directly created, so the amount of data is large, so it is possible to solve the problem where machine learning cannot proceed due to insufficient data.

4 is a block diagram for explaining the configuration of an algorithm comparison system according to an embodiment, and FIG. 5 is a flowchart for explaining a method for comparing machine learning algorithms in an algorithm comparison system according to an embodiment.

The processor of the algorithm comparison system 100 may include a pre-processing unit 410, a data segmentation unit 420, and a performance analysis unit 430. Components of such a processor may be representations of different functions performed by the processor according to control instructions provided by program codes stored in the algorithm comparison system. The processor and components of the processor may control the algorithm comparison system to perform steps 510 to 530 included in the machine learning algorithm comparison method of FIG. 5 . In this case, the processor and components of the processor may be implemented to execute instructions according to the code of an operating system included in the memory and the code of at least one program.

The processor may load a program code stored in a program file for a machine learning algorithm comparison method into a memory. For example, when a program is executed in the algorithm comparison system, the processor may control the algorithm comparison system to load a program code from a program file into a memory under the control of an operating system.

At this time, each of the processor and the pre-processing unit 410, the data division unit 420, and the performance analysis unit 430 included in the processor executes a command of a corresponding part of the program code loaded into the memory to perform subsequent steps (510 to 510). 530) may be different functional representations of the processor.

In step 510, the pre-processing unit 410 may perform pre-processing on cyber attack data. The pre-processor 410 can delete argument features and missing values from LID-DS data, remove colons from event_time features, and use LabelEncoder from event_direction and event_type.

The pre-processing unit 410 may integrate processes used in each cyber attack into one process, and attach labels to each process using LabelEncoder, which is composed of the integrated processes.

The pre-processor 410 converts the minimum value to 0, the maximum value to 1, and the remaining values to values between 0 and 1 for each feature of the LID-DS data using min-max normalization. can

In step 520, the data division unit 420 may divide the preprocessed cyber attack data into training data and test data. The data division unit 420 may divide training data and test data to be used for machine learning at a preset ratio.

In step 530, the performance analyzer 430 may analyze the performance of machine learning algorithms using the divided training data and test data. The performance analyzer 430 receives divided training data as input to each machine learning model, trains each machine learning model using the divided training data, and divides test data into each learned machine learning model. , and intrusion detection accuracy for the divided test data can be derived using each learned machine learning model.

The performance analyzer 430 may evaluate the performance of each learned machine learning model using precision, recall, F1 score, and error rate.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. can be embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

410: pre-processing unit
420: data division
430: performance analysis unit

Claims (16)

In the machine learning algorithm performance analysis method performed by the algorithm comparison system,
Performing pre-processing on cyber attack data;
Dividing the preprocessed cyber attack data into training data and test data; and
Analyzing the performance of machine learning algorithms using the divided training data and the test data
Machine learning algorithm performance analysis method comprising a. According to claim 1,
The cyber attack data,
LID-DS (Leipzig Intrusion Detection-Data Set),
The LID-DS data,
It is classified according to a plurality of cyber attack methods, and includes a plurality of features according to each cyber attack method,
Performing the preprocessing step,
Deleting argument features and missing values for the LID-DS data, removing colons for event_time features, and using LabelEncoder for event_direction and event_type.
Machine learning algorithm performance analysis method comprising a. According to claim 2,
Performing the preprocessing step,
Integrating the processes used in each cyber attack into one process, and attaching labels to each process using LabelEncoder for the labels composed of the integrated process
Machine learning algorithm performance analysis method comprising a. According to claim 1,
Performing the preprocessing step,
Converting the minimum value to 0, the maximum value to 1, and the remaining values to values between 0 and 1 for each feature of the LID-DS data using min-max normalization
Machine learning algorithm performance analysis method comprising a. According to claim 1,
The dividing step is
Dividing the training data and the test data to be used in the machine learning model at a predetermined ratio
Machine learning algorithm performance analysis method comprising a. According to claim 1,
The analysis step is
The divided training data is input to each machine learning model, each machine learning model is trained using the divided training data, and the divided test data is input to each learned machine learning model , Deriving intrusion detection accuracy for the divided test data using each of the learned machine learning models
Machine learning algorithm performance analysis method comprising a. According to claim 6,
The analysis step is
Evaluating the performance of each learned machine learning model using precision, recall, F1 score and error rate
Machine learning algorithm performance analysis method comprising a. A computer-readable recording medium storing a computer program,
When the computer program is executed by a processor,
Performing pre-processing on cyber attack data;
Dividing the preprocessed cyber attack data into training data and test data; and
Analyzing the performance of machine learning algorithms using the divided training data and the test data
A computer-readable recording medium containing instructions for causing the processor to perform an operation including a. As a computer program stored on a computer-readable recording medium,
When the computer program is executed by a processor,
Performing pre-processing on cyber attack data;
Dividing the preprocessed cyber attack data into training data and test data; and
Analyzing the performance of machine learning algorithms using the divided training data and the test data
A computer program comprising instructions for causing the processor to perform an operation comprising: In the machine learning algorithm performance analysis system,
a pre-processing unit that performs pre-processing on cyber attack data;
a data division unit dividing the preprocessed private cyber attack data into training data and test data; and
A performance analyzer for analyzing the performance of machine learning algorithms using the divided training data and the test data.
Algorithm performance analysis system comprising a. According to claim 10,
The cyber attack data,
LID-DS (Leipzig Intrusion Detection-Data Set),
The LID-DS data,
It is classified according to a plurality of cyber attack methods, and includes a plurality of features according to each cyber attack method,
The pre-processing unit,
Delete argument features and missing values for the LID-DS data, remove colons for event_time features, and use LabelEncoder for event_direction and event_type.
Algorithm performance analysis system, characterized in that. According to claim 11,
The pre-processing unit,
The processes used in each cyber attack are integrated into one process, and the labels composed of the integrated processes are attached to each process using LabelEncoder.
Algorithm performance analysis system, characterized in that. According to claim 10,
The pre-processing unit,
Converting the minimum value to 0, the maximum value to 1, and the remaining values to values between 0 and 1 for each feature of the LID-DS data using Min-Max Normalization
Algorithm performance analysis system, characterized in that. According to claim 10,
The data partitioning unit,
Splitting the training data and the test data to be used in the machine learning model at a predetermined ratio
Algorithm performance analysis system, characterized in that. According to claim 10,
The performance analysis unit,
The divided training data is input to each machine learning model, each machine learning model is trained using the divided training data, and the divided test data is input to each learned machine learning model , Deriving intrusion detection accuracy for the divided test data using each learned machine learning model
Algorithm performance analysis system, characterized in that. According to claim 15,
The performance analysis unit,
Evaluating the performance of each trained machine learning model using precision, recall, F1 score and error rate
Algorithm performance analysis system, characterized in that.

Images