JP2016024506A - Data processing apparatus, data processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently perform a correlation analysis in a case in which high frequency events and low frequency events are mixed up in relation to an analysis technique for combining stream monitoring with storage search.SOLUTION: A procedure list generator 120 generates a conditional procedure list 131 in which a first starting procedure for starting storage search when stream monitoring is completed, and a second starting procedure for starting storage search when a specific event occurs in the stream monitoring before the completion of the stream monitoring are defined. A monitoring-search engine 140 starts storage search in accordance with either the first starting procedure or the second starting procedure, upon determining whether the specific event occurs before the completion of the stream monitoring in accordance with the conditional procedure list 131.SELECTED DRAWING: Figure 1

Description

  The present invention relates to technology for detecting cyber attacks and unauthorized access.

In recent years, information leakage due to cyber attacks and unauthorized access has become a social problem, and there is an increasing demand for detecting cyber attacks and unauthorized access.
As a method for detecting unauthorized access, a method is disclosed in which key data is extracted from a packet, collated with ongoing scenario storage, and reported as unauthorized access based on the result (Patent Document 1).
In addition, a method is disclosed in which an alert is raised when matching with a database is performed, and log information with a high suspicion of intrusion is uploaded to an analysis center (Patent Document 2).
Furthermore, when noticeable content is detected in the partial log information extracted at regular intervals, relevant log information in other corresponding network devices can be quickly collected and efficient analysis processing can be realized. A method for optimizing the extraction period is disclosed (Patent Document 3).

JP 2005-136526 A JP 2005-189996 A JP 2004-288110 A

When detecting a cyber attack, it is difficult to determine that it is a cyber attack based only on a log indicating a certain event. If only the log is used, leakage and false detections increase.
This is because individual logs often do not differ from normal communication logs.
For this reason, it is necessary to judge the cyber attack detection by comprehensively analyzing the contents of various logs by analyzing the correlation of logs.
The conventional detection method for cyber attacks has a problem that it takes time to perform correlation analysis including events that occur extremely frequently.
For example, the following is an example.
(A) An access originating from an external IP address not listed in the whitelist occurs (recorded in the firewall log)
(B) 5 IDS Medium alerts originating from the same IP address as in (a) within one hour after the occurrence of (a) event. In this example, either (a) or (b) Is detected, the source IP address is fixed, and the other detection process can be performed with the IP address limited.
As another feature, the frequency varies greatly depending on the detection target.
In this example, it is assumed that (a) occurs very frequently, but (b) rarely occurs.

Since the conventional monitoring method requires processing in time order, in the above example, (a) is detected and (b) is detected.
Since (a) must be detected for every IP address, it occurs very frequently, and every time (a) is detected, a process for monitoring (b) is issued at the corresponding IP address. The number of processes to be monitored becomes enormous and takes too much time.

  The main object of the present invention is to solve the above-mentioned problems. In an analysis method combining stream monitoring and storage search, it is possible to efficiently perform correlation analysis in which high-frequency events and low-frequency events are mixed. Is the main purpose.

The data processing apparatus according to the present invention
A scenario generation unit that performs stream monitoring for monitoring data transmitted and received on the network, and further generates a monitoring search scenario that is defined to perform storage search for searching a log of data transmitted and received on the network;
In accordance with the monitoring search scenario, the stream monitoring, and further, a scenario execution unit for performing the storage search,
The scenario generation unit
A first start procedure for starting the storage search when the stream monitoring is completed, and a second start for starting the storage search when a specific event occurs in the stream monitoring before the completion of the stream monitoring. Generate a supervised search scenario with defined procedures,
The scenario execution unit
The storage search is started according to one of the first start procedure and the second start procedure based on whether or not the specific event has occurred before completion of the stream monitoring.

  In the present invention, since the second start procedure for starting the storage search before the completion of the stream monitoring is defined, the storage search for the frequently-occurring event can be started at an early stage. Correlation analysis with low events can be performed efficiently.

FIG. 3 is a diagram illustrating a configuration example of a correlation analyzer according to the first embodiment. FIG. 3 is a flowchart showing an operation example of a procedure list generator according to the first embodiment. FIG. 3 is a flowchart showing an operation example of a monitoring unit according to the first embodiment. FIG. 6 is a flowchart showing an operation example of a search unit and a result determination unit according to the first embodiment. FIG. 4 is a diagram illustrating an example of a detection rule according to the first embodiment. FIG. 4 shows an example of a conditional procedure list according to the first embodiment. FIG. 6 is a diagram illustrating a configuration example of a monitoring unit according to the second embodiment. FIG. 10 is a diagram illustrating an example of a table according to the second embodiment. FIG. 3 is a diagram illustrating a hardware configuration example of a correlation analyzer according to the first and second embodiments.

Embodiment 1 FIG.
In the present embodiment and Embodiment 2, a configuration for analyzing the correlation of logs for cyber attack detection will be described.
More specifically, in this embodiment and Embodiment 2, a combination of stream monitoring (hereinafter also referred to as stream processing or simply monitoring) and storage search (hereinafter also simply referred to as storage) stored in the storage is combined. A configuration for efficiently performing a correlation analysis in which a high-frequency event and a low-frequency event are mixed will be described by performing a search process based on the detection result in the process and the storage search job holding status.
Note that stream monitoring is a process for monitoring data transmitted and received on the network in real time, and storage search is a process for searching log data that matches a predetermined condition from log data stored in the storage. is there.

FIG. 1 is a configuration diagram illustrating an example of a correlation analyzer 100 according to the present embodiment.
In FIG. 1, the correlation analysis apparatus 100 includes a procedure list generator 120 and a monitoring / search engine 140.
The procedure list generator 120 generates a conditional procedure list 131 based on the detection rule 111, the event frequency information 112, and the system specification information 113 of the system that executes monitoring / searching.
The conditional procedure list 131 is a scenario defined to perform stream monitoring and further to perform storage search, and corresponds to an example of a monitoring search scenario.
The procedure list generator 120 corresponds to an example of a scenario generation unit.
The monitoring / search engine 140 detects an event that matches the detection rule by combining monitoring and searching according to the procedure described in the conditional procedure list 131, and outputs a detection result 151.
That is, the monitoring / search engine 140 performs stream monitoring according to the conditional procedure list 131 and further performs storage search, and corresponds to an example of a scenario execution unit.

  The procedure list generator 120 includes a monitoring / search classification unit 121 and a conditional procedure list generation unit 122.

  The monitoring / search classification unit 121 generates a procedure in which the detection method of the detection rule 111 is classified into monitoring and search.

  The conditional procedure list generation unit 122 generates a conditional procedure list 131 from the procedures classified into monitoring and search.

  The conditional procedure list 131 is a list describing the monitoring / retrieval procedures, in which a monitoring result, the number of held search jobs, and the like are designated as search execution conditions.

  The monitoring / search engine 140 includes a stream DB (stream database) 141, a storage DB (storage database) 142, a monitoring unit 143, a search unit 144, a search condition determination unit 145, a search job holding unit 146, and a result determination unit 147. The

  A stream DB (stream database) 141 is a database for processing received logs as a stream in real time.

  The storage DB (storage database) 142 is a database for accumulating logs and searching later.

  The monitoring unit 143 uses the stream DB 141 to monitor the appearance of a log that matches the conditions.

  The search unit 144 performs a search using the storage DB 142.

  The search condition determination unit 145 determines search execution from the detection result of the monitoring unit 143 based on the conditional procedure list 131.

  The search job holding unit 146 holds an unexecuted search job.

  The result determination unit 147 determines whether or not the correlation defined by the detection rule 111 has been detected by combining the detection result by the monitoring unit 143 and the search result by the search unit 144, and outputs the detection result if detected.

Next, the outline | summary of operation | movement of the correlation analyzer 100 which concerns on this Embodiment is demonstrated with the example given by the subject which invention is going to solve.
Details of general operations will be described later with reference to flowcharts.
In the case of the example mentioned in the problem to be solved by the invention, the detection order is reversed from the time order, first the low frequency (b) is monitored, and (b) is detected and the past data is used to detect (a). By performing a storage search process, it can be detected efficiently.
This is because the target external IP address can be limited by the detection in (b), and the search condition in (a) can be narrowed down.
Further, even if the detection by the monitoring unit 143 has not reached five times, the search condition determination unit 145 makes a determination, and the search is started depending on the case.
For example, if the job search information holds that an address having the same search target log is detected in the situation where the detection is four times, the search is started.
Therefore, there is an advantage that the detection result is output earlier than when the search is started after the detection by the monitoring unit 143 is fixed to five times.
These monitoring and search procedures are described in the conditional procedure list 131, and the monitoring / search engine 140 performs processing according to the procedures.

As described above, in this embodiment, the procedure list generator 120 starts the storage search when the stream monitoring is completed (when the target address is detected five times in the above example). When a specific event occurs in stream monitoring before the completion of stream monitoring (when search job information for the same log exists, the target address is detected four times). A conditional procedure list 131 in which a second start procedure to be started is defined is generated.
Then, the monitoring / search engine 140 performs stream monitoring and monitors whether or not a specific event occurs in the stream monitoring. When the stream monitoring is completed without occurrence of the specific event, the first Start the storage search according to the start procedure.
On the other hand, when a specific event occurs before the completion of the stream monitoring, the storage search is started according to the second start procedure.

Next, a general operation example of correlation analysis apparatus 100 according to the present embodiment will be described in detail.
2 to 4 show flowcharts of operation examples.

  FIG. 2 is a flowchart of the procedure list generator 120.

First, in step 200, the monitoring / search classification unit 121 extracts and aggregates the same processes and processes having an inclusion relationship from the detection rules 111.
Next, in step 201, the monitoring / search classification unit 121 classifies the processing into monitoring / search based on the event frequency information 112 and the system specification information 113.
Next, in step 202, the conditional procedure list generation unit 122 adds a search condition to the procedure list and generates a conditional procedure list 131.

FIG. 3 is a flowchart of monitoring processing, and FIG. 4 is a flowchart of search and result determination, which are executed in parallel.
First, the flow of the monitoring process will be described.

In step 300, an infinite loop is performed.
In step 301, the monitoring unit 143 performs event detection by stream monitoring.
If an event is detected, the process proceeds to step 302; otherwise, the process loops at step 300.
In step 302, the monitoring unit 143 sets a detection flag and outputs the detection result to the search condition determination unit 145.
The detection flag is set every time a target address is detected in stream monitoring.

  Next, the flow of search and result determination will be described.

In step 400, an infinite loop is performed.
In step 401, the search condition determination unit 145 confirms whether a detection flag is set.
When the detection flag is set (when there is a detection), the search condition determination unit 145 performs a search condition determination in step 402.
In the search condition determination, the search condition determination unit 145 determines whether there is a search job that is necessary as a result of detection in accordance with the contents described in the conditional procedure list 131, and if there is, is the same as the search target in the search job Judges whether there is a search job for the log.
Also, control is performed such as performing the overlapping portion first in accordance with the overlapping degree of the search time range.

  As a result of the search condition determination (step 402), if storage search is not defined in the conditional procedure list 131, the process proceeds to step 406.

As a result of the search condition determination (step 402), if it is determined that search execution is unnecessary, the search job holding unit 146 holds search job information in step 403.
That is, the conditional procedure is to perform a storage search when there are five address detections as in the previous example, or when there are four address detections and search job information for the same log is held. If it is defined in the list 131 and the job search information for the same log is not held for the first to third address detection or the fourth address detection, the condition is not satisfied. The storage search is not performed and the search job information is held.
In the search job information, the address detected in step 301 in FIG. 3 is described.

As a result of the search condition determination (step 402), when it is determined that the search execution is necessary, the search unit 144 actually executes the search in step 404.
That is, the conditional procedure is to perform a storage search when there are five address detections as in the previous example, or when there are four address detections and search job information for the same log is held. If it is defined in the list 131, the condition is satisfied if it is the fifth address detection or the fourth address detection in a situation where job search information for the same log is held. A storage search is performed.
After executing the search (step 404), in step 405, the result determination unit 147 confirms the search result.
After the search condition determination and the subsequent processing are completed, in step 406, the result determination unit 147 determines the result as to whether or not the correlation in the log is detected.
If there is detection as a result of the result determination (step 406), the result determination unit 147 outputs the detection result in step 407.
Loops if there is no detection.

FIG. 5 shows a description example of the detection rule in the example given in the problem to be solved by the invention.
In this example, square brackets “[]” are called a section, and a rule number is described therein.
Also, the name and value are connected by “=”, and the value of the element of each name is shown.
After “#”, it is a comment until the end of the line.
In the detection rule file, in addition to the rule number, a log path for the rule, a log type, a detection condition, and the like are described.
Further, a rule number that is a trigger for starting search is designated by the name trigger.
A section having this item indicates that the detection process described in the corresponding section is performed only when a trigger rule is established.
Note that the detection rule does not define a detection method such as whether detection is performed by stream monitoring or storage search.

FIG. 6 shows a description example of the conditional procedure list 131 in the example given in the problem to be solved by the invention.
The conditional procedure list 131 is a list in which detection procedures of log correlations to be detected based on detection rules are described.
A section name enclosed in square brackets “[]” starts with “stream” or “storage”.
A section starting with stream means using stream monitoring as a detection method, and a section starting with storage means using storage search.
Also, a search condition determination rule is defined by the name “condition”.
In this example, if there are two other search jobs that target the same log when it occurs three times out of five detections, search is executed if there are two other search jobs. If there is another job, search is executed.
In addition to the condition of whether or not there is a search for the same log, it may be determined whether or not the search is executed together with the condition of whether or not there is a search for the same content.

  As described above, the stream processing and the storage search are combined, and after the detection by the stream processing, the search is started according to the condition of the search job holding status, so that the correlation of the log can be detected at high speed.

Embodiment 2. FIG.
In the second embodiment, an example will be described in which the monitoring unit 143 in the correlation analyzer 100 of the first embodiment enables detection of a periodic event.
Here, the periodic event is an event that appears regularly, such as beacon communication, which is frequently performed in cyber attacks.

Here, the term “periodic” refers to a state in which the same event (communication or the like) occurs at regular intervals (ΔT).
Specifically, when the number of occurrences k of the same event is an observation period T, a time interval ΔT, and a fluctuation δ of the number of event occurrences, it is defined that the occurrence occurs more than a threshold value {(T / ΔT) −δ} times. To do.
Considering that ΔT also fluctuates (fluctuation amount = α), the event to be counted by the counter is generated at the timing of (ΔT ± α).
If an event occurs due to a missing event or a fluctuation greater than the expected fluctuation, it is possible that a detection failure will occur, but it is considered that a regular event is occurring continuously. We think that we can detect at any timing by continuing monitoring.
ΔT is determined by the time difference between the first two events in the observation period T.

FIG. 7 shows a configuration diagram of the monitoring unit 143 according to the second embodiment.
A table update unit 701, a table determination unit 702, and a table 703 used for detecting a periodic event are added to the monitoring unit 143 of the first embodiment.
The table update unit 701 performs an update process on the table 703 based on the monitoring result.
The table determination unit 702 checks the contents of the table 703 and determines whether a periodic event is detected.

FIG. 8 shows an example of the table 703.
The table 703 has columns of event type 801, ΔT 802, last event occurrence time 803, number of matches 804, cut-off time 805, and deletion flag 806.
The observation information 807 includes information on the start time, fluctuation in event occurrence interval (α), and fluctuation in event occurrence frequency (δ).

The event type 801 is a type for distinguishing events, and is, for example, a set of an external host name and an internal IP address that perform communication.
ΔT802 is an event interval, and is determined by the time difference between the first two events having the same event type in the observation period T as described above.
The last event occurrence time 803 is the time when the event last occurred in the corresponding event type.
The coincidence count 804 is the number of times an event has occurred at a timing (ΔT ± α) from the previous event.
The cut-off time 805 is a time at which the possibility of periodic detection disappears if there is no coincidence by this time in the event type.
The deletion flag 806 is a flag indicating that there is no longer a possibility of detecting a periodic event in the corresponding event type.

In this way, memory usage can be reduced by a method of holding the last event occurrence time and the number of matches instead of holding each event time.
In addition, by generating a cut-off time and deleting event types that are no longer expected to be detected periodically, a deletion flag is set and sequentially removed, thereby enabling high-speed processing.

As mentioned above, although embodiment of this invention was described, you may implement combining these embodiment.
Alternatively, one of these embodiments may be partially implemented.
Or you may implement combining these embodiment partially.
In addition, this invention is not limited to these embodiment, A various change is possible as needed.

Finally, a hardware configuration example of the correlation analyzer 100 shown in the first and second embodiments will be described with reference to FIG.
The correlation analyzer 100 is a computer, and each element of the correlation analyzer 100 can be realized by a program.
As a hardware configuration of the correlation analyzer 100, an arithmetic device 901, an external storage device 902, a main storage device 903, a communication device 904, and an input / output device 905 are connected to the bus.

The arithmetic device 901 is a CPU (Central Processing Unit) that executes a program.
The external storage device 902 is, for example, a ROM (Read Only Memory), a flash memory, or a hard disk device.
The main storage device 903 is a RAM (Random Access Memory).
The communication device 904 is, for example, a NIC (Network Interface Card).
The input / output device 905 is, for example, a mouse, a keyboard, a display device, or the like.

The program is normally stored in the external storage device 902, and is loaded into the main storage device 903 and sequentially read into the arithmetic device 901 and executed.
The program is a program that realizes a function described as “unit” shown in FIG.
Further, an operating system (OS) is also stored in the external storage device 902. At least a part of the OS is loaded into the main storage device 903. ”Is executed.
In the description of the first and second embodiments, “determination of”, “determination of”, “extraction of”, “detection of”, “search of”, “confirmation of”, “ Results of processes described as "setting", "registration of", "analysis of", "selection of", "generation of", "input of", "output of", etc. Information, data, signal values, and variable values are stored in the main storage device 903 as files.

  9 is merely an example of the hardware configuration of the correlation analyzer 100, and the hardware configuration of the correlation analyzer 100 is not limited to the configuration illustrated in FIG. Also good.

  In addition, the data processing method according to the present invention can be realized by the procedure shown in the first and second embodiments.

  100 correlation analysis device, 111 detection rule, 112 event frequency information, 113 system spec information, 120 procedure list generator, 121 monitoring / search classification unit, 122 conditional procedure list generation unit, 131 conditional procedure list, 140 monitoring / search Engine, 141 Stream DB, 142 Storage DB, 143 Monitoring unit, 144 Search unit, 145 Search condition determination unit, 146 Search job holding unit, 147 Result determination unit, 151 Detection result, 701 Table update unit, 702 Table determination unit, 703 table.

Claims (6)

A scenario generation unit that performs stream monitoring for monitoring data transmitted and received on the network, and further generates a monitoring search scenario that is defined to perform storage search for searching a log of data transmitted and received on the network;
In accordance with the monitoring search scenario, the stream monitoring, and further, a scenario execution unit for performing the storage search,
The scenario generation unit
A first start procedure for starting the storage search when the stream monitoring is completed, and a second start for starting the storage search when a specific event occurs in the stream monitoring before the completion of the stream monitoring. Generate a supervised search scenario with defined procedures,
The scenario execution unit
The data processing apparatus, wherein the storage search is started according to one of the first start procedure and the second start procedure based on whether or not the specific event has occurred before completion of the stream monitoring. The scenario execution unit
Performing the stream monitoring and monitoring whether the specific event occurs in the stream monitoring;
When the stream monitoring is completed without the specific event occurring, the storage search is started according to the first start procedure,
2. The data processing apparatus according to claim 1, wherein when the specific event occurs before completion of the stream monitoring, the storage search is started according to the second start procedure. The scenario generation unit
Analyzing a detection rule for detecting an attack, in which neither the stream monitoring nor the storage search is defined, and generating the monitoring search scenario in which the stream monitoring and the storage search are defined The data processing apparatus according to claim 1, wherein: The scenario execution unit
Stores a table that manages the last event occurrence time when an event that occurs at regular intervals last occurs and the number of occurrences of the event that occurred at regular intervals, and updates the table each time the event occurs The data processing apparatus according to claim 1, wherein the event is monitored in the stream monitoring. A scenario generation step for generating a monitoring search scenario in which the computer performs stream monitoring for monitoring data transmitted / received on the network and further performs storage search for searching a log of data transmitted / received on the network; ,
A scenario execution step in which the computer performs the stream monitoring according to the monitoring search scenario, and further performs the storage search;
In the scenario generation step,
The computer
A first start procedure for starting the storage search when the stream monitoring is completed, and a second start for starting the storage search when a specific event occurs in the stream monitoring before the completion of the stream monitoring. Generate a supervised search scenario with defined procedures,
In the scenario execution step,
The computer
A data processing method comprising: starting the storage search according to one of the first start procedure and the second start procedure based on whether or not the specific event has occurred before completion of the stream monitoring. A scenario generation process for generating a monitoring search scenario for performing stream monitoring for monitoring data transmitted and received on the network, and further defining a storage search for searching a log of data transmitted and received on the network;
According to the monitoring search scenario, the program performs a stream monitoring, and further executes a scenario execution process for performing the storage search.
In the scenario generation process,
In the computer,
A first start procedure for starting the storage search when the stream monitoring is completed, and a second start for starting the storage search when a specific event occurs in the stream monitoring before the completion of the stream monitoring. Generate a supervised search scenario with defined procedures,
In the scenario execution process,
In the computer,
The storage search is started according to one of the first start procedure and the second start procedure based on the occurrence of the specific event before the completion of the stream monitoring.

Images