KR20170067118A - Docker container security log analysis method and system based on hadoop distributed file system in cloud environment - Google Patents
Docker container security log analysis method and system based on hadoop distributed file system in cloud environment Download PDFInfo
- Publication number
- KR20170067118A KR20170067118A KR1020160007273A KR20160007273A KR20170067118A KR 20170067118 A KR20170067118 A KR 20170067118A KR 1020160007273 A KR1020160007273 A KR 1020160007273A KR 20160007273 A KR20160007273 A KR 20160007273A KR 20170067118 A KR20170067118 A KR 20170067118A
- Authority
- KR
- South Korea
- Prior art keywords
- container
- containers
- docker
- log data
- log
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
- G06F21/552—Detecting local intrusion or implementing counter-measures involving long-term monitoring or reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
-
- G06F17/30144—
-
- G06F17/30194—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/57—Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
- G06F21/577—Assessing vulnerabilities and evaluating computer system security
Abstract
The HDFS-based method for analyzing a container container security log in a cloud environment includes the steps of: collecting log data of each of the plurality of the container containers in a fluentd agent mounted on each of a plurality of the container containers; Transmitting log data of each of the plurality of the docker containers to at least one fluentd collector mounted in a monitoring docker container; And distributing log data of each of the plurality of the docker containers to a data node included in each of a plurality of nodes of an HDFS (hadoop distributed file system) connected to the monitoring docker container in the monitoring docker container .
Description
The following embodiments are directed to a system and method for security log analysis of a docker container based on hadoop distributed file system (HDFS) in a cloud environment, and more specifically to a system and method for analyzing log data generated when a docker container is executed, Storage, and / or management.
With the recent change to the internet of things (IoT) environment, cloud companies have begun to provide hybrid cloud services that combine infrastructure as a service (IaaS) and platform as a service (PaaS). Here, cloud computing is one of the core technologies for configuring IoT, and it is based on virtualization technology that improves performance by installing applications on various devices.
In particular, because IoT devices have limited computing power and resources compared to existing computers, virtualization technology using virtual machines implemented in existing computers is not suitable for IoT environments.
Accordingly, a container-based virtualization technology, a docker, has been proposed. The docker container can be run under limited computing power by isolating applications and supporting the execution of processes on an application-by-application basis, and can only use limited resources by sharing the kernel of the host operating system.
Since such a container is used by sharing a host operating system, when a malicious attack or intrusion occurs from the outside in the course of execution of an application, there is a disadvantage that it directly damages the host operating system.
Accordingly, there is a demand for a technique for detecting a malicious attack or an intrusion generated from a docker container based on log data generated when the docker container is executed.
However, if you apply the technique of collecting and managing existing log data in the container, log data is collected and analyzed. When the file is created on the local disk due to network delay, database SQL parsing, index update, More costs are incurred.
Therefore, there is a need for a technique for efficiently analyzing and managing log data in the course of processing log data to detect a malicious attack or intrusion.
Therefore, the following embodiments propose a technique for efficiently analyzing and managing log data in the process of processing log data to detect a malicious attack or an intrusion.
One embodiment provides a method and system for analyzing a container container security log that efficiently analyzes and manages log data of a container container using HDFS.
Specifically, one embodiment provides a method and system for analyzing a container container security log that efficiently analyzes and manages log data by distributing log data in an HDFS.
According to one embodiment, a HDFS-based docker container security log analysis method in a cloud environment includes a fluentd agent mounted on each of a plurality of the docker containers, Collecting log data; Transmitting log data of each of the plurality of the docker containers to at least one fluentd collector mounted in a monitoring docker container; And distributing log data of each of the plurality of the docker containers to a data node included in each of a plurality of nodes of an HDFS (hadoop distributed file system) connected to the monitoring docker container in the monitoring docker container .
The step of distributing and storing the log data of each of the plurality of the container containers to the data nodes included in each of the plurality of nodes of the HDFS may include the steps of: And recording metadata information on the log data of each of the plurality of scatterer containers.
Collecting log data of each of the plurality of the container containers may include performing a pre-process of converting log data of each of the plurality of the container containers from an unstructured data format to a fixed data format.
Wherein the transmitting the log data of each of the plurality of the bucket containers to the at least one fluent collector mounted on the monitoring dozer container comprises converting log data of each of the preprocessed plurality of the bucket containers into a javascript object notation (JSON) format Converting; And transmitting log data of each of the plurality of the docker containers converted into the JSON format to the at least one fluent collector.
The step of distributing and storing the log data of each of the plurality of the container containers to the data node included in each of the plurality of nodes of the HDFS may include recording the time at which the log data of each of the plurality of the container containers is distributed and stored .
One embodiment of the present invention can provide a method and system for analyzing a container container security log that efficiently analyzes and manages log data of a container container using HDFS.
In particular, embodiments may provide a method and system for analyzing a container log security log that efficiently analyzes and manages log data by logically storing log data in an HDFS.
Accordingly, one embodiment can provide a method and system for analyzing a container container security log, which detects a malicious attack or an intrusion generated from a container container based on log data that is efficiently analyzed and managed.
FIG. 1 is a diagram illustrating a system for analyzing a security log of a container according to an embodiment of the present invention.
FIG. 2 is a view for explaining the operation of the security log analyzing system of the container according to the embodiment of the present invention.
FIG. 3 is a view for explaining a preprocessing operation of the security log analyzing system of the container according to the embodiment of the present invention.
4 is a flowchart illustrating a method of analyzing a security log of a container in accordance with an embodiment of the present invention.
5 is a block diagram illustrating a system for analyzing the security log of a container in accordance with an embodiment of the present invention.
Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. In addition, the same reference numerals shown in the drawings denote the same members.
Also, terminologies used herein are terms used to properly represent preferred embodiments of the present invention, which may vary depending on the user, intent of the operator, or custom in the field to which the present invention belongs. Therefore, the definitions of these terms should be based on the contents throughout this specification.
FIG. 1 is a diagram illustrating a system for analyzing a security log of a container according to an embodiment of the present invention.
Referring to FIG. 1, a
Here, the
Specifically, instead of supporting the guest operating system for each application, such as the
When a malicious attack or intrusion occurs from the outside in the course of execution of the application by the
Therefore, the
Specifically, the system for analyzing the container
At this time, the container container security
In addition, the
Although the HDFS 131 is illustrated as including the HDFS 131 in the
FIG. 2 is a view for explaining the operation of the security log analyzing system of the container according to the embodiment of the present invention.
Referring to FIG. 2, the
The
At this time, the
The preprocessed log data is transmitted to the at least one
In addition, the
In addition, in the process of transmitting the preprocessed log data to at least one
The
Specifically, the
In addition, the
Accordingly, the
In particular, since not only the log data of each of a plurality of the container containers but also the name or service name of the container container that generated the log data is stored in the
The
That is, the
When the log data of each of the plurality of the docker containers received by the at least one
In this manner, the
In addition, the
FIG. 3 is a view for explaining a preprocessing operation of the security log analyzing system of the container according to the embodiment of the present invention.
3, a fluent agent (the fluent agent is mounted on each of a plurality of the docker containers) included in the docker container security log analysis system according to one embodiment is collected in each of the plurality of docker containers You can perform preprocessing that converts log data from an unstructured data format to a regular data format.
For example, as shown in the drawing, the fluent agent mounted on each of the plurality of the container containers may be configured to include a remote server name, a host indicating a server name, a user indicating a system user account, A method indicating the HTTP method, a path indicating the path of the generated log data, a code indicating the HTTP status code, a size indicating the size of the log data, a referrer indicating an HTTP referer, an agent indicating a fluent agent name, or an HTTP forwarder the log data can be pre-processed to convert the log data from the unstructured data format to the formatted data format based on the attributes such as forwarder.
4 is a flowchart illustrating a method of analyzing a security log of a container in accordance with an embodiment of the present invention. Hereinafter, the method for analyzing the security log of the container container is performed through each component included in the security container analysis log system.
Referring to FIG. 4, the system for analyzing the security log of a container container according to an exemplary embodiment of the present invention collects log data of each of a plurality of the container containers via a fluentd agent installed in each of the plurality of container containers (410).
At this time, in
Also, in
The system for analyzing the container container security log may then send log data of each of the plurality of the container containers to the at least one fluent collector (s) mounted on the monitoring docker container, via the fluent agent mounted on each of the plurality of the docker containers fluentd collector (420).
In
In
Thereafter, the system for analyzing the container container security log analyzes the log data of each of the plurality of the container containers on the data node included in each of the plurality of nodes of the HDFS (Hadoop Distributed File System) connected to the monitoring docker container (430).
At this time, in
In
5 is a block diagram illustrating a system for analyzing the security log of a container in accordance with an embodiment of the present invention.
Referring to FIG. 5, a system for analyzing a container container security log according to an embodiment includes a
The
Here, the
In addition, the
The
At this time, the
In addition, the
The
At this time, the
In addition, the
The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.
The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.
The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.
While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.
Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.
Claims (5)
Collecting log data of each of the plurality of docker containers in a fluentd agent mounted on each of the plurality of docker containers;
Transmitting log data of each of the plurality of the docker containers to at least one fluentd collector mounted in a monitoring docker container; And
Distributing log data of each of the plurality of the docker containers to a data node included in each of a plurality of nodes of an HDFS (Hadoop Distributed File System) connected to the monitoring docker container in the monitoring docker container
A method for analyzing a security log of a container in a container.
The step of distributing and storing the log data of each of the plurality of the container containers to the data node included in each of the plurality of nodes of the HDFS
In the monitoring detour container, recording metadata information on log data of each of the plurality of docker containers, which are distributed and stored, to a name node included in a master node of the monitoring detour container
A method for analyzing a security log of a container in a container.
The step of collecting the log data of each of the plurality of the container containers
Performing preprocessing for converting log data of each of the plurality of the container containers from an unstructured data format to a fixed data format
A method for analyzing a security log of a container in a container.
Wherein transmitting the log data of each of the plurality of the container containers to at least one fluent collector mounted on the monitoring docker container
Converting log data of each of the preprocessed plurality of the container containers into a JavaScript object notation (JSON) format; And
Transmitting log data of each of the plurality of docker containers converted to the JSON format to the at least one fluent collector
The method further comprising:
The step of distributing and storing the log data of each of the plurality of the container containers to the data node included in each of the plurality of nodes of the HDFS
Recording the time at which the log data of each of the plurality of the container containers is distributedly stored
The method further comprising:
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR20150173563 | 2015-12-07 | ||
KR1020150173563 | 2015-12-07 |
Publications (2)
Publication Number | Publication Date |
---|---|
KR20170067118A true KR20170067118A (en) | 2017-06-15 |
KR101810762B1 KR101810762B1 (en) | 2017-12-19 |
Family
ID=59217462
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
KR1020160007273A KR101810762B1 (en) | 2015-12-07 | 2016-01-20 | Docker container security log analysis method and system based on hadoop distributed file system in cloud environment |
Country Status (1)
Country | Link |
---|---|
KR (1) | KR101810762B1 (en) |
Cited By (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107770066A (en) * | 2017-10-20 | 2018-03-06 | 成都精灵云科技有限公司 | It is a kind of across main frame, travelling across VLAN, the Docker container methods of river diversion across cluster |
CN107948203A (en) * | 2017-12-29 | 2018-04-20 | 平安科技(深圳)有限公司 | A kind of container login method, application server, system and storage medium |
CN109684038A (en) * | 2018-12-18 | 2019-04-26 | 网易(杭州)网络有限公司 | Processing method, device and the electronic equipment of Docker service container log |
KR20190066516A (en) * | 2017-12-05 | 2019-06-13 | 숭실대학교산학협력단 | System and method for supervising doker container, computer readable medium for performing the method |
CN109902070A (en) * | 2019-01-22 | 2019-06-18 | 华中师范大学 | A kind of parsing storage searching method towards WiFi daily record data |
CN110941434A (en) * | 2018-09-21 | 2020-03-31 | 中国石油化工股份有限公司 | Seismic processing software deployment method based on container technology |
KR20200052798A (en) * | 2018-11-07 | 2020-05-15 | 숭실대학교산학협력단 | Log analysis framework device of docker container |
CN111930700A (en) * | 2020-07-13 | 2020-11-13 | 车智互联(北京)科技有限公司 | Distributed log processing method, server, system and computing equipment |
CN112764878A (en) * | 2021-01-13 | 2021-05-07 | 中科曙光(南京)计算技术有限公司 | Deep learning-based big data all-in-one machine container cluster risk prediction method |
US11269537B2 (en) | 2018-06-29 | 2022-03-08 | Seagate Technology Llc | Software containers with security policy enforcement at a data storage device level |
US11307980B2 (en) | 2018-04-20 | 2022-04-19 | Seagate Technology Llc | Distributed data storage system with passthrough operations |
KR102426889B1 (en) * | 2022-01-05 | 2022-07-29 | 주식회사 이글루코퍼레이션 | Apparatus, method and program for analyzing and processing data by log type for large-capacity event log |
US11677778B2 (en) | 2020-10-19 | 2023-06-13 | Oracle International Corporation | Protecting data in non-volatile storages provided to clouds against malicious attacks |
US11790252B2 (en) | 2018-10-30 | 2023-10-17 | Samsung Sds Co., Ltd. | Apparatus and method for preprocessing security log |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11238012B1 (en) | 2018-05-15 | 2022-02-01 | Splunk Inc. | Log data extraction from data chunks of an isolated execution environment |
US11113301B1 (en) | 2018-05-15 | 2021-09-07 | Splunk Inc. | Generating metadata for events based on parsed location information of data chunks of an isolated execution environment |
KR102059808B1 (en) | 2018-06-11 | 2019-12-27 | 주식회사 티맥스오에스 | Container-based integrated management system |
US11537627B1 (en) | 2018-09-28 | 2022-12-27 | Splunk Inc. | Information technology networked cloud service monitoring |
US11941421B1 (en) | 2021-07-09 | 2024-03-26 | Splunk Inc. | Evaluating and scaling a collection of isolated execution environments at a particular geographic location |
-
2016
- 2016-01-20 KR KR1020160007273A patent/KR101810762B1/en active IP Right Grant
Cited By (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107770066B (en) * | 2017-10-20 | 2020-06-02 | 成都精灵云科技有限公司 | Cross-host, cross-VLAN and cross-cluster Docker container diversion method |
CN107770066A (en) * | 2017-10-20 | 2018-03-06 | 成都精灵云科技有限公司 | It is a kind of across main frame, travelling across VLAN, the Docker container methods of river diversion across cluster |
KR20190066516A (en) * | 2017-12-05 | 2019-06-13 | 숭실대학교산학협력단 | System and method for supervising doker container, computer readable medium for performing the method |
CN107948203A (en) * | 2017-12-29 | 2018-04-20 | 平安科技(深圳)有限公司 | A kind of container login method, application server, system and storage medium |
US11307980B2 (en) | 2018-04-20 | 2022-04-19 | Seagate Technology Llc | Distributed data storage system with passthrough operations |
US11269537B2 (en) | 2018-06-29 | 2022-03-08 | Seagate Technology Llc | Software containers with security policy enforcement at a data storage device level |
CN110941434A (en) * | 2018-09-21 | 2020-03-31 | 中国石油化工股份有限公司 | Seismic processing software deployment method based on container technology |
US11790252B2 (en) | 2018-10-30 | 2023-10-17 | Samsung Sds Co., Ltd. | Apparatus and method for preprocessing security log |
KR20200052798A (en) * | 2018-11-07 | 2020-05-15 | 숭실대학교산학협력단 | Log analysis framework device of docker container |
CN109684038A (en) * | 2018-12-18 | 2019-04-26 | 网易(杭州)网络有限公司 | Processing method, device and the electronic equipment of Docker service container log |
CN109902070A (en) * | 2019-01-22 | 2019-06-18 | 华中师范大学 | A kind of parsing storage searching method towards WiFi daily record data |
CN109902070B (en) * | 2019-01-22 | 2023-12-12 | 华中师范大学 | WiFi log data-oriented analysis storage search method |
CN111930700A (en) * | 2020-07-13 | 2020-11-13 | 车智互联(北京)科技有限公司 | Distributed log processing method, server, system and computing equipment |
US11677778B2 (en) | 2020-10-19 | 2023-06-13 | Oracle International Corporation | Protecting data in non-volatile storages provided to clouds against malicious attacks |
CN112764878A (en) * | 2021-01-13 | 2021-05-07 | 中科曙光(南京)计算技术有限公司 | Deep learning-based big data all-in-one machine container cluster risk prediction method |
CN112764878B (en) * | 2021-01-13 | 2024-04-23 | 中科曙光(南京)计算技术有限公司 | Deep learning-based big data all-in-one container cluster risk prediction method |
KR102426889B1 (en) * | 2022-01-05 | 2022-07-29 | 주식회사 이글루코퍼레이션 | Apparatus, method and program for analyzing and processing data by log type for large-capacity event log |
Also Published As
Publication number | Publication date |
---|---|
KR101810762B1 (en) | 2017-12-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
KR101810762B1 (en) | Docker container security log analysis method and system based on hadoop distributed file system in cloud environment | |
US10498857B2 (en) | System interaction monitoring and component scaling | |
Barika et al. | Orchestrating big data analysis workflows in the cloud: research challenges, survey, and future directions | |
US10129118B1 (en) | Real time anomaly detection for data streams | |
US10362141B1 (en) | Service group interaction management | |
US9098408B2 (en) | Ticket consolidation for multi-tiered applications | |
US10341355B1 (en) | Confidential malicious behavior analysis for virtual computing resources | |
US20210120012A1 (en) | Detecting malicious beaconing communities using lockstep detection and co-occurrence graph | |
US9977898B1 (en) | Identification and recovery of vulnerable containers | |
US20180039548A1 (en) | Smart virtual machine snapshotting | |
US9766995B2 (en) | Self-spawning probe in a distributed computing environment | |
Pătraşcu et al. | Logging framework for cloud computing forensic environments | |
Solaimani et al. | Online anomaly detection for multi‐source VMware using a distributed streaming framework | |
KR20150056266A (en) | Engine for processing fixed form and non-fixed form bigdata for controlling factory plant method thereof | |
US9325767B2 (en) | Deploying a portion of a streaming application to one or more virtual machines | |
Patrascu et al. | Logging for cloud computing forensic systems | |
US10360614B1 (en) | Assessing and rating deployments of resources | |
US11316879B2 (en) | Security protection for a host computer in a computer network using cross-domain security-relevant information | |
KR101505468B1 (en) | Data comparing processing method and system in cloud computing environment | |
US10536390B1 (en) | Requesting embedded hypermedia resources in data interchange format documents | |
US20150373071A1 (en) | On-demand helper operator for a streaming application | |
US10992742B2 (en) | Managing asset placement with respect to a distributed computing environment having a set of hosts | |
KR101630088B1 (en) | Method and apparatus for monitoring life-cycle of virtual machine | |
US10805180B2 (en) | Enterprise cloud usage and alerting system | |
Hatcher et al. | Edge computing based machine learning mobile malware detection |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AMND | Amendment | ||
E601 | Decision to refuse application | ||
AMND | Amendment | ||
X701 | Decision to grant (after re-examination) | ||
GRNT | Written decision to grant |