CN112035828A - Security situation perception analysis technology and system for large-scale database cluster - Google Patents
Security situation perception analysis technology and system for large-scale database cluster Download PDFInfo
- Publication number
- CN112035828A CN112035828A CN202010713576.8A CN202010713576A CN112035828A CN 112035828 A CN112035828 A CN 112035828A CN 202010713576 A CN202010713576 A CN 202010713576A CN 112035828 A CN112035828 A CN 112035828A
- Authority
- CN
- China
- Prior art keywords
- real
- database
- time information
- analysis
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 70
- 238000005516 engineering process Methods 0.000 title claims abstract description 18
- 230000008447 perception Effects 0.000 title abstract description 11
- 238000012545 processing Methods 0.000 claims abstract description 30
- 238000004364 calculation method Methods 0.000 claims abstract description 29
- 238000000034 method Methods 0.000 claims abstract description 25
- 230000003139 buffering effect Effects 0.000 claims abstract description 16
- 238000004891 communication Methods 0.000 claims abstract description 16
- 238000000605 extraction Methods 0.000 claims abstract description 14
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 238000013500 data storage Methods 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000004140 cleaning Methods 0.000 claims description 2
- 238000002372 labelling Methods 0.000 claims description 2
- 238000012550 audit Methods 0.000 abstract description 12
- 230000000007 visual effect Effects 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
- G06F21/552—Detecting local intrusion or implementing counter-measures involving long-term monitoring or reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a security situation awareness analysis technology and system for a large-scale database cluster, and relates to the field of database auditing. A security posture awareness analysis technique for large-scale database clusters comprising: the method comprises the steps of capturing a database communication data packet, analyzing data flow of the captured data packet, sending an analysis result to a real-time stream processing engine for processing and feature extraction of real-time information, buffering and forwarding the real-time information by a message queue, receiving the real-time information, storing the real-time information in a distributed data warehouse, performing distributed calculation, distributing a calculation result to a subject database for storage, and pulling the calculation result of the subject database for BI analysis and display. In addition, the invention also provides a security situation perception analysis system for the large-scale database cluster. The security audit can be performed on the large-scale database cluster, and the whole security situation perception visual display of the large-scale database cluster is realized.
Description
Technical Field
The invention relates to the field of database auditing, in particular to a security situation perception analysis technology and a security situation perception analysis system for a large-scale database cluster.
Background
The prior art cannot meet the requirement of log information acquisition of a large-scale database cluster, and the data throughput of the large-scale database cluster is high, so that the number of log information in unit time is large, the traditional database security technology cannot analyze data flow in time, data packet loss is caused, and the final result is trial omission.
Log information of a traditional database security audit technology is generally stored in a Relational Database (RDBS) so that data storage capacity is low, and is generally in the GB-TB level. Since the log information amount of a large-scale database cluster is usually PB level, it is difficult for the storage resource of the conventional database security design technology to carry PB level data amount.
The traditional database security audit technology cannot perform complex calculation aiming at mass information (generally referred to as PB level), because the traditional database audit technology generally takes a single machine (a single computer) as a carrier, the calculation efficiency is low.
The traditional database auditing technology is mainly used for searching and displaying data by connecting a local relational database through WEB UI (user interface) through JDBC (java database connectivity) drive, and cannot provide interactive analysis result display in real time.
Disclosure of Invention
The invention aims to provide a security situation awareness analysis technology for a large-scale database cluster, which can realize security audit of the large-scale database cluster, provide multi-dimensional real-time database security analysis for massive security audit logs through distributed computation and rich security algorithms, and realize overall security situation awareness visualization presentation of the large-scale database cluster.
Another object of the present invention is to provide a security posture awareness analysis system for a large-scale database cluster, which is capable of operating a security posture awareness analysis technique for a large-scale database cluster.
The embodiment of the invention is realized by the following steps:
in a first aspect, an embodiment of the present application provides a security situation awareness analysis technique for a large-scale database cluster, which includes capturing a database communication data packet, performing data traffic analysis on the captured data packet, sending an analysis result to a real-time stream processing engine for processing and feature extraction to obtain real-time information, buffering and forwarding the real-time information by a message queue, receiving the real-time information and storing the real-time information in a distributed data warehouse, performing distributed computation on the real-time information, distributing a computation result to a subject database for storage, and pulling a computation result of the subject database for BI analysis and display.
In some embodiments of the present invention, the packet capturing the database communication data packet includes: and receiving a data stream through the network card to recombine the data packet, analyzing the TCP/IP message after the data packet is recombined to extract the information of the database communication protocol, analyzing the database communication protocol to extract the SQL statement, and forwarding the SQL statement.
In some embodiments of the present invention, the analyzing the data traffic of the captured data packet includes: and after receiving the data, carrying out stream processing on the SQL, cleaning and converting the stream data, carrying out data preprocessing on the characteristics, and labeling the information.
In some embodiments of the present invention, the sending the analysis result to the real-time stream processing engine for processing and feature extraction to obtain the real-time information includes: the features are extracted after the tag data is received.
In some embodiments of the present invention, the buffering and forwarding the real-time information through the message queue includes: and receiving the real-time information through kafka and forwarding the real-time information at regular time through a memory buffer mechanism.
In some embodiments of the present invention, the receiving and storing the real-time information in the distributed data warehouse comprises that the ArgoDB distributed database undertakes data storage tasks, and the information forwarded by Kafka is solidified and stored in a two-dimensional table manner.
In some embodiments of the invention, the performing distributed computation includes: the Spark distributed computing engine undertakes computing tasks, a built-in strategy engine scheduling algorithm is used for computing and forwarding a computing result.
In some embodiments of the invention, the distributing the calculation result to the topic database storage comprises: and storing the calculation result in a memory, and generating a multidimensional table according to a set condition.
In some embodiments of the present invention, the performing BI analysis through pulling the calculation result of the topic database includes: the multidimensional tables in the calculation LOAP database can be quickly retrieved and visually displayed through the Pilot engine.
In a second aspect, an embodiment of the present application provides a security situation awareness analysis system for a large-scale database cluster, which includes a log collection module and a security situation analysis module; the log acquisition module realizes acquisition and forwarding of database security logs and comprises the following sub-modules: the data packet capturing and analyzing submodule is used for capturing the data packet of the database communication; the real-time stream processing submodule is used for analyzing the data flow of the captured data packet; the feature extraction submodule is used for sending the analysis result to a real-time stream processing engine for processing and feature extraction to obtain real-time information; the message queue buffering and forwarding submodule is used for buffering and forwarding the real-time information through the message queue; the security situation analysis module realizes the analysis and display of the log and comprises the following sub-modules: the distributed data warehouse submodule is used for receiving the real-time information and storing the real-time information in a distributed data warehouse; the distributed computation submodule is used for carrying out distributed computation on the real-time information; the LOAP database submodule is used for distributing the calculation result to the subject database for storage; and the BI analysis submodule is used for carrying out BI analysis display by pulling the calculation result of the subject database.
Compared with the prior art, the embodiment of the invention has at least the following advantages or beneficial effects:
the technology of the invention can realize the security audit of the large-scale database cluster, provide multi-dimensional real-time database security analysis through distributed computation and rich security algorithm aiming at massive security audit logs, and realize the whole security situation perception visual display of the large-scale database cluster.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a schematic flow chart of a security situation awareness analysis technique for a large-scale database cluster according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a logical structure of a security situation awareness analysis technique for a large-scale database cluster according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a security situation awareness analysis technique module for a large-scale database cluster according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the individual features of the embodiments can be combined with one another without conflict.
Example 1
Referring to fig. 1 and 2 and fig. 3, fig. 1 is a schematic diagram illustrating a security situation awareness analysis technique for a large-scale database cluster according to an embodiment of the present application, including:
s100, packet capturing is carried out on a database communication data packet;
specifically, the data packet capturing and analyzing submodule receives the data stream through the network card to recombine the data packet, analyzes the TCP/IP message after recombination to extract the information of the database communication protocol, analyzes the database communication protocol to extract the SQL statement, and then forwards the SQL statement to the real-time stream processing submodule.
In some embodiments, a database is a collection of large amounts of data that are stored long term within a computer, organized, and sharable, and may be a database cluster that includes database node 1, database node 2, and database node 3. The message can be a TCP/IP message, the TCP is a connection-oriented, reliable and byte stream-based transmission layer communication protocol for dividing the data stream of the application layer into message segments, the segment length of the message segment is influenced by the MTU and is sent to the TCP layer of the target node, the data packets are numbered in sequence, the other side sends ACK and confirms characters when receiving the ACK and retransmits the data if not receiving the ACK and checks whether the data has errors in the transmission process by using a checksum.
Step S110, analyzing data flow of the captured data packet;
specifically, the real-time stream processing submodule performs stream processing on the SQL after receiving the data, and cleans and converts the stream data. And data preprocessing is carried out on the characteristics, and information is labeled.
In some embodiments, the IP information may be parsed out in the captured packets. A data packet is a unit of information whose origin and destination are the network layers.
Step S120, sending the analysis result to a real-time stream processing engine for processing and feature extraction to obtain real-time information;
specifically, the feature extraction sub-module extracts features after receiving the tag data, and in some embodiments, the features include, but are not limited to, a source IP, a destination IP, a client name, an access account, a database name, a table field name, and the like.
In some embodiments, the stream is processed as stream data, which may be a set of sequential, large, fast, continuous arriving data sequences, which may be viewed as a dynamic data set that grows indefinitely over time.
Step S130, buffering and forwarding real-time information by a message queue;
specifically, the message queue buffering and forwarding submodule receives the real-time message through the kafka and forwards the real-time message at regular time through the memory buffering mechanism, so that the IO pressure of the back-end data warehouse is reduced.
In some embodiments, Kafka is an open source stream processing platform developed by the Apache software foundation, which aims to unify message processing both online and offline through a parallel load mechanism, providing buffering for underlying storage.
Step S140, receiving the real-time information and storing the real-time information in a distributed data warehouse;
specifically, the distributed data warehouse submodule is mainly used for the ArgoDB distributed database to undertake data storage tasks, and firstly, the real-time information forwarded by Kafka is solidified and stored in a two-dimensional table mode.
In some embodiments, ArgoDB is a flash and memory based distributed database that can provide distributed storage of data as well as provide MLOAP. Distributed storage: the files are respectively stored in different nodes in a multi-copy mode to ensure the safety of data, and the data extraction speed and the data storage capacity are improved.
Step S150, performing distributed computation on the real-time information;
specifically, the distributed computation submodule is mainly used for undertaking computation tasks through a Spark distributed computation engine, computing the computation tasks through a Spark by a built-in strategy engine scheduling algorithm, and forwarding a computation result.
In some embodiments, distributed computing breaks the application into many small portions, which are distributed to multiple computers for processing. Therefore, the overall calculation time can be saved, and the calculation efficiency is greatly improved.
Step S160, distributing the calculation result to a theme database for storage;
specifically, the LOAP database submodule is a database provided with an online analysis process, and is mainly assumed by ArgoDB, stores a calculation result in a memory, and generates a multidimensional table, that is, a cube table, according to a set condition.
In some embodiments, the LOAP database is used with a large amount of data, tens of TBs being small. In contrast, the number of nodes supported by the traditional MPP and the total data amount processed by the traditional MPP are smaller than those of a Hadoop ecology, and the traditional MPP has high-level analysis capability including high-level bin functions, machine learning, integrated R language and the like. Ecological integrity, OLAP/BI ecology has many mature software, and seamless access is an important capability.
And S170, pulling the calculation result of the theme database to perform BI analysis and display.
Specifically, through the BI analysis submodule, the engine adopts a Pilot engine to undertake analysis tasks, and the Pilot engine can be used for quickly retrieving and visually displaying a multidimensional table in a calculation LOAP database, so that situation perception of the security state of the database cluster is finally achieved.
Example 2
Referring to fig. 2, fig. 2 is a schematic diagram of a logical structure of a security situation awareness analysis technique for a large-scale database cluster according to an embodiment of the present invention, including: the system comprises a database cluster, a distributed log collector cluster and a data security situation perception analysis system;
the database cluster comprises a plurality of database nodes;
the distributed log collector cluster comprises a plurality of log collection modules;
the log collection module comprises four sub-modules, namely a data packet capture and analysis sub-module, a real-time stream processing sub-module, a feature extraction sub-module and a message queue buffering and forwarding sub-module
The data security situation perception analysis system comprises four sub-modules, namely a distributed data warehouse sub-module, a distributed calculation sub-module, an LOAP database sub-module and a BA analysis display sub-module;
example 3
Referring to fig. 3, fig. 3 is a schematic diagram of a security situation awareness analysis technique module for a large-scale database cluster according to an embodiment of the present invention, including: the system comprises a log acquisition module and a security situation analysis module; the log acquisition module realizes acquisition and forwarding of database security logs and comprises the following sub-modules: the data packet capturing and analyzing submodule is used for capturing the data packet of the database communication; the real-time stream processing submodule is used for analyzing the data flow of the captured data packet; the feature extraction submodule is used for sending the analysis result to a real-time stream processing engine for processing and feature extraction to obtain real-time information; the message queue buffering and forwarding submodule is used for buffering and forwarding the real-time information through the message queue; the security situation analysis module realizes the analysis and display of the log and comprises the following sub-modules: the distributed data warehouse submodule is used for receiving the real-time information and storing the real-time information in a distributed data warehouse; the distributed computation submodule is used for carrying out distributed computation on the real-time information; the LOAP database submodule is used for distributing the calculation result to the subject database for storage; and the BI analysis submodule is used for carrying out BI analysis display by pulling the calculation result of the subject database.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In summary, the security situation awareness analysis technology and system for large-scale database clusters provided by the embodiments of the present application mainly solve the problem that the conventional database auditing technology cannot provide unified log collection, unified log storage, large-scale data calculation, and secure dynamic visual presentation for very large database clusters. The method is based on the adoption of a big data technology architecture, and solves the defects of the traditional log audit and the database audit through providing heterogeneous information acquisition, distributed storage, distributed computation and BI analysis through different components in the Hadoopd big data architecture. Safety audit of the large-scale database cluster is realized, multi-dimensional real-time database safety analysis is provided for massive safety audit logs through distributed calculation and rich safety algorithms, and whole safety situation perception visual display of the large-scale database cluster is realized.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Claims (10)
1. A security situation awareness analysis technique for large-scale database clusters is characterized by comprising the following steps:
capturing a database communication data packet;
analyzing the data flow of the captured data packet;
sending the analysis result to a real-time stream processing engine for processing and feature extraction to obtain real-time information;
buffering and forwarding real-time information by the message queue;
receiving real-time information and storing the real-time information in a distributed number bin;
performing distributed calculation on the real-time information;
distributing the calculation result to a theme database for storage;
and pulling the calculation result of the topic database for BI analysis and display.
2. A security posture awareness analysis technique for large scale database clusters as claimed in claim 1 wherein said bale catching database communication data packets comprises:
and receiving a data stream through the network card to recombine the data packet, analyzing the TCP/IP message after the data packet is recombined to extract the information of the database communication protocol, analyzing the database communication protocol to extract the SQL statement, and forwarding the SQL statement.
3. The technology of claim 1, wherein the data traffic parsing of the captured data packets comprises:
after receiving the data, carrying out stream processing on the SQL, and cleaning and converting the stream data;
and carrying out data preprocessing on the characteristics and labeling the information.
4. The technology for security posture awareness analysis of large-scale database clusters according to claim 1, wherein the sending the analysis result to a real-time stream processing engine for processing and feature extraction of real-time information comprises:
the features are extracted after the tag data is received.
5. The security posture aware analysis technique for large scale database clusters as claimed in claim 1, wherein said message queue buffering and forwarding real-time information comprises:
and receiving the real-time information through kafka and forwarding the real-time information at regular time through a memory buffer mechanism.
6. The security posture awareness analysis technique for large scale database clusters of claim 1, wherein the receiving and storing real-time information in distributed bins comprises:
the ArgoDB distributed database undertakes the data storage task, and the information forwarded by Kafka is solidified and stored in a two-dimensional table mode.
7. A security posture awareness analysis technique for large scale database clusters as claimed in claim 1 wherein said performing distributed computation comprises:
the Spark distributed computing engine undertakes computing tasks, a built-in strategy engine scheduling algorithm is used for computing and forwarding a computing result.
8. The security posture awareness analysis technique for large scale database clusters of claim 1, wherein the distributing the computed results to a subject database store comprises:
and storing the calculation result in a memory, and generating a multidimensional table according to a set condition.
9. The technology of claim 1, wherein the pulling of the computation results of the subject database for BI analysis presentation comprises:
the multidimensional tables in the calculation LOAP database can be quickly retrieved and visually displayed through the Pilot engine.
10. A security posture aware analytics system for large-scale database clusters, comprising: the system comprises a log acquisition module and a security situation analysis module;
the log acquisition module realizes acquisition and forwarding of database security logs and comprises the following sub-modules:
the data packet capturing and analyzing submodule is used for capturing the data packet of the database communication;
the real-time stream processing submodule is used for analyzing the data flow of the captured data packet;
the feature extraction submodule is used for sending the analysis result to a real-time stream processing engine for processing and feature extraction to obtain real-time information;
the message queue buffering and forwarding submodule is used for buffering and forwarding the real-time information through the message queue;
the security situation analysis module realizes the analysis and display of the log and comprises the following sub-modules:
the distributed data warehouse submodule is used for receiving the real-time information and storing the real-time information in a distributed data warehouse;
the distributed computation submodule is used for carrying out distributed computation on the real-time information;
the LOAP database submodule is used for distributing the calculation result to the subject database for storage;
and the BI analysis submodule is used for carrying out BI analysis display by pulling the calculation result of the subject database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010713576.8A CN112035828B (en) | 2020-07-22 | 2020-07-22 | Security situation awareness analysis method and system for large-scale database cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010713576.8A CN112035828B (en) | 2020-07-22 | 2020-07-22 | Security situation awareness analysis method and system for large-scale database cluster |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112035828A true CN112035828A (en) | 2020-12-04 |
| CN112035828B CN112035828B (en) | 2024-04-30 |
Family
ID=73582934
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010713576.8A Active CN112035828B (en) | 2020-07-22 | 2020-07-22 | Security situation awareness analysis method and system for large-scale database cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112035828B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113055493A (en) * | 2021-03-26 | 2021-06-29 | 广州虎牙科技有限公司 | Data packet processing method, device, system, scheduling device and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070033233A1 (en) * | 2005-08-05 | 2007-02-08 | Hwang Min J | Log management system and method of using the same |
| US7809826B1 (en) * | 2005-01-27 | 2010-10-05 | Juniper Networks, Inc. | Remote aggregation of network traffic profiling data |
| CN107908651A (en) * | 2017-10-12 | 2018-04-13 | 北京人大金仓信息技术股份有限公司 | A kind of auditing method of distributed type assemblies |
| CN108039959A (en) * | 2017-11-29 | 2018-05-15 | 深信服科技股份有限公司 | Situation Awareness method, system and the relevant apparatus of a kind of data |
| CN108769048A (en) * | 2018-06-08 | 2018-11-06 | 武汉思普崚技术有限公司 | A kind of secure visualization and Situation Awareness plateform system |
| CN110493043A (en) * | 2019-08-16 | 2019-11-22 | 武汉思普崚技术有限公司 | A kind of distribution Situation Awareness call method and device |
| CN111190876A (en) * | 2019-12-31 | 2020-05-22 | 天津浪淘科技股份有限公司 | Log management system and operation method thereof |
-
2020
- 2020-07-22 CN CN202010713576.8A patent/CN112035828B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7809826B1 (en) * | 2005-01-27 | 2010-10-05 | Juniper Networks, Inc. | Remote aggregation of network traffic profiling data |
| US20070033233A1 (en) * | 2005-08-05 | 2007-02-08 | Hwang Min J | Log management system and method of using the same |
| CN107908651A (en) * | 2017-10-12 | 2018-04-13 | 北京人大金仓信息技术股份有限公司 | A kind of auditing method of distributed type assemblies |
| CN108039959A (en) * | 2017-11-29 | 2018-05-15 | 深信服科技股份有限公司 | Situation Awareness method, system and the relevant apparatus of a kind of data |
| CN108769048A (en) * | 2018-06-08 | 2018-11-06 | 武汉思普崚技术有限公司 | A kind of secure visualization and Situation Awareness plateform system |
| CN110493043A (en) * | 2019-08-16 | 2019-11-22 | 武汉思普崚技术有限公司 | A kind of distribution Situation Awareness call method and device |
| CN111190876A (en) * | 2019-12-31 | 2020-05-22 | 天津浪淘科技股份有限公司 | Log management system and operation method thereof |
Non-Patent Citations (3)
| Title |
|---|
| 曾彬;张文沛;: "大数据驱动的网络综合监测系统的设计与实现", 信息技术与网络安全, no. 02 * |
| 王海涛;: "基于大数据和人工智能技术的信息安全态势感知系统研究", 网络安全技术与应用, no. 03 * |
| 王路遥: "机器学习 构建智能网安主动防御体系", 上海信息化, no. 05, pages 25 - 26 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113055493A (en) * | 2021-03-26 | 2021-06-29 | 广州虎牙科技有限公司 | Data packet processing method, device, system, scheduling device and storage medium |
| CN113055493B (en) * | 2021-03-26 | 2023-04-07 | 广州虎牙科技有限公司 | Data packet processing method, device, system, scheduling device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112035828B (en) | 2024-04-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104331435B (en) | A kind of efficient mass data abstracting method of low influence based on Hadoop big data platforms | |
| CN109151078B (en) | Distributed intelligent mail analysis and filtering method, system and storage medium | |
| CN106815338A (en) | A kind of real-time storage of big data, treatment and inquiry system | |
| US11080294B1 (en) | Systems and methods for data analytics | |
| US11301425B2 (en) | Systems and computer implemented methods for semantic data compression | |
| CN111459986B (en) | Data computing system and method | |
| CN101777064A (en) | Image searching system and method | |
| CN110362544A (en) | Log processing system, log processing method, terminal and storage medium | |
| CN107515878A (en) | The management method and device of a kind of data directory | |
| CN108073625A (en) | For the system and method for metadata information management | |
| CN107451208A (en) | A kind of data search method and device | |
| CN112035828B (en) | Security situation awareness analysis method and system for large-scale database cluster | |
| Lu et al. | The design and implementation of configurable news collection system based on web crawler | |
| CN117591532A (en) | Mass data management system of distributed energy system and application method thereof | |
| Maske et al. | A real time processing and streaming of wireless network data using storm | |
| CN111209314A (en) | System for processing massive log data of power information system in real time | |
| CN114168672B (en) | Log data processing method, device, system and medium | |
| CN113821361B (en) | Internet of things platform message processing method and system based on streaming processing | |
| KR101656011B1 (en) | System and method for fault monitoring based on big-data | |
| Phanikanth et al. | A big data perspective of current ETL techniques | |
| CN111078987A (en) | Similar data extraction and arrangement system based on Internet | |
| Deng et al. | Flight test data processing and analysis platform based on new generation information technology Design and Application | |
| Du et al. | A Real-time Big Data Framework for Network Security Situation Monitoring. | |
| CN116610531B (en) | Method for collecting data embedded points and requesting image uploading data based on code probe | |
| CA2803491A1 (en) | Data collection device for monitoring streams in a data network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |