CN112882808B - Method for collecting and transmitting big data audit log of application delivery equipment - Google Patents
Method for collecting and transmitting big data audit log of application delivery equipment Download PDFInfo
- Publication number
- CN112882808B CN112882808B CN202110170542.3A CN202110170542A CN112882808B CN 112882808 B CN112882808 B CN 112882808B CN 202110170542 A CN202110170542 A CN 202110170542A CN 112882808 B CN112882808 B CN 112882808B
- Authority
- CN
- China
- Prior art keywords
- log
- service
- data
- processed
- flow
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000012550 audit Methods 0.000 title claims abstract description 19
- 230000006835 compression Effects 0.000 claims abstract description 18
- 238000007906 compression Methods 0.000 claims abstract description 18
- 230000008569 process Effects 0.000 claims abstract description 13
- 238000005111 flow chemistry technique Methods 0.000 claims abstract description 8
- 239000000284 extract Substances 0.000 claims abstract description 4
- 230000005540 biological transmission Effects 0.000 claims description 6
- 238000005516 engineering process Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000009545 invasion Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/48—Indexing scheme relating to G06F9/48
- G06F2209/484—Precedence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method for collecting and sending big data audit logs of application delivery equipment, which uniformly distributes different CPUs for different service data flows according to service data characteristics, wherein each service data flow and the generated log thereof can only be processed by one CPU; the service flow processing module extracts the required data and writes the data into a log cache queue; after the service flow processing module sends the data message, checking a log buffer queue, and scheduling the log compression sending module to take out the log compression from the log buffer queue and send the log compression after the log buffer reaches the expected configuration; after the log is sent, if the service flow exists, the service flow is processed preferentially, otherwise, the log is sent continuously; the invention distributes different CPUs uniformly through different service data flows, the same service can only be processed by one CPU for forwarding and journaling, and adjusts and distributes the CPU to process service data or process journaling by caching journaling and setting an alarm value for caching journaling, thereby ensuring low delay of service and preventing journaling from losing.
Description
Technical Field
The invention relates to the technical field of communication, in particular to a method for collecting and sending big data audit logs of application delivery equipment.
Background
Application delivery devices are key components of a high availability network infrastructure, often used to distribute workloads to multiple servers to improve the performance and security of services such as websites, applications, and databases.
Under the network structure without application delivery equipment, if a server is down or the service access amount is too large, the use experience of a user can be influenced, the server is directly exposed to the Internet, and if the server has security holes, a hacker can easily invade the server to cause immeasurable loss for enterprises.
Under the network structure of the application delivery device, the user accesses the application delivery device, the application delivery device processes the user request, and forwards the user request to a proper server for processing according to the user request and the background server state mounted by the application delivery device, wherein the technology relates to server health detection, address conversion and service encryption and decryption, and the technology ensures the availability and the high efficiency of the user request and simultaneously ensures the safety of the server.
Because the application delivery device modifies data, such as a source address source port and a destination address destination port of TCP data, a server cannot see real user addresses and ports and cannot see service destination addresses and ports accessed by users, audit log tracking of conversion of the addresses and ports is helpful to positioning problems or tracking hacking invasion, a high-performance application delivery device can receive up to 1000 ten thousand TCP requests per second, each TCP request can establish a new session, the new session can record how to convert the source addresses, the source ports, the destination addresses and the destination ports of data streams, each new session needs to generate an audit log, and how to collect and send a large amount of logs is a technical problem.
The current mainstream technology is that the log is sent to a log server through a network after being generated, and the bottleneck is that the application delivery device occupies a large amount of device memory and processor resources and interface bandwidth for collecting and sending the log.
The existing main stream techniques for collecting and sending audit logs are two, one is that a common data communication equipment manufacturer makes a method, after an operation device generates the audit logs, a syslog general interface is called, log information is transferred to a log processing process through a Socket, the log processing process processes the logs, the logs can be stored in a local hard disk or can be sent to a log server, and if the log generation amount is too large, the log loss condition exists; the other is a high-speed log processing mode, namely, the service data and the log are processed simultaneously, the log is generated simultaneously in the newly built service data flow, and the log is sent to the log server through the service data port.
Disadvantages of the prior art:
1. and the problem of log loss is that if the number of newly-built service sessions of the equipment is too large, a large number of logs are accumulated and overflow the cache area, so that the logs are lost.
2. The service processing delay is affected, and new sessions are processed each time, and meanwhile, logs are generated and sent, so that the service delay is caused.
Once the device opens the audit log function, the user may feel delayed or occasional service failure in processing the service when the user request peaks above 70% of the device's processing capacity.
In the prior art, a mature API interface is generally used, and when the generated log and the transmitted log are triggered according to the service flow, the CPU processing state is not considered, so that the problem of log loss and service data processing delay or failure occurs at the moment of CPU processing peak value.
Therefore, the method for collecting and sending the big data audit log of the application delivery equipment becomes a urgent problem to be solved.
Disclosure of Invention
The invention aims to realize the effective log processing by utilizing the processing capacity of the CPU of the equipment and the minimum influence on the service flow by intelligently adjusting the CPU call to the audit log and the data service.
In order to achieve the above purpose, the technical scheme provided by the invention is as follows: the method for collecting and transmitting the big data audit log of the application delivery equipment comprises the following steps:
step 1: according to the service data characteristics, different CPUs are uniformly distributed to different service data flows, each service data flow can only be processed by one CPU, and logs generated by each data flow can only be processed by the CPU for processing the data flow;
step 2: the log is put into a buffer queue, and the CPU is intelligently scheduled to process the log buffer queue according to the processing requirement of the service data stream;
step 3: after analyzing the data message, the service flow processing module extracts the required data and writes the required data into a log cache queue;
step 4: after the service flow processing module sends the data message, checking a log buffer queue, and if the log buffer reaches the expected configuration, scheduling the log compression sending module to take out the log compression from the log buffer queue and send the log compression;
step 5: judging whether the log contains service flow after the log compression and transmission, if so, preferentially processing the service flow, otherwise, scheduling the log compression and transmission module to continue transmitting the log.
As an improvement, before the log reaches the buffer stop value, the service flow is processed preferentially, and if no service request exists, the log buffer queue is processed.
As an improvement, the log reaches a buffer memory warning, and the log is processed preferentially.
As an improvement, after each processing of the log buffer queue, it is checked whether there is a service request to ensure low latency of the service.
Compared with the prior art, the invention has the advantages that: the invention can only process the forwarding and sending logs by one CPU through the same service flow, and can adjust and allocate the CPU to process the service data or process the logs by caching the logs and setting the warning value for the cached logs, thereby ensuring low delay of the service and preventing the logs from losing.
Drawings
FIG. 1 is a flow chart of a method of collecting and transmitting big data audit logs for an application delivery device of the present invention.
Detailed Description
The method for collecting and transmitting big data audit logs of the application delivery equipment is further described in detail below with reference to the accompanying drawings.
Referring to fig. 1, the method for collecting and sending big data audit logs of the application delivery equipment comprises the following specific implementation processes:
according to the service data characteristics, different CPUs are uniformly distributed to different service data flows, each service data flow can only be processed by one CPU, and logs generated by each data flow can only be processed by the CPU for processing the data flow;
the log is put into a buffer queue, and the CPU is intelligently scheduled to process the log buffer queue according to the condition that the service data processing needs CPU resources;
after analyzing the data message, the service flow processing module extracts the required data and writes the required data into a log cache queue;
after the service flow processing module sends the data message, checking a log buffer queue, and if the log buffer reaches the expected configuration, scheduling the log compression sending module to take out the log compression from the log buffer queue and send the log compression;
judging whether the log contains service flow after the log compression and transmission, if so, preferentially processing the service flow, otherwise, scheduling the log compression and transmission module to continue transmitting the log.
Different CPUs handle different traffic flows, the same traffic flow being not handled across CPUs.
Before the log reaches the forbidden value, the service flow is processed preferentially, and if no service request exists, the log is processed to a log buffer queue.
The log reaches the buffer memory warning, and the log is processed preferentially.
After each processing of the log buffer queue, it is checked whether there is a service request to ensure low latency of the service.
The invention and its embodiments have been described above with no limitation, and the actual construction is not limited to the embodiments of the invention as shown in the drawings. In summary, if one of ordinary skill in the art is informed by this disclosure, a structural manner and an embodiment similar to the technical solution should not be creatively devised without departing from the gist of the present invention.
Claims (4)
1. The method for collecting and transmitting the big data audit log of the application delivery equipment is characterized by comprising the following steps:
step 1: according to the service data characteristics, different CPUs are uniformly distributed to different service data flows, each service data flow can only be processed by one CPU, and logs generated by each data flow can only be processed by the CPU for processing the data flow;
step 2: the log is put into a buffer queue, and the CPU is intelligently scheduled to process the log buffer queue according to the processing requirement of the service data stream;
step 3: after analyzing the data message, the service flow processing module extracts the required data and writes the required data into a log cache queue;
step 4: after the service flow processing module sends the data message, checking a log buffer queue, and if the log buffer reaches the expected configuration, scheduling the log compression sending module to take out the log compression from the log buffer queue and send the log compression;
step 5: judging whether the log contains service flow after the log compression and transmission, if so, preferentially processing the service flow, otherwise, scheduling the log compression and transmission module to continue transmitting the log.
2. The method for collecting and sending big data audit logs of an application delivery device according to claim 1 wherein: before the log reaches the buffer stop value, the service flow is processed preferentially, and if no service request exists, the log buffer queue is processed.
3. The method for collecting and sending big data audit logs of an application delivery device according to claim 1 wherein: and the log achieves buffer memory warning, and the log is processed preferentially.
4. The method for collecting and sending big data audit logs of an application delivery device according to claim 1 wherein: after each processing of the log buffer queue, it is checked whether there is a service request to ensure low latency of the service.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110170542.3A CN112882808B (en) | 2021-02-08 | 2021-02-08 | Method for collecting and transmitting big data audit log of application delivery equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110170542.3A CN112882808B (en) | 2021-02-08 | 2021-02-08 | Method for collecting and transmitting big data audit log of application delivery equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112882808A CN112882808A (en) | 2021-06-01 |
| CN112882808B true CN112882808B (en) | 2023-10-24 |
Family
ID=76057544
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110170542.3A Active CN112882808B (en) | 2021-02-08 | 2021-02-08 | Method for collecting and transmitting big data audit log of application delivery equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112882808B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101763593A (en) * | 2009-12-17 | 2010-06-30 | 中国电力科学研究院 | Method and device for realizing audit log of system |
| US8407335B1 (en) * | 2008-06-18 | 2013-03-26 | Alert Logic, Inc. | Log message archiving and processing using a remote internet infrastructure |
| CN103729442A (en) * | 2013-12-30 | 2014-04-16 | 华为技术有限公司 | Method for recording event logs and database engine |
| CN105119752A (en) * | 2015-09-08 | 2015-12-02 | 北京京东尚科信息技术有限公司 | Distributed log acquisition method, device and system |
| CN105991346A (en) * | 2015-04-28 | 2016-10-05 | 杭州迪普科技有限公司 | Session log processing method and device |
| US9600553B1 (en) * | 2014-05-31 | 2017-03-21 | Veritas Technologies Llc | Distributed replication in cluster environments |
| CN107135088A (en) * | 2016-02-29 | 2017-09-05 | 华为技术有限公司 | The method and apparatus that daily record is handled in cloud computing system |
| CN107249019A (en) * | 2017-05-26 | 2017-10-13 | 北京小米移动软件有限公司 | Data handling system, method, device and server based on business |
| CN107590182A (en) * | 2017-08-03 | 2018-01-16 | 华南理工大学 | A kind of distributed information log collection method |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2008108227A (en) * | 2006-09-25 | 2008-05-08 | Hitachi Ltd | Storage system and audit log management method |
| US9965359B2 (en) * | 2014-11-25 | 2018-05-08 | Sap Se | Log forwarding to avoid deadlocks during parallel log replay in asynchronous table replication |
| US10685034B2 (en) * | 2017-10-17 | 2020-06-16 | Salesforce.Com, Inc. | Systems, methods, and apparatuses for implementing concurrent dataflow execution with write conflict protection within a cloud based computing environment |
-
2021
- 2021-02-08 CN CN202110170542.3A patent/CN112882808B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8407335B1 (en) * | 2008-06-18 | 2013-03-26 | Alert Logic, Inc. | Log message archiving and processing using a remote internet infrastructure |
| CN101763593A (en) * | 2009-12-17 | 2010-06-30 | 中国电力科学研究院 | Method and device for realizing audit log of system |
| CN103729442A (en) * | 2013-12-30 | 2014-04-16 | 华为技术有限公司 | Method for recording event logs and database engine |
| US9600553B1 (en) * | 2014-05-31 | 2017-03-21 | Veritas Technologies Llc | Distributed replication in cluster environments |
| CN105991346A (en) * | 2015-04-28 | 2016-10-05 | 杭州迪普科技有限公司 | Session log processing method and device |
| CN105119752A (en) * | 2015-09-08 | 2015-12-02 | 北京京东尚科信息技术有限公司 | Distributed log acquisition method, device and system |
| CN107135088A (en) * | 2016-02-29 | 2017-09-05 | 华为技术有限公司 | The method and apparatus that daily record is handled in cloud computing system |
| CN107249019A (en) * | 2017-05-26 | 2017-10-13 | 北京小米移动软件有限公司 | Data handling system, method, device and server based on business |
| CN107590182A (en) * | 2017-08-03 | 2018-01-16 | 华南理工大学 | A kind of distributed information log collection method |
Non-Patent Citations (3)
| Title |
|---|
| A scalable, high-performance customized priority queue;Muhuan Huang et.al.;2014 24th International Conference on Field Programmable Logic and Applications (FPL);全文 * |
| 基于Spark Streaming的海量日志实时处理系统的设计;陆世鹏;;电子产品可靠性与环境试验(05);全文 * |
| 基于图的大规模日志处理系统关键技术研究;艾智远;中国优秀硕士学位论文全文数据库 (信息科技辑);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112882808A (en) | 2021-06-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108512885B (en) | Method, system and storage medium for network packet processing | |
| US10666522B2 (en) | Server side content delivery network quality of service | |
| US7747662B2 (en) | Service aware network caching | |
| JP5189974B2 (en) | Load control device and method thereof | |
| US8135850B2 (en) | Systems and methods for load balancing real time streaming | |
| US9794282B1 (en) | Server with queuing layer mechanism for changing treatment of client connections | |
| US9037712B2 (en) | Systems and methods for self-loading balancing access gateways | |
| US9356844B2 (en) | Efficient application recognition in network traffic | |
| US20020007374A1 (en) | Method and apparatus for supporting a multicast response to a unicast request for a document | |
| CA2355286A1 (en) | Optimizing bandwidth consumption for document distribution over a multicast enabled wide area network | |
| WO2018121742A1 (en) | Method and device for transmitting stream data | |
| US20170214625A1 (en) | System and method of providing increased data optimization based on traffic priority on connection | |
| US20160021188A1 (en) | Generic Network Trace with Distributed Parallel Processing and Smart Caching | |
| CN108989420B (en) | Method and system for registering service and method and system for calling service | |
| US20210377294A1 (en) | Constraining resource allocation rate for stateful multi-tenant http proxies and denial-of-service attack prevention | |
| CN116418893A (en) | Data management method and corresponding device | |
| CN112882808B (en) | Method for collecting and transmitting big data audit log of application delivery equipment | |
| CN110855726A (en) | Communication method, communication device, gateway, computing device and medium | |
| US11528187B1 (en) | Dynamically configurable networking device interfaces for directional capacity modifications | |
| Shimano et al. | An information propagation scheme for an autonomous distributed storage system in iSCSI environment | |
| US20190068621A1 (en) | User access rate limiting among content delivery nodes | |
| US20240267406A1 (en) | Bridging between client and server devices using proxied network metrics | |
| Williamson | Dynamic transport-level connection management in a distributed system | |
| Bardinelli et al. | hyDNS: Acceleration of DNS Through Kernel Space Resolution | |
| Ghasemi | Data-Driven Management of CDN Performance |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |