CN112882808A - Method for collecting and sending big data audit log of application delivery equipment - Google Patents
Method for collecting and sending big data audit log of application delivery equipment Download PDFInfo
- Publication number
- CN112882808A CN112882808A CN202110170542.3A CN202110170542A CN112882808A CN 112882808 A CN112882808 A CN 112882808A CN 202110170542 A CN202110170542 A CN 202110170542A CN 112882808 A CN112882808 A CN 112882808A
- Authority
- CN
- China
- Prior art keywords
- log
- service
- data
- processed
- sending
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000012550 audit Methods 0.000 title claims abstract description 19
- 230000006835 compression Effects 0.000 claims abstract description 14
- 238000007906 compression Methods 0.000 claims abstract description 14
- 238000005111 flow chemistry technique Methods 0.000 claims abstract description 8
- 239000000284 extract Substances 0.000 claims abstract description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/48—Indexing scheme relating to G06F9/48
- G06F2209/484—Precedence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method for collecting and sending big data audit logs of application delivery equipment, which evenly distributes different CPUs (central processing units) to different service data streams according to service data characteristics, wherein each service data stream and a log generated by each service data stream can be processed by only one CPU; the service flow processing module extracts the required data and writes the data into a log cache queue; after the service flow processing module sends the data message, a log cache queue is checked, and when the log cache reaches the expected configuration, a log compression sending module is scheduled to take out log compression from the log cache queue and send the compressed log; after the log is sent, if a service flow exists, the service flow is processed preferentially, otherwise, the log is sent continuously; the invention distributes different CPUs uniformly through different service data streams, the same service can be processed by only one CPU for forwarding and logging, and regulates whether the CPU is distributed to process the service data or the log through caching the log and setting an alarm value for the caching log, thereby ensuring low service delay and preventing the log from losing.
Description
Technical Field
The invention relates to the technical field of communication, in particular to a method for collecting and sending big data audit logs of application delivery equipment.
Background
Application delivery devices are a key component of the highly available network infrastructure and are commonly used to distribute workloads across multiple servers to improve the performance and security of services such as websites, applications, and databases.
Under the network structure without application delivery equipment, if a server is down or the service access amount is too large, the use experience of a user is influenced, the server is directly exposed under the Internet, and if the server has security holes, hackers can easily invade the server, so that immeasurable loss is caused to enterprises.
Under the network structure with the application delivery device, a user accesses the application delivery device, the application delivery device processes a user request, and forwards the user request to a proper server for processing according to the user request and the background server state mounted by the application delivery device.
Because the application delivery device modifies data, such as a source address source port and a destination address destination port of TCP data, a server cannot see a real user address and port, and a service destination address and port accessed by a user, an audit log tracks the conversion of the addresses and ports, which is helpful for positioning problems or tracking hacking intrusion, a high-performance application delivery device can receive up to 1000 ten thousand TCP requests per second, each TCP request can create a new session, the new session can record how to convert the source address, the destination address and the destination port of a data stream, each new session needs to generate an audit log, and how to collect and send the large data volume is a technical problem.
The current mainstream technology is that the generated log is sent to a log server through a network, and the bottleneck is that the application delivery device can occupy a large amount of device memory, processor resources and interface bandwidth for collecting and sending the log.
The method comprises the following steps that two main flow technologies of collection and sending of the existing audit logs are provided, one technology is a method of a general data communication equipment manufacturer, after an operation device generates the audit logs, a syslog universal interface is called, log information is transmitted to a log processing process through Socket, the logs are processed by the log processing process, the logs can be stored in a local hard disk or sent to a log server, and if the log generation amount is too large, the log loss condition exists; the other is a high-speed log processing mode, namely simultaneously processing the service data and the log, simultaneously generating the log when a service data stream is newly established, and sending the log to a log server through a service data port.
The prior art has the following disadvantages:
1. the problem of log loss is solved, if the device has too many new service sessions, a large amount of logs can be accumulated and overflow a cache area, and the logs are lost.
2. The service processing delay is affected, and each time a new session is processed, logs are generated and sent, which causes service delay.
Once the device starts the audit log function, when the peak value of the user request reaches more than 70% of the processing capacity of the device, the user feels that a delay or occasional service failure occurs when the service is processed.
The prior art generally uses a mature API interface, and the problems of log loss and service data processing delay or failure occur at the moment of generating logs and sending logs to trigger according to service flow without considering the CPU processing state and causing CPU processing peak values.
Therefore, the collection and sending method of the big data audit log of the application delivery equipment becomes a problem to be solved urgently.
Disclosure of Invention
The invention aims to realize the effective log processing and the minimum influence on the service flow by efficiently utilizing the CPU processing capacity of equipment by intelligently adjusting the CPU call of the audit log and the data service.
In order to achieve the purpose, the technical scheme provided by the invention is as follows: the method for collecting and sending the big data audit log of the application delivery equipment comprises the following steps:
step 1: according to the service data characteristics, different CPUs are uniformly distributed to different service data streams, each service data stream can be processed by only one CPU, and a log generated by each data stream can be processed by only the CPU which processes the data stream;
step 2: the logs are put into a cache queue, and a CPU is intelligently scheduled to process the log cache queue according to the processing requirement of the service data stream;
and step 3: after analyzing the data message, the service flow processing module extracts the required data and writes the data into a log cache queue;
and 4, step 4: after the service flow processing module sends the data message, the log cache queue is checked, and if the log cache reaches the expected configuration, the log compression sending module is dispatched to take out the log compression from the log cache queue and send the log compression;
and 5: after the log is compressed and sent, whether the service flow is contained or not is judged, if the service flow is contained, the service flow is processed preferentially, and otherwise, the log compression sending module is scheduled to continue sending the log.
As an improvement, before the log reaches the value of the buffer forbidden ring, the service flow is processed preferentially, and if no service request exists, the log buffer queue is processed.
As an improvement, the log reaches a cache alert, and the log is processed preferentially.
As an improvement, after the log buffer queue is processed each time, whether a service request exists or not needs to be checked, so that low delay of the service is ensured.
Compared with the prior art, the invention has the advantages that: the invention can process the forwarding and sending logs by only one CPU through the same service flow, and adjusts and allocates the CPU to process the service data or the logs through caching the logs and setting warning values for the caching logs, thereby ensuring low service delay and preventing log loss.
Drawings
FIG. 1 is a flow chart of a method for collecting and sending big data audit logs of an application delivery device according to the present invention.
Detailed Description
The method for collecting and sending big data audit logs of the delivery device according to the present invention is further described in detail with reference to the accompanying drawings.
With reference to fig. 1, the specific implementation process of the method for collecting and sending big data audit logs of delivery equipment according to the present invention is as follows:
according to the service data characteristics, different CPUs are uniformly distributed to different service data streams, each service data stream can be processed by only one CPU, and a log generated by each data stream can be processed by only the CPU which processes the data stream;
the logs are put into a cache queue, and a CPU is intelligently scheduled to process the log cache queue according to the condition that the service data processing needs CPU resources;
after analyzing the data message, the service flow processing module extracts the required data and writes the data into a log cache queue;
after the service flow processing module sends the data message, the log cache queue is checked, and if the log cache reaches the expected configuration, the log compression sending module is dispatched to take out the log compression from the log cache queue and send the log compression;
after the log is compressed and sent, whether the service flow is contained or not is judged, if the service flow is contained, the service flow is processed preferentially, and otherwise, the log compression sending module is scheduled to continue sending the log.
Different CPUs process different service flows, and the same service flow cannot be processed by the cross-CPU.
And (4) before the log reaches the buffer abstinence value, processing the service flow preferentially, and if no service request exists, processing the log buffer queue.
And (4) the log reaches a cache warning, and the log is processed preferentially.
After each log buffer queue is processed, whether a service request exists is checked to ensure low delay of the service.
The present invention and its embodiments have been described above, and the description is not intended to be limiting, and the drawings are only one embodiment of the present invention, and the actual structure is not limited thereto. In summary, those skilled in the art should appreciate that they can readily use the disclosed conception and specific embodiments as a basis for designing or modifying other structures for carrying out the same purposes of the present invention without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (4)
1. The method for collecting and sending the big data audit log of the application delivery equipment is characterized by comprising the following steps of:
step 1: according to the service data characteristics, different CPUs are uniformly distributed to different service data streams, each service data stream can be processed by only one CPU, and a log generated by each data stream can be processed by only the CPU which processes the data stream;
step 2: the logs are put into a cache queue, and a CPU is intelligently scheduled to process the log cache queue according to the processing requirement of the service data stream;
and step 3: after analyzing the data message, the service flow processing module extracts the required data and writes the data into a log cache queue;
and 4, step 4: after the service flow processing module sends the data message, the log cache queue is checked, and if the log cache reaches the expected configuration, the log compression sending module is dispatched to take out the log compression from the log cache queue and send the log compression;
and 5: after the log is compressed and sent, whether the service flow is contained or not is judged, if the service flow is contained, the service flow is processed preferentially, and otherwise, the log compression sending module is scheduled to continue sending the log.
2. The method for collecting and sending big data audit logs of application delivery equipment according to claim 1, wherein: and before the log reaches the value of the buffer forbidding, the service flow is processed preferentially, and if no service request exists, the log buffer queue is processed.
3. The method for collecting and sending big data audit logs of application delivery equipment according to claim 1, wherein: and the log reaches a cache warning, and the log is processed preferentially.
4. The method for collecting and sending big data audit logs of application delivery equipment according to claim 1, wherein: after each log buffer queue is processed, whether a service request exists is checked to ensure low delay of the service.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110170542.3A CN112882808B (en) | 2021-02-08 | 2021-02-08 | Method for collecting and transmitting big data audit log of application delivery equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110170542.3A CN112882808B (en) | 2021-02-08 | 2021-02-08 | Method for collecting and transmitting big data audit log of application delivery equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112882808A true CN112882808A (en) | 2021-06-01 |
| CN112882808B CN112882808B (en) | 2023-10-24 |
Family
ID=76057544
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110170542.3A Active CN112882808B (en) | 2021-02-08 | 2021-02-08 | Method for collecting and transmitting big data audit log of application delivery equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112882808B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115296973A (en) * | 2022-05-06 | 2022-11-04 | 北京数联众创科技有限公司 | Method, device and application for batch collection and sending of front-end logs |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080077752A1 (en) * | 2006-09-25 | 2008-03-27 | Hitachi, Ltd. | Storage system and audit log management method |
| CN101763593A (en) * | 2009-12-17 | 2010-06-30 | 中国电力科学研究院 | Method and device for realizing audit log of system |
| US8407335B1 (en) * | 2008-06-18 | 2013-03-26 | Alert Logic, Inc. | Log message archiving and processing using a remote internet infrastructure |
| CN103729442A (en) * | 2013-12-30 | 2014-04-16 | 华为技术有限公司 | Method for recording event logs and database engine |
| CN105119752A (en) * | 2015-09-08 | 2015-12-02 | 北京京东尚科信息技术有限公司 | Distributed log acquisition method, device and system |
| US20160147858A1 (en) * | 2014-11-25 | 2016-05-26 | Juchang Lee | Log Forwarding to Avoid Deadlocks During Parallel Log Replay in Asynchronous Table Replication |
| CN105991346A (en) * | 2015-04-28 | 2016-10-05 | 杭州迪普科技有限公司 | Session log processing method and device |
| US9600553B1 (en) * | 2014-05-31 | 2017-03-21 | Veritas Technologies Llc | Distributed replication in cluster environments |
| CN107135088A (en) * | 2016-02-29 | 2017-09-05 | 华为技术有限公司 | The method and apparatus that daily record is handled in cloud computing system |
| CN107249019A (en) * | 2017-05-26 | 2017-10-13 | 北京小米移动软件有限公司 | Data handling system, method, device and server based on business |
| CN107590182A (en) * | 2017-08-03 | 2018-01-16 | 华南理工大学 | A kind of distributed information log collection method |
| US20190114350A1 (en) * | 2017-10-17 | 2019-04-18 | Salesforce.Com, Inc. | Systems, Methods, and Apparatuses for Implementing Concurrent Dataflow Execution with Write Conflict Protection Within a Cloud Based Computing Environment |
-
2021
- 2021-02-08 CN CN202110170542.3A patent/CN112882808B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080077752A1 (en) * | 2006-09-25 | 2008-03-27 | Hitachi, Ltd. | Storage system and audit log management method |
| US8407335B1 (en) * | 2008-06-18 | 2013-03-26 | Alert Logic, Inc. | Log message archiving and processing using a remote internet infrastructure |
| CN101763593A (en) * | 2009-12-17 | 2010-06-30 | 中国电力科学研究院 | Method and device for realizing audit log of system |
| CN103729442A (en) * | 2013-12-30 | 2014-04-16 | 华为技术有限公司 | Method for recording event logs and database engine |
| US9600553B1 (en) * | 2014-05-31 | 2017-03-21 | Veritas Technologies Llc | Distributed replication in cluster environments |
| US20160147858A1 (en) * | 2014-11-25 | 2016-05-26 | Juchang Lee | Log Forwarding to Avoid Deadlocks During Parallel Log Replay in Asynchronous Table Replication |
| CN105991346A (en) * | 2015-04-28 | 2016-10-05 | 杭州迪普科技有限公司 | Session log processing method and device |
| CN105119752A (en) * | 2015-09-08 | 2015-12-02 | 北京京东尚科信息技术有限公司 | Distributed log acquisition method, device and system |
| CN107135088A (en) * | 2016-02-29 | 2017-09-05 | 华为技术有限公司 | The method and apparatus that daily record is handled in cloud computing system |
| CN107249019A (en) * | 2017-05-26 | 2017-10-13 | 北京小米移动软件有限公司 | Data handling system, method, device and server based on business |
| CN107590182A (en) * | 2017-08-03 | 2018-01-16 | 华南理工大学 | A kind of distributed information log collection method |
| US20190114350A1 (en) * | 2017-10-17 | 2019-04-18 | Salesforce.Com, Inc. | Systems, Methods, and Apparatuses for Implementing Concurrent Dataflow Execution with Write Conflict Protection Within a Cloud Based Computing Environment |
Non-Patent Citations (3)
| Title |
|---|
| MUHUAN HUANG ET.AL.: "A scalable, high-performance customized priority queue", 2014 24TH INTERNATIONAL CONFERENCE ON FIELD PROGRAMMABLE LOGIC AND APPLICATIONS (FPL) * |
| 艾智远: "基于图的大规模日志处理系统关键技术研究", 中国优秀硕士学位论文全文数据库 (信息科技辑) * |
| 陆世鹏;: "基于Spark Streaming的海量日志实时处理系统的设计", 电子产品可靠性与环境试验, no. 05 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115296973A (en) * | 2022-05-06 | 2022-11-04 | 北京数联众创科技有限公司 | Method, device and application for batch collection and sending of front-end logs |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112882808B (en) | 2023-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7864764B1 (en) | Accelerated packet processing in a network acceleration device | |
| Wolman et al. | ORGANIZ AT ION-BASEDANALYSISOF WEB-OBJECTSHARINGANDCACHI NG | |
| US11223639B2 (en) | Endpoint network traffic analysis | |
| EP3232710A1 (en) | Method, device and system for processing traffic of mobile terminal based on content delivery network | |
| US9356844B2 (en) | Efficient application recognition in network traffic | |
| WO2023077952A1 (en) | Data processing method and system, related device, storage medium and product | |
| WO2004077211A2 (en) | Method and apparatus for increasing file server performance by offloading data path processing | |
| WO2010072081A1 (en) | Method and system for realizing massive terminals access of a streaming media server | |
| US9800662B2 (en) | Generic network trace with distributed parallel processing and smart caching | |
| US10178033B2 (en) | System and method for efficient traffic shaping and quota enforcement in a cluster environment | |
| EP4084415A1 (en) | Data management method and system, associated subsystem and computer readable medium | |
| CN108989420B (en) | Method and system for registering service and method and system for calling service | |
| US8572245B1 (en) | Using the TCP window size for identifying packets and debugging | |
| CN112882808B (en) | Method for collecting and transmitting big data audit log of application delivery equipment | |
| US9215248B1 (en) | User access rate limiting among content delivery nodes | |
| TW201933909A (en) | Network access method, client, network interaction method, and scheduling and network systems | |
| CN110855726A (en) | Communication method, communication device, gateway, computing device and medium | |
| US9055467B2 (en) | Sender device based pause system | |
| KR101776662B1 (en) | Data storage and processing method for collecting and analyzing real-time events, and network system using the same | |
| CN111291367A (en) | Access control method and system, electronic device and storage medium | |
| Shimano et al. | An information propagation scheme for an autonomous distributed storage system in iSCSI environment | |
| CN110519397B (en) | SIP terminal access load balancing system and method based on NGINX | |
| CN109450794B (en) | Communication method and device based on SDN network | |
| CA3129680A1 (en) | Adaptive retrieval of objects from remote storage | |
| Chen et al. | Demystifying Datapath Accelerator Enhanced Off-path SmartNIC |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |