CN112231191B - Log collection method and device - Google Patents
Log collection method and device Download PDFInfo
- Publication number
- CN112231191B CN112231191B CN202011147788.0A CN202011147788A CN112231191B CN 112231191 B CN112231191 B CN 112231191B CN 202011147788 A CN202011147788 A CN 202011147788A CN 112231191 B CN112231191 B CN 112231191B
- Authority
- CN
- China
- Prior art keywords
- log
- node
- target
- identifier
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computer Hardware Design (AREA)
- Mathematical Physics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present application relates to the field of network security technologies, and in particular, to a log collection method and apparatus. The log collection method is applied to log collection equipment comprising a log accepting and rejecting model, the log accepting and rejecting model comprises an annular linked list formed by a plurality of nodes, each node is provided with a corresponding state identifier, if the state identifier of one node is a first identifier, logs received based on the one node are reserved, and if the state identifier of one node is a second identifier, the logs received based on the one node are discarded, and the method comprises the following steps: updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing equipment; receiving a target log sent by network equipment in a current period, and judging a state identifier of a target node corresponding to the target log based on the updated log accepting and rejecting model; if the state identification of the target node is judged to be the first identification, caching the target log into a cache pool; otherwise, the target log is discarded.
Description
Technical Field
The present application relates to the field of network security technologies, and in particular, to a log collection method and apparatus.
Background
With the increasingly deep informatization, the internet is becoming a key information infrastructure of the country, and the internet security is related to the fundamental interests of the country and the society. The network security situation perception technology can synthesize security factors of all aspects, dynamically reflect network security conditions on the whole, predict and early warn development trend of the security conditions, and provide reliable reference basis for enhancing network security. Therefore, security situation awareness research on networks has become a research hotspot in the field of network security at present.
The acquisition and normalization processing of various device logs are the basis of network security situation awareness, and the log acquisition mode is generally divided into two modes, namely passive log acquisition and active log acquisition, wherein the passive log acquisition refers to that a network security situation awareness platform serves as a server to passively receive logs sent by a device. In passive log collection, the network security situation awareness platform cannot control log sending of the device, and a scene that the device continuously sends a large amount of logs may exist. If the sending rate of the logs is higher than the log analysis and processing performance of the situation awareness platform, the abnormality of the situation awareness platform can be caused. Therefore, for receiving the passive log, the network security situation awareness platform needs to make a log receiving speed-limiting strategy to ensure the stable operation of the network security situation awareness platform.
Under the premise, the log collection equipment can pre-cache the received logs into a cache pool, and when the number of the logs in the cache Chi Zhongri reaches a set threshold, the logs are sent to the next link in batches for processing. And combining the speed limit requirement of the logs, when the cache pool is full, the logs are not received again until the batch of logs are processed and written into a batch of logs, and the logs collected in the process are directly discarded. Then, the discarded logs may be continuous, which will affect the analysis result of the network security situation awareness platform later.
Disclosure of Invention
The application provides a log collection method and device, which are used for solving the problem that in the prior art, the analysis result of a network security situation perception platform is inaccurate due to the fact that continuous logs are discarded.
In a first aspect, the present application provides a log collecting method, which is applied to a log collecting device including a log accepting and rejecting model, where the log accepting and rejecting model includes an annular linked list composed of a plurality of nodes, and a corresponding state identifier is provided for each node in the annular linked list, where if a state identifier of a node is a first identifier, a log received based on the node is retained, and if a state identifier of a node is a second identifier, a log received based on the node is discarded, where the method includes:
updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing equipment;
receiving a target log sent by network equipment in a current period, and judging a state identifier of a target node corresponding to the target log based on the updated log accepting and rejecting model;
if the state identifier of the target node is judged to be the first identifier, caching the target log into a cache pool;
and if the state identifier of the target node is judged to be the second identifier, discarding the target log.
Optionally, if the current period is a first period, the state identifiers of the nodes in the circular linked list in the log selection and deletion model are all set as first identifiers.
Optionally, if the current period is not the first period, the step of updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing device includes:
calculating the log reporting rate of the network equipment in the previous period based on the total number of the logs received in the previous period and the period duration;
determining the ratio of the number of logs expected to be reserved and the number of discarded logs in the previous period based on the log reporting rate of the network equipment in the previous period and the preset log processing rate of the log processing equipment;
calculating the ratio of the number of the nodes with the state identifications as the first identifications and the number of the nodes with the state identifications as the second identifications in the log accepting and rejecting model based on the ratio of the number of the logs;
and updating the state identifier of each node in the annular linked list based on the ratio of the number of the nodes and the total number of the nodes in the annular linked list.
Optionally, the nodes whose state identifiers are the first identifiers and the nodes whose state identifiers are the second identifiers are uniformly distributed in the updated circular linked list.
Optionally, the step of determining, based on the updated log accepting/rejecting model, a state identifier of a target node corresponding to the target log includes:
after receiving a target log, judging a state identifier of a target node pointed by a pointer, processing the target log based on the state identifier of the target node, and pointing the pointer to a next node of the target node in the annular linked list.
In a second aspect, the present application provides a log collecting device, which is applied to a log collecting device including a log accepting and rejecting model, where the log accepting and rejecting model includes an annular linked list composed of a plurality of nodes, and is directed to each node in the annular linked list is provided with a corresponding state identifier, where if the state identifier of a node is a first identifier, the log received by the node is retained, and if the state identifier of a node is a second identifier, the log received by the node is discarded, and the log collecting device includes:
the updating unit is used for updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing equipment;
the judging unit is used for receiving a target log sent by network equipment in the current period and judging the state identifier of a target node corresponding to the target log based on the updated log accepting and rejecting model;
the cache unit caches the target log to a cache pool if the judging unit judges that the state identifier of the target node is a first identifier;
and the discarding unit discards the target log if the judging unit judges that the state identifier of the target node is the second identifier.
Optionally, if the current period is a first period, the state identifiers of the nodes in the circular linked list in the log selection and deletion model are all set as first identifiers.
Optionally, if the current period is not the first period, when the log accepting or rejecting model is updated based on the statistical log reporting rate of the previous period and the log processing rate of the log processing device, the updating unit is specifically configured to:
calculating the log reporting rate of the network equipment in the previous period based on the total number of the logs received in the previous period and the period duration;
determining the ratio of the number of logs expected to be reserved and the number of discarded logs in the previous period based on the log reporting rate of the network equipment in the previous period and the preset log processing rate of the log processing equipment;
calculating the ratio of the number of the nodes with the state identifications as the first identifications and the number of the nodes with the state identifications as the second identifications in the log accepting and rejecting model based on the ratio of the number of the logs;
and updating the state identification of each node in the annular linked list based on the ratio of the number of the nodes and the total number of the nodes in the annular linked list.
Optionally, the nodes whose state identifiers are the first identifiers and the nodes whose state identifiers are the second identifiers are uniformly distributed in the updated circular linked list.
Optionally, when the state identifier of the target node corresponding to the target log is determined based on the updated log accepting and rejecting model, the determining unit is specifically configured to:
after receiving a target log, judging a state identifier of a target node pointed by a pointer, processing the target log based on the state identifier of the target node, and pointing the pointer to a next node of the target node in the annular linked list.
In a third aspect, an embodiment of the present application provides a log collecting device, where the log collecting device includes:
a memory for storing program instructions;
a processor for calling program instructions stored in said memory and for executing the steps of the method according to any one of the above first aspects in accordance with the obtained program instructions.
In a fourth aspect, the present application further provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the steps of the method according to any one of the above first aspects.
To sum up, the log collection method provided by the embodiment of the present application is applied to log collection equipment including a log accepting and rejecting model, the log accepting and rejecting model includes an annular linked list composed of a plurality of nodes, and is directed to each node in the annular linked list is provided with a corresponding state identifier, wherein, if the state identifier of a node is a first identifier, the log received by a node is retained, and if the state identifier of a node is a second identifier, the log received by a node is discarded, and the method includes: updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing equipment; receiving a target log sent by network equipment in a current period, and judging a state identifier of a target node corresponding to the target log based on the updated log accepting and rejecting model; if the state identifier of the target node is judged to be the first identifier, caching the target log into a cache pool; and if the state identifier of the target node is judged to be the second identifier, discarding the target log.
By adopting the log collection method provided by the embodiment of the application, the log discarding strategy of the next time period is determined according to the log quantity of the previous time period, the dynamic speed limit of the log is realized, the scene requirements that the actual log quantity does not discard the log in the performance specification of the network security situation awareness platform, more logs are discarded when the actual log quantity exceeds the performance specification of the network security situation awareness platform are met, and the like are met, the log is directly filtered at the log receiving end, and the damage caused by data impact when the network security situation awareness platform runs in the environment exceeding the performance specification of the network security situation awareness platform is effectively avoided. The logs of corresponding entries are discarded by adopting the state identifiers of all the nodes in the annular linked list, the condition of continuously discarding continuous logs for a long time is avoided, and the accuracy of the analysis result of the network security situation perception platform is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments of the present application or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art according to the drawings of the embodiments of the present application.
Fig. 1 is a detailed flowchart of a log collection method according to an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of an annular linked list according to an embodiment of the present disclosure;
fig. 3 is a detailed flowchart of another log collection method according to an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of a log collection device according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of another log collection device according to an embodiment of the present application.
Detailed Description
The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein is meant to encompass any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, the information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. Depending on the context, moreover, the word "if" is used may be interpreted as "at … …" or "at … …" or "in response to a determination".
Exemplarily, referring to fig. 1, a detailed flowchart of a log collecting method provided in an embodiment of the present application is applied to a log collecting device including a log accepting and rejecting model, where the log accepting and rejecting model includes a ring-shaped linked list composed of a plurality of nodes, and a corresponding state identifier is set for each node in the ring-shaped linked list, where if a state identifier of one node is a first identifier, a log received based on the one node is retained, and if a state identifier of one node is a second identifier, a log received based on the one node is discarded, where the method includes the following steps:
step 100: and updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log acquisition equipment.
In practical application, the log collection device receives logs reported by each network device in the network, maintains a log accepting and rejecting model at a log receiving inlet, confirms that each log reported by each network device is discarded or reserved when passing through the log accepting and rejecting model, does not perform any processing on the log when the log is discarded, and stores the log into a cache to wait for the processing of the next link when the log is reserved.
Correspondingly, when the log collection device is started, parameters of a log selection model included in the log collection device can be pre-configured based on a specific application scenario and/or user requirements. Specifically, the log processing rate of the log processing device, the period for updating the log accepting model, and the like may be included, and in this embodiment, the log processing rate may be understood as the maximum log processing capacity of the log processing device.
In the embodiment of the present application, when a cycle ends, an operation of updating a log accepting and rejecting model needs to be started, specifically, in a first case, if the current cycle is a first cycle, state identifiers of nodes in the annular linked list in the log accepting and rejecting model are all set as first identifiers.
That is to say, when the log selection and deletion model is initialized, since there is no previous cycle, in the embodiment of the present application, the state identifiers of the nodes in the above-mentioned circular linked list in the log selection and deletion model are all set as the first identifiers, that is, all logs received in the current cycle are cached in the cache pool.
In the second case, if the current period is not the first period, when the log accepting/rejecting model is updated based on the counted log reporting rate of the previous period and the log processing rate of the log processing device, a preferable implementation manner is to calculate the log reporting rate of the network device in the previous period based on the total number of logs received in the previous period and the period duration; determining the ratio of the number of logs expected to be reserved and the number of discarded logs in the previous period based on the log reporting rate of the network equipment in the previous period and the preset log processing rate of the log processing equipment; calculating the ratio of the number of nodes with the state identifier as a first identifier and the number of nodes with the state identifier as a second identifier in the log accepting and rejecting model based on the ratio of the number of the logs; and updating the state identifier of each node in the annular linked list based on the ratio of the number of the nodes and the total number of the nodes in the annular linked list.
That is, based on the total number of the log pieces received in the previous period and the number of the log pieces that can be processed in the previous period by the log processing device, the ratio between the number of the log pieces expected to be kept and the number of the log pieces expected to be discarded in the next period is determined, and the state identifier of each node in the circular linked list in the log selection and deletion model is adjusted and updated according to the ratio.
Preferably, the nodes whose state identifiers are the first identifiers and the nodes whose state identifiers are the second identifiers are uniformly distributed in the updated circular linked list.
For example, assuming that the duration of the preset period is 2 seconds (S), the log processing rate of the preset log processing device is 4000 pieces/second, and the total number of the log pieces received in the previous period is 20000 pieces, then the ratio of the number of the log expected to be kept and the number of the log discarded in the previous period can be calculated according to the above parameters, where the ratio is: (4000 × 2)/(20000- (4000 × 2)) = (8000 × 12000) = (2:3), that is, 8000 logs are reserved, and 12000 logs are discarded, and in practical applications, the total number of logs reported by the network device in two adjacent cycles fluctuates little, so in the embodiment of the present application, the state identifier of each node in the updated circular linked list can be calculated according to the ratio of the number of logs expected to be reserved and the number of discarded logs calculated in the previous cycle and the total node data (10) of the circular linked list in the log rounding model. For example, referring to fig. 2, as a structural diagram of a circular linked list provided in this embodiment of the present application, assuming that the total number of nodes in the circular linked list is 10, the next node of node 1 is node 2, the next node of node 2 is node 3, … …, the next node of node 9 is node 10, and the next node of node 10 is node 1, if the calculated ratio of the number of logs expected to be kept and the number of log entries to be discarded in the last cycle is (2:3), when updating the log selection model, a preferred implementation manner is to set the state identifiers of nodes 1 and 2 as a first identifier (e.g., true), set the state identifiers of nodes 3 to 5 as a second identifier (e.g., false), set the state identifiers of nodes 6 and 7 as the first identifier, and set the state identifiers of nodes 8 to 10 as a second identifier, that is, a node including 4 state identifiers as the first identifier and a node including 6 state identifiers as the first identifier.
It should be noted that the preset period may be set in a self-defined manner according to different application scenarios and/or different user requirements, for example, set to be 1 second, 2 seconds, 5 seconds, and the like.
In the embodiment of the present application, it is assumed that the circular linked list of the log selection model includes n nodes, and one node corresponds to one log, for example, when receiving log 1, it determines to keep log 1 or discard log 1 based on the state identifier of node 1 corresponding to log 1, when receiving log 2, it determines to keep log 2 or discard log 2 based on the state identifier of node 2 corresponding to log 2, … …, when receiving log n, it determines to keep log n or discard log n based on the state identifier of node n corresponding to log n, and when receiving log n +1, because the next node of node n is node 1, it determines to keep log n +1 or discard log n +1, … … based on the state identifier of node 1 corresponding to log n + 1.
Step 110: and receiving a target log sent by the network equipment in the current period, and judging the state identifier of a target node corresponding to the target log based on the updated log selection model.
In this embodiment of the application, when determining the state identifier of the target node corresponding to the target log based on the updated log selection model, a preferred implementation manner is to determine the state identifier of the target node pointed by the pointer after receiving the target log, process the target log based on the state identifier of the target node, and point the pointer to a node next to the target node in the circular linked list.
It should be noted that, in the embodiment of the present application, the purpose of using the circular linked list is that after each log comes, it can be determined whether the log is reserved only according to the state identifier of the node to which the pointer currently points, and pointer operation in the computer greatly reduces the decision performance loss. When a log is received, if the state identifier of the node currently pointed by the pointer is a first identifier, the log is cached to a cache for subsequent processing, if the state identifier of the node currently pointed by the pointer is a second identifier, the log is directly discarded, and then the pointer is moved to the next node.
Of course, in the embodiment of the present application, when a log is received, the pointer may be moved first, and then the node state identifier may be determined, that is, when a log is received, the pointer is moved to a next node of the currently-pointed node first, then the log is determined to be cached/discarded according to the state identifier of the next node, after the log is processed, the pointer is not moved temporarily, and only when the next log is received, the pointer is moved, and then the node state identifier is determined.
Step 120: and if the state identifier of the target node is judged to be the first identifier, caching the target log into a cache pool.
Step 130: and if the state identifier of the target node is judged to be the second identifier, discarding the target log.
The following describes the log collection method provided in the embodiment of the present application in detail with reference to a specific application scenario. Illustratively, referring to fig. 3, a detailed flowchart of another log collection method provided in an embodiment of the present application is shown, where the method includes the following steps: the method comprises the steps that a network device/a safety device/a host and the like in a networking actively report logs to a log collection device, when the log collection device receives any log, whether the log is reserved or not is judged based on a log accepting and rejecting model, if so, the log is cached in a cache pool, and if not, the log is discarded; further, the log accepting and rejecting model updates the log accepting and rejecting model based on a preset period and the total number of logs reported by the network device/security device/host and the like received in the previous period, for example, when the current period is determined to be finished, the total number of logs received in the previous period is obtained, the log accepting and rejecting model is updated based on the total number of logs and the log processing capacity of the log processing device, and in the next period, whether each received log is reserved is determined by using the updated log accepting and rejecting model.
Exemplarily, referring to fig. 4, a schematic structural diagram of a log collecting device provided in an embodiment of the present application is applied to a log collecting device including a log accepting and rejecting model, where the log accepting and rejecting model includes an annular linked list composed of a plurality of nodes, and a corresponding state identifier is set for each node in the annular linked list, where if a state identifier of one node is a first identifier, a log received based on the one node is retained, and if a state identifier of one node is a second identifier, a log received based on the one node is discarded, where the log collecting device includes:
the updating unit 40 is configured to update the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing device;
a determining unit 41, configured to receive a target log sent by a network device in a current period, and determine, based on the updated log accepting/rejecting model, a state identifier of a target node corresponding to the target log;
a caching unit 42, if the determining unit 41 determines that the status identifier of the target node is the first identifier, the caching unit 42 caches the target log in a cache pool;
a discarding unit 43, wherein if the determining unit 41 determines that the status flag of the target node is the second flag, the discarding unit 43 discards the target log.
Optionally, if the current period is a first period, the state identifiers of the nodes in the circular linked list in the log selection and deletion model are all set as first identifiers.
Optionally, if the current period is not the first period, when the log accepting and rejecting model is updated based on the statistical log reporting rate of the previous period and the log processing rate of the log processing device, the updating unit 40 is specifically configured to:
calculating the log reporting rate of the network equipment in the previous period based on the total number of the logs received in the previous period and the period duration;
determining the ratio of the number of logs expected to be reserved and the number of discarded logs in the previous period based on the log reporting rate of the network equipment in the previous period and the preset log processing rate of the log processing equipment;
calculating the ratio of the number of nodes with the state identifier as a first identifier and the number of nodes with the state identifier as a second identifier in the log accepting and rejecting model based on the ratio of the number of the logs;
and updating the state identifier of each node in the annular linked list based on the ratio of the number of the nodes and the total number of the nodes in the annular linked list.
Optionally, the nodes whose state identifiers are the first identifiers and the nodes whose state identifiers are the second identifiers are uniformly distributed in the updated circular linked list.
Optionally, when determining, based on the updated log accepting/rejecting model, the state identifier of the target node corresponding to the target log, the determining unit 41 is specifically configured to:
after receiving a target log, judging a state identifier of a target node pointed by a pointer, processing the target log based on the state identifier of the target node, and pointing the pointer to a next node of the target node in the annular linked list.
The above units may be one or more integrated circuits configured to implement the above methods, for example: one or more Application Specific Integrated Circuits (ASICs), or one or more microprocessors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), among others. For another example, when one of the above units is implemented in the form of a Processing element scheduler code, the Processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. For another example, these units may be integrated together and implemented in the form of a system-on-a-chip (SOC).
Further, in the log collecting device provided in the embodiment of the present application, from a hardware aspect, a schematic diagram of a hardware architecture of the log collecting device may be shown in fig. 5, where the log collecting device may include: a memory 50 and a processor 51, which,
the memory 50 is used for storing program instructions; the processor 51 calls the program instructions stored in the memory 50 and executes the above-described method embodiments according to the obtained program instructions. The specific implementation and technical effects are similar, and are not described herein again.
Optionally, the present application further provides a log collecting device, which includes at least one processing element (or chip) for executing the above method embodiments.
Optionally, the present application also provides a program product, such as a computer-readable storage medium, having stored thereon computer-executable instructions for causing the computer to perform the above-described method embodiments.
Here, a machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: a RAM (random Access Memory), a volatile Memory, a non-volatile Memory, a flash Memory, a storage drive (e.g., a hard drive), a solid state drive, any type of storage disk (e.g., an optical disk, a dvd, etc.), or similar storage medium, or a combination thereof.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, which may take the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email messaging device, game console, tablet computer, wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Furthermore, these computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (8)
1. A log collection method is applied to log collection equipment comprising a log rejection model, wherein the log rejection model comprises an annular linked list consisting of a plurality of nodes, and a corresponding state identifier is set for each node in the annular linked list, wherein if the state identifier of one node is a first identifier, a log received based on the one node is retained, and if the state identifier of one node is a second identifier, the log received based on the one node is discarded, and the method comprises the following steps:
updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing equipment;
receiving a target log sent by network equipment in a current period, and judging a state identifier of a target node corresponding to the target log based on the updated log accepting and rejecting model;
if the state identifier of the target node is judged to be the first identifier, caching the target log into a cache pool;
if the state identifier of the target node is judged to be a second identifier, discarding the target log; if the current period is not the first period, updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing device comprises the following steps:
calculating the log reporting rate of the network equipment in the previous period based on the total number of the logs received in the previous period and the period duration;
determining the ratio of the number of logs expected to be reserved and the number of discarded logs in the previous period based on the log reporting rate of the network equipment in the previous period and the preset log processing rate of the log processing equipment;
calculating the ratio of the number of the nodes with the state identifications as the first identifications and the number of the nodes with the state identifications as the second identifications in the log accepting and rejecting model based on the ratio of the number of the logs;
and updating the state identifier of each node in the annular linked list based on the ratio of the number of the nodes and the total number of the nodes in the annular linked list.
2. The method of claim 1, wherein if the current cycle is a first cycle, the state flag of each node in the circular linked list in the log rounding model is set to a first flag.
3. The method of claim 1, wherein nodes whose state identities are a first identity and nodes whose state identities are a second identity are evenly distributed in the updated circular linked list.
4. The method of claim 1, wherein the step of determining the status identifier of the target node corresponding to the target log based on the updated log pruning model comprises:
after receiving a target log, judging a state identifier of a target node pointed by a pointer, processing the target log based on the state identifier of the target node, and pointing the pointer to a next node of the target node in the annular linked list.
5. The utility model provides a log collection system which characterized in that is applied to the log collection equipment who includes log accepting model, log accepting model includes the cyclic annular linked list of constituteing by a plurality of nodes, is provided with corresponding state sign to each node in the cyclic annular linked list, wherein, if the state sign of a node is first sign, then keep the log that is received based on a node, if the state sign of a node is the second sign, then abandon the log that is received based on a node, the device includes:
the updating unit is used for updating the log accepting and rejecting model based on the statistical log reporting rate of the previous period and the log processing rate of the log processing equipment;
the judging unit is used for receiving a target log sent by network equipment in a current period and judging a state identifier of a target node corresponding to the target log based on the updated log accepting and rejecting model;
the cache unit caches the target log to a cache pool if the judging unit judges that the state identifier of the target node is a first identifier;
the discarding unit discards the target log if the judging unit judges that the state identifier of the target node is a second identifier; if the current period is not the first period, when the log accepting and rejecting model is updated based on the statistical log reporting rate of the previous period and the log processing rate of the log processing device, the updating unit is specifically configured to:
calculating the log reporting rate of the network equipment in the previous period based on the total number of the logs received in the previous period and the period duration;
determining the ratio of the number of logs expected to be reserved and the number of discarded logs in the previous period based on the log reporting rate of the network equipment in the previous period and the preset log processing rate of the log processing equipment;
calculating the ratio of the number of nodes with the state identifier as a first identifier and the number of nodes with the state identifier as a second identifier in the log accepting and rejecting model based on the ratio of the number of the logs;
and updating the state identifier of each node in the annular linked list based on the ratio of the number of the nodes and the total number of the nodes in the annular linked list.
6. The apparatus of claim 5, wherein the state flags of the nodes in the circular linked list in the log rounding model are all set to a first flag if the current cycle is a first cycle.
7. The apparatus of claim 5, wherein nodes with state identities of a first identity and nodes with state identities of a second identity are evenly distributed in the updated circular linked list.
8. The apparatus according to claim 5, wherein when determining the status identifier of the target node corresponding to the target log based on the updated log-culling model, the determining unit is specifically configured to:
after receiving a target log, judging a state identifier of a target node pointed by a pointer, processing the target log based on the state identifier of the target node, and pointing the pointer to a next node of the target node in the annular linked list.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011147788.0A CN112231191B (en) | 2020-10-23 | 2020-10-23 | Log collection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011147788.0A CN112231191B (en) | 2020-10-23 | 2020-10-23 | Log collection method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112231191A CN112231191A (en) | 2021-01-15 |
| CN112231191B true CN112231191B (en) | 2023-03-31 |
Family
ID=74110771
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011147788.0A Active CN112231191B (en) | 2020-10-23 | 2020-10-23 | Log collection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112231191B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113282920B (en) * | 2021-05-28 | 2023-10-10 | 平安科技（深圳）有限公司 | Log abnormality detection method, device, computer equipment and storage medium |
| CN116346729B (en) * | 2023-02-24 | 2024-02-09 | 安芯网盾(北京)科技有限公司 | Data log reporting current limiting method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101557291A (en) * | 2009-05-25 | 2009-10-14 | 杭州华三通信技术有限公司 | Method for log aggregation and device thereof |
| CN101729295A (en) * | 2009-12-02 | 2010-06-09 | 北京东土科技股份有限公司 | Method for realizing log function |
| CN105138606A (en) * | 2015-08-03 | 2015-12-09 | 上海斐讯数据通信技术有限公司 | Server log management method and system |
| CN111177360A (en) * | 2019-12-16 | 2020-05-19 | 中国电子科技网络信息安全有限公司 | Self-adaptive filtering method and device based on user logs on cloud |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8948020B2 (en) * | 2012-12-11 | 2015-02-03 | International Business Machines Corporation | Detecting and isolating dropped or out-of-order packets in communication networks |
| CN103795577A (en) * | 2014-03-03 | 2014-05-14 | 网神信息技术（北京）股份有限公司 | Log processing method and device of log server |
| CN108616556B (en) * | 2016-12-13 | 2021-01-19 | 阿里巴巴集团控股有限公司 | Data processing method, device and system |
| CN108471387B (en) * | 2018-03-27 | 2022-10-21 | 中国农业银行股份有限公司 | Log flow decentralized control method and system |
| CN109344034A (en) * | 2018-09-29 | 2019-02-15 | 郑州云海信息技术有限公司 | A kind of method and apparatus for managing log |
-
2020
- 2020-10-23 CN CN202011147788.0A patent/CN112231191B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101557291A (en) * | 2009-05-25 | 2009-10-14 | 杭州华三通信技术有限公司 | Method for log aggregation and device thereof |
| CN101729295A (en) * | 2009-12-02 | 2010-06-09 | 北京东土科技股份有限公司 | Method for realizing log function |
| CN105138606A (en) * | 2015-08-03 | 2015-12-09 | 上海斐讯数据通信技术有限公司 | Server log management method and system |
| CN111177360A (en) * | 2019-12-16 | 2020-05-19 | 中国电子科技网络信息安全有限公司 | Self-adaptive filtering method and device based on user logs on cloud |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112231191A (en) | 2021-01-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112231191B (en) | Log collection method and device | |
| CN107104824B (en) | Network topology determination method and device | |
| US20150139074A1 (en) | Adaptive Generation of Network Scores From Crowdsourced Data | |
| CN108156265A (en) | A kind of application control method and mobile equipment | |
| CN112272179B (en) | Network security processing method, device, equipment and machine readable storage medium | |
| CN107305570B (en) | Data retrieval method and system | |
| CN110889132A (en) | Distributed application permission verification method and device | |
| CN113328944A (en) | Flow table management method and device | |
| CN112162912A (en) | Cloud resource monitoring method and system | |
| CN108399175B (en) | Data storage and query method and device | |
| CN113486037A (en) | Cache data updating method, manager and cache server | |
| CN111159009B (en) | Pressure testing method and device for log service system | |
| CN109995834A (en) | Massive dataflow processing method, calculates equipment and storage medium at device | |
| WO2016176919A1 (en) | Attendance checking method and attendance checking server | |
| JP2019016003A (en) | Information processing apparatus, information processing system, information processing method and information processing program | |
| CN108173689B (en) | Output system of load balancing data | |
| CN112367267A (en) | Virtual machine management method and device | |
| CN115190002B (en) | Alarm processing method, device, terminal equipment and storage medium | |
| CN112615971A (en) | Image transmission method and device | |
| JP2021524085A (en) | Message processing methods, devices and systems | |
| CN113055419B (en) | Information sending method and device | |
| CN116132528B (en) | Flight management message pushing method and device and electronic equipment | |
| CN113810864B (en) | Regional short message targeted group sending method and system | |
| CN116149959B (en) | Data processing device, method, monitoring equipment and computer program product | |
| CN115757303B (en) | Index tracking method and device in distributed system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |