CN114020712A - Log rate adjusting method and device - Google Patents
Log rate adjusting method and device Download PDFInfo
- Publication number
- CN114020712A CN114020712A CN202110877419.5A CN202110877419A CN114020712A CN 114020712 A CN114020712 A CN 114020712A CN 202110877419 A CN202110877419 A CN 202110877419A CN 114020712 A CN114020712 A CN 114020712A
- Authority
- CN
- China
- Prior art keywords
- rate
- log
- total
- threshold
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computational Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Physics (AREA)
- Pure & Applied Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Mathematical Optimization (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Operations Research (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Algebra (AREA)
- Life Sciences & Earth Sciences (AREA)
- Software Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application provides a log rate adjusting method and device. The method comprises the following steps: acquiring log rates of logs respectively reported by a plurality of devices of the same log type in a current acquisition period and calculating a total log rate; if the total log rate is greater than the total rate threshold of the log type, calculating the threshold average rate of the log type based on the total rate threshold; for each device, if the log rate of the device is smaller than the threshold average rate, keeping the current log rate unchanged; calculating the compensation deviation of the equipment according to the log rate and the threshold average rate; if the log rate of the equipment is not less than the threshold average rate, calculating the total compensation deviation of the log type based on the compensation deviation of each equipment with the log rate less than the threshold average rate; and increasing the log rate of the equipment according to the total compensation deviation.
Description
Technical Field
The present application relates to the field of network security technologies, and in particular, to a log rate adjustment method and apparatus.
Background
With the rapid development of big data technology, mass log data is often used for service analysis and display in various service scenes. The massive log data analysis model mainly comprises the following three parts: log collection, heterogeneous log normalization (log analysis) and service analysis. Since log collection and heterogeneous log normalization do not involve specific complex traffic analysis, and focus only on collection and normalization functions, their processing performance is usually higher than that of a specific traffic analysis module. Although this inequality of processing performance may be mitigated with caching or message middleware techniques, in the case of a large amount of data, the service analysis may still result in the entire platform being in an unavailable state due to the processing performance not keeping up. For this situation, the platform needs to make a self-protection mechanism, for example, the protection mechanism of the platform usually limits the speed of the log data when the data amount reaches the performance bottleneck.
In the prior art, when the speed limit processing is performed on log data, the log types are usually ignored, and a log type undifferentiated random discarding scheme is adopted, wherein the discarding scheme is implemented in a log acquisition stage, that is, raw data is collected without any information processing, and the log is discarded randomly without distinction according to the magnitude of the rate, for example, whether the reporting rate of the log in the last acquisition period is greater than a threshold value or not is calculated, and if the reporting rate is greater than the threshold value, all logs exceeding the threshold value are discarded in the next acquisition period. The method ignores the log type and carries out indiscriminate random discarding, so that the following problems can be caused: when the number distribution of different reported log types is not uniform, for example, the number of the type a < the number of the type B < the number of the type C (< indicating that the number is much smaller), and the attention degree of the platform to the log is the attention degree of the type a > the attention degree of the type B > is the attention degree of the type C (> > indicating that the number is much larger), in this case, the log corresponding to the type C may be greatly reserved, and the logs corresponding to the more important type a and the type B are discarded.
The prior art also adopts a full discarding scheme based on log types, and the implementation process is as follows: and in the log collection stage, all logs corresponding to the log types needing to be shielded are discarded, and only logs with higher service importance degree are left in the log normalization stage, such as collected log complete sets C (C1, C2 and C3), and the log types needing to be shielded are set to be C1, so that all logs belonging to the C1 types are discarded directly in the normalization analysis process. According to the method, although the logs with low relevance types are discarded, so that no logs of c1 types occupy resources any more during subsequent service analysis, when the log rate with high relevance exceeds the bearing range of the platform, the platform is still in an unavailable state; in addition, if the log reporting rate with high correlation is far lower than the bearing capacity of the platform, or even no log is reported in an extreme case, the logs to be processed in the platform are fewer or even no log exists, so that the log processing resources of the platform are wasted.
Therefore, how to perform log processing to avoid the loss of important information or the waste of log processing resources caused by the existing log processing method is one of the considerable technical problems.
Disclosure of Invention
In view of this, the present application provides a method and an apparatus for adjusting a log rate, so as to process a log according to an adjusted log rate by adjusting the log rate of each log type, thereby solving a problem of loss of important information or waste of log processing resources caused by an existing log processing method.
Specifically, the method is realized through the following technical scheme:
according to a first aspect of the present application, there is provided a log rate adjustment method, including:
acquiring log rates of logs respectively reported by a plurality of devices of the same log type in a current acquisition period and calculating a total log rate;
if the total log rate is greater than the total rate threshold of the log type, calculating the threshold average rate of the log type based on the total rate threshold;
for each device, if the log rate of the device is smaller than the threshold average rate, keeping the current log rate unchanged; calculating the compensation deviation of the equipment according to the log rate and the threshold average rate;
if the log rate of the equipment is not less than the threshold average rate, calculating the total compensation deviation of the log type based on the compensation deviation of each equipment with the log rate less than the threshold average rate; and increasing the log rate of the equipment according to the total compensation deviation.
According to a second aspect of the present application, there is provided a log rate adjustment apparatus, including:
the acquisition module is used for acquiring the log rate of the logs respectively reported by a plurality of devices of the same log type in the current acquisition period and calculating the total rate of the logs;
the calculation module is used for calculating the threshold average speed of the log type based on the total speed threshold if the total speed of the log is greater than the total speed threshold of the log type;
the rate adjusting module is used for keeping the current log rate unchanged for each device if the log rate of the device is smaller than the threshold average rate; calculating the compensation deviation of the equipment according to the log rate and the threshold average rate;
the rate adjustment module is further configured to calculate a total compensation deviation of the log type based on the compensation deviation of each device whose log rate is smaller than the threshold average rate if the log rate of the device is not smaller than the threshold average rate; and increasing the log rate of the equipment according to the total compensation deviation.
According to a third aspect of the present application, there is provided an electronic device comprising a processor and a machine-readable storage medium, the machine-readable storage medium storing a computer program executable by the processor, the processor being caused by the computer program to perform the method provided by the first aspect of the embodiments of the present application.
According to a fourth aspect of the present application, there is provided a machine-readable storage medium storing a computer program which, when invoked and executed by a processor, causes the processor to perform the method provided by the first aspect of the embodiments of the present application.
The beneficial effects of the embodiment of the application are as follows:
by comparing the total log rate with the total rate threshold value and comparing the log rate of each device with the average rate of the threshold value, the log rate of each device under each log type is dynamically adjusted, and the logs are processed according to the adjusted log rate, so that the aim of accurate log control is fulfilled, and the situations of important information loss and the like caused by a rough log processing mode are avoided.
Drawings
Fig. 1 is a schematic flowchart of a log rate adjustment method according to an embodiment of the present disclosure;
fig. 2 is a schematic view of an application scenario of a log rate adjustment method according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a log rate adjustment apparatus according to an embodiment of the present disclosure;
fig. 4 is a schematic diagram of a hardware structure of an electronic device implementing a log rate adjustment method according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with aspects such as the present application.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the corresponding listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
Before introducing the log rate adjustment method provided by the present application, the technical terms related to the present application are introduced:
log rate: because the log access is stable and basically has no large mutation, the log rate in the previous period is taken as the log rate of the current time, for example, the previous period is 1 minute, 120w logs are reported, and the log rate of the current time is 2 w/s.
The following describes the log rate adjustment method provided by the present application in detail.
Referring to fig. 1, fig. 1 is a flowchart of a log rate adjustment method provided in the present application, where the method may be applied to a network device having a security management platform, and the method may include the following steps:
s101, obtaining log rates of logs respectively reported by a plurality of devices with the same log type in the current acquisition period and calculating the total log rate.
In this step, the security management platform may count the total number of logs reported by each device in each acquisition period, and then calculate the log rate of the reported logs of the device in the acquisition period according to the acquisition period, that is, the ratio of the total number of logs to the acquisition period. Assuming that N devices reporting logs of the same log type exist, the log rate of the ith device Di calculated according to the method is recorded as speedDi. Therefore, the total log rate total of the N devices under the log type can be calculated, namely
Alternatively, the log types in this application may include, but are not limited to, traffic flow types, and the like.
And S102, if the total log speed is greater than the total speed threshold of the log type, calculating the threshold average speed of the log type based on the total speed threshold.
In this step, the security management platform will determine whether the total log rate total is greater than the total rate threshold value flod corresponding to the log type. If the total is greater than the flow, the log speed of the log uploaded by the equipment in the log type needs to be subjected to speed limiting processing, and based on the fact that the average speed avg of the log type lower threshold is calculated, the formula is avg which is flow/N.
In addition, if the total is not greater than the flow, it indicates that the log rates of the logs respectively reported by the N devices reporting the logs of the log type are currently within a controllable range, which does not cause a large processing pressure of the security management platform or even influence the log processing mechanism of the security management platform, and then the log rates of the logs reported by the N devices can be kept unchanged, and then the logs of the devices are reported according to the respective log rates.
Optionally, the total rate thresholds corresponding to different log types are different, and the total rate thresholds of different log types are adjusted in a linkage manner when being adjusted, specifically, the total rate thresholds of the log types may be adjusted according to the following process: determining a first target log type with a total log rate smaller than a total rate threshold, and confirming a total rate allowance corresponding to the first target log type according to the total log rate of the first target log type and the total rate threshold; determining a second target log type with the total log rate not less than the total rate threshold; and adjusting the total rate threshold of the second target log type according to the total rate allowance.
Specifically, after the total log rate of each log type is determined in the current acquisition cycle, the log type with the total log rate smaller than the total rate threshold value can be screened out, for convenience of description, the screened out log type is recorded as a first target log type, which indicates that the total rate threshold value corresponding to the log type included in the screened out first target log type is set to be large enough, and redundant amount can be allocated to other log types. For example, assuming that there are M log types, a first target log type screened from the M log types includes k log types, and for convenience of understanding, the k log types are denoted as 1 to k, and it should be noted that the order of the k log types in the M log types is not limited in the present application; then, based on the total log rate and the total rate threshold value respectively corresponding to the log types included in the first target log type, the total rate allowance corresponding to the first target log type is confirmed, that is, the total rate allowance corresponding to each log type in the k log types is determinedRate margin y for this log typekI.e. total rate threshold flod for kth log typekTotal log rate total with kth log typekThe difference between them, the formula can be expressed as: y isk＝flodk-totalk(ii) a The sum of the rate margins for the k log types is then determined as the total rate margin y for the first target log type, formulated as
Then, a second target log type with the total log rate not less than the total rate threshold is determined, which indicates that the total rate threshold set by the log type included in the second target log type is a little small, and in order to avoid the occurrence of a log discarding situation caused by reducing the log rate of the log reported by each device under the log type by using a smaller total rate threshold, the total rate thresholds of some second target log types may be slightly increased, that is, the total rate threshold of the second target log type is adjusted based on the determined total rate margin.
It should be noted that the sum of the total rate thresholds for all log types is fixed.
On this basis, in one possible embodiment, the adjusting of the total rate threshold of the second target log type according to the total rate margin may be performed according to the following procedure: determining the number of log types of which the total log rate is not less than a total rate threshold; determining a ratio between the total rate margin and the number; and determining the sum of the total speed threshold corresponding to the second target log type and the ratio as the total speed threshold after the second target log type is adjusted.
Specifically, the number of the log types whose total log rate is not less than the total rate threshold of each log type is the number of the log types included in the second target log type, which is denoted as h, then the ratio y/h between the total rate margin y and the number h is determined, and then, for the log types included in the second target log type, the adjusted total rate threshold corresponding to the log type may be represented as: and flod is flod + y/h.
In another possible embodiment, adjusting the total rate threshold of the second target log type according to the total rate margin may also be performed according to the following procedure: determining the ratio of the total speed threshold values respectively corresponding to the second target log types; and adjusting the total rate threshold of the second target log type according to the ratio and the total rate margin.
Specifically, assuming that there are h log types included in the second target log type, the ratio of the total rate threshold flod corresponding to the h log types may be determined and recorded as: flod1:flod2:……:flodhThen, the adjusted total rate threshold for the jth log type in the second target log type may be expressed as: flodj＝flodj+(flodj/(flod1+flod2+……+flodh))*y。
By implementing any one of the above adjusting schemes of the total rate threshold, the dynamic adjustment of the total rate threshold of each log type meeting the conditions is realized, so that the log rate of the device reporting the log in the corresponding log type is ensured not to be excessively limited, and the log in the corresponding log type is prevented from being excessively discarded.
Optionally, the total rate threshold of the log types may also be adjusted manually, which may be determined according to the actual situation, that is, the security management platform receives the total rate threshold of each log type entered by the user. It should be noted that, when the security management platform is used for the first time, the total rate threshold of each log type is configured by the user based on the log reporting condition of each log type.
S103, aiming at each device, if the log rate of the device is smaller than the threshold average rate, keeping the current log rate unchanged; and calculating a compensation deviation for the device based on the log rate and the threshold average rate.
In this step, when it is determined based on step S102 that the total log rate is greater than the total rate threshold of the log type, it indicates that the log rate of the log reported by each device in the log type is within a controllable range, and continuing to report the log at each log rate does not greatly affect the log processing performance of the security management platform, and does not cause the loss of log information of a certain log type.
In addition, because there are N devices reporting logs in the log type, there may be a case that the log rate of other devices is greater than the threshold average rate, and therefore, in order to compensate the log rate of other devices, a compensation deviation of the device needs to be calculated and recorded in this step to prepare for the other devices to compensate the log rate.
Alternatively, the compensation offset for the device may be calculated from the log rate and the threshold average rate as follows: the difference between the threshold average rate and the log rate is determined as the compensation deviation for the device.
Specifically, assuming that there are t devices having a log rate smaller than the average rate of the threshold value in a certain log type, the compensation deviation of the i-th device is compensatedDiExpressed, the formula for compensating for the deviation can be expressed as: compensatingDi＝avg-speedDiI is 1, … …, t. The compensation deviations for the t devices can be calculated according to this formula.
S104, if the log rate of the equipment is not less than the threshold average rate, calculating the total compensation deviation of the log type based on the compensation deviation of each equipment of which the log rate is less than the threshold average rate; and according to the total compensation deviation, the log speed of the equipment is increased.
Specifically, the compensation deviation of the single device can be calculated by referring to the formula in step S103, and on the basis of this, the total compensation deviation compensation in the log type can be calculated, and the expression is recorded as
On this basis, scaling up the log rate to the device based on the total compensation offset may be performed as follows: determining the sum of the log rates of all the devices with the log rates not less than the threshold average rate; determining a ratio of the logging rate of the device to the sum; determining a product value of the ratio and the total compensation rate; and determining the sum value of the threshold average speed and the product value of the log type as the adjusted log speed of the device.
Specifically, the adjusted log rate may be used as new _ limit _ speedDjIf the number of devices whose logging rate is not less than the threshold average rate is g, the expression of the adjusted logging rate of the jth device may be:
wherein j is 1, … …, g. Therefore, the dynamic adjustment of the log rate of the equipment with the same log type is realized, then the log of the corresponding equipment can be processed by using the adjusted log rate, specifically, the log collection is carried out in the next collection period according to the adjusted log rate, when the number of the collected logs reaches the product value of the adjusted log rate and the next collection period, the rest logs are not collected any more, and the rest logs are directly discarded.
Optionally, the present application further provides the following method: and adjusting the next acquisition period according to the current acquisition period.
Specifically, in order to avoid the situation that the change of the log rate in a short time cannot be perceived in time, the current acquisition period is used for adjusting the next acquisition period. Specifically, the next acquisition period is the current acquisition period + a set number. For example, taking the current acquisition period as 1 minute as an example, the next acquisition period is the current acquisition period +10 seconds × random number (-3 to 3 random numbers) to control the next acquisition period to be 30 seconds to 90 seconds.
By adopting the log rate adjusting method provided by any embodiment of the application, the log rate of each device under each log type is dynamically adjusted by comparing the total rate of the log with the total rate threshold value and comparing the log rate of each device with the average rate of the threshold value, so that the aim of accurate log control is fulfilled, and the occurrence of situations such as loss of important information caused by a rough log processing mode is avoided.
For better understanding of the embodiment, the application scenario shown in fig. 2 is taken as an example for explanation, the log type is taken as a flow type for explanation, 3 devices (D1/D2/D3) report the flow type in fig. 2, the total rate threshold of this flow type is 4500/s, then when log rate adjustment is performed according to the method provided in this application, the log rates of D1, D2 and D3 are calculated respectively, for example, in FIG. 2, the log rate of D1 is 500/s, the log rate of D2 is 3000/s, the log rate of D3 is 3000/s, the total log rate for this flow type can be calculated to be 500+3000+ 3000-6500/s, then judging whether 6500/s is larger than 4500/s, judging that 6500/s is larger than 4500/s, the threshold average rate avg-4500/3-1500/s is calculated. Then for D1, it can be determined that the logging rate 500/s of D1 is less than the threshold average rate 1500/s, then the logging rate of D1 device is kept unchanged while the compensation deviation composition of D1 device is calculatedD11500-; for D2 and D3, it can be determined that both the logging rates 3000/s for D2 and D3 are greater than the threshold average rate 1500/s, respectively, and then the logging rates for D2 and D3 need to be increased. When the log rates of D2 and D3 are adjusted, only D1 can be determined as the device with the log rate smaller than the threshold log rate, so that the total compensation deviation can be determined as the compensation deviation of D1, namely the compensation deviation D11000/s, so that the D2 adjusted log rate can be obtained
Similarly, a D3 adjusted logging rate may be obtained
Therefore, the dynamic adjustment of the log rate of each device under each type is realized, and the loss of important information or the waste of log processing resources caused by the existing log processing mode is avoided.
Based on the same inventive concept, the application also provides a log rate adjusting device corresponding to the log rate adjusting method. The implementation of the log rate adjustment apparatus can refer to the above description of the log rate adjustment method, which is not discussed herein.
Referring to fig. 3, fig. 3 is a log rate adjustment apparatus according to an exemplary embodiment of the present application, including:
an obtaining module 301, configured to obtain log rates of logs respectively reported by multiple devices of the same log type in a current acquisition period, and calculate a total log rate;
a calculating module 302, configured to calculate a threshold average rate of the log type based on the total rate threshold if the total rate of the log is greater than the total rate threshold of the log type;
a rate adjustment module 303, configured to, for each device, if the log rate of the device is smaller than the threshold average rate, keep the current log rate unchanged; calculating the compensation deviation of the equipment according to the log rate and the threshold average rate;
the rate adjustment module 303 is further configured to calculate a total compensation deviation of the log type based on the compensation deviation of each device whose log rate is smaller than the threshold average rate if the log rate of the device is not smaller than the threshold average rate; and increasing the log rate of the equipment according to the total compensation deviation.
Optionally, the total rate thresholds corresponding to different log types are different; on this basis, the log rate adjusting apparatus provided in this embodiment further includes:
a margin determining module (not shown in the figure), configured to determine a first target log type of which a total log rate is smaller than a total rate threshold, and determine, according to the total log rate of the first target log type and the total rate threshold, a total rate margin corresponding to the first target log type;
a type determining module (not shown in the figure) for determining a second target log type of which the total log rate is not less than the total rate threshold;
a threshold adjustment module (not shown in the figure) for adjusting the total rate threshold of the second target log type according to the total rate margin.
Optionally, the rate adjustment module 303 is specifically configured to determine a sum of log rates of the devices whose log rates are not less than the threshold average rate; determining a ratio of the logging rate of the device to the sum; determining a product value of the ratio and the total compensation rate; and determining the sum value of the threshold average speed of the log type and the product value as the adjusted log speed of the equipment.
In a possible embodiment, the threshold adjustment module (not shown in the figure) is specifically configured to determine a sum of log rates of the devices whose log rates are not less than the threshold average rate; determining a ratio of the logging rate of the device to the sum; determining a product value of the ratio and the total compensation rate; and determining the sum value of the threshold average speed of the log type and the product value as the adjusted log speed of the equipment.
In another possible embodiment, the threshold adjustment module (not shown in the figure) is specifically configured to determine a ratio of total rate thresholds respectively corresponding to the second target log types; and adjusting the total rate threshold of the second target log type according to the ratio and the total rate margin.
Optionally, the rate adjustment module 303 is specifically configured to determine a difference between the threshold average rate and the log rate as a compensation deviation of the device.
Optionally, the rate adjustment module 303 is specifically configured to determine a sum of the compensation deviations of the devices whose log rates are smaller than the threshold average rate as a total compensation deviation of the log type.
Therefore, the dynamic adjustment of the log rate of each device under each type is realized, and the loss of important information or the waste of log processing resources caused by the existing log processing mode is avoided.
Based on the same inventive concept, the embodiment of the present application provides an electronic device, as shown in fig. 4, the electronic device includes a processor 401 and a machine-readable storage medium 402, the machine-readable storage medium 402 stores a computer program capable of being executed by the processor 401, and the processor 401 is caused by the computer program to execute the log rate adjustment method provided in any embodiment of the present application. In addition, the electronic device further comprises a communication interface 403 and a communication bus 404, wherein the processor 401, the communication interface 403 and the machine-readable storage medium 402 are in communication with each other via the communication bus 404.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM), a DDR SRAM (Double Data Rate Dynamic Random Access Memory), and a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
In addition, the embodiment of the present application provides a machine-readable storage medium, which stores a computer program, and when the computer program is called and executed by a processor, the computer program causes the processor to execute the log rate adjustment method provided by the embodiment of the present application.
For the embodiments of the electronic device and the machine-readable storage medium, since the contents of the related methods are substantially similar to those of the foregoing embodiments of the methods, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the embodiments of the methods.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The implementation process of the functions and actions of each unit/module in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, wherein the units/modules described as separate parts may or may not be physically separate, and the parts displayed as units/modules may or may not be physical units/modules, may be located in one place, or may be distributed on a plurality of network units/modules. Some or all of the units/modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (12)
1. A method for log rate adjustment, comprising:
acquiring log rates of logs respectively reported by a plurality of devices of the same log type in a current acquisition period and calculating a total log rate;
if the total log rate is greater than the total rate threshold of the log type, calculating the threshold average rate of the log type based on the total rate threshold;
for each device, if the log rate of the device is smaller than the threshold average rate, keeping the current log rate unchanged; calculating the compensation deviation of the equipment according to the log rate and the threshold average rate;
if the log rate of the equipment is not less than the threshold average rate, calculating the total compensation deviation of the log type based on the compensation deviation of each equipment with the log rate less than the threshold average rate; and increasing the log rate of the equipment according to the total compensation deviation.
2. The method of claim 1, wherein the total rate thresholds for different log types are different; the method further comprises the following steps:
determining a first target log type with a total log rate smaller than a total rate threshold, and confirming a total rate allowance corresponding to the first target log type according to the total log rate of the first target log type and the total rate threshold;
determining a second target log type with the total log rate not less than the total rate threshold;
and adjusting the total rate threshold of the second target log type according to the total rate allowance.
3. The method of claim 1, wherein scaling up the log rate of the device based on the total compensation rate comprises:
determining the sum of the log rates of all the devices with the log rates not less than the threshold average rate;
determining a ratio of the logging rate of the device to the sum;
determining a product value of the ratio and the total compensation rate;
and determining the sum value of the threshold average speed of the log type and the product value as the adjusted log speed of the equipment.
4. The method of claim 2, wherein adjusting the total rate threshold for the second target log type based on the total rate margin comprises:
determining the number of log types of which the total log rate is not less than a total rate threshold;
determining a ratio between the total rate margin and the number;
and determining the sum of the total speed threshold corresponding to the second target log type and the ratio as the total speed threshold after the second target log type is adjusted.
5. The method of claim 2, wherein adjusting the total rate threshold for the second target log type based on the total rate margin comprises:
determining the ratio of the total speed threshold values respectively corresponding to the second target log types;
and adjusting the total rate threshold of the second target log type according to the ratio and the total rate margin.
6. The method of claim 1, wherein calculating a compensated deviation for the device based on the log rate and the threshold average rate comprises:
determining a difference between the threshold average rate and the log rate as a compensation deviation for the device.
7. The method of claim 1, wherein calculating the total compensation skew for the log type based on the compensation skews for devices having log rates less than the threshold average rate comprises:
and determining the sum of the compensation deviations of the devices with the log speed less than the threshold average speed as the total compensation deviation of the log type.
8. An apparatus for adjusting a log rate, comprising:
the acquisition module is used for acquiring the log rate of the logs respectively reported by a plurality of devices of the same log type in the current acquisition period and calculating the total rate of the logs;
the calculation module is used for calculating the threshold average speed of the log type based on the total speed threshold if the total speed of the log is greater than the total speed threshold of the log type;
the rate adjusting module is used for keeping the current log rate unchanged for each device if the log rate of the device is smaller than the threshold average rate; calculating the compensation deviation of the equipment according to the log rate and the threshold average rate;
the rate adjustment module is further configured to calculate a total compensation deviation of the log type based on the compensation deviation of each device whose log rate is smaller than the threshold average rate if the log rate of the device is not smaller than the threshold average rate; and increasing the log rate of the equipment according to the total compensation deviation.
9. The apparatus of claim 8, wherein the total rate thresholds for different log types are different; the device, still include:
the allowance determination module is used for determining a first target log type of which the total log rate is smaller than a total rate threshold, and determining the total rate allowance corresponding to the first target log type according to the total log rate of the first target log type and the total rate threshold;
the type determining module is used for determining a second target log type of which the total log rate is not less than the total rate threshold;
and the threshold adjusting module is used for adjusting the total rate threshold of the second target log type according to the total rate allowance.
10. The apparatus of claim 8,
the rate adjustment module is specifically configured to determine a sum of log rates of each device whose log rate is not less than a threshold average rate; determining a ratio of the logging rate of the device to the sum; determining a product value of the ratio and the total compensation rate; and determining the sum value of the threshold average speed of the log type and the product value as the adjusted log speed of the equipment.
11. The apparatus of claim 9,
the threshold adjusting module is specifically configured to determine a sum of log rates of each device whose log rate is not less than a threshold average rate; determining a ratio of the logging rate of the device to the sum; determining a product value of the ratio and the total compensation rate; and determining the sum value of the threshold average speed of the log type and the product value as the adjusted log speed of the equipment.
12. The apparatus of claim 9,
the threshold adjusting module is specifically configured to determine a ratio of total rate thresholds corresponding to the second target log types, respectively; and adjusting the total rate threshold of the second target log type according to the ratio and the total rate margin.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110877419.5A CN114020712A (en) | 2021-07-31 | 2021-07-31 | Log rate adjusting method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110877419.5A CN114020712A (en) | 2021-07-31 | 2021-07-31 | Log rate adjusting method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114020712A true CN114020712A (en) | 2022-02-08 |
Family
ID=80054242
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110877419.5A Pending CN114020712A (en) | 2021-07-31 | 2021-07-31 | Log rate adjusting method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114020712A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115437881A (en) * | 2022-08-25 | 2022-12-06 | 北京万里开源软件有限公司 | Database process exception handling method based on user mode file system |
| US11847038B1 (en) * | 2022-07-15 | 2023-12-19 | Vmware, Inc. | System and method for automatically recommending logs for low-cost tier storage |
-
2021
- 2021-07-31 CN CN202110877419.5A patent/CN114020712A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11847038B1 (en) * | 2022-07-15 | 2023-12-19 | Vmware, Inc. | System and method for automatically recommending logs for low-cost tier storage |
| CN115437881A (en) * | 2022-08-25 | 2022-12-06 | 北京万里开源软件有限公司 | Database process exception handling method based on user mode file system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114020712A (en) | Log rate adjusting method and device | |
| CN109190070B (en) | Data processing method, device and system and application server | |
| CN109005056B (en) | CDN application-based storage capacity evaluation method and device | |
| CN112751726B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN107623862B (en) | Multimedia information push control method and device and server | |
| US20190361607A1 (en) | Providing combined data from a cache and a storage device | |
| CN106339331B (en) | A kind of data buffer storage stratification scaling method based on user activity | |
| CN110209502B (en) | Information storage method and device, electronic equipment and storage medium | |
| US20230070276A1 (en) | Prediction method and apparatus, readable medium and electronic device | |
| CN110647400B (en) | Node preheating method based on positive feedback | |
| CN111142942A (en) | Window data processing method and device, server and storage medium | |
| CN111626498B (en) | Equipment running state prediction method, device, equipment and storage medium | |
| WO2020036763A1 (en) | Testing data changes in production systems | |
| CN114528231A (en) | Data dynamic storage method and device, electronic equipment and storage medium | |
| CN108810832B (en) | Short message issuing method and device and computer readable storage medium | |
| WO2023115875A1 (en) | Hardware device maintenance method and apparatus, and electronic device | |
| CN112367384A (en) | Kafka cluster-based dynamic speed limiting method and device and computer equipment | |
| CN111369084A (en) | Resource quality scoring method in content network | |
| CN117519608B (en) | Big data server with Hadoop as core | |
| CN117033004B (en) | Load balancing method and device, electronic equipment and storage medium | |
| CN111563115B (en) | Statistical method and device for data distribution information in distributed database | |
| CN111783107B (en) | Multi-source trusted data access method, device and equipment | |
| CN111400045B (en) | Load balancing method and device | |
| CN106991101B (en) | Data table analysis processing method and device | |
| CN112102040A (en) | Global inventory control method and system in distributed environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |