CN109597800B - Log distribution method and device - Google Patents
Log distribution method and device Download PDFInfo
- Publication number
- CN109597800B CN109597800B CN201811513533.4A CN201811513533A CN109597800B CN 109597800 B CN109597800 B CN 109597800B CN 201811513533 A CN201811513533 A CN 201811513533A CN 109597800 B CN109597800 B CN 109597800B
- Authority
- CN
- China
- Prior art keywords
- log processing
- processing server
- probability value
- determining
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the application discloses a method and a device for distributing logs. Then, the probability value of the log processing server is determined as the hit interval length of the log processing server. And finally, generating a random number, determining the log processing server corresponding to the hit interval to which the random number belongs as the current log processing server, and sending the log data to the current log processing server. The method comprises the steps that the writing speed corresponding to the log processing server is used for determining the hit interval corresponding to the log processing server, the writing speed is in direct proportion to the probability value, the higher the writing speed is, the higher the probability value is, the larger the hit interval corresponding to the log processing server is, so that the probability that the log processing server is selected is increased, and the effect of automatic load balancing of each log processing server is achieved.
Description
Technical Field
The application relates to the technical field of internet, in particular to a log distribution method and device.
Background
The network device, the service program and the like generate logs during operation, and the logs need to be collected and processed in some scenes so as to analyze the logs.
In the prior art, an Agent proxy server can be generally adopted to collect logs, and the logs are sent to a log processing server by the Agent server so as to realize processing such as cleaning, matching, extraction and the like of the logs. Because the log processing process is complex, the logs collected by one Agent server are generally required to be distributed to a plurality of log processing servers for processing, so that the performances of the Agent server and the log processing servers are fully exerted.
Existing log distribution may employ a pre-established distribution strategy, such as allocating log processing servers according to log source. However, this method has a high requirement on the configuration policy, and when the log characteristics change greatly, some log processing servers are under excessive pressure, and if the log processing servers are down, a large amount of logs cannot be processed.
Disclosure of Invention
In view of this, embodiments of the present application provide a log distribution method and apparatus, so as to solve the technical problem in the prior art that log distribution cannot reasonably utilize a log processing server.
In order to solve the above problem, the technical solution provided by the embodiment of the present application is as follows:
a method of log distribution, the method comprising:
acquiring the writing speed corresponding to each log processing server;
determining a probability value of a first target log processing server according to the writing speed corresponding to each log processing server, wherein the probability value of the first target log processing server is in direct proportion to the writing speed corresponding to the first target log processing server, and the first target log processing server is respectively each log processing server;
determining the probability value of the first target log processing server as the length of a hit interval of the first target log processing server, and sequentially determining the hit intervals of all the log processing servers;
and generating a random number belonging to any hit interval, determining a log processing server corresponding to the hit interval to which the random number belongs as a current log processing server, and sending log data to the current log processing server.
In a possible implementation manner, the obtaining the writing speed corresponding to each log processing server includes:
calculating an average value of multiple writing speeds of a target log processing server, and determining the average value of the multiple writing speeds of the target log processing server as the writing speed of the target log processing server.
In a possible implementation manner, the determining a probability value of a first target log processing server according to a writing speed corresponding to each log processing server includes:
calculating the sum of the writing speeds corresponding to the log processing servers as the sum of the writing speeds;
and dividing the writing speed of the first target log processing server by the sum of the writing speeds to obtain a probability value of the first target log processing server.
In one possible implementation, the method further includes:
recording the writing times corresponding to each log processing server;
the determining a probability value of a first target log processing server according to the writing speed corresponding to each log processing server includes:
determining the reciprocal of the total number of the log processing servers as an initial probability value of each log processing server;
if the write-in times of any log processing server do not reach a preset threshold value, determining the log processing server with the write-in times not reaching the preset threshold value as an initial log processing server, and determining the probability value of the initial log processing server as the initial probability value;
calculating a sum of writing speeds corresponding to the log processing servers which are not determined as the initial log processing servers as a sum of the writing speeds;
and dividing the writing speed of a second target log processing server by the sum of the writing speeds, and multiplying the result by a residual probability value to obtain a probability value of the second target log processing server, wherein the residual probability value is obtained by subtracting the product of the number of the initial log processing servers and the initial probability value from 1, and the second target log processing server is each log processing server which is not determined as the initial log processing server.
In a possible implementation manner, the determining a probability value of a first target log processing server according to a writing speed corresponding to each log processing server includes:
and if receiving an error identifier sent by any log processing server, determining the log processing server sending the error identifier as an abnormal log processing server, and determining the probability value of the abnormal log processing server as zero.
In one possible implementation, the method further includes:
recording the last write-in time corresponding to each log processing server;
the determining a probability value of a first target log processing server according to the writing speed corresponding to each log processing server includes:
and according to the last write-in time corresponding to each log processing server, determining the log processing server with the last write-in time exceeding a preset time threshold value as a log processing server to be recovered, and determining the probability value of the log processing server to be recovered as the initial probability value.
An apparatus for distributing logs, the apparatus comprising:
the acquisition unit is used for acquiring the writing speed corresponding to each log processing server;
a first determining unit, configured to determine a probability value of a first target log processing server according to a writing speed corresponding to each log processing server, where the probability value of the first target log processing server is proportional to the writing speed corresponding to the first target log processing server, and the first target log processing server is each of the log processing servers;
a second determining unit, configured to determine the probability value of the first target log processing server as the length of a hit interval of the first target log processing server, and sequentially determine hit intervals of the log processing servers;
and the third determining unit is used for generating a random number belonging to any hit section, determining a log processing server corresponding to the hit section to which the random number belongs as a current log processing server, and sending log data to the current log processing server.
In a possible implementation manner, the obtaining unit includes:
a first calculating subunit, configured to calculate an average value of write speeds of the target log processing server for a plurality of times;
a first determining subunit, configured to determine an average value of the multiple writing speeds of the target log processing server as the writing speed of the target log processing server.
In a possible implementation manner, the first determining unit includes:
a second calculating subunit, configured to calculate a sum of write speeds corresponding to the respective log processing servers as a write speed sum;
and the third calculation subunit is used for dividing the writing speed of the first target log processing server by the sum of the writing speeds to obtain a probability value of the first target log processing server.
In one possible implementation, the apparatus further includes:
the first recording unit is used for recording the writing times corresponding to each log processing server;
the first determination unit includes:
a second determining subunit, configured to determine an inverse of the total number of the log processing servers as an initial probability value of each log processing server;
a third determining subunit, configured to determine, if the write-in frequency of any log processing server does not reach a preset threshold, the log processing server whose write-in frequency does not reach the preset threshold as an initial log processing server, and determine the probability value of the initial log processing server as the initial probability value;
a fourth calculating subunit configured to calculate, as a write speed sum, a sum of write speeds corresponding to each log processing server that is not determined as the initial log processing server;
and the fifth calculating subunit is configured to divide the writing speed of a second target log processing server by the sum of the writing speeds, and then multiply a remaining probability value by the sum of the writing speeds to obtain a probability value of the second target log processing server, where the remaining probability value is obtained by subtracting a product of the number of the initial log processing servers and the initial probability value from 1, and the second target log processing server is each of the log processing servers that are not determined as the initial log processing servers.
In a possible implementation manner, the first determining unit is further configured to determine, if it is received that any log processing server sends an error identifier, the log processing server that sends the error identifier as an abnormal log processing server, and determine the probability value of the abnormal log processing server as zero.
In one possible implementation, the apparatus further includes:
the second recording unit is used for recording the last write-in time corresponding to each log processing server;
the first determining unit is specifically configured to determine, according to the last write time corresponding to each log processing server, the log processing server whose last write time exceeds a preset time threshold as a log processing server to be restored, and determine the probability value of the log processing server to be restored as the initial probability value.
A computer readable storage medium having stored therein instructions which, when run on a terminal device, cause the terminal device to perform the above-described method of log distribution.
A computer program product which, when run on a terminal device, causes the terminal device to perform the method of log distribution described above.
Therefore, the embodiment of the application has the following beneficial effects:
according to the method and the device, the writing speed corresponding to each log processing server is firstly obtained, and the probability value of each log processing server is determined according to the writing speed of each log processing server. Then, the probability value of the log processing server is determined as the hit section length of the log processing server. And finally, generating a random number, determining the log processing server corresponding to the hit interval to which the random number belongs as the current log processing server, and sending the log data to the current log processing server for log processing. It can be seen that in the embodiment of the present application, the hit interval corresponding to each log processing server is determined by the write-in speed corresponding to each log processing server, and since the write-in speed is proportional to the probability value, the greater the write-in speed is, the greater the probability value is, the greater the hit interval corresponding to the log processing server is, so that the probability that the log processing server is selected is increased, thereby implementing that the write-in speed of different log processing servers is used as a reference, the log processing servers are randomly selected to be sent, and achieving the effect of automatic load balancing of each log processing server.
Drawings
Fig. 1 is a schematic diagram of a framework of an exemplary application scenario provided in an embodiment of the present application;
fig. 2 is a flowchart of a log distribution method according to an embodiment of the present application;
fig. 3 is a flowchart of a method for determining a probability value according to a speed of a log processing server according to an embodiment of the present application;
fig. 4 is a structural diagram of a log distribution apparatus according to an embodiment of the present application.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, embodiments accompanying the drawings are described in detail below.
In order to facilitate understanding of the technical solutions provided in the present application, the following first describes the background art of the present application.
The inventor finds that the traditional log distribution method comprises three strategies through research on the traditional log distribution method. The first is a main and standby strategy, an Agent server sends logs to one log processing server first, and when the log processing server is down or has low performance, the Agent server is switched to other log processing servers. However, this distribution strategy is prone to the main log processing server being fully loaded, while the standby log processing server is empty. The second is a round-robin strategy, which fairly sends the logs to each log processing server. Due to the sequential transmission, when the speed of one log processing server is low, the subsequent distribution operation is applied, so that the overall transmission speed is affected by the reduction of the speed of one log processing server. The third is to configure a policy in advance, which can configure a distribution policy in advance according to the characteristics of the path, the address, etc., and perform log distribution according to the configured distribution policy. However, such a policy configuration requires a high experience value, and when the data characteristics change, the log processing server cannot cope with the change, so that the log is lost.
Based on this, the embodiment of the application provides a log distribution method and device, and for an Agent server, an important purpose of load balancing is to send out logs as soon as possible. The writing speed of the log processing server can reflect the network condition and the processing performance of the log processing server, and can also accurately reflect the speed of the log leaving the Agent server. Meanwhile, the calculation complexity can be reduced to the maximum extent. Therefore, according to the method, the probability value of each log processing server is determined according to the writing speed of each log processing server, and the probability value is determined as the length of the hit interval of the log processing server, namely, the larger the probability value is, the longer the length of the hit interval corresponding to the log processing server is, the higher the probability of being selected is. Meanwhile, the agent server determines the log processing server corresponding to the hit section to which the random number belongs as the current log processing server by generating the random number, and sends the log data to the current log processing server. Therefore, by the method provided by the embodiment of the application, the current log processing server can be selected according to the writing speed corresponding to each log processing server, and when the writing speed changes, the length of the corresponding hit interval can also change, so that the selected probability changes, thereby dynamically selecting the log processing servers and realizing the automatic load balancing effect.
Referring to fig. 1, the figure is a schematic diagram of a framework of an exemplary application scenario provided in an embodiment of the present application. The log distribution method provided by the embodiment of the application can be applied to the Agent server 10.
In actual application, the Agent server 10 performs a write operation on the log processing server 20, the log processing server 30, and the log processing server 40, so that a write speed corresponding to each log processing server can be obtained, a probability value of each log processing server can be determined according to the write speed corresponding to each log processing server, and the hit interval length of each log processing server can be determined according to the probability value. Then, the Agent server 10 generates a random number, and determines the log processing server corresponding to the hit section in which the random number falls as the current log processing server.
Those skilled in the art will appreciate that the block diagram shown in fig. 1 is only one example in which embodiments of the present application may be implemented. The scope of applicability of the embodiments of the present application is not limited in any way by this framework.
It should be noted that the Agent server 10 in the embodiment of the present application may be an example of an existing, developing or future-developed device capable of collecting logs and distributing log services. The log processing server 20, the log processing server 30, and the log processing server 40 may be one example of existing, developing, or future-developed devices capable of providing log processing services. The embodiments of the present application are not limited in any way in this respect.
In order to facilitate understanding of the technical solutions provided by the present application, a log distribution method provided by the embodiments of the present application will be described below with reference to the accompanying drawings.
Referring to fig. 2, which is a flowchart of a log distribution method provided in an embodiment of the present application, as shown in fig. 2, the method may include:
s201: and acquiring the writing speed corresponding to each log processing server.
In this embodiment, the Agent server that collects log data may forward the log data to each log processing server. In practical application, the Agent server can write log data into the log processing server through a buffer area with a fixed size between the Agent server and the log processing server, namely, the Agent server writes data into the buffer area, the log processing server reads data from the buffer area, when the processing speed of the log processing server is higher than the speed of the Agent server for writing data, the buffer area cannot be blocked, and when the processing speed of the log processing server is not fast enough, the buffer area can be blocked, so that the writing speed of the Agent server is influenced.
Therefore, the Agent server can obtain the writing speed corresponding to each log processing server so as to select the log processing server subsequently.
In practical application, for any log processing server, the corresponding writing speed within a certain period of time may reflect the processing performance of the log processing server. Therefore, when the write speed corresponding to the log processing server is acquired, the average value of the write speeds corresponding to the log processing server for a plurality of times can be used as the write speed corresponding to the log processing server. Specifically, an average value of the multiple writing speeds of the target log processing server is calculated, and the average value of the multiple writing speeds of the target log processing server is determined as the writing speed of the target log processing server.
In practical application, an average value of the latest M writing speeds corresponding to the target log processing server may be calculated, and the average value is determined as the writing speed of the target log processing server, where M is an integer greater than 1. For example, the Agent server may record the write speed corresponding to the last 5 times of each log processing server, and thus, an average value of the write speeds of the last 5 times may be calculated, and the average value may be taken as the write speed of the target log processing server.
S202: and determining the probability value of the first target log processing server according to the writing speed corresponding to each log processing server.
In this embodiment, after the writing speed corresponding to each log processing server is determined, the probability value of the log processing server is determined according to the writing speed corresponding to the log processing server.
The first target log processing server is each log processing server; the probability value of the first target log processing server is in direct proportion to the writing speed corresponding to the first target log processing server. That is, the greater the write speed corresponding to the first target processing server, the greater the probability value. The probability value may characterize a probability that the log processing server may be selected for log data writing.
In this embodiment, two implementation manners for determining the probability value according to the writing speed corresponding to the log processing server are provided, and will be described in the following embodiments, which will not be described for the moment.
S203: and determining the probability value of the first target log processing server as the length of the hit interval of the first target log processing server, and sequentially determining the hit intervals of the log processing servers.
In this embodiment, after the probability value of each first target log processing server is determined, the probability value is determined as the length of the hit interval of the first target log processing server, so as to obtain the hit interval of each log processing server.
It can be understood that the greater the probability value corresponding to the log processing server, the longer the hit interval length of the log processing service is, the greater the probability of being hit is. The hit sections corresponding to the adjacent log processing servers are continuous, and the lengths of all the hit sections are added to 1. The hit interval of the current log processing server can be obtained according to the previous hit interval and the probability value of the current log processing server, specifically, the maximum value of the previous hit interval is used as the minimum value of the current hit interval; and adding the maximum value of the previous hit interval with the probability value of the current log processing server to obtain the maximum value of the current hit interval, thereby forming the hit interval of the current log processing server.
For example, if the probability value of the first log processing server is 0.25, the hit interval corresponding to the first log processing server is [0,0.25 ], if the probability value of the second log processing server is 0.3, the hit interval corresponding to the second log processing server is [0.25,0.55 ], if the probability value of the third log processing server is 0.45, the hit interval corresponding to the third log processing server is [0.55, 1).
S204: and generating a random number belonging to any hit interval, determining a log processing server corresponding to the hit interval to which the random number belongs as a current log processing server, and sending the log data to the current log processing server.
In this embodiment, when the hit interval of each log processing server is obtained, the Agent server may select a log processing server that processes log data by generating a random number. After the random number is generated, determining a hit interval where the random number is located, determining a log processing server corresponding to the hit interval to which the random number belongs as a current log processing server, and sending log data to the current log processing server for log data processing.
For example, the Agent server generates 0.47654 random number, and the random number is located at [0.25,0.55), then the second log processing server is determined as the current log processing server, and the log data is sent to the second log processing server for processing.
According to the embodiment, the Agent server firstly acquires the writing speed corresponding to each log processing server, and determines the probability value of each log processing server according to the writing speed of each log processing server. Then, the probability value of the log processing server is determined as the hit section length of the log processing server. And finally, generating a random number, determining the log processing server corresponding to the hit interval to which the random number belongs as the current log processing server, and sending the log data to the current log processing server for log processing. It can be seen that in the embodiment of the present application, the hit interval corresponding to each log processing server is determined by the write-in speed corresponding to each log processing server, and since the write-in speed is proportional to the probability value, the greater the write-in speed is, the greater the probability value is, the greater the hit interval corresponding to the log processing server is, so that the probability that the log processing server is selected is increased, thereby implementing that the write-in speed of different log processing servers is used as a reference, the log processing servers are randomly selected to be sent, and achieving the effect of automatic load balancing of each log processing server.
In a possible implementation manner of the embodiment of the present application, two implementation manners for determining a probability value according to a writing speed corresponding to a log processing server are provided, one is to calculate a sum of writing speeds corresponding to each log processing server as a writing speed sum; and then dividing the writing speed corresponding to the first target log processing server by the sum of the writing speeds to obtain a probability value of the first target log processing server. For example, if three log processing servers coexist and the corresponding writing speeds are v1, v2, and v3, respectively, the probability value p1 of the first log processing server is v1/(v1+ v2+ v3), the probability value p2 of the second log processing server is v2/(v1+ v2+ v3), and the probability value p3 of the third log processing server is v3/(v1+ v2+ v 3).
The second method is to determine according to the number of writes corresponding to each log processing server, and for facilitating understanding of the implementation of the second method for determining the probability value of the log processing server, the implementation will be described below with reference to the accompanying drawings.
Referring to fig. 3, which is a flowchart of a method for determining a probability value of a log processing server according to an embodiment of the present application, as shown in fig. 3, the method may include:
s301: and recording the writing times corresponding to each log processing server.
In this embodiment, the Agent server may record the number of times of writing to each log processing server. That is, when the Agent server performs a write operation on a certain log processing service, the number of writes is increased by 1, so that the number of writes corresponding to each log processing server is recorded.
S302: and determining the reciprocal of the total number of the log processing servers as the initial probability value of each log processing server.
In this embodiment, the reciprocal of the total number of the log processing servers is used as the initial probability value of each log processing server. For example, if there are N log processing servers, the initial probability value is 1/N.
S303: and if the writing times of any log processing server do not reach the preset threshold value, determining the log processing server with the writing times not reaching the preset threshold value as an initial log processing server, and determining the probability value of the initial log processing server as an initial probability value.
In this embodiment, when the probability value is determined according to the writing speed of the log processing servers, it is determined whether the writing frequency of each log processing server reaches a preset threshold, and if not, the log processing server whose writing frequency does not reach the preset threshold is determined as an initial log processing server, and the initial probability value is used as the probability value of the initial log processing server.
The preset threshold value can be preset, and the specific numerical value can be set according to actual requirements. For example, if the preset threshold is 5, it is determined whether the number of times of writing to the log processing server reaches 5 times, and if not, the initial probability value is taken as the probability value of the log processing server.
And when the writing times of the log processing server reach a preset threshold, executing the subsequent steps, and determining a probability value according to the writing speed corresponding to the log processing server.
In addition, when the Agent server receives an error identifier sent by a certain log processing server, the log processing server is abnormal, and the Agent server cannot write, the log processing server sending the error identifier is determined as an abnormal log processing server, and the probability value of the abnormal log processing server is determined as 0. That is, when the log processing server is selected for log data processing, the abnormal log processing server is not considered.
S304: the sum of the write speeds corresponding to the respective log processing servers that are not determined as the initial log processing servers is calculated as the write speed sum.
S305: and dividing the writing speed of the second target log processing server by the sum of the writing speeds, and multiplying the result by the residual probability value to obtain the probability value of the second target log processing server.
In this embodiment, the sum of the writing speeds corresponding to the log processing servers whose writing times reach the preset threshold is calculated, and the sum is used as the sum of the writing speeds. And dividing the writing speed of each log processor which is not determined as the initial log processing server by the sum of the total writing number degrees and multiplying the sum by the residual probability value to obtain the probability value of each log processor which is not determined as the initial log processing server.
Wherein the remaining probability value is 1 minus a product of the number of the initial log processing servers and the initial probability value, and the second target log processing server is each of the log processing servers that are not determined as the initial log processing servers.
For example, if there are four log processing servers, the initial probability value is 1/4, wherein if the first log processing server is abnormal, the probability value is 0; if the writing times of the second log processing server do not reach 5 times, the probability value is an initial probability value 1/4; the speed of the third log processing server is 400b/ms, the speed of the fourth log processing server is 600b/ms, and the probability value of the third log processing server is (1-0.25) × 400/(400+600) ═ 0.3; the probability value of the fourth log processing server is (1-0.25) × 600/(400+600) ═ 0.45. Further, the hit section of each log processing server may be determined according to the probability value of the log processing server, and according to the above example, the hit section of the first log processing server is [0,0 ], the hit section of the second log processing server is [0,0.25 ], and the hit section of the third log processing server is: [0.25,0.55 ], the hit section of the fourth log processing server is: [0.55,1).
According to the embodiment, when the writing times of the log processing server do not reach the preset threshold value, the initial probability value is taken as the probability value of the log processing server; when the writing times of the log processing server reach a preset threshold value, a probability value is determined according to the writing speed of the log processing server, so that the written log processing server is selected by taking the writing speed of the log processing server as a parameter, and load balance is achieved.
It can be understood that, when the number of writes of a log processing server reaches a preset threshold, but the corresponding write speed is small, the length of the corresponding hit interval may also be small, and the probability of being selected is small. Moreover, the abnormal log processing server is not selected because the corresponding probability value is 0. When the write speed corresponding to the log processing server with the lower write speed is increased and/or the abnormal log processing server is recovered to be normal, the probability value of the log processing server can be updated in order to ensure that the log processing server is selected at an opportunity.
During specific implementation, recording the last write-in time corresponding to each log processing server; according to the last write-in time corresponding to each log processing server, determining the log processing server with the last write-in time exceeding a preset time threshold value as a log processing server to be recovered, and determining the probability value of the log processing server to be recovered as an initial probability value.
In this embodiment, the Agent server may record the last write time corresponding to each log processing server, and then determine whether the difference value exceeds a preset time threshold according to the difference value between the current time and the last write time, if the difference value exceeds the preset time threshold, determine the log processing server as a to-be-recovered log processing server, and determine the probability value of the to-be-recovered log processing server as an initial probability value, thereby updating the probability value of each log processing server to ensure that each log processing server is given an opportunity to be selected.
In practical application, the preset threshold value can be set according to practical requirements. For example, the preset time threshold is 10 seconds, when the last write time corresponding to a certain log processing server exceeds 10 seconds to the current time, the log processing server is determined as a to-be-recovered log processing server, and the probability value of the to-be-recovered log processing server is determined as the initial probability value.
Based on the above method embodiment, the present application provides a log distribution apparatus, which will be described below with reference to the accompanying drawings.
Referring to fig. 4, which is a structural diagram of a log distributing apparatus according to an embodiment of the present application, as shown in fig. 4, the apparatus may include:
an obtaining unit 401, configured to obtain a writing speed corresponding to each log processing server;
a first determining unit 402, configured to determine a probability value of a first target log processing server according to a writing speed corresponding to each log processing server, where the probability value of the first target log processing server is in direct proportion to the writing speed corresponding to the first target log processing server, and the first target log processing server is each of the log processing servers;
a second determining unit 403, configured to determine the probability value of the first target log processing server as the length of the hit interval of the first target log processing server, and sequentially determine the hit intervals of the log processing servers;
a third determining unit 403, configured to generate a random number belonging to any of the hit sections, determine a log processing server corresponding to the hit section to which the random number belongs as a current log processing server, and send log data to the current log processing server.
In a possible implementation manner, the obtaining unit includes:
a first calculating subunit, configured to calculate an average value of write speeds of the target log processing server for a plurality of times;
a first determining subunit, configured to determine an average value of the multiple writing speeds of the target log processing server as the writing speed of the target log processing server.
In a possible implementation manner, the first determining unit includes:
a second calculating subunit, configured to calculate a sum of write speeds corresponding to the respective log processing servers as a write speed sum;
and the third calculation subunit is used for dividing the writing speed of the first target log processing server by the sum of the writing speeds to obtain a probability value of the first target log processing server.
In one possible implementation, the apparatus further includes:
the first recording unit is used for recording the writing times corresponding to each log processing server;
the first determination unit includes:
a second determining subunit, configured to determine an inverse of the total number of the log processing servers as an initial probability value of each log processing server;
a third determining subunit, configured to determine, if the write-in frequency of any log processing server does not reach a preset threshold, the log processing server whose write-in frequency does not reach the preset threshold as an initial log processing server, and determine the probability value of the initial log processing server as the initial probability value;
a fourth calculating subunit configured to calculate, as a write speed sum, a sum of write speeds corresponding to each log processing server that is not determined as the initial log processing server;
and the fifth calculating subunit is configured to divide the writing speed of a second target log processing server by the sum of the writing speeds, and then multiply a remaining probability value by the sum of the writing speeds to obtain a probability value of the second target log processing server, where the remaining probability value is obtained by subtracting a product of the number of the initial log processing servers and the initial probability value from 1, and the second target log processing server is each of the log processing servers that are not determined as the initial log processing servers.
In a possible implementation manner, the first determining unit is further configured to determine, if it is received that any log processing server sends an error identifier, the log processing server that sends the error identifier as an abnormal log processing server, and determine the probability value of the abnormal log processing server as zero.
In one possible implementation, the apparatus further includes:
the second recording unit is used for recording the last write-in time corresponding to each log processing server;
the first determining unit is specifically configured to determine, according to the last write time corresponding to each log processing server, the log processing server whose last write time exceeds a preset time threshold as a log processing server to be restored, and determine the probability value of the log processing server to be restored as the initial probability value.
It should be noted that, the implementation of each unit or module in this embodiment may refer to the implementation of fig. 1 to fig. 3, and details of this embodiment are not described herein again.
In addition, an embodiment of the present application further provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are executed on a terminal device, the terminal device is caused to execute the log distribution method.
The embodiment of the present application further provides a computer program product, and when the computer program product runs on a terminal device, the terminal device is enabled to execute the log distribution method.
According to the embodiment, the writing speed corresponding to each log processing server is firstly obtained, and the probability value of the log processing server is determined according to the writing speed of each log processing server. Then, the probability value of the log processing server is determined as the hit section length of the log processing server. And finally, generating a random number, determining the log processing server corresponding to the hit interval to which the random number belongs as the current log processing server, and sending the log data to the current log processing server for log processing. It can be seen that in the embodiment of the present application, the hit interval corresponding to each log processing server is determined by the write-in speed corresponding to each log processing server, and since the write-in speed is proportional to the probability value, the greater the write-in speed is, the greater the probability value is, the greater the hit interval corresponding to the log processing server is, so that the probability that the log processing server is selected is increased, thereby implementing that the write-in speed of different log processing servers is used as a reference, the log processing servers are randomly selected to be sent, and achieving the effect of automatic load balancing of each log processing server.
It should be noted that, in the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. For the system or the device disclosed by the embodiment, the description is simple because the system or the device corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the method part for description.
It should be understood that in the present application, "at least one" means one or more, "a plurality" means two or more. "and/or" for describing an association relationship of associated objects, indicating that there may be three relationships, e.g., "a and/or B" may indicate: only A, only B and both A and B are present, wherein A and B may be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of single item(s) or plural items. For example, at least one (one) of a, b, or c, may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.
It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (8)
1. A method of log distribution, the method comprising:
acquiring the writing speed corresponding to each log processing server;
recording the writing times corresponding to each log processing server;
determining a probability value of a first target log processing server according to the writing speed corresponding to each log processing server, wherein the probability value of the first target log processing server is in direct proportion to the writing speed corresponding to the first target log processing server, and the first target log processing server is respectively each log processing server;
determining the probability value of the first target log processing server as the length of a hit interval of the first target log processing server, and sequentially determining the hit intervals of all the log processing servers;
generating a random number belonging to any hit interval, determining a log processing server corresponding to the hit interval to which the random number belongs as a current log processing server, and sending log data to the current log processing server;
the determining a probability value of a first target log processing server according to the writing speed corresponding to each log processing server includes:
determining the reciprocal of the total number of the log processing servers as an initial probability value of each log processing server;
if the write-in times of any log processing server do not reach a preset threshold value, determining the log processing server with the write-in times not reaching the preset threshold value as an initial log processing server, and determining the probability value of the initial log processing server as the initial probability value;
calculating a sum of writing speeds corresponding to the log processing servers which are not determined as the initial log processing servers as a sum of the writing speeds;
and dividing the writing speed of a second target log processing server by the sum of the writing speeds, and multiplying the result by a residual probability value to obtain a probability value of the second target log processing server, wherein the residual probability value is obtained by subtracting the product of the number of the initial log processing servers and the initial probability value from 1, and the second target log processing server is each log processing server which is not determined as the initial log processing server.
2. The method of claim 1, wherein the obtaining the writing speed corresponding to each log processing server comprises:
calculating an average value of multiple writing speeds of a target log processing server, and determining the average value of the multiple writing speeds of the target log processing server as the writing speed of the target log processing server.
3. The method of claim 1, wherein determining the probability value of the first target log processing server according to the writing speed corresponding to each log processing server comprises:
calculating the sum of the writing speeds corresponding to the log processing servers as the sum of the writing speeds;
and dividing the writing speed of the first target log processing server by the sum of the writing speeds to obtain a probability value of the first target log processing server.
4. The method of claim 1, wherein determining the probability value of the first target log processing server according to the writing speed corresponding to each log processing server comprises:
and if receiving an error identifier sent by any log processing server, determining the log processing server sending the error identifier as an abnormal log processing server, and determining the probability value of the abnormal log processing server as zero.
5. The method of claim 1 or 4, further comprising:
recording the last write-in time corresponding to each log processing server;
the determining a probability value of a first target log processing server according to the writing speed corresponding to each log processing server includes:
and according to the last write-in time corresponding to each log processing server, determining the log processing server with the last write-in time exceeding a preset time threshold value as a log processing server to be recovered, and determining the probability value of the log processing server to be recovered as the initial probability value.
6. An apparatus for distributing logs, the apparatus comprising:
the acquisition unit is used for acquiring the writing speed corresponding to each log processing server;
the first recording unit is used for recording the writing times corresponding to each log processing server;
a first determining unit, configured to determine a probability value of a first target log processing server according to a writing speed corresponding to each log processing server, where the probability value of the first target log processing server is proportional to the writing speed corresponding to the first target log processing server, and the first target log processing server is each of the log processing servers;
a second determining unit, configured to determine the probability value of the first target log processing server as the length of a hit interval of the first target log processing server, and sequentially determine hit intervals of the log processing servers;
a third determining unit, configured to generate a random number belonging to any of the hit sections, determine a log processing server corresponding to the hit section to which the random number belongs as a current log processing server, and send log data to the current log processing server;
the first determination unit includes:
a second determining subunit, configured to determine an inverse of the total number of the log processing servers as an initial probability value of each log processing server;
a third determining subunit, configured to determine, if the write-in frequency of any log processing server does not reach a preset threshold, the log processing server whose write-in frequency does not reach the preset threshold as an initial log processing server, and determine the probability value of the initial log processing server as the initial probability value;
a fourth calculating subunit configured to calculate, as a write speed sum, a sum of write speeds corresponding to each log processing server that is not determined as the initial log processing server;
and the fifth calculating subunit is configured to divide the writing speed of a second target log processing server by the sum of the writing speeds, and then multiply a remaining probability value by the sum of the writing speeds to obtain a probability value of the second target log processing server, where the remaining probability value is obtained by subtracting a product of the number of the initial log processing servers and the initial probability value from 1, and the second target log processing server is each of the log processing servers that are not determined as the initial log processing servers.
7. The apparatus of claim 6, wherein the obtaining unit comprises:
a first calculating subunit, configured to calculate an average value of write speeds of the target log processing server for a plurality of times;
a first determining subunit, configured to determine an average value of the multiple writing speeds of the target log processing server as the writing speed of the target log processing server.
8. A computer-readable storage medium having stored therein instructions that, when run on a terminal device, cause the terminal device to perform the method of log distribution of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811513533.4A CN109597800B (en) | 2018-12-11 | 2018-12-11 | Log distribution method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811513533.4A CN109597800B (en) | 2018-12-11 | 2018-12-11 | Log distribution method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109597800A CN109597800A (en) | 2019-04-09 |
| CN109597800B true CN109597800B (en) | 2020-11-06 |
Family
ID=65961739
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811513533.4A Active CN109597800B (en) | 2018-12-11 | 2018-12-11 | Log distribution method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109597800B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111061697B (en) * | 2019-12-25 | 2023-06-13 | 中国联合网络通信集团有限公司 | Log data processing method and device, electronic equipment and storage medium |
| CN112199043A (en) * | 2020-09-30 | 2021-01-08 | 深圳壹账通智能科技有限公司 | Server selection method and device, electronic equipment and storage medium |
| CN112926822A (en) * | 2021-01-19 | 2021-06-08 | 北京电解智科技有限公司 | Resource allocation method and device |
| CN112633996B (en) * | 2021-03-05 | 2021-09-14 | 中邮消费金融有限公司 | Credit order distribution method, computer equipment and readable storage medium thereof |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101391268A (en) * | 2008-11-07 | 2009-03-25 | 东北大学 | Reverse optimization method of steel plate rolling and cooling controlling-process temperature institution |
| CN101551846A (en) * | 2009-04-24 | 2009-10-07 | 华中科技大学 | Dynamic load equilibria method for radio frequency identifying middleware |
| CN102055644A (en) * | 2009-11-11 | 2011-05-11 | 中兴通讯股份有限公司 | Method, device and system for load management in distributed directory service system |
| CN104579996A (en) * | 2013-10-17 | 2015-04-29 | 中国电信股份有限公司 | Cluster load balancing method and system |
| CN106294775A (en) * | 2016-08-11 | 2017-01-04 | 合智能科技（深圳）有限公司 | Content delivery method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6477469B2 (en) * | 2001-01-08 | 2002-11-05 | Halliburton Energy Services, Inc. | Coarse-to-fine self-organizing map for automatic electrofacies ordering |
-
2018
- 2018-12-11 CN CN201811513533.4A patent/CN109597800B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101391268A (en) * | 2008-11-07 | 2009-03-25 | 东北大学 | Reverse optimization method of steel plate rolling and cooling controlling-process temperature institution |
| CN101551846A (en) * | 2009-04-24 | 2009-10-07 | 华中科技大学 | Dynamic load equilibria method for radio frequency identifying middleware |
| CN102055644A (en) * | 2009-11-11 | 2011-05-11 | 中兴通讯股份有限公司 | Method, device and system for load management in distributed directory service system |
| CN104579996A (en) * | 2013-10-17 | 2015-04-29 | 中国电信股份有限公司 | Cluster load balancing method and system |
| CN106294775A (en) * | 2016-08-11 | 2017-01-04 | 合智能科技（深圳）有限公司 | Content delivery method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109597800A (en) | 2019-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109597800B (en) | Log distribution method and device | |
| CN109104336B (en) | Service request processing method and device, computer equipment and storage medium | |
| CN108075934B (en) | Network quality monitoring method, device and system | |
| CN106815254B (en) | Data processing method and device | |
| CN109525500B (en) | Information processing method and information processing device capable of automatically adjusting threshold | |
| CN108492150B (en) | Method and system for determining entity heat degree | |
| CN109831508B (en) | Caching method and equipment and storage medium | |
| CN109560976B (en) | Message delay monitoring method and device | |
| CN111562884B (en) | Data storage method and device and electronic equipment | |
| CN105471938B (en) | Server load management method and device | |
| CN110222484B (en) | User identity recognition method and device, electronic equipment and storage medium | |
| US9225608B1 (en) | Evaluating configuration changes based on aggregate activity level | |
| CN107329832B (en) | Data receiving method and device | |
| CN105554113A (en) | Game processing method and system capable of accommodating a large number of players | |
| CN106021026B (en) | Backup method and device | |
| CN111338903B (en) | Alarm method and device for transaction abnormality | |
| CN106603631B (en) | Distributed message system and capacity expansion method | |
| US20170249232A1 (en) | Storage medium storing performance degradation cause estimation program, performance degradation cause estimating device, and performance degradation cause estimation method | |
| CN109857805B (en) | Block chain based data storage method, related equipment and storage medium | |
| CN108200185B (en) | Method and device for realizing load balance | |
| CN109462510B (en) | CDN node quality evaluation method and device | |
| CN113746920B (en) | Data forwarding method and device, electronic equipment and computer readable storage medium | |
| CN111885354B (en) | Service improvement discrimination method and device for bank outlets | |
| CN110554916B (en) | Distributed cluster-based risk index calculation method and device | |
| CN114513469A (en) | Traffic shaping method and device for distributed system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |