CN113810307A - Data flow control method, system and storage medium - Google Patents
Data flow control method, system and storage medium Download PDFInfo
- Publication number
- CN113810307A CN113810307A CN202111182206.7A CN202111182206A CN113810307A CN 113810307 A CN113810307 A CN 113810307A CN 202111182206 A CN202111182206 A CN 202111182206A CN 113810307 A CN113810307 A CN 113810307A
- Authority
- CN
- China
- Prior art keywords
- data
- binlog
- token
- binlog data
- token bucket
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000000670 limiting effect Effects 0.000 claims abstract description 70
- 238000012544 monitoring process Methods 0.000 claims abstract description 15
- 239000000284 extract Substances 0.000 claims abstract description 10
- 238000004364 calculation method Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 25
- 230000005540 biological transmission Effects 0.000 abstract description 9
- 230000000903 blocking effect Effects 0.000 abstract description 6
- 230000003139 buffering effect Effects 0.000 abstract description 6
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000001824 photoionisation detection Methods 0.000 description 25
- 238000004422 calculation algorithm Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000011143 downstream manufacturing Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 101100190462 Caenorhabditis elegans pid-1 gene Proteins 0.000 description 1
- 101100190464 Caenorhabditis elegans pid-2 gene Proteins 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/12—Avoiding congestion; Recovering from congestion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/215—Flow control; Congestion control using token-bucket
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a data flow control method, which comprises the following steps: the receiving module receives the Binlog data sent by the Kafka queue and sends the Binlog data to the current limiting module; the current limiting module sends Binlog data to a Kafka downstream device according to a preset flow, and sends the Binlog data exceeding the preset flow to a waiting queue; the monitoring module periodically extracts the Binlog data in the waiting queue to the current limiting module for retransmission. The invention realizes the buffering of the burst flow by prolonging the transmission path length and the waiting time of the burst flow in the Kakfa queue, effectively avoids the blocking of the burst flow to the Kafka queue, and further improves the overall data processing efficiency of the data processing system. The invention also provides a data flow control system and a storage medium, which have the beneficial effects.
Description
Technical Field
The present invention relates to the field of networks, and in particular, to a method, a system, and a storage medium for controlling data traffic.
Background
Kafka is a distributed publish-subscribe messaging system. In the related art, Kafka may send Binlog data (binary log) generated by a database to consumers located downstream of Kafka for consumption processing. However, the consumption rate of each consumer is limited, when a large amount of Binlog data is generated in a short time by the Kafka upstream database, the consumers cannot consume the Binlog data, and further, a Kafka consumption queue is blocked, so that the overall data processing efficiency of the data processing system is influenced.
Disclosure of Invention
The invention aims to provide a data flow control method, a data flow control system and a storage medium, which can realize the buffering of burst flow by means of prolonging the transmission path length and waiting time of the burst flow in a Kakfa queue, effectively avoid the blocking of the burst flow to the Kafka queue and further improve the overall data processing efficiency of a data processing system.
To solve the above technical problem, the present invention provides a data traffic control method, including:
the receiving module receives the Binlog data sent by the Kafka queue and sends the Binlog data to the current limiting module;
the current limiting module sends the Binlog data to a Kafka downstream device according to a preset flow, and sends the Binlog data exceeding the preset flow to a waiting queue;
and the monitoring module periodically extracts the Binlog data in the waiting queue to the current limiting module for retransmission.
Optionally, the receiving module receives Binlog data sent by a Kafka queue, including:
and the receiving module receives JSON data sent by the Kafka queue and analyzes the JSON data to obtain the Binlog data.
Optionally, the method for sending Binlog data to a Kafka downstream device at a preset flow rate and sending Binlog data exceeding the preset flow rate to a waiting queue includes:
the flow limiting module sends the Binlog data to a token bucket so that the token bucket distributes tokens for the Binlog data according to the preset flow;
sending the Binlog data with the token to the Kafka downstream device if the Binlog data received from the token bucket has the token;
and if the Binlog data received from the token bucket does not have the token, sending the Binlog data without the token to the waiting queue.
Optionally, the throttling module sends the Binlog data to a token bucket, including:
the current limiting module extracts PID from the Binlog data and judges whether the PID has a corresponding exclusive token bucket;
if so, sending the Binlog data to the exclusive token bucket;
and if not, performing hash calculation on the PID, determining a target token bucket in a preset number of public token buckets according to a calculation result, and sending the Binlog data to the target token bucket.
Optionally, the method further comprises:
and the current limiting module configures the number of the token buckets and the preset flow of each token bucket through Apollo.
The present invention also provides a data flow control system, including: a receiving module, a current limiting module and a monitoring module, wherein,
the receiving module is used for receiving the Binlog data sent by the Kafka queue and sending the Binlog data to the current limiting module;
the flow limiting module is used for sending the Binlog data to a Kafka downstream device according to a preset flow and sending the Binlog data exceeding the preset flow to a waiting queue;
and the monitoring module is used for periodically extracting the Binlog data in the waiting queue to the current limiting module for resending.
Alternatively,
the receiving module is further configured to receive JSON data sent by the Kafka queue, and analyze the JSON data to obtain the Binlog data.
Alternatively,
the flow limiting module is further configured to send the Binlog data to a token bucket, so that the token bucket distributes tokens to the Binlog data according to the preset flow; sending the Binlog data with the token to the Kafka downstream device if the Binlog data received from the token bucket has the token; and if the Binlog data received from the token bucket does not have the token, sending the Binlog data without the token to the waiting queue.
Optionally, the current limiting module is further configured to extract a PID from the Binlog data, and determine whether the PID has a corresponding exclusive token bucket; if so, sending the Binlog data to the exclusive token bucket; and if not, performing hash calculation on the PID, determining a target token bucket in a preset number of public token buckets according to a calculation result, and sending the Binlog data to the target token bucket.
The present invention also provides a storage medium having a computer program stored thereon, which when executed by a processor implements the steps of the data traffic control method as described above.
The invention provides a data flow control method, which comprises the following steps: the receiving module receives the Binlog data sent by the Kafka queue and sends the Binlog data to the current limiting module; the current limiting module sends the Binlog data to a Kafka downstream device according to a preset flow, and sends the Binlog data exceeding the preset flow to a waiting queue; and the monitoring module periodically extracts the Binlog data in the waiting queue to the current limiting module for retransmission.
Therefore, the current limiting module, the waiting queue and the monitoring module are additionally arranged in the Kafka queue, wherein the current limiting module can firstly send the Binlog data in the Kafka queue to a Kafka downstream device according to the preset flow, so that downstream consumers can consume the Binlog data at a constant speed, and the downstream processing efficiency is guaranteed; in addition, the current limiting module can also send Binlog data exceeding the preset flow in the Kakfa queue to the waiting queue for temporary storage, and the data in the queue is periodically re-extracted to the current limiting module by the monitoring module for processing, because the Binlog data exceeding the preset flow additionally passes through the waiting queue and is temporarily stored in the waiting queue for a period of time, in other words, the invention realizes the buffering of the burst flow by prolonging the transmission path length and the waiting time of the burst flow in the Kakfa queue, effectively avoids the blocking of the burst flow to the Kafka queue, and further improves the overall data processing efficiency of the data processing system. The invention also provides a data flow control system and a storage medium, which have the beneficial effects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a data flow control method according to an embodiment of the present invention;
fig. 2 is a block diagram of a data traffic control system according to an embodiment of the present invention;
fig. 3 is a block diagram of another data traffic control system according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Kafka is a distributed publish-subscribe messaging system. In the related art, Kafka may send Binlog data (binary log) generated by a database to consumers located downstream of Kafka for consumption processing. However, the consumption rate of each consumer is limited, when a large amount of Binlog data is generated in a short time by the Kafka upstream database, the consumers cannot consume the Binlog data, and further, a Kafka consumption queue is blocked, so that the overall data processing efficiency of the data processing system is influenced. In view of this, the present invention provides a data flow control method, which can implement buffering of burst flow by means of prolonging the transmission path length and waiting time of burst flow in the Kakfa queue, so as to effectively avoid blocking of the burst flow on the Kafka queue, thereby improving the overall data processing efficiency of the data processing system. Referring to fig. 1, fig. 1 is a flowchart of a data traffic control method according to an embodiment of the present invention, where the method includes:
s101, the receiving module receives the Binlog data sent by the Kafka queue and sends the Binlog data to the current limiting module.
The Binlog data is a binary log and is used for recording the change condition in the database. In Kafka, Binlog data is typically transmitted in JSON form, where JSON (JSON object notation) is a lightweight data exchange format. In order to limit the output flow of Binlog data and ensure that the Kakfa downstream equipment can process the Binlog data efficiently, in the embodiment of the present invention, firstly, JSON data transmitted by a Kafka queue needs to be parsed, and the Binlog data in the JSON data is extracted, so as to limit the output of the Binlog data according to the number of the Binlog data.
In one possible case, the receiving module receives the Binlog data sent by the Kafka queue, and may include:
step 11: and the receiving module receives the JSON data sent by the Kafka queue and analyzes the JSON data to obtain the Binlog data.
S102, the flow limiting module sends Binlog data to a Kafka downstream device according to preset flow, and sends the Binlog data exceeding the preset flow to a waiting queue.
The embodiment of the invention is additionally provided with a current limiting module which is used for limiting the flow of the Binlog data transmitted to Kafka downstream equipment, wherein the flow is the number of the Binlog data transmitted in unit time. The embodiment of the present invention does not limit the time unit on which the preset flow rate is based and the specific numerical value of the preset flow rate, for example, the time unit may be 10 pieces/second or 2 pieces/millisecond. Because the current limiting module can only send Binlog data to downstream equipment at preset flow, data accumulation occurs when burst flow occurs. For example, when the output flow rate of the current limiting module is 10/s, if the current limiting module receives 12 Binlog data within a certain second, 2 Binlog data which cannot be sent and exceed the preset flow rate will appear. For the part of Binlog data exceeding the preset flow, the embodiment of the invention outputs the part of Binlog data to a separate waiting queue for temporary storage, and inputs the part of Binlog data to the current limiting module again for sending processing after waiting for a period of time. In other words, the embodiment of the present invention performs buffering processing on the burst traffic by extending the length of the traffic transmission path and the waiting time, and meanwhile, since the burst traffic is buffered to a separate waiting queue and does not occupy the transmission path of the Kafka queue, blocking of the Kafka queue by the burst traffic can be effectively avoided. It should be noted that, the embodiment of the present invention does not limit the length of the wait queue, and can be set according to the actual application requirement.
It should be further noted that, the embodiment of the present invention does not limit the manner in which the current limiting module achieves the current limiting effect, for example, a leaky bucket algorithm (leakbucket) or a token bucket algorithm (TokenBucket) may be used, and both the algorithms may be used to limit the data transmission rate. Considering that the token bucket algorithm is more suitable for processing the burst traffic, the token bucket algorithm is adopted to realize the current limiting effect of the current limiting module in the embodiment of the invention. Specifically, the general principle of the token bucket nulling algorithm is as follows: the system will place tokens into the bucket at a constant rate and retrieve a token from the bucket when data needs to be serviced by the system. If the data can get tokens from the bucket, it can be processed by the system, otherwise it cannot. For embodiments of the present invention, the current limit module may transmit Binlog data with tokens to Kafka downstream devices, and send Binlog data without tokens to the wait queue.
In one possible case, the flow limiting module sending Binlog data to the Kafka downstream device at a preset flow rate and sending Binlog data exceeding the preset flow rate to the waiting queue may include:
step 21: the current limiting module sends the Binlog data to a token bucket so that the token bucket distributes tokens for the Binlog data according to preset flow;
step 22: if the Binlog data received from the token bucket has a token, sending the Binlog data with the token to a Kafka downstream device;
step 23: and if the Binlog data received from the token bucket does not have the token, sending the Binlog data without the token to the waiting queue.
It should be noted that, the embodiments of the present invention do not limit the specific forms of the token bucket and the tokens, nor the ways of allocating the tokens to the token bucket and injecting the tokens into the token bucket, and refer to the related technologies of the token bucket algorithm. Of course, in order to improve processing efficiency, a plurality of token buckets may also be used to process Binlog data, and the embodiment of the present invention does not limit the specific number of token buckets, and may be set according to actual application requirements.
Further, it will be appreciated that the database typically stores data for a plurality of users, and that each user's manipulation of the database results in corresponding Binlog data. The embodiment of the invention does not limit whether the Binlog data of each user can be classified, for example, a special user is processed by using a special token bucket, and a common user is processed by using a common token bucket. If the number of the Binlog data generated by each user is more average, classification processing is not needed; if there are some users that produce significantly higher amounts of Binlog data than other users, a classification process can be performed. In the embodiment of the invention, considering that in a production environment, the number of the Binlog data generated by each user is usually large, and the operation of part of users on the database is frequent, so that the number of the Binlog data generated by the users is obviously higher than that of other users, the Binlog data of each user is classified, the data transmission efficiency can be effectively improved, and the Binlog data of various users can be processed with different priorities. Further, in order to effectively distinguish Binlog data of each user, a user identifier, such as ID information, is usually added to the Binlog data. In one possible case, the Binlog data contains PID, which can be used to distinguish the merchants using the database system, and the current limiting module can process with different token buckets according to the PID information in the Binlog data. Specifically, a dedicated token bucket corresponding to a special PID can be set, and when the current limiting module determines that the PID has the corresponding dedicated token bucket, the Binlog data can be input into the dedicated token bucket for processing; similarly, a plurality of public token buckets can be set, and after determining that the PID of the Binlog data does not have a dedicated token bucket, the current limiting module calculates the PID by using a hash algorithm, determines the public token bucket corresponding to the PID, and sends the Binlog data to the corresponding public token bucket for processing. The embodiment of the invention does not limit the specific form of PID, and can be set according to the actual application requirement; the embodiment of the invention also does not limit the number of the exclusive token buckets and the public token buckets, and can be set according to the actual application requirements; the embodiment of the invention also does not limit how to distribute the Binlog data to the target token buckets in the plurality of public token buckets by utilizing the Hash algorithm and PID, and can refer to the related technology of Hash load balancing.
In one possible scenario, the throttling module sending Binlog data to the token bucket may include:
step 31: the current limiting module extracts PID from the Binlog data and judges whether the PID has a corresponding exclusive token bucket; if yes, go to step 32; if not, go to step 33;
step 32: sending the Binlog data to a special token bucket;
step 33: and performing hash calculation on the PID, determining a target token bucket in a preset number of public token buckets according to a calculation result, and sending the Binlog data to the target token bucket.
Further, it should be noted that the embodiment of the present invention does not limit the preset flow of each token bucket, and the preset flows between the token buckets may be different and may be set according to actual application requirements. Furthermore, in order to improve the preset flow and the setting efficiency of the token buckets, the current limiting module can also dynamically configure the number of the token buckets and the preset flow of each token bucket through Apollo, so that the number of the token buckets is ensured to be sufficient, and the proper preset flow can be set according to the load conditions of different token buckets, wherein Apollo is an open-source configuration management center, so that the configuration of different environments and different clusters can be managed in a centralized manner, and the configuration can be pushed to an application side in real time after the configuration is modified, so that the configuration is effective.
In one possible case, the method may further include:
step 41: the current limit module configures the number of token buckets and the preset flow of each token bucket through Apollo.
S103, the monitoring module periodically extracts the Binlog data in the waiting queue to the current limiting module for resending.
After the data is temporarily stored in the waiting queue, a monitoring thread can be additionally arranged, and Binlog data in the waiting queue is periodically input into the current limiting module again, so that the temporarily stored data can be ensured to be continuously sent to the Kafka downstream device. It should be noted that, the embodiment of the present invention does not limit the specific period for extracting the Binlog data, and may be set according to the actual application requirements.
Based on the embodiment, the current limiting module, the waiting queue and the monitoring module are additionally arranged in the Kafka queue, wherein the current limiting module can firstly send the Binlog data in the Kafka queue to a Kafka downstream device according to the preset flow, so that downstream consumers can consume the Binlog data at a constant speed, and the downstream processing efficiency is guaranteed; in addition, the current limiting module can also send Binlog data exceeding the preset flow in the Kakfa queue to the waiting queue for temporary storage, and the data in the queue is periodically re-extracted to the current limiting module by the monitoring module for processing, because the Binlog data exceeding the preset flow additionally passes through the waiting queue and is temporarily stored in the waiting queue for a period of time, in other words, the invention realizes the buffering of the burst flow by prolonging the transmission path length and the waiting time of the burst flow in the Kakfa queue, effectively avoids the blocking of the burst flow to the Kafka queue, and further improves the overall data processing efficiency of the data processing system.
The data traffic control method is described below based on a specific structural block diagram. Referring to fig. 2, fig. 2 is a block diagram of a data traffic control system according to an embodiment of the present invention. The control of data traffic can be divided into two parts:
(one) limiting sudden large flows
1. Assuming that the received Kafka message is message-a, the current limit module sets the preset flow rate to be 10/s, and the wait queue (slow-Kafka) stores data outside the limited flow rate.
2. And analyzing the received message-a, extracting Binlog data (performing anti-json execution), sending the Binlog data to the current limiting module, and if the data amount in the current limiting module is less than 10, sending the Binlog data to the downstream through Kafka.
3. If the data quantity in the current limiting module is larger than 10, the data quantity is temporarily stored in the waiting queue.
4. The slow-queue consumption listening thread (i.e., the listening module) will periodically fetch the data in the waiting queue, and the data is analyzed again through the process of step 2.
5. The rate at which slow-listen pulls the slow-kafka can be dynamically controlled by Apollo configuration.
(II) isolation by flow type
1. Assume that token buckets are partitioned according to merchant PIDs, with 2 exclusive token buckets (PID 1 for token bucket P1 and PID 2 for token bucket P2), and N common token buckets (token bucket L1-LN).
2. In step (one), the parsed Binlog message extracts the PID thereof, and if the PID thereof is 1 or 2, the message enters a dedicated token bucket to acquire tokens (a certain amount of tokens are issued per second in the token bucket).
3. If the PID of the public token bucket does not match with the exclusive token bucket, the public token bucket corresponding to the Binlog information is determined through hash allocation (hash allocation), for example, L3 is obtained, and tokens are obtained from the public token bucket L3.
4. If a token can be acquired from the token bucket, it is transmitted downstream via Kafka.
5. If the token cannot be acquired from the token bucket, the token is temporarily stored in the wait queue.
6. The number of the special token buckets, the matching identifiers of the special token buckets and the token bucket issuing rate can be dynamically controlled through Apollo configuration.
7. The number of generic token buckets, and the rate at which the token buckets issue tokens, can be dynamically controlled through Apollo configuration.
In the following, a data traffic control system and a storage medium according to embodiments of the present invention are introduced, and the data traffic control system and the storage medium described below and the data traffic control method described above may be referred to correspondingly.
Referring to fig. 3, fig. 3 is a block diagram of another data traffic control system according to an embodiment of the present invention, where the system may include: a receiving module 301, a current limiting module 302, and a listening module 303, wherein,
the receiving module 301 is configured to receive Binlog data sent by the Kafka queue, and send the Binlog data to the current limiting module;
the flow limiting module 302 is used for sending Binlog data to a Kafka downstream device according to a preset flow and sending the Binlog data exceeding the preset flow to a waiting queue;
and the monitoring module 303 is configured to periodically extract the Binlog data in the waiting queue to the current limiting module for retransmission.
Alternatively,
the receiving module 301 is further configured to receive JSON data sent by the Kafka queue, and parse the JSON data to obtain Binlog data.
Alternatively,
the current limiting module 302 is further configured to send the Binlog data to a token bucket, so that the token bucket distributes tokens for the Binlog data according to a preset flow; if the Binlog data received from the token bucket has a token, sending the Binlog data with the token to a Kafka downstream device; and if the Binlog data received from the token bucket does not have the token, sending the Binlog data without the token to the waiting queue.
Alternatively,
the current limiting module 302 is further configured to extract a PID from the Binlog data, and determine whether the PID has a corresponding exclusive token bucket; if so, sending the Binlog data to a dedicated token bucket; and if not, performing hash calculation on the PID, determining a target token bucket in a preset number of public token buckets according to the calculation result, and sending the Binlog data to the target token bucket.
Alternatively,
the throttling module 302 is further configured to configure the number of token buckets and the preset flow rate of each token bucket through Apollo.
The embodiment of the present invention further provides a storage medium, where a computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the data flow control method in any of the above embodiments are implemented.
Since the embodiment of the storage medium portion corresponds to the embodiment of the data flow control method portion, please refer to the description of the embodiment of the data flow control method portion for the embodiment of the storage medium portion, which is not repeated here.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The above description details a data flow control method, system and storage medium provided by the present invention. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (10)
1. A method for controlling data traffic, comprising:
the receiving module receives the Binlog data sent by the Kafka queue and sends the Binlog data to the current limiting module;
the current limiting module sends the Binlog data to a Kafka downstream device according to a preset flow, and sends the Binlog data exceeding the preset flow to a waiting queue;
and the monitoring module periodically extracts the Binlog data in the waiting queue to the current limiting module for retransmission.
2. The method of claim 1, wherein the receiving module receives Binlog data sent by a Kafka queue, and the method comprises:
and the receiving module receives JSON data sent by the Kafka queue and analyzes the JSON data to obtain the Binlog data.
3. The data flow control method according to claim 1 or 2, wherein the flow limiting module sends the Binlog data to a Kafka downstream device at a preset flow rate, and sends the Binlog data exceeding the preset flow rate to a waiting queue, and the method comprises:
the flow limiting module sends the Binlog data to a token bucket so that the token bucket distributes tokens for the Binlog data according to the preset flow;
sending the Binlog data with the token to the Kafka downstream device if the Binlog data received from the token bucket has the token;
and if the Binlog data received from the token bucket does not have the token, sending the Binlog data without the token to the waiting queue.
4. The method of claim 3, wherein the throttling module sends the Binlog data to a token bucket, comprising:
the current limiting module extracts PID from the Binlog data and judges whether the PID has a corresponding exclusive token bucket;
if so, sending the Binlog data to the exclusive token bucket;
and if not, performing hash calculation on the PID, determining a target token bucket in a preset number of public token buckets according to a calculation result, and sending the Binlog data to the target token bucket.
5. The data traffic control method according to claim 3, further comprising:
and the current limiting module configures the number of the token buckets and the preset flow of each token bucket through Apollo.
6. A data flow control system, comprising: a receiving module, a current limiting module and a monitoring module, wherein,
the receiving module is used for receiving the Binlog data sent by the Kafka queue and sending the Binlog data to the current limiting module;
the flow limiting module is used for sending the Binlog data to a Kafka downstream device according to a preset flow and sending the Binlog data exceeding the preset flow to a waiting queue;
and the monitoring module is used for periodically extracting the Binlog data in the waiting queue to the current limiting module for resending.
7. The data flow control system of claim 6,
the receiving module is further configured to receive JSON data sent by the Kafka queue, and analyze the JSON data to obtain the Binlog data.
8. The data flow control system of claim 6 or 7,
the flow limiting module is further configured to send the Binlog data to a token bucket, so that the token bucket distributes tokens to the Binlog data according to the preset flow; sending the Binlog data with the token to the Kafka downstream device if the Binlog data received from the token bucket has the token; and if the Binlog data received from the token bucket does not have the token, sending the Binlog data without the token to the waiting queue.
9. The data flow control system of claim 8,
the current limiting module is also used for extracting a PID from the Binlog data and judging whether the PID has a corresponding exclusive token bucket; if so, sending the Binlog data to the exclusive token bucket; and if not, performing hash calculation on the PID, determining a target token bucket in a preset number of public token buckets according to a calculation result, and sending the Binlog data to the target token bucket.
10. A storage medium having stored thereon a computer program for implementing the steps of the data flow control method according to any one of claims 1 to 5 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111182206.7A CN113810307A (en) | 2021-10-11 | 2021-10-11 | Data flow control method, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111182206.7A CN113810307A (en) | 2021-10-11 | 2021-10-11 | Data flow control method, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113810307A true CN113810307A (en) | 2021-12-17 |
Family
ID=78939385
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111182206.7A Withdrawn CN113810307A (en) | 2021-10-11 | 2021-10-11 | Data flow control method, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113810307A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115883472A (en) * | 2022-11-29 | 2023-03-31 | 重庆紫光华山智安科技有限公司 | Token issuing method and related device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7095753B1 (en) * | 2000-09-19 | 2006-08-22 | Bbn Technologies Corp. | Digital network processor-based multi-protocol flow control |
| CN109194586A (en) * | 2018-08-31 | 2019-01-11 | 重庆富民银行股份有限公司 | Peak clipping processing method based on distributed token bucket |
| CN109286573A (en) * | 2018-08-31 | 2019-01-29 | 重庆富民银行股份有限公司 | Peak clipping system based on distributed token bucket |
| CN110932994A (en) * | 2019-12-31 | 2020-03-27 | 深圳云天励飞技术有限公司 | Data traffic processing method and device, electronic equipment and storage medium |
| CN111045810A (en) * | 2019-12-17 | 2020-04-21 | 浙江大华技术股份有限公司 | Task scheduling processing method and device |
| CN111314476A (en) * | 2020-02-24 | 2020-06-19 | 苏宁云计算有限公司 | Message transmission method and system for enterprise asynchronization |
| WO2020142867A1 (en) * | 2019-01-07 | 2020-07-16 | 华为技术有限公司 | Traffic shaping method and related device |
| CN111639902A (en) * | 2020-04-29 | 2020-09-08 | 深圳壹账通智能科技有限公司 | Data auditing method based on kafka, control device, computer equipment and storage medium |
-
2021
- 2021-10-11 CN CN202111182206.7A patent/CN113810307A/en not_active Withdrawn
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7095753B1 (en) * | 2000-09-19 | 2006-08-22 | Bbn Technologies Corp. | Digital network processor-based multi-protocol flow control |
| CN109194586A (en) * | 2018-08-31 | 2019-01-11 | 重庆富民银行股份有限公司 | Peak clipping processing method based on distributed token bucket |
| CN109286573A (en) * | 2018-08-31 | 2019-01-29 | 重庆富民银行股份有限公司 | Peak clipping system based on distributed token bucket |
| WO2020142867A1 (en) * | 2019-01-07 | 2020-07-16 | 华为技术有限公司 | Traffic shaping method and related device |
| CN111045810A (en) * | 2019-12-17 | 2020-04-21 | 浙江大华技术股份有限公司 | Task scheduling processing method and device |
| CN110932994A (en) * | 2019-12-31 | 2020-03-27 | 深圳云天励飞技术有限公司 | Data traffic processing method and device, electronic equipment and storage medium |
| CN111314476A (en) * | 2020-02-24 | 2020-06-19 | 苏宁云计算有限公司 | Message transmission method and system for enterprise asynchronization |
| CN111639902A (en) * | 2020-04-29 | 2020-09-08 | 深圳壹账通智能科技有限公司 | Data auditing method based on kafka, control device, computer equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115883472A (en) * | 2022-11-29 | 2023-03-31 | 重庆紫光华山智安科技有限公司 | Token issuing method and related device |
| CN115883472B (en) * | 2022-11-29 | 2024-05-03 | 重庆紫光华山智安科技有限公司 | Token issuing method and related device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107592345B (en) | Transaction current limiting device, method and transaction system | |
| US7818313B1 (en) | Method for distributing processing of queries over a cluster of servers in a continuous processing system | |
| Eryilmaz et al. | Asymptotically tight steady-state queue length bounds implied by drift conditions | |
| US20190012218A1 (en) | Event handling in distributed event handling systems | |
| US20210224907A1 (en) | System and method for adaptive information dissemination | |
| CN109218355A (en) | Load equalizing engine, client, distributed computing system and load-balancing method | |
| Liu et al. | D-storm: Dynamic resource-efficient scheduling of stream processing applications | |
| US8150994B2 (en) | Providing flow control and moderation in a distributed message processing system | |
| CN104283643A (en) | Message speed limiting method and device | |
| CN113810307A (en) | Data flow control method, system and storage medium | |
| CN115442373A (en) | Data processing method, device, equipment and storage medium | |
| Salah et al. | Improving snort performance under linux | |
| CN113890842B (en) | Information transmission delay upper bound calculation method, system, equipment and storage medium | |
| US8180823B2 (en) | Method of routing messages to multiple consumers | |
| CN112073327B (en) | Anti-congestion software distribution method, device and storage medium | |
| CN116302863A (en) | Alarm processing method, device, equipment and computer readable medium | |
| CN110781950A (en) | Message processing method and device | |
| CN115168030A (en) | Dynamic regulation and control log acquisition and processing method and device and storage medium | |
| Tsenos et al. | Dynamic rate control in the kafka system | |
| CN110809062B (en) | Public cloud voice recognition resource calling control method and device | |
| CN109508433B (en) | Load fluctuation coping method and system based on performance adjustment of matching algorithm | |
| CN111858019B (en) | Task scheduling method and device and computer readable storage medium | |
| CN113391927A (en) | Method, device and system for processing business event and storage medium | |
| CN104917692A (en) | Method and device for distributing tokens | |
| Wei et al. | Research on Task-Offloading Delay in the IoV Based on a Queuing Network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20211217 |