EP4573726A1

EP4573726A1 - Methods and systems for request traffic management

Info

Publication number: EP4573726A1
Application number: EP23862181.7A
Authority: EP
Inventors: Yantao Zheng; Wenkai ZHOU; Liang Chen; Xin Jin; Yongchao LIU; Ning Huang; Xing Yu
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2022-09-06
Filing date: 2023-08-22
Publication date: 2025-06-25
Also published as: EP4573726A4; CN115665045A; CN115665045B; WO2024051485A1

Abstract

The present disclosure provides a method and a system for request traffic management. The method may include for any cycle, obtaining a pending request; in response to a count of tokens in a target token set corresponding to the cycle being greater than 0, taking a target token from the target token set and issuing the target token to the pending request to facilitate a service system to process the pending request; and in response to the pending request being processed within the cycle, reclaiming one or more tokens held by the pending request and adding the one or more tokens to another token set corresponding to the cycle; wherein a total count of tokens in the cycle is a preset value.

Description

METHODS AND SYSTEMS FOR REQUEST TRAFFIC MANAGEMENT

CROSS REFERENCE TO RELATED APPLICATIONS
This application claims priority of the Chinese Patent Application No. 202211081754.5, filed on September 6, 2022, the entire contents of which are hereby incorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to the field of computer technology, and in particular to methods and systems for request traffic management.

BACKGROUND

At present, network-based service systems (such as database systems that provide data query services, shopping platform systems that provide shopping services, social platform systems that provide online social services, etc. ) can provide various services, and users or various devices can initiate requests to the service systems, and the service systems can process the received requests accordingly to provide the corresponding service. Usually, the count of requests that the device where the service system is located (such as a server) can withstand is limited. If a high concurrency or large traffic of requests (alarge count of requests) occurs, and exceeds the tolerance range of the device where the service system is located, the stability of the service system may be affected, and system failure may occur. Accordingly, it is desirable to manage the request traffic accessing the service system (also called traffic management, traffic limiting) , to ensure the stability of the service system.
Therefore, how to reasonably manage the request traffic of the service system and ensure the stable operation of the service system is an urgent technical problem in this field.

SUMMARY

One of the embodiments of the present disclosure provides a method for request traffic management. The method may be applied to one or more processing devices, comprising: for any cycle, obtaining a pending request; in response to a count of tokens in a target token set corresponding to the cycle being greater than 0, taking a target token from the target token set and issuing the target token to the pending request, to facilitate the one or more processing devices to process the pending request; and in response to the pending request being processed within the cycle, reclaiming one or more tokens held by the pending request and adding the one or more tokens to another token set corresponding to the cycle; wherein a total count of tokens in the cycle may be a preset value, and the total count of tokens may be a sum of the one or more tokens held by the pending request, the count of tokens in the target token set and a count of tokens in the another token set.
One of the embodiments of the present disclosure provides a system for request traffic management. The system may comprise at least one computer-readable storage medium storing a set of instructions; and at least one processing device in communication with the computer-readable storage medium, wherein, when executing the set of instructions, the at least one processing device may be configured to cause the system to perform operations including: for any cycle, obtaining a pending request; in response to a count of tokens in a target token set corresponding to the cycle being greater than 0, taking a target token from the target token set and issuing the target token to the pending request, to facilitate one or more processing devices to process the pending request; and in response to the pending request being processed within the cycle, reclaiming one or more tokens held by the pending request and adding the one or more tokens to another token set corresponding to the cycle; wherein a total count of tokens in the cycle may be a preset value, and the total count of tokens may be a sum of the one or more tokens held by the pending request, the count of tokens in the target token set, and a count of tokens in the another token set.
One of the embodiments of the present disclosure provides a computer-readable storage medium storing computer instructions. The instructions, when read by a computer, causing the computer to implement a method comprising: for any cycle: obtaining a pending request; in response to a count of tokens in a target token set corresponding to the cycle being greater than 0, taking a target token from the target token set and issuing the target token to the pending request, to facilitate one or more processing devices to process the pending request; and in response to the pending request being processed within the cycle, reclaiming one or more tokens held by the pending request and adding the one or more tokens to another token set corresponding to the cycle; wherein a total count of tokens in the cycle may be a preset value, and the total count of tokens may be a sum of the one or more tokens held by the pending request, the count of tokens in the target token set, and a count of tokens in the another token set.
One of the embodiments of the present disclosure provides a method for high concurrency traffic limiting, applied to a client, comprising: in response to triggering of a token application event, obtaining a token from a token issuing bucket of a first traffic limiting cycle, a value of a token issuing cycle being updated to a first traffic limiting cycle; executing a business request of the token application event based on the token, and uploading the generated business data to a server; detecting whether the current traffic limiting cycle within which the business request corresponding to the token is completed is the first traffic limiting cycle; and if yes, putting the token back into a token reclaiming bucket of the first traffic limiting cycle, a total count of tokens in each traffic limiting cycle being a fixed value.
After the generated business data is uploaded to the server, the method for high concurrency traffic limiting may further include: if the current traffic limiting cycle within which the business request corresponding to the token is completed is a second traffic limiting cycle, putting the token into a token issuing bucket of the second traffic limiting cycle. The second traffic limiting cycle and the first traffic limiting cycle may be used to distinguish different traffic limiting periods without limiting the order of the first traffic limiting period and the second traffic limiting period. In some embodiments, the second traffic limiting cycle and the first traffic limiting cycle may be adjacent traffic limiting cycles or the second traffic limiting cycle and the first traffic limiting cycle may be non-adjacent traffic limiting cycles (e.g., the second traffic limiting cycle follows the first traffic limiting cycle and is separated from the first cycle by 2 cycles. ) .
After the generated business data is uploaded to the server, the method for high concurrency traffic limiting method may further include: if the current traffic limiting cycle within which the business request corresponding to the token is completed is the second traffic limiting cycle, obtaining a business request queue; detecting whether there is a valid historical request in the business request queue; and if yes, obtaining a first valid request in the business request queue, updating the issuing cycle value of the token to the second traffic limiting cycle, and executing the first valid request based on the token.
After the business request queue is obtained, the method for high concurrency traffic limiting may further include: if there is no valid historical request in the business request queue, putting the token back into the token issuing bucket of the second traffic limiting cycle.
The second traffic limiting cycle and the first traffic limiting cycle may be adjacent traffic limiting cycles. The method for high concurrency traffic limiting may further include: if the current traffic limiting cycle within which the business request corresponding to the token is completed is the second traffic limiting cycle, putting the token in the token issuing bucket of the first traffic limiting cycle into the token reclaiming bucket of the second traffic limiting cycle, and putting the token in the token reclaiming bucket of the first traffic limiting cycle into the token issuing bucket of the second traffic limiting cycle. In some embodiments, putting a token into a token bucket may be replaced by changing the role of the token to a token of the token bucket, so that it is not necessary to move the token.
A cycle iteration between the first traffic limiting cycle and the adjacent second traffic limiting cycle may be triggered by a pre-registered timer.
The method for high concurrency traffic limiting may further include: in response to the triggering of a token application event, if the token issuing bucket of the first traffic limiting cycle cannot obtain the token, putting the business request of the token application event into a fixed-length business request queue.
One of the embodiments of the present disclosure further provides a device for high concurrency traffic limiting, comprising an obtaining module, an execution module, and a reclaiming module. The obtaining module may be configured to obtain a token from a token issuing bucket of a first traffic limiting cycle in response to the triggering of a token application event, a value of a token issuing cycle being updated to the first traffic limiting cycle; the execution module may be configured to execute a business request of the token application event based on the token, and upload the generated business data to a server; and the reclaiming module may be configured to detect whether the current traffic limiting cycle within which the business request corresponding to the token is completed is the first traffic limiting cycle; if yes, put the token back into a token reclaiming bucket of the first traffic limiting cycle, a total count of tokens in each traffic limiting cycle being a fixed value.
One of the embodiments of the present disclosure further provides a device for high concurrency traffic limiting, comprising a processor and a memory coupled to the processor. The memory may be configured to store program data. The processor may be configured to execute the program data to implement the above-mentioned method for high concurrency traffic limiting.
One of the embodiments of the present disclosure further provides a computer storage medium storing program data, when executed by a processor, causing the processor to implement the above-mentioned method for high concurrency traffic limiting

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will be further illustrated by way of exemplary embodiments, which will be described in detail with the accompanying drawings. These embodiments are non-limiting. In these embodiments, the same count indicates the same structure, and wherein:
FIG. 1 is a schematic diagram illustrating an application scenario of a system for request traffic management according to some embodiments of the present disclosure;
FIG. 2 is a block diagram illustrating a system for request traffic management according to some embodiments of the present disclosure;
FIG. 3 is an exemplary flowchart illustrating a process for request traffic management according to some embodiments of the present disclosure;
FIG. 4 is an exemplary flowchart illustrating a limiting processing according to some embodiments of the present disclosure;
FIG. 5 is a schematic diagram illustrating an hourglass model of a cycle n according to some embodiments of the present disclosure;
FIG. 6 is a schematic diagram illustrating an hourglass model of a cycle n+1 according to some embodiments of the present disclosure;
FIG. 7 is a schematic flowchart illustrating an exemplary process of high concurrency traffic limiting according to some embodiments of the present disclosure;
FIG. 8 is a schematic flowchart illustrating a process of obtaining and reclaiming of an hourglass model token according to some embodiments of the present disclosure;
FIG. 9 is a schematic flowchart illustrating another exemplary process of high concurrency traffic limiting according to some embodiments of the present disclosure;
FIG. 10 is a schematic structural diagram illustrating an exemplary device for high concurrency traffic limiting according to some embodiments of the present disclosure;
FIG. 11 is a schematic structural diagram illustrating another exemplary device for high concurrency traffic limiting according to some embodiments of the present disclosure;
FIG. 12 is a schematic structural diagram illustrating an exemplary computer storage medium according to some embodiments of the present disclosure;
FIG. 13 is a schematic diagram illustrating a three-color rolling model of a cycle n according to the present disclosure;
FIG. 14 is a schematic diagram illustrating a three-color rolling model of a cycle n+1 according to the present disclosure; and
FIG. 15 is a schematic diagram illustrating a three-color rolling model of a cycle n+2 according to the present disclosure.

DETAILED DESCRIPTION

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings that need to be used in the description of the embodiments. Apparently, the accompanying drawings in the following description are only some examples or embodiments of the present disclosure, and those skilled in the art can also apply the present disclosure to other similar scenarios according to the drawings without creative efforts. Unless obviously obtained from the context or the context illustrates otherwise, the same numeral in the drawings refers to the same structure or operation.
It should be understood that "system" , "device" , "unit" and/or "module" as used herein is a method for distinguishing different components, elements, parts, portions or assemblies of different levels. However, the words may be replaced by other expressions if other words can achieve the same purpose.
As indicated in the disclosure and claims, the terms "a" , "an" , "an" and/or "the" are not specific to the singular form and may include the plural form unless the context clearly indicates an exception. Generally speaking, the terms "comprising" and "including" only suggest the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list, and the method or device may also contain other steps or elements.
The flowchart is used in the present disclosure to illustrate the operations performed by the system according to the embodiments of the present disclosure. It should be understood that the preceding or following operations are not necessarily performed in the exact order. Instead, various steps may be processed in reverse order or simultaneously. Meanwhile, other operations may be added to these procedures, or a certain step or steps may be removed from these procedures.
In some embodiments, traffic limiting of a service system may include estimating a count of the requests for the overall service according to an equivalent response time of a single service at the beginning of the system design, thereby estimating the count of the requests that the service system may withstand, and accordingly setting a request traffic limit for the service system. For example, a token bucket may be used to implement traffic limiting (e.g., a certain number of tokens may be stored in the token bucket, and the processing device may issue token (s) in the token bucket to request (s) for accessing the service system, and the service system may process the request (s) that hold the token (s)) .
However, if the traffic limiting is implemented simply by issuing tokens, the traffic limiting may still be without a closed loop of traffic relationship. Only the inflow of requests may be considered but the outflow of requests may be not considered, resulting in poor effect and inaccuracy of traffic limiting.
Some embodiments of the present disclosure provide a method for request traffic management. By issuing token (s) to request (s) and reclaiming token (s) from request (s) , and controlling the total count of tokens in each traffic limiting cycle to a fixed value, management, and control of the access traffic to the service system may be implemented. The method for request traffic management provided by some embodiments of the present disclosure may form a closed loop of token traffic relationship, and consider the inflow and outflow of token request traffic while controlling traffic (i.e., receiving the request for processing and accomplishing the request processing, wherein the end of request processing may be referred to as the completion of the request) . The method may achieve a better effect of traffic limiting and precise traffic limiting.
FIG. 1 is a schematic diagram illustrating an application scenario of a system for request traffic management according to some embodiments of the present disclosure. A scenario 100 may relate to various scenarios related to request traffic management. For example, the scenario 100 may include traffic management scenarios of online service systems such as a shopping platform, an online reply platform, a social network platform, etc.
As shown in FIG. 1, the scenario 100 may include a terminal 110, a processing device 120, and a network 130. The terminal 110 and the processing device 120 may be in communication connection through the network 130.
The terminal 110 may refer to a processing device for accessing the online service system and initiating a request to the online service system, which may be used by a user or set by the user to run automatically to implement the required functions. The terminal 110 may access the online service system, for example, initiate a request to the online service system (such as a data access request, a data storage request, a data processing request, etc. ) . In some embodiments, the terminal 110 may be used by one or more users, including users who directly use the service, or other related users. In some embodiments, the terminal 110 may be one of a mobile device 110-1, a tablet computer 110-2, a laptop computer 110-3, a desktop computer 110-4, and other devices with input and/or output functions, or any combination thereof.
In some embodiments, the mobile device 110-1 may include a wearable device, a smart mobile device, or the like, or any combination thereof.
In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant (PDA) , a gaming device, a navigation device, a handheld terminal (POS) , or the like, or any combination thereof.
In some embodiments, the desktop computer 110-4 may be a vehicle computer, a vehicle television, or the like.
For example, the mobile device 110-1 may include other devices with the input and/or output functions (e.g., a dedicated question and answer terminal set in a public place) .
The foregoing examples may be only used to illustrate the wide range of the terminal 110 rather than limit the scope thereof.
The processing device 120 may be configured to manage request traffic of the service system (also referred to as the online service system) . When the terminal 110 accesses the service system, the request for the service system sent by the terminal 110 may be received by the processing device 120. And the processing device 120 may manage the received request, to control whether the request is processed by the service system, thereby implementing the management and control of the request traffic of the service system. One or more functions of the method for request traffic management proposed in some embodiments of the present disclosure may be stored in a storage medium in the form of instructions. The processing device 120 may read the instructions stored in the storage medium, and when the instructions are read and executed by the processing device 120, one or more functions of the method for request traffic management proposed in some embodiments of the present disclosure may be implemented.
In some embodiments, the processing device 120 may be local or remote. In some embodiments, the processing device 120 may be implemented on a cloud platform, or provided in a virtual manner. Merely by way of example, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, or the like, or any combination thereof.
The processing device 120 may be a single processing device or a device cluster (e.g., the processing device 120 may be a distributed system) composed of multiple processing devices. One processing device may include one or more sub-processing devices. In some embodiments, one processing device may include various types of processing devices, systems, platforms, or the like. or any combination thereof. The processing device 120 may obtain data of other devices (such as the terminal 110, etc. ) , and may perform calculation and processing on the data. As an example only, the processor device 120 may include a central processing unit (CPU) , an application specific integrated circuit (ASIC) , an application specific instruction processor (ASIP) , a graphics processing unit (GPU) , a physical processing unit (PPU) , a digital signal processor (DSP) , a field programmable gate array (FPGA) , a programmable logic circuit (PLD) , a controller, a microcontroller unit, a reduced instruction set computer (RISC) , a microprocessor, or the like, or any combination thereof.
The network 130 may connect various components of the system and/or connect the system to an external resource part. The network 130 may enable communication between various components and with other components outside the system, facilitating the exchange of data and/or information. In some embodiments, the network 130 may be any one or more of a wired network or a wireless network. For example, the network 130 may include a cable network, a fiber optic network, a telecommunications network, the Internet, a local area network (LAN) , a wide area network (WAN) , a wireless local area network (WLAN) , a metropolitan area network (MAN) , a public switched telephone network (PSTN) , a Bluetooth network, a ZigBee network (ZigBee) , near field communication (NFC) , an in-device bus, an in-device line, cable connection, or the like, or any combination thereof. The network connection among the various parts may adopt one of the above-mentioned methods, or may adopt multiple methods. In some embodiments, the network may be in various topologies such as a point-to-point topology, a shared topology, and a central topology, or a combination of multiple topologies. In some embodiments, the network 130 may include one or more network access points. For example, the network 130 may include wired or wireless network access points, such as a base station and/or a network switching point 130-1, 130-2, ..., through which one or more components of scenario 100 may be connected to the network 130 to exchange data and/or information.
In some embodiments, the service system may be deployed on the processing device 120 or other processing devices. When the service system is deployed on other processing device (s) , the processing device 120 may interact with other processing device (s) , so that the service system (s) on other processing device (s) may receive request (s) for token (s) issued via the processing device 120 and process the request (s) .
It should be noted that the scenario 100 in FIG. 1 may be a stand-alone traffic limiting scenario, i.e., the processing device 120 may manage the request traffic of the stand-alone service system based on the method for request traffic management provided in the present disclosure, wherein the stand-alone service system may include a single processing device (e.g., a server) . In some embodiments, the scenario 100 may also be a request traffic management scenario of a distributed service system. The distributed service system may include multiple processing devices (e.g., multiple servers) , and the multiple processing devices may jointly provide services. In such cases, the processing device 120 may manage and allocate requests to be processed by each processing device based on the method for request traffic management provided in the present disclosure. In some embodiments, loads of multiple processing devices in the distributed service system may be in a balanced state.
In some embodiments, the processing device 120 may include multiple processing devices, and the multiple processing devices may implement the method for request traffic management and the distributed service system simultaneously. For example, one of the multiple processing devices may receive a request from the terminal 110 for the distributed service system, and implement one or more functions of the method for request flow management, and multiple processing devices may serve as a distributed service system to provide services (i.e., requests to the distributed service system may be assigned to each of multiple processing devices for processing to provide services) .
In some embodiments, the terminal 110 may also serve as the processing device 120 to implement one or more functions of the method for request traffic management. For example, the service system may be deployed on a certain server, the terminal 110 may initiate a request to the service system, and the terminal 110 may also implement one or more functions of the method for request traffic management to implement the management and control of the request traffic of the service system.
The above information transfer relationship between the components of the scenario 100 is only an example, and the present disclosure is not limited thereto.
It should be noted that the application scenarios are provided for illustrative purposes only, and are not intended to limit the scope of the present disclosure. For those skilled in the art, various modifications or changes may be made according to the description of the present disclosure. For example, the scenario 100 may also include a storage device. As another example, the scenario 100 may be implemented on other devices to implement similar or different functions. However, changes and modifications will not depart from the scope of the present disclosure.
FIG. 2 is a block diagram illustrating a system for request traffic management according to some embodiments of the present disclosure. The system 200 for request traffic management may be implemented on the processing device 120.
As shown in FIG. 2, the system 200 for request traffic management may include an obtaining module 210, a distribution module 220, and a reclaiming module 230.
The obtaining module 210 may be configured to obtain a pending request.
The distribution module 220 may be configured to take a token from a target token set and issue the token to the pending request in response to a count of tokens in a target token set corresponding to a cycle greater than 0, to facilitate one or more processing devices to process the pending request.
The reclaiming module 230 may be configured to reclaim the token held by the pending request and add the token to another token set corresponding to the cycle in response to the pending request being processed within the cycle.
A total count of tokens in the cycle may be a preset value, and the total count of tokens may be a sum of the one or more tokens held by the pending request, the count of tokens in the target token set, and a count of tokens in the another token set.
For a specific description of each module of the system shown above, please refer to the flowcharts of the present disclosure, for example, FIG. 3 and related descriptions thereof.
It should be understood that the system and modules thereof shown in FIG. 2 may be implemented in various ways. For example, in some embodiments, the system and modules thereof may be implemented by hardware, software, or a combination of software and hardware. The hardware part may be implemented by using a dedicated logic; and the software portion may then be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor device or specially designed hardware. Those skilled in the art will appreciate that the method and the system described above can be implemented using computer-executable instructions and/or contained in processing device control codes, for example on a carrier medium such as a magnetic disk, CD or DVD-ROM, such as a programmable read-only memory (firmware) or on a data carrier such as an optical or electronic signal carrier. The system and modules thereof in the present disclosure may not only be implemented by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., may also be implemented by, for example, software executed by various types of processing devices, or may also be implemented by a combination (e.g., firmware) of the above hardware circuits and software.
It should be noted that the above description of the system for request traffic management and modules thereof is only for convenience of description, and does not limit the present disclosure to the scope of the illustrated embodiments. It can be understood that for those skilled in the art, after understanding the principle of the system, it is possible to combine various modules arbitrarily, or form a subsystem to connect with other modules without departing from this principle. For example, in some embodiments, the distribution module 220 and the reclaiming module 230 may be different modules in one system, or one module may implement the functions of the above-mentioned two or more modules. For example, two or more modules may share one storage module, or each module may have its own storage module. Such deformations are within the protection scope of the present disclosure.
FIG. 3 is an exemplary flowchart illustrating a process for request traffic management according to some embodiments of the present disclosure. In some embodiments, one or more operations of the process 300 shown in FIG. 3 may be implemented in the system for request traffic management. In some embodiments, the process 300 shown in FIG. 3 may be performed by the system 200 shown in FIG. 2. In some embodiments, the process 300 shown in FIG. 3 may be performed by the processing device 120. For example, the process 300 shown in FIG. 3 may be stored in a storage medium in the form of instructions, and invoked or executed by the processing device 120. For illustrative purposes, the execution of the process 300 may be described below using the processing device 120 as an example.
A defined time range may be referred to as a cycle. The duration of the cycle may be set according to requirements. In the process 300 in this description, one cycle is mainly taken as an example to describe the method for request traffic management. In some embodiments, for any one of one or more cycles, request traffic management within the cycle may be implemented according to the process 300. For example, for any time period (e.g., 1 min) , the time period may be divided into multiple cycles (e.g., each cycle may be 10 s long, then 1 min may be divided into 6 cycles according to a time sequence: 0s-10s, 10s-20s, 20s-30s, 30s-40s, 40s-50s, and 50s-60s) , in any one of the cycles, the request traffic management in the cycle may be implemented through the process 300, to implement the request flow management in the time period.
In some embodiments, the cycle update/cycle alternation may be triggered by a timer. For example, the first cycle may start when the timing starts, and the current cycle may end every 10s thereafter and a new cycle may start. For any cycle, a cycle preceding the cycle in the time sequence may be referred to as a previous cycle of the cycle, and a cycle following the cycle in the time sequence may be referred to as the next cycle of the cycle. Two cycles adjacent to each other in the time sequence may be referred to as adjacent cycles.
In some embodiments, the aforementioned cycle may also be referred to as a traffic limiting cycle.
As shown in FIG. 3, the process 300 may include one or more of the following operations.
In S310, a pending request may be obtained (e.g., by a processing device (e.g., the processing device 120)) .
The request may refer to a request to a service platform (or a service system) , which is used to request the service platform to perform certain processing to achieve the purpose of providing business services (e.g., a data access request, a data storage request, a data processing request, etc. ) . Specifically, for example, for a social service platform, the request may include a request sent by a user terminal to transmit a certain message to a designated contact. The social service platform may transmit the message to the terminal of the designated contact after receiving the request.
In some embodiments, when the terminal establishes a connection with the service platform, the processing device 120 may obtain the request sent by the terminal to the service platform. In some embodiments, the request obtained by the processing device 120 and waiting to be processed by the service platform may be referred to as a pending request. In some embodiments, the processing device 120 may cyclically manage the request based on a preset cycle time.
In some embodiments, the processing device 120 may also obtain unprocessed pending requests in a historical time from a request waiting queue. For more description about the request waiting queue, please refer to FIG. 4 and related descriptions thereof.
In S320, in response to a count of tokens in a target token set corresponding to the cycle being greater than 0, a token may be taken from the target token set and issued to the pending request, to facilitate a service system (e.g., one or more processing devices) to process the pending request.
A token set (also referred to as a token bucket) may be a system service most commonly used in network traffic shaping and rate limiting. In some embodiments, the processing device 120 may be used as a producer, and may add a token to the bucket (or a container, or a specific storage space) . The request may be used as a consumer. Each request may need to take a token from the bucket. If the request takes the token, the request continues to be executed. If there is no token in the bucket, the processing device may trigger a request deny strategy (e.g., the request may wait timeout, or the request may be directly denied) , thereby achieving the purpose of traffic limiting.
The token may be understood as a passport for a request in a traffic limiting environment. When the request holds the token, the service platform that issues the token may provide a corresponding business service. Otherwise, processing may be delayed or a specified error code may be returned.
During the traffic limiting process, the token may have a life cycle, and its stage may be represented by a state of the token in the full life cycle of the token. In some embodiments, any token in the life cycle may have one or more states (e.g., unissued, issued, reclaimed) . The state "unissued" may mean that the token is in the token bucket of the current traffic limiting cycle. The state "issued" may mean that the token is not in any token bucket and is held by the request and is in service. The state "reclaimed" may mean that the request that holds the token has been completed or terminated, and then the token is reclaimed into a specified token reclaiming bucket.
In some embodiments, considering that there is a maximum load capacity of an online service platform, the total count of tokens issued by the processing device may be a fixed value or may have a maximum value. In some embodiments, the count of tokens may be a constant. In some embodiments, when no external business adjusts the count of tokens in the token bucket, the total count of stock tokens in the token bucket may remain unchanged, and only the state of each token may be changed. Based on the constant total count of tokens, it can be known that the count of tokens assigned to the request is less than or equal to the total count of tokens. Since the total count of tokens may represent the maximum load capacity of the platform, it can be known that a load capacity induced by the request may be less than or equal to the maximum load capacity of the platform, thus ensuring stable operation of the online service platform.
In some embodiments, the processing device may be provided with a corresponding target token set and another token set (e.g., a reclaimed token set) for each cycle. For convenience of description only, the target token set and the reclaimed token set corresponding to a certain cycle may be referred to as the target token set and the reclaimed token set of the cycle. The target token set of the cycle may refer to a storage space for tokens to be issued in the current cycle. The reclaimed token set of the cycle may refer to a storage space for tokens reclaimed in the current cycle.
In some embodiments, the count of tokens in the target token set and the count of tokens in the reclaimed token set within the cycle may change with the issuing and reclaiming of tokens in the current cycle. For example, at the beginning of an Nth cycle, the target token set may include 3 tokens. If 2 new requests are received in the current cycle, then 2 tokens in the target token set may be distributed to the 2 new requests, respectively. At this time, the count of tokens in the target token set may be reduced to 1. As another example, at the beginning of the Nth cycle, the reclaimed token set may include 0 token. If 1 token is reclaimed in the current cycle, then the count of tokens in the reclaimed token set may be increased to 1.
In some embodiments, a total count of tokens in any cycle may be a preset value. It can also be understood that the total count of tokens may remain constant in any cycle. The total count of tokens in the cycle may be a sum of the token (s) held by the pending request (s) and the count of tokens in the target token set and the reclaimed token set in the cycle.
In some embodiments, the preset value may represent a maximum count of concurrent requests that the service platform may process, and may also be referred to as a traffic limiting threshold. For example, if the online service platform is set to process a maximum count of 5000 requests per traffic limiting cycle (e.g., 10s) , the preset value may be 5000. The preset value may be set according to requirements or device conditions of the device where the service platform is implemented.
In some embodiments, at the beginning of a cycle, the count of tokens in the target token set corresponding to the cycle may be a preset value or may be determined based on the count of tokens in the reclaimed token set in a previous cycle of the current cycle. For example, at the beginning of a first cycle, the target token set may be initialized according to the aforementioned preset value, i.e., a corresponding count of tokens may be generated according to the preset value and put into the target token set. In some embodiments, at the beginning of a current cycle other than the first cycle (i.e., a cycle following the first cycle) , the count of tokens in the target token set may be equal to the count of tokens in the reclaimed token set at the end of the previous cycle of the current cycle.
In some embodiments, at the beginning of the current cycle, the count of tokens in the reclaimed token set corresponding to the current cycle may be determined based on the count of tokens in the target token set at the end of the previous cycle of the current cycle. For example, at the beginning of the Nth cycle, the count of tokens in the reclaimed token set in the nth cycle may be equal to the count of tokens in the target token set at the end of the (n-1) th cycle.
In some embodiments, when the nth cycle alternate to the (n+1) th cycle, the target token set of the nth cycle may be transformed into the reclaimed token set of the (n+1) th cycle, and the reclaimed token set of the nth cycle may be transformed into the target token set of the (n+1) th cycle. Exemplarily, if at the end of the Nth cycle, the count of tokens in the target token set is 3, and the count of tokens in the reclaimed token set is 20, then at the beginning of the (n+1) th cycle, the target token set and the reclaimed token set may be exchanged, that is, the count of tokens in the target token set may be 20, and the count of tokens in the reclaimed token set may be 3.
In some embodiments, the target token set and the reclaimed token set may be determined based on a three-color rolling model. For more descriptions about the three-color rolling model, please refer to the related descriptions in FIGs. 13-15.
In some embodiments, when S320 is executed, if there are one or more tokens in the target token set of the current cycle (i.e., the count of tokens is greater than or equal to 1) , the processing device may issue the token (s) to the pending request (s) . If there is no token in the target token set in the current cycle (i.e., the count of tokens is 0) , the processing device may not issue any token to the pending request (s) .
In some embodiments, in one cycle, the token (s) may be obtained from the target token set one by one and issued to the pending request (s) according to an order of the pending requests until the count of tokens in the target token set is changed to 0. When a token is issued to a request, the state of the token may be changed from unissued to issued.
In some embodiments, in one cycle, the processing device may perform a limiting processing on the received pending requests in response to the count of tokens in the target token set of the cycle being 0. For example, when the count of tokens in the target token set is 0, the processing device may stop issuing tokens to pending requests. When a request is being processed, the token (s) may not be issued to the pending request (s) , then the processing device may perform the limiting processing (e.g., stop processing or suspend processing) on the pending request (s) . For more descriptions about the limiting processing, please refer to FIG. 4 and relevant contents thereof.
In some embodiments, issuing the token (s) to the request (s) (i.e., to make the request (s) hold the token (s) ) may be understood as establishing a connection between the request (s) and the token (s) (e.g., writing a storage address of the corresponding token into the request, or adding information about the held token to the request) . In some embodiments, when the service system processes the request, the service system may first check whether the request holds a token. If yes, the service system may process the request; if not, the service system may not process the request.
In S330, in response to the pending request being processed within the cycle, one or more tokens held by the pending request (s) may be reclaimed and added to the reclaimed token set corresponding to the cycle.
In some embodiments, the processing device may detect the completion of the pending request (s) . If the pending request (s) holding the token (s) are processed (i.e., the request processing is completed) , the token (s) held by the request (s) may be released. The processing device may reclaim the token (s) , and adjust the state (s) of the reclaimed token (s) to reclaimed.
In some embodiments, when the processing device reclaims a token, the processing device may determine a reclaiming position of the token based on the cycle in which the token is issued (which may be referred to as a token issuing cycle) and the cycle in which the token is reclaimed (which may be referred to as a token reclaiming cycle) .
In some embodiments, if the token reclaiming cycle of a token is inconsistent with the token issuing cycle of the token, the token may be directly reclaimed into the target token set of the reclaiming cycle, so that the token is reissued in the reclaiming cycle. For example, for any cycle, if a token is issued to a pending request m in the current cycle, in response to the pending request m being processed in another cycle (e.g., a cycle following the current cycle) , the processing device may reclaim the token held by the pending request m and add the token to the target token set of the another cycle.
In some embodiments, when the token reclaiming cycle of a token is consistent with the token issuing cycle of the token, the token may be reclaimed to the reclaimed token set in the token issuing cycle of the token, so that the token is reissued in a next cycle. For example, for any cycle, if a token is issued to a pending request m in the current cycle, in response to the pending request m being processed in the current cycle, the processing device may reclaim the token held by the pending request m and add the token to the reclaimed token set of the current cycle.
According to some embodiments of the present disclosure, it can be guaranteed that the token is issued only once in each cycle, thereby ensuring that the count of requests for the service of the server is controlled no larger than the preset value, and ensuring the normal load of the server.
According to the system for request traffic management provided in the present disclosure, the total count of tokens can be guaranteed to be constant by issuing and reclaiming tokens, so that the system for request traffic management can adapt to different environments. When encountering environmental fluctuations, the present system may have a better speed of convergence to a target speed limiting threshold; and when the environment deteriorates, the environment may not deteriorate continuously. When the environment starts to recover, the real-time bottleneck value of the system to provide services may be quickly touched after the overall traffic limiting threshold is satisfied after at most one speed limiting cycle.
FIG. 4 is an exemplary flowchart illustrating a limiting processing according to some embodiments of the present disclosure. In some embodiments, one or more operations of a process 400 shown in FIG. 4 may be implemented in the system for request traffic management. In some embodiments, the process 400 shown in FIG. 4 may be performed by the system 200 shown in FIG. 2. In some embodiments, the process 400 shown in FIG. 4 may be performed by the processing device 120. For example, the process 400 shown in FIG. 4 may be stored in a storage medium in the form of instructions, and invoked and/or executed by the processing device 120. For illustrative purposes, the execution of process 400 is described below using processing device 120 as an example.
In the process 400 of the present disclosure, one cycle is mainly taken as an example to describe the limiting processing involved in one cycle. In some embodiments, for any one of one or more cycles, the limiting processing within the cycle may be implemented according to the process 400.
As shown in FIG. 4, the process 400 may include one or more of the following operations.
In S410, the processing device may add a pending request to a request waiting queue, to facilitate subsequent obtaining of the pending request from the request waiting queue.
The request waiting queue may be a cache space for unexecuted pending request (s) , such as a FIFO queue. In some embodiments, if there is no token in the target token set, the processing device may write one or more pending requests that are not assigned token (s) into the request waiting queue according to a sequence. The sequence may be a receiving sequence of the pending requests or any other sequence set according to requirements.
In some embodiments, after the processing device adds the pending request (s) to the request waiting queue, the processing device may generate prompt information and send the prompt information to a request initiator to explain that the server is busy at present.
In some embodiments, a count of requests that can be accommodated in the request waiting queue may be a fixed value. The count of requests that can be accommodated in the request waiting queue may reflect a count of pending requests allowed to wait by the server.
In some embodiments, the count of requests that can be accommodated in the request waiting queue may be preset according to actual conditions. For example, the count of requests that can be accommodated in the request waiting queue may be a total count of requests that can be processed by the processing device within one cycle. If the online service platform can process 5000 requests per traffic limiting cycle (e.g., 10s) , the count of requests that can be accommodated in the request waiting queue may be 5000.
In some embodiments, the processing device may add the pending request (s) to the request waiting queue in response to a request storage quantity in the request waiting queue not meeting a limiting condition. The limiting condition may be represented as a correlation between the request storage quantity and the fixed value. For example, if the count of pending requests in the request waiting queue is greater than or equal to the fixed value, the request storage quantity may be deemed to satisfy the limiting condition. If the count of pending requests in the request waiting queue is less than the fixed quantity, the request storage quantity may be deemed not to satisfy the limiting condition.
In some embodiments, when a token can be issued to a pending request, the pending request may be obtained from the request waiting queue, and the token may be issued to the pending request.
In some embodiments, the request waiting queue may include pending requests that are unprocessed historically. If there are too many requests in the current traffic limiting cycle, the processing device may process one or more pending requests in the current cycle, and then store the remaining pending requests in the request waiting queue.
In some embodiments, in response to the request waiting queue including a plurality of pending requests that are unprocessed historically, one or more pending requests may be obtained from the request waiting queue according to a request sequence of the plurality of pending requests. For example, the pending requests in the request waiting queue may be numbered according to an adding sequence of the pending requests, and each of the pending requests may have a serial number. When one or more tokens are assigned to one or more of the pending requests, the tokens may be assigned in sequence according to the serial numbers of the pending requests in the request waiting queue.
In some embodiments, when one or more tokens are assigned to one or more of the pending requests and/or one or more pending requests are added to the request waiting queue, the validity of the pending request (s) may be detected. That is, the processing device may verify whether the pending request (s) are valid; and deny the pending request (s) in response to determining that the pending request (s) are invalid. Whether a pending request is valid may be determined according to request content of the pending request and the identity of a requester of the pending request. For example, if the request content of a pending request is within the service scope of the online service platform and the requester of the pending request is a normal user who does not attack the platform, it can be determined that the pending request is valid.
For more descriptions about token issuing and subsequent processing of the pending request (s) , please refer to S320 and S330 in FIG. 3, which is not repeated herein.
In S420, in response to the request storage quantity in the request waiting queue meeting the limiting condition, the processing device may deny the pending request.
In some embodiments, if the request storage quantity satisfies the limiting condition, the processing device may stop adding pending request (s) to the request waiting queue, and deny request (s) that are not assigned token (s) . At this time, the processing device may prompt the requester that the server is busy at present, and/or prompt the requester to visit later.
Based on the above-mentioned limiting processing, when the server cannot process a large load of requests, the server can cache one or more pending requests that are not assigned token (s) , so that when the load of requests decreases subsequently, the cached pending request (s) can continue to be processed, thus improving the user experience for the requester.
In different parts of the present disclosure (e.g., the relevant descriptions in FIGs. 5-12) , the request traffic management may also be called high concurrency traffic limiting, an application for token may also be called a token application event, the target token set may be a token issuing bucket, the reclaimed token set may be a reclaiming bucket, a token issuing and reclaiming mode based on the target token set and the reclaimed token set of one cycle may be an hourglass model, and the request waiting queue may be called a business request queue.
FIG. 5 is a schematic diagram illustrating an hourglass model of a cycle n according to some embodiments of the present disclosure. FIG. 6 is a schematic diagram illustrating an hourglass model of a cycle n+1 according to some embodiments of the present disclosure.
In FIG. 5 and FIG. 6, the entire hourglass model may include 2 token buckets. For example, during initialization, i.e., in a cycle n, the role of a token bucket 1 may be a token issuing bucket (i.e., the tokens inside the bucket are in the "unissued" state) , and no token may be issued before traffic limiting starts. The role of a token bucket 2 may be a token reclaiming bucket (i.e., the tokens inside the bucket are in the "reclaimed" state) .
The first cycle after the beginning of traffic limiting may be tentatively named as a cycle 1. The request may obtain a token from the token issuing bucket (currently the bucket 1) . The service may be provided for the request that obtains the token, and the token may be in the "issued" state. The token may be reclaimed after the request holding the token is completed or terminated.
It should be noted that since the life cycle of the request does not match the traffic limiting cycle, the value of the traffic limiting cycle when the token is issued may be recorded. If the traffic limiting cycle is consistent with the current traffic limiting cycle, the token may be put into the token reclaiming bucket. If the traffic limiting cycle is inconsistent with the current traffic limiting cycle, the token may be put into the token issuing bucket of the current cycle.
In this way, it can effectively guarantee that the token in the same cycle can only be used once at most, so as to achieve the effect of traffic limiting.
A registered timer may trigger the iteration of the traffic limiting cycle. Like the flipping of an hourglass, the roles of the two buckets, i.e., the role of the token issuing bucket and the role of the token reclaiming bucket, may be reversed. As shown in FIG. 5 and FIG. 6, in the cycle n, the token bucket 1 may be the token issuing bucket, and the token bucket 2 may be the token reclaiming bucket, while in the cycle n+1, the token bucket 1 may be the token reclaiming bucket, and the token bucket 2 may be the token issuing bucket.
When request (s) are traffic-limited, the request (s) may enter a fixed-length FIFO queue (i.e., a traffic limiting deny strategy) to deal with traffic limiting of the current cycle caused by a relatively small number of tokens in the token issuing bucket in the cycle n (correspondingly the token issuing bucket in the cycle n+1 may probability have a relatively large number of tokens) after the requests in the cycle n-1 is greatly reduced. The requests taken out of the FIFO queue may be processed first when the traffic limiting cycle is iterated, and the validity of the requests may be verified (to prevent invalid request identification such as disconnection of the requester) .
FIG. 7 is a schematic flowchart illustrating an exemplary process of high concurrency traffic limiting according to some embodiments of the present disclosure. FIG. 8 is a schematic flowchart illustrating a process of obtaining and reclaiming of an hourglass model token according to some embodiments of the present disclosure.
The high concurrency traffic limiting provided by the present disclosure may be applied to a client, for example, an independent producer of data such as disk-oriented and network IO traffic limiting, or a server proxy. For example, if a web crawler captures data and needs to upload the data to the server for aggregation, it is more appropriate to set the traffic limiting function in the crawler. Through the above hourglass model for traffic limiting, the effect of peak shaving can be achieved, i.e., a certain batch delay can be tolerated in a business scenario.
As shown in FIG. 7, an exemplary process for high concurrency traffic limiting may include one or more of the following operations.
In S710, in response to triggering of a token application event, a token may be obtained from a token issuing bucket of a first traffic limiting cycle, and the value of the token issuing cycle may be updated to the first traffic limiting cycle.
In some embodiments of the present application, when a device for high concurrency traffic limiting enters a traffic limiting state, a high concurrency traffic limiting strategy may be activated. Specifically, after the device for high concurrency traffic limiting receives an instruction of triggering the token application event, the device may take out a token from the token issuing bucket of the current traffic limiting cycle, i.e., the first traffic limiting cycle. After a service request of the token application event obtains the token, the authority to execute the business request may be obtained.
In some embodiments, the total count of tokens in each traffic limiting cycle may be a fixed value.
Further, the device for high concurrency traffic limiting device may need to update the value of the token issuing cycle to the first traffic limiting cycle, so as to record the traffic limiting cycle within which the token is taken out.
When the tokens in the token issuing bucket in the first traffic limiting cycle have been extracted for other business requests, i.e., when there is no token to be taken out, it means that the current business processing load has reached a traffic limiting threshold in the first traffic limiting cycle, and no more tokens can be assigned to business requests (i.e., pending requests) in the same cycle. At this time, the device for high concurrency traffic limiting may implement a traffic limiting deny strategy. As shown in FIG. 8, the service request may be put into a fixed-length FIFO queue for queuing. The service requests in the FIFO queue may be sorted according to the time when the requests enter the queue, i.e., the earlier the time is, the higher the sorting may be.
In S720, a business request of a token application event may be executed based on the token, and the generated business data may be uploaded to the server.
In some embodiments of the present disclosure, after the service request corresponding to the application token event obtains a token, the authority to execute the service request may be obtained. At the client, e.g., an edge terminal of a distributed remote procedure call (RPC) system, as an independent producer of data, the process of the client executing the business request may be regarded as generating relevant data of the business request, and uploading the relevant data of the business request to a central side, e.g., the server, of the distributed RPC system.
After the execution of the business request is completed, the token may be released, i.e., the token may be put back into a token reclaiming bucket of the original traffic limiting cycle, and may not be taken out again during the original traffic limiting cycle, thereby realizing closed-loop monitoring of traffic relationship. Alternatively, the token may be put into a token issuing bucket of a subsequent traffic limiting cycle, and may be taken out for use in the subsequent traffic limiting cycle. Therefore, the device for high concurrency traffic limiting may need to compare the value of the token issuing cycle with the current traffic limiting cycle, and determine the whereabouts of the token according to the result of whether the value of the token issuing cycle and the current traffic limiting cycle are consistent.
In S730, whether the current traffic limiting cycle within which the business request corresponding to the token is completed is the first traffic limiting cycle is detected.
In some embodiments of the present disclosure, the device for high concurrency traffic limiting may obtain the value of the token issuing cycle, e.g., the first traffic limiting cycle, and may also obtain the current traffic limiting cycle within which the service request corresponding to the token is completed. Then, whether the current traffic limiting cycle within which the business request corresponding to the token is completed and the first traffic limiting cycle are the same traffic limiting cycle may be determined. If yes, the process 700 may proceed to S740. If not, the process 700 may proceed to S750.
In S740, the token may be put back into the token reclaiming bucket of the first traffic limiting cycle.
In some embodiments of the present disclosure, if the current traffic limiting cycle within which the token needs to be put back and the traffic limiting cycle within which the token is taken out are the same traffic limiting cycle, it means that the token may not be taken again within the current traffic limiting cycle. Therefore, the device for high concurrency traffic limiting may need to put the token back into the token reclaiming bucket of the first traffic limiting cycle.
In S750, the token may be put into the token issuing bucket of the second traffic limiting cycle.
In some embodiments of the present disclosure, if the current traffic limiting cycle within which the token needs to be put back and the traffic limiting cycle within which the token is taken out are not the same traffic limiting cycle, it means that the token may still be taken to use within the current traffic limiting cycle. Therefore, the device for high concurrency traffic limiting may need to put the token back into the token issuing bucket of the second traffic limiting cycle. At this time, the time sequence relationship between the first traffic limiting cycle and the second traffic limiting cycle may not be limited.
Further, it should be noted that in some embodiments, the first traffic limiting cycle and the second traffic limiting cycle may be adjacent traffic limiting cycles. When entering the second traffic limiting cycle, the device for high concurrency traffic limiting device may reverse the roles of the two token buckets (the role of the token issuing bucket and the role of the token reclaiming bucket) in the first traffic limiting cycle. As shown in FIG. 5 and FIG. 6, the first traffic limiting cycle may be an hourglass model in FIG. 5, the token bucket 1 may be a token issuing bucket, and the token bucket 2 may be a token reclaiming bucket. The second traffic limiting cycle may be the hourglass model in FIG. 6, the token bucket 2 may be the token issuing bucket, and the token bucket 1 may be the token reclaiming bucket.
The roles of the above-mentioned token buckets may reverse. The iteration of the traffic limiting cycle may be triggered through the registered timer. Similar to flipping the hourglass according to a preset cycle, the upper token bucket of the hourglass may be kept as the token issuing bucket, and the lower token bucket of the hourglass may be kept as the token reclaiming bucket.
Specifically, if the current traffic limiting cycle within which the business request corresponding to the token is completed is the second traffic limiting cycle, the device for high concurrency traffic limiting may put the tokens in the token issuing bucket in the first traffic limiting cycle into the token reclaiming bucket in the second traffic limiting cycle, and put the tokens in the token reclaiming bucket in the first traffic limiting cycle into the token issuing bucket in the second traffic limiting cycle, so as to realize the hourglass flipping and role exchange.
In some embodiments of the present disclosure, in response to triggering of a token application event, the device for high concurrency traffic limiting may obtain the tokens from the token issuing bucket of the first traffic limiting cycle, and the value of the token issuing cycle may be updated to the first traffic limiting cycle. The business request of the token application event may be executed based on the token, and the generated business data may be uploaded to the server. Whether the current traffic limiting cycle within which the business request corresponding to the token is completed is the first traffic limiting cycle may be detected. If yes, the token may be put back into the token reclaiming bucket of the first traffic limiting cycle. In some embodiments, the total count of tokens in each traffic limiting cycle may be a fixed value. The above-mentioned process for high concurrency traffic limiting can form a closed loop of token traffic relationship by issuing tokens and reclaiming tokens, and the inflow and outflow of token traffic may be set to realize the effect of precise traffic limiting.
In order to improve precise traffic limiting and efficient traffic limiting, the present disclosure also provides a method for quickly obtaining tokens, i.e., assigning tokens of historical traffic limiting cycles to valid historical requests in the business request queue, to efficiently utilize available tokens. FIG. 9 is a schematic flowchart illustrating another embodiment of the method for high concurrency traffic limiting provided by the present disclosure.
As shown in FIG. 9, an exemplary process for high concurrency traffic limiting may include one or more of the following operations.
In S910, if the current traffic limiting cycle within which the business request corresponding to the token is completed is the second traffic limiting cycle, a business request queue may be obtained.
In some embodiments of the present disclosure, if the current traffic limiting cycle within which the business request corresponding to the token is completed (e.g., the second traffic limiting cycle) and the value of the token issuing cycle (e.g., the first traffic limiting cycle) is not the same traffic limiting cycle, the device for high concurrency traffic limiting may further obtain a fixed-length FIFO queue.
In S920, whether there is a valid historical request in the business request queue may be detected.
In some embodiments of the present disclosure, as shown in FIG. 4, the device for high concurrency traffic limiting may search for a business request from the business request queue, and check the validity of the business request. If the valid historical request is successfully searched out, the process 900 may proceed to S930. If there is no valid historical request in the business request queue, the process 900 may proceed to S940.
In S930, the first valid request in the business request queue may be obtained, the value of the token issuing cycle may be updated to the second traffic limiting cycle, and the first valid request may be executed based on the token.
In some embodiments of the present disclosure, the FIFO queue may pop out the header (e.g., the first valid request) . The token may be issued to the first valid request. The business of the first valid request may be executed, and the value of the token issuing cycle may be updated to the second traffic limiting cycle.
In this way, the device for high concurrency traffic limiting does not need to return the token (s) to the token issuing bucket of the current traffic limiting cycle, but directly issues the token (s) to the valid request (s) of the FIFO queue, improving the token allocation efficiency.
In S940, the token may be put back into the token issuing bucket of the second traffic limiting cycle.
In some embodiments of the present disclosure, the device for high concurrency traffic limiting may return the token (s) to the current traffic limiting cycle (e.g., the token issuing bucket of the second traffic limiting cycle) for waiting to be taken out.
In some embodiments of the present disclosure, the device for high concurrency traffic limiting uses an hourglass model to realize inflow and outflow monitoring of closed-loop traffic, and truly realize the precise traffic limiting of the traffic limiting effect. The hourglass model of the present disclosure does not have the cycle critical problem in the traditional traffic limiting algorithm, and supports burst traffic. Compared with using the indirect index to calculate a cycle token threshold to delivery, the process for high concurrency traffic limiting of some embodiments of the present disclosure directly uses a traffic outflow index, the overall implementation is relatively simple, and the effect is direct and obvious.
It should be noted that the above description is merely provided for the purposes of illustration, and not intended to limit the scope of the present disclosure. For persons having ordinary skills in the art, multiple variations and modifications may be made under the teachings of the present disclosure. However, those variations and modifications do not depart from the scope of the present disclosure.
The present disclosure also proposes a device for high concurrency traffic limiting. FIG.
10 is a schematic structural diagram illustrating an exemplary device for high concurrency traffic limiting according to some embodiments of the present disclosure.
The device 1000 for high concurrency traffic limiting of some embodiments may include an obtaining module 1010, an execution module 1020 and a reclaiming module 1030.
The obtaining module 1010 may be configured to obtain a token from a token issuing bucket of a first traffic limiting cycle in response to triggering of a token application event. The value of the token issuing cycle may be updated to the first traffic limiting cycle.
The execution module 1020 may be configured to execute a business request of the token application event based on the token, and upload the generated business data to a server.
The reclaiming module 1030 may be configured to detect whether the current traffic limiting cycle within which the business request corresponding to the token is completed is the first traffic limiting cycle; and if yes, put the token back into a token reclaiming bucket of the first traffic limiting cycle. A total count of tokens in each traffic limiting cycle may be a fixed value.
FIG. 11 is a schematic structural diagram illustrating another exemplary device for high concurrency traffic limiting according to some embodiments of the present disclosure.
The device 1100 may include a processor 1110, a memory 1120, an input/output device 1130 and a bus 1140.
The processor 1110, the memory 1120, and the input/output device 1130 may be respectively connected to the bus 1140. The memory 1120 may store program data. The processor 1110 may be configured to execute the program data to implement the method for high concurrency traffic limiting described in the present disclosure.
In some embodiments of the present disclosure, the processor 1110 may also be referred to as a CPU (Central Processing Unit) . The processor 1110 may be an integrated circuit chip with signal processing capability. The processor 1110 may also be a general-purpose processor, a digital signal processor (DSP) , an application specific integrated circuit (ASIC) , a field programmable gate array (FPGA) , or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor, or the processor 1110 may be any conventional processor, or the like.
The present disclosure also provides a computer storage medium. FIG. 12 is a schematic structural diagram illustrating an exemplary computer storage medium according to some embodiments of the present disclosure. The computer storage medium 1200 may store program data 1210. When executed by the processor, the program data 1210 is used to implement the method for high concurrency traffic limiting of the present disclosure.
In some embodiments, the method for request traffic management provided in the present disclosure may also be performed based on a three-color rolling model.
In some embodiments, the three-color rolling model may include a past token bucket, a current token bucket, and a future token bucket. The past token bucket may be used to store unissued tokens in the previous cycle. The current token bucket may be used to store tokens to be issued in the current cycle and the tokens issued in the previous cycle. The future token bucket may be used to store tokens that are issued and reclaimed during the current cycle.
In some embodiments, if the method for request traffic management provided in the present disclosure is implemented based on the three-color rolling model, the target token set and the reclaimed token set can be determined from the three-color rolling model according to actual situations, and the method for request traffic management provided by the present disclosure can be executed accordingly. When issuing token (s) , the token (s) in the past token bucket (at this time, the past token bucket may be the target token set) may be issued by priority. If the past token bucket is empty, the token (s) in the current token bucket (at this time, the current token bucket may be the target token set) may be started to be issued. When reclaiming token (s) , if the token issuing cycle is consistent with the current cycle, the token (s) may be put into the future token bucket (at this time, the future token bucket may be the reclaimed token set) . If the token issuing cycle is consistent with the current cycle, the token (s) may be put into the current token bucket (at this time, the current token bucket may be the reclaimed token set) . More descriptions of the three-color scrolling model may be found elsewhere in the present disclosure (e.g., FIGs. 13-15 and descriptions thereof) .
FIG. 13 is a schematic diagram illustrating a three-color rolling model of a cycle n according to the present disclosure. FIG. 14 is a schematic diagram illustrating a three-color rolling model of a cycle n+1 according to the present disclosure. FIG. 15 is schematic diagram illustrating a three-color rolling model of a cycle n+2 according to the present disclosure.
As shown in FIG. 13 to FIG. 15, the target token set and the reclaimed token set may be re-divided into three token buckets. That is, the entire three-color rolling model may include three token buckets, e.g., a past token bucket, a current token bucket, and a future token bucket. Blue (block A) represents the current token bucket of a current traffic limiting cycle, green (block B) represents the future token bucket of a future traffic limiting cycle, and beige (block C) represents the past token bucket of a past traffic limiting cycle. For example, as shown in FIG. 13, bucket 1 is in the current traffic limiting cycle. All tokens may be initialized, and the state of the tokens may be "unissued" .
In token issuing operation, a token may be obtained from the past token bucket preferentially, and if there is a token in the past token bucket, the token issuing cycle may be refreshed. If there is no token in the past token bucket, the token may be obtained from the current token bucket. If there is a token in the past token bucket, the token issuing cycle may be refreshed, and the state of the obtained token may be changed to "issued" and issued to the request to hold.
In token reclaiming operation, whether the token issuing cycle and the traffic limiting cycle implemented by the current token bucket are the same is determined. If not, the token may be returned to the current token bucket, and if yes, the token may be return to the future token bucket.
The registered timer may trigger the iteration of the traffic limiting cycle, the role of each bucket may be changed (e.g., among past, current, future) , and the token (s) in the past token bucket may be put into the current token bucket at one time. The state transition process may include from future to current, from current to past, from past to future, and the cycle may be repeated.
As shown in FIGs. 13-15, in the traffic limiting cycle n, the bucket 1 may be the current token bucket, the bucket 2 may be the future token bucket, and the bucket 3 may be the past token bucket. In the traffic limiting cycle n+1, the bucket 1 may be the past token bucket, the bucket 2 may be the current token bucket, and the bucket 3 may be the future token bucket. In the traffic limiting cycle n+2, the bucket 1 may be the future token bucket, the bucket 2 may be the past token bucket, and the bucket 3 may be the current token bucket.
In some embodiments, the three-color scrolling model introduced above in combination with middleware such as redis may be applied to a distributed traffic limiting scenario. Specifically, the three-color scrolling model may have three actions (e.g., issuing tokens, reclaiming tokens, and iteration of the traffic limiting cycle) . The three actions may be cured by lua scripts, i.e., the key may record the iteration cycle ID, and the value may be a string (i.e., the count of tokens in the bucket may be recorded) . By cooperating with the Incrby command, replacing the incr command of the cycle ID issuer, and recording the count of tokens used, the corresponding model may be implemented in redis. The supporting user survival TTL mechanism may be used to reclaim tokens during the iterative process of the traffic limiting cycle when the user is abnormal.
An example is described to illustrate the state transition of the traffic limiting cycle of the three-color rolling model.
Taking the state of the traffic limiting cycle in FIGs. 13-15 as an example. All tokens of the three-color rolling model may be initialized in a cycle n, e.g., the count of tokens in the past token bucket may be 0, the count of tokens in the current token bucket may be 100, and the count of tokens in the future token bucket may be 0. In the cycle n, there may be 10 business requests to be executed. After the execution is completed, the count of tokens in the past token bucket may be 0, the count of tokens in the current token bucket may be 90, and the count of tokens in the future token bucket may be 10.
In a cycle n+1, the roles of the token buckets of the three-color rolling model may be switched, and thus, the count of tokens in the past token bucket may be 90, the count of tokens in the current token bucket may be 10, and the count of tokens in the future token bucket may be 0. In the cycle n+1, there may be 50 business requests to be executed. The tokens in the past token bucket may be taken out first (at this time, the past token bucket may be used as the target token set) . After the execution of the business requests is completed (at this time, the future token bucket may be used as the reclaimed token set) , the count of tokens in the past token bucket may be 40, the count of tokens in the current token bucket may be 10, and the count of tokens in the future token bucket may be 50.
In a cycle n+2, the roles of the token buckets of the three-color rolling model may be further switched, and thus, the count of tokens in the past token bucket may be 10, the count of tokens in the current token bucket may be 50, and the count of tokens in the future token bucket may be 40. At this time, the tokens in the future token bucket may need to be merged into the past token bucket, i.e., the state of the token bucket may be updated as follows: the count of tokens in the past token bucket may be 50, the count of tokens in the current token bucket may be 50, and the count of tokens in the future token bucket may be 0. In the cycle n+2, there may be 80 business requests to be executed. The tokens in the past token bucket are taken out first. After the business requests is completely executed, the count of tokens in the past token bucket may be 0, the count of tokens in the current token bucket may be 20, and the count of tokens in the future token bucket may be 80.
In the subsequent traffic limiting cycle, the token buckets of the three-color rolling model may be iterated in turn, which is not listed herein.
In some embodiments, the processing device may evaluate the performance of the service system, and adjust the traffic limiting threshold based on a performance evaluation result, to dynamically modify the total count of cycle tokens (i.e., the preset value in process 300) . The processing device may apply a new total count of cycle tokens in a traffic limiting cycle (hereinafter referred to as a new cycle) after the total count of cycle tokens is modified. In some embodiments, the aforementioned process may include one or more of the following operations.
In S1, the processing device may obtain the device data of the device where the service system is located in the historical time, and may obtain a performance evaluation result of the device where the service system is located based on the device data, and may determine an adjustment range of a traffic limiting threshold based on the performance evaluation result.
Device performance evaluation may be implemented through various existing device performance evaluation methods. The performance evaluation result may include an estimated request load capacity of the device (referring to the estimated maximum concurrent requests that the device may carry in one cycle) . If the difference (positive) between the estimated request load capacity and the original traffic limiting threshold m is greater than a set positive threshold, the adjustment range may be (m, m+difference) . Similarly, if the difference (negative) between the estimated request load capacity and the original traffic limiting threshold m is less than the set negative threshold, the adjustment range may be (m+difference, m) .
In S2, the total count of cycle tokens is adjusted according to the adjustment range.
1) If the total count of the original cycle tokens is increased by n, the total count of the cycle tokens may be adjusted according to one or more operations in the following 1.1-1.2.
1.1. At the beginning of a new cycle, the count of tokens in the target token set (e.g., token issuing bucket) of the new cycle may be increased by n.
1.2. At the beginning of a new cycle, the count of tokens in the target token set (e.g., token issuing bucket) of the new cycle may be increased by a, and the count of tokens in the reclaimed token set (e.g., token reclaiming bucket) of the new cycle may be increased by b, where a+b=n. The operation in 1.2 may make the increase of the device's request load capacity more gradual, and may go through 2 rounds of buffering. After 2 rounds of cycles, the effect of increasing the request load capacity by n may be achieved, avoiding the possible impact on the stability of device operation caused by a sudden increase in the device load capacity.
2) If the total count of the original tokens needs to be decreased by n, there may be different cases.
In a first case, if the count of tokens in the target token set (e.g., token issuing bucket) of the new cycle is greater than n, then the count of tokens in the target token set (e.g., token issuing bucket) of the new cycle may be decreased by n. Otherwise, the count of tokens in the target token set (e.g., token issuing bucket) of the new cycle may be decreased by a, and the count of tokens in the reclaimed token set (e.g., reclaiming bucket) of the new cycle may be decreased by b, where a+b=n. Similar to the previous effect of a gentle increase in the carrying capacity, the present operations can make the request load decrease more smoothly, and can better balance the processing requirements of the business request and the device performance.
In a second case, if the count of tokens in the target token set (e.g., the token issuing bucket) of the new cycle is less than or equal to n, then in the next multiple rounds (e.g., j rounds) , the total count of the cycle tokens may be decreased. The total count of the cycle tokens in the jth round cycle may be decreased by nj, where n1+... +nj=n, and the total count of the cycle tokens may be decreased according to the operations of the first case in each round cycle.
For S1, the processing device may obtain the performance evaluation result of the device (e.g., the estimated request load capacity of the device) by analyzing the device data in the historical time.
In S1.1, the processing device may obtain request traffic processing data of the device where the business service is located in multiple historical cycles. For each cycle, the request traffic processing data may include the total count of requests received in the cycle (referred to as the count of received cycles) , the total count of requests received in the cycle and processed within the cycle (referred to as the count of processed cycles) , an average processing time of requests received and processed within the cycle (referred to as request processing time in the cycle) , a maximum count of concurrent requests in the cycle, the count of tokens issued in the cycle, the count of tokens reclaimed in the cycle, the total count of tokens in the cycle, etc. The aforementioned request traffic processing data may be obtained based on device log data or obtained through other feasible methods.
In S1.2, the processing device may determine a performance evaluation result of the device through a prediction model based on the request traffic processing data of multiple historical cycles.
The prediction model may include a sequence model (e.g., long short term memory (LSTM) and other sequence models) . The processing device may generate a data sequence of the request traffic processing data of multiple historical cycles, and may input the data sequence into a sequence model. The sequence model may output the performance evaluation result of the device (including the estimated request load capacity of the device) .
The prediction model may be obtained through training. The training process may include performing supervised training of the prediction model based on multiple training samples and corresponding labels thereof. The training samples may be the request traffic processing data samples of multiple historical cycles of the device in a historical time period (such as T1) . The labels corresponding to the training samples may be the actual request load capacity (referring to the maximum concurrent requests that the device may actually process in one cycle) of the subsequent historical time period T2 (T2 is the same length as the traffic limiting cycle) of the historical time period T1 obtained by technicians through various device testing methods.
It should be noted that although several modules or sub-modules of the system are mentioned in the above detailed description, this division is only exemplary and not mandatory. Actually, according to the embodiment of the present disclosure, the features and functions of two or more modules described above may be embodied in one module. Instead, the features and functions of one module described above may be further divided to be embodied by a plurality of modules.
In addition, although the operations of the modules of the disclosed system are described in a particular order in the drawings, this does not require or imply that the operations must be performed in that particular order, or that all illustrated operations must be performed to achieve the desired result. Additionally or alternatively, certain operations may be omitted, multiple operations may be combined for execution as one operation, and/or one operation may be decomposed for execution as multiple operations.
The basic concept has been described above. Obviously, for those skilled in the art, the above detailed disclosure is only an example, and does not constitute a limitation to the present disclosure. Although not expressly stated here, those skilled in the art may make various modifications, improvements and corrections to the present disclosure. Such modifications, improvements and corrections are suggested in this disclosure, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of the present disclosure.
Meanwhile, the present disclosure uses specific words to describe the embodiments of the present disclosure. For example, "one embodiment" , "an embodiment" , and/or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present disclosure. Therefore, it should be emphasized and noted that references to "one embodiment" or "an embodiment" or "an alternative embodiment" two or more times in different places in the present disclosure do not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of the present disclosure may be properly combined.
In addition, unless clearly stated in the claims, the sequence of processing elements and sequences described in the present disclosure, the use of counts and letters, or the use of other names are not used to limit the sequence of processes and methods in the present disclosure. While the foregoing disclosure has discussed by way of various examples some embodiments of the invention that are presently believed to be useful, it should be understood that such detail is for illustrative purposes only and that the appended claims are not limited to the disclosed embodiments, but rather, the claims are intended to cover all modifications and equivalent combinations that fall within the spirit and scope of the embodiments of the present disclosure. For example, although the implementation of various components described above may be embodied in a hardware device, it may also be implemented as a software only solution, e.g., an installation on an existing server or mobile device.
In the same way, it should be noted that in order to simplify the expression disclosed in this disclosure and help the understanding of one or more embodiments of the invention, in the foregoing description of the embodiments of the present disclosure, sometimes multiple features are combined into one embodiment, drawings or descriptions thereof. This method of disclosure does not, however, imply that the subject matter of the disclosure requires more features than are recited in the claims. Rather, claimed subject matter may lie in less than all features of a single foregoing disclosed embodiment.
In some embodiments, counts describing the quantity of components and attributes are used. It should be understood that such counts used in the description of the embodiments use the modifiers "about" , "approximately" or "substantially" in some examples. Unless otherwise stated, "about" , "approximately" or "substantially" indicates that the stated figure allows for a variation of ±20%. Accordingly, in some embodiments, the numerical parameters used in the disclosure and claims are approximations that can vary depending upon the desired characteristics of individual embodiments. In some embodiments, numerical parameters should consider the specified significant digits and adopt the general digit retention method. Although the numerical ranges and parameters used in some embodiments of the present disclosure to confirm the breadth of the range are approximations, in specific embodiments, such numerical values are set as precisely as practicable.
Each of the patents, patent applications, publications of patent applications, and other material, such as articles, books, specifications, publications, documents, things, and/or the like, referenced herein is hereby incorporated herein by this reference in its entirety for all purposes, excepting any prosecution file history associated with same, any of same that is inconsistent with or in conflict with the present document, or any of same that may have a limiting affect as to the broadest scope of the claims now or later associated with the present document. By way of example, should there be any inconsistency or conflict between the description, definition, and/or the use of a term associated with any of the incorporated material and that associated with the present document, the description, definition, and/or the use of the term in the present document shall prevail.
In closing, it is to be understood that the embodiments of the application disclosed herein are illustrative of the principles of the embodiments of the application. Other modifications that may be employed may be within the scope of the application. Thus, by way of example, but not of limitation, alternative configurations of the embodiments of the application may be utilized in accordance with the teachings herein. Accordingly, embodiments of the present application are not limited to that precisely as shown and described.

Claims

A method implemented on at least one machine each of which has at least one processor and at least one storage device for request traffic management, the method comprising:

for any cycle,

obtaining a pending request;

in response to a count of tokens in a target token set corresponding to the cycle being greater than 0, taking a target token from the target token set and issuing the target token to the pending request to facilitate a service system to process the pending request; and

in response to the pending request being processed within the cycle, reclaiming one or more tokens held by the pending request and adding the one or more tokens to another token set corresponding to the cycle;

wherein:

a total count of tokens in the cycle is a preset value, and

the total count of tokens is a sum of the one or more tokens held by the pending request, the count of tokens in the target token set, and a count of tokens in the another token set.
The method of claim 1, wherein at the beginning of the cycle, the count of tokens in the target token set corresponding to the cycle is the preset value or is determined based on the count of tokens in the another token set corresponding to a previous cycle of the cycle.
The method of claim 1, wherein at the beginning of the cycle, the count of tokens in the another token set corresponding to the cycle is determined based on the count of tokens in the target token set corresponding to a previous cycle of the cycle.
The method of claim 1, further comprising:

for the any cycle, in response to the pending request being processed within another cycle, reclaiming the one or more tokens held by the pending request and adding the one or more tokens to the target token set corresponding to the another cycle.
The method of claim 1, further comprising:

in response to the count of tokens in the target token set corresponding to the cycle being 0, performing a limiting processing on the pending request.
The method of claim 5, wherein the limiting processing includes:

adding the pending request to a request waiting queue, to facilitate subsequent obtaining of the pending request from the request waiting queue.
The method of claim 5, wherein the limiting processing includes:

in response to a request storage quantity in the request waiting queue meeting a limiting condition, denying the pending request.
The method of claim 1, wherein the obtaining a pending request includes:

obtaining the pending request from a request waiting queue, the request waiting queue including one or more unprocessed pending requests in a historical time period.
The method of claim 8, wherein the obtaining the pending request from a request waiting queue includes:

in response to the request waiting queue including a plurality of unprocessed pending requests in the historical time period, obtaining the pending request from the request waiting queue according to a request sequence of the plurality of unprocessed pending requests.
The method of claim 8, wherein the obtaining the pending request further includes:

verifying whether the pending request is valid; and

in response to determining that the pending request is invalid, denying the pending request.
The method of claim 1, further comprising:

in response to the pending request being processed in another cycle, detecting whether there is a valid pending request in the request waiting queue;

in response to a determination that there is a valid pending request in the request waiting queue, obtaining a first valid pending request in the request waiting queue, reclaiming the one or more tokens held by the pending request, and issuing the one or more tokens to the first valid pending request, to facilitate the at least one processor to process the first valid pending request; and

in response to a determination that there is no valid pending request in the request waiting queue, reclaiming the one or more tokens held by the pending request, and adding the one or more tokens to the target token set corresponding to another cycle.
A system for request traffic management, comprising:

at least one storage device storing a set of instructions; and

at least one processor in communication with the storage device, wherein when executing the set of instructions, the at least one processor is configured to cause the system to perform operations including:

for any cycle,

obtaining a pending request;

in response to a count of tokens in a target token set corresponding to the cycle being greater than 0, taking a target token from the target token set and issuing the target token to the pending request to facilitate a service system to process the pending request; and

in response to the pending request being processed within the cycle, reclaiming one or more tokens held by the pending request and adding the one or more tokens to another token set corresponding to the cycle;

wherein:

a total count of tokens in the cycle is a preset value, and

the total count of tokens is a sum of the one or more tokens held by the pending request, the count of tokens in the target token set, and a count of tokens in the another token set.
The system of claim 12, wherein at the beginning of the cycle, the count of tokens in the target token set corresponding to the cycle is the preset value or is determined based on the count of tokens in the another token set corresponding to a previous cycle of the cycle.
The system of claim 12, wherein at the beginning of the cycle, the count of tokens in the another token set corresponding to the cycle is determined based on the count of tokens in the target token set corresponding to a previous cycle of the cycle.
The system of claim 12, wherein the operations further including:

for any cycle, in response to the pending request being processed within another cycle, reclaiming the one or more tokens held by the pending request and adding the one or more tokens to the target token set corresponding to the another cycle.
The system of claim 12, wherein the operations further including:

in response to the count of tokens in the target token set corresponding to the cycle being 0, performing a limiting processing on the pending request.
The system of claim 16, wherein the limiting processing includes:

adding the pending request to a request waiting queue, to facilitate subsequent obtaining of the pending request from the request waiting queue.
The system of claim 16, wherein the limiting processing includes:

in response to a request storage quantity in the request waiting queue meeting a limiting condition, denying the pending request.
The system of claim 12, wherein the obtaining the pending request includes:

obtaining the pending request from a request waiting queue, the request waiting queue including one or more unprocessed pending requests in a historical time period.
A non-transitory computer readable medium storing instructions, the instructions, when executed by at least one processor, causing the at least one processor to implement a method comprising:

for any cycle,

obtaining a pending request;

in response to a count of tokens in a target token set corresponding to the cycle being greater than 0, taking a target token from the target token set and issuing the target token to the pending request to facilitate a service system to process the pending request; and

in response to the pending request being processed within the cycle, reclaiming one or more tokens held by the pending request and adding the one or more tokens to another token set corresponding to the cycle;

wherein:

a total count of tokens in the cycle is a preset value, and

the total count of tokens is a sum of the one or more tokens held by the pending request, the count of tokens in the target token set, and a count of tokens in the another token set.