CN113360565B

CN113360565B - Method and apparatus for flow control

Info

Publication number: CN113360565B
Application number: CN202110911853.0A
Authority: CN
Inventors: 汪龙重; 潘毅; 刘彬; 余璜
Original assignee: ANT Financial Hang Zhou Network Technology Co Ltd
Current assignee: Beijing Oceanbase Technology Co Ltd
Priority date: 2021-08-10
Filing date: 2021-08-10
Publication date: 2021-11-02
Anticipated expiration: 2041-08-10
Also published as: CN113360565A

Abstract

The embodiment of the specification discloses a method and a device for flow control, which are used for effectively improving the reliability of the flow control and the data transmission efficiency. The method comprises the following steps: acquiring the data volume of data currently issued by a producer, the current performance index information of a consumer and the storage resource usage amount required by processing the data volume; determining a target distribution object of the flow to be adjusted from distribution objects based on the data volume of the data currently issued by the producer and the current performance index information of the consumer, wherein the distribution objects comprise the producer and the consumer; determining an amount of fixed resources and an amount of available floating resources allocated to the allocation target based on the amount of storage resource usage; allocating the storage resource with the fixed resource amount to the allocation object; and allocating corresponding storage resources for the target allocation object based on the available floating resource amount.

Description

Method and apparatus for flow control

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a method and an apparatus for flow control.

Background

In a business scenario such as an Online Analytical Processing (OLAP) database, it is generally necessary to improve business Processing efficiency through Parallel Execution (Parallel Execution). In the parallel execution process, a large amount of data interaction (Exchange) is needed between different producers and consumers to Exchange data, and as a part of producer threads send more data or consumer threads process more data, the problem of Network Convey Effect (Network Convey Effect) is easily caused, and the overall execution efficiency is reduced. To avoid this problem, it is often necessary to control the transmission flow between the producer and the consumer.

The traditional flow control scheme mainly comprises a token bucket algorithm, a leaky bucket algorithm, a counter algorithm and the like. However, this kind of solution can only control the flow after the transmission flow between the producer and the consumer reaches a certain bottleneck, and still cannot guarantee the data transmission efficiency. Therefore, a more reliable and efficient flow control scheme is needed.

Disclosure of Invention

An object of the embodiments of the present specification is to provide a method and an apparatus for flow control, so as to effectively improve reliability of flow control and data transmission efficiency.

In order to achieve the above purpose, the embodiments of the present specification adopt the following technical solutions:

in a first aspect, a method for flow control is provided, including:

acquiring the data volume of data currently issued by a producer, the current performance index information of a consumer and the storage resource usage amount required by processing the data volume;

determining a target distribution object of the flow to be adjusted from distribution objects based on the data volume of the data currently issued by the producer and the current performance index information of the consumer, wherein the distribution objects comprise the producer and the consumer;

determining an amount of fixed resources and an amount of available floating resources allocated to the allocation target based on the amount of storage resource usage;

allocating the storage resource with the fixed resource amount to the allocation object;

and allocating corresponding storage resources for the target allocation object based on the available floating resource amount.

In a second aspect, there is provided an apparatus for flow control, comprising:

the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring the data volume of data currently issued by a producer, the current performance index information of a consumer and the storage resource usage amount required by processing the data volume;

a first determining unit, configured to determine a target allocation object to be adjusted in flow from allocation objects based on a data amount of data currently issued by the producer and current performance index information of the consumer, where the allocation object includes the producer and the consumer;

a second determination unit that determines the fixed resource amount and the available floating resource amount allocated to the allocation target based on the storage resource usage amount;

a first allocation unit that allocates the storage resource of the fixed resource amount to the allocation target;

and the second allocation unit allocates the corresponding storage resource to the target allocation object based on the available floating resource amount.

In a third aspect, an electronic device is provided, including:

a processor; and

a memory arranged to store computer executable instructions that, when executed, cause the processor to:

In a fourth aspect, a computer-readable storage medium is provided that stores one or more programs that, when executed by an electronic device that includes a plurality of application programs, cause the electronic device to:

According to the scheme of the embodiment of the specification, the differences of different producers and consumers in processing performance are considered, the producer and/or the consumer needing to adjust the flow are determined based on the data volume of data currently issued by the producer and the current performance index information of the consumer, the storage resource volume required by data transmission is divided into a fixed volume and a floating volume, the fixed volume of the storage resource is distributed to the producers and the consumers, and the normal operation of the data transmission between the producer and the consumer can be ensured; meanwhile, the floating amount of storage resources are distributed to the producers and/or consumers needing to adjust the flow, so that producers issuing more data and/or consumers with poor processing performance can obtain more storage resources to store data, the data issued by the producers can be quickly transmitted to the consumers, the producers are guaranteed not to be blocked when issuing the data, the effect of dynamically controlling the data flow in the data transmission process is achieved, the data transmission efficiency between the producers and the consumers is improved, and the overall throughput capacity of a business system comprising the producers and the consumers is also improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the specification and are incorporated in and constitute a part of this specification, illustrate embodiments of the specification and together with the description serve to explain the specification and not to limit the specification in a non-limiting sense. In the drawings:

fig. 1 is a schematic flow chart of a method for flow control according to an embodiment of the present disclosure;

FIG. 2A is a schematic diagram of data transmission between a producer and a consumer according to an embodiment of the present disclosure;

FIG. 2B is a schematic diagram of data transmission between a producer and a consumer according to another embodiment of the present disclosure;

FIG. 3 is a schematic structural diagram of a device for flow control according to an embodiment of the present disclosure;

fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.

Detailed Description

In order to make the objects, technical solutions and advantages of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below with reference to the specific embodiments of the present disclosure and the accompanying drawings. It is to be understood that the embodiments described are only a few embodiments of the present disclosure, and not all embodiments. All other embodiments obtained by a person skilled in the art without making any inventive step based on the embodiments in this description shall fall within the scope of protection of this document.

Description of partial concepts:

and (3) online analysis processing: the method is mainly applied to a data warehouse system, supports complex query analysis operation, and provides intuitive and understandable query results.

And (3) executing in parallel: in a business scenario such as an OLAP database, a processing statement (e.g., SQL statement) is divided into a plurality of tasks, and the tasks are processed in parallel, so as to improve the execution efficiency of the processing statement. Parallel execution is particularly effective when processing queries with large computation and long execution time, such as large table queries, multi-table join queries, and the like.

Data Transfer Layer (Data Transfer Layer): the method is an abstraction of data transmission between different threads of different machines in a database, and is used for processing data transmission.

Convoy Effect (Convey Effect): in an operating system, a short process may be behind a long process, which may result in a long average latency and a reduced utilization of a Central Processing Unit (CPU).

Network convoying Effect (Network Convey Effect): when the network sends the data, the producer sending the small amount of data needs to wait for the producer sending the large amount of data or the consumer processing the large amount of data to finish execution and then continue to operate, so that the overall execution efficiency is reduced.

As described above, in the parallel execution process, a large amount of data interaction (Exchange) is required between different producers and consumers to Exchange data, and since a part of producer threads send more data or consumer threads process more data, the problem of the network convoy effect is easily caused, which leads to a decrease in the overall execution efficiency. To avoid this problem, it is often necessary to control the transmission flow between the producer and the consumer. The traditional flow control scheme mainly comprises a token bucket algorithm, a leaky bucket algorithm, a counter algorithm and the like. However, this kind of solution can only control the flow after the transmission flow between the producer and the consumer reaches a certain bottleneck, and still cannot guarantee the data transmission efficiency.

For this reason, embodiments of the present specification aim to provide a scheme for dynamically controlling transmission flow during data transmission, which considers differences in processing performance between different producers and consumers, determines a producer and/or a consumer that needs to adjust the flow rate based on the data amount of data currently issued by the producer and the current performance index information of the consumer, and can ensure normal data transmission between the producer and the consumer by dividing the storage resource amount required for data transmission into a fixed amount and a floating amount, and allocating the fixed amount of storage resource to each producer and consumer; meanwhile, the floating amount of storage resources are distributed to the producers and/or consumers needing to adjust the flow, so that producers issuing more data and/or consumers with poor processing performance can obtain more storage resources to store data, the data issued by the producers can be quickly transmitted to the consumers, the producers are guaranteed not to be blocked when issuing the data, the effect of dynamically controlling the data flow in the data transmission process is achieved, the data transmission efficiency between the producers and the consumers is improved, and the overall throughput capacity of a business system comprising the producers and the consumers is also improved.

It should be understood that the method for flow control provided by the embodiments of the present specification may be executed by an electronic device or software installed in the electronic device, and specifically may be executed by a terminal device or a server device.

The technical solutions provided by the embodiments of the present description are described in detail below with reference to the accompanying drawings.

Referring to fig. 1, a flow chart of a method for controlling flow according to an embodiment of the present disclosure is schematically shown, where the method may include:

s102, acquiring the data volume of the data currently issued by the producer, the current performance index information of the consumer and the storage resource usage amount required for processing the data volume of the data currently issued by the producer.

The performance index information of the consumer is an index value of an index that can represent the data processing performance of the consumer. The performance indicator information of the consumer may for example include, but is not limited to, at least one of the following information: the amount of data to be processed by the consumer, and the data processing efficiency of the consumer. Specifically, the amount of data to be processed by the consumer may be determined according to the message topic subscribed by the consumer and the data published by the producer, and the data processing efficiency of the consumer may be determined based on the amount of data processed by the consumer in the historical time period. Of course, the performance index information of the consumer may also be obtained by other common manners in the art, and this is not limited in this specification.

The storage resource usage amount is used to represent the size of the storage resource required for processing the data issued by the producer, and may be specifically determined according to the data amount of the data issued by the producer. Illustratively, as shown in fig. 2A, the data currently published by the producer P1 includes data 11 to data 13, data 2 and data 3, the total amount of these data is 5, and the storage resource usage amount required for processing these data is 5.

It should be noted that the storage resource in this embodiment of the present specification may be any storage unit having a data storage function, such as a memory, and the type of the storage resource is not specifically limited in this embodiment of the present specification.

S104, determining a target distribution object of the flow to be adjusted from the distribution objects based on the data volume of the data currently issued by the producer and the current performance index information of the consumer, wherein the distribution objects comprise the producer and the consumer.

For the reason that both a producer with a large published data amount and a consumer with poor processing performance may cause a network convoying effect, in order to improve the data transmission efficiency between the producer and the consumer, the producer with the large published data amount can rapidly publish data and the consumer with the poor processing performance can rapidly receive data from the producer, a target distribution object of the flow to be adjusted can be determined from the producer and the consumer based on the data amount of the data currently published by the producer and the current performance index information of the consumer.

Specifically, as an alternative, the above S104 may include the following steps: determining a producer with the data volume of the currently issued data exceeding a first data volume threshold value as a target distribution object of the flow to be adjusted; and determining the consumers with the data volume to be processed exceeding the second data volume threshold or the data processing efficiency being less than the processing efficiency threshold as target distribution objects of the flow to be adjusted. The first data volume threshold, the second data volume threshold, and the processing efficiency threshold may be preset according to actual needs, and specific numerical values are not limited in the embodiments of this specification.

It can be understood that, by the above manner, a producer who issues data at present with a large data volume and a consumer who waits for processing with a small data volume or has low processing efficiency are used as target allocation objects for adjusting the flow volume, and differences between different producers and consumers in terms of processing performance are considered, so that storage resources are allocated based on the determined target allocation objects subsequently, and it can be ensured that the producers and consumers which are easy to cause the network convoy effect can be allocated to storage resources with a proper size, so that a certain storage space can be provided for data of a large data volume issued by the producer, and it is ensured that the producer who issues a large data volume can issue data quickly, and the network convoy effect caused by the producer who issues a small data volume waiting for issuing data of a large data volume is avoided; and moreover, a certain storage space can be provided for the consumers with more data to be processed or low processing efficiency to cache the received data, so that the consumers can receive more data from the producer, the data release of the producer and the data reception of other consumers are prevented from being blocked by the consumers, and the network navigation protection effect caused by the fact that the producer waits for the consumers to process the data is avoided. Therefore, the data transmission efficiency between the producer and the consumer is further improved, and the overall throughput capacity of the business system including the producer and the consumer is also improved.

S106, based on the storage resource usage, the fixed resource amount and the available floating resource amount allocated to the allocation object are determined.

As an alternative, S106 may include the steps of:

step A1, based on the occupation ratio of the target allocation object in the allocation object, determines the maximum available total floating resource amount allocated to the target allocation object.

The occupation ratio of the target distribution objects in the distribution objects is the ratio between the number of the target distribution objects and the number of the distribution objects.

Considering that the larger the occupation ratio of the target allocation object in the allocation object is, the more the target allocation object occupies more storage resources, and further, the smaller the available floating resource amount that can be allocated to a single target allocation object is, in order to ensure that the resource amount of the storage resources allocated to the target allocation object is reasonable, the available resource coefficient may be determined based on the occupation ratio of the target allocation object in the allocation object, and then the product of the storage resource usage amount and the available resource coefficient is determined as the maximum available floating resource total amount. Wherein, the larger the ratio is, the smaller the available resource coefficient is; the smaller the duty ratio, the larger the available resource coefficient.

Alternatively, the available resource coefficient may be

Wherein dfc _ cnt indicates the number of target allocation objects, and total _ cnt indicates the number of allocation objects. Further, to ensure smooth variation of the available resource coefficientsFurther, the amount of the storage resource allocated to the target allocation object is ensured to be reasonable, and the available resource coefficient can be

。

Step a2, determining the amount of available floating resources allocated to a single allocation object based on the maximum total amount of available floating resources and the number of allocation objects.

Specifically, the ratio between the maximum available total floating resource amount and the number of allocation objects may be determined as the available floating resource amount allocated to a single allocation object. More specifically, the amount of available floating resources allocated to a single allocation object is shown in the following equation (1):

（1）

wherein the content of the first and second substances,

representing the amount of available floating resources allocated to a single allocation object;

representing the maximum total amount of available floating resources.

Step a3, determining the fixed resource amount allocated to a single allocation object based on the difference between the storage resource usage amount and the maximum available total floating resource amount and the number of allocation objects.

Specifically, the ratio between the above difference value and the number of allocation objects may be determined as a fixed resource amount allocated to a single allocation object.

It can be understood that, through the above manner, the maximum available total floating resource amount is determined according to the proportion of the target allocation object in the allocation object, and then the fixed resource amount allocated to a single allocation object is determined according to the maximum available total floating resource amount and the number of the allocation objects, so that it can be ensured that the resource amount of the storage resource allocated to each allocation object is reasonable when resource allocation is subsequently performed, and the processing requirement of the allocation object can be met. In addition, because the data volume of the data issued by the producer and the performance index information of the consumer are dynamically changed, the occupation ratio of the target distribution object in the distribution object is also dynamically changed, and furthermore, the determined fixed resource quantity and the available floating resource quantity are changed along with the change of the difference between the producer and the consumer in the aspect of processing performance, so that the flow control requirement in the data transmission process can be better adapted.

Alternatively, the amount of available resources allocated to a single allocation object may be adjusted by setting a ratio between the fixed amount of resources and the floating amount of resources. Specifically, the step S106 may include the following steps:

step B1, determining the total amount of fixed resources allocated to the allocation object based on the storage resource usage amount.

Wherein the total amount of the fixed resources is less than the usage amount of the storage resources. For example, any value less than the storage resource usage may be determined as the fixed total amount of resources.

Step B2, based on the total amount of fixed resources and the number of allocation objects, determines the amount of fixed resources allocated to a single allocation object.

Specifically, the ratio between the total amount of fixed resources and the number of allocation objects may be determined as the amount of fixed resources allocated to a single allocation object.

Step B3, determining the amount of available floating resources allocated to a single allocation object based on the amount of fixed resources and a predetermined ratio between the amount of fixed resources and the amount of floating resources.

It should be noted that the predetermined ratio may be set according to actual needs, and this is not limited in the embodiments of the present specification. For example, the predetermined ratio may be set to 1:2, and if the fixed resource amount determined through the above steps B1 through B2 is 1 unit capacity of storage resources, it may be determined that the floating resource amount is 2 unit capacity of storage resources.

It can be understood that the proportion between the fixed resource amount and the floating resource amount is set to adjust the available resource amount allocated to a single allocation object, so that the logic is simple to realize, and the data transmission efficiency between a producer and a consumer can be further improved.

As still another alternative, a certain buffer ratio may be set for each target allocation object of the flow rate to be adjusted to adjust the amount of storage resources allocated to the target allocation object. Specifically, the step S106 may include the steps of:

step C1, based on the storage resource usage amount, determines the total amount of fixed resources allocated to the allocation object.

Step C2, based on the total amount of fixed resources and the number of allocation objects, determines the amount of fixed resources allocated to a single allocation object.

And step C3, aiming at the single target allocation object, determining the available floating resource amount allocated to the target allocation object based on the buffer proportion and the fixed resource amount corresponding to the target allocation object.

The buffer ratio may be a ratio of an available floating resource amount to a fixed resource amount, which may be specifically set according to actual needs, and the buffer ratios corresponding to different target allocation objects may be the same or may also be different, which is not limited in this specification.

Optionally, if the target allocation object is a producer, determining a buffer proportion corresponding to the producer based on the data amount of the data currently issued by the producer. For example, if the data volume of the data issued by the producer as the target allocation object is larger, the buffer proportion corresponding to the producer can be set to a larger value, so that it can be ensured that enough storage space is provided for the producer issuing more data to buffer the issued data, the producer can issue the data quickly, and the network convoy effect caused by long-time waiting of other producers is avoided. Otherwise, the buffer ratio corresponding to the producer can be set to a smaller value, so that the waste of resources caused by excessive storage resources allocated to the producer is avoided.

And if the target distribution object is a consumer, determining the corresponding buffer proportion of the consumer based on the current performance index information of the consumer. For example, if the amount of data to be processed by a consumer serving as a target allocation object is larger or the data processing efficiency is lower, the buffer ratio corresponding to the consumer may be set to a larger value to ensure that a sufficient storage space is provided for the consumer with poor processing performance to buffer the received data, so that the consumer can receive more data from the producer, and further the producer can rapidly issue the data, thereby avoiding blocking data issuance by the producer and data processing by other consumers, and avoiding a network convoy effect caused by the producer waiting for the consumer to process the data. Therefore, the data transmission efficiency between the producer and the consumer is further improved, and the overall throughput capacity of the business system including the producer and the consumer is also improved. Otherwise, the buffer ratio corresponding to the consumer can be set to a smaller value, so that the resource waste caused by excessive storage resources allocated to the consumer is avoided.

In order to further avoid the resource waste caused by too much storage resources allocated to the target allocation object, after S106, the method of the embodiment of the present specification may further include: if the ratio of the amount of available floating resources to the amount of storage resource usage exceeds a ratio threshold, the amount of available floating resources is updated based on the ratio threshold. Specifically, the product of the ratio threshold and the storage resource usage may be determined as the new amount of available floating resources.

It should be noted that, in this embodiment of the present specification, the ratio threshold may be set according to actual needs, for example, the ratio threshold may be set to 1/8, which is not limited in this embodiment of the present specification.

S108, allocating storage resources with fixed resource quantity for the allocation object.

And S110, distributing corresponding storage resources for the target distribution object based on the available floating resource amount.

For example, as shown in fig. 2A and 2B, assuming that 1 unit capacity of storage resources is used for storing one piece of data, the data to be currently released by the producer P1 includes data 11 to 13, data 2, and data 3, where the data 11 to 13 are data to be processed by the consumer C1, the data 2 is data to be processed by the consumer C2, and the data 3 is data to be processed by the consumer C3. By determining the fixed resource amount and the available floating resource amount allocated to each allocation object to be 1 and 2, respectively, and the determined target allocation object to be the consumer C1 through the above steps, it is possible to allocate 1 unit capacity of storage resources to the producer P1 and the consumers C1 to C3, respectively, and further allocate 2 unit capacity of storage resources to the consumer C1. Thereby, the consumer C1 is enabled to receive the data 11 to 13 at once without causing a jam to the data 2 and the data 3 to be issued by the producer, so that at the same time, the consumer C2 is enabled to receive and process the data 2 and the consumer C3 is enabled to receive and process the data 3. Thus, the producer P1, the consumer C1 and the consumer C2 can finish the processing quickly and release more resources to execute a new data processing operation, thereby improving the data transmission efficiency between the producer P1 and each consumer and improving the throughput of the whole business system including the producer and the consumer.

It should be noted that the above method for controlling flow of use case is executed in real time, that is, by monitoring the data volume of data currently issued by a producer, the current performance index information of a consumer, and the storage resource usage amount required for processing the currently issued data volume according to a predetermined time interval, determining a target allocation object of the flow to be adjusted, and a fixed resource amount and a floating resource amount allocated to a single allocation object in real time according to the monitoring result, then allocating a storage resource of the fixed resource amount to each allocation object, and further allocating a corresponding storage resource to the target allocation object based on the floating resource amount, thereby implementing dynamic adaptive flow control. The time interval may be set according to actual needs, for example, the time interval may be set to a small value.

According to the method for flow control provided by the embodiment of the specification, differences of different producers and consumers in processing performance are considered, the producer and/or the consumer needing to adjust the flow are determined based on the data volume of data currently issued by the producer and the current performance index information of the consumer, the storage resource volume required by data transmission is divided into a fixed volume and a floating volume, the fixed volume of storage resources is distributed to the producer and the consumer, and normal data transmission between the producer and the consumer can be ensured; meanwhile, the floating amount of storage resources are distributed to the producers and/or consumers needing to adjust the flow, so that producers issuing more data and/or consumers with poor processing performance can obtain more storage resources to store data, the data issued by the producers can be quickly transmitted to the consumers, the producers are guaranteed not to be blocked when issuing the data, the effect of dynamically controlling the data flow in the data transmission process is achieved, the data transmission efficiency between the producers and the consumers is improved, and the overall throughput capacity of a business system comprising the producers and the consumers is also improved.

It should be noted that the method for flow control provided in the embodiment of the present specification may also be used in a scenario of controlling any computer resource, such as a CPU, a network broadband, and the like, and this is not specifically limited in the embodiment of the present specification.

The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

In addition, corresponding to the method for controlling flow rate shown in fig. 1, an embodiment of the present specification further provides a device for controlling flow rate. Fig. 3 is a schematic structural diagram of a device 300 for controlling flow according to an embodiment of the present disclosure, including:

an obtaining unit 310, configured to obtain a data size of data currently issued by a producer, current performance index information of a consumer, and a storage resource usage amount required for processing the data size;

a first determining unit 320, configured to determine a target allocation object to be adjusted in flow from allocation objects based on a data amount of data currently issued by the producer and current performance index information of the consumer, where the allocation object includes the producer and the consumer;

a second determining unit 330 that determines the fixed resource amount and the available floating resource amount allocated to the allocation object based on the storage resource usage amount;

a first allocation unit 340, configured to allocate the storage resource of the fixed resource amount to the allocation target;

and a second allocating unit 350, configured to allocate corresponding storage resources to the target allocation object based on the available floating resource amount.

Optionally, the second determining unit 330 includes:

the maximum floating amount determining subunit determines the maximum available floating resource total amount based on the proportion of the target distribution object in the distribution object;

a first available float amount determination subunit that determines an amount of available float resources to be allocated to a single allocation object based on the maximum available float resource total amount and the number of allocation objects;

and a first fixed quantity determining subunit which determines the fixed resource quantity allocated to the single allocation object based on the difference value between the storage resource usage quantity and the maximum available floating resource total quantity and the quantity of the allocation objects.

Optionally, the maximum floating amount determining subunit determines an available resource coefficient based on a square of the ratio, and determines a product of the storage resource usage amount and the available resource coefficient as a maximum total amount of available floating resources.

Optionally, the second determining unit 330 includes:

a first fixed total amount determination subunit that determines a fixed resource total amount allocated to the allocation target based on the storage resource usage amount, the fixed resource total amount being smaller than the storage resource usage amount;

a second fixed amount determination subunit that determines the amount of fixed resources allocated to a single allocation object based on the total amount of fixed resources and the number of allocation objects;

and a second available floating amount determination subunit that determines an amount of floating resources available to be allocated to the single allocation object based on the fixed resource amount and a predetermined ratio between the fixed resource amount and the floating resource amount.

Optionally, the second determining unit 330 includes:

a second fixed total amount determination subunit that determines a fixed resource total amount allocated to the allocation target based on the storage resource usage amount, the fixed resource total amount being smaller than the storage resource usage amount;

a third fixed amount determination subunit that determines the amount of fixed resources allocated to a single allocation object based on the total amount of fixed resources and the number of allocation objects;

and a third available floating amount determining subunit, configured to determine, for a single target allocation object, an available floating resource amount allocated to the target allocation object based on the buffer proportion corresponding to the target allocation object and the fixed resource amount.

Optionally, the second determining unit 330 further includes:

a first proportion determining subunit, configured to determine, if the target allocation object is a producer, a buffer proportion corresponding to the producer based on a data amount of data currently issued by the producer;

and the second proportion determining subunit determines the buffer proportion corresponding to the consumer based on the current performance index information of the consumer if the target distribution object is the consumer.

Optionally, the apparatus 300 further comprises:

an updating unit configured to update the available floating resource amount based on a ratio threshold if the ratio of the available floating resource amount to the storage resource usage amount exceeds the ratio threshold after the second determining unit 330 determines the available floating resource amount allocated to the allocation object.

Optionally, the performance index information includes a data amount to be processed and/or a data processing efficiency;

the first determining unit 320 includes:

the first target distribution object determining subunit determines the producer whose data volume of the currently issued data exceeds the first data volume threshold value as the target distribution object of the flow to be adjusted;

and the second target distribution object determining subunit determines the consumers with the data volume to be processed exceeding the second data volume threshold or the data processing efficiency being less than the processing efficiency threshold as the target distribution objects of the flow volume to be adjusted.

The device for flow control provided by the embodiment of the present specification considers the differences in processing performance between different producers and consumers, determines a producer and/or a consumer that needs to adjust the flow rate based on the data volume of data currently issued by the producer and the current performance index information of the consumer, and allocates a fixed amount of storage resources to each producer and consumer by dividing the amount of storage resources required for data transmission into a fixed amount and a floating amount, so as to ensure the normal operation of data transmission between the producer and the consumer; meanwhile, the floating amount of storage resources are distributed to the producers and/or consumers needing to adjust the flow, so that producers issuing more data and/or consumers with poor processing performance can obtain more storage resources to store data, the data issued by the producers can be quickly transmitted to the consumers, the producers are guaranteed not to be blocked when issuing the data, the effect of dynamically controlling the data flow in the data transmission process is achieved, the data transmission efficiency between the producers and the consumers is improved, and the overall throughput capacity of a business system comprising the producers and the consumers is also improved. Obviously, the apparatus for flow control according to the embodiment of the present specification may be used as an execution subject of the method for flow control shown in fig. 1, and thus can implement the function of the method for flow control implemented in fig. 1. Since the principle is the same, it is not described herein again.

Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. Referring to fig. 4, at a hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory, such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.

The processor, the network interface, and the memory may be connected to each other via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 4, but that does not indicate only one bus or one type of bus.

And the memory is used for storing programs. In particular, the program may include program code comprising computer operating instructions. The memory may include both memory and non-volatile storage and provides instructions and data to the processor.

The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a means for flow control on a logical level. The processor is used for executing the program stored in the memory and is specifically used for executing the following operations:

The method performed by the apparatus for flow control according to the embodiment shown in fig. 1 of the present specification may be applied to or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present specification may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present specification may be embodied directly in a hardware decoding processor, or in a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.

It should be understood that the electronic device of the embodiments of the present description can implement the functions of the apparatus for flow control in the embodiment shown in fig. 1. Since the principle is the same, the embodiments of the present description are not described herein again.

Of course, besides the software implementation, the electronic device in this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or logic devices.

Embodiments of the present specification also propose a computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a portable electronic device comprising a plurality of application programs, are capable of causing the portable electronic device to perform the method of the embodiment shown in fig. 1, and in particular to perform the following:

In short, the above description is only a preferred embodiment of the present disclosure, and is not intended to limit the scope of the present disclosure. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present specification shall be included in the protection scope of the present specification.

The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

Claims

1. A method for flow control, comprising:

determining a target distribution object of the to-be-regulated flow from distribution objects based on the data volume of the data currently issued by the producer and the current performance index information of the consumer, wherein the performance index information comprises the data volume to be processed and/or the data processing efficiency, the distribution objects comprise the producer and the consumer, and the target distribution object comprises at least one of the following distribution objects: the data processing method comprises the steps that a producer of currently issued data with the data volume exceeding a first data volume threshold, a consumer of which the data volume to be processed exceeds a second data volume threshold and a consumer of which the data processing efficiency is smaller than a processing efficiency threshold are provided;

2. The method of claim 1, wherein determining the fixed amount of resources and the available amount of floating resources allocated to the allocation object based on the storage resource usage amount comprises:

determining the maximum available total floating resource amount based on the proportion of the target distribution object in the distribution object;

determining an amount of available floating resources allocated to a single allocation object based on the maximum total amount of available floating resources and the number of allocation objects;

determining a fixed amount of resources allocated to a single allocation object based on a difference between the amount of storage resource usage and the maximum total amount of available floating resources and the number of allocation objects.

3. The method of claim 2, wherein determining the maximum total amount of available floating resources based on the proportion of the target allocation object in the allocation object comprises:

determining an available resource coefficient based on a square of the occupancy;

and determining the product of the storage resource usage and the available resource coefficient as the maximum available total floating resource amount.

4. The method of claim 1, wherein determining the fixed amount of resources and the available amount of floating resources allocated to the allocation object based on the storage resource usage amount comprises:

determining a total amount of fixed resources allocated to the allocation object based on the storage resource usage amount, the total amount of fixed resources being less than the storage resource usage amount;

determining the amount of the fixed resources allocated to a single allocation object based on the total amount of the fixed resources and the number of the allocation objects;

an amount of available floating resources allocated to a single allocation object is determined based on the amount of fixed resources and a predetermined ratio between the amount of fixed resources and the amount of floating resources.

5. The method of claim 1, wherein determining the fixed amount of resources and the available amount of floating resources allocated to the allocation object based on the storage resource usage amount comprises:

and for a single target allocation object, determining the available floating resource amount allocated to the target allocation object based on the buffer proportion corresponding to the target allocation object and the fixed resource amount.

6. The method of claim 5, further comprising, prior to determining the amount of available floating resources allocated to the target allocation object based on the corresponding buffer fraction of the target allocation object and the fixed amount of resources:

if the target distribution object is a producer, determining a buffer proportion corresponding to the producer based on the data volume of the data currently issued by the producer;

and if the target distribution object is a consumer, determining the corresponding buffer proportion of the consumer based on the current performance index information of the consumer.

7. The method of claim 1, after determining the amount of available floating resources allocated to the allocation object, the method further comprising:

updating the amount of available floating resources based on a ratio threshold if the ratio of the amount of available floating resources to the amount of storage resource usage exceeds the ratio threshold.

8. A device for flow control, comprising:

a first determining unit, configured to determine a target allocation object for traffic adjustment from allocation objects based on a data amount of data currently published by the producer and current performance index information of the consumer, where the performance index information includes a data amount to be processed and/or data processing efficiency, the allocation objects include the producer and the consumer, and the target allocation object includes at least one of the following allocation objects: the data processing method comprises the steps that a producer of currently issued data with the data volume exceeding a first data volume threshold, a consumer of which the data volume to be processed exceeds a second data volume threshold and a consumer of which the data processing efficiency is smaller than a processing efficiency threshold are provided;

9. An electronic device, comprising:

a processor; and

10. A computer-readable storage medium storing one or more programs that, when executed by an electronic device including a plurality of application programs, cause the electronic device to: