CN115664818A

CN115664818A - Independent station access flow control method and device, equipment, medium and product thereof

Info

Publication number: CN115664818A
Application number: CN202211321482.1A
Authority: CN
Inventors: 尹亮
Original assignee: Guangzhou Huanju Shidai Information Technology Co Ltd
Current assignee: Guangzhou Huanju Shidai Information Technology Co Ltd
Priority date: 2022-10-26
Filing date: 2022-10-26
Publication date: 2023-01-31

Abstract

The application relates to a method for controlling access flow of an independent station, and a device, equipment, medium and product thereof, wherein the method comprises the following steps: receiving a data query request, initializing a flow control level corresponding to the request to be the highest level in a plurality of member levels, wherein different member levels respectively indicate data source operation authorities of different data scales; judging whether the total flow requested by the current independent station in unit time exceeds an independent station flow control threshold value or not, and setting the flow control level as a second-level when the total flow requested by the current independent station in unit time exceeds the independent station flow control threshold value; recursively judging whether the total flow of the requests of the page corresponding to the requests in unit time exceeds a page flow control threshold value or not, and when the total flow exceeds the page flow control threshold value, reducing the flow control level by one level when the flow control level does not reach the lowest level; and judging whether the flow control level obtained by the data query request reaches a level threshold required by the data source retrieval service corresponding to the request, and when the flow control level is reached, executing a data query task by the data source retrieval service according to the flow control level. The application improves the stability of the independent station by implementing flow control.

Description

Independent station access flow control method and device, equipment, medium and product thereof

Technical Field

The present application relates to network security technologies, and in particular, to a method, an apparatus, a medium, and a product for controlling access flow of an independent station.

Background

The e-commerce platform based on the independent station needs to provide network access related security services for each independent station, wherein the security services include that the independent station is prevented from being attacked by saturation or illegally accessed, the independent station can be ensured to operate more robustly under the support of the platform, and a user can obtain good operation experience when accessing the independent station.

In daily life, the problems that the independent station is frequently encountered and causes that the user cannot normally access the independent station include, besides malicious attack events, slow response caused by page grabbing by a crawler technology, single-page access traffic surge in scenes such as killing seconds, partial or all of data sources such as databases for storing commodity information being unavailable, and the like, which belong to the problems of generalized network security, and an effective prevention and control means needs to be provided so as to ensure the healthy operation of the independent station.

The traditional prevention and control mode does not consider the characteristic of relatively isolated maintenance of each independent station, focuses on the safety maintenance of data source retrieval service corresponding to a data source, and only performs proper control when the flow of the data source retrieval service is too large.

Disclosure of Invention

The present application aims to solve the above problems and provide a method for controlling access flow of an independent station, and a corresponding apparatus, device, non-volatile readable storage medium, and computer program product.

According to an aspect of the present application, there is provided an access flow control method of an independent station, including the steps of:

receiving a data query request, initializing a flow control level corresponding to the request to be the highest level in a plurality of member levels, wherein different member levels respectively indicate data source operation authorities of different data scales;

executing a flow control task of the independent station, judging whether the total flow requested by the current independent station in unit time exceeds a flow control threshold of the independent station, and setting the flow control level as the next lower level in the plurality of member levels when the total flow requested by the current independent station in unit time exceeds the flow control threshold of the independent station;

executing a page flow control task, recursively judging whether the total flow of unit time requests of a page corresponding to the data query request exceeds a page flow control threshold value, and when the total flow exceeds the page flow control threshold value, descending the flow control level by one level when the flow control level does not reach the lowest level;

and executing a data source flow control task, judging whether the flow control level obtained by the data query request reaches a level threshold required by the data source retrieval service corresponding to the request, and controlling the data source retrieval service to execute the data query task on the corresponding data source according to the flow control level when the flow control level obtained by the data query request reaches the level threshold required by the data source retrieval service corresponding to the request.

Optionally, executing the flow control task of the independent station, determining whether a total flow requested by the current independent station in unit time exceeds a flow control threshold of the independent station, and if so, setting the flow control level as a lower level of the multiple member levels, including:

when the total flow requested in unit time of the current independent station exceeds the flow control threshold of the independent station, triggering a corresponding flow control event of the independent station;

in response to the independent station flow control event, setting the flow control level to be the next lower level in the plurality of member levels, wherein the next lower level is used for indicating the data source operation authority which is endowed to the data query request with the minimum data size;

responding to the independent station flow control event, skipping the step of executing the page flow control task and entering the step of executing the data source flow control task;

and when the total flow requested in unit time of the current independent station does not exceed the flow control threshold of the independent station, the flow control level is not modified, and the step of executing the page flow control task is sequentially carried out.

Optionally, executing a page flow control task, and recursively determining whether a total requested flow per unit time of a page corresponding to the data query request exceeds a page flow control threshold, including:

judging whether the total requested flow per unit time of the page corresponding to the data query request exceeds a page flow control threshold, and triggering a corresponding page flow control event when the total requested flow per unit time of the page corresponding to the data query request exceeds the page flow control threshold;

responding to the page flow control event, judging whether the flow control level is the lowest level in the plurality of member levels, and when the flow control level is not the lowest level, descending the flow control level by one level;

and iterating the processes until the flow control level is the lowest level, responding to the data service request and returning a null value, and terminating the execution of the subsequent steps, or until the data service request is asynchronously processed by the step of executing the data source flow control task.

Optionally, executing a data source flow control task, and determining whether a flow control level obtained by the data query request reaches a level threshold required by a data source retrieval service corresponding to the request, includes:

when the flow control level obtained by the data query request reaches a level threshold required by the data source retrieval service corresponding to the request, triggering a data query event;

responding to the data query event, judging whether the deviation proportion of the average request time delay of the corresponding data source retrieval service relative to a set value reaches a preset threshold value, and controlling the data source retrieval service to execute a data query task based on local data content query on the corresponding data source when the deviation proportion reaches the preset threshold value;

and when the preset threshold value is not reached, controlling the data source retrieval service to execute a data query task based on all data content queries on the corresponding data source.

Optionally, when the preset threshold is not reached, controlling the data source retrieval service to execute a data query task based on a query of all data contents on a corresponding data source, including:

judging whether the total flow requested by the corresponding data source retrieval service in unit time reaches a set threshold or throws out an exception, and triggering a fusing event when the total flow reaches the set threshold or throws out the exception;

responding to the fusing event, controlling the data source retrieval service to execute a data query task based on local data content query on a corresponding data source;

and when the fusing event is not triggered, controlling the data source retrieval service to execute a data query task based on the whole data content query on the corresponding data source.

Optionally, after executing the data source flow control task, the method includes:

judging whether the flow control level obtained by the data query request reaches a level threshold required by the data sorting service corresponding to the request, and triggering a sorting event when the level threshold is reached;

ordering result data obtained after the data source retrieval service executes a data query task in response to the ordering event to obtain an ordering result;

and responding the data query request with the sequencing result.

Optionally, the flow control level includes the following member levels:

the highest level is used for indicating that the data source operation authority is shared in the scale range of the whole data content;

a secondary high level for indicating that the data source operation right is enjoyed within the scale range of the local data content;

a second lowest level indicating that the data source operation authority is enjoyed within a range set smaller than the scale of the local data content;

and the lowest level is used for indicating that the data source operation authority is not shared by any data content.

According to another aspect of the present application, there is provided an independent station access flow control apparatus including:

the initialization module is used for receiving a data query request, initializing the flow control level corresponding to the request to be the highest level in a plurality of member levels, and indicating the data source operation authority of different data scales by different member levels;

the station flow control module is configured to execute a flow control task of the independent station, judge whether the total flow requested by the current independent station in unit time exceeds a flow control threshold of the independent station, and set the flow control level as the lower level of the plurality of member levels when the total flow exceeds the flow control threshold of the independent station;

the page flow control module is set to execute a page flow control task, recursively judges whether the total flow of unit time requests of a page corresponding to the data query request exceeds a page flow control threshold value or not, and when the total flow exceeds the page flow control threshold value, the flow control level is reduced by one level when the flow control level does not reach the lowest level;

and the retrieval flow control module is set to execute a data source flow control task, judges whether the flow control level obtained by the data query request reaches a level threshold required by the data source retrieval service corresponding to the request, and controls the data source retrieval service to execute the data query task on the corresponding data source according to the flow control level when the flow control level is reached.

According to another aspect of the present application, there is provided a standalone station access flow control apparatus, comprising a central processing unit and a memory, the central processing unit being configured to invoke execution of a computer program stored in the memory to perform the steps of the standalone station access flow control method described herein.

According to another aspect of the present application, there is provided a non-transitory readable storage medium storing a computer program implemented according to the method for controlling access flow of an individual station in the form of computer readable instructions, the computer program, when invoked by a computer, performing the steps included in the method.

According to another aspect of the present application, there is provided a computer program product comprising computer programs/instructions which, when executed by a processor, implement the steps of the method described in any one of the embodiments of the present application.

Compared with the prior art, the method has the advantages that a flow control level is initialized for the independent station after a data query request of the independent station arrives, the highest level of a plurality of member levels is kept, then whether the flow control level is adjusted or not can be determined according to the total unit time request flow of the independent station on the whole station level, the page level corresponding to the data query request can be specifically selected, the flow control level is adjusted according to the total unit time request flow condition of the page, and finally the data source flow control task execution process is entered, wherein the corresponding data source retrieval service is controlled to execute the data query task according to the flow control level.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic diagram of a network architecture used in an exemplary application scenario for implementing the method for controlling access flow of an independent station according to the present application;

fig. 2 is a schematic flowchart of an embodiment of a method for controlling access flow of an independent station according to the present application;

fig. 3 is a schematic flowchart of executing a flow control task of an independent station in an embodiment of the present application;

fig. 4 is a schematic flowchart illustrating a process of executing a page flow control task in an embodiment of the present application;

fig. 5 is a schematic flowchart of executing a retrieve flow control task in an embodiment of the present application;

FIG. 6 is a flow chart illustrating a fusing mechanism implemented in the embodiments of the present application;

FIG. 7 is a flow diagram illustrating an implementation ordering mechanism in an embodiment of the present application;

FIG. 8 is a functional block diagram of the independent station access flow control apparatus of the present application;

fig. 9 is a schematic diagram of a structure of an access flow control apparatus for a stand-alone station used in the present application.

Detailed Description

Referring to fig. 1, an exemplary network architecture adopted by the application scenario of the present application includes a centralized control server 80, a data server 81, and a plurality of independent stations 82, where the centralized control server 80 may be configured to deploy a computer program product implemented by programming according to the independent station access flow control method of the present application, so as to run the flow control service of the present application. The data server 81 may be used to deploy one or more data sources of the present application. The individual stations also operate in respective servers, each typically for operating an online store, each individual station may be equipped with a corresponding plug-in to the flow control service to enable invocation of the flow control service, and the centralized control server 80 may perform access to data sources in the data server 81 by executing a data retrieval service.

In some embodiments, the flow control service may run directly in each of the individual stations, and directly access the data sources in the data server 81. In some other embodiments, the flow control service may also be run in each of the data servers, and provide a corresponding interface for each independent station to call. As such, it can be seen that the flow control service of the present application can operate in a variety of ways to provide flow control services for each individual station.

The following description will be given taking as an example the operation of the independent station after programming the access flow control method according to the present application into a computer program product.

Referring to fig. 1, a method for controlling access flow of an independent station according to the present application, in one embodiment, includes the following steps:

step S1100, receiving a data query request, initializing a flow control level corresponding to the request to be the highest level of a plurality of member levels, wherein different member levels respectively indicate data source operation authorities of different data scales;

when an independent station receives a data query request, under the default condition, the independent station firstly carries out initialization operation on the data query request, and in the initialization operation, the flow control level Leve l of the data query request is set as the highest level Leve l _0.

The flow control level may include a plurality of member levels, each of which indicates the operation authority of the data source with different data sizes, for example, in an embodiment, four member levels may be set as follows:

a highest level Leve l _3, configured to indicate that the data source operation authority is enjoyed within a scale range of all data contents, and accordingly, all logics corresponding to the data access service are implemented by an independent station for each data service request, where the data access service includes a data recall stage and a result processing stage, where the recall stage mainly performs a coarse recall process of the data access service, a multi-channel recall process may be implemented in the coarse recall process, and the result processing stage may be used to integrate data of the coarse recall, for example, sort the data, form a report, and so on;

the secondary high-level leave _2 is used for indicating that the local data content is entitled to the data source operation authority within the scale range, and correspondingly, the logic corresponding to the corresponding data access service can be adjusted to only realize partial logic, for example, only a part of recall channels are used in the course of rough recall, or only a part of data slices are recalled, and the like, and the business logic corresponding to sorting is not executed;

and a lower level Leve l _1 for indicating that the data source operation authority is shared within a set range smaller than the size of the local data content, and correspondingly, when providing the data access service for the data query request, providing the data access service according to a preset minimum data content size, so that the data source which can be recalled only contains a range smaller than the local data content, even if the data query request can only be queried within a range of the specified smaller-scale data source.

And the lowest level Leve l _0 is used for indicating that no data content is entitled with data source operation authority, correspondingly, the service logic of the data access service can not be skipped for the data query request, no operation is executed, and the data query request is directly responded by adopting a null value.

In other embodiments, a plurality of member levels with different numbers from those in the above example may also be set for the flow control level according to actual needs, and as long as a corresponding relationship is established between each member level and a specific service logic of the data access service, so that data contents relied on by each member level are different in scale and present a certain gradient relationship. In this regard, one skilled in the art will be able to devise various arrangements that are consistent with the principles disclosed above.

Step S1200, executing a flow control task of the independent station, judging whether the total flow requested by the current independent station in unit time exceeds a flow control threshold of the independent station, and setting the flow control level as the lower level of the plurality of member levels when the total flow requested by the current independent station in unit time exceeds the flow control threshold of the independent station;

the step is mainly used for realizing the flow control of the independent station layer, so that the flow control task of the independent station is correspondingly executed. In the flow control task of the independent station, firstly, whether the total flow of requests per unit time (QPS) of the current independent station exceeds a flow control threshold preset for the current independent station is judged, and when the flow control threshold is exceeded, it is indicated that the current independent station is not suitable to respond to the data query request any more, so that the flow control level corresponding to the data query request needs to be lowered, and the flow control level is directly set from its original level to a lower level, and the lower level is usually constrained to correspond to a minimum data content size, so that a front-end protection mechanism can be formed for the total station of the independent station, especially for the back-end service thereof. The mechanism is used for resisting saturation attacks of a large number of data query requests at the same time, and a first safety barrier of the independent station is formed.

In some embodiments, the individual station flow control threshold may be a measured threshold or an empirical threshold, which may be set by one skilled in the art as desired.

After the flow control level is set to the lower level of the plurality of member levels, step S1300 may be performed to further implement flow control based on the page corresponding to the data query request, however, in a preferred embodiment, it is considered that the total flow rate per unit time of the independent station has reached a higher state, and the flow control level of the data query request has been reduced to the lower level corresponding to the minimum data content size, and thus step S1400 may be performed directly in excess of step S1300, so that a traffic link may be reduced by reducing the service level of the data query request, and unnecessary overhead of the independent station may be avoided from being aggravated.

If the total flow rate requested by the current individual station per unit time does not exceed the flow control threshold of the individual station, it indicates that the current individual station is still in a state of robust operation, so that the data query request can be fully served with data access service, and therefore, step S1300 can be directly continued.

Step S1300, executing a page flow control task, recursively judging whether the total flow of unit time requests of a page corresponding to the data query request exceeds a page flow control threshold value, and when the total flow of unit time requests of the page corresponding to the data query request exceeds the page flow control threshold value, reducing the flow control level by one level when the flow control level does not reach the lowest level;

this step is mainly used to implement flow control at the page level, and therefore, it is determined that the corresponding page is first determined according to the data query request, and specifically, the corresponding page can be obtained from the requested website, and then it is determined whether the total flow of the unit time request of the page exceeds a page flow control threshold, when the total flow of the unit time request of the page exceeds the page flow control threshold, it is indicated that the corresponding page is being accessed at high frequency, and the existing access flow will cause unstable access to the page and even the independent station, and therefore the flow control level corresponding to the data query request needs to be lowered, for example, if the flow control level of the data query request is the highest level, the flow control level is set to the next-highest level after the flow control threshold is determined to be exceeded by the above-mentioned detection; if the flow control level of the data query request is the next lower level, and if the flow control level of the data query request is degraded again, the flow control level is set to be the lowest level.

In some embodiments, the page flow control threshold may be a measured threshold or an empirical threshold, which may be set by one skilled in the art as desired.

In some embodiments, the data query request is added to an asynchronous queue, and the data source flow control task is executed in step S1400 when it is dequeued from the asynchronous queue for consumption. During the queuing, the step S1300 may be executed iteratively, and through iteration, it is detected whether the total requested flow per unit time of the page exceeds the page flow control threshold, and when it is determined in each iteration that the total requested flow per unit time exceeds the page flow control threshold, the flow control level of the data query request is reduced by one level, so as to implement recursive adjustment of the flow control level.

In some embodiments, when the total flow requested by the page per unit time does not exceed the page flow control threshold through multiple recursive detections, if the flow control level does not reach the highest level at this time, the flow control level may be further incremented by one level to enhance the elasticity of the page-based flow control mechanism.

It can be seen that, in combination with this step S1300, the flow control level of the data query request can be dynamically set according to the real-time traffic condition of the page to which the data query request points, so that the flow control level can effectively reflect the busy degree of the corresponding page to make a change in responsiveness, and a second security shielding of the independent station is formed by using this mechanism.

Step S1400, executing a data source flow control task, determining whether a flow control level obtained by the data query request reaches a level threshold required by a data source retrieval service corresponding to the request, and if so, controlling the data source retrieval service to execute the data query task on the corresponding data source according to the flow control level.

The step is mainly used for realizing the flow control task of a data source layer, so that corresponding data source retrieval service is executed for the data query request according to the finally determined flow control level of the data query request, so that corresponding result data can be obtained within the range of the data content scale corresponding to the flow control level.

When a data source flow control task is executed, it is first determined whether a flow control level finally obtained by the data query request reaches a level threshold required by a data source retrieval service corresponding to the data query request, where the level threshold may be set by a person skilled in the art as needed, and is usually lower than a highest level, for example, set to a next lower level, which means that when the flow control level reaches to the next lower level or higher, the data source retrieval service is executed for the corresponding data query request.

And when the flow control level is judged to reach the grade threshold, controlling the data source retrieval service to execute a data query task on a corresponding data source according to the flow control level. The data source retrieval service is generally provided corresponding to a corresponding data source, for example, when the data source retrieval service is used for executing a data rough recall service corresponding to a plurality of recall channels, each recall channel may be packaged as one data source retrieval service, each data source retrieval service recalls data from its corresponding data source, and the data sources on which different data source retrieval services depend when executing the rough recall may be the same data source, for example, the same database stored in a distributed system, in addition, a plurality of recall channels may also be packaged in a single data source retrieval service, and the like, which can be flexibly implemented. The data source may refer to a complete database or may be a data slice in a distributed storage system.

The data source retrieval service is implemented in advance to be capable of executing a corresponding data query task in response to a specific member level specified by the flow control level, so that corresponding retrieval can be performed in a data content scale range corresponding to the member level specified by the flow control level, the data content scale can be embodied by controlling the number of recall channels, or by controlling the number of data sources which depend on the data content scale, and the like, which can be flexibly implemented.

And after the data source retrieval service executes a corresponding data query task and obtains the rough recall data, the rough recall data can be used for responding the data query request, so that the whole business process is completed. Of course, the situation that the data source retrieval service throws an exception in the process of executing the data query task and returns a null value to respond to the data query request is not excluded.

In some embodiments, when the data source retrieval service obtains result data, the coarse recall data may be sorted before the data query request is answered, and the data query request is answered with sorted result data.

In some embodiments, during the operation of the data source retrieval service, the total requested flow per unit time corresponding to the data source may be further detected to adjust the flow control level, so as to construct a third security barrier against a flow attack for the independent station.

In some embodiments, the data source may be a product information library specifically storing product information corresponding to product items, or a product picture library specifically storing product pictures corresponding to product items, but should not be limited thereto.

According to the above embodiments, after a data query request of an independent station arrives, a flow control level is initialized for the independent station, so that the independent station holds the highest level of a plurality of member levels, then whether the flow control level is adjusted or not can be determined according to the total flow requested by the independent station in a unit time of a total station level, and specifically, the flow control level can be adjusted according to the total flow requested by the independent station in a unit time, and finally, a data source flow control task execution process is entered, in which a corresponding data source retrieval service is controlled to execute a data query task according to the flow control level.

On the basis of any embodiment of the present application, referring to fig. 3, executing a flow control task of an independent station, determining whether a total flow requested by a current independent station in unit time exceeds a flow control threshold of the independent station, and setting the flow control level to a lower level of the plurality of member levels when the total flow requested by the current independent station exceeds the flow control threshold, includes:

step S1210, when the total flow requested by the current independent station in unit time exceeds the flow control threshold of the independent station, triggering a corresponding flow control event of the independent station;

when the detection confirms that the total flow requested by the current independent station per unit time exceeds the independent station flow control threshold value, an independent station flow control event is correspondingly triggered so as to execute the subsequent steps based on the event response mechanism. When the independent station flow control threshold is not exceeded, the step S1240 may be directly skipped.

Step S1220, in response to the independent station flow control event, setting the flow control level to a second lower level among the plurality of member levels, where the second lower level is used to indicate a data source operation authority that is given to the data query request in a minimum data size;

and responding to the independent station flow control event, directly degrading and setting the flow control level of the data query request from the highest level to the lower level, wherein the lower level only allows the data query request to have the data source operation authority within the range of the minimum data content scale, so that the control of the subsequent service flow of the data access service corresponding to the data query request is also realized through the setting of the lower level.

Step S1230, responding to the independent station stream control event, skipping the step of executing the page stream control task and entering the step of executing the data source stream control task;

after step S1220 is performed, the flow control level is lowered to the next lower level in response to the independent station flow control event, and thus step S1300 corresponding to the page flow control task is not required to be performed, and step S1400 is directly skipped to perform the source flow control task.

And step S1240, when the total flow requested by the current independent station in unit time does not exceed the flow control threshold of the independent station, not modifying the flow control level, and sequentially entering the step of executing the page flow control task.

When the total flow requested by the current individual station in unit time does not exceed the flow control threshold of the individual station, because the individual station does not enter the busy state represented by the flow control threshold, the process may directly jump to execute step S1240 after executing step S1210, and directly enter step S1300 to execute the page flow control task without modifying the flow control level.

According to the embodiments, in the present application, the total flow requested in a unit time corresponding to the current time of the independent station is compared with the independent station flow control threshold set corresponding to the total flow requested in the unit time to determine the flow to be skipped for the data query request, so that the subsequent service flow can be optimized according to the actual situation, and thus, not only the load of the independent station is reduced by setting the flow control level when necessary, but also the flow control processing at the page level can be skipped when necessary by flexible avoidance of the service flow, thereby saving the system overhead and further reducing the load of the independent station.

On the basis of any embodiment of the present application, referring to fig. 4, executing a page flow control task, and recursively determining whether a total flow of requests per unit time of a page corresponding to the data query request exceeds a page flow control threshold includes:

step S1310, determining whether the total requested flow per unit time of the page corresponding to the data query request exceeds a page flow control threshold, and triggering a corresponding page flow control event when the total requested flow per unit time of the page corresponding to the data query request exceeds the page flow control threshold;

in the process of implementing flow control based on the page, whether the total flow requested by the page corresponding to the data query request in unit time exceeds a preset page flow control threshold is judged, and when the total flow exceeds the page flow control threshold, the flow of the corresponding page is represented to be too heavy, so that a corresponding page flow control event can be triggered, and the subsequent steps can be executed based on an event response mechanism.

Step S1320, responding to the page flow control event, judging whether the flow control level is the lowest level of the plurality of member levels, and when the flow control level is not the lowest level, reducing the flow control level by one level;

responding to the page flow control event, further judging whether the flow control level reaches the lowest level of the plurality of member levels, if so, theoretically terminating iterative detection, otherwise, if not, namely, the flow control level is not the lowest level, then according to the gradient relation of the member levels, descending the original member level by one level, for example, descending the original member level from the highest level to the next highest level, descending the original member level from the next highest level to the next lower level, and then descending the original member level to the lowest level.

Step S1330, iterating the above processes until the flow control level is the lowest level, returning a null value in response to the data service request, and terminating execution of the subsequent step, or until the data service request is asynchronously processed by the step of executing the data source flow control task.

Since the data query request is usually waiting for dequeue consumption in an asynchronous mechanism, the above process may be iterated before it is consumed, and the process is executed from step S1310 again, and loop detection is performed to determine whether the flow control level needs to be set for demotion, and the process is performed recursively.

In the iteration process, if the flow control level reaches the lowest level, the page to which the data query request points is too busy, and the data query request is not suitable to be responded, and at the moment, a null value is directly adopted to respond to the data service request without executing a subsequent business process. If the flow control level has not reached the lowest level in the above iteration process, the data query request is dequeued for asynchronous consumption, and at this time, the above iteration process is naturally terminated, and step S1400 is executed instead.

According to the embodiments, on the page level, the busy state of the corresponding page is monitored in real time by using the time interval of asynchronous queuing and queuing of the data query requests, and when the page is excessively busy, the flow control level of the page can be timely reduced to form a self-adaptive dynamic regulation mechanism, so that the data query requests of the whole independent station can be efficiently scheduled according to the flow of the page under the action of the mechanism, and the independent station is ensured to realize fine and effective flow control on the whole.

On the basis of any embodiment of the present application, referring to fig. 5, executing a data source flow control task, and determining whether a flow control level obtained by the data query request reaches a level threshold required by a data source retrieval service corresponding to the request includes:

step S1410, when the flow control level obtained by the data query request reaches the level threshold required by the data source retrieval service corresponding to the request, triggering a data query event;

in the process of executing the data source flow control task, when the flow control level finally obtained by the data query request is determined to reach the level threshold required by the data source retrieval service corresponding to the data query request through detection, a corresponding data query event can be triggered so as to execute a subsequent service flow based on an event response mechanism.

Step S1420, responding to the data query event, determining whether a deviation ratio of an average request delay of the corresponding data source retrieval service with respect to a set value reaches a preset threshold, and when the deviation ratio reaches the preset threshold, controlling the data source retrieval service to execute a data query task based on local data content query on the corresponding data source;

in response to the data query event, the access scale of the data query request to the corresponding data source can be further regulated according to the concrete representation of the average request delay of the data source retrieval service.

Specifically, a set value and a preset threshold value are set for the data source retrieval service in advance, the set value is used for representing a response delay threshold value of a corresponding data source, when the average response delay generated when the data source retrieval service accesses the data source exceeds the response delay threshold value, the data source can be considered to be in a busy state, otherwise, the data source is not busy; the preset threshold is a sufficient condition for measuring whether the amplitude of the average response delay exceeding the response delay threshold reaches a value for confirming that the data source is in a busy state. And then, comparing the deviation proportion with the preset threshold value, when the deviation proportion is higher than the preset threshold value, determining that the data source is in a busy state, otherwise, determining that the data source is in a non-busy state.

And comparing the deviation proportion of the average request time delay relative to the set value with the preset threshold value to confirm the busy state of the data source, wherein the deviation proportion is used for representing the variation amplitude, has the capability of self-adaptive adjustment in response to the set value and the preset threshold value and keeping adjustment within a certain amplitude range, and has higher robustness.

And after the data source is confirmed to be in a busy state, the data source retrieval service can be controlled to enter a business process of local data content query, so that the data source retrieval service executes a data query task based on local data content query on the corresponding data source. The local data content may be a partial data slice generated by a data source stored in the distributed storage system, rather than a full data slice. It follows that by local data content querying, the load of the distributed storage coefficients can be reduced, which is also reflected in a reduced load of the whole individual station.

And step S1430, when the preset threshold value is not reached, controlling the data source retrieval service to execute a data query task based on all data content query on the corresponding data source.

When the step S1420 is judged, it is determined that the deviation ratio of the average request delay of the corresponding data source retrieval service with respect to the set value does not reach the preset threshold, which indicates that the corresponding data source is in a non-busy state, at this time, the data source retrieval service may be controlled to execute a data query task based on a query of all data contents on the corresponding data source according to a normal service flow, for example, execute the data query task in all data slices generated for the data source by the distributed storage system, so as to perform a full query.

According to the embodiment, after the business process of the data source retrieval service is entered, whether the corresponding data source is in a busy state or not can be further evaluated according to the average request time delay obtained by statistics after the data source retrieval service queries the data source, and when the data source is in the busy state, the corresponding data query task is controlled to narrow the data query range, so that the load of whole data service access of the independent station is reduced, the robustness of the whole system of the independent station is improved, and good access experience is ensured.

On the basis of any embodiment of the present application, referring to fig. 6, when a preset threshold is not reached, controlling the data source retrieval service to execute a data query task based on a query of all data contents on a corresponding data source includes:

step S1421, judging whether the total flow requested in unit time of the corresponding data source retrieval service reaches a set threshold or throws an exception, and triggering a fusing event when the total flow reaches the set threshold or throws the exception;

in an embodiment, when the data source retrieval service confirms that the corresponding data source is in a non-busy state and needs to start a query process of a whole data content range, the response capability of the data source can be further confirmed by detecting the total flow of unit-time requests of the data source. Therefore, whether the total flow of the requests of the data source retrieval service in unit time reaches a set threshold is judged, and when the total flow reaches the set threshold, the requests are indicated to be processing a large number of requests, so that a fusing event is triggered to start a fusing mechanism.

In another embodiment, when the data source retrieval service is in the process of executing the data query task, the fusing event may also be triggered if an exception is thrown.

Step S1422, responding to the fusing event, and controlling the data source retrieval service to execute a data query task based on local data content query on a corresponding data source;

in response to the fusing event, the query is not performed in a global data content manner any more, but is performed in a local data content-based manner, that is, in the last step, the system fusing is further prevented according to the busyness of the data source retrieval service, and a specific data query task is adjusted in time in response to the fusing event of the system, so that the data query request can obtain corresponding rough recall data in a limited local data content range.

Step S1423, when the fusing event is not triggered, controlling the data source retrieval service to execute a data query task based on a query of all data contents on a corresponding data source.

If the fusing event is not triggered in the process of executing the data query task by the data source retrieval service, the flow of the data source retrieval service is moderate, and the execution process is smooth, so that the data source retrieval service is continuously controlled to execute the data query task based on all data content queries on the corresponding data source. So as to obtain the coarse recall data over the full amount of data content.

According to the embodiment, in the process of executing the data source retrieval service, a fusing mechanism is established, so that when the data source retrieval service is busy or abnormal, timely adjustment can be made, the query scale of the data query request can be timely controlled, the load of the data source of the independent station can be effectively adjusted, and the operation robustness of the independent station can be further ensured.

On the basis of any embodiment of the present application, referring to fig. 7, after the data source flow control task is executed, the method includes:

step S1500, judging whether the flow control level obtained by the data query request reaches a level threshold required by the data sorting service corresponding to the request, and triggering a sorting event when the flow control level reaches the level threshold;

after the steps S1100 to S1400 of the present application, the data query request will obtain corresponding rough recall data after being responded normally, and for these result data, according to the mechanism of the commodity recommendation system, the data sorting service may be further adopted to sort and prefer so as to output the final sorting result.

And in the same way as before, a level threshold suitable for processing can also be set for the data sorting service, when the data sorting service is called, whether the flow control level obtained by the data query request at the latest reaches the level threshold required by the data sorting service is judged, when the flow control level does not reach the level threshold, the data sorting service is skipped to terminate execution without difficulty, and the data service request can be responded by null value. When the flow control level reaches the level threshold, it indicates that the data query request at least obtains coarse recall data of a part of data sources as result data, so that a data sorting service can be used, and accordingly, a sorting event can be triggered so as to execute a subsequent business process based on an event response mechanism.

Step S1600, responding to the sorting event, sorting result data obtained after the data source retrieval service executes the data query task, and obtaining a sorting result;

in response to the sorting event, the result data can be sorted according to a preset sorting service logic through the data sorting service, preference is given after sorting, and data with a part of the data arranged in front is selected as a sorting result, for example, the commodity data of a plurality of commodity items with the highest recommendation scores are selected to construct a commodity recommendation list.

And S1700, responding the data query request according to the sorting result.

After the sorting result is determined, the sorting result can be pushed to a requester corresponding to the data query request, and the response to the data query request is completed. The requester may be, for example, a terminal device of a user of the independent station or any other device that initiates the data query request.

According to the embodiments, it can be seen that the method and the device ensure that various data query requests can obtain corresponding query sequencing results with more stable reliability through flow control, ensure that the independent station runs stably, and enable users to obtain good access experience.

Referring to fig. 8, an apparatus for controlling independent station access traffic according to an aspect of the present application includes an initialization module 1100, an intra-station traffic control module 1200, a page traffic control module 1300, and a retrieval traffic control module 1400, where: the initialization module 1100 is configured to receive a data query request, initialize a flow control level corresponding to the request to a highest level among a plurality of member levels, and respectively indicate data source operation permissions of different data scales by different member levels; the intra-station traffic control module 1200 is configured to execute a traffic control task of an independent station, determine whether a total traffic requested by the current independent station in unit time exceeds a traffic control threshold of the independent station, and set the traffic control level as a lower level among the plurality of member levels when the total traffic exceeds the traffic control threshold; the page flow control module 1300 is configured to execute a page flow control task, recursively determine whether a total flow of requests per unit time of a page corresponding to the data query request exceeds a page flow control threshold, and if so, reduce the flow control level by one level when the flow control level does not reach a lowest level; the retrieval flow control module 1400 is configured to execute a data source flow control task, determine whether a flow control level obtained by the data query request reaches a level threshold required by a data source retrieval service corresponding to the request, and control the data source retrieval service to execute the data query task on the corresponding data source according to the flow control level when the flow control level obtained by the data query request reaches the level threshold.

On the basis of any embodiment of the present application, the intra-station flow control module 1200 includes: the total station event triggering unit is set to trigger a corresponding independent station flow control event when the total flow requested by the current independent station in unit time exceeds an independent station flow control threshold; the total station setting unit is used for setting the flow control level to be the lower level of the plurality of member levels in response to the independent station flow control event, and the lower level is used for indicating the data source operation authority which is given to the data query request in the minimum data scale; the total station event response unit is set to respond to the independent station flow control event, skip the step of executing the page flow control task and enter the step of executing the data source flow control task; and the total station normal execution unit is set to not modify the flow control level and sequentially enter the step of executing the page flow control task when the total flow requested by the current independent station in unit time does not exceed the flow control threshold of the independent station.

On the basis of any embodiment of the present application, the page flow control module 1300 includes: the page event triggering unit is configured to judge whether the total requested flow per unit time of the page corresponding to the data query request exceeds a page flow control threshold, and trigger a corresponding page flow control event when the total requested flow per unit time of the page corresponding to the data query request exceeds the page flow control threshold; the page event response unit is set to respond to the page flow control event, judge whether the flow control level is the lowest level in the plurality of member levels, and reduce the flow control level by one level when the flow control level is not the lowest level; and the page iteration processing unit is set to iterate the processes until the flow control level is the lowest level, responds to the data service request and returns a null value, and terminates the execution of the subsequent steps, or until the data service request is asynchronously processed by the step of executing the data source flow control task.

On the basis of any embodiment of the present application, the retrieving flow control module 1400 includes: a query event triggering unit configured to trigger a data query event when a flow control level obtained by the data query request reaches a level threshold required by a data source retrieval service corresponding to the request; the query event response unit is used for responding to the data query event, judging whether the deviation proportion of the average request time delay of the corresponding data source retrieval service relative to a set value reaches a preset threshold value or not, and controlling the data source retrieval service to execute a data query task based on local data content query on the corresponding data source when the deviation proportion reaches the preset threshold value; and the normal query execution unit is set to control the data source retrieval service to execute a data query task based on all data content queries on the corresponding data source when the preset threshold value is not reached.

On the basis of any embodiment of the present application, the normal query execution unit includes: the system comprises a fusing event triggering unit, a fusing event processing unit and a data source retrieval service processing unit, wherein the fusing event triggering unit is used for judging whether the total flow requested by a corresponding data source retrieval service in unit time reaches a set threshold or throws out abnormity, and triggering a fusing event when the total flow reaches the set threshold or throws out abnormity; the fusing event response unit is used for responding the fusing event and controlling the data source retrieval service to execute a data query task based on local data content query on a corresponding data source; and the circular detection processing unit is used for controlling the data source retrieval service to execute a data query task based on all data content query on a corresponding data source when the fusing event is not triggered.

On the basis of any embodiment of the present application, the retrieval flow control module 1400 includes: a sorting event triggering unit, configured to determine whether a flow control level obtained by the data query request reaches a level threshold required by a data sorting service corresponding to the request, and trigger a sorting event when the flow control level reaches the level threshold; the ordering event response unit is used for responding to the ordering event, ordering the result data obtained after the data source retrieval service executes the data query task and obtaining an ordering result; and the sequencing result pushing unit is used for responding the data query request by the sequencing result.

On the basis of any embodiment of the present application, the flow control levels include the following member levels: the highest level is used for indicating that the data source operation authority is shared in the scale range of the whole data content; a secondary high level for indicating that the data source operation right is enjoyed within the scale range of the local data content; a second lowest level for indicating that a data source operation right is enjoyed within a set range smaller than the scale of the local data content; and the lowest level is used for indicating that the data source operation authority is not shared by any data content.

Another embodiment of the present application also provides an independent station access flow control apparatus. As shown in fig. 9, the internal structure of the individual station access flow control apparatus is schematically illustrated. The independent station access flow control device includes a processor, a computer readable storage medium, a memory, and a network interface connected by a system bus. Wherein the computer readable non-transitory readable storage medium of the standalone station access flow control device stores an operating system, a database, and computer readable instructions, the database having stored therein a sequence of information, the computer readable instructions, when executed by the processor, cause the processor to implement a standalone station access flow control method.

The processor of the independent station access flow control device is used for providing calculation and control capability and supporting the operation of the whole independent station access flow control device. The memory of the individual station access flow control device may have computer readable instructions stored thereon that, when executed by the processor, cause the processor to perform the individual station access flow control method of the present application. The independent station accesses the network interface of the flow control device for connection communication with the terminal.

Those skilled in the art will appreciate that the structure shown in fig. 9 is a block diagram of only a portion of the structure relevant to the present application, and does not constitute a limitation on the individual station access flow control devices to which the present application is applied, and a particular individual station access flow control device may include more or less components than those shown in the figure, or combine certain components, or have a different arrangement of components.

In this embodiment, the processor is configured to execute specific functions of each module in fig. 8, and the memory stores program codes and various data required for executing the modules or sub-modules. The network interface is used for realizing data transmission between user terminals or servers. The nonvolatile readable storage medium in the present embodiment stores program codes and data necessary for the independent station access flow control device of the present application to execute all modules, and the server can call the program codes and data of the server to execute the functions of all modules.

The present application further provides a non-transitory readable storage medium having stored thereon computer readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of the individual station access flow control method of any of the embodiments of the present application.

The present application also provides a computer program product comprising computer programs/instructions which, when executed by one or more processors, implement the steps of the method as described in any of the embodiments of the present application.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments of the present application may be implemented by hardware related to instructions of a computer program, which may be stored in a non-volatile readable storage medium, and when executed, may include the processes of the embodiments of the methods as described above. The storage medium may be a computer-readable storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).

In conclusion, the flow rate of the independent station in the e-commerce platform is organically adjusted, the operation robustness of the independent station, the page, the data source and the like is stronger, the daily operation smoothness of the independent station can be ensured, the throughput of the independent station can be improved, and various illegal attacks can be powerfully resisted.

Claims

1. An independent station access flow control method, comprising:

executing a page flow control task, recursively judging whether the total flow of requests of a page corresponding to the data query request in unit time exceeds a page flow control threshold value, and if so, descending the flow control level by one level when the flow control level does not reach the lowest level;

2. The individual station access flow control method according to claim 1, wherein performing an individual station flow control task, determining whether a total flow requested per unit time of a current individual station exceeds an individual station flow control threshold, and when exceeding, setting the flow control level to a next lower level among the plurality of member levels comprises:

3. The method of claim 1, wherein performing a page flow control task to recursively determine whether a total flow of requests per unit time for a page corresponding to the data query request exceeds a page flow control threshold comprises:

4. The method of claim 1, wherein performing a data source flow control task, and determining whether a flow control level obtained by the data query request reaches a level threshold required for a data source retrieval service corresponding to the request comprises:

5. The method as claimed in claim 4, wherein when the preset threshold is not reached, controlling the data source retrieval service to perform a data query task based on a full data content query on the corresponding data source comprises:

responding to the fusing event, and controlling the data source retrieval service to execute a data query task based on local data content query on a corresponding data source;

and when the fusing event is not triggered, controlling the data source retrieval service to execute a data query task based on the query of all data contents on the corresponding data source.

6. The method according to any one of claims 1 to 5, wherein after performing the data source flow control task, the method comprises:

ordering result data obtained after the data source retrieval service executes the data query task in response to the ordering event to obtain an ordering result;

and responding the data query request with the sequencing result.

7. An individual station access flow control method according to any of claims 1 to 5, characterized in that said flow control levels comprise the following membership levels:

the highest grade is used for indicating that the data source operation authority is shared in the scale range of the whole data content;

a secondary high level for indicating that the data source operation authority is enjoyed within the scale range of the local data content;

a second lowest level for indicating that a data source operation right is enjoyed within a set range smaller than the scale of the local data content;

8. An individual station access flow control apparatus, comprising:

the initialization module is configured to receive a data query request, initialize a flow control level corresponding to the request to a highest level of a plurality of member levels, and respectively indicate data source operation permissions of different data scales by different member levels;

9. A standalone station access flow control apparatus comprising a central processor and a memory, wherein the central processor is adapted to invoke execution of a computer program stored in the memory to perform the steps included in the method of any of claims 1 to 7.

10. A non-transitory readable storage medium storing a computer program in the form of computer readable instructions, the computer program when invoked by a computer performing the steps comprising the method of any one of claims 1 to 7.