CN113595921B - Data stream processing method and device, electronic equipment and computer readable storage medium - Google Patents
Data stream processing method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN113595921B CN113595921B CN202110732655.8A CN202110732655A CN113595921B CN 113595921 B CN113595921 B CN 113595921B CN 202110732655 A CN202110732655 A CN 202110732655A CN 113595921 B CN113595921 B CN 113595921B
- Authority
- CN
- China
- Prior art keywords
- data stream
- transmitted
- current limiting
- user
- preference degree
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/29—Flow control; Congestion control using a combination of thresholds
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/215—Flow control; Congestion control using token-bucket
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/24—Traffic characterised by specific attributes, e.g. priority or QoS
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
Abstract
According to the data stream processing method, the data stream processing device, the electronic equipment and the computer readable storage medium, the data stream to be transmitted is differentiated and limited according to the preference degree of the user on the target object by analyzing the user corresponding to the data stream to be transmitted. In the embodiment of the application, after a data stream request sent by a user is obtained, a corresponding data stream to be transmitted is obtained; and predicting the preference degree of the user to a target object in the data stream to be transmitted, processing the corresponding data stream to be transmitted according to the magnitude relation between the preference degree of the user and a current limiting threshold value, transmitting the data stream requested by the user with higher preference degree, intercepting the data stream requested by the user with lower preference degree, and performing intelligent differential current limiting according to the preference degree of the user to the target object.
Description
Technical Field
The present disclosure relates to the field of information processing, and in particular, to a method and an apparatus for processing a data stream, an electronic device, and a computer-readable storage medium.
Background
In a high-concurrency mass traffic system, in order to ensure availability of the system, when a traffic sudden increase or a system load is large and cannot satisfy all traffic, data streams requested by a part of users are generally limited, that is, a corresponding service is not provided for the part of users, so that the use experience of the part of users is necessarily affected. For example, in an e-commerce system, during a promotion event, a user request suddenly increases, which results in an overload of the system, and in order to ensure the availability of the system, a data stream requested by a part of users needs to be limited.
However, the existing current limiting schemes perform indiscriminate random interception on a data stream requested by a user, and indiscriminate interception affects the use experience of a part of high-quality users, resulting in loss of the high-quality users.
Therefore, how to consider the usability of the system and the use experience of a high-quality user and realize more intelligent data stream processing is an urgent problem to be solved.
Content of application
The application provides a data stream processing method, a data stream processing device, electronic equipment and a computer-readable storage medium, which are used for predicting the preference degree of a user for a target object in a data stream to be transmitted and processing the data stream to be transmitted according to the preference degree of the user, so that the processing of the data stream to be transmitted is more intelligent.
In order to solve the foregoing problem, in a first aspect, an embodiment of the present application provides a data stream processing method, where the method includes:
when a data stream request sent by a user terminal is received, acquiring a data stream to be transmitted;
predicting the preference degree of a user using the user terminal to a target object in the data stream to be transmitted;
and processing the data stream to be transmitted according to the size relation between the preference degree and the current limiting threshold value.
Optionally, processing the data stream to be transmitted according to a size relationship between the preference and a current limiting threshold includes:
under the condition that the preference degree is smaller than a current limiting threshold value, limiting the current of the data stream to be transmitted according to a first current limiting mode;
and transmitting the data stream to be transmitted to the user terminal under the condition that the preference degree is not less than the current limiting threshold value.
Optionally, processing the data stream to be transmitted according to a size relationship between the preference and the current limit threshold includes:
limiting the data stream to be transmitted under the condition that the preference degree is smaller than a current limiting threshold value;
under the condition that the preference degree is not smaller than the current limiting threshold value, detecting whether the data stream to be transmitted meets the condition of current limiting according to a second current limiting mode;
transmitting the data stream to be transmitted to the user terminal under the condition that the data stream to be transmitted does not meet the condition of limiting the current according to a second current limiting mode;
and under the condition that the data stream to be transmitted meets the condition of current limiting according to a second current limiting mode, limiting the current of the data stream to be transmitted according to the second current limiting mode.
Optionally, the method further comprises:
periodically updating the current limiting threshold value by taking unit time as a period according to the number of all data streams to be transmitted in the unit time, the number of the data streams to be transmitted which are triggered to carry out current limiting according to a first current limiting mode in the unit time and the number of the data streams to be transmitted which are triggered to carry out current limiting according to a second current limiting mode in the unit time;
processing the data stream to be transmitted according to the size relationship between the preference degree and the current limiting threshold, wherein the processing comprises the following steps:
and processing the data stream to be transmitted according to the magnitude relation between the preference degree and the current limiting threshold value which is updated last time.
Optionally, the method further comprises:
comparing the number of the data streams to be transmitted which are triggered to be limited in the unit time in a second current limiting mode with a preset threshold value;
under the condition that the number of the data streams to be transmitted, which are triggered to be subjected to current limiting according to the second current limiting mode in unit time, is smaller than the preset threshold value, judging whether the number of all the data streams to be transmitted in two adjacent unit times is reduced or not;
and under the condition that the number of all data streams to be transmitted in two adjacent unit times is not reduced, stopping updating the current limiting threshold value.
Optionally, the method further comprises:
under the condition that the number of all data streams to be transmitted in two adjacent unit times is reduced, attenuating the current limiting threshold value which is updated for the last time according to a preset attenuation ratio;
processing the data stream to be transmitted according to the size relationship between the preference degree and the current limiting threshold, wherein the processing comprises the following steps:
and processing the data stream to be transmitted according to the magnitude relation between the preference degree and the attenuated current limiting threshold value.
Optionally, detecting whether the to-be-transmitted data stream meets a condition of performing current limitation according to a second current limiting manner includes:
generating tokens at a preset rate and putting the tokens into a token bucket, wherein the generation of the tokens is stopped after the total number of the tokens contained in the token bucket reaches an upper limit;
and detecting whether the data stream to be transmitted acquires tokens from a token bucket.
Optionally, predicting a preference of a user using the user terminal for a target object in the data stream to be transmitted includes:
collecting historical operation behavior data of a user using the user terminal on a target object in each transmitted data stream, wherein the historical operation behavior data comprises at least one of the following data: click rate, exposure duration;
and predicting the preference degree of a user using the user terminal to a target object in the data stream to be transmitted according to the historical operation behavior data.
In a second aspect, an embodiment of the present application provides a data stream processing apparatus, where the apparatus includes:
the data stream acquisition module is used for acquiring a data stream to be transmitted when receiving a data stream request sent by a user terminal;
the preference degree prediction module is used for predicting the preference degree of a user using the user terminal to a target object in the data stream to be transmitted;
and the data stream processing module is used for processing the data stream to be transmitted according to the size relation between the preference degree and the current limiting threshold value.
Optionally, the data stream processing module includes:
the current limiting submodule is used for limiting the current of the data stream to be transmitted according to a first current limiting mode under the condition that the preference degree is smaller than a current limiting threshold value;
and the transmission sub-module is used for transmitting the data stream to be transmitted to the user terminal under the condition that the preference degree is not less than the current limiting threshold value.
Optionally, the data stream processing module includes:
the first current limiting submodule is used for limiting the current of the data stream to be transmitted under the condition that the preference degree is smaller than a current limiting threshold value;
the detection submodule is used for detecting whether the data stream to be transmitted meets the condition of current limiting according to a second current limiting mode under the condition that the preference degree is not smaller than the current limiting threshold value;
the first transmission submodule is used for transmitting the data stream to be transmitted to the user terminal under the condition that the data stream to be transmitted does not meet the condition of current limiting according to a second current limiting mode;
and the second current limiting submodule is used for limiting the current of the data stream to be transmitted according to a second current limiting mode under the condition that the data stream to be transmitted meets the condition of limiting the current according to the second current limiting mode.
Optionally, the apparatus further comprises:
the updating module is used for periodically updating the current limiting threshold value according to the number of all data streams to be transmitted in unit time, the number of the data streams to be transmitted which are triggered to carry out current limiting according to a first current limiting mode in unit time and the number of the data streams to be transmitted which are triggered to carry out current limiting according to a second current limiting mode in unit time by taking unit time as a period;
the data stream processing module comprises:
and the first processing submodule is used for processing the data stream to be transmitted according to the magnitude relation between the preference degree and the current limiting threshold value which is updated last time.
Optionally, the apparatus further comprises:
the comparison module is used for comparing the number of the data streams to be transmitted which are triggered to carry out current limiting according to the second current limiting mode in the unit time with a preset threshold value;
the judging module is used for judging whether the number of all data streams to be transmitted in two adjacent unit times is reduced or not under the condition that the number of the data streams to be transmitted which are triggered to carry out current limiting according to the second current limiting mode in the unit time is smaller than the preset threshold value;
and the stopping module is used for stopping updating the current limiting threshold under the condition that the quantity of all the data streams to be transmitted in two adjacent unit times is not reduced.
Optionally, the method further comprises:
the attenuation module is used for attenuating the current limiting threshold value which is updated last time according to a preset attenuation ratio under the condition that the number of all data streams to be transmitted in two adjacent unit times is reduced;
the data stream processing module comprises:
and the second processing sub-module is used for processing the data stream to be transmitted according to the magnitude relation between the preference degree and the attenuated current limiting threshold value.
Optionally, the detection submodule includes:
the system comprises a token generating unit, a token generating unit and a token bucket, wherein the token generating unit is used for generating tokens at a preset rate and putting the tokens into the token bucket, and the token generating unit stops generating the tokens after the total number of the tokens contained in the token bucket reaches an upper limit;
and the detection unit is used for detecting whether the data stream to be transmitted acquires tokens from a token bucket.
Optionally, the preference prediction module includes:
an acquisition submodule, configured to acquire historical operation behavior data of a user using the user terminal on a target object in each transmitted data stream, where the historical operation behavior data includes at least one of: click rate, exposure duration;
and the prediction submodule is used for predicting the preference degree of the user using the user terminal to the target object in the data stream to be transmitted according to the historical operation behavior data.
In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the data stream processing method provided in the embodiment of the present application.
In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor, and the computer program performs the steps of the data stream processing method provided in the present application.
In the embodiment of the application, after a data stream request sent by a user is obtained, a corresponding data stream to be transmitted is obtained, the preference degree of the user for a target object in the data stream to be transmitted is predicted, the data stream to be transmitted corresponding to the user is processed according to the size relation between the preference degree of the user and a current limiting threshold value, the data stream to be transmitted of the user with higher preference degree is transmitted, the data stream to be transmitted of the user with lower preference degree is intercepted, intelligent differential data stream processing is carried out according to the preference degree of the user for the target object, and differential current limiting is achieved when the system load is higher.
Taking the target object as an advertisement as an example, in the embodiment of the application, when the system load is too high, whether to limit the current of the user can be determined according to the advertisement click rate of the user, differential current limiting is realized according to the advertisement click rate of the user, service stopping is performed on the user with low advertisement click rate, normal service is performed on the user with high advertisement click rate, and the current limiting is realized to ensure the availability of the system, and meanwhile, the advertisement click rate is ensured to be higher.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the description of the embodiments or the related art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.
Fig. 1 is a flowchart of a data stream processing method provided in an embodiment of the present application;
fig. 2 is a flowchart of a data stream processing method provided in an embodiment of the present application;
fig. 3 is a flowchart of a data stream processing method provided in an embodiment of the present application;
fig. 4 is a diagram illustrating an example of a data stream processing method according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a text processing apparatus according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
A flowchart of a data stream processing method provided in an embodiment of the present application is applied to an application server, as shown in fig. 1. The data stream processing method comprises the following steps:
step S110, when receiving a data stream request sent by a user terminal, obtaining a data stream to be transmitted.
In the embodiment of the application, the user terminal may initiate a data stream request to the application server, and the server obtains a corresponding data stream to be transmitted according to the data stream request.
In this embodiment of the present application, the data stream to be transmitted refers to, after the user terminal initiates a data stream request, responding to the request, the data stream to be transmitted that is acquired by the application server and needs to be delivered to the user terminal, and it can be understood that the data stream to be transmitted is a data stream requested by a user.
In practical application, if the load of the current application system is low and the transmission requirements of all data streams can be met, after the data stream to be transmitted is obtained, the data stream to be transmitted is timely issued to the user terminal so as to respond to a data stream request sent by the user terminal.
If the load of the current application system is large and the transmission requirements of all data streams cannot be met, executing the following steps S120 to S130:
step S120, predicting the preference degree of the user using the user terminal to the target object in the data stream to be transmitted.
In the embodiment of the application, the user identity of the user terminal can be determined according to a data stream request initiated by the user terminal, so that the relevant information of the user using the user terminal can be acquired. And predicting the preference degree of the target object in the data stream to be transmitted by the user according to the relevant information of the user.
For example, the data stream to be transmitted is a video stream, and the target object is an advertisement video, which is taken as an example to further explain:
after a data stream request initiated by a user is obtained, the user ID of the user can be determined, historical behavior data of the user is further obtained according to the user ID, and therefore the preference degree of the user to the advertisement video is predicted according to the historical behavior data of the user.
And step S130, processing the data stream to be transmitted according to the size relation between the preference degree and the current limiting threshold value.
In the embodiment of the application, after the preference degree of the user for the target object is obtained, the preference degree is compared with a preset current limiting threshold, the data stream to be transmitted of the user with the preference degree smaller than the current limiting threshold is limited, and the data stream to be transmitted of the user with the preference degree larger than the preset threshold is transmitted.
In the embodiment of the present application, the current limit threshold may be preset by a technician in advance according to experience and historical data, or may be updated in real time according to a change of a data amount of the server.
In the embodiment of the application, after a data stream request sent by a user is obtained, a corresponding data stream to be transmitted is obtained, and the preference degree of the user for a target object in the data stream to be transmitted is predicted, so that the data stream to be transmitted corresponding to the user is processed according to the size relation between the preference degree of the user and a current limiting threshold value, the data stream to be transmitted of the user with higher preference degree is transmitted, the data stream to be transmitted of the user with lower preference degree is intercepted, intelligent differentiated data stream processing is performed according to the preference degree of the user for the target object, and differentiated current limiting is achieved when the system load is higher.
Taking the target object as an advertisement as an example, in the embodiment of the application, when the system load is too high, whether to limit the current of the user can be determined according to the advertisement click rate of the user, differential current limiting is realized according to the advertisement click rate of the user, service stopping is performed on the user with low advertisement click rate, normal service is performed on the user with high advertisement click rate, and the current limiting is realized to ensure the availability of the system, and meanwhile, the advertisement click rate is ensured to be higher.
Referring to fig. 2, an embodiment of the present application provides a flowchart of a data stream processing method, where the data stream processing method includes:
step S210, when receiving a data stream request sent by a user terminal, obtaining a data stream to be transmitted.
This step is similar to step S110 described above and will not be described herein again.
Step S220, predicting the preference degree of the user using the user terminal to the target object in the data stream to be transmitted.
In the embodiment of the application, the preference degree of a user to a target object in a data stream to be transmitted can be predicted according to historical behavior data of the user using the user terminal.
In an alternative embodiment, step S220 includes the following sub-steps:
substep S221, collecting historical operation behavior data of a user using the user terminal on a target object in each transmitted data stream, wherein the historical operation behavior data includes at least one of the following data: click rate, exposure duration.
In this embodiment of the present application, the method for acquiring relevant information of the user according to user information carried in a data flow request initiated by a user terminal may specifically include: user representation, historical user operational behavior data on target objects in the transmitted data stream, and the like.
In a specific application process, historical data related to a target object can be acquired from historical operation behavior data of a user, so that the click rate, the exposure duration and the like of the user for the target object can be obtained according to statistical analysis of the historical data related to the target object. In the embodiment of the present application, the target object may be video data, web page data, text data, or the like.
And a substep S222, predicting the preference degree of the user using the user terminal to the target object in the data stream to be transmitted according to the historical operation behavior data.
In the embodiment of the application, after the historical operation behavior data of the user is collected, the historical operation behavior data of the user can be analyzed by adopting a rule statistic method or a model estimation method, and the preference degree of the user to the target object is predicted.
Specifically, the rule statistical method comprises the following steps: and carrying out statistical analysis on historical operation behavior data of the user by using a preset rule to determine the preference of the user on the target object. The model estimation method comprises the following steps: and predicting to obtain the preference degree of the user to the target object by utilizing a pre-trained preference degree prediction model and the historical operation behavior data of the user.
In practical application, the preference degree of the user for the target object can be counted off line, and the user is graded according to the preference degree of the user so as to determine the processing method of the data to be processed according to the gear of the user.
Illustratively, the target object is an advertisement video:
in practical application, relevant data about clicking, playing and playing time of the advertisement in the historical operation behaviors of the user can be collected, and the clicking rate, the exposure rate and the exposure duration of the advertisement video by the user are calculated according to the collected relevant data, so that the preference degree of the user to the advertisement video is determined according to the historical operation behavior data of the user.
Step S230, limiting the data stream to be transmitted according to a first current limiting mode under the condition that the preference degree is smaller than a current limiting threshold value; and transmitting the data stream to be transmitted to the user terminal under the condition that the preference degree is not less than the current limiting threshold value.
Specifically, after determining the preference of the user for the target object, the preference may be compared with a current limiting threshold, and if the preference is smaller than the current limiting threshold, the data stream to be transmitted corresponding to the user is intercepted, and if the preference is larger than the current limiting threshold, the data stream to be transmitted corresponding to the user is transmitted.
In the embodiment of the application, the first current limiting manner may be understood as that the server actively limits the data stream to be transmitted of the user whose preference is smaller than the current limiting threshold, so as to actively limit the data stream to be transmitted of a part of users when the system load is large, and the service experience of the high-quality user, and the click rate and the exposure rate of the target object may be ensured according to the preference of the user to the target object.
Referring to fig. 3, an embodiment of the present application provides a flowchart of another data stream processing method, where the data stream processing method includes:
step S310, when receiving a data stream request sent by a user terminal, obtaining a data stream to be transmitted.
This step is similar to step S210 described above and will not be described herein again.
Step S320, predicting the preference of the user using the user terminal for the target object in the data stream to be transmitted.
This step is similar to step S220 described above and will not be described herein again.
Step S330, detecting whether the data stream to be transmitted meets the condition of current limiting according to a second current limiting mode under the condition that the preference degree is not smaller than the current limiting threshold value.
In the embodiment of the application, under the condition that the preference degree of the user is smaller than the current limiting threshold value, the server actively limits the current of the data stream to be transmitted of the user, does not provide service for the user, and limits the transmission of the data stream to be transmitted of the user. After the server actively limits the current of the data stream to be transmitted of the user with the preference degree smaller than the current limiting threshold, further detecting whether the data stream to be transmitted of the user with the preference degree not smaller than the current limiting threshold needs to be limited according to a second current limiting mode.
Step S340, transmitting the data stream to be transmitted to the user terminal when the data stream to be transmitted does not satisfy the condition of current limiting according to the second current limiting manner.
In the embodiment of the present application, after performing differentiated current limiting according to the preference of the user for the target object, there may still be a situation that the system load is too high, and therefore it is required to detect whether a large amount of data streams to be transmitted need to be further intercepted according to the second current limiting manner. In the embodiment of the present application, the second current limiting manner is an undifferentiated random current limiting manner, and specifically, any current limiting scheme in the prior art may be adopted.
The data stream to be transmitted meets the requirement of current limiting according to a second current limiting mode, and the indication is that: after the transmission of part of the data streams is limited according to the first current limiting mode, the current system load is still overloaded, and part of the data streams to be transmitted still exceed the system load, so that the current limitation needs to be performed on the part of the data streams to be transmitted which exceed the system load.
And step S350, limiting the data stream to be transmitted according to a second current limiting mode under the condition that the data stream to be transmitted meets the condition of limiting the current according to the second current limiting mode.
In the embodiment of the application, after determining that a part of data streams to be transmitted exceeds the system load, limiting the part of data streams to be transmitted according to a second current limiting mode.
The data stream processing method provided by the embodiment of the present application may be applied to a throttling scheme in the related art (for example, a token bucket, a leaky bucket, a counter algorithm, and the like), and taking an improvement on a token bucket throttling algorithm as an example, the step S330 may include the following sub-steps:
in this embodiment, the second current limiting manner is a token bucket current limiting manner.
And a substep S331 of generating tokens at a preset rate and putting the tokens into a token bucket, wherein the generation of the tokens is stopped after the total number of tokens contained in the token bucket reaches an upper limit.
In this embodiment of the present application, the system continuously generates tokens, puts the tokens into a token bucket, and stops generating the tokens after the capacity of the token bucket reaches an upper limit, where a data stream needs to be transmitted to obtain the tokens from the token bucket so as to transmit the data stream, and each data stream is transmitted to consume the tokens in the token bucket. When the tokens in the token bucket are insufficient, the number of the data streams to be transmitted is represented to be overloaded, and the data streams to be transmitted need to be limited.
And a substep S332, detecting whether a token is obtained from the token bucket for the data stream to be transmitted.
In the embodiment of the application, the data stream to be transmitted can be detected in real time, and whether the token can be acquired from the token bucket is determined so as to determine whether the data stream to be transmitted meets the condition of current limiting according to the second current limiting mode. If the token cannot be obtained for the data stream to be transmitted, it indicates that the data stream to be transmitted needs to be limited.
In an optional implementation manner, the data stream processing method further includes a step of updating a current limit threshold, which specifically includes:
the method comprises the following steps of S1, with unit time as a period, periodically updating a current limiting threshold value according to the number of all data streams to be transmitted in unit time, the number of the data streams to be transmitted which are triggered to carry out current limiting according to a first current limiting mode in unit time, and the number of the data streams to be transmitted which are triggered to carry out current limiting according to a second current limiting mode in unit time.
In this embodiment of the present application, the current limit threshold may also be periodically updated, specifically including: and counting the number of all data streams to be transmitted, the number of the data streams to be transmitted which are subjected to current limiting in a first current limiting mode and the number of the data streams to be transmitted which are subjected to current limiting in a second current limiting amplifying mode in a unit time period. And updating the current limiting threshold according to the three values so as to realize that all data streams to be transmitted which need to be limited are limited in current by a first current limiting mode. And the current limiting threshold is periodically updated, so that all data streams to be transmitted exceeding the server load are subjected to differential current limiting.
In the embodiment of the present application, after the current limiting threshold is periodically updated, the data stream to be transmitted may be processed according to a size relationship between the preference of the user for the target object and the current limiting threshold that is updated last time.
S2, comparing the number of the data streams to be transmitted which are triggered to carry out current limiting according to a second current limiting mode in unit time with a preset threshold value;
in the embodiment of the application, the preset threshold is preset by a developer according to historical experience, and may be 0, and when the preset threshold is 0, the preset threshold is used to determine whether there is a data stream to be transmitted that triggers current limiting according to the second current limiting mode.
S3, under the condition that the number of the data streams to be transmitted, which are triggered to be subjected to current limiting according to the second current limiting mode in unit time, is smaller than the preset threshold value, judging whether the number of all the data streams to be transmitted in two adjacent unit times is reduced or not;
in this embodiment of the application, when the preset threshold is 0, and when it is determined that the number of the data streams to be transmitted, which trigger the second current limiting mode for current limiting, is less than the preset threshold, it is indicated that the number of the data streams to be transmitted, which trigger the second current limiting mode for current limiting, is 0, that is, all the current-limited data streams to be transmitted are current-limited in the first current limiting mode. At this time, whether the number of all data streams to be transmitted in two adjacent unit times is reduced is judged.
And S4, under the condition that the number of all data streams to be transmitted in two adjacent unit times is not reduced, stopping updating the current limiting threshold.
In the embodiment of the application, when the number of the data streams to be transmitted, which trigger the second current limiting mode, is smaller than the preset threshold and the number of all the data streams to be transmitted in two adjacent unit times is not reduced, it is indicated that the updating of the current limiting threshold approaches the steady state, and the application server achieves intelligent differential current limiting on all the data streams to be transmitted, which exceed the system load and need current limiting, so as to achieve the purpose of current limiting on all the data streams to be transmitted, which need current limiting, according to the preference of a user on a target object.
In an optional implementation manner, the data stream processing method further includes a current limit threshold attenuation step, specifically including:
under the condition that the number of all data streams to be transmitted in two adjacent unit times is reduced, attenuating the current limiting threshold value which is updated for the last time according to a preset attenuation ratio;
in the embodiment of the application, when the number of the data streams to be transmitted, which are triggered to be subjected to current limiting according to the second current limiting mode in unit time, is smaller than the preset threshold, and when the number of all the data streams to be transmitted in two adjacent unit times is reduced, it is indicated that the load of the server is reduced at this time, the current limiting threshold is large, and excessive data streams to be transmitted are subjected to current limiting. In order to improve the overall user experience of the application server, the current limit threshold needs to be attenuated at this time.
In an embodiment of the present application, the attenuating step comprises: specifically, the developer presets the attenuation ratio in advance according to historical experience, for example, the attenuation ratio may be set to 0.5, and according to the attenuation ratio, the current limiting threshold updated most recently is attenuated, so that the number of data streams to be transmitted which trigger the first current limiting mode is reduced, and the application server can provide services for more users.
Referring to fig. 4, an exemplary diagram of a data stream processing method is shown, in which a target object is used as an advertisement click rate for explanation, and the method specifically includes:
first, the advertisement click rates of all users are counted offline, and the ranking is performed, and if the ranking is P total gears from low to high [ 1,P ], in the present application, the current limiting threshold (i.e., the admission threshold) may be a gear value.
After a user initiates a data stream request, a gear to which the advertisement click rate belongs can be determined according to user information, and whether the data stream to be transmitted of the user needs to be limited or not is determined according to the gear value to which the advertisement click rate belongs and the gear value of the admission threshold. Specifically, if the advertisement click rate of the user is not greater than the admission threshold, the flow limit 1 is actively triggered, that is, the server actively limits the data stream to be transmitted of a part of users with lower advertisement click rate according to the advertisement click rate of the user; if the advertisement click rate is larger than the admission threshold value, a token is tried to be acquired for the data stream to be transmitted of the user, and after acquisition fails, the flow limitation 2 is triggered, namely that after the application server performs active flow limitation, part of the data stream to be transmitted still exceeds the system load, and passive undifferentiated flow limitation needs to be performed.
In practical application, the current limit threshold value needs to be periodically updated, and specifically, the ratio of the amount of all the current-limited data streams to be transmitted to the total amount of the data streams to be transmitted in unit time can be obtained according to the ratio of the sum of the amount of the data streams to be transmitted which trigger the current limit 1, the sum of the amount of the data streams to be transmitted which trigger the current limit 2, and the total amount of the data to be transmitted in unit time. And then according to the advertisement click rate gear divided in advance, calculating to obtain the gear to which the data stream to be transmitted needing to be limited belongs. And the gear value obtained by calculation is utilized to update the current limiting threshold value, so that all data streams to be transmitted, which exceed the system load and need to be limited, are limited by the current limiting 1, and the purpose of differential intelligent current limiting is achieved.
Specifically, it can be expressed as the following formula:
wherein the current limit Threshold value in the next unit time period is represented as Threshold t+1 The number of data streams to be transmitted that trigger the current limit 1 is denoted Q1 t Trigger limitThe number of data streams to be transmitted for stream 2 is denoted as Q2 t The total data volume to be transmitted in the current unit time period t is represented as Q t And P is the number of gears divided in advance.
For example, when Q is 1t ＝2000，Q2 t ＝100，Q t 10000, p is 100, that is, in the current unit time period of the system, 2% of data to be transmitted is limited by the current limit 1, and 1% of data to be transmitted is limited by the current limit 2; the current limit Threshold in the next unit time period t+1 The update is required to be:
that is, in the next unit time period, the data stream to be transmitted of the user with the grade lower than 3 is actively limited by the system through the current limit 1.
Referring to fig. 5, a block diagram of a data stream processing apparatus 500 according to the present application is shown, specifically, the data stream processing apparatus 500 may include the following modules:
a data stream acquiring module 501, configured to acquire a data stream to be transmitted when receiving a data stream request sent by a user terminal;
a preference prediction module 502, configured to predict a preference of a user using the user terminal for a target object in the data stream to be transmitted;
and the data stream processing module 503 is configured to process the data stream to be transmitted according to a size relationship between the preference and the current limit threshold.
Optionally, the data stream processing module 503 includes:
the current limiting submodule is used for limiting the current of the data stream to be transmitted according to a first current limiting mode under the condition that the preference degree is smaller than a current limiting threshold value;
and the transmission submodule is used for transmitting the data stream to be transmitted to the user terminal under the condition that the preference degree is not smaller than the current limiting threshold value.
Optionally, the data stream processing module 503 includes:
the first current limiting submodule is used for limiting the current of the data stream to be transmitted under the condition that the preference degree is smaller than a current limiting threshold value;
the detection submodule is used for detecting whether the data stream to be transmitted meets the condition of current limiting according to a second current limiting mode under the condition that the preference degree is not smaller than the current limiting threshold value;
the first transmission submodule is used for transmitting the data stream to be transmitted to the user terminal under the condition that the data stream to be transmitted does not meet the condition of current limiting according to a second current limiting mode;
and the second current limiting submodule is used for limiting the current of the data stream to be transmitted according to a second current limiting mode under the condition that the data stream to be transmitted meets the condition of limiting the current according to the second current limiting mode.
Optionally, the apparatus further comprises:
the updating module is used for periodically updating the current limiting threshold value according to the number of all data streams to be transmitted in unit time, the number of the data streams to be transmitted which are triggered to carry out current limiting according to a first current limiting mode in unit time and the number of the data streams to be transmitted which are triggered to carry out current limiting according to a second current limiting mode in unit time by taking unit time as a period;
the data stream processing module 503 includes:
and the first processing submodule is used for processing the data stream to be transmitted according to the magnitude relation between the preference degree and the current limiting threshold value which is updated last time.
Optionally, the apparatus further comprises:
the comparison module is used for comparing the number of the data streams to be transmitted which are triggered to be subjected to current limiting in the unit time in the second current limiting mode with a preset threshold value;
the judging module is used for judging whether the number of all data streams to be transmitted in two adjacent unit times is reduced or not under the condition that the number of the data streams to be transmitted which are triggered to carry out current limiting according to the second current limiting mode in the unit time is smaller than the preset threshold value;
and the stopping module is used for stopping updating the current limiting threshold under the condition that the number of all the data streams to be transmitted in two adjacent unit times is not reduced.
Optionally, the method further comprises:
the attenuation module is used for attenuating the current limiting threshold value which is updated last time according to a preset attenuation ratio under the condition that the number of all data streams to be transmitted in two adjacent unit times is reduced;
the data stream processing module 503 includes:
and the second processing submodule is used for processing the data stream to be transmitted according to the magnitude relation between the preference degree and the attenuated current limiting threshold value.
Optionally, the detection submodule includes:
the system comprises a token generating unit, a token generating unit and a token bucket, wherein the token generating unit is used for generating tokens at a preset rate and putting the tokens into the token bucket, and the token generating unit stops generating the tokens after the total number of the tokens contained in the token bucket reaches an upper limit;
and the detection unit is used for detecting whether the data stream to be transmitted acquires tokens from the token bucket.
Optionally, the preference prediction module 502 includes:
an acquisition submodule, configured to acquire historical operation behavior data of a user using the user terminal on a target object in each transmitted data stream, where the historical operation behavior data includes at least one of: click rate, exposure duration;
and the prediction submodule is used for predicting the preference degree of the user using the user terminal to the target object in the data stream to be transmitted according to the historical operation behavior data.
In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the data stream processing method provided in the embodiment of the present application.
For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
Correspondingly, the present application further provides an electronic device, which includes a memory, a processor, and a computer program that is stored in the memory and can be run on the processor, and when the processor executes the computer program, the data stream processing method according to the embodiment of the present application is implemented, and the same technical effect can be achieved, and for avoiding repetition, details are not repeated here. The electronic device can be a PC, a mobile terminal, a personal digital assistant, a tablet computer and the like.
The present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the data stream processing method according to the embodiments of the present application, and can achieve the same technical effects, and details are not repeated here to avoid repetition. The computer-readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.
The embodiments in the present specification are all described in a progressive manner, and each embodiment focuses on differences from other embodiments, and portions that are the same and similar between the embodiments may be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
The data stream processing method, apparatus, electronic device and computer-readable storage medium provided by the present application are described in detail above, and a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the above embodiment is only used to help understand the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment may be implemented by software plus a necessary general hardware platform, and may also be implemented by hardware. Based on such understanding, the above technical solutions substantially or contributing to the related art may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
Claims (11)
1. A method for processing a data stream, the method comprising:
when a data stream request sent by a user terminal is received, acquiring a data stream to be transmitted;
collecting historical operation behavior data of a user using the user terminal on a target object in a transmitted data stream;
predicting the preference degree of a user using the user terminal to a target object in the data stream to be transmitted according to the historical operation behavior data;
processing the data stream to be transmitted according to the size relationship between the preference degree and the current limiting threshold, wherein the processing at least comprises the following steps:
transmitting the data stream to be transmitted to the user terminal under the condition that the preference degree is not smaller than the current limiting threshold value; or
And transmitting the data stream to be transmitted to the user terminal under the conditions that the preference degree is not less than the current limiting threshold value and the data stream to be transmitted does not meet the condition of current limiting according to a second current limiting mode.
2. The method of claim 1, further comprising:
and under the condition that the preference degree is smaller than a current limiting threshold value, limiting the data stream to be transmitted according to a first current limiting mode.
3. The method of claim 1, further comprising:
limiting the current of the data stream to be transmitted under the condition that the preference degree is smaller than a current limiting threshold value;
under the condition that the preference degree is not smaller than the current limiting threshold value, detecting whether the data stream to be transmitted meets the condition of current limiting according to a second current limiting mode;
and under the condition that the data stream to be transmitted meets the condition of current limiting according to a second current limiting mode, limiting the current of the data stream to be transmitted according to the second current limiting mode.
4. The method of claim 3, further comprising:
periodically updating the current limiting threshold value by taking unit time as a period according to the number of all data streams to be transmitted in the unit time, the number of the data streams to be transmitted which are triggered to carry out current limiting according to a first current limiting mode in the unit time and the number of the data streams to be transmitted which are triggered to carry out current limiting according to a second current limiting mode in the unit time;
processing the data stream to be transmitted according to the size relationship between the preference degree and the current limiting threshold, wherein the processing comprises the following steps:
and processing the data stream to be transmitted according to the magnitude relation between the preference degree and the current limiting threshold value which is updated last time.
5. The method of claim 4, further comprising:
comparing the number of the data streams to be transmitted which are triggered to carry out current limiting according to a second current limiting mode in unit time with a preset threshold value;
under the condition that the number of the data streams to be transmitted, which are triggered to be subjected to current limiting according to the second current limiting mode in unit time, is smaller than the preset threshold value, judging whether the number of all the data streams to be transmitted in two adjacent unit times is reduced or not;
and under the condition that the quantity of all the data streams to be transmitted in two adjacent unit times is not reduced, stopping updating the current limiting threshold value.
6. The method of claim 5, further comprising:
under the condition that the number of all data streams to be transmitted in two adjacent unit times is reduced, attenuating the current limit threshold value which is updated last time according to a preset attenuation ratio;
processing the data stream to be transmitted according to the size relationship between the preference degree and the current limiting threshold, wherein the processing comprises the following steps:
and processing the data stream to be transmitted according to the size relation between the preference degree and the attenuated current limiting threshold value.
7. The method according to claim 3, wherein detecting whether the data stream to be transmitted meets a condition for limiting the current according to a second current limiting manner comprises:
generating tokens at a preset rate and putting the tokens into a token bucket, wherein the generation of the tokens is stopped after the total number of tokens contained in the token bucket reaches an upper limit;
and detecting whether the data stream to be transmitted acquires tokens from a token bucket.
8. The method of any of claims 1-7, wherein the historical operational behavior data comprises at least one of: click rate, exposure duration.
9. A data stream processing apparatus, characterized in that the apparatus comprises:
the data stream acquisition module is used for acquiring a data stream to be transmitted when receiving a data stream request sent by a user terminal;
the preference degree prediction module is used for acquiring historical operation behavior data of a user using the user terminal on a target object in a transmitted data stream; predicting the preference degree of a user using the user terminal to a target object in the data stream to be transmitted according to the historical operation behavior data;
a data stream processing module, configured to process the data stream to be transmitted according to a size relationship between the preference and a current limiting threshold, where the data stream processing module at least includes:
transmitting the data stream to be transmitted to the user terminal under the condition that the preference degree is not smaller than the current limiting threshold value; or
And transmitting the data stream to be transmitted to the user terminal under the condition that the preference degree is not less than the current limiting threshold value and the data stream to be transmitted does not meet the condition of current limiting according to a second current limiting mode.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the data stream processing method of any one of claims 1 to 8 when executing the computer program.
11. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the steps of the data stream processing method of one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110732655.8A CN113595921B (en) | 2021-06-29 | 2021-06-29 | Data stream processing method and device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110732655.8A CN113595921B (en) | 2021-06-29 | 2021-06-29 | Data stream processing method and device, electronic equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113595921A CN113595921A (en) | 2021-11-02 |
| CN113595921B true CN113595921B (en) | 2022-12-13 |
Family
ID=78245077
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110732655.8A Active CN113595921B (en) | 2021-06-29 | 2021-06-29 | Data stream processing method and device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113595921B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104518981A (en) * | 2013-09-27 | 2015-04-15 | 深圳市腾讯计算机系统有限公司 | Flow control method and device |
| CN109842565A (en) * | 2018-12-15 | 2019-06-04 | 平安科技（深圳）有限公司 | Interface current-limiting method, device, electronic equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170171585A1 (en) * | 2015-12-15 | 2017-06-15 | Le Holdings (Beijing) Co., Ltd. | Method and Electronic Device for Recording Live Streaming Media |
-
2021
- 2021-06-29 CN CN202110732655.8A patent/CN113595921B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104518981A (en) * | 2013-09-27 | 2015-04-15 | 深圳市腾讯计算机系统有限公司 | Flow control method and device |
| CN109842565A (en) * | 2018-12-15 | 2019-06-04 | 平安科技（深圳）有限公司 | Interface current-limiting method, device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113595921A (en) | 2021-11-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110880984B (en) | Model-based flow anomaly monitoring method, device, equipment and storage medium | |
| CN108337563B (en) | Video evaluation method, device, equipment and storage medium | |
| CN111459783B (en) | Application program optimization method and device, electronic equipment and storage medium | |
| Li-yuan et al. | The research of quality of experience evaluation method in pervasive computing environment | |
| CN112333556B (en) | Control method for monitoring video transmission bandwidth, terminal equipment and readable storage medium | |
| CN108923996B (en) | Capacity analysis method and device | |
| CN108390775B (en) | User experience quality evaluation method and system based on SPICE | |
| CN101662773A (en) | Method and equipment for realizing purpose of reducing communication deception risk by using computer | |
| CN108683605A (en) | A kind of current-limiting method and device of service call | |
| CN109783337B (en) | Model service method, system, apparatus and computer readable storage medium | |
| CN115460153B (en) | Dynamic adjustment method and device for token bucket capacity, storage medium and electronic device | |
| CN111476510A (en) | Method and system for identifying risk user, storage medium and equipment | |
| CN112596985B (en) | IT asset detection method, device, equipment and medium | |
| CN113595921B (en) | Data stream processing method and device, electronic equipment and computer readable storage medium | |
| CN106445715A (en) | Method and device for reporting pedometer messages | |
| CN116566696A (en) | Security assessment system and method based on cloud computing | |
| US10439919B2 (en) | Real time event monitoring and analysis system | |
| CN112187870A (en) | Bandwidth smoothing method and device | |
| CN111278039B (en) | User perception suppression identification method, device, equipment and medium | |
| CN112994934B (en) | Data interaction method, device and system | |
| CN113590325A (en) | Hotel distributor interface current limiting method and device, computer equipment and storage medium | |
| CN112422370B (en) | Method and device for determining voice call quality | |
| CN114896296A (en) | Cloud service resource configuration method and device, electronic equipment and computer readable medium | |
| CN105591980B (en) | A kind of bandwidth consumption management method and device | |
| CN113835905A (en) | Message queue load balancing method and device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |