CN115118782A - Data control method and device, electronic equipment and computer readable storage medium - Google Patents
Data control method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN115118782A CN115118782A CN202210742923.9A CN202210742923A CN115118782A CN 115118782 A CN115118782 A CN 115118782A CN 202210742923 A CN202210742923 A CN 202210742923A CN 115118782 A CN115118782 A CN 115118782A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- access request
- message queue
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/26—Flow control; Congestion control using explicit feedback to the source, e.g. choke packets
- H04L47/263—Rate modification at the source after receiving feedback
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application relates to the technical field of data processing, and provides a data control method, a data control device, electronic equipment and a computer-readable storage medium. The method comprises the following steps: storing the access requests received from any client into a message queue, and pushing each currently stored access request in the message queue to a target server so that the target server acquires corresponding target traffic data from a traffic database for feedback according to the received access requests; and determining that the quantity of the request data currently stored in the message queue reaches a preset value, and suspending receiving an access request from any client until the quantity of the request data is smaller than the preset value. The data control method provided by the embodiment of the application can limit the speed of the external access target server and improve the stability of the external access request interface.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data control method, an apparatus, an electronic device, and a computer-readable storage medium.
Background
With the development of the mobile internet, the storage amount of data is larger and larger. The data level has been developed from TB to PB to ZB, and the types of data are many, and more semi-structured and unstructured data information such as web pages, pictures, video images, etc. are stored. And these data are usually stored in a target server, such as a traffic playback platform, so that when the traffic playback platform receives an external access request from a client, the requested relevant data are fed back to the client.
However, because the relationship between the traffic playback platform and the client is a one-to-many relationship, when the number of accessed clients increases, and the frequency of receiving and recording external requests by the clients increases rapidly, or when the client executes conventional pressure measurement, the external access requests initiated by a plurality of clients to the traffic playback platform are instantaneous and highly concurrent, which causes the external access request interface of the traffic playback platform to crash, and further causes a chain reaction to crash the upstream and downstream of the whole system. Therefore, how to avoid the external access request interface breakdown caused by the too high external request frequency is one of the problems to be solved urgently at present.
Disclosure of Invention
The present application is directed to solving at least one of the technical problems occurring in the related art. Therefore, the data control method is provided, which can limit the speed of the external access target server and improve the stability of the external access request interface.
The application also provides a data control device.
The application also provides an electronic device.
The application also provides a computer readable storage medium.
The data control method according to the embodiment of the first aspect of the application comprises the following steps:
storing access requests received from any client into a message queue, and pushing each access request currently stored in the message queue to a target server so that the target server acquires corresponding target traffic data from a traffic database for feedback according to the received access requests;
and determining that the amount of the request data currently stored in the message queue reaches a preset value, and suspending receiving the access request from any client until the amount of the request data is smaller than the preset value.
According to the data control method provided by the embodiment of the application, the access requests received from any client are stored in the message queue, the currently stored access requests in the message queue are pushed to the target server, and when the quantity of the currently stored request data in the message queue reaches the preset value, the access requests are suspended from any client, so that the access requests are limited by using the storage capacity of the message queue, the rate of receiving the access requests by the target server is limited, the target server is protected from being washed down by the increasing access requests and the burst access requests, the peak clipping of data flow is realized, the stability of an external access request interface is improved, the stability between systems and services is further ensured, the usability of the systems is improved, and the service quality is ensured.
According to an embodiment of the present application, pushing each access request currently stored in the message queue to a target server includes:
and asynchronously pushing each access request currently stored in the message queue to the target server according to a preset pushing rate.
According to an embodiment of the present application, further comprising:
acquiring the average response time of the target server for the received access request;
determining a response time interval to which the average response time belongs from a data table in which mapping relations between a plurality of response time intervals and a plurality of push rates are recorded according to the average response time;
and determining the push rate corresponding to the response time interval as the preset push rate.
According to an embodiment of the present application, after determining that the amount of the request data currently stored in the message queue reaches a preset value, the method further includes:
and sending prompt information for prompting to suspend the access to any client side which sends the access request at the current moment.
According to an embodiment of the present application, further comprising:
determining that the frequency of sending an access request by any client in a fixed time period is greater than a preset frequency, and marking the client as a target client;
and acquiring each historical access request sent by the target client within a historical preset time period, determining that each historical access request is the same, and stopping receiving the access request sent by the target client.
According to an embodiment of the present application, further comprising:
acquiring any traffic data to be stored sent to the target server;
determining that the target traffic data which is the same as the traffic data to be stored does not exist in the traffic database, and marking the traffic data to be stored as the target traffic data to be stored in the traffic database.
According to an embodiment of the present application, the determining that the target traffic data identical to the traffic data to be stored does not exist in the traffic database includes:
extracting a first target field from the traffic data to be stored according to a preset configuration rule;
performing hash calculation on the first target field to obtain a first hash code of the first target field;
matching the first hash code with each second hash code corresponding to each target flow data, and determining that the target flow data same as the to-be-stored flow data does not exist in the flow database under the condition that the second hash code matched with the first hash code does not exist;
the preset configuration rule comprises a main call entry parameter and a main call return value of the flow data;
and after extracting a second target field from the target flow data through the preset configuration rule, the second hash code is obtained by performing hash calculation on the second target field.
The data control device according to the embodiment of the second aspect of the application comprises:
the data receiving module is used for storing the access requests received from any client into a message queue and pushing each access request currently stored in the message queue to a target server so that the target server can obtain corresponding target traffic data from a traffic database for feedback according to the received access requests;
and the data control module is used for determining that the quantity of the request data currently stored in the message queue reaches a preset value, and suspending receiving the access request from any client until the quantity of the request data is smaller than the preset value.
The electronic device according to the third aspect of the present application includes a processor and a memory storing a computer program, and the processor implements the data control method according to any one of the above embodiments when executing the computer program.
A computer-readable storage medium according to a fourth aspect of the present application, having stored thereon a computer program which, when executed by a processor, implements the data control method of any of the above embodiments.
The computer program product according to an embodiment of the fifth aspect of the application comprises: the computer program, when executed by a processor, implements a data control method as in any of the embodiments described above.
One or more technical solutions in the embodiments of the present application have at least one of the following technical effects:
the method comprises the steps of storing access requests received from any client into a message queue, pushing each access request currently stored in the message queue to a target server, and suspending receiving of the access request from any client when the current stored request data volume in the message queue reaches a preset value, so that the access requests are limited by using the storage capacity of the message queue, the rate of receiving the access requests by the target server is limited, the target server is protected from being washed down by increasingly-increased access requests and burst access requests, peak clipping of data streams is realized, the stability of an external access request interface is improved, the stability between systems and between services is further ensured, the availability of the systems is improved, and the service quality is ensured.
Furthermore, data are read from the message queue at a preset push rate, and the operation of asynchronously pushing the access request in the message queue to the target server is executed, so that the smooth processing of the access request recorded in the message queue is realized. Meanwhile, as the access requests in the message queue are pushed according to a certain preset pushing rate, when the message queue is full, if the server still receives a plurality of access requests from the customer service end, the plurality of access requests can be inserted into the message queue according to the preset pushing rate, so that the smooth processing of the access requests added into the message queue is realized.
Furthermore, by obtaining the average response time of the target server to the access request and determining the preset push rate corresponding to the average response time from the data table recorded with the mapping relationship between the multiple response time intervals and the multiple push rates according to the average response time, the processing efficiency of the target server to the access request is prevented from being influenced due to the fact that the access request pushed to the target server is too fast, and the target server can respond to the access request more smoothly.
Furthermore, the frequency of sending the access requests by the client in a fixed time interval is determined to be greater than the preset frequency, and the access requests of the client stop being received when the historical access requests sent by the client in the historical preset time interval are the same, so that the invasion of a large number of malicious access requests to the message queue is avoided, and the processing efficiency of normal access requests is improved.
Furthermore, the traffic data to be stored is matched with each target traffic data in the traffic database, so that when it is determined that the target traffic data which is the same as the traffic data to be stored does not exist in the traffic database, the traffic data to be stored is stored in the traffic database, repeated target traffic data in the traffic database is avoided, and the purpose of data deduplication is achieved, thereby reducing the storage pressure of the target server and improving the processing efficiency of the target server on the access request.
Furthermore, after a first hash code of the first target field is determined through hash calculation, the first hash code of the flow data to be stored is compared with the first hash code of each target flow data to judge whether the flow database has the same target flow data as the flow data to be stored, so that duplication removal of the flow data is refined to field duplication removal of the main call entry and return value levels, the number of fields needing to be matched in the flow data is reduced, and duplication removal efficiency of the flow data is improved.
Drawings
In order to more clearly illustrate the technical solutions in the present application or prior art, the drawings used in the embodiments or the description of the prior art are briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic flowchart of a data control method provided in an embodiment of the present application;
FIG. 2 is a schematic flow chart diagram illustrating a data control method according to another embodiment of the present application;
FIG. 3 is a schematic flow chart diagram illustrating a data control method according to another embodiment of the present application;
fig. 4 is a schematic structural diagram of a data control device according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.
Detailed Description
To make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Hereinafter, the data control method, the data control apparatus, the electronic device, and the computer-readable storage medium according to the embodiments of the present application will be described in detail and explained with reference to several specific embodiments.
In one embodiment, a data control method is provided, and the method is applied to a server and used for data control. The server can be an independent server or a server cluster formed by a plurality of servers, and can also be a cloud server providing basic cloud computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN (content delivery network), big data and artificial intelligence sampling point equipment and the like.
As shown in fig. 1, a data control method provided in this embodiment includes:
and 102, determining that the quantity of the request data currently stored in the message queue reaches a preset value, and suspending receiving the access request from any client until the quantity of the request data is smaller than the preset value.
The method comprises the steps of storing access requests received from any client into a message queue, pushing each access request currently stored in the message queue to a target server, and suspending receiving of the access request from any client when the current stored request data volume in the message queue reaches a preset value, so that the access requests are limited by using the storage capacity of the message queue, the rate of receiving the access requests by the target server is limited, the target server is protected from being washed down by increasingly-increased access requests and burst access requests, peak clipping of data streams is realized, the stability of an external access request interface is improved, the stability between systems and between services is further ensured, the availability of the systems is improved, and the service quality is ensured.
In one embodiment, a server is connected with a plurality of clients and a target server, wherein the target server is a traffic playback platform. If a message queue with a fixed Capacity is arranged in the server, for example, a message queue ArrayBlockingqueue is defined, and the queue Capacity indicates the Capacity size of the message queue. The message queue ArrayBlockingQueue is used for receiving and storing the access requests sent by the client, and simultaneously pushing each currently stored access request of the message queue ArrayBlockingQueue to the target server, so that the target server responds to the received access request and obtains the target traffic data accessed by the access request from the traffic database for feedback.
When receiving an access request of any client, the server detects the number of access requests currently stored in the message queue, namely the request data volume, and judges whether the current request data volume reaches the queue capacity, namely a preset value. If the access request does not reach the preset value, judging that the message queue can also insert the access request at the moment, and inserting the access request into the message queue at the moment; if the current request data volume of the message queue is full, the insertion of the access request is refused, and the received access request is inserted into the message queue again until the access request recorded in the message queue is reduced because of being received by the target server, so that the request data volume in the message queue does not exceed the preset value all the time, and the purpose of limiting the flow is achieved.
For example, assuming that the preset value of the message queue is a preset value, that is, the number of insertable access requests is X, after receiving an access request from a client, the server determines whether the number of the inserted access requests in the current message queue reaches X, if not, the access request is inserted into the message queue, otherwise, the insertion of the access request is rejected. However, since the access requests in the message queue of the server are pushed to the target server, the number of access requests currently inserted in the message queue changes in real time, and at this time, when the access requests in the message queue are pushed to the target server, the recorded access requests are reduced, so that the message queue can receive new access requests again.
In order to make a data interface of a target server not affected by a request data amount recorded in a message queue while throttling access requests, so as to make data transmission more stable, in an embodiment, pushing each access request currently stored in the message queue to the target server includes:
and asynchronously pushing each access request currently stored in the message queue to the target server according to a preset pushing rate.
In an embodiment, the server performs a loop operation on the message queue, reads data from the message queue at a preset push rate, and performs an operation of asynchronously pushing the access request in the message queue to the target server, thereby implementing smooth processing on the access request recorded in the message queue.
Meanwhile, as the access requests in the message queue are pushed according to a certain preset pushing rate, when the message queue is full, if the server still receives a plurality of access requests from the customer service end, the plurality of access requests can be inserted into the message queue according to the preset pushing rate, so that the smooth processing of the access requests added into the message queue is realized.
The preset push rate can be set according to actual conditions. In order to make the response of the access request more smooth, in an embodiment, the determining manner of the preset push request includes:
according to the average response time, determining a response time interval to which the average response time belongs from a data table in which mapping relations between a plurality of response time intervals and a plurality of push rates are recorded;
and determining the push rate corresponding to the response time interval as the preset push rate.
In an embodiment, the server stores a data table in advance, and the data table records push rates respectively corresponding to different response time intervals. The server obtains the response time of the target server to each access request, then averages the response times, and determines the average response time. After the average response time is determined, traversing the average response time in a data table recorded with a plurality of response time intervals, determining the response time interval to which the average response time belongs, and determining the push rate corresponding to the response time interval in the data table as the preset push rate.
By acquiring the average response time of the target server to the access request and determining the preset push rate corresponding to the average response time from the data table recorded with the mapping relation between the multiple response time intervals and the multiple push rates according to the average response time, the processing efficiency of the target server to the access request is prevented from being influenced due to the fact that the access request pushed to the target server is too fast, and the target server can respond to the access request more smoothly.
In an embodiment, after determining that the amount of the request data currently stored in the message queue reaches a preset value, the method further includes:
and sending prompt information for prompting the suspension of the access to any client side which sends the access request at the current moment.
When the amount of the request data currently stored in the message queue is full, the prompt information is sent to the client side sending the access request at the current moment to prompt the user to suspend access, so that the user can timely know the reason why the flow data cannot be obtained according to the access request, and the user experience is improved.
Since the access data capacity of the message queue is limited, in order to avoid the encroachment of a large number of malicious access requests on the message queue, in an embodiment, as shown in fig. 2, a flowchart of a data control method provided in another embodiment of the present application is shown. In addition to the steps shown in fig. 1, the method further comprises the following steps:
In an embodiment, each client sending an access request is detected in a fixed time period, such as a peak time period during which the access request is received on the same day, or any specified time period on the same day, and the frequency of sending the access request by each client is obtained. Illustratively, the peak period for receiving an access request is 10: m-11: 00a.m takes this period as a fixed period. The peak period of the access request can be determined by obtaining the access request received every hour of the day and then determining the certain hour with the largest number of received access requests as the peak period of the access request. If the frequency of the access request sent by the client is greater than the preset frequency, the client is judged to be possibly a client attacked maliciously, and the client is marked as a target client at the moment. Then, from the history data, each history access request sent by the target client in a history preset time period, such as a peak period of the server receiving the access request in the previous day, is obtained. If the historical access requests are the same, the target client is judged to repeatedly initiate the same access request for multiple times, and the target client can be judged to be a suspicious client at the moment, so that the access request sent by the target client is stopped being received.
By determining that the frequency of sending the access requests by the client in the fixed time period is greater than the preset frequency and that the historical access requests sent by the client in the historical preset time period are the same, the access requests of the client are stopped from being received, so that the invasion of a large number of malicious access requests to a message queue is avoided, and the processing efficiency of normal access requests is improved.
Considering that the target server may run for several months, it is necessary to use a storage space of up to several hundred TBs to store semi-structured and unstructured traffic data such as web pages, pictures, video images, etc., the amount of stored data is huge, and the valuable data therein is less than a dozen G, that is, a large amount of repeated data occupies storage, so that the storage space of the target server is wasted, and the processing efficiency of the access request is reduced as the amount of stored data increases. Therefore, in order to reduce the storage pressure of the target server and improve the processing efficiency of the target server for the access request, in an embodiment, as shown in fig. 3, in addition to the steps shown in fig. 1 or fig. 2, the method further includes:
The traffic data to be stored is the traffic data to be stored to the target server. When the server acquires the traffic data to be stored which is sent to the target server, matching all fields of the traffic data to be stored with all fields of any target traffic data stored in a traffic database of the target server, if the target traffic data which is identical to the fields of the traffic data to be stored exists in the traffic database, judging that the traffic data to be stored is repeated traffic data, and filtering the traffic data to be stored; otherwise, the traffic data to be stored is stored to the traffic database of the target server as target traffic data.
The flow data to be stored is matched with each target flow data in the flow database, so that when the flow database does not have the target flow data which is the same as the flow data to be stored, the flow data to be stored is stored in the flow database, repeated target flow data in the flow database is avoided, the purpose of data deduplication is achieved, the storage pressure of a target server is reduced, and the processing efficiency of the target server on an access request is improved.
In order to improve deduplication efficiency of traffic data, in an embodiment, the determining that the target traffic data that is the same as the traffic data to be stored does not exist in the traffic database includes:
extracting a first target field from the traffic data to be stored according to a preset configuration rule;
performing hash calculation on the first target field to obtain a first hash code of the first target field;
matching the first hash code with each second hash code corresponding to each target flow data, and determining that the target flow data same as the to-be-stored flow data does not exist in the flow database under the condition that the second hash code matched with the first hash code does not exist;
the preset configuration rule comprises a main call entry parameter and a main call return value of the flow data;
and after extracting a second target field from the target flow data through the preset configuration rule, the second hash code is obtained by performing hash calculation on the second target field.
In an embodiment, since a structure of one piece of traffic data includes a master call entry parameter and a master call return value, after the traffic data to be stored is obtained, a preset configuration rule is entered for deduplication, that is, the traffic data to be stored is analyzed first, after the master call entry parameter and the master call return value in the traffic data to be stored are obtained, corresponding relevant field information is extracted from the obtained relevant field information, and the corresponding field and the collected master call entry parameter and master call return value are combined into a standard Json object, that is, a first target field. And then carrying out hash calculation on the combined JSON object by utilizing a hash algorithm added with the disturbance parameters to obtain a first hash code of the first target field, and storing the first hash code in a bitmap data structure of Redis.
Considering that the traffic data may further include one or n (n >1) sub-calls, each of which is composed of a sub-call entry value and a sub-call return value, the preset configuration rule may further include the entry value and the return value of the sub-call. Correspondingly, the first target field is: after main call incoming parameters, main call return values, sub call incoming parameters and sub call return values in the flow data to be stored are obtained, corresponding relevant field information is extracted from the main call incoming parameters, the sub call incoming parameters and the sub call return values, and then the corresponding fields and the collected main call incoming parameters and the collected main call return values are combined to form a standard Json object.
Similarly, the second hash codes of the target traffic data can be obtained by obtaining the first hash codes, and then the first hash codes are compared with the second hash codes. If the second hash codes with the same first hash code exist in the second hash codes, the flow data to be stored is repeated, and the flow data to be stored is discarded, so that the aim of duplicate removal is fulfilled; otherwise, the target flow data which is not repeated with the flow data to be stored is represented, and the flow data to be stored is stored in the flow database.
The method comprises the steps of extracting a first target field of flow data to be stored through a preset configuration rule comprising a main call input parameter and a return value, comparing the first hash code of the flow data to be stored with the first hash code of each target flow data after determining the first hash code of the first target field through hash calculation to judge whether the target flow data same as the flow data to be stored exists in a flow database, and accordingly, the duplication elimination of the flow data is refined to the field duplication elimination of the main call input parameter and the return value level, the number of fields needing to be matched in the flow data is reduced, and the duplication elimination efficiency of the flow data is improved.
The data control device provided by the present application is described below, and the data control device described below and the data control method described above may be referred to in correspondence with each other.
In one embodiment, as shown in fig. 4, there is provided a data control apparatus including:
the data receiving module 210 is configured to store access requests received from any client into a message queue, and push each currently stored access request in the message queue to a target server, so that the target server obtains corresponding target traffic data from a traffic database according to the received access request to perform feedback;
a data control module 220, configured to determine that the amount of requested data currently stored in the message queue reaches a preset value, and suspend receiving the access request from any one of the clients until the amount of requested data is smaller than the preset value.
The method comprises the steps of storing access requests received from any client into a message queue, pushing each access request currently stored in the message queue to a target server, and suspending receiving of the access request from any client when the current stored request data volume in the message queue reaches a preset value, so that the access requests are limited by using the storage capacity of the message queue, the rate of receiving the access requests by the target server is limited, the target server is protected from being washed away by increasing access requests and burst access requests, peak clipping of data streams is realized, the stability of an external access request interface is improved, the stability between systems and services is further ensured, the availability of the systems is improved, and the service quality is ensured.
In an embodiment, the data receiving module 210 is specifically configured to:
and asynchronously pushing each access request currently stored in the message queue to the target server according to a preset pushing rate.
In an embodiment, the data receiving module 210 is further configured to:
acquiring the average response time of the target server for the received access request;
determining a response time interval to which the average response time belongs from a data table in which mapping relations between a plurality of response time intervals and a plurality of push rates are recorded according to the average response time;
and determining the push rate corresponding to the response time interval as the preset push rate.
In one embodiment, the data control module 220 is further configured to:
and after determining that the amount of the request data currently stored in the message queue reaches a preset value, sending prompt information for prompting the suspension of access to any client of the access request at the current moment.
In one embodiment, the data control module 220 is further configured to:
determining that the frequency of sending an access request by any client in a fixed time period is greater than a preset frequency, and marking the client as a target client;
and acquiring each historical access request sent by the target client within a historical preset time period, determining that each historical access request is the same, and stopping receiving the access request sent by the target client.
In one embodiment, the data control module 220 is further configured to:
acquiring any traffic data to be stored sent to the target server;
determining that the target traffic data which is the same as the traffic data to be stored does not exist in the traffic database, and marking the traffic data to be stored as the target traffic data to be stored in the traffic database.
In an embodiment, the data control module 220 is specifically configured to:
extracting a first target field from the traffic data to be stored according to a preset configuration rule;
performing hash calculation on the first target field to obtain a first hash code of the first target field;
matching the first hash code with each second hash code corresponding to each target flow data, and determining that the target flow data same as the to-be-stored flow data does not exist in the flow database under the condition that the second hash code matched with the first hash code does not exist;
the preset configuration rule comprises a main call entry parameter and a main call return value of the flow data;
and after extracting a second target field from the target flow data through the preset configuration rule, the second hash code is obtained by performing hash calculation on the second target field.
Fig. 5 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 5: a processor (processor)810, a Communication Interface 820, a memory 830 and a Communication bus 840, wherein the processor 810, the Communication Interface 820 and the memory 830 communicate with each other via the Communication bus 840. The processor 810 may invoke computer programs in the memory 830 to perform data control methods, including, for example:
storing access requests received from any client into a message queue, and pushing each access request currently stored in the message queue to a target server so that the target server acquires corresponding target traffic data from a traffic database for feedback according to the received access requests;
and determining that the amount of the request data currently stored in the message queue reaches a preset value, and suspending receiving the access request from any client until the amount of the request data is smaller than the preset value.
In addition, the logic instructions in the memory 830 may be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, an embodiment of the present application further provides a storage medium, where the storage medium includes a computer program, where the computer program is storable on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, a computer can execute the data control method provided in the foregoing embodiments, for example, the method includes:
storing access requests received from any client into a message queue, and pushing each access request currently stored in the message queue to a target server so that the target server acquires corresponding target traffic data from a traffic database for feedback according to the received access requests;
and determining that the amount of the request data currently stored in the message queue reaches a preset value, and suspending receiving the access request from any client until the amount of the request data is smaller than the preset value.
On the other hand, embodiments of the present application further provide a processor-readable storage medium, where a computer program is stored in the processor-readable storage medium, where the computer program is configured to cause a processor to execute the method provided in each of the above embodiments, for example, the method includes:
storing access requests received from any client into a message queue, and pushing each access request currently stored in the message queue to a target server so that the target server acquires corresponding target traffic data from a traffic database for feedback according to the received access requests;
and determining that the amount of the request data currently stored in the message queue reaches a preset value, and suspending receiving the access request from any client until the amount of the request data is smaller than the preset value.
The processor-readable storage medium can be any available medium or data storage device that can be accessed by a processor, including but not limited to magnetic memory (e.g., floppy disks, hard disks, magnetic tape, magneto-optical disks (MOs), etc.), optical memory (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor memory (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND FLASH), Solid State Disks (SSDs)), etc.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (10)
1. A data control method, comprising:
storing access requests received from any client into a message queue, and pushing each access request currently stored in the message queue to a target server so that the target server acquires corresponding target traffic data from a traffic database for feedback according to the received access requests;
and determining that the amount of the request data currently stored in the message queue reaches a preset value, and suspending receiving the access request from any client until the amount of the request data is smaller than the preset value.
2. The data control method of claim 1, wherein pushing each access request currently stored in the message queue to a target server comprises:
and asynchronously pushing each access request currently stored in the message queue to the target server according to a preset pushing rate.
3. The data control method according to claim 2, further comprising:
acquiring the average response time of the target server for the received access request;
determining a response time interval to which the average response time belongs from a data table in which mapping relations between a plurality of response time intervals and a plurality of push rates are recorded according to the average response time;
and determining the push rate corresponding to the response time interval as the preset push rate.
4. The data control method according to claim 1, wherein after determining that the amount of requested data currently stored in the message queue reaches a preset value, the method further comprises:
and sending prompt information for prompting the suspension of the access to any client side which sends the access request at the current moment.
5. The data control method according to claim 1, further comprising:
determining that the frequency of sending an access request by any client in a fixed time period is greater than a preset frequency, and marking the client as a target client;
and acquiring each historical access request sent by the target client within a historical preset time period, determining that each historical access request is the same, and stopping receiving the access request sent by the target client.
6. The data control method according to any one of claims 1 to 5, characterized by further comprising:
acquiring any traffic data to be stored sent to the target server;
determining that the target traffic data which is the same as the traffic data to be stored does not exist in the traffic database, and marking the traffic data to be stored as the target traffic data to be stored in the traffic database.
7. The data control method of claim 6, wherein the determining that the target traffic data that is the same as the pending traffic data does not exist in the traffic database comprises:
extracting a first target field from the traffic data to be stored according to a preset configuration rule;
performing hash calculation on the first target field to obtain a first hash code of the first target field;
matching the first hash code with each second hash code corresponding to each target flow data, and determining that the target flow data same as the to-be-stored flow data does not exist in the flow database under the condition that the second hash code matched with the first hash code does not exist;
the preset configuration rule comprises a main call entry parameter and a main call return value of the flow data;
and after extracting a second target field from the target flow data through the preset configuration rule, the second hash code is obtained by performing hash calculation on the second target field.
8. A data control apparatus, comprising:
the data receiving module is used for storing the access requests received from any client into a message queue and pushing each access request currently stored in the message queue to a target server so that the target server can obtain corresponding target traffic data from a traffic database for feedback according to the received access requests;
and the data control module is used for determining that the quantity of the request data currently stored in the message queue reaches a preset value, and suspending receiving the access request from any client until the quantity of the request data is smaller than the preset value.
9. An electronic device comprising a processor and a memory storing a computer program, wherein the processor implements the data control method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the data control method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210742923.9A CN115118782B (en) | 2022-06-27 | 2022-06-27 | Data control method, device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210742923.9A CN115118782B (en) | 2022-06-27 | 2022-06-27 | Data control method, device, electronic equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115118782A true CN115118782A (en) | 2022-09-27 |
| CN115118782B CN115118782B (en) | 2023-08-22 |
Family
ID=83330962
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210742923.9A Active CN115118782B (en) | 2022-06-27 | 2022-06-27 | Data control method, device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115118782B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118260236A (en) * | 2024-03-29 | 2024-06-28 | 北京微核芯科技有限公司 | Data access request sending method, device and equipment for multi-core processor |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2008092404A1 (en) * | 2007-01-25 | 2008-08-07 | Huawei Technologies Co., Ltd. | An output queue-based flow control realization method and apparatus |
| CN107733812A (en) * | 2017-11-02 | 2018-02-23 | 东软集团股份有限公司 | A kind of data packet dispatching method, device and equipment |
| CN112104731A (en) * | 2020-09-11 | 2020-12-18 | 北京奇艺世纪科技有限公司 | Request processing method and device, electronic equipment and storage medium |
-
2022
- 2022-06-27 CN CN202210742923.9A patent/CN115118782B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2008092404A1 (en) * | 2007-01-25 | 2008-08-07 | Huawei Technologies Co., Ltd. | An output queue-based flow control realization method and apparatus |
| CN107733812A (en) * | 2017-11-02 | 2018-02-23 | 东软集团股份有限公司 | A kind of data packet dispatching method, device and equipment |
| CN112104731A (en) * | 2020-09-11 | 2020-12-18 | 北京奇艺世纪科技有限公司 | Request processing method and device, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 刘静涛 等: "云存储分级服务缓冲队列技术", 指挥信息系统与技术, no. 05 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118260236A (en) * | 2024-03-29 | 2024-06-28 | 北京微核芯科技有限公司 | Data access request sending method, device and equipment for multi-core processor |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115118782B (en) | 2023-08-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112165436B (en) | Flow control method, device and system | |
| CN111756644B (en) | Hot spot current limiting method, system, equipment and storage medium | |
| CN108540533B (en) | Request answering method and device | |
| CN113568756B (en) | Cooperative dynamic scheduling method and system for password resources | |
| CN113794650B (en) | Concurrent request processing method, computer device and computer readable storage medium | |
| CN111371772A (en) | Intelligent gateway current limiting method and system based on redis and computer equipment | |
| CN111159195A (en) | Data storage control method and equipment in block chain system | |
| CN112052088A (en) | Adaptive process CPU resource limitation method, device, terminal and storage medium | |
| CN113238861A (en) | Task execution method and device | |
| CN115118782B (en) | Data control method, device, electronic equipment and computer readable storage medium | |
| CN113517985A (en) | File data processing method and device, electronic equipment and computer readable medium | |
| CN113687781A (en) | Method, device, equipment and medium for pulling up thermal data | |
| CN106612263B (en) | Method and equipment for processing application access request | |
| CN113852490A (en) | Method, device and equipment for realizing quality of service (QoS) of samba user in distributed storage system | |
| CN111193700B (en) | Safety protection method, safety protection device and storage medium | |
| CN108400884A (en) | A kind of method, system and device improving website service stability | |
| CN112969172A (en) | Communication flow control method based on cloud mobile phone | |
| WO2023082765A1 (en) | Server state control method and system, and storage medium | |
| CN114374657B (en) | Data processing method and device | |
| CN110781500A (en) | Data wind control system and method | |
| CN116915606A (en) | Fusing method and device, electronic equipment and storage medium | |
| CN115051867A (en) | Detection method and device for illegal external connection behaviors, electronic equipment and medium | |
| CN115002033A (en) | Flow control method, device, equipment, storage medium and computer product | |
| CN114827157A (en) | Cluster task processing method, device and system, electronic equipment and readable medium | |
| CN109903022B (en) | Resource distribution method, device, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |