CN111782732A - Data processing method and device and computer readable storage medium - Google Patents

Data processing method and device and computer readable storage medium Download PDF

Info

Publication number
CN111782732A
CN111782732A CN202010703197.0A CN202010703197A CN111782732A CN 111782732 A CN111782732 A CN 111782732A CN 202010703197 A CN202010703197 A CN 202010703197A CN 111782732 A CN111782732 A CN 111782732A
Authority
CN
China
Prior art keywords
key
keys
target item
consumption
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010703197.0A
Other languages
Chinese (zh)
Inventor
朱龙云
袁文頔
李子达
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Tencent Domain Computer Network Co Ltd
Original Assignee
Shenzhen Tencent Domain Computer Network Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Tencent Domain Computer Network Co Ltd filed Critical Shenzhen Tencent Domain Computer Network Co Ltd
Priority to CN202010703197.0A priority Critical patent/CN111782732A/en
Publication of CN111782732A publication Critical patent/CN111782732A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems

Abstract

The embodiment of the application provides a data processing method, a data processing device and a computer readable storage medium, wherein the method comprises the following steps: receiving an acquisition request of a target object sent by terminal equipment; determining a target key from a plurality of keys corresponding to a target item in response to an acquisition request, wherein each key in the plurality of keys is used for correspondingly recording the consumption and a consumption threshold of the target item on each key, and the sum of the consumption thresholds recorded corresponding to the plurality of keys is a preset total consumption threshold of the target item; acquiring a first consumption and a first consumption threshold of a target item on a target key, which are recorded corresponding to the target key, from a storage server; if the first consumption is less than the first consumption threshold, determining that the request is passed. By the method, data access can be uniformly distributed to a plurality of keys when the acquisition request of the article is processed, and hot data which are frequently read in the storage server are avoided, so that parallel capacity expansion of the interface service is facilitated.

Description

Data processing method and device and computer readable storage medium
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus, and a computer-readable storage medium.
Background
A hotspot is data that is accessed at high frequency in a distributed environment. When a large number of access requests for a certain data occur in the network, the data can become a hot spot. The storage service may use a Key (Key) to record some data, for example, record the total consumption of an item, and the calculation processing for the total consumption is performed by a corresponding calculation unit (e.g., CPU core unit).
At present, for a scenario with a limited total consumption, when an interface server receives an acquisition request for a certain item, the interface server needs to access the total consumption recorded by a key corresponding to the item from a storage service. When the access request is too much, the data of the total consumption recorded by the key becomes hot data, and is frequently accessed, and the load of the computing unit corresponding to the key is too large, which may cause the system to have a large processing delay, even a processing failure. Therefore, how to avoid hot data in the storage service when performing data access is a problem that needs to be solved urgently at present.
Disclosure of Invention
Embodiments of the present invention provide a data processing method, an apparatus, and a computer-readable storage medium, which can implement uniform distribution of data access to a plurality of keys when processing an acquisition request of an article, and avoid occurrence of frequently read hot data in a storage server, thereby facilitating parallel capacity expansion of an interface service.
In a first aspect, an embodiment of the present application provides a data processing method, which is applied to an interface server included in an interface service cluster, and the method includes:
receiving an acquisition request of a target object sent by terminal equipment;
determining a target key from a plurality of keys corresponding to a target item in response to an acquisition request, wherein each key in the plurality of keys is used for correspondingly recording the consumption and a consumption threshold of the target item on each key, and the sum of the consumption thresholds recorded corresponding to the plurality of keys is a preset total consumption threshold of the target item;
acquiring a first consumption and a first consumption threshold of a target item on a target key, which are recorded corresponding to the target key, from a storage server;
if the first consumption is less than the first consumption threshold, determining that the request is passed.
In a second aspect, an embodiment of the present application provides a data processing method, which is applied to a control server, and the method includes:
acquiring performance parameters configured by a management system and configuration parameters of a target article, wherein the performance parameters comprise the maximum concurrency request quantity of an interface service cluster and the maximum concurrency number of a storage server per second, and the configuration parameters comprise a total consumption threshold value of the target article;
determining a plurality of keys corresponding to a target item and consumption thresholds of the target item corresponding to each key in the plurality of keys according to the performance parameters and the configuration parameters, wherein the sum of the consumption thresholds corresponding to the plurality of keys is a preset total consumption threshold of the target item;
and sending a plurality of keys corresponding to the target items to an interface server included in the interface service cluster, and sending the consumption threshold of the target items corresponding to each key to a storage server, so that the storage server stores the consumption threshold of the target items corresponding to each key.
In a third aspect, an embodiment of the present application provides a data processing apparatus, which is applied to an interface server included in an interface service cluster, and the apparatus includes:
the receiving and sending unit is used for receiving an acquisition request of a target object sent by the terminal equipment;
the processing unit is used for responding to the acquisition request to determine a target key from a plurality of keys corresponding to a target item, wherein each key in the plurality of keys is used for correspondingly recording the consumption and consumption threshold values of the target item on each key, and the sum of the consumption threshold values recorded corresponding to the plurality of keys is a preset total consumption threshold value of the target item;
the acquisition unit is used for acquiring a first consumption and a first consumption threshold of the target item on the target key, which are recorded corresponding to the target key, from the storage server;
the processing unit is further configured to determine that the request is passed if the first consumption is less than the first consumption threshold.
In a fourth aspect, an embodiment of the present application provides a data processing apparatus, which is applied to a control server, and includes:
the system comprises an acquisition unit, a storage unit and a management unit, wherein the acquisition unit is used for acquiring performance parameters configured by a management system and configuration parameters of a target article, the performance parameters comprise the maximum concurrency request quantity of an interface service cluster and the maximum concurrency number per second of a storage server, and the configuration parameters comprise a total consumption threshold value of the target article;
the processing unit is used for determining a plurality of keys corresponding to a target article and consumption thresholds of the target article corresponding to each key in the plurality of keys according to the performance parameters and the configuration parameters, wherein the sum of the consumption thresholds corresponding to the plurality of keys is a preset total consumption threshold of the target article;
and the transceiving unit is used for sending a plurality of keys corresponding to the target articles to an interface server included in the interface service cluster, and sending the consumption threshold of the target articles corresponding to each key to the storage server, so that the storage server stores the consumption threshold of the target articles corresponding to each key.
In a fifth aspect, embodiments of the present application provide a data processing apparatus, which includes a processor and a memory, where the processor and the memory are connected to each other, where the memory is used to store a computer program, and the computer program includes program instructions, and the processor is configured to call the program instructions to execute the data processing method described in the first aspect, or the data processing method described in the second aspect.
In a sixth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, the computer program including program instructions that, when executed by a processor, cause the processor to execute the data processing method described in the first aspect and the data processing method described in the second aspect.
In a seventh aspect, the present application provides a computer program product or a computer program, where the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the data processing method provided in the various alternatives of the first aspect or the second aspect described above.
In the embodiment of the application, an interface server in an interface service cluster receives an acquisition request of a target article sent by a terminal device; determining a target key from a plurality of keys corresponding to a target item in response to an acquisition request, wherein each key in the plurality of keys is used for correspondingly recording the consumption and a consumption threshold of the target item on each key, and the sum of the consumption thresholds correspondingly recorded by the plurality of keys is a preset total consumption threshold of the target item; acquiring a first consumption and a first consumption threshold of a target item on a target key, which are recorded corresponding to the target key, from a storage server; if the first consumption is less than the first consumption threshold, determining that the request is passed. By the method, data access can be uniformly distributed to a plurality of keys when the acquisition request of the article is processed, and hot data which are frequently read in the storage server are avoided, so that parallel capacity expansion of the interface service is facilitated.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is an architecture diagram of an article management and control system according to an embodiment of the present disclosure;
fig. 2 is a schematic flowchart of a data processing method according to an embodiment of the present application;
FIG. 3 is a schematic view of a plurality of keys corresponding to a target object according to an embodiment of the present disclosure;
fig. 4 is an architecture diagram of another article management and control system according to an embodiment of the present application;
fig. 5 is an architecture diagram of another article management and control system according to an embodiment of the present application;
fig. 6 is a schematic flowchart of another data processing method according to an embodiment of the present application;
FIG. 7 is a schematic diagram of a process for generating a plurality of keys corresponding to a target item according to an embodiment of the present application;
fig. 8 is a schematic flowchart of another data processing method according to an embodiment of the present application;
FIG. 9 is a block diagram of a data processing apparatus according to an embodiment of the present disclosure;
fig. 10 is a simplified schematic diagram of an entity structure of a data processing apparatus according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.
In the prior art, because no control server generates multiple keys for the interface server and the storage server, after receiving an item acquisition request sent by a user, the interface server determines whether the item acquisition request can pass through by acquiring the current consumption and the consumption threshold corresponding to only one Key corresponding to the item stored in the storage server. And each time the interface server receives an article acquisition request, the storage server acquires the current consumption and consumption threshold of the Key for judgment. In addition, the storage server only uses one computing unit to process the Key, and if the article acquisition requests sent by the user are too many within a period of time, the computing unit becomes a bottleneck of the whole system due to limited computing capacity of the computing unit, so that the Key becomes a hot spot, and parallel capacity expansion of the interface service is difficult. There are currently two solutions to this phenomenon.
One of the two methods is to increase the frequency of a computing unit in a storage server or optimize the program performance, but the main frequency of the computing unit has a physical upper limit and the program performance also has a limit value, and both the main frequency and the program performance can only be vertically expanded, and the ascending space is limited.
And secondly, the total quantity limit of the articles is cancelled, the total quantity consumption of the articles is calculated in a near real-time manner through a background, and when the total quantity consumption of the articles reaches the limit, an alarm is given to enable related operators to increase the total quantity limit or stop the article getting activity of the user. However, the alarm after the total consumption of the articles reaches the limit may be delayed or omitted, so that the total dispensing amount of the articles cannot be accurately controlled, and meanwhile, part of system operation risks can be distributed to system users.
Therefore, the above two solutions fail to solve the problem of hotspot keys well.
In order to better understand the embodiments of the present application, the following description refers to the terms and the technical knowledge involved in the embodiments of the present application:
cloud technology (Cloud technology) is based on a general term of network technology, information technology, integration technology, management platform technology, application technology and the like applied in a Cloud computing business model, can form a resource pool, is used as required, and is flexible and convenient. Cloud computing technology will become an important support. Background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture-like websites and more web portals. With the high development and application of the internet industry, each article may have its own identification mark and needs to be transmitted to a background system for logic processing, data in different levels are processed separately, and various industrial data need strong system background support and can only be realized through cloud computing.
A distributed cloud storage system (hereinafter, referred to as a storage system) refers to a storage system that integrates a large number of storage devices (storage devices are also referred to as storage nodes) of different types in a network through application software or application interfaces to cooperatively work by using functions such as cluster application, grid technology, and a distributed storage file system, and provides a data storage function and a service access function to the outside.
At present, a storage method of a storage system is as follows: logical volumes are created, and when created, each logical volume is allocated physical storage space, which may be the disk composition of a certain storage device or of several storage devices. The client stores data on a certain logical volume, that is, the data is stored on a file system, the file system divides the data into a plurality of parts, each part is an object, the object not only contains the data but also contains additional information such as data Identification (ID), the file system writes each object into a physical storage space of the logical volume, and the file system records storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.
The process of allocating physical storage space for the logical volume by the storage system specifically includes: physical storage space is divided in advance into stripes according to a group of capacity measures of objects stored in a logical volume (the measures often have a large margin with respect to the capacity of the actual objects to be stored) and Redundant Array of Independent Disks (RAID), and one logical volume can be understood as one stripe, thereby allocating physical storage space to the logical volume.
In order to better understand the embodiments of the present application, a system architecture to which the embodiments of the present application can be applied is described below.
Referring to fig. 1, fig. 1 is an architecture diagram of an article management and control system according to an embodiment of the present disclosure. As shown in fig. 1, the article management and control system includes a management system, a control server, an interface service cluster, and a storage service. The interface service cluster may include a plurality of interface servers, and fig. 1 includes a first interface server, a second interface server, and a third interface server, where each interface server is configured with a routing module. In practical application, the interface service cluster may include more than three interface servers, and in the embodiment of the present application, the interface service cluster includes three interface servers as an example. The storage server may assign a plurality of keys to the target item, each key for recording consumption of the target item on the key. The storage server may also assign a calculation unit, i.e., "K1/L1" to "Kn/Ln" in the drawing, to each Key of the plurality of keys (keys), respectively. K1-Kn are respectively indexes of each Key, L1-Ln are respectively consumption thresholds configured by each Key in K1-Kn, and the sum of the consumption thresholds (namely L1, L2, … … and Ln) corresponding to the n keys from K1-Kn is a preset total consumption threshold (Lmax) of the target item.
The management system shown in fig. 1 may be a system in which an operator sets parameters such as a total consumption amount threshold (Lmax) of items, a single-user limit amount, a maximum concurrency request amount (Qmax) of an interface service cluster, and a maximum concurrency request amount per second (Pmax) of a storage server for each item. The total article limit amount refers to the total issued amount of a certain article, and limits the maximum issued amount of the article; the single-user limit quantity refers to the quantity of the articles which can be received by one user at most; the maximum concurrent requests per second refers to the maximum number of item requests that the storage server can receive in one second of time. The item can be a prop in a virtual game product or other items or data which can be picked up and accessed. The management system may include an interactive interface for configuring each of the performance parameters and the configuration parameters by obtaining the parameter values entered by the operator for the parameters. Of course, the management system may also configure each performance parameter and configuration parameter by an operator entering a recognizable computer language.
The control server shown in fig. 1 is used to provide control services, and may perform control-type operations on one or more interface servers and storage servers in the interface service cluster. The control server can generate Key according to the performance parameters and configuration parameters configured by the management system and by combining the working capacity of the specific server. Wherein the control service can control the number of generated keys, as well as the threshold of each Key; the storage server can also be controlled to assign a corresponding computing unit to the generated Key. For example, if the control server determines that 10 keys are to be generated according to the performance parameters and the configuration parameters obtained from the management system, the control server may send the 10 keys to each interface server in the interface service cluster, and may further control the storage service to allocate 10 computing units to the 10 keys.
Each interface service in the interface service cluster shown in fig. 1 is configured to receive an item acquisition request initiated by a user, and determine whether the item acquisition request can pass through according to information acquired from a storage service. Specifically, after receiving an article acquisition request sent by a user, the interface service may determine a target Key from a plurality of keys in the storage server through the routing module, and then determine whether the article acquisition request can pass through according to the current consumption and the consumption threshold of the target Key acquired from the storage server, and may evenly distribute the request received by the interface service to a plurality of keys in a scene where the total amount of articles is limited, thereby avoiding hot data frequently read in the storage server, increasing the number of the interface service without causing hot data, and facilitating parallel capacity expansion of the interface service. Each interface server is provided with a routing module, and the routing module can enable each interface server to exchange data with the storage service. For example, the interface server can obtain, via the routing module, the current consumption of one or more keys and the consumption threshold for each Key.
A storage server as shown in fig. 1 may assign a computing unit to a plurality of keys generated by a control server. The storage server may also add a count unit to the total consumption of the item after the interface server receives an item acquisition request from a user. For example, for an item that can only be picked up by a single user and the total consumption of the item is 99, if the storage server receives a notification message that the interface server passes an item acquisition request, the total consumption of the item may be increased by one count unit, that is, the total consumption becomes 100; accordingly, the Key in the storage server that was selected by the interface server as the target of the item acquisition request also increases by an amount of consumption.
In order to eliminate a hot spot in a server, embodiments of the present application provide a data processing method, a data processing apparatus, and a computer-readable storage medium, and the data processing method and the communication apparatus provided in embodiments of the present application are further described in detail below.
Referring to fig. 2, fig. 2 is a schematic flowchart illustrating a data processing method applied to an interface server according to an embodiment of the present disclosure. The execution subject of the embodiment is any one interface server in the interface service cluster. The data processing method may include the steps of:
210. and receiving an acquisition request of the target object sent by the terminal equipment.
And the user clicks to get the target object through the interactive page of the terminal equipment, and then the terminal equipment can generate an acquisition request of the target object according to the operation of the user and send the acquisition request of the target object to the interface server.
Before the interface server receives the acquisition request of the target item sent by the terminal device, the interface server also needs to receive a plurality of keys corresponding to the target item sent by the control server, wherein the plurality of keys are determined by the control server according to the performance parameters and the configuration parameters configured by the management system. The performance parameters configured by the management system comprise a maximum concurrency request amount (Qmax) of the interface service cluster and a maximum concurrency number per second (Pmax) of the storage server, and the configuration parameters comprise a total consumption threshold value (Lmax) of the target item.
220. And determining a target key from a plurality of keys corresponding to the target item in response to the acquisition request, wherein each key in the plurality of keys is used for correspondingly recording the consumption and consumption threshold values of the target item on each key, and the sum of the consumption threshold values recorded corresponding to the plurality of keys is a preset total consumption threshold value of the target item.
Specifically, after receiving the acquisition request of the target item, the interface server may determine the target key from a plurality of keys corresponding to the target item in response to the acquisition request. The interface server may first process the acquisition request using the old keys, correspondingly consuming one of the old keys. The interface server may then convert the keys consumed in the old keys to a plurality of keys sent by the control server.
In one possible implementation, the interface service may determine the target key by a polling method. The interface server may first respond to the acquisition request to acquire the recorded first index of the key. And determining a target key from a plurality of keys corresponding to the target item according to the first index. Wherein, the first index may be recorded and saved by the interface server. The saved first index may be a number of a key, and indicates the interface server to determine that the key corresponding to the number is the target key. For example, the first index is 8, the interface server may determine that the 7 th key is the target key. The indexes of the plurality of keys, i.e., K1 to Kn, are counted from 0, i.e., the index of K1 is 0, and the index of Kn is n-1.
In a possible implementation manner, before the interface server obtains the first index of the recorded key, it needs to first determine whether the interface server has the first index, and in a case that the first index exists, the interface server may directly determine the target key according to the first index. If the first index does not exist in the interface server, the interface server needs to generate a random number r, which may be any positive integer greater than or equal to 0. The absence of the first index in the interface server may be caused by the interface server currently receiving the acquisition request of the target item sent by the user through the terminal device for the first time, or by the first index being lost in the interface server and being unreadable. After the interface service generates a random number r, the number n of the plurality of keys is left by the random number r, so that a second index can be obtained, and the target key is determined from the plurality of keys corresponding to the target object through the second index. The number n of the keys is determined by the control server according to the performance parameters and the configuration parameters configured by the management system. For example, if the random number generated by the interface service is 99 and the number n of the plurality of keys is 10, the first index is equal to r% n, that is, 99% 10, and the result is 9, the second index is 9, and the interface server may determine that the 10 th key of the plurality of keys corresponding to the target item is the target key. And after the interface server determines the target key from the plurality of keys corresponding to the target item according to the second index, generating the first index according to the second index and the number of the plurality of keys, and storing the first index. The first index may be generated by performing an operation of adding one to the second index, and then performing a remainder operation on the number n of the plurality of keys based on a sum obtained by the operation of adding one to the second index. For example, if the second index is 9, the interface server determines a target key from a plurality of keys corresponding to the target item according to the second index, then calculates 10% 10 from 9+1, and obtains 0 from 10%, and then the first index is 0. If the interface server receives the next acquisition request of the target item, it may determine that the target key is the key with index 0 as the target key.
In a possible implementation, the interface server may also determine the target key by a probabilistic access method. The interface server may respond to the acquisition request to acquire consumption of the target item recorded for each key of a plurality of keys corresponding to the target item. And finally, determining the target key from the plurality of keys according to the access probability of each key. Referring to fig. 3, fig. 3 is a schematic structural diagram of a plurality of keys corresponding to a target object according to an embodiment of the present disclosure. As shown in fig. 3, the number of keys is 5, that is, K1 to K5, the consumption amounts of the target items recorded for each key are a, b, c, d, and e, respectively, and the threshold values for each key are L1, L2, L3, L4, and L5, respectively. Let s be a + b + c + d + e, the access probability of K1 is e/s, the access probability of K2 is d/s, the access probability of K3 is c/s, the access probability of K4 is b/s, and the access probability of K5 is a/s. It will be appreciated that each key corresponds to a recorded consumption of the target item, arranged in reverse order in the numerator of the probability of access. Therefore, the probability of accessing the keys with low consumption is increased, the probability of accessing the keys with high consumption is reduced, each key can be accessed more uniformly, and some keys are prevented from becoming hot spots. For example, if the number of keys is 5, i.e., K1 to K5, the threshold value for each key is 20. The interface server acquires that the consumption of the target item recorded corresponding to each key is 6, 8, 10, 12 and 14 respectively. The interface server can calculate that the access probability of K1 is 14/50, the access probability of K2 is 12/50, the access probability of K3 is 10/50, the access probability of K4 is 8/50, and the access probability of K5 is 6/50. According to the fact that the threshold value of each key is 20, the residual consumption amount of K1 is the largest, and therefore the access probability of K1 is the largest; the remaining consumption of K5 is minimal, so the access probability of K5 is minimal. By this method, the access of K1 to K5 can be made more uniform.
In a possible implementation manner, the interface server may obtain the locally cached data, and query, from the locally cached data, consumption of the target item recorded corresponding to each key of the plurality of keys corresponding to the target item. And if the consumption of the target item of each key exists in the local cache, calculating the access probability of each key by using the consumption of the target item of the local cache. If the consumption of the target item of each key does not exist in the local cache, the interface server may obtain the consumption of the target item of each key from the storage server.
In a possible implementation manner, the interface server may also obtain only the size arrangement relationship of the consumption amounts of the target items recorded corresponding to each key in the plurality of keys, that is, a specific value does not need to be obtained. For example, in fig. 3, if the consumption of the target item obtained by the interface server from K1 to K5 is a > b > c > d > e, the interface server can calculate that the access probability of K5 is the minimum and the access probability of K1 is the maximum. The interface server may determine the target key according to the magnitude-permutation relationship of the access probabilities.
230. And acquiring a first consumption and a first consumption threshold of the target item on the target key, which are recorded corresponding to the target key, from the storage server.
After the interface server determines the target key, it may determine whether the acquisition request of the target item, sent by the terminal device, received by the interface server passes, and then need to acquire, from the storage server, the first consumption amount and the first consumption threshold value of the target item on the target key, which are recorded corresponding to the target key.
240. If the first consumption is less than the first consumption threshold, determining that the request is passed.
The interface server may determine a magnitude relationship between the first consumption amount and the first consumption amount threshold, and if the first consumption amount is smaller than the first consumption amount, the interface server may determine that the target key has a consumption amount of the remaining target item available, and the interface server may determine the acquisition request of the target item sent by the terminal device. After determining the acquisition request of the target item sent by the terminal equipment, the interface server sends a notification message for updating the target key to the storage server, wherein the notification message is used for indicating the storage server to update the consumption of the target item corresponding to the stored target key. For example, the current consumption of the target key in the storage server is 19, and when the storage server receives the notification message sent by the reverse access server, the current consumption of the target key may be updated. If the user has picked up 1 item, the current consumption of the target key in the storage server is updated to 20; if the user has picked up 2 items, the current consumption of the target key in the storage server is updated to 21, and so on.
After the interface server determines the acquisition request of the target item sent by the terminal device, the interface server may update the consumption of the target item corresponding to the target key included in the locally cached data. When the next target item acquisition request comes, the interface server can record the consumption of the target item according to the consumption of each key in a plurality of keys corresponding to the target item cached locally, and does not need to acquire the consumption again from the storage server. It should be noted that, the consumption of the target item recorded by each key of the plurality of keys corresponding to the target item cached locally by the interface server and the consumption of the target item recorded by each key of the plurality of keys corresponding to the target item in the storage server may have a certain error, because the storage server also receives the notification message of the updated key of the other interface server. The error only has a certain influence on the accuracy of the access probability, that is, the error affects the average degree of each key in the storage server, which is determined by the interface server as the target key, at most, so that the average degree does not reach a high accuracy. For example, in the case where the average degree is very high, the number of times each key in the storage server is determined as the target key as shown in fig. 3 may be 19 times, 20 times, 19 times, 20 times, respectively. And the error may only cause the number of times each key is respectively determined as the target key to become less uniform than the above, for example, 17 times, 18 times, 20 times, 22 times, 21 times. The total number of times that the target key is determined from the plurality of keys is the same. The error does not affect the overall consumption of the target item calculated by the storage server.
In a possible implementation manner, when the interface server detects that the first consumption amount of the target item at the target key, recorded corresponding to the target key, just reaches the first consumption amount threshold after the current acquisition request of the target item is passed, that is, the target key just finishes consumption after the current acquisition of the target item, the interface server will automatically skip the target key when determining the target key next time. For example, if the first consumption amount of the target key determined by the interface server is 99 and the first consumption amount threshold is 100, the first consumption amount will become 100 after the target item is acquired this time, that is, the consumption amount of the target key will reach the threshold. The interface server will no longer select the key as the target key the next time.
In a possible implementation manner, for a case where the user receives 2 items at a time, if the first consumption amount of the target key determined by the interface server is 99 and the first consumption amount threshold is 100, the number of the target items that the user wants to receive is greater than the remaining consumption amount of the target key. In this case, the interface server may consume one consumption amount of the current target key and then determine another target key from the storage server again to complete the request for 2 target items sent by the user.
In one possible implementation manner, in the prior art, in the case that the target item is limited by a single user, the storage server may assign a key to each user individually to calculate whether the target item pickup amount of each user is over-limited. Therefore, the system can be combined with the architecture of the total quantity of articles limitation in the embodiment of the present application, and the system can refer to fig. 4, where fig. 4 is an architecture diagram of another article management system provided in the embodiment of the present application. The system can be applied to the situation that the total quantity of the articles is limited, and the single user is limited. That is, the article management and control system may provide two storage services, i.e., a storage service limited by a single user and a storage service limited by the total amount of articles, using one storage server. Specifically, after receiving an acquisition request of a target item sent by a terminal device, an interface server may first determine whether the target item received by a user is over-limited on a plurality of keys limited by a single user, and if not, may determine whether the consumption of the target item reaches a consumption threshold on a plurality of keys limited by a total amount of items, thereby determining whether the acquisition request of the target item sent by the terminal device is passed. For example, the consumption threshold of the target item is 1000, and a single user can receive at most 3 target items (that is, the single user receives the upper limit of the number), and if the interface server knows that the number of the target items received by a user is 2 on the single user limited keys, it is determined that the target items received by the user do not exceed the limit; further, the interface server may pass the acquisition request of the target item if the total consumption of the target item is 900, which means that the target item has not been picked up, by continuing the determination on the plurality of keys of the item total amount limit.
In a possible implementation manner, when an interface server fails or crashes, the interface server can be directly turned off or off-line, and each interface server is independent and does not affect each other.
In a possible implementation manner, if an operator increases the total amount of the target item through the management system, the control server may newly generate a plurality of keys according to the performance parameters and the configuration parameters configured by the management system, and after receiving the key, the interface server may determine the target key according to the original plurality of keys and the newly added plurality of keys, and further determine whether the acquisition request of the target item can pass through.
In a possible implementation manner, the embodiment of the application can be applied to effect advertisements, such as limiting the playing amount of the advertisements according to the exposure amount, the click rate, the cost consumption amount, or the like of the advertisements.
In a possible implementation manner, please refer to fig. 5, and fig. 5 is an architecture diagram of another article management and control system provided in an embodiment of the present application. The article management and control system shown in fig. 5 includes an interface service cluster and a real-time computing server, where the real-time computing server may compute a remaining consumption amount corresponding to each current interface server in real time, and push a value of the real-time remaining consumption amount to each interface server. And each interface server judges whether the acquisition request of the target object reaches the upper limit or not according to the residual consumption of the interface server.
In a possible implementation manner, based on the article management and control system shown in fig. 5, each interface server may report the consumption of each interface server to the real-time calculation server, and the real-time calculation server collects the consumption to calculate the total consumption.
In a possible implementation manner, the embodiment of the present application may also be applied to a single IP, a single Open ID, and other scenarios, that is, the number of articles that can be taken by one device corresponding to one IP or one Open ID is limited. In addition, the consumption threshold may extend to a 1 year consumption threshold, a 1 month consumption threshold, a 1 week consumption threshold, and the like, as well as periodic consumption thresholds, such as a weekly consumption threshold, and the like.
According to the embodiment of the application, after receiving the acquisition request of the target item sent by the terminal equipment, the interface server determines the target key from the plurality of keys corresponding to the target item, and judges whether the first consumption of the target item on the target key is smaller than the first consumption threshold. If yes, a notification message for updating the target key is further sent to the storage server through the acquisition request of the target item, and the storage server updates the consumption of the target item corresponding to the stored target key. By the method, the keys corresponding to the target object can be prevented from becoming access hotspots, the requests received by the interface server can be uniformly distributed into a plurality of keys, and the hot data which are frequently read in the storage server are prevented from occurring, so that the hot data cannot be caused by increasing the number of interface services, and the parallel capacity expansion of the interface services is facilitated.
Referring to fig. 6, fig. 6 is a schematic flowchart illustrating a data processing method applied to a control server according to an embodiment of the present disclosure. The execution subject of this embodiment is the control server. The data processing method may include the steps of:
610. the method comprises the steps of obtaining performance parameters configured by a management system and configuration parameters of a target article, wherein the performance parameters comprise the maximum concurrency request quantity of an interface service cluster and the maximum concurrency number per second of a storage server, and the configuration parameters comprise a total consumption threshold value of the target article.
The maximum concurrency request amount of the interface service cluster is Qmax, the maximum concurrency number per second of the storage server is Pmax, and the total consumption threshold value of the target goods is Lmax.
620. Determining a plurality of keys corresponding to a target item and consumption thresholds of the target item corresponding to each key in the plurality of keys according to the performance parameters and the configuration parameters, wherein the sum of the consumption thresholds corresponding to the plurality of keys is a preset total consumption threshold of the target item.
Specifically, as shown in fig. 7, the control server may generate n keys according to Lmax (total consumption threshold of target item), Qmax (maximum concurrent request amount of interface service cluster), and Pmax (maximum concurrent request amount per second of storage server). The n generated keys are K1/L1-Kn/Ln in FIG. 7, where K1-Kn are the indices of each newly generated key and L1-Ln are the thresholds of each newly generated key. The control server first determines the number n of keys corresponding to the target item to be generated, based on Qmax and Pmax. Then the control server calculates an intermediate parameter t, wherein t is Qmax/Pmax, and when t is 0, n is 1; when t >0 and t is not an integer, n ═ int (t) + 1; when t >0 and t is an integer, n ═ t.
After the number of keys corresponding to the target item is determined, the consumption threshold L of the target item corresponding to each key of the plurality of keys needs to be determined according to the Lmax and the number n of the plurality of keys. A threshold value L is int (Lmax/n) + delta, where m is Lmax% n, and delta of the first m keys is 1; delta of the last n-m keys is 0.
For example, if Qmax is 1000, Pmax is 200, Lmax is 2000, t is 5, n is 5, and m is 2000% 5 is 0; the 5 key correspondence point thresholds L are all the same, i.e., L1-L2-L3-L4-L5-2000/5-400, and L1+ L2+ L3+ L4+ L5-400 +400+400+400+ 400-2000 (i.e., the total consumption threshold Lmax of the target item).
When Qmax is 1100, Pmax is 200, Lmax is 2000, t is 5.5, n is int (5.5) +1 is 6, and m is 2000% 6 is 2; the threshold L1 ═ L2 ═ int (2000/6) +1 ═ 334 for the first 2 keys; the threshold value L3 ═ L4 ═ L5 ═ L6 ═ int (2000/6) ═ 333 for the last 4 keys, and L1+ L2+ L3+ L4+ L5+ L6 ═ 334+334+333+333+333 ═ 2000 (i.e., the total consumption threshold value Lmax for the target item).
630. And sending a plurality of keys corresponding to the target items to an interface server included in the interface service cluster, and sending the consumption threshold of the target items corresponding to each key to a storage server, so that the storage server stores the consumption threshold of the target items corresponding to each key.
By the embodiment of the application, the control server can obtain the performance parameters configured by the management system and the configuration parameters of the target object, and determine a plurality of keys corresponding to the target object and a consumption threshold of the target object corresponding to each key in the plurality of keys according to the performance parameters and the configuration parameters of the target object. After the determination, a plurality of keys corresponding to the target item may be sent to each interface server, and the consumption threshold of the target item corresponding to each key may be sent to the storage server, so that the storage server stores the consumption threshold of the target item corresponding to each key. By the method, the control server can split the keys corresponding to the articles into the keys, so that the interface server and the storage server are controlled to perform data processing, hot spot data which are frequently read in the storage server can be avoided, the hot spot data cannot be caused by increasing the number of interface services, and parallel capacity expansion of the interface services is facilitated.
Referring to fig. 8, fig. 8 is a schematic flowchart of another data processing method according to an embodiment of the present application. The execution main body of the embodiment includes a control server and an interface server. The data processing method may include the steps of:
810. the control server obtains performance parameters configured by the management system and configuration parameters of the target articles, wherein the performance parameters comprise the maximum concurrency request quantity of the interface service cluster and the maximum concurrency number per second of the storage server, and the configuration parameters comprise a total consumption threshold value of the target articles.
820. The control server determines a plurality of keys corresponding to a target item and consumption thresholds of the target item corresponding to each key in the plurality of keys according to the performance parameters and the configuration parameters, wherein the sum of the consumption thresholds corresponding to the plurality of keys is a preset total consumption threshold of the target item.
830. And the control server sends a plurality of keys corresponding to the target article to the interface server included in the interface service cluster.
840. The control server sends the consumption threshold of the target item corresponding to each key to the storage server, so that the storage server stores the consumption threshold of the target item corresponding to each key.
850. And the interface server receives an acquisition request of the target object sent by the terminal equipment.
860. And the interface server responds to the acquisition request to determine a target key from a plurality of keys corresponding to the target item, wherein each key in the plurality of keys is used for correspondingly recording the consumption and the consumption threshold of the target item on each key.
870. The interface server acquires a first consumption and a first consumption threshold of the target item on the target key, which are recorded corresponding to the target key, from the storage server.
880. If the first consumption is less than the first consumption threshold, the interface server determines that the request is passed.
The specific implementation manner of steps 810 to 880 can refer to the related description of the foregoing embodiments, and is not described herein again.
According to the embodiment of the application, the control server determines a plurality of keys corresponding to a target object and a consumption threshold of the target object corresponding to each key in the plurality of keys through performance parameters and configuration parameters of the management system, and the interface server and the storage server perform data processing according to the plurality of keys corresponding to the target object and the consumption threshold of the target object corresponding to each key generated by the control server, so that hot spot data which are frequently read in the storage server are avoided, the hot spot data cannot be caused when the number of interface services is increased, and parallel capacity expansion of the interface services is facilitated.
Referring to fig. 9, fig. 9 is a schematic unit diagram of a data processing apparatus according to an embodiment of the present disclosure. The data processing apparatus shown in fig. 9 may be used to perform some or all of the functions in the method embodiments described above with respect to fig. 2, 5 and 8. The device may be a server, a device in a server, or a device capable of being used in cooperation with a server. The logical structure of the apparatus may include: a transceiving unit 910, a processing unit 920 and an obtaining unit 930. When the apparatus is applied to an interface server included in an interface service cluster:
a transceiving unit 910, configured to receive an acquisition request of a target item sent by a terminal device;
a processing unit 920, configured to determine, in response to an acquisition request, a target key from a plurality of keys corresponding to a target item, where each key in the plurality of keys is used to correspondingly record a consumption amount and a consumption threshold of the target item on each key, and a sum of the consumption thresholds recorded corresponding to the plurality of keys is a preset total consumption threshold of the target item;
an obtaining unit 930, configured to obtain, from the storage server, a first consumption amount and a first consumption amount threshold of the target item on the target key, which are recorded in correspondence with the target key;
the processing unit 920 is further configured to determine that the acquisition request is passed if the first consumption amount is smaller than the first consumption amount threshold.
In a possible implementation manner, the obtaining unit 930 is further configured to obtain a first index of a key of the record in response to the obtaining request; the processing unit 920 is further configured to determine a target key from a plurality of keys corresponding to the target item according to the first index.
In a possible implementation manner, if there is no first index, the processing unit 920 is further configured to generate a random number; determining a second index from the random number and the number of keys; determining a target key from a plurality of keys corresponding to the target item according to the second index; a first index is generated based on the second index and the number of keys, and the first index is stored.
In a possible implementation manner, the obtaining unit 930 is further configured to obtain, in response to the obtaining request, consumption of the target item recorded corresponding to each key of a plurality of keys corresponding to the target item; the processing unit 920 is further configured to determine an access probability of each key according to the consumption of the target item recorded corresponding to each key; a target key is determined from the plurality of keys based on the access probability of each key.
In a possible implementation manner, the obtaining unit 930 is further configured to obtain the locally cached data in response to the obtaining request; the processing unit 920 is further configured to query, from the locally cached data, consumption of the target item recorded corresponding to each key of a plurality of keys corresponding to the target item.
In a possible implementation manner, before receiving an acquisition request of a target item sent by a terminal device, the transceiver unit 910 is further configured to receive a plurality of keys corresponding to the target item sent by a control server, where the plurality of keys are determined by the control server according to performance parameters configured by a management system, and the performance parameters include a maximum concurrency request amount of an interface service cluster and a maximum concurrency number per second of a storage server.
When the apparatus is applied to a control server:
an obtaining unit 930, configured to obtain a performance parameter configured by the management system and a configuration parameter of the target item, where the performance parameter includes a maximum concurrency request amount of the interface service cluster and a maximum concurrency number per second of the storage server, and the configuration parameter includes a total consumption threshold of the target item;
a processing unit 920, configured to determine, according to the performance parameter and the configuration parameter, consumption thresholds of a plurality of keys corresponding to a target item and a target item corresponding to each key in the plurality of keys, where a sum of the consumption thresholds corresponding to the plurality of keys is a preset total consumption threshold of the target item;
a transceiving unit 910, configured to send a plurality of keys corresponding to the target item to an interface server included in the interface service cluster, and send a consumption threshold of the target item corresponding to each key to the storage server, so that the storage server stores the consumption threshold of the target item corresponding to each key.
In a possible implementation manner, the processing unit 920 is further configured to determine a plurality of keys corresponding to the target item according to the maximum concurrency request amount of the interface service cluster and the maximum concurrency number per second of the storage server; determining a consumption threshold for the target item corresponding to each of the plurality of keys based on the total consumption threshold and the number of the plurality of keys.
Referring to fig. 10, fig. 10 is a simplified schematic diagram of an entity structure of a data processing apparatus according to an embodiment of the present disclosure, where the apparatus includes a processor 1010, a memory 1020, and a communication interface 1030, and the processor 1010, the memory 1020, and the communication interface 1030 are connected by one or more communication buses.
The processor 1010 is configured to support the communication device to perform the functions corresponding to the methods of fig. 2, 5 and 8. The processor 1010 may be a Central Processing Unit (CPU), a Network Processor (NP), a hardware chip, or any combination thereof.
The memory 1020 is used to store program codes and the like. Memory 1020 may include volatile memory (volatile), such as Random Access Memory (RAM); the memory 1020 may also include a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory (flash memory), a Hard Disk Drive (HDD), or a solid-state drive (SSD); memory 1020 may also include a combination of memories of the sort described above.
Communication interface 1030 is used to transmit and receive data, information, messages and the like, and may also be described as a transceiver, transceiving circuitry and the like.
In the embodiment of the present application, when the data processing apparatus is applied to an interface server included in an interface service cluster, the processor 1010 may call the program code stored in the memory 1020 to perform the following operations:
the control communication interface 1030 receives an acquisition request of a target item sent by the terminal equipment;
the processor 1010 calls the program code stored in the memory 1020 to respond to the acquisition request to determine a target key from a plurality of keys corresponding to the target item, wherein each key in the plurality of keys is used for correspondingly recording the consumption of the target item on each key and a consumption threshold, and the sum of the consumption thresholds corresponding to the plurality of keys is a preset total consumption threshold of the target item;
the processor 1010 calls the program code stored in the memory 1020 to acquire a first consumption amount and a first consumption threshold value of the target item on the target key, which are recorded corresponding to the target key, from the storage server;
the processor 1010 calls program code stored in the memory 1020 to determine that the request is passed if the first consumption is less than the first consumption threshold.
In one possible implementation, processor 1010 invokes program code stored in memory 1020 to retrieve a first index of a key of a record in response to a retrieval request; processor 1010 invokes program code stored in memory 1020 to determine a target key from a plurality of keys corresponding to the target item based on the first index.
In one possible implementation, if the first index is not present, processor 1010 invokes program code stored in memory 1020 to generate a random number; determining a second index from the random number and the number of keys; determining a target key from a plurality of keys corresponding to the target item according to the second index; a first index is generated based on the second index and the number of keys, and the first index is stored.
In one possible implementation, the processor 1010 invokes the program code stored in the memory 1020 to respond to the acquisition request to acquire the consumption amount of the target item recorded corresponding to each of a plurality of keys corresponding to the target item; processor 1010 invokes program code stored in memory 1020 to determine the probability of access for each key based on the recorded consumption of the target item for each key; a target key is determined from the plurality of keys based on the access probability of each key.
In one possible implementation, the processor 1010 invokes program code stored in the memory 1020 to obtain locally cached data in response to a fetch request; processor 1010 invokes program code stored in memory 1020 to look up the consumption of the target item recorded for each of a plurality of keys corresponding to the target item from the locally cached data.
In a possible implementation manner, before receiving an acquisition request of a target item sent by a terminal device, the control communication interface 1030 receives a plurality of keys corresponding to the target item sent by a control server, where the plurality of keys are determined by the control server according to performance parameters configured by a management system, and the performance parameters include a maximum concurrency request amount of an interface service cluster and a maximum concurrency number per second of a storage server.
In the embodiment of the present application, when the apparatus is applied to a control server, the processor 1010 may call the program code stored in the memory 1020 to perform the following operations:
the processor 1010 calls the program codes stored in the memory 1020 to obtain performance parameters configured by the management system and configuration parameters of the target item, wherein the performance parameters comprise the maximum concurrency request amount of the interface service cluster and the maximum concurrency number per second of the storage server, and the configuration parameters comprise a total consumption threshold value of the target item;
the processor 1010 calls the program code stored in the memory 1020 to determine a plurality of keys corresponding to the target item and consumption thresholds of the target item corresponding to each key of the plurality of keys according to the performance parameter and the configuration parameter, wherein the sum of the consumption thresholds corresponding to the plurality of keys is a preset total consumption threshold of the target item;
the control communication interface 1030 transmits a plurality of keys corresponding to the target item to an interface server included in the interface service cluster, and transmits a consumption amount threshold of the target item corresponding to each key to the storage server, so that the storage server stores the consumption amount threshold of the target item corresponding to each key.
In one possible implementation, the processor 1010 invokes the program code stored in the memory 1020 to determine a plurality of keys corresponding to the target item according to the maximum concurrency request amount of the interface service cluster and the maximum concurrency number per second of the storage server; determining a consumption threshold for the target item corresponding to each of the plurality of keys based on the total consumption threshold and the number of the plurality of keys.
Embodiments of the present application also provide a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the functions corresponding to the methods of fig. 2, 5 and 8.
It should be noted that, in the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to relevant descriptions of other embodiments for parts that are not described in detail in a certain embodiment.
The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs.
The units in the processing equipment of the embodiment of the invention can be merged, divided and deleted according to actual needs.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the present application are all or partially generated when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, memory Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.

Claims (10)

1. A data processing method applied to an interface server included in an interface service cluster, the method comprising:
receiving an acquisition request of a target object sent by terminal equipment;
determining a target key from a plurality of keys corresponding to the target item in response to the acquisition request, wherein each key in the plurality of keys is used for correspondingly recording consumption and consumption thresholds of the target item on each key, and the sum of the consumption thresholds recorded corresponding to the plurality of keys is a preset total consumption threshold of the target item;
acquiring a first consumption and a first consumption threshold of the target item on the target key, which are recorded corresponding to the target key, from a storage server;
and if the first consumption is smaller than the first consumption threshold, determining that the acquisition request passes.
2. The method of claim 1, wherein determining a target key from a plurality of keys corresponding to the target item in response to the acquisition request comprises:
responding to the acquisition request to acquire a first index of a recorded key;
and determining a target key from a plurality of keys corresponding to the target item according to the first index.
3. The method of claim 2, further comprising:
if the first index does not exist, generating a random number;
determining a second index according to the random number and the number of the plurality of keys;
determining a target key from a plurality of keys corresponding to the target item according to the second index;
and generating the first index according to the second index and the number of the plurality of keys, and storing the first index.
4. The method of claim 1, wherein determining a target key from a plurality of keys corresponding to the target item in response to the acquisition request comprises:
responding to the acquisition request to acquire the consumption of the target item recorded corresponding to each key in a plurality of keys corresponding to the target item;
determining the access probability of each key according to the consumption of the target item recorded by the corresponding key;
and determining a target key from the plurality of keys according to the access probability of each key.
5. The method according to claim 4, wherein the obtaining the consumption amount of the target item recorded for each key of a plurality of keys corresponding to the target item in response to the obtaining request comprises:
responding to the acquisition request to acquire the data of the local cache;
and inquiring consumption of the target item recorded corresponding to each key in a plurality of keys corresponding to the target item from the locally cached data.
6. The method according to claim 1, wherein before receiving the acquisition request of the target item sent by the terminal device, the method further comprises:
receiving a plurality of keys corresponding to the target item sent by a control server, wherein the plurality of keys are determined by the control server according to performance parameters configured by a management system, and the performance parameters include a maximum concurrency request amount of the interface service cluster and a maximum concurrency number per second of the storage server.
7. A data processing method is applied to a control server, and the method comprises the following steps:
acquiring performance parameters configured by a management system and configuration parameters of a target article, wherein the performance parameters comprise the maximum concurrency request quantity of an interface service cluster and the maximum concurrency number per second of a storage server, and the configuration parameters comprise a total consumption threshold value of the target article;
determining a plurality of keys corresponding to the target item and consumption thresholds of the target item corresponding to each key of the plurality of keys according to the performance parameter and the configuration parameter, wherein the sum of the consumption thresholds corresponding to the plurality of keys is a preset total consumption threshold of the target item;
sending a plurality of keys corresponding to the target item to an interface server included in the interface service cluster, and sending the consumption threshold of the target item corresponding to each key to the storage server, so that the storage server stores the consumption threshold of the target item corresponding to each key.
8. The method of claim 7, wherein the determining a plurality of keys corresponding to the target item and a consumption threshold of the target item for each of the plurality of keys based on the performance parameter and the configuration parameter comprises:
determining a plurality of keys corresponding to the target object according to the maximum concurrent request quantity of the interface service cluster and the maximum concurrent number per second of the storage server;
determining a consumption threshold of the target item corresponding to each of the plurality of keys based on the total consumption threshold and the number of the plurality of keys.
9. A data processing apparatus, comprising:
a processor and a memory, the processor and the memory being interconnected, wherein the memory is configured to store a computer program comprising program instructions, the processor being configured to invoke the program instructions, to perform the data processing method of any one of claims 1 to 6, and to perform the data processing method of claim 7 or 8.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions that, when executed by a processor, cause the processor to perform the data processing method according to any one of claims 1 to 6, and the data processing method according to claim 7 or 8.
CN202010703197.0A 2020-07-21 2020-07-21 Data processing method and device and computer readable storage medium Pending CN111782732A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010703197.0A CN111782732A (en) 2020-07-21 2020-07-21 Data processing method and device and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010703197.0A CN111782732A (en) 2020-07-21 2020-07-21 Data processing method and device and computer readable storage medium

Publications (1)

Publication Number Publication Date
CN111782732A true CN111782732A (en) 2020-10-16

Family

ID=72763647

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010703197.0A Pending CN111782732A (en) 2020-07-21 2020-07-21 Data processing method and device and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN111782732A (en)

Similar Documents

Publication Publication Date Title
US11290360B2 (en) Analyzing resource placement fragmentation for capacity planning
US9489137B2 (en) Dynamic storage tiering based on performance SLAs
EP2830284A1 (en) Caching method for distributed storage system, node and computer readable medium
US20130332693A1 (en) Allocating storage memory based on future file size or use estimates
US20140181035A1 (en) Data management method and information processing apparatus
US11416166B2 (en) Distributed function processing with estimate-based scheduler
CN110147203B (en) File management method and device, electronic equipment and storage medium
CN111708755A (en) Data migration method, device, system, electronic equipment and readable storage medium
US11416176B2 (en) Function processing using storage controllers for load sharing
WO2020094064A1 (en) Performance optimization method, device, apparatus, and computer readable storage medium
CN112084173A (en) Data migration method and device and storage medium
CN111562884A (en) Data storage method and device and electronic equipment
US11429311B1 (en) Method and system for managing requests in a distributed system
CN110909978A (en) Resource processing method, device, server and computer readable storage medium
US10594620B1 (en) Bit vector analysis for resource placement in a distributed system
CN109977074B (en) HDFS-based LOB data processing method and device
CN111782732A (en) Data processing method and device and computer readable storage medium
CN115993932A (en) Data processing method, device, storage medium and electronic equipment
CN115981559A (en) Distributed data storage method and device, electronic equipment and readable medium
CN112083886B (en) Storage management method, system and device of NAS (network attached storage) equipment
US9710183B2 (en) Effectively limitless apparent free space on storage device
CN113672652A (en) Data access method, device, equipment and storage medium
US20200159706A1 (en) Object Storage System with Control Entity Quota Usage Mapping
CN113760940A (en) Quota management method, device, equipment and medium applied to distributed system
US11816088B2 (en) Method and system for managing cross data source data access requests

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination