CN111611067A - Load balancing method and device and block chain system - Google Patents
Load balancing method and device and block chain system Download PDFInfo
- Publication number
- CN111611067A CN111611067A CN201910135696.1A CN201910135696A CN111611067A CN 111611067 A CN111611067 A CN 111611067A CN 201910135696 A CN201910135696 A CN 201910135696A CN 111611067 A CN111611067 A CN 111611067A
- Authority
- CN
- China
- Prior art keywords
- data access
- data
- storage device
- average
- data storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000013500 data storage Methods 0.000 claims abstract description 82
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 29
- 230000008569 process Effects 0.000 claims abstract description 14
- 238000009826 distribution Methods 0.000 claims abstract description 10
- 238000010801 machine learning Methods 0.000 claims description 11
- 238000012549 training Methods 0.000 claims description 9
- 238000000605 extraction Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 description 16
- 238000010586 diagram Methods 0.000 description 15
- 238000003860 storage Methods 0.000 description 13
- 230000008859 change Effects 0.000 description 7
- 238000004590 computer program Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 230000009467 reduction Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000032683 aging Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000003066 decision tree Methods 0.000 description 2
- 230000007812 deficiency Effects 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000010187 selection method Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 241001522296 Erithacus rubecula Species 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000009193 crawling Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application discloses a load balancing method, a load balancing device and a block chain system. The device includes: a receiving unit, configured to receive a data access request of at least one data access object; the distribution unit is used for distributing the data access request to the data storage device meeting the access performance constraint condition according to the weight polling algorithm; and the updating unit is used for dynamically updating the weight coefficient of the data storage device based on a data access time parameter corresponding to the data access object in the distribution process, wherein the data access time parameter is used for representing the predicted trend of the read average number and the write average number of the data access object in the statistical period changing along with the time. According to the technical scheme of the embodiment of the application, the weight coefficient of the data storage device is dynamically updated based on the data access time parameter corresponding to the data access object, so that the performance of the system is prevented from being reduced when the system deals with a large number of access requests.
Description
Technical Field
The present application relates generally to the field of distributed storage technologies, and in particular, to a load balancing method, apparatus, and blockchain system.
Background
The block chain technology is also called as distributed ledger technology, and is a distributed internet database technology. A network constructed based on the blockchain technique may be referred to as a blockchain network, in which network nodes (also referred to as blockchain nodes) are included. The block chain network has the characteristics of decentralization, public transparency, no tampering, trustiness and the like. Block chain technology is also increasingly being applied to the field of logistics.
Compared with the traditional database, the application of the block chain technology in the field of logistics has the obvious defects of poor performances in the aspects of throughput, read-write delay and the like because the block chain needs to be subjected to calculation such as encryption, signature and the like.
Disclosure of Invention
In view of the above-mentioned drawbacks and deficiencies of the prior art, it is desirable to provide a load balancing method, apparatus and block chain system to avoid performance degradation of a data storage device when handling a large number of data access requests without incurring increased access latency.
In a first aspect, an embodiment of the present application provides a load balancing apparatus, where the apparatus includes:
a receiving unit, configured to receive a data access request of at least one data access object;
the distribution unit is used for distributing the data access request to the data storage device meeting the access performance constraint condition according to the weight polling algorithm;
and the updating unit is used for dynamically updating the weight coefficient of the data storage device based on a data access time parameter corresponding to the data access object in the distribution process, wherein the data access time parameter is used for representing the predicted trend of the read average number and the write average number of the data access object in the statistical period changing along with the time.
In a second aspect, an embodiment of the present application provides a blockchain system, including:
a load balancing apparatus as described in the first aspect;
and the side chain is used as a data storage device and used for storing and verifying the transaction data corresponding to the data access request.
In a third aspect, an embodiment of the present application provides a load balancing method, where the method includes:
receiving a data access request of at least one data access object;
distributing the data access request to a data storage device meeting the access performance constraint condition according to a weight polling algorithm;
and dynamically updating the weighting coefficients of the data storage device during the allocation process based on a data access time parameter corresponding to the data access object, wherein the data access time parameter is used for representing the predicted trend of the read average number and the write average number of the data access object in the statistical period changing along with the time.
According to the load balancing device and method provided by the embodiment of the application, the weight coefficient of the data storage device is dynamically updated based on the data access time parameter corresponding to the data access object, so that the performance of a system is prevented from being reduced when the system deals with a large number of access requests.
Furthermore, a data access time parameter of a data access object is predicted through a pre-established access amount prediction model, and the weight coefficient of the data storage device is updated by using the data access time parameter, so that the dynamic adjustment of load balance is realized, and the performance reduction of the system when a large number of data access requests occur is avoided.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
fig. 1 shows a schematic structural diagram of a load balancing apparatus 100 provided in an embodiment of the present application;
fig. 2 shows a schematic structural diagram of an updating unit 103 according to another embodiment of the present application;
FIG. 3 is a schematic diagram showing the average number of reads and writes of a commodity object in a statistical period as a function of time;
fig. 4 is a schematic structural diagram of a blockchain system 400 provided in an embodiment of the present application;
fig. 5 is a schematic flowchart illustrating a load balancing method according to an embodiment of the present application;
FIG. 6 shows a flowchart of step 503 provided by another embodiment of the present application
Fig. 7 shows a schematic structural diagram of a computer system suitable for implementing the terminal device or the server according to the embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant disclosure and are not limiting of the disclosure. It should be further noted that, for the convenience of description, only the portions relevant to the disclosure are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
In the prior art, in some areas of application load balancing, for example block chaining. For security, the blockchain needs to use encryption, signature and other computations, resulting in significant performance deficiencies in terms of throughput, read-write latency and the like.
For example, when the data storage device is a side chain of the blockchain system, the prior art simply performs transaction mapping according to the service type. That is, corresponding side chains are allocated according to the service types for processing the services corresponding to the service types. For example, side chain a is used for handling food type traffic and side chain B is used for handling automotive type traffic. Since business related access requests for food types can be soared at certain times, such as during twenty-one activities, twenty-two activities, or other times when requests are suddenly increased, soaring requests can result in degraded side chain a access performance. In addition, assigning sidechains according to traffic type for handling certain low volume traffic may result in the sidechains being underutilized. For example, side chain B.
For another example, similar problems as described above also exist for a scenario in which servers are assigned by traffic type for processing access requests. The method comprises the steps of assuming that a scene of certain keywords is accessed through a certain website, taking the keywords as data access objects, and crawling relevant content corresponding to the keywords as sources of the data access objects, wherein the relevant content corresponding to the keywords is characterized by URL link addresses. If the servers are distributed according to the access service types, the server A is used for acquiring the related content of the first keyword, the server B is used for acquiring the related content of the second keyword, the first keyword may have the sudden increase of the access amount in some time, and the corresponding server of the second keyword cannot be fully utilized due to the small access amount of some keywords.
Therefore, in view of the above problems, embodiments of the present application aim to provide a load balancing apparatus and method, which effectively use the time characteristics of data access objects to predict the future access amount of a data storage apparatus, so as to overcome the problem of load imbalance caused by mapping the existing data storage apparatus according to the service types, thereby avoiding performance degradation when a system deals with a large number of data access requests, and at the same time, not causing an increase in access delay.
Referring to fig. 1, fig. 1 is a schematic structural diagram illustrating a load balancing apparatus 100 according to an embodiment of the present disclosure.
As shown in fig. 1, the apparatus 100 includes:
a receiving unit 101, configured to receive a data access request of at least one data access object;
the allocation unit 102 is configured to allocate the data access request to the data storage device that satisfies the access performance constraint condition according to a weighted round robin algorithm;
an updating unit 103, configured to dynamically update the weighting coefficients of the data storage device during the allocation process based on a data access time parameter corresponding to the data access object, where the data access time parameter is used to characterize a predicted trend of the read average and the write average of the data access object in the statistical period over time.
In the embodiment of the application, a data access request is received, a target data storage device meeting the access performance constraint condition is selected from a plurality of data storage devices according to a weight polling algorithm, and the target data storage device is responsible for processing the service corresponding to the data access object. The data storage device may be a side chain of the block chain, or may be another server or service processing device.
Taking the block chain tracing system as an example, the system takes the commodity object as a data access object, and data after entering the supply chain is written into the block chain. The supply chain includes a number of links such as production, warehousing, shipping, transportation, receipt, and the like. Wherein the data access object may be a commodity object processed in a supply chain link. For example, the commodity object a is a living article. The commodity object B is an automobile type article.
The data access request may be a read request of the data access object, a write request of the data access object, or an inquiry request of the data access object.
When a data access request of a certain data access object is received, a target side chain meeting the access performance constraint condition is selected from a plurality of side chains through a weight polling algorithm, and the target side chain is used for processing a service corresponding to the data access request. Wherein the access performance constraint condition can be set according to the application scene. For example, in the blockchain traceability system, the access performance constraint condition may be that the access amount does not exceed the allowed maximum number of reads and writes within a predetermined time period, or the capacity index of the allowed access admission, etc. The capability index of admission access may be, for example, TPS: transactions Per Second (number of Transactions transmitted Per Second), i.e., number of Transactions processed Per Second by the server. That is, the client starts timing when sending the request, and finishes timing after receiving the server response, so as to calculate the used time and the number of completed transactions. Generally, the overall system processing capability depends on the TPS value of the lowest processing capability module. It may also be QPS: the query per second QPS is a measure of how much traffic a particular query server processes within a specified time, i.e., the number of requests per second, i.e., the maximum throughput.
Wherein the data access time parameter is used for characterizing the predicted trend of the read average number and the write average number of the data access object in the statistical period changing along with the time. For example, in the scenario of the blockchain applied to the logistics supply chain, the average number of reads/writes for a certain commodity object a on the first day of the statistical period, the average number of reads/writes for the second day, and the average number of reads/writes for the last day of the statistical period. And abstracting the read/write average numbers according to time change to obtain a data access time parameter of the commodity object A. The trend of the commodity object a along with the change of time in the statistical period is statistically analyzed, so as to obtain a schematic diagram of the average read-write times of a certain commodity object along with the change of time in the statistical period, which is shown in fig. 3. As shown in fig. 3, the average number of reads on the first day (D1) of the statistical cycle is about 4 times, and the average number of writes on the first day is about 7 times, where the peak of the average number of writes occurs from the second day to the third day of the statistical cycle, which indicates that the sudden increase in the number of accesses occurring from the second day to the third day may degrade the performance of the data storage device for processing the commodity object. Wherein, the statistical period can be one week, one month or other set values.
The weighted polling algorithm is not described herein in detail for the prior art.
According to the embodiment of the application, in the distribution process of the data access requests, the weight coefficient of the data storage device is dynamically updated based on the data access time parameter corresponding to the data access object, and the problem that the performance of the data storage device is reduced due to the concentrated occurrence of a large number of data access requests is effectively solved.
Referring to fig. 2, fig. 2 is a schematic structural diagram of an update unit 103 according to another embodiment of the present application.
The updating unit 103 may include:
an extracting subunit 201, configured to extract a set of feature values of the data access request;
the prediction subunit 202 is configured to input the feature value set to a pre-established access quantity prediction model to obtain a data access time parameter corresponding to the data access object;
and the calculating subunit 203 is used for calculating the weight coefficient of the data storage device based on the data access time parameter.
In the embodiment of the application, the data access time parameter of the data access object is obtained through a pre-established access amount prediction model. For example, the characteristic value set of the data access request can be input into a pre-established access quantity prediction model to obtain a data access time parameter corresponding to the characteristic value set of the data access request. The access quantity prediction model is pre-established and can be obtained through training and learning of a machine learning algorithm, and the machine learning algorithm can be, for example, a Gradient Boosting Decision Tree (GBDT) algorithm.
The feature value set may be extracted from historical data related to the data access object by an existing feature extraction algorithm, and further obtained by a feature selection method. The set of feature values obtained by the present application may include, for example, waybill features associated with merchandise objects. For example, the set of feature values for a commodity object may include waybill information associated with the commodity object. The waybill information may include, for example, recipient information, sender information, recipient address information, sender address information, consignment type, waybill aging identification, commodity classification identification, and the like.
The data access time parameter may be a data access rule of a data access object obtained by performing statistical analysis on read-write access records of the data storage device. And (4) performing off-line or near-line analysis to obtain the time characteristic of the average read-write times of the data access object by collecting the historical access records of the data access object. This prediction of the temporal characteristics enables dynamic optimization of the performance of the system in time. The data access time parameter of the data access object may also be obtained based on a machine learning algorithm.
The pre-establishing of the visit quantity prediction model may include:
acquiring historical data of at least one data access object;
extracting a characteristic value set and a label value set based on historical data;
and training and predicting by using the characteristic value set and the label value set by adopting a machine learning algorithm to obtain an access quantity prediction model.
The historical data of the data access object is acquired, and for example, the historical data may be subjected to tag processing in a manual tagging manner, so as to obtain a tag value set corresponding to the data access object. The characteristic value set and the label value set of the data access object are divided into a training set and a verification set according to a proportion, and the access amount prediction model with better fitting performance is obtained according to machine learning algorithm training and learning. The visit amount prediction model can be used for predicting the visit amount of a new commodity object at the future time when the commodity object arrives.
According to the data access method and device, the time characteristic corresponding to the data access object is obtained through the access amount prediction model which is established in advance, the access amount of the data access request related to the data access object in the future time is effectively predicted, and when the access amount is close to the performance threshold value of the data storage device, the data access request distributed to the data storage device is reduced, so that the performance of the data storage device is prevented from being reduced when a large number of data access requests occur.
Further, the calculating subunit 203 provided in the embodiments of the present application may be implemented in various ways. For example, a first read average and a first write average of data access objects on a data storage device are obtained based on a data access time parameter; and summing the first reading average and the first writing average, then calculating the reciprocal, and using the reciprocal as a new weight coefficient of the data storage device corresponding to the reciprocal. As another example, a second read average and a second write average of data access objects on the data storage device are obtained based on the data access time parameter; calculating an exponential form read average and an exponential form write average for each data access object using the second read average and the second write average, respectively; and summing the exponential form reading average and the exponential form writing average, calculating the reciprocal, and respectively taking the reciprocal as a new weight coefficient of the data storage device corresponding to the reciprocal.
Further describing the above embodiments, a first read average and a first write average for data access objects on a data storage device are obtained based on a data access time parameter; and summing the first reading average and the first writing average, then calculating the reciprocal, and using the reciprocal as a new weight coefficient of the data storage device corresponding to the reciprocal.
Assuming that N commodity objects exist on the ith side chain, the average number of the read of the jth (j belongs to 1-N) commodity object on the first day is represented as r through the visit quantity prediction modelijThe average number of writes is denoted as wij。
The weighting factor of the ith side chain can be updated according to the following formula (1):
when data of a certain new commodity object arrives, polling is carried out according to the current weight coefficient so as to distribute the new commodity object. And updating the weight coefficient of the target side chain by using the formula (1) after the data of the commodity is linked to the allocated target side chain.
Obtaining a second read average number and a second write average number of data access objects on the data storage device based on the data access time parameter; calculating an exponential form read average and an exponential form write average for each data access object using the second read average and the second write average, respectively; and summing the exponential form reading average and the exponential form writing average, calculating the reciprocal, and respectively taking the reciprocal as a new weight coefficient of the data storage device corresponding to the reciprocal. It can be understood that the weight coefficients of the side chains are updated as an integral average along the time axis.
Assuming that N commodity objects exist on the ith side chain, the jth (j belongs to 1-N) commodity object obtains the t-th reading average number r through the visit quantity prediction modelijtAverage number of writes of wijt。
The exponential read number r of the jth commodity on the ith side chainijAnd the exponential write average wijRespectively expressed as:
the weighting factor of the ith side chain can be updated according to the above formula (1). When data of a certain new commodity object arrives, polling is carried out according to the current weight coefficient so as to distribute the new commodity object. And updating the weight coefficient of the target side chain by using the formula (1) after the data of the commodity is linked to the allocated target side chain.
In the embodiment of the application, the access amount of the data of the existing commodity object of the data storage device is predicted according to the read-write time characteristic of the commodity object in a certain time period, so that the data storage device can meet the access performance constraint condition in a specified time period in the future, and the performance reduction caused when a system deals with a large number of data access requests is avoided.
Further, referring to fig. 4, fig. 4 is a schematic structural diagram of a block chain system 400 according to an embodiment of the present disclosure.
As shown in fig. 4, the system 400 includes:
a load balancing device 401, configured to receive a data access request of at least one data access object; distributing the data access request to a data storage device meeting the access performance constraint condition according to a weight polling algorithm; and dynamically updating the weight coefficient of the data storage device based on a data access time parameter corresponding to the data access object in the distribution process, wherein the data access time parameter is used for representing the predicted trend of the read average number and the write average number of the data access object in the statistical period changing along with the time.
The description of the load balancing apparatus 401 in the embodiment of the present application refers to the description of the foregoing embodiment. The data access time parameter is pre-constructed according to the read-write time characteristic corresponding to the read-write request of the user. The visit amount of the side chain is predicted through the data visit time parameter, and the data visit assigned to the side chain is reduced when the maximum visit amount is approached.
At least one side chain 402, which is a data storage device, is used to store and verify transaction data corresponding to the data access request. I.e., each sidechain 402 is used to store and verify sidechain block data and transaction data. Each side chain 402 may also include a plurality of side chain nodes. Each side chain node is used to store side chain data.
The block chain system provided by the embodiment of the application comprises the load balancing device, and the load balancing device can effectively solve the problem of load imbalance caused by simply distributing the service processing devices based on service types in the prior art.
The system may further comprise: and a main chain 403 for recording the hash value of each side chain data and cross-chain transaction data. The backbone may also include a plurality of backbone nodes, each backbone node for storing backbone data.
Compared with the scheme that a multilevel cache mode is adopted to overcome the burst traffic of the block chain in the prior art, the load balancing method provided by the embodiment of the application updates the weight coefficient of the side chain through prediction of future access amount, cannot cause increase of access delay, and effectively avoids the problem of performance reduction of the side chain of the block chain system when the side chain is in response to burst access.
Further referring to fig. 5, fig. 5 is a schematic flowchart illustrating a load balancing method according to an embodiment of the present application.
As shown in fig. 5, the method includes:
502, distributing the data access request to a data storage device meeting the access performance constraint condition according to a weight polling algorithm;
in the allocation process, the weighting coefficients of the data storage device are dynamically updated based on a data access time parameter corresponding to the data access object, wherein the data access time parameter is used for representing the predicted trend of the read average number and the write average number of the data access object in the statistical period along with the change of the time.
In the embodiment of the present application, taking the block chain traceability system as an example, the system writes data of the commodity object after entering the supply chain into the block chain. And various links of the supply chain, such as production, warehousing, delivery, transportation, signing and the like. Wherein the data access object may be a commodity object processed in a supply chain link. For example, the commodity object a is a living article. The commodity object B is an automobile type article.
The data access request of the data access object can be a read request of the data access object or a write request of the data access object.
The data storage means may, for example, be a side chain of a block chain, or other server or business processing device.
When a data access request of a data access object is received, selecting a target meeting an access performance constraint condition from a plurality of data storage devices through a weight polling algorithm, and processing a service corresponding to the data access request. The access performance constraints may be set according to the application scenario. For example, in a blockchain traceability system, the access performance constraint condition may be that the trend of the access amount does not exceed the maximum allowed number of reads and writes within a predetermined time period, or the capability index of allowing access admission, etc.
Wherein the data access time parameter is used for characterizing the predicted trend of the read average number and the write average number of the data access object in the statistical period changing along with the time. For example, in the scenario of the blockchain applied to the logistics supply chain, the average number of reads/writes for a certain commodity object a on the first day of the statistical period, the average number of reads/writes for the second day, and the average number of reads/writes for the last day of the statistical period. And abstracting the read/write average numbers according to time change to obtain a data access time parameter of the commodity object A.
The weighted polling algorithm is not described herein in detail for the prior art.
According to the embodiment of the application, in the distribution process, the weight coefficient of the data storage device is dynamically updated based on the data access time parameter corresponding to the data access object, so that the problem of performance reduction of the data storage device caused by centralized occurrence of a large number of data access requests is effectively solved.
Referring to fig. 6, fig. 6 is a flowchart illustrating a step 503 according to another embodiment of the present application.
Step 503 may include:
In the embodiment of the application, the data access time parameter of the data access object is obtained through a pre-established access amount prediction model. For example, the characteristic value set of the data access request can be input into a pre-established access quantity prediction model to obtain a data access time parameter corresponding to the characteristic value set of the data access request. The access quantity prediction model is pre-established and can be obtained through training and learning of a machine learning algorithm, and the machine learning algorithm can be, for example, a Gradient Boosting Decision Tree (GBDT) algorithm.
The feature value set may be extracted from historical data related to the data access object by an existing feature extraction algorithm, and further obtained by a feature selection method. The set of feature values obtained by the present application may include, for example, waybill features associated with merchandise objects. For example, the set of feature values for a commodity object may include waybill information associated with the commodity object. The waybill information may include, for example, recipient information, sender information, recipient address information, sender address information, consignment type, waybill aging identification, commodity classification identification, and the like.
The data access time parameter may be a data access rule of a data access object obtained by performing statistical analysis on read-write access records of the data storage device. And (4) performing off-line or near-line analysis to obtain the time characteristic of the average read-write times of the data access object by collecting the historical access records of the data access object. This prediction of the temporal characteristics enables dynamic optimization of the performance of the system in time. The data access time parameter of the data access object may also be obtained based on a machine learning algorithm.
The pre-establishing of the visit quantity prediction model may include:
acquiring historical data of at least one data access object;
extracting a characteristic value set and a label value set based on historical data;
and training and predicting by using the characteristic value set and the label value set by adopting a machine learning algorithm to obtain an access quantity prediction model.
The historical data of the data access object is acquired, and for example, the historical data may be subjected to tag processing in a manual tagging manner, so as to obtain a tag value set corresponding to the data access object. The characteristic value set and the label value set of the data access object are divided into a training set and a verification set according to a proportion, and the access amount prediction model with better fitting performance is obtained according to machine learning algorithm training and learning. The visit amount prediction model can be used for predicting the visit amount of a new commodity object at the future time when the commodity object arrives.
According to the data access method and device, the time characteristic corresponding to the data access object is obtained through the access amount prediction model which is established in advance, the access amount of the data access request related to the data access object in the future time is effectively predicted, and when the access amount is close to the performance threshold value of the data storage device, the data access request distributed to the data storage device is reduced, so that the performance of the data storage device is prevented from being reduced when a large number of data access requests occur.
Further, step 603 provided in the embodiments of the present application may be implemented in various ways. For example, a first read average and a first write average of data access objects on a data storage device are obtained based on a data access time parameter; and summing the first reading average and the first writing average, then calculating the reciprocal, and using the reciprocal as a new weight coefficient of the data storage device corresponding to the reciprocal. As another example, a second read average and a second write average of data access objects on the data storage device are obtained based on the data access time parameter; calculating an exponential form read average and an exponential form write average for each data access object using the second read average and the second write average, respectively; and summing the exponential form reading average and the exponential form writing average, calculating the reciprocal, and respectively taking the reciprocal as a new weight coefficient of the data storage device corresponding to the reciprocal.
Further describing the above embodiments, a first read average and a first write average for data access objects on a data storage device are obtained based on a data access time parameter; and summing the first reading average and the first writing average, then calculating the reciprocal, and using the reciprocal as a new weight coefficient of the data storage device corresponding to the reciprocal.
Suppose there are N commercial objects on the ith side chainThe average number of the readings of the jth (j belongs to 1-N) commodity object on the first day obtained by the visit quantity prediction model is represented as rijThe average number of writes is denoted as wij。
The weighting factor of the ith side chain can be updated according to the following formula (1):
when data of a certain new commodity object arrives, polling is carried out according to the current weight coefficient so as to distribute the new commodity object. And updating the weight coefficient of the target side chain by using the formula (1) after the data of the commodity is linked to the allocated target side chain.
Obtaining a second read average number and a second write average number of data access objects on the data storage device based on the data access time parameter; calculating an exponential form read average and an exponential form write average for each data access object using the second read average and the second write average, respectively; and summing the exponential form reading average and the exponential form writing average, calculating the reciprocal, and respectively taking the reciprocal as a new weight coefficient of the data storage device corresponding to the reciprocal. It can be understood that the weight coefficients of the side chains are updated as an integral average along the time axis.
Assuming that N commodity objects exist on the ith side chain, the jth (j belongs to 1-N) commodity object obtains the t-th reading average number r through the visit quantity prediction modelijtAverage number of writes of wijt。
The exponential read number r of the jth commodity on the ith side chainijAnd the exponential write average wijRespectively expressed as:
the weighting factor of the ith side chain can be updated according to the above formula (1). When data of a certain new commodity object arrives, polling is carried out according to the current weight coefficient so as to distribute the new commodity object. And updating the weight coefficient of the target side chain by using the formula (1) after the data of the commodity is linked to the allocated target side chain.
In the embodiment of the application, the access amount of the data of the existing commodity object of the data storage device is predicted according to the read-write time characteristic of the commodity object in a certain time period, so that the data storage device can meet the access performance constraint condition in a specified time period in the future, and the performance reduction caused when a system deals with a large number of data access requests is avoided. The specified time period may be, for example, one week, one month, or other setting.
It should be noted that while the operations of the disclosed methods are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Rather, the steps depicted in the flowcharts may change the order of execution. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
It should be understood that the units or modules described in the apparatus 100-200 correspond to the various steps in the method described with reference to fig. 5-6. Thus, the operations and features described above with respect to the method are equally applicable to the apparatus 100 and 200 and the units included therein, and are not described in detail herein. The apparatus 100-200 may be implemented in a browser or other security applications of the electronic device in advance, or may be loaded into the browser or other security applications of the electronic device by downloading or the like. The corresponding units in the apparatus 100 and 200 may cooperate with units in the electronic device to implement the solution of the embodiment of the present application.
The division into several modules or units mentioned in the above detailed description is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Referring now to FIG. 7, shown is a block diagram of a computer system 700 suitable for use in implementing a terminal device or server of an embodiment of the present application.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU)701, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data necessary for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.
In particular, the processes described above with reference to the flow diagrams of fig. 5-6 may be implemented as computer software programs, according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a machine-readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program executes the above-described functions defined in the system of the present application when executed by the Central Processing Unit (CPU) 701.
It should be noted that the computer readable media shown in the present disclosure may be computer readable signal media or computer readable storage media or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units or modules described in the embodiments of the present application may be implemented by software or hardware. The described units or modules may also be provided in a processor, and may be described as: a processor includes a receiving unit, an allocating unit, and an updating unit. Where the names of such units or modules do not in some cases constitute a limitation of the unit or module itself, for example, a receiving unit may also be described as a "unit for receiving a data access request for at least one data access object".
As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may be separate and not incorporated into the electronic device. The computer-readable storage medium stores one or more programs that, when executed by one or more processors, perform the load balancing method described herein.
For example, the electronic device may implement the following as shown in fig. 5: step 501, receiving a data access request of at least one data access object;
502, distributing the data access request to a data storage device meeting the access performance constraint condition according to a weight polling algorithm;
in the allocation process, the weighting coefficients of the data storage device are dynamically updated based on a data access time parameter corresponding to the data access object, wherein the data access time parameter is used for representing the predicted trend of the read average number and the write average number of the data access object in the statistical period along with the change of the time. As another example, the electronic device may implement the various steps as shown in fig. 6 and 6.
The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the disclosure. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.
Claims (10)
1. A load balancing apparatus, comprising:
a receiving unit, configured to receive a data access request of at least one data access object;
the distribution unit is used for distributing the data access request to the data storage device meeting the access performance constraint condition according to a weight polling algorithm;
and the updating unit is used for dynamically updating the weight coefficient of the data storage device based on a data access time parameter corresponding to the data access object in the distribution process, wherein the data access time parameter is used for representing the predicted trend of the read average number and the write average number of the data access object in the statistical period changing along with the time.
2. The load balancing apparatus according to claim 1, wherein the updating unit further comprises:
the extraction subunit is used for extracting the characteristic value set of the data access request;
the prediction subunit is used for inputting the characteristic value set into a pre-established access quantity prediction model so as to obtain a data access time parameter corresponding to the data access object;
a calculating subunit, configured to calculate a weight coefficient of the data storage device based on the data access time parameter.
3. The load balancing apparatus of claim 2, wherein the computing subunit is further configured to:
obtaining a first read average and a first write average for the data access objects on the data storage device based on the data access time parameter;
and summing the first reading average and the first writing average, then calculating the reciprocal, and using the reciprocal as a new weight coefficient of the data storage device corresponding to the reciprocal.
4. The load balancing apparatus of claim 2, wherein the computing subunit is further configured to:
obtaining a second read average number and a second write average number of the data access objects on the data storage device based on the data access time parameter;
calculating an exponential form read average and an exponential form write average for each of the data access objects using the second read average and the second write average, respectively;
and summing the exponential form reading average and the exponential form writing average, calculating the reciprocal, and respectively using the reciprocal as a new weight coefficient of the data storage device corresponding to the reciprocal.
5. A blockchain system, comprising:
the load balancing device of any one of claims 1-4;
and the side chain is used as a data storage device and used for storing and verifying the transaction data corresponding to the data access request.
6. A method for load balancing, the method comprising:
receiving a data access request of at least one data access object;
distributing the data access request to a data storage device meeting the access performance constraint condition according to a weight polling algorithm;
and dynamically updating the weight coefficient of the data storage device based on a data access time parameter corresponding to the data access object in the distribution process, wherein the data access time parameter is used for representing the predicted trend of the read average number and the write average number of the data access object in the statistical period changing along with the time.
7. The method of claim 6, wherein updating the weighting coefficients of the data storage device based on the data access time parameter corresponding to the data access object comprises:
extracting a set of feature values of the data access request;
inputting the characteristic value set into a pre-established access quantity prediction model to obtain a data access time parameter corresponding to the data access object;
calculating a weight coefficient of the data storage device based on the data access time parameter.
8. The method of claim 7, wherein said calculating a weight coefficient for the data storage device based on the data access time parameter comprises:
obtaining a first read average and a first write average for the data access objects on the data storage device based on the data access time parameter;
and summing the first reading average and the first writing average, then calculating the reciprocal, and using the reciprocal as a new weight coefficient of the data storage device corresponding to the reciprocal.
9. The method of claim 7, wherein said calculating a weight coefficient for the data storage device based on the data access time parameter comprises:
obtaining a second read average number and a second write average number of the data access objects on the data storage device based on the data access time parameter;
calculating an exponential form read average and an exponential form write average for each of the data access objects using the second read average and the second write average, respectively;
and summing the exponential form reading average and the exponential form writing average, calculating the reciprocal, and respectively using the reciprocal as a new weight coefficient of the data storage device corresponding to the reciprocal.
10. The method according to any one of claims 7 to 9, wherein pre-establishing the visit volume prediction model comprises:
acquiring historical data of the at least one data access object;
extracting a set of feature values and a set of label values based on the historical data;
and training and predicting by using the characteristic value set and the label value set by adopting a machine learning algorithm to obtain an access quantity prediction model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910135696.1A CN111611067B (en) | 2019-02-22 | 2019-02-22 | Load balancing method, device and blockchain system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910135696.1A CN111611067B (en) | 2019-02-22 | 2019-02-22 | Load balancing method, device and blockchain system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111611067A true CN111611067A (en) | 2020-09-01 |
| CN111611067B CN111611067B (en) | 2024-06-14 |
Family
ID=72204576
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910135696.1A Active CN111611067B (en) | 2019-02-22 | 2019-02-22 | Load balancing method, device and blockchain system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111611067B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106648456A (en) * | 2016-09-18 | 2017-05-10 | 重庆邮电大学 | Dynamic save file access method based on use page view and prediction mechanism |
| CN106873907A (en) * | 2017-01-09 | 2017-06-20 | 中国电子科技集团公司第五十二研究所 | A kind of multi-controller memory array reads and writes load-balancing method and device |
| CN106909613A (en) * | 2017-01-11 | 2017-06-30 | 华南理工大学 | A kind of method of adaptive equalization database access service |
| CN107426332A (en) * | 2017-08-10 | 2017-12-01 | 华南理工大学 | The load-balancing method and system of a kind of web server cluster |
-
2019
- 2019-02-22 CN CN201910135696.1A patent/CN111611067B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106648456A (en) * | 2016-09-18 | 2017-05-10 | 重庆邮电大学 | Dynamic save file access method based on use page view and prediction mechanism |
| CN106873907A (en) * | 2017-01-09 | 2017-06-20 | 中国电子科技集团公司第五十二研究所 | A kind of multi-controller memory array reads and writes load-balancing method and device |
| CN106909613A (en) * | 2017-01-11 | 2017-06-30 | 华南理工大学 | A kind of method of adaptive equalization database access service |
| CN107426332A (en) * | 2017-08-10 | 2017-12-01 | 华南理工大学 | The load-balancing method and system of a kind of web server cluster |
Non-Patent Citations (2)
| Title |
|---|
| JAGDISH CHANDRA PATNI等: "Distributed load balancing model for grid computing environment", 《2015 1ST INTERNATIONAL CONFERENCE ON NEXT GENERATION COMPUTING TECHNOLOGIES (NGCT)》, 11 January 2016 (2016-01-11), pages 123 - 126 * |
| 申泽星: "集群服务器在混合请求下的负载均衡优化算法", 《计算机工程与应用》, vol. 54, no. 18, 30 September 2018 (2018-09-30), pages 99 - 104 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111611067B (en) | 2024-06-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9870251B2 (en) | Semantic-aware and user-aware admission control for performance management in data analytics and data storage systems | |
| CN110134516B (en) | Financial data processing method, apparatus, device and computer readable storage medium | |
| CN111427971B (en) | Business modeling method, device, system and medium for computer system | |
| CN113037877B (en) | Optimization method for time-space data and resource scheduling under cloud edge architecture | |
| CN110209677A (en) | The method and apparatus of more new data | |
| US8849745B2 (en) | Decision support methods and apparatus | |
| CN108897626A (en) | Resource scheduling method and server | |
| WO2021180056A1 (en) | Method for resource migration, system and device | |
| CN109767274B (en) | Method and system for carrying out associated storage on massive invoice data | |
| CN109521943A (en) | The distribution method and Related product of cloud database instance | |
| CN111008767B (en) | Internet financial technology architecture evaluation method, device, electronic equipment and medium | |
| CN114266281A (en) | Method, device and system for training graph neural network | |
| CN113780611A (en) | Inventory management method and device | |
| CN113761003A (en) | User portrait data processing method and device, electronic equipment and readable storage medium | |
| CN110738508A (en) | data analysis method and device | |
| CN112162762B (en) | Gray level distribution method, gray level distribution device and electronic equipment | |
| US10931776B2 (en) | Adaptive data fetching from network storage | |
| CN116737373A (en) | Load balancing method, device, computer equipment and storage medium | |
| CN111611067A (en) | Load balancing method and device and block chain system | |
| CN110704488B (en) | Method for managing data and corresponding system, computer device and medium | |
| CN110706051B (en) | Order data aggregation method and device and server | |
| CN112131257A (en) | Data query method and device | |
| CN110019165A (en) | A kind of method and apparatus for cleaning abnormal data | |
| CN113760521B (en) | Virtual resource allocation method and device | |
| CN116257175A (en) | Automatic driving data loading method, system and storage medium for caching based on memory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |