CN114328425A - High-concurrency data issuing request-oriented power data sharing device and method - Google Patents
High-concurrency data issuing request-oriented power data sharing device and method Download PDFInfo
- Publication number
- CN114328425A CN114328425A CN202111425160.7A CN202111425160A CN114328425A CN 114328425 A CN114328425 A CN 114328425A CN 202111425160 A CN202111425160 A CN 202111425160A CN 114328425 A CN114328425 A CN 114328425A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- client
- module
- time interval
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention belongs to the field of electric power information processing, and provides an electric power data sharing device and method for a high concurrent data issuing request. The device comprises an application module, a service module and an interface module; the application module is used for acquiring different service application requests for the data to be issued of the electricity consumption information acquisition system from the peripheral system, decomposing the required data into minimum data objects, correspondingly extracting and converting the data, and storing the data in an interface database of the interface module in advance; the service module comprises a computing cluster, a service module and a service module, wherein the computing cluster is used for carrying out data configuration according to a data release request, organizing minimum data objects in the interface database into corresponding data packets, transmitting the data packets to the service cluster and carrying out data release based on the size of the data packets; the application module is further used for estimating the downloading time interval of each data packet for the client, acquiring the Gaussian confidence determined by the optimal release strategy, releasing the downloading time interval of each data packet for the client and storing the downloading time interval in the interface database.
Description
Technical Field
The invention belongs to the field of electric power information processing, and particularly relates to a high concurrent data issuing request-oriented electric power data sharing device and method.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
The electricity consumption information acquisition system is used as a basic data platform of a full-service data center marketing domain, so that on one hand, the data acquisition period is continuously shortened, ten-million-level equipment access is supported, and data integration with the full-service data center marketing domain is realized; on the other hand, service interaction and data service such as fee control instruction issuing, table code uploading and the like are provided to the outside in a unified manner.
However, the inventor finds that the current power business data sharing mode is limited by the initial design target and bearing capacity, the timeliness of developing business applications based on the data sharing service cannot be met, the time for overall data acquisition and data landing is slowed down, and the current situations that the computing requirements of various types of businesses of the current peripheral system are increased and the sharing quantity of acquired data is increased cannot be met.
Disclosure of Invention
In order to solve the technical problems in the background art, the invention provides a power data sharing device and method for a high concurrent data issue request, which can improve the interactivity and stability of data sharing.
In order to achieve the purpose, the invention adopts the following technical scheme:
the invention provides a high-concurrency data issuing request-oriented power data sharing device, which comprises an application module, a service module and an interface module;
the application module is configured to: acquiring different service application requests for data to be issued of the electricity consumption information acquisition system from a peripheral system, decomposing the required data into minimum data objects, correspondingly extracting and converting the data, and storing the data in an interface database of an interface module in advance;
the service module comprises a computing cluster, a service module and a service module, wherein the computing cluster is used for carrying out data configuration according to a data release request, organizing minimum data objects in an interface database into corresponding data packets according to different requests, transmitting the data packets to the service cluster, carrying out data release based on the size of the data packets and determining data release time;
the application module is further configured to: and estimating the downloading time interval of each data packet for the client based on the data release time, acquiring the Gaussian confidence determined by the optimal release strategy, and then releasing the downloading time interval of each data packet for the client and storing the downloading time interval in an interface database.
The application module comprises a demand management module, a configuration management module and a monitoring management module;
the demand management module is used for decomposing data required by the application request according to the application request;
the configuration management module is used for providing data object configuration service based on data decomposition;
the monitoring management module is used for carrying out online monitoring management and control on the shared service information.
As an implementation manner, the computing cluster includes a static data publishing computing cluster and a real-time data publishing computing cluster, where the static data publishing computing cluster is used to publish historical offline data, and the real-time data publishing computing cluster is used to obtain real-time data and push subscribed data to an external system in real time.
As an implementation manner, the service cluster includes an interface service cluster and a file service cluster, the interface service cluster is responsible for providing a uniform interface service to the outside in a distributed load balancing manner, and the file service cluster provides a big data file download service to the outside.
As an implementation manner, when the service cluster issues data, if the issued data amount is less than a set issuing number threshold, directly returning the data through the interface service cluster; otherwise, the data packets are placed on the file service nodes through the file service cluster, and the downloading time interval of each data packet for the client is estimated based on the downloading record data of the client.
As an implementation manner, the file service cluster predicts the download time interval of each data packet for the client based on the client download record data and the gaussian regression prediction model.
As an embodiment, the application module is further configured to: judging the Gaussian confidence determined by the optimal release strategy, judging whether the mean difference of the Gaussian confidence is smaller than a set threshold value, and if so, directly releasing the predicted download time interval of each data packet available for the client; otherwise, adjusting the download time interval of each data packet for the client.
In an embodiment, the application module is further configured to predict an environmental state of the interface database in a next time period by using a markov decision process before obtaining the gaussian confidence determined by the optimal publishing policy, so as to map the environmental state to a downloading time interval of the client.
A second aspect of the present invention provides a power data sharing method for a high concurrent data issue request, including:
the application module acquires different service application requests for the data to be issued of the electricity consumption information acquisition system from the peripheral system, decomposes the required data into minimum data objects, correspondingly extracts and converts the data, and stores the data in an interface database of the interface module in advance;
the computing cluster performs data configuration according to the data publishing request, organizes the minimum data object in the interface database into a corresponding data packet according to different requests, transmits the data packet to the service cluster, performs data publishing based on the size of the data packet and determines the data publishing time;
and the application module estimates the downloading time interval of each data packet for the client based on the data release time, acquires the Gaussian confidence determined by the optimal release strategy, and then releases the downloading time interval of each data packet for the client and stores the downloading time interval in the interface database.
As an embodiment, the power data sharing method facing to the high concurrent data issue request further includes:
judging the Gaussian confidence determined by the optimal release strategy, judging whether the mean difference of the Gaussian confidence is smaller than a set threshold value, and if so, directly releasing the predicted download time interval of each data packet available for the client; otherwise, adjusting the download time interval of each data packet for the client.
As an implementation manner, before obtaining the gaussian confidence determined by the optimal publishing policy, a markov decision process is adopted to predict an environment state of the interface database in a next time period, so as to map the environment state to a downloading time interval of the client.
Compared with the prior art, the invention has the beneficial effects that:
(1) the invention provides a power data sharing method and device facing to a high concurrent data issuing request, which adopt data sharing to hierarchically accelerate timeliness of data from acquisition to application and improve data sharing efficiency; on the other hand, real-time control and management of the whole cluster resources are enhanced, and interactivity and stability of data sharing are improved based on service performance and resource consumption prediction, so that a data base is laid for constructing a novel power system taking new energy as a supply main body and a strong smart grid as a hub platform, and realizing dynamic characteristics of safety, controllability, flexibility, high efficiency, intelligence, friendliness and open interaction.
(2) Aiming at the problem of low data sharing timeliness of the electricity consumption information acquisition system, the data sharing module is constructed, data caching service is provided through the interface database, resource consumption caused by frequent access of peripheral system application to the original production database is reduced, and a data sharing layering idea is adopted, so that on one hand, data are classified and issued based on priority, and processing timeliness of core services is prevented from being influenced by non-core services; on the other hand, the real-time control and management of the whole resources of the data sharing module are realized through the feedback of the monitoring information, idle resources are avoided, and the utilization rate of cluster arrangement resources is improved.
(3) Aiming at non-real-time data packet downloading service with large data volume, the invention adopts a Gaussian regression prediction model to predict the downloading time zones of various data packets according to the downloading records of the client, on one hand, the difference and the tendency of the client to the downloading time zones are evaluated based on multidimensional downloading characteristics, thereby ensuring that the downloading time zones provided for the client of the peripheral system meet the requirement of the downloading effective time zones; on the other hand, the Gaussian regression prediction has the advantages of strong generalization, easy realization, output probability and the like, and can adjust the downloading time zone to avoid the occupation of the core service resources when the network bandwidth resource environment changes or the input and the output of the interface database are limited.
(4) In order to realize real-time control and management of the whole resources of the data sharing module, the environment state (including the storage state of an interface database and the network resource state) in the next time period is predicted by adopting a Markov decision process, and feedback adjustment is carried out based on the prediction result, so that when the resources are idle or tense, strategy adjustment is carried out in advance, idle resources are avoided, and the sharing service efficiency is accelerated.
Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the invention and together with the description serve to explain the invention and not to limit the invention.
Fig. 1 is a schematic structural diagram of a power data sharing device for high-concurrency data distribution requests according to an embodiment of the present invention;
fig. 2 is a flowchart of a power data sharing method for a high-concurrency data issue request according to an embodiment of the present invention;
FIG. 3 is a flow chart of a time interval prediction based on a Gaussian regression prediction model according to an embodiment of the present invention;
fig. 4 is a flowchart of an environmental state estimation model based on a markov decision process according to an embodiment of the present invention.
Detailed Description
The invention is further described with reference to the following figures and examples.
It is to be understood that the following detailed description is exemplary and is intended to provide further explanation of the invention as claimed. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
Example one
Referring to fig. 1 and fig. 2, the present embodiment provides a schematic structural diagram of a power data sharing apparatus facing a high concurrent data issue request, which specifically includes an application module, a service module, and an interface module.
In a specific implementation process, the application module is configured to: and acquiring different service application requests for the data to be issued of the power utilization information acquisition system from the peripheral system, decomposing the required data into minimum data objects, correspondingly extracting and converting the data, and storing the data in an interface database of the interface module in advance.
In fig. 1, the application modules include a demand management module, a configuration management module, and a monitoring management module;
the demand management module is used for decomposing data required by the application request according to the application request; for example: carrying out standard management and control on the processes of application, approval, execution and the like of a sharing request of data sharing required by the power utilization information acquisition system;
the configuration management module is used for providing data object configuration service based on data decomposition; for example: carrying out standard control on the access authority of the shared service, the operation parameters of the data sharing module and the configuration data release model;
the monitoring management module is used for carrying out online monitoring management control on the shared service information; for example: and carrying out online monitoring and control on the information such as the running state, the calling condition, the interactive flow, the shared log, the database storage resource and the like of the shared service.
The demand management module acquires different service application requests for the data to be issued of the electricity consumption information acquisition system from the peripheral system, and decomposes the required data into each minimum data object according to the application requests.
Specifically, a physical architecture for data sharing of the power consumption information acquisition system relies on virtualized resources of Aliyun, the power consumption information acquisition system is divided into a computing node container group, a service node container group and a file storage service group according to functions, and data sharing services are provided for peripheral systems such as a marketing service system through firewall equipment and network equipment which are uniformly deployed by the power consumption information acquisition system. More than 10 ten thousand pieces of data containing 52 types of service application requests and corresponding request data downloading records from 2021 month to 2021 month and 9 month are obtained by screening based on a power consumption information acquisition system production library and serve as example samples, as shown in table 1, the example samples are part of service request examples:
table 1 example of partial service requests and related to collecting data
And decomposing the 52-type service application requests to obtain each minimum data object, extracting and converting required data from the power utilization information acquisition system production database by the data object configuration service of the configuration management module, storing the required data into the interface database of the interface module in advance, and uniformly storing and managing the data to be issued.
In a specific implementation process, the service module includes a computing cluster, which is used for performing data configuration according to a data publishing request, organizing minimum data objects in the interface database into corresponding data packets according to different requests, transmitting the data packets to the service cluster, publishing the data based on the size of the data packets, and determining data publishing time.
The computing clusters comprise static data publishing computing clusters and real-time data publishing computing clusters, the static data publishing computing clusters are used for publishing historical offline data (including publishing time computation, publishing data object sets and the like), and the real-time data publishing computing clusters are used for acquiring real-time data and pushing subscribed data to an external system in real time.
For example: the real-time data publishing and calculating cluster acquires real-time data through technologies of one-sending and double-receiving of a front-end processor of the power utilization information acquisition system, independent application and grabbing and the like, and pushes subscribed data to an external system in real time through a client.
The service cluster comprises an interface service cluster and a file service cluster, the interface service cluster is responsible for providing uniform interface service for the outside in a distributed load balancing mode, and the file service cluster provides big data file downloading service for the outside.
Specifically, the service module acquires different data release requests and distributes the different data release requests to the computing cluster, the computing cluster performs data configuration according to dimensions such as data range, data frequency, data content combination and the like of the data release requests, minimum data units in the interface database are organized into required data packets according to different requests, and the data packets are transmitted to the service cluster;
in some embodiments, when the service cluster issues data, if the issued data amount is less than a set issue number threshold (for example: 1TB), the service cluster directly returns data through the interface service cluster; otherwise, the data packets are placed on the file service nodes through the file service cluster, and the downloading time interval of each data packet for the client is estimated based on the downloading record data of the client.
In some embodiments, the file service cluster predicts a download time interval for each data packet available to the client based on the client download log data and the gaussian regression prediction model. For example: if the issued data volume is larger than or equal to 1TB, the data packets are placed on the file service nodes through the file service cluster, downloading record data of the client side are obtained based on historical records in log files of the monitoring management module, and a Gaussian regression prediction model is adopted to predict downloading time intervals of the data packets for the client side.
Specifically, as shown in fig. 3, the method for calculating the earliest download time and the latest download time of the data packet by using the gaussian regression prediction model specifically includes the following steps:
step 1.1, acquiring client download record data based on a historical record in a log file, wherein the client download record data comprises d (for example, d is 6) dimensional feature vectors which are respectively a data packet size, a data packet weight value, historical download duration of a data packet, historical download time of the data packet, historical download times of the data packet and a download process network transmission rate, and a historical download record sample set and sample feature data are formed;
step 1.2, clustering historical download record samples by adopting a K-means clustering method, relatively uniformly and randomly selecting a plurality of (such as 26) samples as initial clustering central points based on download time and download data quantity, calculating the sum of each characteristic distance among the samples, sequentially dividing the current cluster and iteratively updating the mean vector of each cluster by adopting a greedy strategy until the clustering result is kept unchanged after the iterative updating, outputting the clustering result of the historical download record sample set, and dividing each cluster into a training sample set and a testing sample set according to the proportion;
step 1.3, according to the clustering result, performing data definition of a Gaussian regression model based on the clustering sample set;
a clustered set of training samples is defined as
Wherein X ═ X1,x2,…,xd]If d is 5-dimensional feature vector for the historical download record sample, the input feature vector of the ith sample is Xi＝[xi1,xi2,…,xid]，Y＝[y1,y2,…,yr]Is the actual output value of the sample, and its corresponding set of outputs satisfying the Gaussian process regression is { f (x)1),f(x2),…,f(xr) The relationship between the two is as follows:
wherein ε is independent white Gaussian noise and is also in accordance with Gaussian distribution, the mean is 0, and the variance is σ2Is marked as epsilon-N (0, sigma)2)；
Step 1.4, constructing a kernel function based on historical sample characteristic data of the clustered sample set, and calculating to obtain a clustered sample set kernel matrix;
the linear combination of two independent Gaussian distributions still satisfies the Gaussian distribution, so here Y satisfies the Gaussian distribution, i.e.
Y～GP(m(X),C(X,X)) (3)
Wherein, m (X) ═ E (f (X)) is obtained by calculating a mean value based on historical sample data of the clustered sample set, and C (X, X) ═ K (X, X) + σ2I, I is an r X r identity matrix, C (X, X) is an r X r covariance matrix,
k (X, X) is a kernel function matrix of r X r:
wherein, the element Kij＝k(xi,xj)，k(xi,xj)＝E[(f(xi)-m(xi))(f(xj)-m(xj))]The method is calculated and obtained based on historical sample data of the clustered sample set.
The kernel function being a covariance of the squared exponential type, i.e.
Wherein, theta0Is a hyper-parameter, represents the average distance between each sample mean function and the cluster sample mean function,
θ1representing the length of the function "wobble", this parameter is determined as follows:
setting the objective function to logp (Y | X) ═ logN (μ, K)y) And (3) solving an optimal value by a gradient descent method, namely:
when the gradient decrease value is smaller than the set threshold value, it is judged that p (Y | X) is the largest under the conditions of Y-GP (m (X), C (X, X)), and at this time, the fixed parameter theta is1；
Step 1.5, taking the kernel matrix as a covariance matrix of joint Gaussian distribution, and performing joint calculation on the sample data to be tested and the historical sample data of the clustered sample set to obtain a joint prior distribution matrix of the sample to be tested and the historical sample set;
data to be tested { X*,y*The joint Gaussian density distribution obtained by the Gaussian process with the same distribution as the training data { X, Y } "is
Wherein K is a covariance matrix;
step 1.6, according to the probability distribution of the clustered historical sample set and the posterior probability of the sample to be detected, Gaussian regression prediction of the sample set to be detected is obtained by adopting a Bayesian formula;
calculating output vector y according to Bayes formula*Conditional probability of (2)
y*|y～N(μ*+K*K-1(y-μ),K**-K*K-1K*T) (8)
y*Is predicted to be
y*Variance, i.e., prediction uncertainty, of
v(y*)＝Var[y*|X,y,X*]＝K(X*,X*)-K(X*,X)(K(X,X)+σ2I)-1K(X,X*) (10)
By training the mapping f (-) between the input data set X and the output data set Y: rd→ R, to realize the input data x under test to be tested*When the sample data satisfies the mean and variance of the Gaussian distribution, f (x) is output*)；
And step 1.7, setting a confidence phi of the Gaussian regression model, and determining a time confidence interval of data packet downloading based on Gaussian regression prediction of a sample to be tested, namely determining the earliest downloading time and the latest downloading time of the data packet.
In a specific implementation, the application module is further configured to: and estimating the downloading time interval of each data packet for the client based on the data release time, acquiring the Gaussian confidence determined by the optimal release strategy, and then releasing the downloading time interval of each data packet for the client and storing the downloading time interval in an interface database.
Specifically, the application module is further configured to predict an environment state of the interface database in a next time period by using a markov decision process before obtaining the gaussian confidence determined by the optimal publishing policy, so as to map the environment state to a download time interval of the client.
For example: the method comprises the steps of obtaining a request data volume, a request priority, a resource storage state, a network resource state and a downloading time interval of each data packet which is estimated by a service cluster and is available to a client side from 10 ten thousand historical downloading records of 2021 month to 2021 month and 9 month based on a monitoring management module, predicting an environment state of an interface database in a next time period by adopting a Markov decision process, and obtaining a Gauss confidence coefficient determined by an optimal release strategy.
As shown in fig. 4, the specific process of obtaining the gaussian confidence determined by the optimal issuing policy includes:
step 2.1. define state decision quadruplet M ═ (S, a, P)sa,R,γ)；
Wherein S represents a set of states, and S ═ S (S)1,s2,…si,…)，siIndicating that the interface database in the ith step is in the resource state, and the resource state is expressed in percentage; a represents a set of actions, A ═ a1,a2,…ai,…)，aiRepresenting the action of the ith step, representing three states of data storage, data downloading and data deletion of the interface database; p represents the probability of a state transition, PsaRepresenting the state from the data environment state to the state S after taking the action a ∈ A under the current state of S ∈ SiTransition to the next state si+1At state siLower execution action aiTransition to the next state si+1Is expressed as p(s)i+1|si,ai) The data is stored in the earliest downloading time based on the division result of the time interval, the data is for downloading in the time interval, the table data is deleted after the latest downloading time, and the state transition probability is 1; r represents a return function of
If one group(s)i,ai) Move to the next state si+1Then the return function is denoted as r(s)i+1|si,ai) The r value is set by the interface database to be 40% of the residual capacity and the satisfaction degree of the download processing result of the client is equal to 0, 100%]Determining, wherein the optimal remaining capacity is calculated by Euclidean distanceDegree is Euclidean distance index; gamma is belonged to 0,1]Is a fold-down factor;
step 2.2, initializing a value function V (S) ═ 0, and determining downloading time by setting a confidence coefficient phi, so that loading, downloading and deleting are performed based on time mapping action (namely, action strategy), which is marked as strategy pi: S → A, and a strategy pi (S) is formed;
defining a function V of state valuesπ(s) represents the value function of state s under strategy π:
wherein r isiRepresenting the immediate return of step i in the future, s' representing the status of the next step, Vπ(s) represents the slave state s0Long-term effects, V, arising from the onset of strategy ππ(s ') long-term effects of strategy pi are adopted in the next state, r (s ' | s, a) represents that action a is adopted in state s, and immediate return is generated to the next state s ';
given a strategy pi and an initial state s, the initial action a is determined by the strategy pi and the state s, i.e. the action a ═ pi(s), the next moment will go to the next state s 'with a probability p (s' | s, a), then the above equation is expressed based on the transition probability as:
at the current state s and the current action a, following the policy π in the future, the function Q of the action value will be defined with a probability p (s' | s, a) to turn to the next state sπ(s, a) is as follows:
based on Vπ(s) and QπThe above expression of (s, a), whose bellman equation is expressed as follows:
thus, in the dynamic programming, the relationship of the value function of the current state to the value function of the next state is indicated.
The optimization objective pi is then expressed as:
respectively recording optimal strategies pi*The corresponding state value function and behavior value function are V*(s) and Q*(s, a) in the following relationship:
the state value function and the behavior value function respectively satisfy the following Bellman optimality equations as follows:
step 2.3. calculating its state value function V based on arbitrary strategy piπ(s) performing a strategy estimation, calculating v(s) to converge;
based on equation (14), under a certain policy π, there are three possible actions a corresponding to π(s), each possible action being denoted as π (a | s), then the state value function extension is defined as follows:
first, all V are put togetherπThe initial value of(s) is assigned to 0, and the value function for all states s is then updated based on equation (21):
in the formula, k represents the number of iterations, only one array is used for storing each state value function, and each time a new value is obtained, the old value is covered, and the shape is [ V ]k+1(s1),Vk+1(s2),Vk(s3),…]Denotes the state s1、s2Are all updated, state s3The new value is not updated, so that the new value can be utilized in time in the iteration process, and faster convergence is realized;
setting a constant threshold value theta, and judging each state s, | Vk(s)-Vk+1(s)|<Theta, if so, V is determinedk+1(s) is the state value function V of the current strategy piπ(s)；
Step 2.4, obtaining state value function V of strategy pi based on strategy estimation of step 2.3π(s) in order to find a better strategy, changing the action a adopted in the state s by changing the confidence coefficient phi, and carrying out strategy improvement based on a greedy thought maxq (s, a);
traversing all states and all possible actions a, and adopting a greedy strategy to obtain a new strategy pi ', namely changing the confidence coefficient phi, thereby changing the action a adopted in the state s, generating a new strategy pi', and calculating the action value function Q based on the formula (15)π′(s,a)；
Judgment of Qπ′Whether or not (s, a) is greater than Vπ(s) if Qπ′(s,a)>Vπ(s), the new policy pi' (action a is only used in state s, and the old policy pi is followed in other states) is higher than the old policy (policy pi is followed in all states) function value, so the policy is updated by the following formula:
obtaining a new strategy through a greedy strategy adopting a value function;
and 2.5, if a new strategy pi ' exists, returning the strategy pi ' to the step E3, and if the new strategy pi ' does not exist, outputting the current strategy pi as the optimal strategy pi*；
Step 2.6. optimal strategy pi*And correspondingly mapping action transfer generated based on the environment state into a client downloading time interval, and outputting the Gaussian confidence phi mapped in the step 1.7 based on the highest proportion symmetrical interval.
In some embodiments, the application module is further to: judging the Gaussian confidence determined by the optimal release strategy, judging whether the average difference of the Gaussian confidence is smaller than a set threshold epsilon, if so, directly releasing the predicted download time interval of each data packet available for the client; otherwise, adjusting the download time interval of each data packet for the client.
Based on the estimated download time interval of each data packet available to the client, 50 and 500 cases of data sharing request data are respectively subjected to simulation tests based on the method of the embodiment on 10, 15 and 18 of 2021, and the earliest download time and the latest download time of each client for downloading the data packet based on the method of the embodiment are informed, and the experimental comparison results are as follows:
as can be seen from the above table, it is difficult to consider the real-time evolution state of the environmental resources and the preference of each client for downloading data time in the fixed client downloading time in the conventional process, so that when the daily sharing service request amount increases, on one hand, there is a risk of saturation of the database storage resources and the network transmission resources, and on the other hand, part of the data sharing service is incomplete and includes two parts, namely poor timeliness of the core service and incomplete downloading data of the client, the data sharing process in the patent can allocate downloading time based on the client downloading time preference, and realize the resource occupation estimation of the next time period based on the environmental state, so that the data sharing service can be guaranteed to adopt an optimized execution strategy, and when the shared data amount increases suddenly, the high-quality sharing service can still be provided;
and finally, the peripheral system takes the downloaded data away in a determined time interval through the client, and the interface database clears the expired data.
Example two
As shown in fig. 2, the present embodiment further provides a power data sharing method facing to a high concurrent data issue request, which includes:
the application module acquires different service application requests for the data to be issued of the electricity consumption information acquisition system from the peripheral system, decomposes the required data into minimum data objects, correspondingly extracts and converts the data, and stores the data in an interface database of the interface module in advance;
the computing cluster performs data configuration according to the data publishing request, organizes the minimum data object in the interface database into a corresponding data packet according to different requests, transmits the data packet to the service cluster, performs data publishing based on the size of the data packet and determines the data publishing time;
and the application module estimates the downloading time interval of each data packet for the client based on the data release time, acquires the Gaussian confidence determined by the optimal release strategy, and then releases the downloading time interval of each data packet for the client and stores the downloading time interval in the interface database.
The power data sharing method facing the high concurrent data issuing request further comprises the following steps:
judging the Gaussian confidence determined by the optimal release strategy, judging whether the mean difference of the Gaussian confidence is smaller than a set threshold value, and if so, directly releasing the predicted download time interval of each data packet available for the client; otherwise, adjusting the download time interval of each data packet for the client.
In the implementation process, a gaussian regression prediction model is used to estimate the downloading time interval of each data packet for the client, and the specific steps are as shown in fig. 3 in the first embodiment, which will not be described again here.
And before the Gaussian confidence determined by the optimal release strategy is obtained, predicting the environment state of the interface database in the next time period by adopting a Markov decision process so as to map the environment state into a downloading time interval of the client.
Specifically, based on the download time interval of each data packet available to the client, which is obtained by the monitoring management module and obtained by the historical download record, the request priority, the resource storage state, the network resource state and the service cluster prediction, the environment state of the interface database in the next time period is predicted by adopting the markov decision process, and the gaussian confidence determined by the optimal release strategy is obtained. The specific process of obtaining the gaussian confidence determined by the optimal issuing policy is shown in fig. 4 in the first embodiment, and will not be described here again.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A power data sharing device facing to a high concurrent data issuing request is characterized by comprising an application module, a service module and an interface module;
the application module is configured to: acquiring different service application requests for data to be issued of the electricity consumption information acquisition system from a peripheral system, decomposing the required data into minimum data objects, correspondingly extracting and converting the data, and storing the data in an interface database of an interface module in advance;
the service module comprises a computing cluster, a service module and a service module, wherein the computing cluster is used for carrying out data configuration according to a data release request, organizing minimum data objects in an interface database into corresponding data packets according to different requests, transmitting the data packets to the service cluster, carrying out data release based on the size of the data packets and determining data release time;
the application module is further configured to: and estimating the downloading time interval of each data packet for the client based on the data release time, acquiring the Gaussian confidence determined by the optimal release strategy, and then releasing the downloading time interval of each data packet for the client and storing the downloading time interval in an interface database.
2. The high concurrent data publication request oriented power data sharing device according to claim 1, wherein the application modules include a demand management module, a configuration management module and a monitoring management module;
the demand management module is used for decomposing data required by the application request according to the application request;
the configuration management module is used for providing data object configuration service based on data decomposition;
the monitoring management module is used for carrying out online monitoring management and control on the shared service information.
3. The power data sharing device facing to the highly concurrent data publishing request according to claim 1, wherein the computing clusters include a static data publishing computing cluster and a real-time data publishing computing cluster, the static data publishing computing cluster is configured to publish the historical offline data, and the real-time data publishing computing cluster is configured to obtain the real-time data and push subscribed data to an external system in real time.
4. The power data sharing device facing to the high concurrent data issue request according to claim 1, wherein the service cluster includes an interface service cluster and a file service cluster, the interface service cluster is responsible for providing a unified interface service to the outside in a distributed load balancing manner, and the file service cluster provides a big data file download service to the outside.
5. The power data sharing device facing to the high concurrent data issuing request according to claim 4, wherein when the service cluster issues data, if the issued data amount is less than a set issuing number threshold, the service cluster directly returns data through the interface service cluster; otherwise, the data packets are placed on the file service nodes through the file service cluster, and the downloading time interval of each data packet for the client is estimated based on the downloading record data of the client.
6. The high-concurrency data publication request-oriented power data sharing device according to claim 5, wherein the file service cluster estimates a download time interval of each data packet available to the client based on the client download record data and a Gaussian regression prediction model.
7. The high concurrent data publication request oriented power data sharing device according to claim 1, wherein the application module is further configured to: judging the Gaussian confidence determined by the optimal release strategy, judging whether the mean difference of the Gaussian confidence is smaller than a set threshold value, and if so, directly releasing the predicted download time interval of each data packet available for the client; otherwise, adjusting the download time interval of each data packet for the client.
8. The power data sharing device for high concurrent data publication requests according to claim 1, wherein the application module is further configured to predict an environment state of the interface database in a next time period by using a markov decision process before obtaining the gaussian confidence determined by the optimal publication policy, so as to map the environment state to a download time interval of the client.
9. A power data sharing method facing to a high concurrent data issuing request is characterized by comprising the following steps:
the application module acquires different service application requests for the data to be issued of the electricity consumption information acquisition system from the peripheral system, decomposes the required data into minimum data objects, correspondingly extracts and converts the data, and stores the data in an interface database of the interface module in advance;
the computing cluster performs data configuration according to the data publishing request, organizes the minimum data object in the interface database into a corresponding data packet according to different requests, transmits the data packet to the service cluster, performs data publishing based on the size of the data packet and determines the data publishing time;
and the application module estimates the downloading time interval of each data packet for the client based on the data release time, acquires the Gaussian confidence determined by the optimal release strategy, and then releases the downloading time interval of each data packet for the client and stores the downloading time interval in the interface database.
10. The high-concurrency data publication request-oriented power data sharing method according to claim 9, wherein the high-concurrency data publication request-oriented power data sharing method further comprises:
judging the Gaussian confidence determined by the optimal release strategy, judging whether the mean difference of the Gaussian confidence is smaller than a set threshold value, and if so, directly releasing the predicted download time interval of each data packet available for the client; otherwise, adjusting the download time interval of each data packet for the client;
or
And before the Gaussian confidence determined by the optimal release strategy is obtained, predicting the environment state of the interface database in the next time period by adopting a Markov decision process so as to map the environment state into a downloading time interval of the client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111425160.7A CN114328425A (en) | 2021-11-26 | 2021-11-26 | High-concurrency data issuing request-oriented power data sharing device and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111425160.7A CN114328425A (en) | 2021-11-26 | 2021-11-26 | High-concurrency data issuing request-oriented power data sharing device and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114328425A true CN114328425A (en) | 2022-04-12 |
Family
ID=81045861
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111425160.7A Pending CN114328425A (en) | 2021-11-26 | 2021-11-26 | High-concurrency data issuing request-oriented power data sharing device and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114328425A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115242879A (en) * | 2022-06-29 | 2022-10-25 | 浪潮通信技术有限公司 | Data sharing system and method |
| CN117252655A (en) * | 2023-11-20 | 2023-12-19 | 畅捷通信息技术股份有限公司 | Invoice downloading method, invoice downloading device, computing equipment and storage medium |
-
2021
- 2021-11-26 CN CN202111425160.7A patent/CN114328425A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115242879A (en) * | 2022-06-29 | 2022-10-25 | 浪潮通信技术有限公司 | Data sharing system and method |
| CN115242879B (en) * | 2022-06-29 | 2024-04-02 | 浪潮通信技术有限公司 | Data sharing system and method |
| CN117252655A (en) * | 2023-11-20 | 2023-12-19 | 畅捷通信息技术股份有限公司 | Invoice downloading method, invoice downloading device, computing equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Bhattacharjee et al. | Barista: Efficient and scalable serverless serving system for deep learning prediction services | |
| CN114328425A (en) | High-concurrency data issuing request-oriented power data sharing device and method | |
| CN113037877B (en) | Optimization method for time-space data and resource scheduling under cloud edge architecture | |
| Hamrouni et al. | A survey of dynamic replication and replica selection strategies based on data mining techniques in data grids | |
| Hu et al. | An efficient online computation offloading approach for large-scale mobile edge computing via deep reinforcement learning | |
| Nguyen et al. | User-aware and flexible proactive caching using LSTM and ensemble learning in IoT-MEC networks | |
| Donida Labati et al. | Computational intelligence in cloud computing | |
| CN113822456A (en) | Service combination optimization deployment method based on deep reinforcement learning in cloud and mist mixed environment | |
| CN115408152A (en) | Adaptive resource matching obtaining method and system | |
| Dai et al. | A learning algorithm for real-time service in vehicular networks with mobile-edge computing | |
| Soula et al. | Intelligent tasks allocation at the edge based on machine learning and bio-inspired algorithms | |
| Karthikeyan et al. | COSCO2: AI‐augmented evolutionary algorithm based workload prediction framework for sustainable cloud data centers | |
| Wang | Edge artificial intelligence-based affinity task offloading under resource adjustment in a 5G network | |
| Li et al. | Neighborhood search-based job scheduling for IoT big data real-time processing in distributed edge-cloud computing environment | |
| CN114936708A (en) | Fault diagnosis optimization method based on edge cloud collaborative task unloading and electronic equipment | |
| Tekiyehband et al. | An efficient dynamic service provisioning mechanism in fog computing environment: A learning automata approach | |
| Khaledian et al. | An energy-efficient and deadline-aware workflow scheduling algorithm in the fog and cloud environment | |
| Zhou et al. | Load balancing prediction method of cloud storage based on analytic hierarchy process and hybrid hierarchical genetic algorithm | |
| Devagnanam et al. | Design and development of exponential lion algorithm for optimal allocation of cluster resources in cloud | |
| Afzali et al. | An efficient resource allocation of IoT requests in hybrid fog–cloud environment | |
| CN114693141B (en) | Transformer substation inspection method based on end edge cooperation | |
| CN116109058A (en) | Substation inspection management method and device based on deep reinforcement learning | |
| CN115562940A (en) | Load energy consumption monitoring method and device, medium and electronic equipment | |
| CN111784029A (en) | Fog node resource allocation method | |
| ANKITA et al. | Solving grid scheduling problem using an amalgam of swarm and traditional approach |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |