CN111695967A - Method, device, equipment and storage medium for determining quotation - Google Patents
Method, device, equipment and storage medium for determining quotation Download PDFInfo
- Publication number
- CN111695967A CN111695967A CN202010501337.6A CN202010501337A CN111695967A CN 111695967 A CN111695967 A CN 111695967A CN 202010501337 A CN202010501337 A CN 202010501337A CN 111695967 A CN111695967 A CN 111695967A
- Authority
- CN
- China
- Prior art keywords
- quotation
- model
- target
- resource
- measured
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 50
- 238000012549 training Methods 0.000 claims abstract description 45
- 230000002787 reinforcement Effects 0.000 claims abstract description 31
- 230000008901 benefit Effects 0.000 claims abstract description 20
- 230000006870 function Effects 0.000 claims description 23
- 230000009471 action Effects 0.000 claims description 18
- 238000011156 evaluation Methods 0.000 claims description 4
- 230000000875 corresponding effect Effects 0.000 description 36
- 238000010586 diagram Methods 0.000 description 11
- 238000012545 processing Methods 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 6
- 238000004088 simulation Methods 0.000 description 6
- 239000000446 fuel Substances 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 239000003795 chemical substances by application Substances 0.000 description 4
- 230000006399 behavior Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 230000005611 electricity Effects 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000010187 selection method Methods 0.000 description 2
- 229910000831 Steel Inorganic materials 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 235000013305 food Nutrition 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 239000008267 milk Substances 0.000 description 1
- 210000004080 milk Anatomy 0.000 description 1
- 235000013336 milk Nutrition 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000010959 steel Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/06—Buying, selling or leasing transactions
- G06Q30/0601—Electronic shopping [e-shopping]
- G06Q30/0611—Request for offers or quotes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Economics (AREA)
- Health & Medical Sciences (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- General Health & Medical Sciences (AREA)
- Finance (AREA)
- Accounting & Taxation (AREA)
- Human Resources & Organizations (AREA)
- Biophysics (AREA)
- Public Health (AREA)
- Development Economics (AREA)
- Primary Health Care (AREA)
- Tourism & Hospitality (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Water Supply & Treatment (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The embodiment of the invention discloses a method, a device, equipment and a storage medium for determining quotation. The method comprises the following steps: acquiring a measured state parameter and a measured resource parameter of a target resource, wherein the measured resource parameter comprises at least one measured price coefficient; and inputting the measured state parameters and the measured resource parameters into a pre-trained target quotation model to obtain an output target quotation coefficient and a target income corresponding to the target quotation coefficient, wherein the target quotation model is obtained by training based on a deep reinforcement learning algorithm. According to the embodiment of the invention, the target quotation model is trained based on the deep reinforcement learning algorithm, so that the problem that the accuracy of quotation is influenced by incomplete information is solved, and an optimal quotation decision is provided for a supplier of target resources, so that the market benefit of the supplier is maximized.
Description
Technical Field
The embodiment of the invention relates to the technical field of electric power, in particular to a method, a device, equipment and a storage medium for determining quotations.
Background
In the market environment, resource providers always optimize their bidding strategies to obtain higher profits. Since resource providers are not familiar with the market environment and cannot grasp complete market information, a perfect quotation strategy theory is required as guidance. An efficient quotation decision tool can help decision makers and quotation staff to make a successful quotation and thereby obtain a high amount of revenue. In addition, the research and the deduction of the quotation behaviors of the resource suppliers are facilitated, and the market supervision authorities can investigate the behaviors of the resource suppliers, so that the existing loopholes in the market rules can be identified, and the policy and the regulation of the market are continuously improved.
Since the market information is not complete for the resource provider, the resource provider has a great difficulty in optimizing its own quotation strategy. The traditional quotation strategy research method of the resource provider is mainly based on a game theory method which is very useful for theoretically discussing the optimal bidding strategy of market members and relatively roughly researching the bidding behavior of the resource provider, but the game theory method has low practicability due to inherent defects, so the game theory method is not suitable for researching a complete bidding strategy, and the obtained simulation result is inaccurate.
Disclosure of Invention
The embodiment of the invention provides a method, a device, equipment and a storage medium for determining quotation, which are used for providing an optimal quotation decision by a supply object of target resources so as to maximize the market income of the supply object.
In a first aspect, an embodiment of the present invention provides a method for determining an offer, where the method includes:
acquiring a measured state parameter and a measured resource parameter of a target resource, wherein the measured resource parameter comprises at least one measured price coefficient;
and inputting the measured state parameters and the measured resource parameters into a pre-trained target quotation model to obtain an output target quotation coefficient and a target income corresponding to the target quotation coefficient, wherein the target quotation model is obtained by training based on a deep reinforcement learning algorithm.
In a second aspect, an embodiment of the present invention further provides an apparatus for determining an offer, where the apparatus includes:
the system comprises a measured resource parameter acquisition module, a resource evaluation module and a resource evaluation module, wherein the measured resource parameter acquisition module is used for acquiring a measured state parameter and a measured resource parameter of a target resource, and the measured resource parameter comprises at least one measured price coefficient;
and the target quotation coefficient output module is used for inputting the measured state parameters and the measured resource parameters into a pre-trained target quotation model to obtain an output target quotation coefficient and target income corresponding to the target quotation coefficient, wherein the target quotation model is obtained based on deep reinforcement learning algorithm training.
In a third aspect, an embodiment of the present invention further provides an apparatus, where the apparatus includes:
one or more processors;
a memory for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement any of the offer determination methods referred to above.
In a fourth aspect, embodiments of the present invention also provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform any of the above mentioned methods of determining an offer.
According to the embodiment of the invention, the target quotation model is trained based on the deep reinforcement learning algorithm, so that the problem that the accuracy of quotation is influenced by incomplete information is solved, and an optimal quotation decision is provided for a supplier of target resources, so that the market benefit of the supplier is maximized.
Drawings
Fig. 1 is a flowchart of a method for determining an offer according to an embodiment of the present invention.
Fig. 2 is a flowchart of a method for determining an offer according to a second embodiment of the present invention.
Fig. 3 is a flowchart of a method for determining an offer according to a third embodiment of the present invention.
Fig. 4A is a schematic diagram of a specific example of a simulated power market according to a third embodiment of the present invention.
Fig. 4B is a schematic diagram of a market user load according to a third embodiment of the present invention.
Fig. 4C is a schematic diagram of a training result of an initial offer model according to a third embodiment of the present invention.
Fig. 5 is a schematic diagram of an apparatus for determining an offer according to a fourth embodiment of the present invention.
Fig. 6 is a schematic structural diagram of an apparatus according to a fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a method for determining an offer according to an embodiment of the present invention, where the present embodiment is applicable to a case of performing offer simulation on a target resource, and the method may be performed by an offer determining device, which may be implemented in software and/or hardware, and may be configured in a terminal device. The method specifically comprises the following steps:
s110, acquiring the measured state parameter and the measured resource parameter of the target resource.
The target resource may be, for example, a hardware device, such as an ultrasound device, an energy source material, such as steel, or a food, such as milk.
The reinforcement learning algorithm consists of five parts, namely an agent, a state parameter, an action parameter, an incentive parameter and an environment. The intelligent agent can make corresponding action according to the input state parameters, and the environment can receive the action and return the next state parameters and the reward corresponding to the action. The above process is repeated continuously in order to train the agent to select the action that will result in the greatest reward at any state parameter. In this embodiment, the measured status parameters may be a market clearing price and a market resource demand. The market clearing price refers to a balance price obtained by the supply quantity of the market balance target resource and the demand quantity of the market resource. At this price, the resource provider is not over-or under-supplied, nor is the market over-or under-demanded.
In one embodiment, optionally, the target resource is a power resource, and the measured state parameters of the power resource are a market clearing price and a market electricity load level. In this embodiment, the agent is a target pricing model and the environment is the electricity market. Wherein, the market electric quantity load level refers to the electric quantity demand of the user. For example, the measured state parameter may be a state parameter fed back by the power market after last actual quoted price.
Exemplary measured resource parameters include, but are not limited to, a fuel cost function, a marginal cost function, a maximum value and a minimum value of the amount of power supply, among others. Specifically, the fuel cost function satisfies the formula:
wherein, Ci(PGi) A fuel cost function for resource object i; pGiThe amount of power supplied to resource object i; a isi、biAnd ciRespectively, a first order coefficient, a second order coefficient and a constant term coefficient of the fuel cost, and G represents a resource object set. The unit of the power supply amount can be megawatt-hour.
Correspondingly, a derivative calculation is performed on the fuel cost function to obtain a marginal cost function, and specifically, the marginal cost function satisfies a formula:
wherein the content of the first and second substances,
is the marginal cost function of resource object i.
Further, the electric energy bidding function of the resource object satisfies the formula:
wherein, P (P)Gi) An electric energy bidding curve of the resource object i; k is a radical ofiThe power bid factor selected for resource object i.
In one embodiment, optionally, the measured resource parameter includes at least one measured pricing factor. Wherein, the measured price coefficient is the electric energy bidding coefficient. In practical application, after the resource object determines the target quotation coefficient, the electric energy bidding function constructed based on the target quotation coefficient is provided for the electric power market. In the present embodiment, the action parameter in the reinforcement learning algorithm is the measured price coefficient.
And S120, inputting the measured state parameters and the measured resource parameters into a pre-trained target price quotation model to obtain an output target price quotation coefficient and a target income corresponding to the target price quotation coefficient.
The target offer model is used for predicting a target offer coefficient corresponding to the maximum profit. The target price quotation coefficient is one of the measured price quotation coefficients in the measured resource parameters, and the target income is the maximum income value in the income values corresponding to all the measured price quotation coefficients.
In one embodiment, optionally, the target offer model is trained based on a deep reinforcement learning algorithm. The deep reinforcement learning algorithm is a neural network algorithm combining deep learning and reinforcement learning. And outputting the reward expectation value corresponding to each action parameter and state parameter by the reinforcement learning algorithm to form a reward expectation value list. However, the reinforcement learning algorithm is limited by the dimension and cannot obtain all reward expectation values obtained after actions are executed under all state parameters, and the reinforcement learning algorithm trains the model parameters of the current network model based on the training samples obtained by the current network model under the current state parameters, so that the training samples have strong relevance. The deep learning algorithm is a model for predicting unknown data based on a current data set, so that the deep reinforcement learning algorithm is used for training another neural network model to fit an incentive expectation list and training the incentive expectation value obtained based on the reinforcement learning algorithm according to the incentive expectation value output by the neural network model until the incentive expectation value output by the reinforcement learning model approaches to a real incentive expectation value.
According to the technical scheme, the target offer model is trained on the basis of the deep reinforcement learning algorithm, the problem that the offer accuracy is affected by incomplete information is solved, and an optimal offer decision is provided for a supply object of target resources, so that the market income of the supply object is maximized.
Example two
Fig. 2 is a flowchart of a method for determining a quote price according to a second embodiment of the present invention, and the technical solution of the present embodiment is further detailed based on the above-mentioned embodiment. Optionally, the target offer model is obtained by training based on a deep reinforcement learning algorithm, and includes: acquiring a sample state parameter and a sample resource parameter of a target resource, and inputting the sample state parameter and the sample resource parameter into an initial quotation model; wherein the sample resource parameters include at least one sample quotation coefficient; determining sample data according to an output result of the initial quotation model, and storing the sample data in an experience playback pool; wherein the sample data comprises a prediction benefit output by the initial quotation model; and determining the comparison yield output by the comparison quotation model based on the sample data stored in the experience playback pool according to a preset training frequency, and performing iterative training on the initial quotation model based on the comparison yield, the prediction yield and the standard yield until a trained target quotation model is obtained.
The specific implementation steps of this embodiment include:
s210, obtaining the sample state parameters and the sample resource parameters of the target resources, and inputting the initial sample state parameters and the sample resource parameters into an initial quotation model.
In one embodiment, optionally, the sample resource parameters include at least one sample quotation factor.
And S220, determining sample data according to an output result of the initial quotation model, and storing the sample data in an experience playback pool.
Specifically, the initial quotation model selects a sample quotation coefficient as a current sample quotation coefficient under the current sample state parameter, and outputs a prediction gain corresponding to the current sample quotation coefficient.
In one embodiment, optionally, the initial offer model is used to: determining a preset action selection mode according to the iteration times corresponding to the current iteration training; the preset action selection mode comprises a uniform random selection mode or a selection mode based on a greedy algorithm; and determining a current sample quotation coefficient based on a preset action selection mode, and outputting current prediction benefits corresponding to the current sample quotation coefficient.
In an exemplary embodiment, a mapping relationship between the iteration number and the preset action selection mode is established, and the preset action selection mode corresponding to the iteration number is determined according to the mapping relationship. In one embodiment, optionally, the current sample offer coefficient is determined by a uniform random selection method, and then determined by a selection method of a greedy algorithm. Illustratively, when the iteration times are 0-1000 times, the current sample quotation coefficient is determined by adopting a uniform random selection mode, and when the iteration times are 1001-iteration time threshold, the current sample quotation coefficient is determined by adopting a selection mode of a greedy algorithm.
Wherein, the uniform random selection refers to randomly selecting the current sample quotation coefficient a from at least one sample quotation coefficientrCurrent sample quote factor arSatisfies the following formula:
wherein, p (a)r|sn) Representing the selection of the current sample quotation coefficient arProbability of(s)nAnd the sample state parameter in the nth iteration is represented, and N represents the number of sample quotation coefficients.
Wherein, the greedy algorithm refers to the maximum prediction income pair corresponding to the current sample state parameterTaking the corresponding sample quotation coefficient as the current sample quotation coefficient apCurrent sample quote factor apSatisfies the following formula:
ap＝argmaxQn-1(sn,a)
wherein s isnRepresenting the state parameter of the sample at the nth iteration, Qn-1(snAnd a) represents the state s of the network model updated in the last iterationnThe prediction gain when the sample quotation coefficient a is selected; wherein the sample quotation coefficient a comprises the current sample quotation coefficient ap。
In one embodiment, optionally, the sample data includes predicted revenue output by the initial price quote model. In one embodiment, optionally, the sample data further includes a current sample quotation coefficient, a current sample state parameter, and a next sample state parameter output by the initial quotation model. Wherein, the next sample state parameter can be obtained based on a simulated market clearing algorithm.
And S230, determining comparison benefits output by the comparison quotation model based on sample data stored in the experience playback pool according to the preset training frequency, and performing iterative training on the initial quotation model based on the comparison benefits, the prediction benefits and the standard benefits until a trained target quotation model is obtained.
For example, the preset training frequency may be 10 times or 20 times. Specifically, a process of outputting the one-time predicted profit by the initial quotation model is defined as a simulation process, and after the initial quotation model executes the simulation process for 10 times, the comparison profit output by the comparison quotation model is determined based on sample data stored in the empirical playback pool. In an embodiment, optionally, when the storage space of the experience playback pool is zero, the comparison benefit output by the comparison offer model is determined based on the sample data stored in the experience playback pool according to the preset training frequency. Specifically, when the storage space of the experience playback pool is 100 data, after the experience playback pool has been filled with 100 sample data, the comparison benefit output by the comparison offer model is determined based on the sample data stored in the experience playback pool according to the preset training frequency.
In one embodiment, optionally, determining a control benefit output against the offer model based on sample data stored in the empirical playback pool comprises: and selecting sample data from the experience playback pool, and inputting the sample data into the comparison quotation model to obtain output comparison benefits corresponding to the sample data. Specifically, the reference quotation model determines a profit value corresponding to each sample quotation coefficient based on the next sample state parameter in the sample data, and the reference profit is the maximum profit value among all the profit values.
In one embodiment, optionally, the model parameters of the comparison offer model are updated based on the model parameters of the initial offer model according to a preset interval; wherein, the network structure of the initial quotation model is the same as that of the contrast quotation model. The preset time interval may be 1 minute or 10 minutes, for example. Specifically, the network parameter of the initial quotation model is recorded as θ, and the network parameter of the comparison quotation model is recorded as θ'. And when the model parameters of the initial quotation model update the model parameters of the comparison quotation model, the model parameters of the initial quotation model satisfy theta-theta'. Wherein the model parameters of the initial quote model are updated in each iteration training. In another embodiment, optionally, when the model parameters of the initial offer model are iteratively updated a preset number of times, the model parameters of the comparison offer model are updated based on the model parameters of the initial offer model. Here, the preset number may be 10 or 50, for example.
In one embodiment, optionally, a loss function is calculated based on the control gains, the predicted gains, and the standard gains, and the initial bid model is iteratively trained based on the loss function and a gradient descent algorithm until a trained target bid model is obtained.
Specifically, the loss function satisfies the following formula:
L(θ)＝([rn+γmaxQn(sn+1,a'|θ')]-Qn(s,a|θ))2
wherein r isnFor standard gain, gamma is the attenuation coefficient, Qn(s, a | θ) as the initial quote model at state snSample offers for down selectionPredicted benefit of coefficient a output, maxQn(sn+1A '| θ') is in the state s of the comparison quotation modeln+1Next, the maximum comparison gain is output among all the offer coefficients, and a' is a sample offer coefficient corresponding to the maximum comparison gain.
Specifically, the standard profit may be calculated based on a simulated market clearing algorithm. Specifically, the function value of the loss function is minimized based on a gradient descent algorithm, and a trained target price quotation model is obtained.
S240, acquiring the measured state parameter and the measured resource parameter of the target resource.
And S250, inputting the measured state parameters and the measured resource parameters into a pre-trained target quotation model to obtain an output target quotation coefficient and a target income corresponding to the target quotation coefficient.
According to the technical scheme of the embodiment, model parameters of the initial quotation model are trained based on the comparison yield output by the comparison quotation model, so that the problem of strong correlation among sample data of the reinforcement learning algorithm is solved, the dimension disaster of the reinforcement learning algorithm is avoided, and the convergence of the initial quotation model is accelerated.
EXAMPLE III
Fig. 3 is a flowchart of a method for determining a quoted price according to a third embodiment of the present invention, and the technical solution of the present embodiment is further detailed based on the above-mentioned embodiments. Optionally, determining a next sample state parameter according to a current sample quotation coefficient provided by different resource objects based on the current sample state parameter; wherein the resource object comprises a resource object corresponding to the initial offer model; and determining the standard income corresponding to the current sample state parameter according to the next sample state parameter.
The specific implementation steps of this embodiment include:
s310, obtaining the sample state parameters and the sample resource parameters of the target resources, and inputting the initial sample state parameters and the sample resource parameters into an initial quotation model.
And S320, determining sample data according to an output result of the initial quotation model, and storing the sample data in an experience playback pool.
And S330, determining comparison benefits output by the comparison quotation model based on the sample data stored in the experience playback pool according to the preset training frequency.
S340, determining a next sample state parameter according to a current sample quotation coefficient provided by different resource objects based on the current sample state parameter, and determining a standard gain corresponding to the current sample state parameter according to the next sample state parameter.
Wherein the resource objects include resource objects corresponding to the initial offer model. The manner of providing the current sample quotation coefficient by other resource objects besides the initial quotation model described in this embodiment is not limited, and for example, the current sample quotation coefficient provided by other resource objects may be defined, or the current sample quotation coefficient may be provided by other target quotation models.
In practical application, the power market integrates the quotation coefficients provided by a plurality of resource objects, the market power load condition and other factors, and obtains the market clearing price after bidding and the corresponding winning power of each resource object. In this embodiment, a simulated market clearing algorithm is constructed, and the next sample state parameter is obtained based on the simulated market clearing algorithm. The simulated market clearing algorithm specifically comprises the following steps: and determining the next sample state parameter according to the current sample quotation coefficient, the power supply amount range and the market load information provided by different resource objects. Wherein, the simulated market clearing algorithm satisfies the formula:
wherein f is the cost of all resource objects; l is a network node set; branch is a branch set; lambda [ alpha ]elClearing the price for the market of the node l; pDhThe load requirement of the h-th user; xxyIs a branchReactance value of xy; thetaxAnd thetayThe phase angles corresponding to the nodes x and y respectively; pxymaxFor the current limit of branch ij, PGimaxAnd PGiminRespectively representing the upper limit and the lower limit of the generating capacity of the generator i;
specifically, the market clearing price of each network node and the bid amount corresponding to each resource object can be calculated through the simulated market clearing algorithm. Wherein the bid amount for each resource object is summed to equal the market electricity load level in the sample status parameter.
Specifically, the standard profit is calculated according to the market clearing price and the winning bid amount of the network node corresponding to the initial quotation model, and a calculation formula of the standard profit satisfies:
wherein λ iseAnd clearing the price for the market of the network node where the resource object i corresponding to the initial quotation model is located. PGiIs the winning bid amount of resource object i.
And S350, performing iterative training on the initial quotation model based on the comparison income, the prediction income and the standard income until a trained target quotation model is obtained.
In an embodiment, optionally, when the iteration number of the initial offer model reaches a preset iteration number threshold, ending the iterative training of the initial offer model to obtain a trained target offer model.
And S360, acquiring the measured state parameter and the measured resource parameter of the target resource.
And S370, inputting the measured state parameters and the measured resource parameters into a pre-trained target quotation model to obtain an output target quotation coefficient and a target income corresponding to the target quotation coefficient.
Fig. 4A is a schematic diagram of a specific example of a simulated power market according to a third embodiment of the present invention. As shown in fig. 3, the simulated power market includes 3 network nodes, node 1, node 2, and node 3, respectively, G1, G2, G3, G4, and G5 representing resource object 1, resource object 2, resource object 3, resource object 4, and resource object 5, respectively. Specifically, G1 access node 1, G2, G3, G4 and G5 all access node 2, and the user load access node 3.
Table 1 shows parameter information of a sample resource parameter provided in the third embodiment of the present invention.
Fig. 4B is a schematic diagram of a market user load according to a third embodiment of the present invention. The abscissa in fig. 4B represents the period in units of h and the ordinate represents the user load in units of MW. The 24 hour market user load demand is shown in FIG. 4B.
In this specific example, the storage space of the experience playback pool is set to 6000, the iteration number of randomly selecting the current sample offer coefficient is set to 2400 times before, and the selection of the current sample offer coefficient based on the greedy algorithm is set to 1200 times after, so that the preset iteration number threshold of the initial offer model is 9600 times. And setting the preset training frequency to 10 times, namely after the initial quotation model executes 10 times of simulation processes, performing iterative updating on model parameters of the initial quotation model. And after the model parameters of the initial quotation model are updated for 5 times in an iteration mode, updating the model parameters of the comparison quotation model based on the model parameters of the initial quotation model.
Fig. 4C is a schematic diagram of a training result of an initial offer model according to a third embodiment of the present invention. The lower abscissa of fig. 4C represents the number of deep dual-Q network iterations, i.e., the number of iterations of the initial quote model trained based on the deep reinforcement learning algorithm, and the upper abscissa of fig. 4C represents the number of Q-learning iterations, i.e., the number of iterations of the initial quote model trained based on the reinforcement learning algorithm. The thicker curve in fig. 4C represents the training results based on the deep dual-Q network, and the thinner curve represents the training results based on the Q-learning network. From FIG. 4C, the baseThe number of iterations in the Q-learning network satisfies N500 × 24 12000, and the profit (i.e., profit) based on the Q-learning network is stabilized at 0.55 × 106The iteration number of the network based on the depth double-Q is equal to N150 × 24+6000 and 9600, and the profit value (i.e. profit) of the network based on the depth double-Q is stabilized at 0.95 × 106Left and right. Compared with a reinforcement learning algorithm, the initial offer model obtained based on deep reinforcement learning algorithm training has fewer iteration times, the profit is about 72% higher, and a better prediction profit value is achieved.
According to the technical scheme of the embodiment, the next sample state parameter and the standard profit are obtained through calculation based on the simulated market clearing algorithm, iterative training is carried out on the initial quotation model, the problem of environmental feedback in the deep reinforcement learning algorithm is solved, and the training effect and the convergence speed of the initial quotation model are improved.
Example four
Fig. 5 is a schematic diagram of an apparatus for determining an offer according to a fourth embodiment of the present invention. The embodiment can be applied to the case of performing offer simulation on the target resource, and the device can be implemented in a software and/or hardware manner, and can be configured in the terminal device. The bid amount determination device includes: a measured resource parameter acquisition module 410 and a target quotation coefficient output module 420.
The measured resource parameter obtaining module 410 is configured to obtain a measured state parameter and a measured resource parameter of a target resource, where the measured resource parameter includes at least one measured pricing coefficient;
and the target quotation coefficient output module 420 is configured to input the measured state parameters and the measured resource parameters into a pre-trained target quotation model to obtain an output target quotation coefficient and a target profit corresponding to the target quotation coefficient, where the target quotation model is obtained by training based on a deep reinforcement learning algorithm.
According to the technical scheme, the target offer model is trained on the basis of the deep reinforcement learning algorithm, the problem that the offer accuracy is affected by incomplete information is solved, and an optimal offer decision is provided for a supplier of target resources, so that the market income of the supplier is maximized.
On the basis of the technical scheme, optionally, the target resource is the power resource, and the measured state parameters of the power resource are the market clearing price and the market electric quantity load level.
On the basis of the above technical solution, optionally, the apparatus further includes a target offer model training module, and the target offer model training module includes:
the system comprises a sample state parameter acquisition unit, a data processing unit and a data processing unit, wherein the sample state parameter acquisition unit is used for acquiring a sample state parameter and a sample resource parameter of a target resource and inputting the sample state parameter and the sample resource parameter into an initial quotation model; wherein the sample resource parameters include at least one sample quotation coefficient;
the sample storage unit is used for determining sample data according to the output result of the initial quotation model and storing the sample data in the experience playback pool; the sample data comprises the prediction income output by the initial quotation model;
and the target quotation model training unit is used for determining the comparison yield output by the comparison quotation model based on the sample data stored in the experience playback pool according to the preset training frequency, and performing iterative training on the initial quotation model based on the comparison yield, the prediction yield and the standard yield until the trained target quotation model is obtained.
On the basis of the above technical solution, optionally, the apparatus further includes:
the next sample state parameter determining module is used for determining the next sample state parameter according to the current sample quotation coefficient provided by different resource objects based on the current sample state parameter; the resource objects comprise resource objects corresponding to the initial quotation model;
and the standard profit determining module is used for determining the standard profit corresponding to the current sample state parameter according to the next sample state parameter.
On the basis of the above technical solution, optionally, the apparatus further includes:
the comparison quotation model updating module is used for updating model parameters of the comparison quotation model based on the model parameters of the initial quotation model according to the preset interval time; wherein, the network structure of the initial quotation model is the same as that of the contrast quotation model.
On the basis of the above technical solution, optionally, the target offer model training unit is specifically configured to:
and calculating a loss function based on the comparison gain, the prediction gain and the standard gain, and performing iterative training on the initial quotation model based on the loss function and a gradient descent algorithm until a trained target quotation model is obtained.
On the basis of the above technical solution, optionally, the initial offer model is used for:
determining a preset action selection mode according to the iteration times corresponding to the current iteration training; the preset action selection mode comprises a uniform random selection mode or a selection mode based on a greedy algorithm;
and determining a current sample quotation coefficient based on a preset action selection mode, and outputting current prediction benefits corresponding to the current sample quotation coefficient.
The device for determining the quotation provided by the embodiment of the invention can be used for executing the method for determining the quotation provided by the embodiment of the invention, and has corresponding functions and beneficial effects of the executing method.
It should be noted that, in the embodiment of the quote price determination apparatus, the included units and modules are only divided according to the functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
EXAMPLE five
Fig. 6 is a schematic structural diagram of an apparatus provided in the fifth embodiment of the present invention, and the fifth embodiment of the present invention provides a service for implementing the method for determining a price quote according to the foregoing embodiment of the present invention, and may configure a price quote determining device in the foregoing embodiment. Fig. 6 illustrates a block diagram of an exemplary device 12 suitable for use in implementing embodiments of the present invention. The device 12 shown in fig. 6 is only an example and should not bring any limitations to the functionality and scope of use of the embodiments of the present invention.
As shown in FIG. 6, device 12 is in the form of a general purpose computing device. The components of device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 6, and commonly referred to as a "hard drive"). Although not shown in FIG. 6, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
The processing unit 16 executes various functional applications and data processing, such as implementing a method of determining a price quote provided by an embodiment of the present invention, by running a program stored in the system memory 28.
Through the equipment, the problem that the accuracy of quotation is influenced by incomplete information is solved, and an optimal quotation decision is provided for a supplier of target resources, so that the market income of the supplier is maximized.
EXAMPLE six
An embodiment of the present invention further provides a storage medium containing computer-executable instructions, which when executed by a computer processor, perform a method for determining an offer, the method including:
acquiring a measured state parameter and a measured resource parameter of a target resource, wherein the measured resource parameter comprises at least one measured price coefficient;
and inputting the measured state parameters and the measured resource parameters into a pre-trained target quotation model to obtain an output target quotation coefficient and a target income corresponding to the target quotation coefficient, wherein the target quotation model is obtained by training based on a deep reinforcement learning algorithm.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, or the like, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the above method operations, and may also perform related operations in the method for determining a price quote provided by any embodiment of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method for determining an offer, comprising:
acquiring a measured state parameter and a measured resource parameter of a target resource, wherein the measured resource parameter comprises at least one measured price coefficient;
and inputting the measured state parameters and the measured resource parameters into a pre-trained target quotation model to obtain an output target quotation coefficient and a target income corresponding to the target quotation coefficient, wherein the target quotation model is obtained by training based on a deep reinforcement learning algorithm.
2. The method of claim 1, wherein the target resource is a power resource and the measured state parameters of the power resource are a market clearing price and a market charge loading level.
3. The method of claim 2, wherein the target offer model is trained based on a deep reinforcement learning algorithm, comprising:
acquiring a sample state parameter and a sample resource parameter of a target resource, and inputting the sample state parameter and the sample resource parameter into an initial quotation model; wherein the sample resource parameters include at least one sample quotation coefficient;
determining sample data according to an output result of the initial quotation model, and storing the sample data in an experience playback pool; wherein the sample data comprises a prediction benefit output by the initial quotation model;
and determining comparison benefits output by comparison quotation models based on sample data stored in the experience playback pool according to a preset training frequency, and performing iterative training on the initial quotation models based on the comparison benefits, the prediction benefits and the standard benefits until a trained target quotation model is obtained.
4. The method of claim 3, further comprising:
determining a next sample state parameter according to a current sample quotation coefficient provided by different resource objects based on the current sample state parameter; wherein the resource object comprises a resource object corresponding to the initial offer model;
and determining the standard income corresponding to the current sample state parameter according to the next sample state parameter.
5. The method of claim 3, further comprising:
updating model parameters of a comparison quotation model based on the model parameters of the initial quotation model according to a preset interval time; wherein the network structure of the initial offer model is the same as the network structure of the comparison offer model.
6. The method of claim 5, wherein iteratively training the initial offer model until a trained target offer model is obtained based on the control revenue, the predicted revenue, and a standard revenue, comprises:
and calculating a loss function based on the comparison income, the prediction income and the standard income, and performing iterative training on the initial quotation model based on the loss function and a gradient descent algorithm until a trained target quotation model is obtained.
7. The method of claim 3, wherein the initial offer model is used to:
determining a preset action selection mode according to the iteration times corresponding to the current iteration training; the preset action selection mode comprises uniform random selection or a selection mode based on a greedy algorithm;
and determining a current sample quotation coefficient based on a preset action selection mode, and outputting a current prediction income corresponding to the current sample quotation coefficient.
8. An offer determination device, comprising:
the system comprises a measured resource parameter acquisition module, a resource evaluation module and a resource evaluation module, wherein the measured resource parameter acquisition module is used for acquiring a measured state parameter and a measured resource parameter of a target resource, and the measured resource parameter comprises at least one measured price coefficient;
and the target quotation coefficient output module is used for inputting the measured state parameters and the measured resource parameters into a pre-trained target quotation model to obtain an output target quotation coefficient and target income corresponding to the target quotation coefficient, wherein the target quotation model is obtained based on deep reinforcement learning algorithm training.
9. An apparatus, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a method of determining an offer as recited in any of claims 1-7.
10. A storage medium containing computer-executable instructions for performing the method of determining a bid of any of claims 1-7 when executed by a computer processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010501337.6A CN111695967A (en) | 2020-06-04 | 2020-06-04 | Method, device, equipment and storage medium for determining quotation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010501337.6A CN111695967A (en) | 2020-06-04 | 2020-06-04 | Method, device, equipment and storage medium for determining quotation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111695967A true CN111695967A (en) | 2020-09-22 |
Family
ID=72478958
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010501337.6A Pending CN111695967A (en) | 2020-06-04 | 2020-06-04 | Method, device, equipment and storage medium for determining quotation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111695967A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113240459A (en) * | 2021-04-27 | 2021-08-10 | 东南大学 | Market member quotation method based on deep reinforcement learning algorithm and module thereof |
| CN116051206A (en) * | 2023-03-27 | 2023-05-02 | 阿里健康科技(杭州)有限公司 | Advertisement delivery request sending method, device, equipment and storage medium |
-
2020
- 2020-06-04 CN CN202010501337.6A patent/CN111695967A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113240459A (en) * | 2021-04-27 | 2021-08-10 | 东南大学 | Market member quotation method based on deep reinforcement learning algorithm and module thereof |
| CN116051206A (en) * | 2023-03-27 | 2023-05-02 | 阿里健康科技(杭州)有限公司 | Advertisement delivery request sending method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Woodford | Monetary policy analysis when planning horizons are finite | |
| JP4661250B2 (en) | Prediction method, prediction device, and prediction program | |
| CN111401940B (en) | Feature prediction method, device, electronic equipment and storage medium | |
| CN112270547A (en) | Financial risk assessment method and device based on feature construction and electronic equipment | |
| CN111340221B (en) | Neural network structure sampling method and device | |
| CN112200489B (en) | Integrated optimization system, method and device for nonferrous metal smelting production and supply marketing | |
| CN113159835A (en) | Power generation side electricity price quotation method and device based on artificial intelligence, storage medium and electronic equipment | |
| CN111695967A (en) | Method, device, equipment and storage medium for determining quotation | |
| WO2022246843A1 (en) | Software project risk assessment method and apparatus, computer device, and storage medium | |
| CN116992151A (en) | Online course recommendation method based on double-tower graph convolution neural network | |
| CN110827069A (en) | Data processing method, device, medium, and electronic apparatus | |
| CN113344647B (en) | Information recommendation method and device | |
| CN114818913A (en) | Decision generation method and device | |
| CN111353872A (en) | Credit granting processing method and device based on financial performance value and electronic equipment | |
| CN112434872A (en) | Hotel yield prediction method, system, equipment and storage medium | |
| CN111957053A (en) | Game player matching method and device, storage medium and electronic equipment | |
| CN115760201A (en) | Intelligent sales lead recommendation reason generation method and system | |
| CN113094602B (en) | Hotel recommendation method, system, equipment and medium | |
| JP7060130B1 (en) | Operation support equipment, operation support methods and programs | |
| Su et al. | A combined model based on secondary decomposition technique and grey wolf optimizer for short-term wind power forecasting | |
| CN113570204A (en) | User behavior prediction method, system and computer equipment | |
| CN115496431A (en) | Order form and transport capacity matching method and device and electronic equipment | |
| CN112580885A (en) | Method, device and equipment for predicting accessory qualification rate and storage medium | |
| CN108984680B (en) | Information recommendation method and device, server and storage medium | |
| CN113627513A (en) | Training data generation method and system, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |