KR20190132193A - A Dynamic Pricing Demand Response Method and System for Smart Grid Systems - Google Patents

Abstract

Demand response (DR) has become a method effective in increasing grid reliability and saving energy in a smart grid system as information and communication technologies have advanced, and can promptly respond to the imbalance of demand and supply through flexible load adjustment on a demand side. The present invention suggests a dynamic pricing DR algorithm for energy management considering both the profits of a service provider (SP) and the expenses of a client in a power system market. Reinforcement learning is used to describe a hierarchical decision-making system. Here, a dynamic pricing problem is formalized as a discrete finite Markov decision process, and Q-learning is adopted to solve this decision-making problem. Using the reinforcement learning, the SP can adaptively determine a retail electricity price in an online learning process while resolving uncertainty about the load demand profile of a customer and the flexibility of a wholesale electricity price. A simulation result shows that the DR algorithm according to the present invention can increase the profitability of the SP, reduce customer the energy costs of the customer, balance energy supply and demand in an electricity market, and increase the reliability of a power system, which can thus be a win-win strategy for both. The dynamic pricing DR method comprises the steps of: selecting a retail electricity price; and updating a Q value of a Q-learning algorithm based on the retail electricity price and reward and energy information.

Description

A Dynamic Pricing Demand Response Method and System for Smart Grid Systems

The present invention relates to the demand response (DR) in the smart grid. Specifically, a service provider having a subscription customer in the electric power market provides a profitable service provider's profitability by providing an optimal electric retail price to the subscription customer using the reinforcement learning model. It relates to dynamic pricing demand response methods and systems in a smart grid that can increase the reliability of the power system by increasing energy costs of subscribers, balancing energy supply and demand in the power market.

As information and communication technology advances in smart grid systems, demand response has become an effective way to improve grid reliability and energy savings, and flexible load balancing on the demand side makes it possible to respond immediately to imbalances in supply and demand. According to the United States Department of Energy, DR said, "Providing incentives to drive changes in electricity prices over time or to lower power usage when market prices are high or grid stability is threatened. Tariffs or programs established for the purpose of

The prior art generally discusses two categories of DR, price and incentive base. Price-based DR motivates customers to change their energy usage patterns in response to changing electricity prices over time, while incentive-based DR provides fixed or time-varying incentives to customers as they reduce energy consumption during stress periods in their power systems. Both categories have inherent advantages and take advantage of many aspects of their potential for flexible demand. This study focuses on price-based DR where efficiency has been evaluated in several studies.

In order to maximize the social welfare of the smart grid system, there are many studies on price-based DR that focus on controlling home appliances directly from the customer's point of view. For example, energy consumption planning for home appliances has been studied in consideration of time-of-use pricing to reduce customer costs and increase energy efficiency. Similarly, some studies have evaluated the impact of large field deployments of TOU pricing on commercial and industrial customers' energy use. The authors of other studies examined commercial and industrial DR on major peak plots with predetermined time and duration of price increases. Some studies describe deterministic DR models that know the next day's electricity prices and predefine the optimal energy consumption schedule while minimizing daily costs. Other authors also proposed a real-time incentive-based strategy based on daily prices for large electrical customers. However, it is assumed that the duration and value of the incentive rate is determined ahead of schedule. In another study, the two previous price-based DR plans were designed for industrial loads, taking into account both current and future load control over a period of time. However, despite the author's modeling future price uncertainties, the mathematical formulas of these two papers are complex and the actual implementation is complex. To date, most of the previous studies on energy consumption schedules are based on given pricing policies and cannot accommodate the uncertainty of the dynamic electricity market environment. It is therefore essential to devise innovative dynamic pricing mechanisms for energy management.

Dynamic pricing is a business strategy that adjusts product prices in a timely manner to provide the right services to the right customers. There has been some research on dynamic pricing DR algorithms through smart grids. Previous studies have investigated dynamic pricing strategies using DR for microgrid retailers in integrated energy systems where retail prices and microgrid transmissions are formulated as mixed integer secondary programming issues to maximize retailer benefits. In another study, the Stackelberg game was used to model energy transactions between retailers and customers, who based on energy pricing to determine dynamic retail prices to maximize profits and then to customers at published prices. To manage the energy usage of the device. However, in this work, dynamic pricing distributed by retailers was predetermined. To some extent, these studies are still deterministic and cannot cope with the customer's demand profile and the flexibility of wholesale electricity prices in the power market.

Recently, with the rapid development of artificial intelligence, there is increasing interest in adopting reinforcement learning (RL) to solve decision problems in smart grid. Many breakthroughs have been reported in reinforcement learning, especially the deep Q-network developed by Atari and AlphaGo. Reinforcement learning is an area of machine learning inspired by behaviorist psychology, as shown in Figure 1, in which a software service provider must take action in a probabilistic environment to maximize the concept of cumulative rewards. The service provider is in a discrete time step and at each time step the service provider selects a job from a series of available jobs and sends them to the customer afterwards. The service provider is then rewarded and moves to a new customer. The goal of service providers is to have as many rewards as possible. Reinforcement learning algorithms have been used to plan energy storage systems and to obtain optimal charging policies for, for example, batteries or electric vehicles. This scenario is relatively easy due to the limited number of behaviors and states, making it the focus of many research papers. Other studies have used reinforcement learning to obtain energy schedule management methods for specific devices in the DR, such as electric water heaters, thermostatic loads, or other devices. The authors of other studies considered microgrids as a whole by giving each microgrid the capacity to buy or sell energy to other microgrids. Reinforcement learning was used between the microgrids as a means of selecting a purchasing / sales strategy for energy trading to maximize average income.

Korea Patent Registration No. 10-1647060

The present invention was devised to solve the above problems, and an object of the present invention is to increase the profitability of a service provider in the electric power market, reduce the energy cost of energy demand customers, and balance power supply and demand in the electric power market, thereby improving reliability of the electric power system. To improve.

To this end, the dynamic pricing demand response method in the smart grid according to the present invention is a dynamic pricing demand response method by a service provider, and selects an electric retail price arbitrarily from an electric retail price range based on an electric wholesale price obtained from a grid manager. Updating the Q value of the Q-learning algorithm based on the retail price, the compensation and the energy information, and the second step of acquiring the customer's energy information and the compensation based on the service provider's revenue and the customer's cost. And a third step of repeating the first to third steps for each time slot from the start to the end of the time slot.

In addition, the dynamic pricing demand response method in the smart grid according to the present invention is a dynamic pricing demand response method by the service provider in the smart grid, the service provider determines the electrical retail price for the subscription customer, the electrical retail price determined Iteratively repeats the process of acquiring compensation based on the energy consumption of the subscribed customers and the service provider's profits and customer costs according to the responsiveness, and optimizes the pricing policy when the cumulative compensation obtained by repeating the above process is maximized It is characterized by the selection of a pricing policy.

In addition, a program for executing the dynamic pricing demand response method in the smart grid according to the present invention is stored in a computer-readable recording medium, the program is a first step of receiving the electricity wholesale price of the grid manager and the energy demand of subscribing customers And a second step of initializing the Q value of the Q-learning algorithm to 0, the time slot t and the iteration i to 1, and selecting an electric retail price arbitrarily from the electric retail price range based on the electric wholesale price in the iteration i. A third step of calculating a compensation amount based on the service provider's revenue and the customer's cost and the customer's energy information, and updating the Q value of the Q-learning algorithm based on the retail price, the compensation amount and the energy information. Step 5, determining whether the time slot t is the last end time T; and if the time slot is not the end time, the third to sixth steps. A seventh step of repeating the steps, an eighth step of determining whether the Q value has reached the maximum value if the time slot is the end time, and repeating steps 3-7 if the Q value has not reached the maximum value A ninth step, and if the Q value reaches the maximum value, outputting the retail price with the pricing policy corresponding to the maximum Q value.

The present invention provides the following main contents.

1) In the power system market, we proposed a dynamic pricing algorithm for energy management that considers both the service provider's profit and the customer's cost.

2) The dynamic retail pricing problem is formulated into a finite discrete Markov decision process, and Q-learning is adopted to solve this decision problem. There is no need for a pre-specified model at retail pricing.

3) Uncertainty in the customer's load demand profile and flexibility in wholesale electricity prices are taken into account. This is achieved by each exam in the online learning process.

4) The effect of individual customer preferences on the electricity market, such as the dissatisfaction cost function.

Accordingly, the present invention has the effect of improving the reliability of the power system by increasing the profitability of the service provider in the power market, reducing the energy costs of the energy demand customers, and balancing the supply and demand of the power market.

1 shows the structure of reinforcement learning.
2 illustrates a configuration of a hierarchical electricity market according to the present invention.
Figure 3 shows the configuration of the electricity market model to which reinforcement learning (RL) is applied according to the present invention.
4 shows a flowchart implementing the Q-learning mechanism according to the present invention.
5 shows the customer energy demand profile of the simulation.
6 shows the wholesale price of electricity in the simulation.
7 shows the convergence of Q values of three customers.
8 shows the optimal retail price and energy consumption in each time slot.
9 shows the energy savings of three customers.
10 shows the effect of the weighting factor on the average retail price.
11 illustrates the effect of weighting factors on the average revenue of service providers and customers.
12 shows the learning rate for different numbers of customers.

The present invention contemplates the power system market as shown in FIG. 2 in which a service provider determines a dynamic pricing strategy based on a customer's energy demand profile and dissatisfaction level.

Similarly, the grid operator's wholesale electricity prices are taken into account, enabling more efficient energy use. In particular, the service provider determines the retail price sent to the customer at each time and bills the customer for electricity. Reinforcement learning is used to analyze how service providers learn and acquire dynamic retail policies while interacting with a variety of customers in order to maximize service providers' benefits and minimize customer costs. Employing reinforcement learning to solve dynamic pricing problems has three main advantages in providing the best solution to these problems.

Firstly, the reinforcement learning model is free. The customer does not need a prespecified model to choose the retail price. Instead, the relationship between retail price and profit is learned through dynamic interactions with customers.

Second, reinforcement learning is flexible. It can respond to dynamically changing environments through continuous learning and adaptation, taking into account the uncertainty and flexibility of the electricity market.

Third, reinforcement learning is concise. The algorithm's entire calculation process is based on the lookup table and the update mechanism shown in Table 1.

Detailed description of the invention is as follows. First, we introduce the power system market model and describe the mathematical formula of this model. Next, we detail the reinforcement learning methodology, including formulating dynamic pricing issues into the MDP and adopting Q-learning to solve these decision problems. Finally, the numerical simulation results are explained.

System model

 As shown in FIG. 2, a power system market model that includes a grid manager 10, a service provider 20, and a customer 30 is considered. The grid is installed, managed and maintained by the grid manager. Grid managers operate nationwide high voltage grids, while service providers transmit low voltage electricity. The service provider purchases electricity from the grid manager at the wholesale market price and sells the electricity to the customer at the retail price.

The present invention focuses on a DR algorithm between a service provider and a customer. Service providers decide what retail pricing policies to use to more efficiently use energy and maximize profits, and customers participate in the energy dynamic pricing DR program to balance energy demand and reduce energy costs. The service provider can flexibly determine the retail electricity price based on the customer's load demand profile and the cost of purchasing electricity from the grid manager.

A. Customer Model

Customer load profiles can be categorized as either critical or standby based on their priority and requirements.

Critical loads: Critically meeting customer load demands, such as data center power usage, is critical.

(One)

는 시간 슬롯 t를 나타내고, T는 하루의 최종 시간 슬롯 즉, T=24이고, 가격은 매시간 갱신되고,

고객은 n으로 나타낸다. 시간 슬롯 t에서의 고객 n의 에너지 수요를

및 에너지 소비를

으로 각각 나타낸다.here

Represents the time slot t, T is the last time slot of the day, T = 24, the price is updated hourly,

Customer is represented by n. The energy demand of customer n in time slot t

And energy consumption

Respectively.

를 소비하면, 고객 n의 부하 수요의 에너지량

이 충족되고 부하 수요

의 나머지는 만족되지 않는다. 이러한 감소된 에너지는 시간슬롯 t에서 고객 n의 불만을 야기하며, 이것은 불만족 비용 함수

로 표시된다. 이 모델은 고객이 에너지 수요를 줄일 때 경험할 수 있는 불쾌감의 정도를 모델링하고 볼록 함수로 정의되며 에너지를 크게 줄임으로써 극적으로 증가한다.Atmospheric loads: Customers' demand for electricity, such as heating, ventilation, and cooling (HVAC), usually decreases with rising electricity prices. Customer n has energy in time slot t

, The amount of energy in load demand of customer n

To meet the load demand

The rest of is not satisfied. This reduced energy causes customer n dissatisfaction in timeslot t, which is a function of dissatisfaction cost

Is displayed. This model models the degree of discomfort that a customer may experience when reducing energy demand, is defined as a convex function, and increases dramatically by significantly reducing energy.

The energy consumption that can be reduced by customer n in time slot t is defined as follows.

(2)

(3)

(4)

는 시간 슬롯 t에서의 탄력성 계수,

는 시간 슬롯 t에서의 고객 n에 대한 전력 소매 가격을 나타내고, 시간 슬롯 t에서의

는 도매 전력 가격을 나타낸다.here,

Is the elasticity coefficient in time slot t,

Represents the power retail price for customer n in time slot t,

Represents wholesale power prices.

이란 한 경제 변수가 다른 경제 변수의 변화에 어떻게 반응하는지를 나타내는 척도이다. 특정 상황에서 수요의 가격 탄력성은 재화의 가격 변화에 대한 재화나 용역의 수요량의 반응성 또는 탄력성을 나타내는 척도이다. 보다 정확하게 스마트 그리드에서 이 매개 변수는 1% 가격 변동에 대한 에너지 수요 변화를 나타낸다. 탄력성은 일반적으로 부정적 의미이며, 이는 전력 수요와 전기 가격 사이의 역 관계를 나타낸다. 몇 가지 연구는 스마트 그리드에서 수요의 가격 탄력성을 조사했다. 마시모(Massimo)는 하루 중 시간대와 계획 기간에 따른 가격 탄력성의 변화를 조사한 결과 전력 수요가 피크 시간대에는 비 피크 시간대에 비해 더 탄력적이며 장기간 탄력성은 일반적으로 단기 탄력성보다 크다고 결론 지었다. Miller와 Alberini는 미국의 전국 주택 조사, 양식 ELA-861 및 주거 에너지 소비 조사와 같은 3개의 전국 데이터 세트 실험을 통해 -0.2에서 -0.7의 가격 탄력성을 발견했다. 이 연구에서 에너지 관리를 위한 실시간 결정을 내릴 때 강화 학습을 채택할 타당성을 탐구하는데 초점을 맞췄다. 따라서 탄력성 값은 기존 연구에서 직접 얻어진다.Resilience in Economics

Is a measure of how one economic variable responds to changes in another. In certain circumstances, the price elasticity of demand is a measure of the responsiveness or elasticity of the quantity demanded of a good or service to changes in the price of the good. More precisely in a smart grid, this parameter represents a change in energy demand for a 1% price change. Resilience is generally negative, indicating the inverse relationship between power demand and electricity prices. Some studies have examined the price elasticity of demand in smart grids. Massimo examined changes in price elasticity over time and planning periods throughout the day and concluded that power demand is more elastic at peak times than non-peak times, and long-term elasticity is generally greater than short-term elasticity. Miller and Alberini found price elasticities of -0.2 to -0.7 in three national dataset experiments, the US National Housing Survey, Form ELA-861, and Residential Energy Consumption Survey. This study focused on exploring the feasibility of adopting reinforcement learning when making real-time decisions for energy management. Thus, elasticity values are obtained directly from existing studies.

The dissatisfaction cost function of customer n at timeslot t is defined as

(5)

(6)

(7)

(8)

와

는 고객에 따라 달라지는 파라미터이며,

는 다른 고객들 사이에서 변하는 고객 선호도 값이며,

는 미리 결정된 상수이다.

은 전기 수요 감소에 대한 고객의 태도를 반영 한다.

이 더 크다는 것은 고객이 만족도를 높이기 위해 수요 감축을 적게하는 것을 선호한다는 것을 나타내고 더 적다는 것은 반대의 의미이다.

과

는 소매 가격이 실효값일 때 수요 감축의 범위를 나타낸다.In (5),

Wow

Is a customer-specific parameter.

Is a customer preference value that changes among other customers.

Is a predetermined constant.

Reflects the customer's attitude toward reduced electricity demand.

This greater indicates that the customer prefers less demand reduction to increase satisfaction and less means the opposite.

and

Represents the extent of demand reduction when the retail price is the effective value.

The goal of customer n is to minimize costs as described below.

(9)

B. Service Provider Model

Suppose a service provider participates in a wholesale power market organized by a grid manager. In each time slot, the service provider purchases energy from the grid manager at the wholesale price determined by the grid manager and then sells energy to the customer at its own determined retail price. Therefore, the service provider's goal is to perform dynamic retail pricing that maximizes profits as follows:

10

(11)

은

보다 크지만, 범위 내에 있어야 한다. 식(11)에서

와

는 소매 가격 경계의 미리 결정된 계수이다. 이 속성은 가격을 공정하게 유지하고 이익을 보호하기 위해 서비스 제공자와 고객 간의 규제 요구조건이나 상호 합의를 반영할 수 있다.Generally

silver

Greater than, but within range. In equation (11)

Wow

Is a predetermined coefficient of the retail price boundary. This attribute may reflect regulatory requirements or mutual agreement between the service provider and the customer in order to keep prices fair and protect interests.

C. Objective Function

The present invention considers the revenue of the service provider and the cost of the customer as follows.

(12)

(13)

는 서비스 제공자의 이익과 고객 비용 간의 상대적 중요성을 나타내는 가중치 계수이다.

의 값은 서비스 제공자의 정책에 따라 결정되어야 한다.

의 영향은 후술하기로 한다. here

Is a weighting factor that indicates the relative importance between the service provider's benefits and customer costs.

The value of shall be determined according to the service provider's policy.

The influence of will be described later.

Reinforcement Learning Methodology

Reinforcement learning (RL) is suitable for describing the hierarchical decision framework described above, as shown in FIG. The service provider 20 acts as an agent, the customer 30 is an environment, the retail price represents the action the service provider sends to the customer in each time slot, and the customer's energy information (energy demand). And consumption) represent a state, and the revenue and customer costs of a service provider represent a reward. First, the dynamic retail pricing problem is formulated into the discrete finite horizon Markov decision process (MDP). We then adopt Q-learning to propose an efficient dynamic pricing algorithm that does not require sufficient knowledge of system dynamics and uncertainty.

A. Formulation of System Model Using Markov Decision Process

, 상태 S

및 보상

이 포함된다.Dynamic retail pricing is modeled as discrete finite-interval MDP because it is a decision problem in probabilistic environments. In this MDP model, compensation and energy consumption depend only on the energy demand and retail price of the time slot, not on historical data. Key components modeled in MDP include discrete time t, behavior

, State S

And reward

This includes.

1) t is a finite discrete time slot in which retail price measures are enforced.

는 고객 n에 대해 서비스 제공자가 시간 슬롯 t에서 선택하는 소매가격이다.2)

Is the retail price that the service provider selects in time slot t for customer n.

은 서비스 제공자로부터 소매가격 신호를 받기 전에 고객의 에너지 수요를 나타낸다.

은 서비스 제공자로부터 소매가격 신호를 받은 후 고객의 실제 에너지 소비량을 나타낸다.3)

Represents the energy demand of the customer before receiving the retail price signal from the service provider.

Represents the actual energy consumption of the customer after receiving a retail price signal from the service provider.

은 상술한 서비스 제공자의 수익 및 고객 비용으로, 주어진 상태에서 소매 가격 책정을 실행하여 얻게 되는 예상되는 즉각적 보상을 나타낸다. MDP의 에피소드(시작부터 종료까지의 기록)는 다음과 같이, 시간 슬롯, 상태, 동작(action) 및 보상의 유한한 순서를 형성한다4)

Denotes the expected immediate rewards of implementing retail pricing in a given state, with the revenue and customer costs of the service provider described above. Episodes of MDP (recording from start to end) form a finite sequence of time slots, states, actions, and rewards, as follows:

Once the MDP is implemented, the total compensation for each batch can be easily calculated.

(14)

(15)

Then the total future reward from time slot t is expressed as

(16)

But the environment is stochastic. Therefore, you cannot be sure that you will receive the same reward the next time you take the same action. The more we branch into the future, the more we branch. Therefore, it is common to use discounted future rewards instead.

(17)

는 현재 시스템 보상과 비교하여 미래 시스템 보상의 상대적 중요성을 나타내는 할인 요소이다. 특히

가 0 일 때 시스템은 근시안적이고 현재 보상에만 의존한다. 환경이 결정론적이고 동일한 조치가 항상 동일한 보상으로 나타나는 경우,

는 1로 설정할 수 있다. 현재 보상 및 향후 보상의 균형을 유지하려면

값을 실제 소수(예: 0.9)로 설정해야 한다. 시간 슬롯 t에서의 할인된 미래 보상이 시간 슬롯 t+1에서 동일한 표현으로 표현될 수 있음을 쉽게 알 수 있다.here

Is a discount factor that indicates the relative importance of future system rewards compared to current system rewards. Especially

Is 0, the system is shortsighted and depends only on the current compensation. If the environment is deterministic and the same action always results in the same reward,

Can be set to 1. To balance current and future rewards

You must set the value to the actual decimal number (eg 0.9). It can be readily seen that the discounted future reward in time slot t can be represented by the same representation in time slot t + 1.

(18)

Mark υ as the policy that maps state to behavior. The goal of the dynamic pricing problem is to find the optimal policy v that always selects an action (retail price) to maximize the expected discount reward.

B. Adopt Q-learning for dynamic pricing issues

Reinforcement learning is an approach to making decisions sequentially in an unknown environment. Policies can be changed in real time based on online learning of past experiences.

을 할당하고 반복에서 이를 업데이트하여 양호한 행동을 강화하는 것이다. 최적의 Q값

은 시작 상태

에서 조치

를 취하고 최적 정책을 계속해서 따를 때 최대 할인 미래 보상을 나타내며, 이것은 식 (18)을 기반으로 Bellman 식을 충족시킨다.Q-learning, a type of modelless reinforcement learning technique, is used to obtain an optimal policy (order of retail prices in the present invention). The basic principle of Q-learning is the Q value for each state behavior pair in time slot t.

Is assigned and updated in the iteration to reinforce good behavior. Optimal Q value

Is the starting state

Action from

The maximum discounted future reward is obtained by taking, and continuing to follow the optimal policy, which satisfies the Bellman equation based on equation (18).

(19)

As shown in Table 1, updating the mechanism for obtaining the Q value using the Bellman equation is abbreviated. When updating the mechanism, θ is a learning rate indicating how far the Q value can replace the previous Q value. 0 means the service provider learns nothing, while 1 means the service provider only considers the latest information.

Table 1. Q-LEARNING TECHNIQUE

가 상태

에서 행동

으로 최대 기대되는 시스템 이익이 되므로, 다음과 같은 최적의 정책을 얻을 수 있다.In the algorithm of Q-learning, a service provider performs a series of tasks to interact with the environment (customer). The environment then changes and the service provider receives new status and reward signals. In this process, learning takes place through trial and error. The Q value is stored and updated during the learning process. After updating through a sufficient number of iterations, the Q value converges to the maximum value.

State

Act on

This is the maximum expected system benefit, so the following optimal policy can be obtained.

20

The best retail price is then determined.

4 shows how the Q-learning algorithm shown in Table 1 is implemented to obtain the maximum Q-value (optimal retail price). Here the input and output of the algorithm are specified.

The Q-learning algorithm is operated at the beginning of the day, as shown in FIG. Inputs to the algorithm include the customer's energy demand over time slot T, wholesale power price, coefficient of retail price range, and other related parameters. Upon receiving this parameter, the service provider initializes the Q value to 0 and the time slot t and the repetition value i to 1. The service provider will then calculate the optimal retail price repeatedly.

That is, in each iteration i, the service provider observes the customer's energy demand information at each time slot and then selects the retail price using the ε-greedy policy within the retail price boundary. Reinforcement learning requires a clever exploration mechanism. Selecting a random behavior generally degrades performance without referring to the estimated probability distribution. The most common way is to use the ε-greedy policy, which selects behaviors that are distributed evenly within the set of available behaviors. This policy can be used to select either the behavior of probability ε (ε is a fraction between 0 and 1) or the behavior of probability 1-ε from the Q value of a given state in each iteration. Where the random selection indicates that the service provider randomly selects the retail price within the price range of a given state, and the selection from the Q value indicates that the service provider searches the stored Q value in a given state to find the maximum Q value, and then the corresponding retail It means choosing a price. Note that the maximum Q value is not fixed and can be replaced by subsequent iterations. This gives the system randomness, but prevents complete randomness and facilitates exploration of the behavior space. After choosing the retail price, the service provider can get immediate compensation by equation (12), observe the customer's energy demand in time slot t + 1, and update the Q value using the Q-learning mechanism in Table 1 Then repeat the process until the last time slot T is reached. The service provider then compares the current Q value with the previous Q value to see if it has converged to the maximum Q value, otherwise it moves the system to the next iteration i + 1 and repeats this process.

. 현재 Q값과 이전 Q값 사이의 차이가

보다 작으면 Q 값은 최대값으로 수렴한다

의 값은 시스템 설계에 달려 있다. 마지막으로, 서비스 제공자는 하루의 t=1부터 t-=T 까지 시간 슬롯에 대한 최적의 소매 가격을 얻고 등록된 고객에게 이 가격을 알린다.The iteration termination condition is as follows.

. The difference between the current Q value and the previous Q value

If less, the Q value converges to the maximum value.

The value of depends on the system design. Finally, the service provider obtains the optimal retail price for the time slot from t = 1 to t- = T of the day and informs the registered customer of this price.

Numerical Simulation Results

Numerical simulation results for evaluating the performance of the dynamic pricing DR program according to the present invention are presented. For simplicity, the simulation is based on the service provider and three customers. The total time period is represented by 24 time zones representing 24 hours a day. Thus the value of T defined above is 24, which represents the last time slot of the day. An example load load profile of three customers in each time slot was obtained on June 22, 2017 from SDG & E, as shown in FIG. 5, and was used as input to the flowchart shown in FIG. 4. Table 2 shows the dissatisfaction related parameters of these three customers. Elasticity values are shown in Table 3 and the response was divided into off-peak / mid-peak / on-peak.

Table 2 CUSTOMERS 'DISSATISFACTION RELATED PARAMETERS

Table 3 ELASTICITY

과

는 서비스 제공자의 이익과 고객의 비용을 모두 고려한 수용 가능한 값인 1.5로 설정되고 [2.4, 8.2]의 소매 가격 범위를 제공한다. 본 발명에서 가중치

는 0.9로 서비스 제공자의 이익은 고객의 비용보다 상대적으로 중요함을 나타낸다. 0에서 1까지 변화하는

의 영향은 후술한다. 이 파라미터들은 또한 도 4에 도시된 순서도(흐름도)의 입력으로 사용된다.Taking the electricity wholesale price shown in FIG. 6, the online data provided by ComEd was used on June 22, 2017. The retail electricity price range was formulated using specific factors of wholesale electricity prices. In this simulation

and

Is set to 1.5, an acceptable value that takes into account both the service provider's benefit and the customer's cost, and provides a retail price range of [2.4, 8.2]. Weights in the present invention

Is 0.9, which indicates that the service provider's profit is more important than the customer's cost. Varying from 0 to 1

The influence of will be described later. These parameters are also used as input to the flowchart (flow chart) shown in FIG.

All parameter values in the simulation scenario are specific and can vary depending on the design of the electrical market or the characteristics of the service provider and customer. However, this does not distort the analysis of the results of the simulation.

Based on the scenario defined above, the simulation is run through iterations where the optimal retail price is calculated. For example, at the beginning of the day, the service provider calculates the Q value (retail price) using the learning algorithm of Figure 4 from the grid manager's wholesale electricity price, the customer's electrical load demand, and other parameters defined in the scenario. The maximum Q value (optimum retail price) is finally obtained during the next 24 hours of the day. 7 shows the convergence of Q values for the scenario on June 22, 2017. At the beginning of the day, the service provider does not know how to choose an action to obtain a high Q value. However, the Q value is repeatedly trial and error, increasing every time the service provider learns in the environment, and finally converges to the maximum value. The calculation time and convergence iteration will be described later.

Once optimal electricity prices are obtained, the optimal energy consumption of each customer is determined by equations (1) and (2). Next, the performance of the dynamic pricing algorithm is detailed in various aspects.

A. Best Retail Price

After running the simulation, the main output is the optimal retail electricity price for each Gauguin. 8 shows the optimal retail price, the signal of the wholesale price and the standby load energy demand and energy consumption for three customers. Since critical load demand does not change with retail prices, only the available load energy information is displayed.

It can be seen from FIG. 8 that the retail price trend is similar to the wholesale price reflecting the energy purchase cost of the grid manager. However, it does not exceed the price range. The retail price for all customers increases from slot 6 to slot 12 to gain more benefit to the service provider, but a sudden decrease is observed in slot 13. This occurs to reflect the peak time with modulus of elasticity from -0.3 to -0.5 in slot 13, which significantly reduces energy during this period as retail prices continue to rise. However, this value should not exceed the reduced energy range (8), so the retail price will decrease during time zone 13. The price difference (retail price-wholesale price) between the peak time period (from 17:00 to 21:00) and the non-peak time (from 7 to 12) is the difference between energy reduction (energy demand-energy) Consumption) is greater. Based on Eq, this is because the electricity demand is more elastic during peak times.

The average retail price of customer 3 is higher than the average retail price of the other two customers, which increases the average retail price because customer 3 has less dissatisfaction and reduces energy demand.

B. Total Energy Reduction

=0.8)은 다른 두 고객과 비교하여 훨씬 적은 에너지 절감을 선택한다. 이 현상은

의 물리적 의미와 일치한다. 즉,

가 큰 고객은 불만을 적게하기 위해 더 작은 에너지 감소를 선호하지만, 반면

이 더 작은 고객은 DR동안 더 큰 에너지 감소를 선택한다. 이러한 상황에서 DR은 전력 시장에서 에너지 공급과 수요의 균형을 맞출 수 있는 기회를 제공하여 시스템 과부하를 효과적으로 제거하고 전력 시스템의 신뢰성을 향상시킬 수 있다.9 shows the total energy reduction of each customer participating in the proposed dynamic pricing algorithm. Where the yellow bar represents energy demand and the green bar represents energy consumption. Energy consumption is reduced by 43.820, 51.733 and 58.947 kWh at customers 1, 2 and 3 respectively. Customer 1 (

= 0.8) chooses much less energy savings compared to the other two customers. This phenomenon

Is consistent with its physical meaning. In other words,

While larger customers prefer smaller energy reductions to make less complaints,

This smaller customer chooses a larger energy reduction during DR. In this situation, DR can provide an opportunity to balance energy supply and demand in the power market, effectively eliminating system overload and improving power system reliability.

C. Influence of weighting factor

의 변수 값을 0에서 1로 변경하여 시뮬레이션을 수행했다. 그림 10.과 11은 서비스 제공자 및 고객의 평균 소매가격과 평균 수익을 개별적으로 보여준다.

가 0에서 1로 증가하면 평균 소매 가격과 서비스 제공자의 평균 이익이 증가한다. 그러나 고객의 평균 이익은 감소한다. 그 이유는 분명하다:

이 증가함에 따라 서비스 제공자의 자체 이익을 극대화하고 고객의 비용을 고려하지 않으며 상대적으로 높은 소매가격을 선택한다. 대조적으로,

=0일 때, 시스템은 고객의 비용을 최소화 하는 경향이 있다. 따라서 서비스 제공자는 상대적으로 낮은 소매 가격을 선택한다.To investigate the effect of weighting factors

The simulation was performed by changing the value of the variable from 0 to 1. Figures 10 and 11 show the average retail price and average revenue for service providers and customers separately.

Increases from 0 to 1, the average retail price and average service provider's profits increase. But average profits for customers decline. The reason is clear:

This increase maximizes the service provider's own interests, does not take into account customer costs, and chooses a relatively high retail price. In contrast,

When = 0, the system tends to minimize customer costs. The service provider therefore chooses a relatively low retail price.

D. System Performance

에 거의 최적 값으로 수렴된다. 이렇게 긴 계산 시간을 갖는 이유는 각각의 고객에 대해 각각 최대 59개의 작업이 있는 총 24개의 시간 슬롯을 고려할 때, 행동 횟수의 값은

의 식으로 계산된다. 본 발명에서 소매 가격의 최소 간격은 도매 전기 가격과 동일한 최소 간격을 유지하기 위해 0.1로 설정되어 (8.2- 2.4) /0.1 + 1의 값을 제공한다. 따라서 오직 하나의 고객의 경우

-greedy 정책을 사용하면 M이 [1,….59]에 속하는

순열이 있다. 그러나 클라우드 컴퓨팅과 같은 최신 첨단 기술은 이것이 중요한 문제가 되지 않을 것임을 의미한다.To evaluate system performance, the number of customers increased from three to ten, and the simulation was run 10 ⁵ times. This simulation was performed using software with source code programmed in Java in Eclipse tools and a 3.30Ghz, 4-core i5-6600 CPU 8GB RAM Window PC hardware. Figure 12 shows the learning rate for different numbers of customers, and Table 4 lists the corresponding computation time and the number of convergence iterations. As the number of customers increases, the computation time and convergence iterations increase. Specifically, if you set the number of customers to 10, the simulation would take 98 minutes for 10 ⁵ iterations

Converges to an almost optimal value. The reason for this long computation time is to consider the total of 24 time slots with up to 59 tasks each for each customer.

Calculated by In the present invention, the minimum interval of retail prices is set to 0.1 to provide a value of (8.2-2.4) /0.1 + 1 to maintain the minimum interval equal to the wholesale electricity price. So for one customer only

-greedy policy causes M to be [1,…]. .59]

There is a permutation. But modern advanced technologies like cloud computing mean that this won't be an issue.

Table 4. COMPUTATION TIME AND NUMBER OF CONVERGENCE ITERATIONS

The present invention studies dynamic pricing algorithms between service providers and customers in the power system market, allowing service providers to flexibly determine electricity retail prices using reinforcement learning methodologies based on customer load demand profiles and dissatisfaction levels and wholesale power. have.

We first formulate a dynamic pricing problem based on discrete finite MDP and then use Q-learning to solve the decision problem. Through the use of reinforcement learning methodologies, service providers do not require a customer's pre-specified model for which retail prices should be chosen. Instead, the relationship between status, behavior and rewards is learned through dynamic online interactions with customers. Continuous learning and adaptation, taking into account the uncertainty of the customer's load demand profile and the flexibility of wholesale electricity prices, can respond to a dynamically changing environment. Numerical simulation results show that the proposed dynamic pricing algorithm can increase service provider profitability, reduce customer energy costs, balance energy supply and demand in the power market, and improve the reliability of the power system. It can be seen that this is a win-win strategy for both providers and customers.

The above description is merely illustrative of the present invention, and various modifications may be made by those skilled in the art without departing from the technical spirit of the present invention.

Therefore, the embodiments disclosed in the specification of the present invention are not intended to limit the present invention. The scope of the present invention should be construed by the claims below, and all techniques within the scope equivalent thereto will be construed as being included in the scope of the present invention.

10: grid manager 20: service provider
30: Customer

Claims (14)

In the dynamic pricing demand response method by the service provider in the smart grid,
A first step of arbitrarily selecting the electric retail price in the electric retail price range based on the electric wholesale price obtained from the grid manager;
A second step of obtaining compensation and energy information of the customer based on the service provider's revenue and customer cost,
A third step of updating a Q value of a Q-learning algorithm based on the retail price, reward and energy information,
Dynamic pricing demand response method characterized in that for repeating the first step to the third step for each time slot from the beginning to the end of the time slot. The method of claim 1,
A fourth step of determining whether the Q value reaches a maximum value after the repetition of the first to third steps is completed;
If the Q value does not reach the maximum value, repeating the first step to the third step for each time slot from the beginning to the end of the time slot,
Dynamic pricing demand response method characterized in that for adopting a policy corresponding to the maximum Q value as the optimum price policy for determining the retail price when the Q value reaches the maximum value. The method of claim 1,
The energy information of the customer is a dynamic pricing demand response method characterized in that it comprises the energy demand of the customer and the actual energy consumption. The method of claim 3,
Dynamic customer demand consumption method is characterized in that calculated from the energy demand of the customer based on the difference between the electricity retail price and the wholesale price of electricity. The method of claim 1,
Revenue of the service provider is a dynamic pricing demand response method characterized in that the difference between the retail price and electricity wholesale price multiplied by the actual energy consumption of the customer. The method of claim 1,
The customer cost is a dynamic pricing demand response method, characterized in that the retail price multiplied by the actual energy consumption sum of the energy cost and the customer dissatisfaction cost. The method of claim 1,
The reward is a dynamic pricing, characterized in that the difference between the value of the weight of the service provider's revenue and the relative importance of the customer's cost multiplied by the service provider's profit and (1-weighted) the customer's cost multiplied. Demand response method. The method of claim 1,
And determining that the current iteration Q value has reached a maximum value when the difference between the Q value of the previous iteration and the Q value of the current iteration is less than or equal to a predetermined value. In the dynamic pricing demand response method by the service provider in the smart grid,
The service provider repeatedly performs a process of determining the electric retail price for the subscribing customer and acquiring compensation amount based on the energy consumption of the subscribing customer and the service provider's profit and the customer's cost according to the determined electric retail price,
Dynamic pricing demand response method characterized in that for selecting the optimal pricing policy pricing policy when the cumulative compensation amount obtained by repeatedly performing the above process. The method of claim 9,
Wherein said service provider selects an electric retail price in an electric retail price range based on an electric wholesale price of a grid manager. The method of claim 10,
Dynamic pricing demand response method characterized in that the selection of the ε-greedy method. The method of claim 9,
The compensation amount is a dynamic price, characterized in that the difference between the value multiplied by the service provider's revenue multiplied by the customer's cost multiplied by the weight (ρ) representing the relative importance between the service provider's revenue and customer's cost Demand response method. A computer readable medium storing a program for executing a dynamic pricing demand response method in a smart grid,
The program includes the first step of inputting the wholesale price of electricity of the grid manager and the energy demand of subscribed customers;
A second step of initializing the Q value of the Q-learning algorithm to 0, the time slot t and the iteration i to 1,
A third step of arbitrarily selecting an electric retail price in the electric retail price range based on the electric wholesale price in an iteration i;
A fourth step of calculating compensation amount and energy information of the customer based on the service provider's revenue and customer's cost,
A fifth step of updating a Q value of a Q-learning algorithm based on the retail price, compensation amount and energy information;
Determining whether a time slot t is a last ending time T;
A seventh step of repeating the third to sixth steps if the time slot is not the end time;
An eighth step of determining whether the Q value has reached the maximum value if the time slot is the end time;
A ninth step of repeating steps 3 to 7 if the Q value has not reached the maximum value,
And if the Q value has reached the maximum value, outputting the retail price with a pricing policy corresponding to the maximum Q value. The method of claim 13,
And the eighth step determines whether the difference between the Q value of the repetition i and the Q value of the previous repetition i-1 is less than or equal to a predetermined value.

Images