KR102480521B1 - HEMS optimization method and device using reinforcement learning - Google Patents

Abstract

The present invention discloses an HEMS optimization apparatus and method using reinforcement learning. According to the present invention, it includes a processor and a memory connected to the processor, wherein the memory, based on a pre-learned artificial neural network, measures the indoor temperature at time t-1, the upper and lower limits of the user's preferred temperature and the time t. The current temperature is predicted using the predicted external temperature and the energy consumption of the air conditioner, the predicted current temperature is input to the Q-learning module for the air conditioner agent, and the Q-learning module acts according to the predicted current temperature. In case of outputting , a pair of states and actions according to the predicted current temperature are stored and updated in a Q-value table, and agents corresponding to the air conditioner agent, a device with an unblockable load, and a device with a load that can be blocked An HEMS optimization apparatus using reinforcement learning that stores program instructions executable by the processor to search for an optimal policy maximizing the sum of rewards according to each action is provided.

Description

HEMS optimization method and device using reinforcement learning {HEMS optimization method and device using reinforcement learning}

The present invention relates to a HEMS optimization method and apparatus using reinforcement learning.

With residential households accounting for one-third of total electricity consumption, Home Energy Management Systems (HEMS) have become an essential technology for energy management.

The main goal of HEMS is to reduce electricity costs while ensuring comfort by scheduling optimal energy consumption for smart appliances (such as air conditioners and washing machines, for example).

More recently, distributed energy resources (DERs), e.g., rooftop solar photovoltaic (PV) and energy storage systems (ESS), advanced metering infrastructure with smart meters, and demand management. Smart grid technologies, including

The core technology of HEMS is an optimization method used to perform economical load reduction and load shifting of smart home appliances in addition to operation scheduling (charge/discharge) of DER.

However, the conventional HEMS optimization algorithm is made based on a model, but there is a problem of not providing a somewhat appropriate solution in an environment including recently increasing smart home appliances, and in particular, there is a problem that optimal energy scheduling of air conditioners is not achieved. .

Korean Patent Publication No. 10-2015-0040894

In order to solve the problems of the prior art, the present invention proposes an HEMS optimization method and apparatus using reinforcement learning that can provide an optimized solution in consideration of a rooftop PV system, ESS, and smart home appliances.

In order to achieve the above object, according to an embodiment of the present invention, an HEMS optimization apparatus using reinforcement learning includes a processor; and a memory connected to the processor, wherein the memory includes an indoor temperature at time t-1, an upper/lower limit of a user's preferred temperature, an external temperature predicted at time t, and an air conditioner based on a pre-learned artificial neural network. If the current temperature is predicted using the energy consumption of , the predicted current temperature is input to the Q-learning module for the air conditioner agent, and the Q-learning module outputs an action according to the predicted current temperature, the prediction A pair of states and actions according to the current temperature are stored and updated in a Q-value table, and compensation according to each action of the agents corresponding to the air conditioner agent, a device with a load that cannot be blocked, and a device with a load that can be blocked An HEMS optimization apparatus using reinforcement learning that stores program instructions executable by the processor to search for an optimal policy that maximizes the sum of .

The device having the non-breakable load may include a washing machine, and the device having the interruptable load may include an energy storage system.

The agent of the washing machine may perform one binary action of on/off, and the agent of the air conditioner and energy storage system may perform one action of a plurality of levels having a difference in unit energy consumption.

The artificial neural network may learn how much energy consumption of the air conditioner affects the current indoor temperature.

The artificial neural network may be composed of one input data layer having a plurality of neurons, a plurality of hidden layers having a plurality of neurons, and an output layer having one neuron.

According to another aspect of the present invention, as a method for optimizing a reinforcement learning-based home energy management system in a device including a processor and a memory, based on a pre-learned artificial neural network, the indoor temperature at time t-1 and the user's preferred convenience temperature predicting a current temperature using upper/lower limits, external temperature predicted at time t, and energy consumption of the air conditioner; inputting the predicted current temperature to a Q-learning module for an air conditioner agent; storing and updating a pair of states and actions according to the predicted current temperature in a Q-value table when the Q-learning module outputs an action according to the predicted current temperature; and searching for an optimal policy maximizing the sum of rewards according to the actions of the air conditioner agent, a device having an unblockable load, and a device corresponding to a device having a blockable load. (HEMS) optimization method is provided.

According to another aspect of the present invention, a computer readable program for performing the method described above is provided.

According to the present invention, since the current temperature predicted through the artificial neural network is input to the Q-learning module to determine the optimal policy, there is an advantage in that the scheduling efficiency of the smart home appliance can be further increased.

1 is a diagram illustrating a reinforcement learning-based HEMS framework according to a preferred embodiment of the present invention.
2 is a diagram for explaining a reinforcement learning process.
3 is a diagram illustrating the configuration of an artificial neural network model for predicting indoor temperature according to the present embodiment.
4 is a diagram illustrating an HEMS optimization algorithm according to this embodiment.
5 is a diagram illustrating a Q-learning and ANN-based framework for optimally controlling a home appliance and an ESS.
6 is a diagram showing the configuration of an apparatus for HEMS optimization based on reinforcement learning according to a preferred embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

In the present invention, a situation in which automatic energy management is performed by HEMS is considered, and it is assumed that HEMS schedules and controls the following (home appliance) devices according to time-to-use (TOU) rates.

): 제어 가능 기기는 HEMS에 의해 동작이 스케줄링되고 제어되는 기기이다. Controllable devices (

): A controllable device is a device whose operation is scheduled and controlled by HEMS.

)와 시프트 가능 기기(

)로 분류된다. Devices whose operating characteristics can be reduced (

) and shift-capable devices (

) is classified as

For example, devices that can be reduced are devices that can reduce energy consumption to reduce electricity costs, such as air conditioners.

However, according to the TOU pricing scheme, the energy consumption of shift-capable devices can be shifted to different time zones to minimize the total cost of electricity.

Shift-capable devices are divided into two load types.

) 및 차단 가능 부하(interruptible load,

)이다. First, non-interruptible load,

) and interruptible load,

)to be.

During the machine's task period, the operation of the shiftable machine with non-blockable load must not be interrupted by the HEMS control, and the washing machine must complete washing to the drying gun.

A shift-capable machine with a breakable load may cease operation at any time.

For example, when the PV generated output is greater than the load demand, the HEMS must stop the discharging process and immediately start charging the ESS.

)는 TV, PC 또는 조명과 같이 HEMS에 의해 스케줄링되지 않는 기기이다. 따라서,

는 고정된 에너지 소비 스케줄링을 유지한다. On the other hand, uncontrollable devices (

) is a device that is not scheduled by HEMS, such as a TV, PC, or lighting. thus,

maintains fixed energy consumption scheduling.

Hereinafter, the conventional HEMS optimization algorithm will be first described, and the reinforcement learning-based HEMS optimization algorithm according to this embodiment will be described in detail.

를 포함한다. As shown in Equation 1, the objective function for the HEMS optimization problem consists of two parts, each with a different decision variable.

includes

는 TOU 가격

및 시간 t에서 순 에너지 소비량에 따른 전체 에너지 비용이다. here,

is the TOU price

and total energy cost as a function of net energy consumption at time t.

은 제어 가능 기기/제어 불가능 기기의 에너지 소비 및 예측된 PV 생성 출력에 관하여 기재된다. Also,

is described in terms of energy consumption and predicted PV generated output of controllable/uncontrollable devices.

는 소비자 불편 비용과 관련된 총 패널티 상수이다.

is the total penalty constant associated with consumer inconvenience costs.

로부터 원하는 소비자 온도

와의 차이를 의미한다. Discomfort at room temperature

Desired consumer temperature from

means the difference between

는 소비자 불편 비용에 대한 패널티 기간이다.

is the penalty period for consumer inconvenience costs.

는 소비자의 전기 요금을 희생하여 소비자의 선호 편의 수준을 만족시키기 위해 HEMS 운영자에 의해 결정될 수 있다

may be determined by the HEMS operator to satisfy the consumer's preferred convenience level at the expense of the consumer's electricity bill.

In the following, equations and inequality constraints for the HEMS optimization problem are described.

와 예측된 PV 생성 출력

의 차이이다. Equation 2 is a constraint on the net energy consumption, i.e. the total consumption of all devices

and the predicted PV generated output

is the difference between

In Equation 3, the total consumption of all devices can be divided into four different types of devices, including reduceable devices, shiftable devices with non-blockable loads, shiftable devices with breakable loads, and non-controllable devices. .

에서의, 실내 온도(

), 외부 온도(

), 기기의 에너지 소비량 및 내부 열적 조건을 나타내는 환경 파라미터

로 표현되는 시간 t

에서의 온도 역학에 대한 제약을 나타낸다. For devices that can be reduced (eg air conditioners), Equation 4 is

at room temperature (

), external temperature (

), environmental parameters representing the energy consumption and internal thermal conditions of the device

time expressed as t

represents a constraint on the temperature dynamics at

및

로 제한되는 것을 나타낸다. Equation 5 represents the upper and lower limits of the indoor temperature range desired by the consumer, and Equation 6 shows the energy consumption capacity of the device that can be reduced.

and

indicates that it is limited to

를 갖는 세탁기와 같이 차단 불가능 부하를 갖는 시프트 가능 기기의 선호 동작을 보장한다.Equations 7 to 9 are binary decision variables

Ensures preferred operation of shiftable appliances with non-breakable loads, such as washing machines with

와

는 소비자가 원하는 시작 시간 및 종료시간인 정지 기간 동안, (ii) 수학식 8에서 하루 중 동작 기간

시간 동안, (iii) 수학식 9에서

시간의 연속 동작 기간 동안 (i) in Equation 7

Wow

During the stop period, which is the start time and end time desired by the consumer, (ii) the operation period of the day in Equation 8

For time, (iii) in Equation 9

over a period of continuous operation of time

이다. In Equation 10, the energy consumption capacity of a shiftable device with a non-blockable load is

to be.

에 대한 에너지 상태(SOE)의 동역학을 이전 시간(t-1)에서의 SOE로 나타낸 것이고, 충전 및 방전 효율은

,

, 충방전 에너지는

및

이다. Equation 11 is the ESS at the current time (t)

The dynamics of the energy state (SOE) for is represented by the SOE at the previous time (t-1), and the charging and discharging efficiency is

,

, the charge and discharge energy is

and

to be.

및

를 갖는 SOE 용량 제약을 나타낸다. Equation 12 is

and

Represents the SOE capacity constraint with .

는 ESS 온/오프를 결정하는 이진 변수이다. Equations 13 and 14 represent constraints on the charging and discharging energy of the ESS, respectively, where

Is a binary variable that determines ESS on/off.

의 선형화를 통해 MILP 최적화 문제로 변환될 수 있다. Finally, the mixed integer nonlinear programming (MINLP) based HEMS optimization problem has the following nonlinear objective function

It can be converted to a MILP optimization problem through linearization of

Hereinafter, the HEMS optimization algorithm based on reinforcement learning and artificial intelligence according to the present embodiment will be described in detail.

1 is a diagram illustrating a reinforcement learning-based HEMS framework according to a preferred embodiment of the present invention.

As shown in FIG. 1, the reinforcement learning-based HEMS (RL-Based HEMS) according to this embodiment is based on the power supply company's TOU rate policy, weather information (eg, outside temperature) of the Korea Meteorological Administration, and consumer's convenience level and Considering PV, Q-learning among reinforcement learning techniques is used to schedule the operation of the air conditioner (AC), washing machine (WM), and ESS.

In addition, in scheduling the operation of the AC, an artificial neural network (ANN) is used to determine the optimum temperature, which will be described in detail again.

Prior to the description of the reinforcement learning-based HEMS optimization algorithm, reinforcement learning will be explained in detail.

Reinforcement learning is one of the machine learning techniques for optimal decision-making in a non-deterministic environment.

2 is a diagram for explaining a reinforcement learning process.

Referring to FIG. 2, while the agent interacts with the environment, the agent learns an action type dependent on the state of the environment and sends the learned action to the environment.

The environment then returns the reward to the agent along with the new environment state.

This learning process continues until the agent maximizes the total cumulative reward received from the environment.

A policy is defined by how an agent behaves in a particular state, and the agent's main goal is to determine the optimal policy that maximizes the reward.

In this embodiment, it is assumed that the environment is described by a Markov decision-making process in which agent state transitions do not consider all past states and actions, but rely only on the current state, with actions selected in the current state.

를 결정하기 위한 대표적인 강화학습 기법 중 하나이다. Q-learning is an optimal policy for decision-making problems.

It is one of the representative reinforcement learning techniques for determining .

와 행동

의 Q-값(

)을 계산하고, 총 보상을 최대화하는 방향으로 Q-값을 업데이트한다. The general process of Q-learning is to compute a pair of states at time t using the Bellman equation

and act

Q-value of (

), and update the Q-value in the direction that maximizes the total reward.

에 기초하여 최적 Q-값(

)은 현재 보상

및 디스카운트된 최대 미래 보상

의 합에 의해 얻어지며, 여기서,

는 현재 및 미래 보상의 상대적 중요도를 설명하는 디스카운트 팩터를 나타낸다. In Equation 18, the optimal policy

Based on the optimal Q-value (

) is the current reward

and discounted maximum future compensation

It is obtained by the sum of , where

represents a discount factor describing the relative importance of current and future rewards.

가 감소함에 따라 에이전트는 현재 보상에 점점 더 집중하기 때문에 근시안적으로 된다. 그러나, 더 큰

를 사용하면, 에이전트를 향후 보상에 점점 더 집중할 수 있어 원시안적으로 된다. 현재와 미래의 보상의 균형을 맞추기 위해

값은 Q-러닝을 이용하여 시스템 운영자에 의해 조정될 수 있다. discount factor

As β decreases, the agent becomes myopic because it focuses more and more on the current reward. However, larger

, the agent becomes more and more far-sighted as it can focus more and more on future rewards. To balance present and future rewards

The value can be adjusted by the system operator using Q-learning.

)이 시간 t에서 한 쌍의 상태 및 행동에 따라 업데이트 될 때마다,

는 상태-행동 테이블, 즉 Q-값 테이블에 저장된다. Q-value (

) is updated according to the state and behavior of a pair at time t,

is stored in a state-action table, i.e., a Q-value table.

The agent selects its own action using the Q-value table at every time t, and each element (Q-value) in the Q-value table associated with the selected pair of states and actions is updated through the following Bellman equation: .

는 새로운 Q-값이 기존 Q-값을 재정의하는 정도를 결정하는 학습 속도를 나타낸다. In Equation 19,

denotes the learning rate that determines the degree to which a new Q-value overrides an existing Q-value.

에서 에이전트는 아무것도 학습하지 않고 Q-러닝 프로세스의 탐험(exploration) 없이 과거 Q-값만을 사용한다.

In , the agent learns nothing and uses only past Q-values without exploration of the Q-learning process.

에서 에이전트는 현재 보상 및 탐사(exploitation) 없이 디스카운트된 최대 미래 보상만 사용하여 Q-값을 업데이트한다. however,

At , the agent updates the Q-value using only the current reward and the discounted maximum future reward without exploration.

의 선택과 유사하게, 탐험과 탐사 사이의 트레이드오프는 0 내지 1 범위 내에서 시스템 운영자에 의해 결정될 수 있다. discount factor

Similar to the choice of , the tradeoff between exploration and exploration can be determined by the system operator in the range of 0 to 1.

를 반복적인 방식으로 업데이트 함으로써 Q-값은 점점 커지고, 에이전트는 다음과 같이 가장 큰 Q-값을 갖는 최적 정책

를 얻게 된다. Finally, using Equation 19

By updating in an iterative way, the Q-value gradually increases, and the agent selects the optimal policy with the largest Q-value as

will get

According to one embodiment of the present invention, the above-mentioned Q-learning method is applied to individual devices (air conditioners, washing machines or ESSs) to calculate the optimal operation schedule of smart home devices with PV systems and ESSs, and as a result The optimization technique according to the present embodiment can reduce the consumer's electricity bill within the consumer's preferred device scheduling and convenience level.

Hereinafter, states, actions, and rewards in the Q-learning-based HEMS optimization according to the present embodiment will be described in detail.

The Q-learning algorithm according to this embodiment considers a situation in which it runs for 24 hours with a scheduling resolution of 1 hour.

에서, 세탁기(WM), 에어컨(AC) 및 ESS의 상태 공간(state space)은 각각 다음과 같이 표현된다.

In , the state spaces of the washing machine (WM), air conditioner (AC) and ESS are respectively expressed as follows.

는 시간 t에서 WM, AC의 에너지 소비량이고

는 ESS의 SOE를 나타낸다. Here, state

is the energy consumption of WM and AC at time t,

represents the SOE of the ESS.

The optimal action for each device depends on the agent's environment, including the current state, as described above.

The action space of WM, AC and ESS is as follows.

)를 소비한다. 반면, "OFF" 행동에서 세탁기 에이전트는 세탁기를 끈다. 에어컨 에이전트의 행동은 수학식 23과 같이 복수의 레벨(예를 들어, 10 레벨)의 에너지 소비량으로 구분된다. 여기서,

는 에어컨의 단위 에너지 소비량을 나타낸다. In Equation 22, the washing machine agent performs a binary action. In "ON" action, the washing machine agent turns on the washing machine, and the washing machine has constant energy (

) is consumed. On the other hand, in the "OFF" action, the washing machine agent turns off the washing machine. The behavior of the air conditioner agent is divided into a plurality of levels (eg, 10 levels) of energy consumption as shown in Equation 23. here,

represents the unit energy consumption of the air conditioner.

의 단위로 복수의 셋으로 구분된다. Similar to the air conditioner, the ESS agent is also

It is divided into a plurality of sets in units of .

와

와 같이 충전 및 방전 행동으로 분류된다. Each of these distinct actions

Wow

It is classified into charging and discharging behavior as

The reinforcement learning-based HEMS optimization algorithm according to the present embodiment calculates hourly energy scheduling of the device for 24 hours.

,

및

행렬을 사용하여 설명되고,

,

,

및

이다. As described above, for a given state and behavior, the Q-value tables of WM, AC, and ESS agents are

,

and

described using matrices,

,

,

and

to be.

는 집합 A의 카디널리티(cardinality)이다. here,

is the cardinality of set A.

The reward function for each appliance agent is formulated as the sum of the negative electricity cost and the negative dissatisfaction cost related to consumer preference convenience and appliance operating characteristics.

은 다음과 같이 정의된다. Comprehensive reward for HEMS

is defined as:

,

및

는 다음과 같은 관점에서 HEMS 성능을 평가하는 것을 목표로 한다. In Equation 25, the three reward functions

,

and

aims to evaluate HEMS performance from the following perspectives.

(i) WM's electricity costs and penalties for undesirable behavior by consumers;

(ii) Penalties for AC electricity costs and thermal discomfort for consumers

(iii) Penalty due to electricity cost and energy shortage due to overcharging and undercharging of ESS

Finally, the reward function for the WM agent is expressed as

및

는 각각 WM의 소비자 선호 시작 및 종료 시간이고,

및

는 선호 동작 시간 간격과 비교하여 이른 동작 및 늦은 동작에 대한 패널티이다. here,

and

are the consumer preferred start and end times of WM, respectively,

and

is a penalty for early and late actions compared to the preferred action time interval.

이전 또는

이후에 WM 에너지 소비를 스케줄링하면 불만족 비용이 보상 함수에 음수 값으로 추가된다. WM agent

before or

When scheduling WM energy consumption later, the cost of dissatisfaction is added as a negative value to the compensation function.

Otherwise, only negative electricity costs are added to the compensation function.

The reward function of the AC agent is

는 소비자 열적 불편함에 대한 패널티이다.

is the penalty for consumer thermal discomfort.

와

및

의 차이로 정의되며,

가

및

의 범위를 벗어난 경우에만 음의 부호가 있는 보상으로 간주된다. The cost of dissatisfaction is the consumer preference temperature

Wow

and

is defined as the difference between

go

and

Only if it is outside the range of , it is regarded as a compensation with a negative sign.

Finally, the compensation function of the ESS agent consists of negative electricity cost and negative energy underutilization cost as follows.

와

는 ESS 과충전 및 부족충전에 대한 패널티이다. 이러한 경우, ESS의 에너지 부족 이용은 SOE가

보다 낮거나(부족충전) 또는

(과충전)보다 커지면 발생하고, ESS의 에너지 부족 이용 중에 에너지 비용과 함께 보상에 반영된다.

Wow

is a penalty for ESS overcharging and undercharging. In this case, the ESS's use of the energy deficit is the SOE's

less than (undercharged) or

It occurs when it exceeds (overcharge), and is reflected in compensation along with energy cost during energy shortage use of ESS.

This embodiment considers a situation in which HEMS schedules the energy consumption of air conditioners based on indoor and outdoor temperatures according to thermal conditions preferred by consumers.

Typically, HEMS calculates the current indoor temperature using an approximate model (ie, the equivalent thermal parameters (ETP) model) as shown in Equation 4 in relation to the previous indoor temperature, current outdoor temperature, and air conditioner energy consumption.

In contrast, this embodiment proposes an Artificial Neural Network (ANN)-based method for predicting the indoor temperature in relation to the energy consumption of the air conditioner.

In the ANN model according to this embodiment, the AC agent learns the extent to which the energy consumption of AC affects the current indoor temperature, which means estimation of the function f representing the relationship between indoor temperature and AC energy consumption as follows do.

는 이전 실내 온도

, 사용자 선호 실내 열적 조건

, 일기 예보(

) 및 AC 에너지 소비량

과 같은 입력 데이터와 예측된 현재 실내 온도에 대한 출력 사이의 관계를 설명하기 위한 근사화된 함수이다. here,

is the previous room temperature

, user-preferred indoor thermal conditions

, weather forecast (

) and AC energy consumption

It is an approximated function for explaining the relationship between input data such as <RTI ID=0.0>and</RTI>

As shown in FIG. 3, the ANN model according to this embodiment is composed of one input data layer with 5 neurons, 3 hidden layers with 17 neurons, and an output layer with 1 neuron.

를 통해 입력 벡터의 가중합과 고정 바이어스

를 계산하고, 가중합은 전달 합수를 통해 다음 레이어로 전달된다. Each layer is weighted

The weighted sum of the input vectors and the fixed bias

is calculated, and the weighted sum is passed to the next layer through the transfer function.

In this embodiment, the ReLu function is used as a transfer function. Also, the Adam optimization algorithm is used for training of the ANN model, and the learning rate of the optimization algorithm is set to 0.005.

The approximate temperature prediction by the ANN model is input to the Q-learning module for the AC agent. The approximated model allows the AC agent to more accurately calculate the cost of dissatisfaction during the Q-learning process and more efficiently determine the optimal energy consumption scheduling.

Finally, the HEMS with the PV system, ESS, and equipment learns an energy management policy that optimizes electricity rates and consumer convenience levels using the algorithm shown in FIG. 4 .

HEMS receives the previous indoor temperature, the consumer's preferred indoor temperature range, predicted outdoor temperature and AC energy consumption, and predicts the current indoor temperature using the ANN model.

Then, Q-learning according to this embodiment is initialized to schedule optimal energy consumption and ESS charge/discharge of the device.

5 shows a Q-learning and ANN-based framework for optimally controlling home appliances and ESS, and FIG. 6 shows the configuration of a device for HEMS optimization based on reinforcement learning according to a preferred embodiment of the present invention. it is a drawing

As shown in FIG. 5 , the reinforcement learning-based HEMS optimization apparatus according to the present embodiment may include a processor 500 and a memory 502.

The processor 500 may include a central processing unit (CPU) capable of executing a computer program or other virtual machines.

Memory 502 may include a non-volatile storage device such as a non-removable hard drive or a removable storage device. The removable storage device may include a compact flash unit, a USB memory stick, and the like. Memory 502 may also include volatile memory, such as various random access memories.

The memory 502 stores program instructions executable by the processor 500 .

The program instructions according to the present embodiment are based on the pre-learned artificial neural network, using the indoor temperature at time t-1, the upper and lower limits of the user's preferred temperature, the external temperature predicted at time t, and the energy consumption of the air conditioner. When the current temperature is predicted, the predicted current temperature is input to the Q-learning module for the air conditioner agent, and the Q-learning module outputs an action according to the predicted current temperature, as long as according to the predicted current temperature The state and behavior of the pair are stored and updated in the Q-value table, and the optimum for maximizing the sum of the rewards according to the actions of the agents corresponding to the air conditioner agent, the device with the non-blockable load, and the device with the load that can be blocked Explore policy.

Here, the device having the non-blockable load may include a washing machine, and the device having the non-blockable load may include an energy storage system.

In addition, the agent of the washing machine may perform one binary action of on/off, and the agent of the air conditioner and energy storage system may perform one action of a plurality of levels having a difference in unit energy consumption.

The embodiments of the present invention described above have been disclosed for illustrative purposes, and those skilled in the art having ordinary knowledge of the present invention will be able to make various modifications, changes, and additions within the spirit and scope of the present invention, and such modifications, changes, and additions will be considered to fall within the scope of the following claims.

Claims (7)

As a HEMS optimization device using reinforcement learning,
processor; and
Including a memory coupled to the processor,
the memory,
Based on the pre-learned artificial neural network, the current temperature is predicted using the indoor temperature at time t-1, the upper and lower limits of the user's preferred temperature, the external temperature predicted at time t, and the energy consumption of the air conditioner,
Input the predicted current temperature to the Q-learning module for the air conditioner agent,
When the Q-learning module outputs an action according to the predicted current temperature, a pair of states and actions according to the predicted current temperature are stored in a Q-value table and updated;
To search for an optimal policy that maximizes the sum of rewards according to the actions of each of the agents corresponding to the air conditioner agent, a device having a non-blockable load, and a device having a blockable load,
storing program instructions executable by the processor;
The device with the non-breakable load includes a washing machine, the device with the interruptable load includes an energy storage system,
The agent of the washing machine performs one binary action of on/off,
The agent of the air conditioner and energy storage system performs one action among a plurality of levels having a difference in unit energy consumption;
The compensation function for the behavior of the agent of the washing machine, air conditioner, and energy storage system is defined as the sum of negative electricity costs and negative dissatisfaction costs related to consumer preference convenience and device operation characteristics,
The artificial neural network is an HEMS optimization device using reinforcement learning to learn the extent to which the energy consumption of the air conditioner affects the current temperature. delete delete delete According to claim 1,
The artificial neural network is an HEMS optimization device using reinforcement learning composed of one input data layer having a plurality of neurons, a plurality of hidden layers having a plurality of neurons, and an output layer having one neuron. A HEMS optimization method using reinforcement learning in a device including a processor and memory,
Predicting the current temperature using the indoor temperature at time t-1, the upper and lower limits of the user's preferred temperature, the external temperature predicted at time t, and the energy consumption of the air conditioner based on the pre-learned artificial neural network;
inputting the predicted current temperature to a Q-learning module for an air conditioner agent;
storing and updating a pair of states and actions according to the predicted current temperature in a Q-value table when the Q-learning module outputs an action according to the predicted current temperature; and
Searching for an optimal policy that maximizes the sum of rewards according to the actions of each of the agents corresponding to the air conditioner agent, a device having a non-blockable load, and a device having a blockable load,
The device with the non-breakable load includes a washing machine, the device with the interruptable load includes an energy storage system,
The agent of the washing machine performs one binary action of on/off,
The agent of the air conditioner and energy storage system performs one action among a plurality of levels having a difference in unit energy consumption;
The compensation function for the behavior of the agent of the washing machine, air conditioner, and energy storage system is defined as the sum of negative electricity costs and negative dissatisfaction costs associated with consumer preference convenience and device operation characteristics,
The HEMS optimization method using reinforcement learning in which the artificial neural network learns the extent to which the energy consumption of the air conditioner affects the current temperature. delete

Images