KR20230070779A - Demand response management method for discrete industrial manufacturing system based on constrained reinforcement learning - Google Patents

Abstract

Demand response (DR) has been recognized as an effective method of improving the stability and financial efficiency of a power grid. DR realization for the industrial sector as a primary consumer is urgently needed. The present invention provides a novel industrial price-based DR management approach for an industrial discrete manufacturing system by simultaneously considering energy costs and daily production goals. To achieve this, the discrete manufacturing system is modeled as a constrained Markov decision process (CMDP), and a constrained reinforcement learning (CRL) algorithm is adopted to determine an efficient operation strategy for the discrete manufacturing system. A simulation was performed by using a real lithium-ion battery assembly system to verify the performance of the method in accordance with the present invention. The evaluation results showed that the method in accordance with the present invention can optimize energy costs without violating production goals.

Description

Demand response management method for discrete industrial manufacturing system based on constrained reinforcement learning}

The present invention relates to a demand response management method for a discrete industrial manufacturing system, and more specifically, a cost-effective operating strategy for a discrete industrial manufacturing system modeled with a Constrained Markov Decision Process (CMDP) and adopting a Constrained Reinforcement Learning (CRL) algorithm. It is about a demand response management method for a discrete industrial manufacturing system to which constraint reinforcement learning can determine

Demand response (DR) refers to the change in energy demand by customers for a time-varying price. A well-designed DR program can promote economic efficiency, operational flexibility and system reliability in the smart grid. According to a recent study, energy consumption in the industrial sector has increased dramatically over the past few years, accounting for more than 50% of global energy use. In addition, energy consumed by the industrial sector is expected to increase further in the coming years. Therefore, it is important to develop an effective DR program for energy management in the industrial sector.

However, implementing an effective DR program for industrial manufacturing systems is a complex and challenging task. Unlike the residential and commercial sectors, industrial energy consumption patterns typically result from a variety of sequential, interdependent and correlated production operations and a variety of manufacturing equipment. Therefore, a successful industrial DR implementation requires a high-resolution system model capable of acquiring all the physical characteristics of each device in the target system. Additionally, many industrial DR measures can result in lost production or increased costs associated with shifting production, which will limit industrial customers' interest in participating in DR programs.

Over the past few years, reinforcement learning (RL) methods have attracted more and more attention for solving complex sequential decision-making problems. A series of successes in RL have been achieved with the application of deep Q-learning. RL, a branch of artificial intelligence (AI) inspired by behavioral psychology, explores how software agents can act in uncertain environments to maximize cumulative rewards. Use of RL includes identification of agents and environments. The agent interacts with the environment and receives feedback from the environment. This feedback (compensation) is used to evaluate each state-action pair. In general, the benefits of applying RL to address decision-making problems can be summarized in two main aspects.

First, RL has no models. There is no need for predefined rules or predefined rules to determine how to select an action. Second, RL is adaptive. Valuable knowledge can be learned from historical data to deal with highly uncertain system dynamics, and the extracted knowledge can be generalized and applied to newly emerging situations. The impressive advantages of RL can potentially solve smart grid decision-making problems such as DR energy management, electric vehicle charging and dynamic economic dispatch.

Much attention is being paid to the application of RL to solve DR management problems in the residential and commercial sectors. For example, previous studies have proposed a DRL-based energy management algorithm to obtain an optimal charging strategy for an electric vehicle by probabilistically considering user behavior and electricity price. In addition, other conventional studies have developed a DRL-based DR scheduling algorithm to optimally control a set of home appliances by considering the uncertainty of residents' behavior, real-time electricity rates, and outdoor temperature.

In addition, other prior studies have applied batch RL-based day-ahead DR planning to optimize the energy consumption of energy-varying loads such as heat pump thermostats or electric heaters under limited information on system dynamics. In addition, other prior studies explored the dynamic pricing problem at the service provider side using the RL method, and the pricing problem was modeled with the Markov Decision Process (MDP) and then the Q-leaning algorithm was used to determine the ideal value. In addition, in other prior studies, RL has been applied to explore ideal incentive ratios for various customers with the goal of optimizing both the service provider's benefit and the customer's cost. In addition, in another prior study, a multi-agent RL-based DR algorithm was developed to minimize the sum of power cost and user dissatisfaction cost considering smart homes with various types of home appliances.

The studies highlighted above have demonstrated a series of successes in addressing residential and commercial DR challenges, but that success is due to simple application scenarios and/or low device diversity. Moreover, the devices in these applications are generally considered to operate independently of each other rather than in conjunction with each other. With these capabilities, various RL approaches such as Q-learning and Actor-Critic can be applied to residential and commercial scenarios to solve cost minimization problems or dynamic pricing problems.

Unusual work has been undertaken to explore the potential of DR in the industrial sector using RL-related technologies for several reasons:

i) Industrial assets are usually closely linked and work together.

ii) Continuous equipment in the manufacturing process must operate in a specific sequence without breaking the sequence.

iii) Industrial DR management should consider how to maintain production requirements as well as reducing energy consumption in general.

In a recent study, a real-time based DR method was proposed to control the energy consumption of a Li-ion battery assembly manufacturing system. Here, the manufacturing system is modeled as a Markov game. A multi-agent DRL algorithm was then used to solve this Markov game. However, this study only tries to optimize energy consumption, and no constraints on production targets are mentioned in terms of actual production. While minimizing energy costs is attractive, manufacturing systems should not achieve this at a cost that violates production requirements.

Korean Patent Publication No. 10-2019-0132193

To overcome these problems, the present invention aims to provide a Constraint Reinforcement Learning (CRL) based DR management algorithm for an industrial manufacturing system that seeks to optimize energy costs while ensuring production targets.

To this end, the demand response management method of a discrete industrial manufacturing system to which constrained reinforcement learning is applied according to the present invention is a method performed in an energy management device of a discrete industrial manufacturing system _. It executes the action (task) (a _t ) and acquires the reward (R _t ) and cost (R ^c _t ) to move to the next state (S _t+1 ), the current state (S _t ), and the action (a _t ) , an experience accumulation step of storing a training set of total time steps (t=1 to T) by storing samples consisting of the next state (S _t+1 ), reward (R _t ), and cost (R ^c _t ), and , Randomly sampling a mini-batch from the training set and calculating a target label of a state-action value function and a state value function for the mini-batch, updating the parameters of the action value function and the parameters of the state value function according to the gradient descent method so that an error between the function value and the target value is minimized; Updating parameters of a policy function; Updating parameters of a target state value function; and a parameter update step including a step of updating a Lagrange multiplier, and an operation of executing an action according to a policy in the current state based on the parameter determined in the parameter update step, obtaining a reward, and moving to the next state. It is characterized in that the parameter update step is repeated until the cumulative compensation is maximized, including an accumulation compensation calculation step of repeatedly executing and calculating an accumulation compensation.

As described above, the present invention has the effect of determining a cost-effective operating strategy for a discrete industrial manufacturing system.

With the advent of artificial intelligence (AI), there is growing interest in adopting reinforcement learning (RL) to handle complex decision-making processes. The present invention proposes a CRL-based DR algorithm for discrete manufacturing systems to minimize energy costs while achieving production targets. In particular, discrete manufacturing systems are formulated into CMDP and CRL algorithms are adopted to identify optimal operating schedules for all machines.

Finally, as a simulation result of applying the CDRL algorithm according to the present invention to the lithium ion battery assembly process, it was demonstrated that the algorithm according to the present invention can minimize the energy cost of the process without violating the production target.

In the future, the DR scheme according to the present invention can be evaluated in many other types of industrial facilities that integrate energy storage systems and renewable energy resources (eg solar and wind power), and can also be implemented in actual industrial manufacturing systems to evaluate their performance. you will be able to receive

1 is a diagram showing the general configuration of an industrial discrete manufacturing system according to the present invention.
2 is a diagram showing the configuration of an actor-critic scheme;
3 is a view showing a lithium ion battery assembly process to which the method according to the present invention is applied.
4 is a diagram showing a hierarchical structure of a lithium ion battery module.
5 is a diagram showing accumulated compensation in a learning process according to the present invention.
6 is a diagram showing total energy demand corresponding to electricity price;
Figure 7 shows the storage capacity of final battery production at each time step (interval).
Figure 8 shows the total cost obtained by CSAC and Gurobi solver.
9 is a diagram showing accumulated compensation in the learning process from June 2 to 4, 2019.
10 is a diagram showing total energy demand corresponding to electricity prices from June 2 to June 4, 2019.
11 is a diagram showing the storage amount of final battery production in each time interval from June 2 to 4, 2019.
12 is a flow chart showing processing steps in the method according to the present invention.

To apply the CRL algorithm according to the present invention, the operating cost optimization problem of an industrial manufacturing process is formulated as a Constrained Markov Decision Process (CMDP) without prediction of unknown variables or dynamic models that consider both energy consumption and resource management.

Compensation and cost functions are elaborately designed and integrated into the construed CDMP to achieve production targets while minimizing energy costs.

Based on the CMDP according to the present invention, we design a new CRL algorithm that solves the CMDP to determine the optimal operating policy for all equipment in the industrial manufacturing system.

Finally, the efficiency of the CRL algorithm according to the present invention is evaluated using a typical lithium ion battery assembly manufacturing process, which is an actual industrial case. The evaluation results show that the CRL algorithm according to the present invention can balance energy demand and supply and reduce energy costs while ensuring production requirements. This indicates that the algorithm according to the present invention has the potential to handle complex industrial DR management problems.

The present invention is the first application of CRL to DR management of an industrial manufacturing system that simultaneously considers the operation of several continuous machines and the use of various resources (electricity and production materials).

The specification of the present invention is structured as follows.

First, the problem formulation of a typical individual manufacturing system is detailed, including an energy demand model, a production resource balance model and a target function, and then the CRL algorithm according to the present invention for solving the conventional DR problem is described. The evaluation results for the CRL algorithm will be described.

Formulate the problem

We look at the energy management issues of a typical discrete manufacturing system in a price-based DR environment. Assume that the facility has an energy management center (EMC) that receives real-time pricing (RTP) data from the utility company on a periodic basis (eg hourly). EMC then determines the optimal control policy to manage the energy consumption of the various machines in the discrete manufacturing system. The CRL-based DR system according to the present invention is pre-installed in the EMC.

1 shows a typical industrial discrete manufacturing system.

Referring to FIG. 1 , M _i,j and B _i,j denote a machine and a corresponding buffer used to process and store intermediate products, respectively. where i denotes the ith serial production line branch and j denotes the jth machine or buffer of the ith branch. The mathematical formulas of the energy demand model, the production resource balance model, and the target function can be described as follows.

A. Energy Demand Model

Typically, in a discrete manufacturing system, each machine has two operating options: running and idle. Running means the machine is in full operating mode and Idle means the machine is going into sleep mode.

z _i,j ^t is the binary variable of machine M _i,j . That is, z _i,j =1 when M _i,j is the operating mode, otherwise z _i,j =0.

During step t, each machine can select only one operating option. Therefore, the energy consumption of machine M _i,j during step t can be expressed as Equation 1.

Here, e _i,j ^op and e _i,j ^idle denote the energy consumption of machine M _i,j in operating or sleep mode, respectively.

Therefore, the energy consumption of the entire manufacturing system during step t can be summed using Equation 2.

In addition, the total energy consumption E ^t of the entire system depends on the maximum capacity E _max of the local power grid as shown in Equation 3.

B. Production resource balance model

Between the two sequential machines, the buffer B _i,j serves as a storage space for the different parts of the product along the production process. In step t, the material storage of buffer B _i,j is expressed as Equation 4.

Here, P _i,j ^t (C _i,j ^t ) represents the amount produced (consumed) in buffer B _i,j during step t. P _i,j ^t and C _i,j ^t are the same as Equations 5 and 6, respectively.

p _i,j ^t (c _i,j ^t ) denotes the production (consumption) rate of machine M _i,j in operating mode z _i,j ^t .

In order to ensure normal operation of the process machine, the amount of material in the buffer B _i,j during step t must maintain constraints as shown in Equation 7.

where B _i,j ^min and B _i,j ^max represent the minimum and maximum amounts of material in buffer B _i,j .

C. Objective function

The goal of real-world industrial discrete manufacturing systems is to perform production operations with minimal energy costs. The objective function of the system can be expressed by Equations 8 and 9.

보다 작을 수 없음을 의미한다.Equation 8 defines the energy cost minimization for the day where v ^t represents the electricity price at step t. Equation 9 defines the production target constraint, which means that the residual value B _final ²⁴ of the final buffer B _final is the predefined target output at the end of the last step (i.e. t=24).

This means that it cannot be smaller than

CRL methodology

First, CMDP is briefly described, then the industrial DR management problem is formulated as CMDP, and finally, the CRL methodology is applied to solve CMDP.

A.CMDP

As an extension of MDP, CMDP features six pairs: S , A , π , P _r , R , R ^c . Here, S represents the state space in which there are available states. A represents an action space with available actions. π represents the distribution of actions in a given state.

에 종속되는 예상 할인 수익 J를 최대화하는 최적 정책 π를 식별하는 것이다. P _r : S × A × S → [0,1] represents the transition probability function, R (or R ^c ) : S × A × S → R (or R ^c ) represents the reward function (or cost function) . CDMP generally includes the concept of agents and environments interacting with each other in discrete time steps. During each step t∈[0,T], the agent observes the state s_t∈S of the environment and chooses an action a _t∈ A according to policy π . At the next step t+1, the agent gets the reward R(s _t ,a _t ,s _t+1 )⊂ R and the cost R ^c (s _t ,a _t ,s _t+1 )⊂ R ^c . The environment moves to the next state s t ₊₁ according to the transition probability function P _r (s _t+1 |s _t ,a _t ). The agent's target is an upper bound constraint on the expected discounted total cost J ^c

is to identify the optimal policy π that maximizes the expected discounted return J that depends on

이고, γ∈[0,1]은 할인율를 나타내고, R과 R_t ^c는 각각 R(s_t,a_t,s_t+1)와 R^c(s_t,a_t,s_t+1)의 약자이다. where τ is the path

, γ∈[0,1] represents the discount rate, and R and R _t ^c are abbreviations of R(s _t ,a _t ,s _t+1 ) and R ^c (s _t ,a _t ,s _t+1 ), respectively. am.

Finally, the state value function V ^π (s) and action value function Q ^π (s) are defined as follows.

Here, π is a decision policy mapped from state space S to action space A , or a probability policy mapped to the probability of choosing another action in a state.

The state value function and the action value function can be decomposed into the immediate reward and the discount value of the subsequent state according to the Bellman equation.

B. CMDP Formulation of Industrial DR Problems

The industrial DR management problem is established as CMDP and EMC is considered as a learning agent that interacts with the environment, i.e. the entire discrete manufacturing system. The formalized CMDP includes state space, action space, reward function, and cost function. The optimization horizon consists of 24 steps with a total of 24 decisions to be made based on hourly price.

The goal of the present invention is to find an optimal strategy to meet production target constraints while minimizing the total energy cost of a discrete manufacturing system.

1) state space

State s observed at the beginning of each stage in industrial DR management includes four parts: time indicator t, electricity price v ^t , mechanical energy consumption e ^t _ij , and buffer storage B ^t _i,j .

Equation 18 represents a sample of the state s _t at step t, including time indicator t, electricity price v ^t , mechanical energy consumption e ^t _ij , and buffer storage B ^t _i,j .

2) action space

At the beginning of each phase, the EMC schedules each machine's operation with a binary decision variable z ^t _i,j ∈{0,1}. Action space A thus contains all the machine's actions.

Equation 20 represents a sample of action (a _t ) at step t. where z ^t _i,j represents the selection of the operating point of machine M _i,j at step t.

3) reward function and cost function

As described above, the objective function of industrial DR consists of two parts: satisfying the production task and minimizing energy costs. Therefore, in this CMDP framework, the reward function R _t is defined as:

where R _t is the reciprocal of the energy cost of the manufacturing assembly system at step t.

The cost function R _t ^c is defined as

where R _t ^c < 0, and B ^t _final is the storage amount of the final buffer B _final at step t.

에서 얼마나 벗어났는지 계산한다. 두 번째 줄은 허용 가능한 최대 저장 B^max _final을 초과하는 최종 생산 저장 B^t _final의 양을 측정한다. 세 번째 줄은 허용 가능한 최소 저장 B^min _final 미만인 최종 생산 저장 B^t _final의 양을 계산한다. 네 번째 줄은 최종 생산 저장 B^t _final이 최소 저장 B^min _final과 최대 저장 B^max _final 사이에 있음을 나타낸다. The first line stores the final production at the end of the last time (t=24) B ^t _final is the target output

Calculate how far out of The second line measures the amount of final production storage B ^t _final that exceeds the maximum allowable storage B ^max _final . The third line calculates the amount of final production storage B ^t _final that is less than the minimum allowable storage B ^min _final . The fourth line indicates that the final production save B ^t _final is between the minimum save B ^min _final and the maximum save B ^max _final .

C. CRL Algorithm

The CRL algorithm according to the present invention represents a constrained soft actor-critic (CSAC) for solving CMDP. In order to understand the CSAC algorithm, the background of the actor-critic algorithm will be explained first, and then the CSAC algorithm according to the present invention will be described in detail.

1) Actor-critic algorithm

According to different learning strategies, RL algorithms can be classified as value-based, policy-based or actor-critic. Value-based approaches such as Q-learning and SARSA use only value functions and do not have explicit formulations for policies. Policy-based approaches, such as policy gradient, try to identify the optimal policy directly without any form of value function. The third type is an actor-critic algorithm such as Fig. 2 that combines the above two approaches. Actors are responsible for generating actions, and critics are responsible for processing rewards. During training, when the agent observes the most recent state in the environment, the actor outputs a set of actions based on the current policy. On the other hand, critics will judge how good the current policy is through the value function. The deviation between the expected value and the reward received is then represented by the time lag (TD) error, which is fed back to actors and critics simultaneously to adjust the policy and value functions.

Conventional actor-critic algorithms generally have poor sample efficiency, especially policy-based learning algorithms such as proximity policy optimization (PPO) and asynchronous actor-critic (A3C). Policy-based learning algorithms such as Deep Deterministic Policy Gradient (DDPG) require new sample collection at each gradient step, which greatly affects the efficiency of the training process. Because the out-of-policy learning algorithm can reuse past samples, the sample efficiency is greatly improved. However, out-of-policy learning algorithms are sensitive to learning hyperparameters, meaning stability and convergence can present greater challenges. To overcome these problems, the soft actor-critic (SAC) algorithm, an out-of-policy maximum entropy DRL algorithm, was designed to achieve robust and sample-efficient performance.

The SAC algorithm improves the stochastic policy through an off-policy method. A salient property of SAC is entropy normalization, where the agent learns a policy to strike a balance between expected reward and entropy. This is closely related to the exploration-exploration mechanism. In other words, as entropy increases, more exploration occurs. This allows SAC to accelerate the learning rate and prevent premature convergence of the policy to a non-ideal local optimum.

In the maximum entropy RL framework, the value function changes to include the entropy bonus at each step.

는 단계 t에서 확률적 정책을 위한 엔트로피이다. here,

is the entropy for the stochastic policy at step t.

Therefore, the Bellman equation in entropy normalized form is:

와

는 수학식 27에 의해 연결된다. Two normalized value functions

and

is connected by Equation 27.

(여기서,

)는 수학식 28과 같이 유도된다. According to Equation 27, the approximate solution of the policy

(here,

) is derived as in Equation 28.

을 산출한다. 최적 정책

을 참조하면 최적값 V_h*(s)도 얻을 수 있다. 수학식 28에 따르면, Q-값 함수의 업데이트 메커니즘은 오프 정책 방식을 통해 달성될 수 있다.When Q _h ^π converges to Q _h ^* , the agent chooses the optimal policy

yields optimal policy

Referring to , the optimal value V _h *(s) can also be obtained. According to Equation 28, the update mechanism of the Q-value function can be achieved through an off-policy method.

The SAC framework is presented in Algorithm 1. Implementation details such as clipped double Q-learning, baseline value function, and deferred update of value function are omitted here.

Algorithm 1: Soft Actor-Critic

1. Initialize policy and normalized value function parameters

2. Repeat section 3~6

3. Generate samples based on current policy

4. Sampling from the data buffer

5. Update the parameters of the value function according to Equation 26

6. Update the policy parameters according to Equation 28

7. Repeat 3-6 until convergence

2) CSAC algorithm

Despite the fact that SAC has had a series of successes in solving complex decision-making tasks, SAC is designed to handle MDP and cannot solve CMDP which has practical constraints. From this point of view, in order to solve the constraints of CMDP, we propose a CSAC algorithm according to the present invention by integrating an adjustable Lagrange multiplier with SAC.

The industry DR problem was formulated as CMDP, where production target constraints were partitioned into each time step. That is, R _t ^c < 0 defined in Equation 22.

Under the SAC framework, the optimal solution of CMDP can be obtained by solving Equation (29).

Here, D is a data sampling buffer, that is, a series of empirical data.

By introducing Lagrange multipliers, the constrained optimization problem can be reformulated as:

는 다음과 같다. here,

is as follows

을 최대화하여 정책 π^k을 얻을 수 있다. Lagrange multipliers can be used to address constrained optimization problems. For the policy domain given a multiplier λ ^k ≥ 0 at the kth iteration

By maximizing , the policy π ^k can be obtained.

Then, set as in Equation 32 and repeat the process.

δ _i is the step size for updating λ. [] ⁺ represents a non-negative real number.

It is demonstrated that the iterative method of updating policy parameters and Lagrange multipliers can converge to an ideal and feasible solution when the following three assumptions are satisfied.

First, V _h ^π (s) is bounded by a range for all policies π∈Π.

Second, all minima of J _c (π) are feasible solutions.

와

는 정책 신경망의 파라미터 θ를 업데이트하기 위한 스텝이다. third,

and

is a step for updating the parameter θ of the policy neural network.

For finite repetition situations, δ _λ can actually be set smaller than δ _θ . The CSAC algorithm according to the present invention is summarized as Algorithm 2.

Algorithm 2: Constrained Soft Actor-Critic

, 상태-행동가치 함수 Q 파라미터

, 대응하는 업데이트 스텝 크기

, 라그랑주 승수 λ, 온도 파라미터 α, 할인율 γInput values: policy neural network parameter θ, state value function V parameter

, the state-action-value function Q parameter

, the corresponding update step size

, the Lagrange multiplier λ, the temperature parameter α, and the discount rate γ

, 현재 단계를 위한 행동만이 실행될 것이다. Output: Optimal control strategy A ^* = for all machines

, only actions for the current step will be executed.

, 상태-행동가치 함수 Q 파라미터

, 대응하는 업데이트 스텝 크기

, 라그랑주 승수 λ, 온도 파라미터 α, 할인율 γ을 초기화한다. 1. Policy neural network parameter θ, state value function V parameter

, the state-action-value function Q parameter

, the corresponding update step size

, the Lagrange multiplier λ, the temperature parameter α, and the discount rate γ are initialized.

을 위한 빈 응답 풀(empty reply pool) D를 초기화한다. 2.

Initializes an empty reply pool D for

3. Each episode i

4. Each sample step t

을 선택한다5. Act on current state s _t

choose

6. Get reward R _t , cost R ^c _t and move to next state s _t+1

7. Save data D

8. End of sample phase

9. Each gradient step n

10. Randomly sample transition mini-batch B

11. For mini-batch B, calculate the target label (target value) as in Equations 33 to 37

를 업데이트한다12. By gradient descent

update

의해 업데이트한다 13. Gradient Descent

updated by

14. Update θ by gradient descent

로 목표 V 함수 파라미터

를 업데이트한다. 15. Latency

as the target V function parameter

update

16. Update target λ with step size δ _λ

17. End of the gradient step

18. Repeat the gradient step until you get the maximum cumulative reward (up to episode I _max )

19. End of Algorithm

The CSAC algorithm consists of 9 neural networks, and the 9 neural networks can be classified into 3 sets.

와 2개의 상태가치 함수

를 라그랑주 함수(수학식 32)에서 가치 함수와 관련있는

으로 근사화하는데 사용된다. The first four neural networks have two action-value functions

and two state-value functions

in relation to the value function in the Lagrange function (Equation 32).

is used to approximate

파라미터를 가진 2개의 행동가치 함수

와 2개의 상태가치 함수

를 근사화하는데 사용된다. The following four neural networks are associated with constraints and

2 action-value functions with parameters

and two state-value functions

is used to approximate

The last neural network with parameters θ is used to approximate the policy function.

, 상태-행동가치 함수 Q 파라미터

및 정책 네트워크 매개변수 θ이다. 또한, 대응하는 경사 하강 업데이트 스텝 크기

, 라그랑주 승수 λ, 온도 파라미터 α, 할인율 γ 역시 CSAC 알고리즘의 입력으로 사용된다.The parameters of the 9 neural networks are used as inputs to the CSAC algorithm. That is, the state value function V parameter

, the state-action-value function Q parameter

and the policy network parameter θ. Also, the corresponding gradient descent update step size

, the Lagrange multiplier λ, the temperature parameter α, and the discount rate γ are also used as inputs to the CSAC algorithm.

As shown in Algorithm 2, the entire learning process is controlled by three time indicators: i (line 3 of Algorithm 2), t (Line 4 of Algorithm 2) and n (Line 9 of Algorithm 2). of which i counts learning episodes. t represents the daily hourly step. n represents a gradient step. From line 3 to line 8, the algorithm enters the process of accumulating experience. As the step counter t increases, the agent executes a _t according to the current policy π, gets the reward R _t , the cost R ^c _t , and moves to the next state s _t+1 . In this process, each pair of (s _t ,a _t ,s _t+1 ,R _t ,R ^c _t ) is stored in response pool D for future agent learning.

는

로 계산된다. 여기서, r_t는 라그랑주 함수와 관련된 신경망에 대한

이고, R^c _t는 제약과 관련된 신경망에 대한 것이고,

는 V 신경망의 추가 카피로 수학식 41에 따라

을 업데이트한다. From lines 9 to 17, the algorithm enters the learning phase, the gradient process. In line 10, the algorithm samples a random batch of B samples from response pool D, i.e. {(s _t ,a _t ,s _t+1 ,R _t ,R ^c _t )}. |B| is the size of the mini-batch. In line 11, the training targets for the Q and V networks are calculated through Equations 33 to 37. Considering Equations 34 and 35, the training target

Is

is calculated as where r _t is for the neural network associated with the Lagrange function

, and R ^c _t is for the neural network associated with the constraint,

Is an additional copy of the V neural network according to Equation 41

update

로 표시된다.Considering Equations 35 and 36, a clipped double Q-learning technique is used to reduce bias in the policy update process. The training labels for the V neural network are

is indicated by

은 별도로 관리되어 학습된다. 수학식 37을 고려하면,

는

에서 샘플링된 행동이라는 것을 나타낸다. 12행에서 행동가치 신경망의 매개변수

를 업데이트하기 위해 평균 제곱 오차(MSE)

를 최소화하기 위해 Adam 옵티마이저를 사용하여 경사하강법 업데이트를 수행한다. 마찬가지로 13행에서 상태가치 신경망의 매개변수

를 업데이트하기 위해 MSE 오차

를 최소화하기 위해 Adam optimizer를 통해 경사하강법 업데이트가 수행된다. where two sets of Q networks

is separately managed and learned. Considering Equation 37,

Is

Indicates that the behavior is sampled from . Parameters of the action-value neural network in line 12

Mean squared error (MSE) to update

To minimize , gradient descent update is performed using the Adam optimizer. Similarly, in line 13, the parameters of the state-valued neural network

To update the MSE error

To minimize , gradient descent update is performed through the Adam optimizer.

In line 14, we perform gradient descent using the Adam optimizer to update the parameter θ of the policy neural network to minimize the loss.

의 지연된 업데이트가 수학식 41에서 수행됩니다. 16행에서, 라그랑주 승수 λ는 수학식 42에 따라 업데이트된다. 17행에서 알고리즘은 특수 미니 배치 B 샘플을 기반으로 현재 그레이디언트 프로세스를 완료한다. 마지막으로 알고리즘은 다음 에피소드에 진입하여 최대 누적 보상을 얻을 때까지 학습 과정을 반복한다. 이는 알고리즘이 최적의 운영 정책을 생성할 수 있음을 의미한다.On line 15, the target V network parameters

A deferred update of is performed in Equation 41. At line 16, the Lagrange multiplier λ is updated according to equation (42). At line 17, the algorithm completes the current gradient process based on the special mini-batch B samples. Finally, the algorithm repeats the learning process until it enters the next episode and obtains the maximum cumulative reward. This means that the algorithm can generate an optimal operating policy.

12 illustrates a process of a demand response management method of a discrete industrial manufacturing system to which constraint reinforcement learning is applied according to the present invention.

The process shown in FIG. 12 is performed in the energy management unit (EMC) of the discrete industrial manufacturing system or a separate computer device, and the optimal operating policy (energy strategy) generated through this process is applied to the discrete industrial manufacturing system. .

Specifically, the process shown in FIG. 12 is a process for generating an optimal operating policy for demand response management of a discrete industrial manufacturing system, and this process is performed by an energy management device (EMC) or a separate computer device of a discrete industrial manufacturing system. will be performed by the processor of Hereinafter, it will be described in detail as being performed by a processor.

Referring to FIG. 12, the method according to the present invention is largely divided into an experience accumulation step (S100), a parameter update step (S102), and an accumulated reward calculation step (S104).

The experience accumulation step (S100) is a process of generating a training set. The processor executes the action (a _t ) according to the policy (π) in the time interval t state (S _t ) and obtains the reward (R _t ) and cost (R ^c _t ) to reach the next state (S _t+1 ) and store samples consisting of the current state (S _t ), action (a _t ), next state (S _t+1 ), reward (R _t ), and cost (R ^c _t ), so that the total time step ( A training set of t = 1 to T) is stored in the response pool D provided in the internal memory.

Here, the state (S _t ) includes the electricity price, energy consumption of each machine, and storage amount of each buffer in the corresponding time interval (Equation 18). The action (a _t ) represents the operation or idleness of each machine in the corresponding time interval (Equation 20).

Parameter update step (S102) is a process of determining the input value of the algorithm 2 described above. The parameter updating step (S102) is a learning process (gradient step) for determining parameters of the neural network.

, 상태-행동가치 함수 Q 파라미터

, 대응하는 업데이트 스텝 크기

, 라그랑주 승수 λ, 온도 파라미터 α, 할인율 γ 등을 포함한다. The input values of Algorithm 2 are the policy neural network (policy function) parameter θ and the state value function V parameter.

, the state-action-value function Q parameter

, the corresponding update step size

, the Lagrange multiplier λ, the temperature parameter α, and the discount rate γ.

The processor randomly samples a mini-batch from the training set generated in the experience accumulation step (S100), and targets the state-action value function and the state value function for the mini-batch. A value (target label) is calculated, and the parameters of the action value function and the parameter of the state value function are updated according to the gradient descent method so that the error between the function value and the target value is minimized.

In addition, the processor performs a process of updating parameters of a policy function (Equation 40), a process of updating parameters of a target state value function (Equation 41), and a Lagrange multiplier. The updating process (Equation 42) is sequentially performed.

The accumulative compensation calculation step (S104) is a process of obtaining an accumulative compensation amount for one episode. Based on the determined parameter, the processor calculates an accumulated reward by repeatedly executing an action according to a policy in the current state, obtaining a reward, and moving to the next state.

Thereafter, the processor determines whether the accumulated compensation is maximum (S106), and if the accumulated compensation has not yet reached the maximum, repeatedly performs the parameter updating step (S102), and determines that the accumulated compensation has reached the maximum policy at that time. is determined as the optimal operating policy (S108).

When the optimal operating policy is determined through the above-described process, the optimal operating policy can be applied to the discrete industrial manufacturing system to achieve demand response management that can minimize energy costs while ensuring production targets.

Experiment evaluation

To verify the validity of the CSAC algorithm according to the present invention, a practical lithium-ion battery assembly process was selected as an example of a discrete manufacturing process. First, the details of the lithium-ion battery assembly process are introduced and the algorithm according to the present invention is evaluated.

A. Case study

As shown in FIG. 3 , the lithium ion battery assembly process includes four processes of assembly, saturation, formation and grading.

Assembling: Components are assembled together to form a battery module having a hierarchical structure of battery modules as shown in FIG. 4 . The components include a side frame (SF), a battery cell (BC), a cooling plate (CP), an intermediate frame (IF) and a compressed foam (CF).

Saturation: The module is injected with an appropriate amount of electrolyte.

Formation: The module is activated into a usable mode by an appropriate charge and discharge process.

Rating: Through a series of resistance and capacitance measurements, battery modules are rated according to their performance.

Figure 3 provides an overview of the battery module assembly process which can be divided into 10 tasks with each task assigned to a related machine. Operational information for each machine, including available operating modes, production speed, power consumption and buffer capacity, is provided in Table 1.

Based on these parameters, the assembly system's target is set to produce 500 lithium-ion battery units per day (i.e., the machine's maximum buffer capacity). The maximum amount of power drawn by the system is set at 500 kW.

B. Numerical results

, 상태-행동가치(state-action value) 신경망 Q 파라미터

, 정책 신경망 파라미터 θ, 라그랑주 승수 λ를 업데이트하기 위해 아담 옵티마이저(Adam optimizer)를 사용한다.As shown in Table 2, the state-valued neural network V parameters to start training

, state-action value neural network Q parameters

, we use the Adam optimizer to update the policy neural network parameter θ, and the Lagrange multiplier λ.

는 각각 5e^-4, e^-3, e^-3 및 e^-5로 설정된다.Step sizes for state-value networks V, state-action-value Q, policy networks

are set to 5e ^-4 , e ^-3 , e ^-3 and e ^-5 , respectively.

The number of hidden layers for each neural network is set to 2 and each layer has 64 neurons. In addition, ReLU (Rectified Linear Unit) is implemented as an activation function to connect the two hidden layers in each neural network. In the normalization of the Q-value neural network, the temperature coefficient α and the discount rate γ are set to 0.02 and 0.95, respectively. The softmax function is applied in the last layer of the policy to create discrete actions. Playback buffer size and batch size are set to 500 and 256 respectively. The weights of the 9 neural networks are randomly initialized and updated iteratively.

After setting all the parameters, the agent starts maximizing the cumulative reward. 5 shows the training process on September 5, 2019. The agent is performing poorly initially, as evidenced by the reward. However, as the number of iterations increased, the agent began to perform actions that provided higher rewards through trial and error. Finally achieved the maximum reward in about 30,000 times. Obtaining the maximum reward determines the corresponding optimal operating strategy. Table 3 lists the operating options of all machines at each stage, with "1" and "0" representing the working and idle modes respectively.

Figure 6 shows the total energy consumption of all machines under the optimal operating policy generated in the method according to the invention. Machines consume more energy when prices are low, consume less when prices are high, and avoid consuming energy during peak hours. In particular, the machine is consuming more energy during steps 1-15 and 19-24 and less during steps 16-18. In particular, most machines are reducing energy consumption to a minimum value because power prices are at their peak at levels 16 and 17. This not only relieves stress on the power grid, but also reduces energy costs for industrial consumers. According to the proposed DR plan to describe real-time battery production, the corresponding final battery production storage at each stage t is shown in FIG. 7 . The system finally calculates 500 battery modules (ie, production target) by executing the optimal schedule table shown in Table 3.

Figure 8 compares the total cost obtained from the MILP Gurobi solver and the CSAC algorithm according to the present invention. The MILP Gurobi solver uses a specific model that takes into account all information from the system, short-sighted measures to meet production targets and minimize energy costs as defined in Equations 8 and 9. However, the CSAC algorithm exhibits the self-learning ability to select different actions to maximize this reward, as described above. As shown in FIG. 8, the CRL algorithm method according to the present invention shows low performance in the initial training stage because it goes through trial and error. However, after experiencing more episodes, the agent adapts to the learning environment and adjusts its policy through a search and exploit mechanism. Finally, the optimal policy is obtained. Since the method according to the present invention is modelless and does not require expertise in complex energy management scenarios, it can provide a promising solution to solve complex industrial energy management problems.

In order to gain insight into the efficiency of the DR algorithm according to the present invention, simulations are performed considering electricity prices for three days from June 2nd to June 4th, 2019 obtained from ComEd. Figures 9 and 10 respectively show the convergence of cumulative rewards in the learning process and the corresponding total energy consumption of all machines under the DR scheme according to the present invention.

As shown in FIG. 10, similar energy consumption trends were observed between the previous day patterns, further proving the validity of the DR method according to the present invention. The final battery production and storage for 3 days at each step t is shown in FIG. As can be seen in Figure 11, the system produces 500 battery modules at the end stage each day.

Claims (4)

As a method performed in an energy management device of a discrete industrial manufacturing system,
In the time interval t state (S _t ), the action (task) (a _t ) is executed according to the policy (π), and the reward (R _t ) and cost (R ^c _t ) are obtained to move to the next state (S _t+1 ). _{The total time} ^step ₍ t ₌ An experience accumulation step of storing a training set of 1 to T);
Randomly sample a mini-batch from the training set, calculate a target label of a state-action value function and a state value function for the mini-batch, and updating the parameters of the action value function and the parameters of the state value function according to gradient descent so that an error between the value and the target value is minimized; Updating parameters of a policy function; Updating parameters of a target state value function; and a parameter updating step including updating a Lagrange multiplier;
Including a cumulative reward calculation step of calculating an accumulated reward by repeatedly executing an action according to a policy in the current state based on the parameter determined in the parameter update step, obtaining a reward, and moving to the next state,
Demand response management method of a discrete industrial manufacturing system to which constraint reinforcement learning is applied, characterized in that the parameter updating step is repeated until the cumulative compensation is maximized. According to claim 1,
The condition includes the electricity price, the energy consumption of each machine, and the storage amount of each buffer in the corresponding time interval. According to claim 1,
The action is a demand response management method of a discrete industrial manufacturing system applied with constraint reinforcement learning, characterized in that indicating the operation or idleness of each machine in the corresponding time interval. According to claim 1,
Demand response management method of a discrete industrial manufacturing system applied with constraint reinforcement learning, characterized in that when the cumulative compensation is maximized, the policy at that time is determined as the optimal operating policy.

Images