KR20230128190A - An intelligent protocol of multi-agent reinforcement learning- based autonomous model calibration for ASM-type model of membrane bioreactor - Google Patents

Abstract

The present invention relates to a sewage treatment plant model autonomous correction system using reinforcement learning, and a process operation multi-optimization system using the system thereof. More specifically, as a sewage treatment plant model correction system, the present invention relates to a multi-optimization system for a sewage treatment plant process operation using reinforcement learning comprising: a modeling part that calculates a sewage treatment plant model by imitating a sewage treatment plant; a modeling correction part that corrects the sewage treatment plant model by correcting a dynamic coefficient value of the sewage treatment plant model using artificial intelligence learning; and an optimization module that optimizes the operation control of sewage treatment processes based on an ASM model corrected by an autonomous correction system.

Description

Sewage treatment plant model autonomous calibration system using reinforcement learning, and process operation multi-optimization system using the system

The present invention relates to a sewage treatment plant model autonomous calibration system using reinforcement learning and a process operation multi-optimization system using the system.

Mathematical process modeling of sewage treatment processes has increased interest and use from the design stage of sewage treatment facilities to operation management. Mathematical modeling provides opportunities including the possibility of testing hypotheses about a system, the transfer of system knowledge between engineers, and the ability to predict the future state of a system. Among mathematical models of sewage treatment processes, the activated sludge model (ASM) developed by the International Water Association has been most commonly used to model biological sewage treatment processes. On the other hand, as ASM is applied to a specific sewage treatment plant using general variables and parameters, its complexity increases, making it difficult to apply ASM and time-consuming work. Therefore, model calibration is important for elucidating the behavior of biological processes in specific plants.

The kinetic parameters of ASM models differ greatly with respect to sewage treatment processes due to the variability of environmental properties and the complexity of the biological phenomena within the process. Therefore, several studies have presented calibration approaches to accurately describe sewage treatment processes. A calibration method for a full-scale sewage treatment plant (WWTP) is proposed by considering the ranking of sensitive parameters and determining values for simulating wastewater pollutants.

Experimental calibration for ASM parameters was performed to model the wastewater chemical oxygen demand (COD) for the sequencing batch reactor. A method for selecting possible parameter combinations was proposed to minimize the deviation between the simulated and measured substrate concentrations of the WWTP. Although these studies presented notable calibration results, they were based on steady-state values of calibrated dynamic parameters. Most WWTPs are subject to significant seasonality and dynamics that can influence changes in biological metabolism. This is because the changed temperature or changed operating conditions change the rate of growth or decay of the biomass, increasing or decreasing pollutant removal efficiency. Therefore, when developing a reliable tool for an ASM calibration protocol, it is essential to consider the dynamic state conditions of the ASM and calibrate its parameters over time.

Considering the dynamic behavior of microorganisms, a dynamic calibration protocol was developed and validated in existing WWTP and membrane bioreactor (MBR) plants. Although these studies obtained reliable results by applying dynamic calibration methods, they were limited to revealing the interactions between calibrated parameters. Most biological behaviors in sewage treatment processes incorporate individual phenomena such as carbon oxidation, nitrification and denitrification. The dynamic parameters of ASM-type models affect not only one process, but also two or more biological processes. This is because biological phenomena are complex and interact with system components. Correlations between the calibrated dynamic parameters should therefore be considered in the calibration method.

Calibration of ASM-type models using mathematical and statistical methods is still considered as one of the bottlenecks in modeling research. This is because mathematical and statistical-based calibration can be time consuming and unstable due to the many parameters of ASM-type models. In contrast, artificial intelligence (AI) is an alternative solution in wastewater treatment processes that provides powerful and scalable insights to solve complex and high-level problems. Multi-Agent Reinforcement Learning (MARL) is a powerful AI algorithm that provides reliable and feasible multi-solutions, but MARL algorithms have rarely been used in wastewater treatment research.

In addition, sewage treatment plants (WWTPs) have complex and energy-intensive systems to continuously mitigate the environmental pollution of wastewater caused by human activities. In addition, discharge standards and regulations for sewage are becoming more stringent worldwide to protect the environment from harmful sewage discharge. Therefore, the current challenge in wastewater treatment is to provide solutions to ensure operational energy cost-efficiency and desired wastewater quality. In terms of energy saving and environmental protection, process optimization of WWTP has recently attracted attention.

Several studies have been reported in recent years on the application of optimization strategies to WWTPs to maximize operational efficiency and balance energy consumption and sustainable sewage quality. A multi-objective genetic algorithm was used to trade off aeration energy and wastewater quality based on a sewage benchmark simulation model (BSM). An interactive multi-target optimization tool is proposed to determine the chemical dosage rate and oxygen concentration in the reactor. Treatment costs and wastewater quality indices were optimized, and backpropagation algorithms were used to identify relationships between optimization targets and determinants. An integrated superstructure method combining conceptual and mathematical programming has been proposed to design and operate a sustainable WWTP. Although these studies have shown remarkable optimization performance, mathematical algorithms require non-linear differential and algebraic equations, which may limit their ability to account for uncertainties while solving optimization problems.

WWTPs are complex systems with non-linear properties with respect to biological-chemical-physical mechanisms. Also, dynamic and non-linear inflow conditions can be a trap for reliable operation of WWTPs. 15 shows the nonlinear characteristics of the influent measured at the Y-city WWTP. It can be seen that the measured influent components exhibit distinct non-linearity in which the concentrations of the influent components are widely scattered (Fig. 15(a) and (b)). The skewness values of influent contaminants were positive values showing a biased distribution as shown in FIG. 15(c). The tails of the influent pollutant distribution diverged from normality indicating a non-linear nature. Therefore, data-driven or model-free algorithms can be an alternative solution for optimizing WWTPs corresponding to non-linear influent properties.

Reinforcement learning (RL) has received great attention for optimizing WWTP through its ability to solve high-dimensional and nonlinear problems. Hernandez-Del-Olmo et al (2012) applied model-free RL to a BSM1-based WWTP to optimize the dissolved oxygen (DO) concentration, thereby simultaneously reducing aeration energy and effluent ammonia concentration. Seo et al (2021) proposed an optimization strategy based on PPO (Proximal Policy Optimization) for the pumping system of WWTP to reduce energy consumption. Although these studies have used RL, they have been limited to solving dynamic multi-objective problems. Most WWTPs are not driven by one operating variable, but by more influential operating variables. Recently, Chen et al (2021) used a multi-agent deep deterministic policy gradient (MADDPG) algorithm to optimize DO concentrations and chemical amounts in a WWTP. The MADDPG algorithm used showed improved performance in optimization, but the MADDPG algorithm risks solving the multi-agent credit allocation problem. In this problem, it is difficult to calculate the agent's contribution to solving the problem in a collaborative environment. In this context, QMIX and the two-level attention network (G2ANet) algorithm-based game abstraction mechanism are advanced multi-agent reinforcement learning (MARL) techniques that provide optimal solutions while maximizing effective self-learning-based task completion success.

Korean Registered Patent No. 10-1927503 Korean registered patent 10-1629240 Republic of Korea Patent No. 10-2041326 Republic of Korea Patent Publication 10-2021-0109161

Therefore, the present invention has been made to solve the above conventional problems, and according to an embodiment of the present invention, an intelligent system that corrects an activated sludge model (ASM) model of a sewage treatment plant using multi-agent reinforcement learning artificial intelligence technology. Its purpose is to provide a model calibration protocol and a multi-optimization system for a process operation strategy using the same.

According to an embodiment of the present invention, an intelligent model calibration protocol for calibrating an ASM model of a sewage treatment plant using multi-agent reinforcement learning artificial intelligence technology, and a process operation strategy multiple optimization system using the same, biological microorganisms for each market value of the sewage treatment plant Reinforcement learning that can improve the environmental performance and economic feasibility of the sewage treatment process by dynamically correcting the process simulation ASM mathematical model according to the change in characteristics and optimizing the aeration intensity, external carbon source injection amount, and sludge circulation flow rate among the operating set values of the process The purpose is to provide a sewage treatment plant model autonomous correction system and a multi-optimization system for process operation using the system.

And according to an embodiment of the present invention, modeling errors can be reduced by 87% and 52% respectively in simulating effluent chemical oxygen demand (COD) and total nitrogen (TN) by verifying using domestic sewage treatment plant data, In addition, as a result of process operation optimization using ASM calibrated through an intelligent model calibration protocol, a sewage treatment plant model autonomous calibration system using reinforcement learning that can improve aeration energy by 25% and effluent water quality by 7%, and a process using the system Its purpose is to provide an operation multi-optimization system.

In addition, according to the embodiment of the present invention, it is expected that it will be applied to various sewage treatment plant operation fields by presenting an accurate process simulation and an economical and environmentally friendly process operation strategy for domestic and foreign sewage treatment plants. For commercialization, we provide a sewage treatment plant model autonomous correction system using reinforcement learning, which can be expected to be used for energy and eco-friendliness improvement at domestic and foreign sewage treatment plants and environmental system companies, and a process operation multi-optimization system using the system. There is a purpose.

On the other hand, the technical problems to be achieved in the present invention are not limited to the above-mentioned technical problems, and other technical problems that are not mentioned will become clear to those skilled in the art from the description below. You will be able to understand.

A first object of the present invention is a sewage treatment plant model correction system, comprising: a modeling unit for calculating a sewage treatment plant model by simulating the sewage treatment plant; And a modeling correction unit for correcting the sewage treatment plant model by correcting the dynamic coefficient value of the sewage treatment plant model using artificial intelligence learning. there is.

And the sewage treatment plant model may be characterized in that it is an activated sludge model (ASM).

In addition, the artificial intelligence learning may be characterized as multi-agent reinforcement learning artificial intelligence.

And the multi-agent reinforcement learning artificial intelligence may be characterized in that it is based on the G2ANet algorithm.

A second object of the present invention is a method for correcting a sewage treatment plant model, comprising: calculating, by a modeling unit, a sewage treatment plant activated sludge model (ASM) by simulating the sewage treatment plant; It can be achieved as an autonomous correction method for a sewage treatment plant model using reinforcement learning, comprising: a modeling correction unit correcting the dynamic coefficient value of the sewage treatment plant model using artificial intelligence learning to generate a calibrated ASM model. there is.

In addition, artificial intelligence learning may be multi-agent reinforcement learning artificial intelligence, and the multi-agent reinforcement learning artificial intelligence may be characterized as being based on the G2ANet algorithm.

A third object of the present invention is a sewage treatment plant operation optimization system, comprising: an autonomous correction system according to the first object mentioned above; and an optimization module for optimizing operation control of the sewage treatment process based on the ASM model corrected by the autonomous correction system.

And the optimization module may be characterized by optimizing process operation based on multi-agent reinforcement learning artificial intelligence.

In addition, the optimization module may be characterized by optimizing the aeration intensity, the external carbon source injection amount, and the sludge circulation flow rate.

A fourth object of the present invention is a sewage treatment plant process operation multi-optimization method, wherein the modeling unit simulates the sewage treatment plant to calculate a sewage treatment plant activated sludge model (ASM); generating a calibrated ASM model by a modeling correction unit correcting the dynamic coefficient value of the sewage treatment plant model using artificial intelligence learning; and an optimization module optimizing the aeration intensity, external carbon source injection amount, and sludge circulation flow rate of sewage treatment process operation through multi-agent reinforcement learning artificial intelligence based on the calibrated ASM model. It can be achieved as a multi-optimization method for operating the sewage treatment plant using

According to an embodiment of the present invention, an intelligent model correction protocol for correcting an activated sludge model (ASM) model of a sewage treatment plant using multi-agent reinforcement learning artificial intelligence technology, and a process operation strategy multiple optimization system using the same can be provided. .

According to the sewage treatment plant model autonomous correction system using reinforcement learning and the process operation multi-optimization system using the system according to an embodiment of the present invention, intelligent correction of the ASM model of the sewage treatment plant using multi-agent reinforcement learning artificial intelligence technology Through a model calibration protocol and a multi-optimization system for process operation strategies using the same, the process simulation ASM mathematical model is dynamically calibrated according to changes in biological and microbial characteristics by market value of the sewage treatment plant, and aeration intensity and external carbon source injection amount among the operating set values of the process In addition, it has the effect of improving the environmental and economic feasibility of the sewage treatment process by optimizing the sludge circulation flow rate.

In addition, according to the sewage treatment plant model autonomous correction system using reinforcement learning and the process operation multi-optimization system using the system according to an embodiment of the present invention, the effluent chemical oxygen demand (COD) and total In simulating nitrogen (TN), modeling errors can be reduced by 87% and 52%, respectively, and as a result of process operation optimization using ASM calibrated through an intelligent model calibration protocol, aeration energy can be reduced by 25% and effluent water quality reduced by 7 % has the potential to improve.

In addition, according to the sewage treatment plant model autonomous correction system using reinforcement learning and the process operation multi-optimization system using the system according to an embodiment of the present invention, accurate process simulation and economical and eco-friendly process operation strategies are provided for domestic and foreign sewage treatment plants. By presenting it, it is expected that it will be applied to various sewage treatment plant operation fields. As for commercialization, it can be expected to be used in domestic and foreign sewage treatment plants and environmental system companies to improve energy and eco-friendliness.

On the other hand, the effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and together with the detailed description of the invention serve to further understand the technical idea of the present invention, the present invention is limited only to those described in the drawings. and should not be interpreted.
1 is a block diagram of a sewage treatment plant model autonomous correction system using reinforcement learning and a process operation multi-optimization system using the system according to an embodiment of the present invention;
2 is a MARL-based self-correction framework of an ASM-type model for MBR plants;
3 is a scheme describing initialization and calibration periods as part of a calibration protocol for the MARL algorithm modified by Guo et al;
4 is a training plan for multi-agent reinforcement learning modified by Wong et al;
Figure 5 shows the structure of QMIX: (a) mixed network structure, (b) overall QMIX architecture, (c) agent network structure
6 is a calculation procedure of two-step attention and graph embedding;
7 is a detailed structure of G2ANet including two-level attention using Central-V;
Figure 8 shows daily measurement data from 2011 to 2018: influent and effluent (a) COD and (b) TN concentrations, (c) MLSS concentrations of four reactors, (d) influent temperature
Figure 9 is the average compensation change of QMIX and G2ANet-based autonomous calibration system
Figure 10 shows a three-dimensional t-SNE embedding of calibrated kinetic parameters assigned by the G2ANet algorithm (points are colored blue to red according to the number of training epochs).
11 is a comparison of the G2ANet-based calibration model for (a) wastewater COD and (b) wastewater TN and the basic model of the MBR plant;
Figure 12 is a histogram and kernel density of kinetic parameter values corrected for 2018 by the G2ANet algorithm (black dotted line represents the default values of kinetic parameters);
Figure 13 shows the results of the G2ANet-based calibration model: (a) wastewater COD and TN and (b) calibration parameters for February, (c) wastewater COD and TN and (b) calibration parameters for July;
Figure 14 shows the change in the biological process rate for February and July in (a) anoxic reactor and (b) aerobic reactor;
15 shows nonlinear characteristics of influent TSS, COD, and TN concentrations: (a) measured influent contaminant concentrations (b) 3-dimensional plots of influent over time and (c) influent skewness values and bias histograms and probability plots;
16 is a graphical framework of a MARL-based autonomous multi-trajectory search system;
17 shows the layout and influent load of the MBR plant;
18 shows changes in influent COD and TN concentrations: (a) monthly and (b) daily trend
19 is a graphical representation of a multi-agent reinforcement learning algorithm-based multi-task trajectory retrieval system;
20 shows influent data generation results: (a) influent scenario cluster for COD and TN composition ratio using k-means clustering and the procedure, and (b) daily influent data generation according to the scenario;
21 is a linear compensation function of the MARL algorithm using a linear combination of figures of merit;
22 shows changes in average reward values according to the number of epochs: (a) QMIX and G2ANet comparison, (b) QMIX, (c) G2ANet comparison,
23 shows optimal operating trajectories of DO concentration, external recycling rate and external carbon flux by the G2ANet algorithm in (a) low, (b) normal and (c) high inflow scenarios;
24 is a k-means clustering-based Voronoi diagram for classifying influent condition characteristics in 2018 of a target MBR plant: (a) influent COD and TN concentrations and (b) daily and (c) monthly influent conditions;
25 shows the self-optimization performance of the G2ANet-based trajectory search system in February (high influent condition): (a) COD and TN concentration changes, (b) optimal trajectory, (c) figure of merit, and (d) effluent component profile according to quality limit,
26 shows self-optimization performance of the G2ANet-based trajectory search system in July (low inflow condition): (a) COD and TN concentration changes, (b) optimal trajectory, (c) figure of merit, and (d) outflow component profile according to quality limit

The above objects, other objects, features and advantages of the present invention will be easily understood through the following preferred embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed content will be thorough and complete and the spirit of the present invention will be sufficiently conveyed to those skilled in the art.

In this specification, when an element is referred to as being on another element, it means that it may be directly formed on the other element or a third element may be interposed therebetween. Also, in the drawings, the thickness of components is exaggerated for effective description of technical content.

Embodiments described in this specification will be described with reference to cross-sectional views and/or plan views, which are ideal exemplary views of the present invention. In the drawings, the thicknesses of films and regions are exaggerated for effective explanation of technical content. Accordingly, the shape of the illustrated drawings may be modified due to manufacturing techniques and/or tolerances. Therefore, embodiments of the present invention are not limited to the specific shape shown, but also include changes in the shape generated according to the manufacturing process. For example, a region shown at right angles may be rounded or have a predetermined curvature. Accordingly, the regions illustrated in the drawings have attributes, and the shapes of the regions illustrated in the drawings are intended to illustrate a specific shape of a region of a device and are not intended to limit the scope of the invention. Although terms such as first and second are used to describe various elements in various embodiments of the present specification, these elements should not be limited by these terms. These terms are only used to distinguish one component from another. Embodiments described and illustrated herein also include complementary embodiments thereof.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. The terms 'comprises' and/or 'comprising' used in the specification do not exclude the presence or addition of one or more other elements.

In describing the specific embodiments below, several specific contents are prepared to more specifically describe the invention and aid understanding. However, readers who have knowledge in this field to the extent that they can understand the present invention can recognize that it can be used without these various specific details. In some cases, it is mentioned in advance that parts that are commonly known in describing the invention and are not greatly related to the invention are not described in order to prevent confusion for no particular reason in explaining the present invention.

Hereinafter, the configuration and functions of a sewage treatment plant model autonomous correction system using reinforcement learning and a process operation multi-optimization system using the system according to an embodiment of the present invention will be described. 1 is a block diagram of a sewage treatment plant model autonomous correction system using reinforcement learning and a process operation multi-optimization system using the system according to an embodiment of the present invention.

First, a sewage treatment plant model autonomous calibration method and system using reinforcement learning according to an embodiment of the present invention will be described.

In the embodiment of the present invention, an intelligent protocol for calibrating an ASM-type model is proposed and biological motion parameters are calibrated by utilizing a multi-agent reinforcement learning (MARL) algorithm. First, we selected a subset of kinetic parameters among 16 kinetic parameters. Second, the structure of the MARL algorithm was chosen to propose the kinetic parameter values. In the embodiment of the present invention, two MARL algorithms, QMIX and game abstraction mechanism based on two-level attentional network (G2ANet), were compared. Based on a selected subset of kinetic parameters, MARL's agents were assigned to calibrate the kinetic parameters. After training the weights of the MARL algorithm, MARL tests were performed on newly measured data that were not used in the algorithm training procedure. MARL-based calibrated kinetic parameters were evaluated to ensure reliable and feasible calibration performance by interpreting biological phenomena.

The self-calibration framework of the ASM model according to an embodiment of the present invention is shown as a block in FIG. 2 . The approach of the present invention consists of two parts: (1) selecting the structure of the MARL algorithm to be used in the MBR plant and (2) proposing the optimal dynamic parameter values of the ASM-SMP model considering the influent and operational data of the target. It consists of

First, the structure of the MARL algorithm was selected as shown in step 1 of FIG. 2, considering the measured influent and operation data set of the MBR plant. Parameters to be calibrated were selected to reflect the biological information of the target sewage treatment plant. It is usually required as a subset containing a small number of kinetic and stoichiometric parameters. Then, in order to increase the accuracy of the calibrated model, the MARL structure was selected using the status, observation, behavior, and reward according to the operation information of the sewage treatment plant. In addition, the state variables of the influent were estimated using the influent information.

Second, as shown in step 2 of FIG. 2, MARL-based self-calibration was trained and applied to determine the optimal value of model parameters. Influent and operational data were collected from 2011 to 2018 at a full-scale MBR plant located in Y-city. Seven years of data from 2011 to 2017 were used as the training set, and the automatic calibration was tested using the 2018 measurement data. In the embodiment of the present invention, two MARL algorithms, QMIX and a two-level attention network (G2ANet) based game abstraction mechanism are used. Given the fluctuating nature of MBR plant influents, suggesting daily calibrated parameters can be a pitfall that increases uncertainty. Therefore, the MARL algorithm suggested calibrated parameter values every 5 days. Figure 2 shows that the MARL-based calibration model was validated against the baseline model for the measured influent and effluent data sets.

First, a method for selecting a subset of dynamic parameters for calibration will be described.

Selection of a subset of parameters is a critical procedure to identify parameters that can affect the process behavior modeled by ASM-SMP. This procedure should be performed prior to optimizing the search for optimal parameter values. The influence of parameters can be influenced by influent characteristics and operating conditions. Parameter selection creates a subset containing a small number of parameters from among the many parameters of an ASM that can be identified. Influential parameters can be selected in two ways ((1) mathematical models to identify the sensitivity of parameters and (2) prior experience and expertise). It can be seen that the influential parameter subsets reported in previous studies can be useful for qualitative modeling. However, a subset of previous studies cannot completely guarantee accurate modeling results for other cases.

Therefore, in the embodiment of the present invention, influential parameters for correction were selected based on existing research on general sewage treatment modeling. It is also essential to select special parameters for the MBR plant. Therefore, additional parameters influencing the biological nutrient removal process of the MBR plant were selected. For the stoichiometry of the model, the stoichiometric parameters should be measured through respiratory system-based experiments. Since stoichiometric parameters in the Monod equations of several biological processes have been consistently identified in a series of ASMs, the examples of the present invention utilize well-established stoichiometric parameters.

[Table 1] Summary of calibrated kinetic parameters of the MBR model in references and the present invention

Table 1 summarizes the review of the calibrated kinetic parameters in the modeling MBR plant based on the ASM series model. We calibrated primarily for heterotrophic and autotrophic growth and decay, and general kinetic parameters associated with the formation of SMPs. The parameters are directly related to heterotrophic and autotrophic populations and nitrification processes. In conclusion, eight kinetic parameters ( ) was selected.

In order to improve the accuracy of ASM-SMP, MARL-based automatic calibration was performed on eight selected kinetic parameters. Other parameters not selected for calibration were provided by default and literature values were found to be acceptable for domestic WWTP and MBR.

Next, the temperature-dependent parameters are explained. Biological reaction rates vary with temperature change similarly to chemical reactions because of the concept of activation energy. Since biological reactions depend on temperature, the Arrhenius equation can be used for biological reactions. Therefore, the temperature dependence related to growth, decay and other rates and constant dynamics of the ASM-SMP model can be formulated according to the Arrhenius equation. The form commonly used in the field of sewage treatment research of the Arrhenius equation is expressed as Equation 1.

[Equation 1]

where k _T is the kinetic parameter at temperature T, k _T0 is the kinetic parameter at the reference temperature (20 °C), and θ is an Arrhenius constant, usually between 1.04 and 1.09.

Table 2 shows a summary of the temperature dependent kinetic parameters and Arrhenius coefficient values. The seven kinetic parameters representing biological reaction rates included in the ASM-SMP model are: The parameter subsets of MARL-based calibration include Since it contains four kinetic parameters such as The remaining three kinetic parameters, including , were adjusted according to the change in influent temperature according to the Arrhenius equation of Equation 1. As a result, the hydrolysis rate, hydrolysis constant, and ammonification rate reflected the activity of various biological microorganisms according to the temperature change.

[Table 2] - Temperature dependent kinetic parameters and Arrhenius coefficients

In an embodiment of the present invention, various ASM-SMP simulations were performed, each with a different sample set of influent and operational values sampled from the measurement data set of the target MBR plant. In each simulation, the MARL-based autonomic correction is trained to search for strategies to find appropriate kinetic parameters, including Each simulation using the calibrated parameters was compared to the measured data set and the error between calibrated and measured runoff could be calculated. Therefore, it is important to select a sampling data set to ensure accurate calibration performance.

In the embodiment of the present invention, the measured set is divided into a training set (from 2011 to 2017) and a test set (2018). The autonomous calibration algorithm was trained using data from the training set, randomly sampling the data set and running the calibration during the sampled calibration period. The sampled training period was 20 days consisting of an initialization period of 5 days and a calibration period of 15 days. Initialization of kinetic parameters was performed during each simulation. This is necessary because the proposed kinetic parameters of the model may affect the initial conditions of the model at the start of different calibration periods. For example, low microbial activity in the cold season can reduce microbial growth and increase decay rate parameters. Initial values using growth and decay rates corrected for the cold season may have the effect of underestimating microbial activity when warm season data are used for correction.

Figure 3 shows the conceptual sampling and implementation of multi-agent reinforcement learning based calibration. One simulation period consists of two subsequent parts: an initialization period of 5 days (default values of the parameters are used) and a 15-day calibration period in which MARL adjusts the kinetic parameters. MARL-based calibration suggested kinetic parameter values every 5 days. Therefore, we proposed three sequential values for each kinetic parameter through autonomous calibration over a 15-day calibration period. The 5-day period for each calibrated parameter represents the size of the window. The window size can be chosen to strike a balance between computation time and the complexity of the target system. If the size of the window is large, information specific to data information may be lost, whereas if the size of the window is small, complexity may increase. A window size of 5 days was chosen to ensure accurate calibration results.

Randomly selecting a total of 20 days including the initialization and calibration period, the automatic calibration system changed the kinetic parameters from the default values of the initialization period as shown in Fig. 3. Training of the calibration system was not active in this initialization period, but the tuned kinetic parameters were suggested as an initial point in the calibration period. The autonomous system then changed the dynamic parameters at the adjusted values in the initialization period. At this point, the autonomous system has been trained to interface with the environment and update the weight values.

Next, self-correcting multi-agent reinforcement learning is described. In recent years, advanced applications of deep learning and reinforcement learning (RL) have been attempted in decision-making problems in hydrological research, such as sewage treatment processes. RL is a self-learning algorithm with unspecified output. It finds the optimal solution to a given problem using a reward policy and a trial-and-error mechanism. Advanced RL has recently been applied to solve multiple problems using one or more single agents systematically modeled as Multi-Agent RL (MARL). MARL addresses the problem of concurrent decision-making by multiple autonomous agents operating in a common environment. Each MARL agent aims to solve and optimize problems by interacting with the environment and other agents.

In an embodiment of the present invention, MARL was used to simultaneously present values of kinetic parameters. Figure 4 shows the structure of the MARL algorithm for calibrating the ASM-SMP model. In the embodiment of the present invention, centralized training and distributed execution are adopted in the MARL architecture. Agents can share their respective information during training, and distributed tasks are executed based on each (local) observation. In the embodiment of the present invention, the calibrated kinetic parameters were used as the actions of n agents and the influent and operating conditions were used as observations. The main algorithm is that an agent can receive external information from other agents' observations, rewards, and gradients. Agents execute policies distributedly, taking local observations into account. In part of centralized training, agents can be trained together using critics or Q-networks. In the embodiment of the present invention, influent and operating conditions were used as critics or Q-network conditions. This structure helps reduce non-stationarity and partial observability. This is because it stabilizes the agent's learning process even if other agents' policies change.

In the embodiment of the present invention, two MARL technologies are used: QMIX and G2ANet. QMIX is a value-based method of training decentralized policies in a centralized, end-to-end manner. G2ANet models the relationship between agents as a complete graph and proposes actions by a two-level attention network-based game abstraction mechanism.

QMIX has a centralized training and distributed execution structure. QMIX uses networks that compute collaborative values as complex, non-linear combinations of the values of each agent that condition only on local observations. QMIX leverages a hybrid network to collect single-agent local value features and includes global state in the training and learning process. This results in better algorithm performance and stable consistency between centralized and distributed policies.

QMIX maintains consistency between deterministic and greedy decentralization policies and centralization policies based on the optimal joint action value function. Therefore, the joint Q function and the local Q function satisfy Equation 3.

[Equation 3]

here Is find the argument u that gives the maximum Q-value in the function, is the centralized action-value Q-function conditioning for joint action-observation records τ and joint action u . The Q-function calculates the expected reward for an action taken in a given state. is the total It is an individual value function for each agent a of an agent, and is conditional only on individual behavioral observation records τ ^a and u ^a . Equation 3 allows each agent to participate singly in the distributed execution by selecting a greed action according to Q _a .

Equation 3 can be generalized to a larger group of monotonic functions. Monotonicity is defined as a constraint function between each of Q _tot and Q _a as expressed by Equation 4. Each update of the local agent Q _a increments the co-agent Q _tot .

[Equation 4]

Figure 5 shows a QMIX architecture comprising a set of agent networks, mixed networks and hypernetworks. is an individual value function of a single agent network a, and uses DRQN, which receives the current individual observation value and previous operation as inputs each time, as shown in FIG. DRQN uses gated recurrent units (GRUs) to transmit hidden state values to facilitate learning on longer time scales. According to the epsilon-schedule with probability 1-ε, the output of DRQN is According to the policy π to choose the maximum value of am.

The mixed network shown in Fig. 5(a) receives the output of individual agents and mix monotonously is a feedforward neural network that generates HyperNetwork W ₁ and W ₂ receives state S as an input and generates weights for each layer of the mixed network. The hypernetwork consists of two fully connected layers using Rectified Linear Units (ReLUs).

G2ANet automatically learns the interaction relationship between agents through game abstraction-based end-to-end model design. All agents are represented as graph structures and grouped into multi-game abstraction algorithms. This multi-game abstraction algorithm consists of a two-level attention mechanism, which is Hard Attention and Soft Attention. It can also be combined with a graph neural network (GNN) to obtain joint encoding for displaying contributions from other agents. The calculation procedure is shown in FIG. 6 .

(1) Hard Attention: Hard Attention selects important elements according to the sampling method and discards the rest. Cut other agents unrelated to the target agent.

(2) Soft Attention: Soft Attention calculates the importance distribution of elements. Importance weights (relationships) between agents are learned.

(3) GNN (graph embedding): GNN calculates the contribution of other agents and outputs information of other agents including coordination relationships.

7 shows the detailed structure of G2ANet. Agent i encodes the observation o _i ^t and the action a _i ^t at each time step t into a feature vector e _i ^t embedded by MLP and GRU. Hard Attention calculates the hard-attention weight W _h ^i,j representing the connection between agent i and agent j as shown in FIG. 7(a).

Through the hard Attention structure, each agent plays a different role to other agents. Here Bi-GRU and gumble-softmax are used to compute the backpropagation of the gradient. 7(b) shows soft attention learning the soft attention weight W _s ^i,j between agents. It consists of a query-key-value structure that calculates the embedding of the agent. The GNN calculates the joint vector encoding to define the contributions of all other agents to agent i as shown in Fig. 7(c). Finally, as shown in Fig. 7(d), a centralized critic and individual actors (Central-V) are used to calculate the Q-value. Central-V is a policy-dependent actor-critic algorithm that conditions states rather than individual observation records.

An embodiment of the present invention was performed at a full-scale MBR plant located in Y-city. From 2011 to 2018, operational data including influent and mixed liquor suspended solids (MLSS) concentrations from the four reactors were collected daily as shown in Figure 8. The average influent and effluent COD concentrations were 331.92 and 14.05 mg/L, respectively. The COD removal efficiency was 95.76%. For TN, the influent and effluent concentrations are 44.23 and 6.24 mg/L, respectively. Therefore, the TN removal efficiency was 85.90%. The MLSS concentration and average concentration of each reactor are shown in FIG. 8(c). Anoxic, aerobic, and membrane reactors have relatively high MLSS concentrations to sustain nitrification and denitrification processes, as in typical MBR processes. 8(d) shows the change in influent wastewater temperature over time. There is a clear trend for water temperatures to rise in the hot season (June to September) and decrease in the cold season (November to February). Therefore, it is important to reflect the operating characteristics as well as the seasonal and temperature dependent characteristics of the target plant through accurate and dynamic calibration.

Eight agents each representing a proposal of value were assigned. In an embodiment of the present invention, agents 1 to 4 are each A specific number corresponding to was given to each agent. Agents 1 to 4 are related to sewage COD concentrations. This is because the parameters represent heterotrophs and BAPs that affect heterotrophic growth and lysis. On the other hand, agents 5 to 8 are respectively showed the correction of These parameters represent TN clearance mechanisms and effects on autotrophic growth and lysis. The detailed structure of the MARL agent is as follows.

- State and observations :

During centralized learning, the global state available only to the agent contains information about influent and biological properties. Specifically, the state vectors include COD, TN, NO, NH, SS, Q, and the influent temperatures and MLSS concentrations of the four reactors for the current and previous days (d and d-1 days). Agents 1 to 8 received regional observations according to target kinetic parameters related to COD and TN removal. Agents 1 to 4 used influent S _i (soluble inert COD), S _s (readily biodegradable COD), X _i (inert suspended COD), X _s (slowly biodegradable COD), COD concentration and temperature as observations. Influent S _NO (soluble nitrogen nitrate), S _NH (soluble ammonia nitrogen), S _ND (soluble biodegradable organic nitrogen), X _ND (slowly biodegradable organic nitrogen), TN concentrations and temperatures were observed for 5 to 8 agents. .

-Actions:

The set of individual actions allowed to the agent consist of -20%, -10%, 0%, +10% and +20%. Percentage represents the magnitude of change in each motion parameter. for example, +20% of is +1.48, which is calculated by multiplying +20% by the difference between the lower (0.6) and upper (8) bounds of the kinetic parameter. Therefore, it was calculated as (8-0.6) × 0.2 = 1.48.

- Rewards :

The overall goal of calibration is to minimize simulation errors. Therefore, a numerical compensation function was assigned to MARL as shown in Equation 5.1.

[Equation 5.1]

where r(d) is the compensation value on day d, varying from -1 to +1, and MAPE _pollutant (d) is the average absolute percentage error of each contaminant on day dl. This normalized the reward so that it can help the agent navigate the task efficiently and promote better calibration performance. MAPE is expressed as Equation 5.2.

[Equation 5.2]

where y is the measured wastewater COD and TN concentrations are the sewage COD and TN concentrations modeled through the calibrated ASM-SMP model. The lower the MAPE value, the higher the accuracy of the model, so if the calibrated model has accurate performance for wastewater COD and TN modeling, the MAPE value decreases and the compensation increases as shown in Equation 5.1.

The structure of MARL for calibrating the kinetic parameters of the aforementioned ASM-SMP model is summarized in Table 3.

[Table 3] Structure of QMIX and G2ANet for kinetic parameter calibration of ASM-SMP

The automatic calibration system's MARL algorithm was trained on a random sampling of past measured influent and operational data. It is important to reliably propose calibrated kinetic parameters through a trained MALR for high and robust performance in response to anomalous changes in influent. In the embodiment of the present invention, MARL training procedures were compared according to the number of epochs in order to verify the stable performance of MARL. Figure 9 shows the average compensation value according to the cumulative number of epochs of QMIX and G2ANet-based autonomous calibration system. The number of epochs represents the number of times we trained by sampling the initialization and calibration period, as mentioned earlier. A total of 20,000 random samplings were performed on 7 years of data to train the MARL algorithm. The average reward value per 100 epochs is shown to explicitly compare the increasing trend of the reward value of the two MARL algorithms.

QMIX was unstable in reward change according to training epoch. At the beginning of algorithm training, around 2,000 epochs, QMIX was trained quickly and received high average reward values. This is because QMIX uses memory to store the algorithm's past state-action-reward information. QMIX is an out-of-policy reinforcement learning algorithm that uses data from a playback buffer of old experiences. However, QMIX showed inconsistent behavior throughout the entire training procedure despite changing the action selection method from exploration to utilization by reducing the probability value. The e probability represents epsilon-greed, which selects whether to search or utilize with a low e probability value and a high e probability value, respectively.

As shown in FIG. 9, the reward of G2ANet tended to increase stably as training progressed. The central-V of the G2ANet-based self-correcting system is an on-policy algorithm, a policy gradient method in which the action-value function is suitable for the current policy and most of them are improved by greedy according to the action-values (O'). G2ANet obtained action probabilities directly from the state and observation of the target MBR plant, and found a way to maximize future expected rewards. As a result, the G2ANet algorithm showed more stable reward values over the entire training period compared to QMIX. During training, a stable converged reward value was observed after 15,000 epochs. The trained weights of G2ANet were used to calibrate the updated motion parameters after 15,000 epochs. In the embodiment of the present invention, G2ANet was selected instead of QMIX to calibrate the motion parameters in consideration of adaptability and stable function.

In an embodiment of the present invention, a G2ANet agent that underpins the calibration performance of multiple agents in a training procedure using visualization of high-dimensional and non-linear dimensional data, called t-distributed stochastic neighbor embedding (t-SNE). 8 types of motion changes were investigated. t-SNE uses the heavy-tailed Student-t distribution method to reduce high-dimensional and non-linear dimensional data at different scales to the center of the map (also called embedding).

Figure 10 shows the results of the t-SNE algorithm that maps similar calibrated dynamical parameters of G2ANet to nearby points. The t-SNE algorithm produced similar embeddings for the kinetic parameters in terms of number of training epochs. As shown in FIGS. 9 and 10 according to the reward increase trend, the training procedure can be largely divided into three stages: random action, training progress, and complete training. Embedding by t-SNE was clustered into three training procedures as shown in Fig. 10(a).

The results of t-SNE showed an obvious trend in which the embedded points progressively progressed along the positive t-SNE1 direction. It can be said that the G2ANet algorithm has learned to increase the accuracy of the model by suggesting appropriate motion parameters throughout the training procedure.

10(b) shows the embedding points by t-SNE in the random operation of the G2ANet training procedure. The points are widely scattered in 3-dimensional t-SNE throughout the random action procedure. This is because the actions of the 8 agents were determined by G2ANet's initial policy of exploration and not exploitation. Therefore, as shown on the left of FIG. 10, the compensation value of the corresponding step also changed. As the training procedure progressed, the embedding points moved in the positive t-SNE 1 direction. During training, the embedding points were mapped to the surrounding space of the 3D t-SNE as shown in FIG. 10(c) compared to the random action procedure. During training, G2ANet encountered randomly selected conditions such as influent and operational information of the MBR plant, and greedily updated the refined policy. At this stage, G2ANet refined the policy and updated the weights in the direction of the gradient of performance.

The embedding points of the complete training are shown in Fig. 10(d). Here, G2ANet validated the performance on the test set using the weights at that stage to evaluate the performance. The mapped points were concentrated in the entire training compared to the random behavior of the training procedure and the training progress. This indicates that the eight agents in G2ANet agreed and suggested consistent kinetic parameters, despite the fact that the state of the MBR plant was randomly selected. Therefore, G2ANet had the ability to robustly calibrate the ASM-SMP model even when complex inlet conditions of MBR plants were observed. As a result, in the present embodiment, G2ANet trained after 15,000 training epochs was used as the main calibration algorithm to propose appropriate motion parameters to reflect the biological process of the MBR plant.

Verification was conducted using a different data set collected in 2018 from the target MBR plant. Figure 11 shows the 2018 calibrated model proposed by the proposed G2ANet algorithm in relation to wastewater COD and TN concentrations. Compared to the base model using default values for all parameters, the G2ANet-based automatic calibration significantly reduced errors. As shown in Fig. 11(a), the remarkable calibration performance of the G2ANet algorithm was observed to simulate the sewage COD concentration. The basic model has limitations in simulating wastewater COD concentrations correctly. The deviation between the measured sewage COD concentration and the simulated sewage COD concentration by the basic model was 21.48 mg/L. On the other hand, the G2ANet-based calibration model reduced the deviation to 2.78 mg/L. Therefore, when G2ANet was used to calibrate the ASM-SMP model, the variance of the wastewater COD concentration simulation was reduced by 87.05%.

The improvement of effluent TN concentration simulation by G2ANet was confirmed in Fig. 11(b). The base model showed a relatively low deviation when simulating the sewage TN concentration as 4.99 mg/L compared to the sewage COD concentration. However, calibration was also required to simulate effluent TN concentrations. Because TN has a relatively high impact on wastewater quality, the effluent charge for discharging high effluent TN concentrations is high compared to other pollutants. In this context, accurate simulation of wastewater TN concentration is important to establish a monitoring-optimization-control strategy to improve the TN removal efficiency of sewage treatment processes. The G2ANet-based calibration model improved the simulation accuracy of effluent TN concentration compared to the basic model. The G2ANet algorithm reduced the simulation deviation from 4.99 mg/L in the basic model to 2.40 mg/L. Therefore, the calibrated model improved the simulation performance by reducing the deviation by 51.85%. As a result, the G2ANet-based autonomous calibration system showed accurate simulation performance in both COD and TN concentrations of wastewater from a full-scale MBR plant.

In addition, in the examples of the present invention, the calibration performance of G2ANet was quantitatively evaluated as summarized in Table 4. MAPE and RMSE (Root Mean Squared Error) were selected as numerical performance evaluators to evaluate model adaptability to measurement data. G2ANet showed lower MAPE and RMSE for both sewage COD and TN concentrations. That is, the model calibrated using the proposed calibration protocol outperformed the base model. It can be seen that the G2ANet algorithm performed robust and accurate corrections to the newly provided state information for 2018, despite the algorithm being trained using only 2011 to 2017 data. Therefore, it can be inferred that the G2ANet algorithm can be used for model calibration on a newly measured data set without any additional training procedure.

Analysis of the kinetic parameters calibrated during the validation period was performed as shown in FIG. 12 . The histogram and kernel density distributions each represent several classes of data and a smooth empirical probability density function based on the discrete location of all data. Figure 12(a) shows the distribution of kinetic parameters corrected by the G2ANet algorithm to reflect COD removal. The black dotted lines are means the default value of

Heterotrophic maximum growth rate ( ) was increased and the heterotrophic decay rate (b _h ) was corrected to a value lower than the default value. In addition, the reduced fraction of BAP produced during cell lysis (f _BAP ) and the increased rate of hydrolysis of BAP (K _h,BAP ) indicate a lower production of BAP, indicating lower production of both heterotrophs and autotrophs. affected dissolution. These results indicated that the proposed correction system increased the activity of heterotrophs by increasing the growth rate and reducing the decay rate. Therefore, the COD removal efficiency of the target MBR plant can be reflected through the G2ANet algorithm.

12(b) shows an analysis of calibrated kinetic parameters related to biological TN clearance mechanisms. The maximum growth rate of an autotroph reflecting the biological activity of the autotroph ) and decreased the heterotroph decay rate (b _a ). increased activated the aerobic growth of autotrophs, and reduction alleviated the lysis of autotrophs, increasing the nitrification mechanism. In addition, both the fraction of UAP generated during cell growth (f _UAP ) and the rate of hydrolysis of UAP (K _{h, UAP} ) were decreased compared to the default values. They accelerated denitrification and thus contributed to the reduction of nitrate concentration. The high TN removal efficiency of the target MBR plant was taken into account by the proposed calibration system. Therefore, the reflected TN removal efficiency decreased the effluent TN concentration in the simulation as shown in Fig. 11(b) and was simulated satisfactorily compared to the basic model. As a result, the G2ANet-based calibration system suggested satisfactory values of eight kinetic parameters to simulate the COD and TN removal process, reflecting the biological mechanisms of the target plants.

Detailed changes in kinetic parameters calibrated to reflect COD and TN removal mechanisms under various operating conditions are shown in Figure 13. As shown in Fig. 13(a) and (c), the simulated wastewater COD and TN concentrations in February and July were in good agreement with the measured influent data. It can be inferred that the proposed G2ANet-based correction system reflects the operating conditions and seasonal characteristics of the target plant. Several environmental factors can affect a wastewater treatment simulation, with temperature being an important factor. An increase in temperature causes an increase in the value of the rate coefficient. Therefore, the values of biological kinetic parameters must be determined at the temperature of the target plant. However, most temperature correction factors have been developed for isolated treatment processes. Therefore, evaluation of the proposed calibration is important to verify the robustness of its application to new data sets by considering temperature-seasonal characteristics.

13 (b) and (d) verify the calibration performance of the G2ANet algorithm for February and July. In the overall case, the G2ANet algorithm increased the growth rate of heterotrophs and autotrophs over two months and reduced their decay rate. In particular, as the temperature decreases, the growth rate and decay rate increase and decrease, respectively, but the activity of microorganisms may decrease. Compared to FIG. 13 (d) showing the results of July, February, when the temperature was low, decreased the activities of heterotrophs and autotrophs. The maximum growth rate and decay rate of heterotrophs in February were similar to those in July. On the other hand, the peak autotrophic growth rate in February was lower than that in July. Autotrophic decay rates in February were corrected for higher values than in July. Therefore, considering the calibrated kinetic parameters, the microbial activity in the hot season (July) was higher than that in the cold season (February) in the simulation.

These results show the advantages of G2ANet-based autonomous calibration considering influent characteristics and operating conditions. The states of the G2ANet algorithm include influent temperature and MLSS concentration. Therefore, the self-calibration system according to the embodiment of the present invention not only reflected the biological microbial mechanism but also improved the model accuracy. These results show that the utilization of G2ANet-based calibration can be an effective solution for modeling and simulation of MBR plants.

In the embodiment of the present invention, the autonomous calibration system was further evaluated by analyzing the biological process rate in the ASM-SMP model to ensure reliability. Figure 14 shows the change in biological process rate calculated by calibrated kinetic parameters. The biological process rate of the anoxic reactor includes anoxic growth of heterotrophs, dissolution of heterotrophs, ammonification, and hydrolysis of substrates, as shown in FIG. 14(a). This is because the activity of heterotrophic microbial organisms mainly occurs in anoxic reactors. In addition, conversion of ammonia to amino acids occurs during heterotrophic and autotrophic biomass synthesis, which is corrected by the temperature-dependent variable k _a . Substrate hydrolysis is the degradation of slowly biodegradable substrates and the formation of readily biodegradable substrates that can be removed by heterotrophic bacteria under anoxic and aerobic conditions. Compared to the treatment rate in February, shown on the left side of Figure 14 (a), the increased wastewater temperature enhanced the anoxic growth, ammonization and hydrolysis of the substrate, while reducing the dissolution of heterotrophs in July. . The biomass of heterotrophic microbial organisms increased on days with higher wastewater temperatures due to increased growth and reduced decay rates. It is also possible to increase ammonification by heterotrophic synthesis, and as a source of heterotrophs, slowly converting biodegradable substrates to easily biodegradable substrates can increase heterotrophic growth. These results indicate that the proposed G2ANet-based calibration system reflects the biological processes of heterotrophic organisms under operating conditions.

14(b) shows the biological process rates generated in the aerobic reactors in February and July. Compared to the low temperature season, the aerobic growth of autotrophs increased in the high temperature season. Dissolution of autotrophs was similar in both February and July. Biomass of autotrophs may increase in July according to calibrated kinetic parameters. Therefore, the TN removal efficiency in July increased from 81.16% in February to 86.93%. Detailed biological mechanisms were not provided to the G2ANet algorithm, but automatic calibration suggested reliable values of the dynamical parameters considering the influent and operating conditions as state information about the structure. As a result, the self-calibration based on G2ANet proved the feasibility and reliability of the calibration performance in a full-scale MBR plant considering the operating characteristics and biomass of microbial organisms.

According to an embodiment of the present invention, a new intelligent calibration protocol based on the G2ANet algorithm is proposed to calibrate the dynamic parameters of the ASM-SMP model. Eight kinetic parameters were selected as a set of calibration targets. Additionally, three temperature-dependent parameters were calibrated with the Arrhenius equation. Agents of the G2ANet algorithm were assigned to calibrate each kinetic parameter by receiving self-observations from the ASM-SMP model to account for the COD and TN removal phenomena. In addition, the centralized reviewer used a centralized training and distributed execution method to increase proofreading performance. The influent and operating conditions of the target MBR plant were considered as conditions and observations for the agent to reflect the biological mechanisms. The G2ANet-based calibration system outperformed the QMIX-based parameter calibration performance when comparing the compensation values of each algorithm with each other.

The ASM-SMP model calibrated by the G2ANet algorithm reduced the error variance of the base model by 87.05% and 51.85% for wastewater COD and TN simulations, respectively. In addition, the proposed calibration system was verified on a seasonal basis and monthly basis to ensure its applicability to the actual plant. G2ANet corrected the kinetic parameters with respect to seasonality. It increases microbial activity in the hot season and reduces biomass in the cold season. The main contribution is a stable and feasible G2ANet-based model calibration protocol for WWTP. The proposed G2ANet-based calibration system can consider the characteristics of dynamic and complex biological nutrient removal processes. In addition, it can be utilized to optimize and control the WWTP based on accurate modeling, as the proposed accurate and reliable model calibration protocol is key to deriving an appropriate and reliable operating strategy of the MBR system mimicking a real plant. .

Hereinafter, the configuration and functions of a multi-agent reinforcement learning-based sewage treatment plant process operation multi-optimization system according to an embodiment of the present invention using the aforementioned calibration system will be described.

In an embodiment of the present invention, an autonomous operating trajectory search system based on an advanced MARL algorithm is provided to achieve sustainable operation in various non-linear influent conditions. First, the k-means clustering algorithm distinguishes inflow conditions according to various inflow scenarios. Second, the optimal operating trajectory including DO, external recycling rate, and external carbon dose was automatically searched by the MARL algorithm to satisfy the desired environmental and economic operation. Third, we applied and evaluated the proposed system on a new one-year data set that was not used to train the MARL algorithm to verify its feasibility and adaptability under various influent conditions.

A framework for retrieving and proposing multiple operational trajectories for MBR is shown in FIG. 16 . The first step is the creation of high, normal, and low influent scenarios with influent loads for chemical oxygen demand (COD) and total nitrogen (TN). Daily changes in the measured influent data set were utilized to generate the influent data set. The generated daily influent data sets provided different influent scenarios for the MARL algorithm. The second step is to select the structure of the MARL algorithm to propose multiple operating trajectories considering influent conditions. Observations, statuses, actions and rewards were selected according to the operating characteristics of the target plant. In addition, when calculating the compensation function, discharged water quality (EQI), aeration energy (AE), pumping energy (PE), and external carbon (EC) were considered. This is an essential factor for the environmental stability and energy efficiency of the sewage treatment process. The third step is the application and training of the MARL algorithm. In this step, the best MARL algorithm is selected according to the change in the reward obtained. This is because the converged reward trend can evaluate the stable performance of the MARL algorithm. The final step is the evaluation of the autonomous trajectory search system for the proposed MBR in terms of effluent quality, energy and operating cost. Since the influent has a diurnal pattern, the operating trajectory is suggested daily by the MARL algorithm. The evaluation was conducted daily and compared with the manual system of the target wastewater treatment plant.

In the embodiment of the present invention, as shown in FIG. 17, the operation strategy of the MBR plant located in Y-city was optimized. The process includes biological treatment and sludge filtration by 5 reactors in sequence. The MBR plant replaced the clarifier with a membrane reactor to reduce footprint and waste sludge. The volumes of the aerobic, stabilizing, anoxic, aerobic and membrane reactors were 820, 220, 2,500, 2,900 and 720 m ³ respectively. External sludge recirculation between the membrane and stabilization reactor was set at 400% of the influent flow rate. The internal recirculation ratio between the anoxic and anaerobic reactors was 100% of the influent flow rate. DO concentrations for the aerobic and membrane reactors were maintained at 4 and 7 mg/L, respectively. The average influent concentrations of COD, TSS and TN were 360, 173 and 52 mg/L, respectively. Standard deviations ranged from 12.9 mg/L (TN) to 69 mg/L (COD). This means that the influent of the target MBR plant was abnormal, making it difficult to optimize the treatment process under dynamic conditions. The MBR plant was simulated with a calibrated activated sludge model soluble microbial products (ASM-SMP) model based on the measured data set. In addition, when proposing the optimal trajectory, influent data generation and various influent scenarios were used to reflect abnormal influent characteristics.

Carbon and nitrogen loads are important input factors in sewage treatment processes. Changes in influent can affect pollutant removal efficiency and wastewater treatment performance for operating energy. In addition, optimization of wastewater treatment without considering dynamic influent cannot guarantee sufficient accuracy and improve plant performance. To address dynamic and unsteady influent conditions for possible process optimization, this study generated influent data sets from measured influent data to reflect the inherent natural variability in pollutant concentrations. 18 shows changes in the concentration of inflow pollutants of the target MBR facility. Figure 18(a) shows a clear pattern of monthly data that pollutant concentrations were high in March-May and October-December and decreased in March-May. Monthly contaminant concentrations increased by 12% under high influent conditions, but decreased by -15% under the lowest influent conditions. Figure 18(b) shows the daily variation of the influent concentration, which is evident as morning and evening peaks corresponding to housework activities. Based on the measured data set and its characteristics, an hourly influent data set was created by considering the monthly pollutant changes in carbon and nitrogen composition ratios and the daily trend of influent pollutants. Gaussian noise was also added to the generated hourly data set to include uncertainties for the time-series influx data.

In the following, influent scenario clustering using the K-means clustering algorithm is described. In order to present the optimal operating trajectory from the generated influent data, it is necessary to cluster similar influent conditions by group. This is because it is impractical to provide all operating trajectories for each type of influent condition. This increases the dimensionality of the optimization problem and leads to time-consuming computations. Therefore, influent scenarios according to various COD and TN composition ratios were proposed to optimize the target plant to reflect complex influent conditions. Different inflow conditions can be grouped into two or more groups using the k-means clustering algorithm.

The k-mean clustering algorithm clusters variables according to their similarity. It selects a random number of centroids and then assigns the data to nearby clusters by calculating the Euclidean distance from the centroid. The two steps of k-means clustering minimize Equation 6.

[Equation 6]

Here, x _i is the COD and TN composition ratio in the present invention, μ _k is the average of cluster c _k , and K is the number of clusters.

A MARL-based multi-operating trajectory retrieval system is designed to determine optimal setpoints for tunable working parameters of a target MBR plant. The main contribution points of the multi-operation navigation system were environmental and cost-effective plant maintenance in response to various influent conditions. Therefore, as shown in FIG. 19, an optimal system integrated with influent scenario clustering by k-means algorithm is proposed to reflect various realistic influent conditions. Agents of the MARL algorithm interacted with each other to propose optimal settings for each and share process information of the target plant. Three agents were assigned to search for the optimal set point of DO concentration in the aerobic reactor, the amount of sludge recycled from the membrane reactor to the stabilization reactor before the anaerobic reactor, and the external carbon dose to the influent stream of the target plant. Three operating parameters manipulated by the MARL algorithm were selected to ensure the performance of the nitrification and denitrification processes. According to the proposed optimal trajectory, four performance indicators were evaluated in daytime operation. The index consists of EQI, AE, PE, and EC and was additionally considered as a reward function of the MARL algorithm.

The driving trajectory search system manipulated three variables including DO concentration, sludge recycling and external carbon usage. Several studies on wastewater quality and operational energy efficiency improvement have investigated the importance of three manipulated variables:

- First, the DO concentration is an important factor in the nutrient removal efficiency related to nitration, which converts ammonia (NH ₄ ) into nitrate (NO ₃ ). DO is also an indicator of the energy consumption of the wastewater treatment process, which consumes the most energy associated with the aeration process. Therefore, appropriate DO setpoints must be optimized to maintain sufficient nitrification and prevent excessive aeration energy consumption.

- Second, sludge recycling enhances the denitrification process by returning nitrate formed under aerobic conditions to the anoxic region. Nitrogen removal efficiency depends on the amount of sludge recycling, with low or high nitrate recycling reducing the denitrification potential.

- Third, adding an external carbon source is an effective way to enhance denitrification, which converts nitrate (NO ₃ ) into nitrogen gas (N ₂ ) in the stream. Excessive external carbon source dosage increases operating costs and triggers subsequent removal processes.

In an embodiment of the present invention, the multi-operating trajectory search system suggested weekly optimal set values of DO concentration, sludge recycling amount and external carbon usage for the improvement of pollutant removal efficiency and operating cost and the most influential operating parameters. Optimized operating trajectories were evaluated compared to manual operating systems. Four indicators, EQI, AE, PE and EC, were used to compare and evaluate the performance of the proposed optimization system.

EQI (kg/d) is the average value over the observation period based on the weighted effluent load of the compound that has a significant effect on water quality. EQI is expressed by Equation 7.

[Equation 7]

where T is the operating time (days) between t ₁ and t ₂ , and TSS _e (t), COD _e (t), Nkj _e (t), NO _e (t), and BOD _e (t) are each the total Suspended solids effluent concentrations (mg/L), chemical oxygen demand, Kjeldahl nitrogen, nitrate and biological oxygen demand (mg/L). B is the weighting factor for the pollutant in the wastewater (B _TSS =2, B _COD =1, B _Nkj =3-, B _NO =10, and B _BOD =2).

AE (kWh/d) is the energy consumed by aeration and is calculated from the K _L a value. This is expressed in Equation 8.

[Equation 8]

where S _O ^sat is the oxygen saturation concentration (8 m/L), V is the total N aerobic reactor volume in the ith reactor (m ³ ), and K _L a is the oxygen transfer coefficient (d ^-1 ).

PE (kWh/d) is calculated by the consumed energy of the internal and external flow recirculation pumps as in Equation 9.

[Equation 9]

where Q _int (t), Q _r (t) and Q _w (t) are the flow rates of internal recycling, external sludge recycling and waste sludge, respectively. EC (kgCOD / d) is the consumption of an external carbon source expressed by Equation 10.

[Equation 10]

where Q _EC is the external carbon addition rate (m ³ /d), and COD _EC is the readily biodegradable substrate concentration from the external carbon source (400,000 gCOD/m ³ ).

A multi-agent system is a group of interacting autonomous entities that share a common target environment. MARL is an artificial intelligence (AI) technology that replaces existing RL (reinforcement learning) algorithms that have limitations in finding common goals for multiple RL agents and coordinating actions with other agents. MARL has the advantage of sharing experiences with other agents and learning similar tasks together. This makes MARL inherently more powerful for solving multipurpose problems. A common goal of MARL practice is to integrate the stability of an agent's learning dynamics with the adaptation to the dynamic behavior of other agents. The common goal of MARL practice is to show the “stability” of an agent’s learning dynamics and interconnections, and the “adaptation” of an agent’s dynamic behavior corresponding to novel events in its environment. These two advantages of MARL make it easy to apply the MARL algorithm to the field of sewage treatment process research. This is because WWTPs are unsteady and highly variable, making them complex to optimize. Therefore, it is highly necessary to use reliable and adaptable optimization algorithms to optimize sewage treatment processes.

In the embodiment of the present invention, the operating trajectories of DO set point, external sludge recycling, and external carbon input were simultaneously optimized as shown in FIG. 19 using the MARL algorithm. The three agents of the MARL algorithm were deployed individually to optimize each manipulating the manipulated variable (distributed run). MARL's centralized training structure is adopted to learn agents together and share information. In the present invention, the optimization problem is abstracted from the time-based process operation into a sequential decision-making problem. The environment is an MBR plant modeled with the ASM-SMP model as described above. The agent was combined with the interaction interface to try to establish DO concentration, external sludge recycling amount and external carbon injection amount. Observations and agent status are related to influent and operating conditions. Reward values are distributed by the environment after each interface (i.e., learning epoch). Compensation functions were determined based on Performance Index, EQI, AE, PE and EC. The MARL algorithm was trained to maximize the reward value to improve the operational performance of MBR plants.

Two MARL algorithms, QMIX and G2ANet, were used to solve multi-target problems. QMIX is a value-based MARL algorithm that trains distributed policies of agents in a centralized, end-to-end manner. G2ANet interprets the agent's relationship based on the game abstraction theory using hard and soft attention. Both algorithms showed excellent performance in solving multi-agent problems such as StarCraft II, Traffic Junction, and Predator-Prey. The detailed algorithms of QMIX and G2ANet are as described in the previous calibration system.

Appropriate clustering of influent conditions is required to propose optimal operating trajectories for sustainable operation of MBR plants under various influent conditions. Influent conditions were created by considering the measured influent data set with composition ratios of influent COD concentrations of 0.85 - 1.12 and influent TN concentrations of 0.84 - 1.07. 20 shows the results of analyzing the generated influent data and the clustered data in three influent scenarios using the k-means clustering algorithm. First, influent data was generated according to the composition ratio of the measured influent COD and TN concentrations. Here, k-means clustering was used because direct use of operational trajectory retrieval systems for highly distributed data can increase the complexity of multi-agent systems.

The cluster area is automatically determined by the k-means clustering algorithm. In the k-means algorithm, centers were initially randomly arranged as shown in FIG. 20(a). After repeating the calculation of Equation 6 several times, the cluster center was representatively determined for each cluster at an appropriate distance from the data set included in the cluster. k-means clustering calculated three clusters using the degree of COD and TN composition ratio. The three clusters represent high, medium and low inflow scenarios. The centers for the high, medium and low scenarios were (1.12, 1.16), (1.01, 0.90) and (0.8, 0.92), respectively.

Figure 20(b) shows a sample of the influent data set generated over a diurnal period. In the present invention, the result of the generated COD concentration is depicted representatively. According to the influent scenario cluster, the generated data showed typical trends in pollutant concentrations of different magnitudes and diurnal time-related variability in household activity. In addition, the noise mixed in the influent data caused a large deviation in the generated data over time. As a result, influent data sets generated according to influent scenarios reflect the various situations of influent and can be used to develop an optimal operating search system under various influent conditions.

The structure of the MARL algorithm that interfaces with the modeled MBR platform was determined to maximize reward value through improvements in environmental and economic efficiency. The interaction interface between MARL and MBR plants with three agents was determined considering the vectors of state, observation, task and reward. The detailed structure of the MARL algorithm is as follows.

- State and observations:

The joint state(s) used when centralized learning is in progress is the time, COD, TN, NO ₃ , NH ₄ , TSS and the current and previous time flow rates (h and h-1). include Influent conditions observed at present and previous times helped reflect the dynamics of influent conditions. A vector of joint states makes it easy for agents to coordinate with each other and reduce the complexity of multi-agent problems. In addition, the three agents received local observation vectors according to which variables they manipulated.

The first agent receives a vector containing time, influent COD (COD _in ), influent TN (TN _in ), influent flow rate (Q _in ), and ammonia concentration in the aerobic reactor (NHaerobic). We propose a working trajectory of the setpoint concentration. The first consideration is that nitrification occurs in an aerobic reactor according to dynamic inlet conditions, so vectors include time, CODin, TNin, and Qin. Since ammonia removal also depends on the DO concentration for nitrification, NHaerobic was additionally included in the observation of the first agent.

The second article suggested the optimal amount of external sludge recirculation from the aerobic membrane reactor to the stabilization reactor. DO, nitrate and ammonia concentrations in anoxic reactors (DO _anoxic , NO _anoxic and NH _anoxic ) and nitrate and ammonia concentrations in membrane reactors (NO _mem and NH _mem ) were considered as observation vectors. Anoxic reactors for the denitrification process require low DO concentrations (DO _anoxic ). The NO _mem and NH _mem concentrations are essential in the recycling process because the denitrification performance is reduced when low nitrate is recycled to the anoxic reactor. Also, both NO _anoxic and NH _anoxic concentrations are indicators of discrimination performance.

The third agent is external carbon using observation vectors including time, influent COD (COD _in ), influent TN (TN _in ), influent flow rate (Q _in ), and influent COD/TN ratio (C/N ⁱⁿ ). It is for input retrieval. Low carbon to nitrogen (C/N _in ) conditions may require an external carbon source edition for fast and efficient denitrification. A small COD _in relative to the TN _in concentration results in incomplete denitrification, whereas a high dose of external carbon results in excessive costs and requires a subsequent process to remove it.

-Actions:

A separate working set was used for each multi-agent. The working set of DO setpoint suggested agents were [-1, -0.5, 0, +0.5 and +1 mg/L] for the DO concentrations. The action of the second agent to determine external sludge recycling was [-0.2, -0.1, 0, +0.1, +0.2 change] for 400% of the influent flow. The external carbon input was investigated by the third agent, which selected the carbon flow rate among [variation of -1, -0.5, 0, +0.5, +1 m ³ /d].

- Rewards:

The reward function was defined based on the operational goals of the MBR plant and rewarded the MARL algorithm for each action taken. Multi-agents always try to maximize their own payoff and generate optimal policies over time. Meanwhile, improvements in environmental and economic operations mean reductions in wastewater pollutants and operating energy in MBR plants. Therefore, in the present invention, a compensation function is assigned to compare the improvement amount of the proposed operating strategy with the passive operating system. The performance of operating system J(t) was calculated using Equation 11 and the reward value r(t) was calculated using Equation 12.

[Equation 11]

[Equation 12]

where ω is the weighting factor for each performance indicator (ω _EQI =200, ω _AE =40, ω _PE =3, and ω _EX =1). As shown in FIG. 19, the linear compensation function was applied by selecting well-known performance characteristics in the sewage treatment process. A linear combination of figures of merit reflects the expert's policy and helps to increase the performance of RL. We also assigned a negative reward value if the proposed operating trajectory could not improve operating performance compared to a passive operating system. Negative reward values help avoid non-optimal actions possible through the MARL algorithm.

The structure of MARL to search the optimal operating trajectory for daytime operation of MBR plant is summarized in Table 4.

[Table 4] Structure of QMIX and G2ANet to find the optimal operation trajectory of MBR plant

22 shows the average reward values of the two MARLs used according to training epochs. To compare the reward growth trend of the two MARL algorithms, the average reward value is shown instead of the reward value over all epochs. 10,000 training epochs were conducted to update and obtain the best policy to optimize the operating conditions of the MBR plant. Generated influent scenarios such as high, normal and low influent conditions were utilized in all training sessions for sustainable wastewater treatment operations. Both algorithms showed converged compensation values according to the number of epochs as shown in FIG. 22(a). However, QMIX has some structural constraints (see Fig. 22(b)) that limit accurate learning of the action-value function for the non-monotonic environment of the MBR process. This algorithm converged to a local optimum by the wrong policy with epsilon decay search. Thus, QMIX learned only sub-optimal policies with very unstable rewards despite higher reward values.

The detailed compensation value of G2ANet is shown in FIG. 22(c). The G2ANet algorithm is noticeably different in convergence stability. This means that G2ANet was able to more accurately handle the non-monotonic and non-stationary of the environment and found the optimal policy with a well-expressed action-value function. Although the acquisition reward values of G2ANet were lower than those of QMIX, G2ANet achieved more stable reward trends in all training procedures despite randomly given influx conditions. Therefore, it can be inferred that the G2ANet agents cooperated well to optimize and improve the MBR plant's operating system in terms of achieving environmental and economic goals in various influent environments. However, in QMIX, the agents learned to cooperate with rather limited coordination, so they did not actively try to optimize the process together, but instead tried to maximize the reward without coordination. Compared with the compensation values of each training process shown in Fig. 22(b) and Fig. 22(c), the G2ANet algorithm was suitable for application to the wastewater treatment process. Consequently, in the present invention, G2ANet was chosen to optimize the multi-operational set-point of the MBR plant by considering the balance between the highly efficient but unstable QMIX and the stable G2ANet with improved efficiency.

Optimal settings of the three manipulated variables should maintain qualitative performance under a variety of influent conditions. The performance of the operating trajectory retrieval system was evaluated over operating time under various influent conditions to identify optimal setpoints for process operation. 23 shows the operating trajectories proposed by the G2ANet algorithm selected under three inflow conditions, such as low (low), normal, and high (high) inflow scenarios, according to COD and TN composition ratios. The results of 20 simulations in the three influent scenarios are plotted as representative of the trend of the entire operating trajectory of the simulation. Influx scenarios of three centroids (1.12, 1.16), (1.01, 0.90), and (0.8, 0.92) were fed to the G2ANet algorithm, and the trained G2ANet retrieved the operating trajectories in time units.

23(a) shows the optimal trajectory proposed by the G2ANet algorithm under low inflow conditions. It was evident that the optimal setpoint for the DO concentration in the aerobic reactor was lower than the manual DO setpoint of 4 mg/L. Therefore, 92.08% of the DO concentration values suggested by G2ANet were lower than the manual DO settings. These results indicate that less aeration energy is consumed through a reduced operating trajectory compared to passive aeration systems. In addition, the optimal operating trajectory of the DO set point showed an obvious changing trend similar to the situation of the influent trend shown in FIG. 20 . The retrieved DO setpoints increased mainly around 10:00 am and 20:00 pm, while decreasing at other times consistent with changes in influent contaminants.

The operating trajectories of the external recycling rate and external carbon flow in the low inflow (low inflow) scenario are shown in Fig. 23(a). The trajectory of the external recycling rate explored under the low inflow condition was not different from the normal and high inflow conditions. On the other hand, the proposed external carbon flux in the low-inflow scenario was slightly lower than in the other scenarios. These results can infer that the G2ANet-based autonomous trajectory search system focuses on reducing excessive aeration energy rather than reducing pump energy and carbon input costs. This is because aeration energy accounts for more than 50% of the energy in the normal operation of a sewage treatment plant.

Fig. 23(b) shows the results of the driving trajectory in the normal inflow scenario. In the normal inflow scenario, 91.45% of the optimal trajectory DO concentration values were lower than the manual DO setpoint. On the other hand, the external recirculation rate was changed based on 4 times the inflow rate. On the other hand, the external carbon flow rate increased according to the inflow load trend. Compared to the results of the low inflow scenario, the G2ANet algorithm retrieved higher operating trajectories of DO concentration and external carbon flux. Higher settings enhance both nitrification and denitrification processes to remove nutrients and increase TN removal efficiency.

The optimal operating trajectory in the high inflow (high inflow) scenario is shown in FIG. 23(c). The G2ANet algorithm suggested the highest setpoint for the high inflow scenario compared to the other scenarios. Therefore, 88.33% of the suggested DO concentration values were lower than the manual DO setting value of 4 mg/L. Also, DO concentrations around 6 mg/L have sometimes been suggested for effective nitrification processes in aerobic reactors. The trajectory of the external carbon flux was suggested by relatively high values, while the external recycling rate was low and similar to the normal inflow scenario. It can be inferred that a higher DO concentration in the recycle stream would reduce the effect of denitrification in the anoxic reactor, and thus the external recycle rate did not increase significantly. As a result, the G2ANet-based autonomous operating trajectory retrieval system provided three setpoints for the MBR plant according to inflow scenarios to effectively enhance the nitrification and denitrification processes.

Rather, the G2ANet algorithm suggested a broader boundary of the three operating trajectories corresponding to very different influent conditions (see Fig. 20). A narrow operating trajectory can be a pitfall in operating a real-world MBR plant, as it risks reacting unreliably to dramatically changing influent conditions and leads to unfavorable outcomes in anomalous events. Therefore, the proposed autonomous driving trajectory search system can be a solution that can economically and environmentally operate the MBR plant by searching for the global optimal point under various influent conditions.

The daily measured influent COD and TN concentrations in 2018 at the target MBR plants are shown in Figure 24(a). Relatively high inlet COD concentrations were observed in early 2018. On the other hand, the concentration decreased between days 150 and 300, corresponding to May and October, respectively. The influent TN concentration increased around day 60 (February) and varied over time.

COD and TN concentrations were highly variable. Therefore, the composition ratio changes significantly over time. Influent conditions measured in 2018 were identified as three clusters: low influent conditions, normal influent conditions, and high influent conditions. The result of the influent condition identified according to k-means clustering is shown as a Voronoi diagram in FIG. 24(b). A Voronoi diagram divides a two-dimensional plane into polygons according to their closest points, facilitating the interpretation of data points in their geometrical properties. The influent up to about day 100 (shown in blue) was mainly located in high influent and normal influent conditions. The influent composition ratio gradually changed to normal and low influent conditions over time. In addition, data points were not collected at specific locations in each influx scenario. These results suggest that influent pollutants tend to change seasonally. However, the influent also had abnormal characteristics and extreme situations in which the composition ratio of the influent to COD and TN was less than (0.6, 0.6) or greater than (1.3, 1.3), as shown in FIG. 24(b). Therefore, it can be seen that influent characteristics are an important factor for a robust and stable optimization strategy in a highly variable influent environment.

24(c) shows monthly influent conditions characterized according to k-means clustering and Voronoi diagram. As the above results, the influent composition ratio in January and February was located in the high influent condition diagram. The influent composition ratio shifted to a condition with a low degree of contamination over time. The composition ratio of COD and TN was the highest in February (1.13, 1.04), and July, when the composition ratio of COD and TN was (0.76, 0.89), was confirmed to be the month with the lowest influent condition. As a result, the G2ANet-based autonomous trajectory search system was applied to months with low or high influent composition ratios to evaluate stability and adaptability in extreme influent conditions.

The optimal operation trajectory search performance by the G2ANet algorithm was evaluated by applying the influent conditions in February and July, when the influent conditions were extremely high and low. 25 shows the optimized performance of the G2ANet-based autonomous trajectory search system in February. An ASM-SMP model was used to simulate the target plant and the previously mentioned dynamically calibrated kinetic parameters were used to accurately simulate the MBR plant. As shown in FIG. 25(a), the influent condition in February showed a relatively high concentration of COD of 600 mg/L or more and TN of 46 mg/L or more, the average TN concentration. Also, the two influent contaminants changed over time. Therefore, it was necessary to search for optimal operating conditions according to various influent conditions.

25(b) shows the optimal trajectories in February including DO concentration, external recycling rate, and external carbon flux. The total DO concentration was lower than the set point of the manually operated system. The average suggested DO concentration was 2.30 mg/L, which is a 42.5% reduction from the manual DO setting of 4 mg/L. Meanwhile, the external recycling rate was investigated according to the fluctuating influent conditions. The average value was 4.04, which was close to the setting value of 4 for the manual driving system. Meanwhile, in order to strengthen the MBR plant's denitrification process, the external carbon amount was increased to an average of 1.53 m3/d. Since the influent COD/TN ratio is 8.48, the nitrogen removal performance may be reduced. The passive driving system added no extraneous carbon, but G2ANet's agent optimized the dosage within a range for improved nitrogen removal.

Performance indicators including EQI, AE, PE and EC are shown in FIG. 25(c). The performance index was calculated through the operation trajectory proposed by G2ANet (see Fig. 25(b)) and compared with the manual operation system. The three agents optimized their actuation trajectories to lower EQI under various inflow conditions, resulting in a 5.61% reduction in EQI values compared to the passive system. These results suggest that the reduced DO setpoint reduced effluent nitrate by preventing excessive nitrification and that the proposed recycling ratio and external carbon input maintained an effective denitrification process. The aeration energy effectively decreased by 28.44% due to the DO setpoint reduction, while the pumping energy increased by 1.01%.

The environmental benefits of a G2ANet-based autonomous system are shown in Figure 25(d). The dotted line represents the wastewater quality limit of the Korean Ministry of Environment. The effluent limits for CODCR and TN are 52 and 20 mg/L, respectively, as 3-hour average values. The COD concentration of discharged water by the manual and autonomous driving system satisfies the discharged water quality standard. However, since 18 effluent TN concentration values for days 40–50 and 52–58 exceeded the 20 mg/L criterion, the passive system could be limited in meeting the effluent criterion for TN. Although the effluent limit was applied as a 3-hour average value, the passive system discharged effluent exceeding 20 mg/L TN 4 times as indicated by the red circles in Fig. 25(d). Passive systems require biological or chemical treatment to reduce wastewater TN concentrations and robust real-time monitoring systems. Additional processing may result in excessive operating costs. On the other hand, the proposed G2ANet-based autonomous system effectively reduced the effluent TN concentration and satisfied the criteria. The three interconnected agents of G2ANet have teamed up to make the MBR plant environmentally and economically viable despite extremely high inflow conditions. A quantitative evaluation of the proposed system is summarized in Table 7.

The influent in July was 396 mg/L for COD and 46 mg/L for TN, similar to or lower than the average influent concentration, as shown in FIG. 26(a). Therefore, it is possible to propose an applicable solution by balancing economic and environmental benefits. 26(b) shows the optimized trajectory of the G2ANet-based autonomous driving system in July. The setpoint of the DO concentration in the aerobic reactor was effectively reduced by 49.3%, so the average DO concentration in the autonomous system was 2.03 mg/L. Compared to the DO setpoint in February, the proposed system further reduced the DO concentration in July by 6.8%. There are two reasons for the low setpoint in July. (1) influx conditions were lower and (2) microbial community growth was more active under high temperature conditions. The activation of microorganisms according to the operating temperature was reflected in the calibrated ASM-SMP model. Therefore, the G2ANet-based autonomous system suggested a lower DO set point in the high temperature season.

The performance of the G2ANet-based autonomous system in July is shown in FIG. 26(c). The EQI value was reduced by 11.11% compared to the passive system with the operating trajectory optimized by G2ANet. In July, the improvement in effluent water quality was higher because the nitrogen removal rate was better improved in the high temperature season. Since temperature is a sensitive factor for microbial growth and influences nitrification-denitrification, wastewater quality can be improved with fewer operating values. The optimized DO setpoint reduced the aeration energy by 30.84% and increased the pumping energy by 0.43%.

The environment improvement of the autonomous system is shown in Fig. 26(d). Sewage COD complied with legal discharge limits and remained below 52 mg/L. However, the manual operation method discharged effluent (TN) that exceeded the limit on a total of four occasions on days 187, 202, and 205. Passive systems can meet effluent limits by considering 3-hour averages, but rigorous operation may be risky in future inflow conditions. From the environmental management point of view, the proposed G2ANet-based autonomous system can be a solution that safely meets the effluent water quality restrictions as shown in the red circle in Figure 26(d). G2ANet has effectively improved the MBR plant's operating system by achieving economic and environmental benefits. A comprehensive summary of the performance improvements is shown in Table 7.

[Table 7] Overall performance of G2ANet-based autonomous driving system in full-scale MBR factory

Table 7 summarizes the quantitative performance improvement of the entire G2ANet-based operating system in February and July 2018. The autonomous system clearly improved effluent quality under both high and low influent conditions, as shown in FIGS. 25 and 26 . Therefore, the system according to the embodiment of the present invention has the ability to satisfy the effluent water quality limit and improve the effluent water quality by 6.97% under various influent conditions in 2018. In addition, the aeration energy decreased by 24.80% for the whole year, while the pumping energy increased by 1.06%. As a result, the proposed autonomous trajectory search system can save 4.805·10 ⁵ kWh during one year of operation. The G2ANet-based autonomous driving system can save $36,039 annually at a price of $0.075 per kWh supplied by KEPCO. External carbon input cost was not included in the performance evaluation because the target plant uses external carbon irregularly. In addition, the present invention assumes that the external carbon source is an easily biodegradable substrate. The optimization system can be extended by controlling other variables such as methanol-based external carbon and DO concentrations. As a result, G2ANet-based autonomous systems can contribute to safe water quality maintenance and energy savings under dynamic and varied influent conditions. In addition, the present invention showed the possibility of being extended to more complex operating systems by considering centralized learning-decentralized execution agents to optimize the manipulation of variables under various influent conditions.

In the embodiment of the present invention, an autonomous driving trajectory search system is proposed for energy saving and environmental operation of MBR plants. Optimal operating settings were provided by the G2ANet algorithm under various inlet conditions. To develop a stable and adaptable operating strategy, influent conditions considering low, normal, and high COD and TN composition ratios were created and clustered into three scenarios based on k-means clustering algorithm. Advanced G2ANet reinforcement learning was used to propose operational trajectories of DO, external sludge recycling, and external carbon input across the three collaborating agents.

The results of the G2ANet-based autonomous system show that the optimal setpoint is adequately proposed to improve the performance of the target MBR plant and clearly outperforms under various influent conditions. In addition, the newly measured one-year influent case shows that the G2ANet-based autonomous system can adapt to new influent conditions by improving aeration energy by 24.80% and effluent quality by 6.97% while maintaining similar pumping energy under various COD-TN composition ratio conditions of dynamic influent. indicates that it can be dealt with. As a result, the G2ANet-based autonomous operating trajectory retrieval system represents an innovative contribution to the smart operation of WWTP operating systems through their stability and adaptability in the domain of digitized mathematical research. It can also serve as an operating guide for actual plant operators by proposing a stable multi-solution in a dynamic and complex wastewater environment.

In addition, the device and method described above are not limited to the configuration and method of the above-described embodiments, but all or part of each embodiment is selectively combined so that various modifications can be made. may be configured.

Claims (11)

As a sewage treatment plant model correction system,
a modeling unit for calculating a sewage treatment plant model by simulating the sewage treatment plant; and
A modeling correction unit for correcting the sewage treatment plant model by correcting the dynamic coefficient value of the sewage treatment plant model using artificial intelligence learning; sewage treatment plant model autonomous correction system using reinforcement learning.
According to claim 1,
The sewage treatment plant model is an activated sludge model (ASM), characterized in that the sewage treatment plant model autonomous correction system using reinforcement learning.
According to claim 2,
The artificial intelligence learning is a sewage treatment plant model autonomous correction system using reinforcement learning, characterized in that multi-agent reinforcement learning artificial intelligence.
According to claim 3,
The multi-agent reinforcement learning artificial intelligence is a sewage treatment plant model autonomous correction system using reinforcement learning, characterized in that based on the G2ANet algorithm.
As a sewage treatment plant model correction method,
Calculating a sewage treatment plant activated sludge model (ASM) by a modeling unit simulating the sewage treatment plant;
A method for self-calibrating a sewage treatment plant model using reinforcement learning, comprising: a modeling correction unit correcting the dynamic coefficient value of the sewage treatment plant model using artificial intelligence learning to generate a calibrated ASM model. According to claim 5,
The artificial intelligence learning is a sewage treatment plant model autonomous correction method using reinforcement learning, characterized in that multi-agent reinforcement learning artificial intelligence.
According to claim 6,
The multi-agent reinforcement learning artificial intelligence is a sewage treatment plant model autonomous correction method using reinforcement learning, characterized in that based on the G2ANet algorithm.
As a sewage treatment plant operation optimization system,
An autonomous correction system according to any one of claims 1 to 4; and
An optimization module for optimizing the sewage treatment process operation control based on the ASM model calibrated by the autonomous correction system; a sewage treatment plant process operation multiple optimization system using reinforcement learning, comprising:
According to claim 8,
The optimization module is a sewage treatment plant process operation multiple optimization system using reinforcement learning, characterized in that for optimizing process operation based on multi-agent reinforcement learning artificial intelligence.
According to claim 9,
The optimization module optimizes the aeration intensity, external carbon source injection amount, and sludge circulation flow rate.
As a multiple optimization method for sewage treatment plant process operation,
Calculating a sewage treatment plant activated sludge model (ASM) by a modeling unit simulating the sewage treatment plant;
generating a calibrated ASM model by a modeling correction unit correcting the dynamic coefficient value of the sewage treatment plant model using artificial intelligence learning; and
The optimization module optimizes the aeration intensity, external carbon source injection amount, and sludge circulation flow rate of sewage treatment process operation through multi-agent reinforcement learning artificial intelligence based on the calibrated ASM model. Sewage treatment plant process operation multiple optimization method.