WO2025246013A1 - 一种基于非参数强化学习的航空发动机维护策略优化方法 - Google Patents

一种基于非参数强化学习的航空发动机维护策略优化方法

Info

Publication number
WO2025246013A1
WO2025246013A1 PCT/CN2024/109213 CN2024109213W WO2025246013A1 WO 2025246013 A1 WO2025246013 A1 WO 2025246013A1 CN 2024109213 W CN2024109213 W CN 2024109213W WO 2025246013 A1 WO2025246013 A1 WO 2025246013A1
Authority
WO
WIPO (PCT)
Prior art keywords
sub
maintenance
network
aero
engine
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2024/109213
Other languages
English (en)
French (fr)
Inventor
张益鸣
张鼎扬
张树有
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University
Original Assignee
Zhejiang University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University filed Critical Zhejiang University
Priority to PCT/CN2024/109213 priority Critical patent/WO2025246013A1/zh
Priority to JP2025549928A priority patent/JP2026521822A/ja
Publication of WO2025246013A1 publication Critical patent/WO2025246013A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00—Computer-aided design [CAD]
    • G06F30/20—Design optimisation, verification or simulation
    • G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]

Definitions

  • This invention relates to the field of aero-engine operation and maintenance technology, and specifically to an aero-engine maintenance strategy optimization method based on nonparametric reinforcement learning.
  • Aero engines are a crucial infrastructure component of logistics and passenger transport. Developing sound maintenance strategies is essential for ensuring the normal operation of aero engine equipment, improving reliability, and saving costs for airlines. However, in routine aero engine maintenance, the complexity of engine equipment, the relatively harsh working environment, the high uncertainty of system sensors, the complex degradation mechanisms of sub-components, and the need for safety-assured design of the engine system all contribute to low efficiency and significant resource waste in aero engine system maintenance.
  • Existing aero-engine maintenance decision-making schemes mainly include: repair maintenance, preventive maintenance, and condition-based maintenance.
  • Repair maintenance is an unplanned maintenance action taken after equipment or system failure, typically occurring when equipment malfunctions or shuts down. It mainly includes fault detection, problem diagnosis, equipment repair, and restoring normal operation.
  • Preventive maintenance is a planned maintenance activity taken before equipment failure occurs, aiming to prevent equipment failure through regular inspections, maintenance, and repairs to ensure efficient operation within a predetermined timeframe.
  • Condition-based maintenance is a maintenance strategy that determines the timing of maintenance based on the actual operating state and health condition of the equipment. It uses sensors, monitoring equipment, and technologies to collect data to predict the equipment's health condition and thus determine when maintenance should be performed.
  • Existing aero-engine maintenance decision-making schemes often rely on static maintenance plans or the current system state, which are insufficient in scenarios requiring rapid response and high safety levels. Furthermore, they struggle to provide correct maintenance solutions for decision-making problems arising from system uncertainties such as sensor failure.
  • the purpose of this invention is to address the shortcomings of existing technologies by providing an aero-engine based on nonparametric reinforcement learning. Maintenance strategy optimization methods.
  • the present invention employs the following technical solution:
  • a method for optimizing aero-engine maintenance strategies based on nonparametric reinforcement learning comprising the following steps:
  • Establish a policy network and a value network randomly select an initial system state s ⁇ sub>0 ⁇ /sub> , and formulate an initial maintenance action a ⁇ sub> 0 ⁇ /sub>. Based on the initial system state s ⁇ sub>0 ⁇ /sub> and the initial maintenance action a ⁇ sub> 0 ⁇ /sub>, interact with the aero-engine model to obtain a set of state-action value groups: system state, maintenance action, system feedback cost, and next system state.
  • Update the training set. Determine whether the system running step size has reached the set system training step size. If it has, stop training, obtain the trained network, and output the aero-engine maintenance optimization strategy. Otherwise, continue to select actions in the policy network and repeat training until the running step size reaches the training step size.
  • the parameter values of the system state include the stagnation pressure corresponding to each component i of the engine.
  • Stagnation temperature Stagnation enthalpy Pressure ratio ⁇ i specific heat of working fluid at constant pressure
  • the specific heat ratio of the working fluid is ⁇ i
  • the efficiency factor is ⁇ i ; the maintenance actions include major repairs and minor repairs.
  • the aero-engine model includes a system state transition model and a system feedback model
  • the system state transition model is fitted to the system state transition probability obtained based on the Markov process using a Bayesian network, wherein the Bayesian network parameter L is represented by the maximum likelihood estimation method; the Bayesian network parameter optimization is performed using the expectation-maximization algorithm; and the system state transition function P(s t+1
  • the system feedback model is based on the system state s t and fits the operating cost r 1 (s t ) through a Gaussian process; according to the maintenance actions determined by the policy network under the system state s t , the sum of the prices corresponding to the maintenance actions of each component is taken as the maintenance cost r 2 (s t ); the sum of the operating cost r 1 (s t ) and the maintenance cost r 2 (s t ) is taken as the system feedback cost r (s t ) under the system state s t ;
  • the policy network selects actions based on the system state s ⁇ sub> t ⁇ /sub> , and obtains the maintenance action execution probability P ⁇ sub>net ⁇ /sub> ( at ) through a Gaussian process:
  • a ⁇ (s t , a i ) is the advantage function of the i-th maintenance action a i predicted by the policy network under the system state s t ;
  • the value network takes the system state s ⁇ sub> t ⁇ /sub> as input and outputs a value assessment V ⁇ sub> ⁇ /sub> (s ⁇ sub> t ⁇ /sub> ) of the system state s ⁇ sub> t ⁇ /sub> based on a Gaussian process.
  • the action selection is carried out by using an ⁇ -greedy algorithm to select maintenance action exploration or maintenance strategy utilization to formulate the next maintenance action; when the ⁇ -greedy algorithm sampling is greater than ⁇ , maintenance action exploration is performed, and the advantage function is obtained by random sampling in the uncertainty range of the Gaussian process through the policy network; when the ⁇ -greedy algorithm sampling is less than ⁇ , maintenance strategy utilization is performed, and the advantage function is output by the Gaussian process mean through the policy network.
  • Bayesian data fusion is employed to combine prior maintenance experience data with maintenance actions obtained through a policy network to assist in decision-making guidance.
  • the updated policy network and value network include the following sub-steps: based on the original sampled point system state
  • the dominance function A ⁇ (s ⁇ sub>t ⁇ /sub> , a ⁇ sub> t ⁇ /sub> ) ⁇ sup> new ⁇ /sup> is used to extend the Gaussian process fitting dataset.
  • ⁇ is the attenuation coefficient
  • r(s t ,at t ) is the system feedback cost function
  • a Gaussian process is constructed based on the covariance matrix and used as the updated policy network:
  • I is the variance of the observation noise
  • K is the identity matrix
  • K is the covariance matrix calculated using the Gaussian process kernel function.
  • the value assessment V ⁇ (s t ) is corrected using the time difference method, and the value network is updated through batch training.
  • updating the training set includes the following sub-steps: obtaining the degree of fit between each training data point and the value network by calculating the negative log-likelihood; re-batch random sampling of the replay buffer, obtaining the degree of fit between the resampled points and the value network by calculating the negative log-likelihood; comparing and replacing data with higher fit in the training set with data with lower fit, so that the model tends to train on data with lower fit, thereby improving the performance. Overall reliability of the model.
  • the present invention also provides an aero-engine maintenance strategy optimization system based on nonparametric reinforcement learning, comprising:
  • the aero-engine model building module is used to build system state transition models and system feedback models based on collected aero-engine operating data and Bayesian networks and Gaussian processes.
  • the network initialization module is used to initialize the policy network and the value network.
  • the maintenance action decision module is used to formulate maintenance strategies based on the current system state of the aero-engine model, so as to avoid the aero-engine from entering a highly deteriorated state and reduce system feedback costs.
  • the replay buffer module is used to store the state-action value group for each step
  • the model training module is used to train the policy network and the value network by using the state-action value groups extracted in batches by the replay buffer module.
  • the dynamic update module is used to dynamically update the training set, replacing data with a high degree of fit to the value network with data with a low degree of fit.
  • the beneficial effects of this invention are that by constructing an aero-engine model and using a nonparametric reinforcement learning method based on Gaussian processes, the sparse engine model can be fitted to a Gaussian process via a Bayesian network, while ensuring similarity to high-precision simulations.
  • This invention improves the fit between the overall training data and the model through dynamic data updates while incorporating system uncertainties, thus increasing algorithm sampling efficiency.
  • this invention enhances system security and sampling efficiency by incorporating uncertain action selection methods based on prior maintenance experience data. The results are superior to existing methods and offer advantages such as strong versatility and ease of use.
  • Figure 1 is a schematic diagram of the intelligent agent learning process described in this invention.
  • Figure 2 is a schematic diagram of the aero-engine model building process described in this invention.
  • Figure 3 is a block diagram of the aero-engine maintenance strategy optimization system based on nonparametric reinforcement learning described in this invention.
  • Figure 4 shows the fitting effect of the aero-engine model, where (a) is a comparison between the model prediction results and the actual results, and (b) is a global sensitivity analysis of the feedback model on the high-pressure turbine, low-pressure turbine, high-pressure compressor, and low-pressure compressor.
  • Figure 5 is a comparison of the training results using this method and the training results of traditional decision-making algorithms.
  • (a) is the convergence result of the algorithm training
  • (b) is the box plot of the five-fold results of the actual application of the algorithm after training with the aero-engine model.
  • Figure 6 shows the specific maintenance strategy for the high-pressure turbine, low-pressure turbine, high-pressure compressor, and low-pressure compressor components of an aero-engine, as described in this method.
  • the core technology of this invention is to mathematically model historical aero-engine data with sparse data features, constructing an aero-engine model.
  • nonparametric reinforcement learning is employed to solve the model.
  • Nonparametric reinforcement learning utilizes Gaussian processes to quantify system uncertainty and leverages nonparametric features to integrate system uncertainty in aero-engine scenarios into the overall reinforcement learning algorithm flow. This solves the problems of sparse datasets and uncertainty in real industrial scenarios.
  • a Bayesian data fusion algorithm is used to incorporate prior maintenance experience into the maintenance strategy exploration process, improving algorithm sampling efficiency and thus solving the predictive maintenance problem of aero-engine systems. This results in a method for formulating aero-engine system operation and maintenance strategies that incorporates system uncertainty. The main idea can be summarized as follows:
  • a nonparametric reinforcement learning method based on Gaussian processes is proposed.
  • the policy and value functions of the reinforcement learning algorithm are constructed based on Gaussian processes. Actions are explored and utilized based on the uncertainty of Gaussian processes.
  • a Bayesian data fusion algorithm is used to integrate prior maintenance experience data with reinforcement learning decision results, incorporating the uncertainty of the engine system into the reinforcement learning process.
  • the deterioration state of the engine subsystem is used as a safety evaluation index, and the system feedback cost is used as an economic evaluation index for the maintenance plan.
  • This method is used to formulate operation and maintenance strategies for the aero-engine system, improving the overall safety and reliability of the system.
  • a dynamic update method for model sampling points is adopted. Based on the negative log-likelihood of the training data used in each model and the value network, the fit between the training data and the value function model is calculated. By dynamically updating the method, training points with lower fit to the model are selected for batch training, which improves the fit of the reinforcement learning network to the overall model sampling points and enhances the reliability of the system.
  • the present invention provides an optimization method for aero-engine maintenance strategies based on nonparametric reinforcement learning, as shown in Figure 1, which specifically includes the following steps:
  • Aero-engine model for algorithm verification is established, which consists of two parts: a system state transition model and a system feedback model.
  • the system state transition model is responsible for judging the engine state deterioration process based on the current aero-engine state, while the system feedback model is responsible for calculating the total cost of each engine operation and maintenance cycle, which serves as the system reward.
  • the following sub-steps are included:
  • the state transition process between different components of the engine is characterized by isentropic relations and calibrated polytropic efficiency: the state parameter value of the i-th component is the stagnant pressure. Stagnation temperature Stagnation enthalpy Pressure ratio ⁇ i , specific heat of working fluid at constant pressure is The specific heat ratio of the working fluid is ⁇ ⁇ sub>i ⁇ /sub> , and the efficiency factor is ⁇ sub> i ⁇ /sub> .
  • the state parameter transition equations between different components are as follows:
  • the system's state is defined using a degenerate simulation based on a gamma distribution.
  • the degenerate equation is:
  • ⁇ represents damage growth
  • ⁇ is the shape parameter
  • ⁇ is the scale parameter
  • ⁇ is the gamma distribution. 100 engine operation segments are generated, each segment consisting of 50 deterioration steps, and the state is discretized into 5 levels, with higher values indicating increasing deterioration.
  • the state transition probability of the aero-engine is obtained based on the Markov process using the following formula:
  • S0 is the state before the transition
  • S1 is the state after the transition
  • the Bayesian network parameters L are expressed using the maximum likelihood estimation method:
  • Historical operating cost data for each component of the engine are collected.
  • the system state s ⁇ sub>t ⁇ /sub> and operating cost r ⁇ sub>1 ⁇ /sub>(s ⁇ sub> t ⁇ /sub> ) of the aero-engine are represented by x (i) and y (i) .
  • the kernel function of the Gaussian process is:
  • This function measures the similarity of input data, where x and x′ are input vectors, and
  • l is a length scale parameter that controls the smoothness of the function. This is the signal variance, which controls the overall magnitude of the kernel function. For a given dataset (X,Y), the marginal likelihood of a Gaussian process is:
  • K is the covariance matrix and n is the number of training data points.
  • the repair prices of each component are summed up as the maintenance cost r2 ( st ) of the engine in the system state s t .
  • the sum of the engine's operating cost r1 ( st ) and maintenance cost r2 ( st ) is taken as the system feedback cost r( st ).
  • Aero-engine System Integration and Initialization The mathematical model of the aero-engine system proposed in this example consists of a system state transition model and a system feedback model. During operation, the model accepts the current state of the system and provides the next state and system feedback cost based on the current state and maintenance actions. After the aero-engine model has been running for a period of time, the aero-engine system uses the system transition function P(s t+1
  • the system feedback model is used to calculate the engine operating cost r 1 (s t ) and maintenance cost r 2 (s t ) corresponding to the current state s t in this period.
  • the operating cost and maintenance cost are integrated as the system feedback cost r (s t ).
  • the aero-engine model and initial state are obtained.
  • This example uses non-parametric networks such as Bayesian networks and Gaussian processes for modeling, which can achieve better fitting results for the characteristics of sparse datasets.
  • a maintenance strategy is formulated for the engine.
  • a reinforcement learning policy network Pnet is established to formulate the system maintenance strategy.
  • the system state s ⁇ sub> t ⁇ /sub> is input into the policy network Pnet to obtain the execution probability P( at ) of the maintenance action formulated by the system.
  • the system state s ⁇ sub> t ⁇ /sub> and the system dominance function A ⁇ sub> ⁇ ⁇ /sub>(s ⁇ sub>t ⁇ /sub> , at ) are represented by x and y to establish a Gaussian process:
  • the kernel function for a Gaussian process is:
  • This function measures the similarity of input data, where x and x′ are input vectors, and
  • l is a length scale parameter that controls the smoothness of the function. This is the signal variance, which controls the overall magnitude of the kernel function. For a given dataset (X,Y), the marginal likelihood of the Gaussian process is:
  • K is the covariance matrix and n is the number of training data points.
  • the policy network selects actions based on the system state. Action selection employs either an ⁇ -greedy algorithm or policy exploitation to obtain the advantage function for each maintenance action, thus determining the next maintenance action.
  • ⁇ is a number between 0 and 1. Random sampling is performed on the range of 0-1. If the sampled value is greater than ⁇ , action exploration is used, meaning the policy network randomly samples within the uncertainty range of the Gaussian process to obtain the advantage function; otherwise, policy exploitation is used, where the policy network outputs the advantage function based on the Gaussian process mean.
  • Action exploration can explore random actions within a certain range to find better maintenance actions.
  • Policy exploitation utilizes the Gaussian process mean output of the policy network obtained through training, inheriting the results of previous training.
  • the execution probability of each maintenance action is calculated based on the dominance function output by the Gaussian process.
  • the execution probability of each maintenance action is:
  • a ⁇ (s t , a i ) is the advantage function of the policy network predicting the i-th maintenance action a i under state s t .
  • the action selection is performed using the maximum probability estimation method.
  • a t argmax(P net (a t ))
  • a ⁇ sub> t ⁇ /sub> refers to the specific maintenance action optimized by the policy network Pnet based on the execution probability of maintenance actions according to the current system state, and is the output of the policy network Pnet.
  • the maintenance actions include major overhauls and minor repairs.
  • Bayesian data fusion is used to combine prior maintenance experience data with the policy network.
  • prior maintenance experience data provides guidance, improves the security of the system, increases sampling efficiency, and accelerates algorithm convergence.
  • Pa is the probability of the decision action in the policy network
  • Pae is the probability of the action to be executed based on the prior maintenance experience data
  • w is the weight of the influence of the prior maintenance experience on the decision.
  • G ⁇ sub>t ⁇ /sub> represents the system feedback.
  • the system state s ⁇ sub> t ⁇ /sub> is input into the value network V ⁇ sub>net ⁇ /sub> to obtain the estimated value of the actions defined by the system, V ⁇ sub>t ⁇ /sub>.
  • the system state s ⁇ sub> t ⁇ /sub> and the estimated value of the system actions, V ⁇ sub> t ⁇ /sub> are represented using x ⁇ sub>i ⁇ /sub> and y ⁇ sub>i ⁇ /sub> , respectively.
  • a Gaussian process is established, with the Gaussian kernel function and marginal likelihood consistent with step (2.1).
  • the output of the Gaussian process is the value assessment of the system state by the value network V ⁇ sub>net ⁇ /sub>, V ⁇ sub> ⁇ ⁇ /sub>(s).
  • step (1) Based on the reinforcement learning policy network Pnet and value network Vnet generated in steps (2.1) and (2.2), the aero-engine model established in step (1) is initialized with random state, the system state s0 is randomly selected and input into the policy network Pnet, and the first maintenance action a0 is formulated.
  • Strategy iteration and network training Based on the initial maintenance action a0 and initial system state s0 formulated in step (2), interact with the aero-engine model established in step (1) to obtain a set of state action value groups; establish a replay buffer, store the state action value group (s t , a t , R(s t ), s t+1 ) of each step into the replay buffer and sample it as a training set; update the policy network Pnet and the value network Vnet; dynamically update the training set through negative log-likelihood.
  • step (3.1) Based on the initial system state s ⁇ sub>0 ⁇ /sub> and initial maintenance action a ⁇ sub>0 ⁇ /sub> obtained in step (2.3), interact with the aero-engine system to obtain the next state of the system and the cost of the state transition process. Organize this process into a state-action group (s ⁇ sub>t ⁇ /sub>, at ⁇ sub>t ⁇ /sub>, R(s ⁇ sub> t ⁇ /sub> ), s ⁇ sub>t+1 ⁇ /sub> ). Since both the policy network Pnet and the value network Vnet require batch data for training, therefore... A data replay buffer needs to be established to store the state-action group for each step, and then extracted in batches for training the policy network and value network. The state-action group for that step is stored in the replay buffer.
  • ⁇ is the attenuation coefficient and r(s t ,at t ) is the system feedback cost function.
  • the dominance function based on the system state at the original sampling point (i.e., the sampling point of the previous period), The dominant function A ⁇ (s ⁇ sub>t ⁇ /sub> , a ⁇ sub> t ⁇ /sub> ) ⁇ sub> old ⁇ /sub> and the system state at the new sampling point (i.e., the sampling point in the current period)
  • the advantage function A ⁇ (s ⁇ sub>t ⁇ /sub> , a ⁇ sub> t ⁇ /sub> ) ⁇ sup> new ⁇ /sup> is used to expand the Gaussian process fitting dataset and update the policy network.
  • the input to the extended Gaussian process is:
  • the output is:
  • a ⁇ sub> ⁇ /sub> (s ⁇ sub>t ⁇ /sub> , a ⁇ sub> t ⁇ /sub> ) is the advantage function when taking action a ⁇ sub> t ⁇ /sub> in state s ⁇ sub> t ⁇ /sub>, and the expanded global covariance matrix is:
  • a Gaussian process is constructed based on the covariance matrix and used as the updated policy network:
  • K is the covariance matrix calculated by the Gaussian process kernel function.
  • I is the variance of the observation noise, and I is the identity matrix.
  • V ⁇ ( st ) is corrected using the time difference method (TD-Error), batch training is performed, and Gaussian process fitting is applied.
  • e is the number of training data points extracted in a batch
  • ⁇ ⁇ sub>i ⁇ /sub> and ⁇ sub> i ⁇ /sub> are the variance and mean of the value function prediction for the i-th training data point, respectively.
  • y ⁇ sub>i ⁇ /sub> is the system state value V ⁇ sub> ⁇ ⁇ /sub>(s ⁇ sub> i ⁇ /sub> ) after correction using the time difference method
  • the negative log-likelihood represents the degree of fit between each training data point and the value function model.
  • the set of points is denoted as e * , and the negative log-likelihood of e * is calculated.
  • a larger negative log-likelihood indicates a poorer fit between the sampled points and the model.
  • the replacement probability Pu is calculated as the ratio of the negative log-likelihood of e to the sum of the negative log-likelihoods of e and e * , and is used as the training data replacement probability to replace training data points.
  • a smaller probability tends to retain the current training data point e, while a larger probability tends to replace the current training data point e with the resampled point e * , making the model more inclined to train on data with a lower model fit, thereby improving the overall reliability of the model.
  • e is the set of sampling points used for training
  • e * is the set of sampling points for resampling
  • An embodiment of the present invention was implemented on a machine equipped with an Intel(R) Xeon(R) Gold 6330 CPU, an NVIDIA GTX3090 graphics processor, and 128GB of memory. Using the above-described embodiment, the results shown in Figures 4-6 were obtained.
  • Aero-engine modeling was performed on the GE90 engine of General Electric, and the state transition matrix and system feedback of the engine were fitted.
  • the fitting results are shown in Figure 4.
  • Figure 4(a) shows the comparison between the model prediction results and the actual results.
  • the vertical axis represents the predicted value
  • the horizontal axis represents the actual value.
  • the circled points are the actual data points
  • the solid plus points are the predicted points.
  • This method can accurately predict the state transition process and system feedback of the aero-engine.
  • Figure 4(b) shows the global sensitivity analysis of the system feedback model on the high-pressure turbine, low-pressure turbine, high-pressure compressor, and low-pressure compressor.
  • Figure 5(a) shows the effect of this method compared with the rule-based maintenance method, the deep value reinforcement learning algorithm, and the actor-critic reinforcement learning algorithm.
  • the bottom curve is the convergence result of this method, and the upper dashed line is the rule-based maintenance algorithm.
  • Nonparametric reinforcement learning can significantly reduce the system feedback cost compared with the rule-based maintenance algorithm.
  • the two convergence curves at the top are the deep value reinforcement learning algorithm and the actor-critic reinforcement learning algorithm, respectively.
  • the convergence results of these two algorithms after 1000 training iterations are still inferior to those of nonparametric reinforcement learning after 100 training iterations.
  • both algorithms exhibit strong oscillations during convergence, indicating low sampling efficiency and inability to train stably.
  • nonparametric reinforcement learning shows relatively stable training, demonstrating high sampling efficiency. The results show that nonparametric reinforcement learning improves data utilization efficiency by more than 100 times compared to traditional reinforcement learning methods.
  • FIG. 5(b) shows the box plot of the average performance of this method compared to rule-based maintenance methods, deep value reinforcement learning algorithms, and actor-critic reinforcement learning algorithms after 5 practical applications following training.
  • the leftmost box plot shows the results of this method.
  • the nonparametric reinforcement learning method has significantly lower upper, lower, and mean system feedback costs compared to the comparison methods, and the actor-critic model also exhibits decision divergence. It can be seen that this method can relatively stably formulate maintenance strategies for aero-engines.
  • Figure 6 shows the results of this method for high-pressure turbines and low-pressure turbines.
  • the specific maintenance strategies for the high-pressure turbine, high-pressure compressor, and low-pressure compressor are defined, with circles representing minor repairs and inverted triangles representing major repairs. The results show that this method can accurately mathematically model aero-engines and correctly formulate reasonable maintenance strategies for different components.
  • This embodiment also provides an aero-engine maintenance strategy optimization system based on nonparametric reinforcement learning, which is used to implement the above embodiments.
  • module refers to combinations of software and/or hardware that perform predetermined functions.
  • This embodiment provides an aero-engine maintenance strategy optimization system based on nonparametric reinforcement learning, as shown in Figure 3, including:
  • the aero-engine model building module is used to build system state transition models and system feedback models based on collected aero-engine operating data and Bayesian networks and Gaussian processes.
  • the network initialization module is used to initialize the policy network and the value network.
  • the maintenance action decision module is used to formulate maintenance strategies based on the current system state of the aero-engine model, so as to avoid the aero-engine from entering a highly deteriorated state and reduce the system feedback cost r(s t );
  • the replay buffer module is used to store the state-action value group for each step, so that it can be extracted in batches to train the policy network and value network.
  • the model training module is used to train the policy network and the value network by using the state-action value groups extracted in batches by the replay buffer module.
  • the dynamic update module is used to dynamically update the training set, replacing data with a high degree of fit to the value network with data with a low degree of fit.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biophysics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Hardware Design (AREA)
  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Biomedical Technology (AREA)
  • Geometry (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

本发明公开了一种基于非参数强化学习的航空发动机维护策略优化方法。首先，针对稀疏航空发动机运行数据，通过贝叶斯网络与高斯过程建立航空发动机模型；然后，建立策略网络和价值网络，与航空发动机模型交互，形成状态动作价值组，存入重放缓冲区，随机采样，作为训练集；更新策略网络和价值网络，更新训练集；最后进行航空发动机维护优化策略决策。本发明首次提出针对航空发动机基于高斯过程的非参数强化学习方法，在融合系统不确定性的同时，通过动态数据更新提高整体训练数据与模型的拟合程度，提高算法采样效率；基于先验维护经验数据融合不确定性进行动作选择，提高系统的安全性与采样效率，进而解决航空发动机系统的预测性维护问题。

Description

一种基于非参数强化学习的航空发动机维护策略优化方法 技术领域
本发明涉及航空发动机运维技术领域，具体涉及一种基于非参数强化学习的航空发动机维护策略优化方法。
背景技术
航空发动机是物流客运重要的基础设施之一，合理制定维护策略对于维护航空发动机设备的正常运行、提升可靠性以及节约航空公司成本至关重要。然而，在日常航空发动机检修工作中，发动机设备复杂，工作环境相对恶劣，系统传感器具有高度的不确定性，子部件具有复杂的退化机理，且发动机系统需要进行安全性保障设计，导致航空发动机系统运维工作效率低下，资源浪费严重的情况。
为了解决上述问题，需要将系统不确定性纳入策略制定的范畴中，并提高算法效率，因此，亟待研究一种新的航空发动机运维方案，能够兼顾航空发动机维护的安全性与经济性，以实现对航空发动机的智能化运维。在这样的背景下，基于非参数强化学习的航空发动机运维策略制定方法应运而生。
现有的航空发动机维护决策方案主要包括：故障修复维护，预防性维护，基于状态的维护。其中，故障修复维护是在设备或系统发生故障后才采取的维护行动，即是一种非计划性维护，通常发生在设备出现故障或停机时，主要包括检测故障、诊断问题、修复设备以及恢复其正常运行状态；预防性维护是在设备发生故障之前采取的计划维护活动，目标是通过定期的检查、维护和修复来预防设备故障，以确保设备在预定的时间内保持高效运行；基于状态的维护是根据设备的实际工作状态和健康状况来决定维护时机的一种维护策略，使用传感器、监测设备和技术收集数据，以此预测设备的健康状况，从而决定何时进行维护。现有的航空发动机维护决策方案通常依赖于静态维护计划或当前系统状态，在要求快速响应和高安全级别的场景中存在不足，并且针对例如传感器失效等系统不确定性场景下的决策问题，很难给出正确的维护方案。
通过引入多智能体强化学习技术，与高斯过程不确定性进行结合，可以更好的解决航空发动机维护系统中的系统不确定性，结合先验维护经验数据，可以针对复杂耦合多子系统发动机进行维护决策，提高设备的运行效率。
发明内容
本发明的目的在于，针对现有技术的不足，提供一种基于非参数强化学习的航空发动机 维护策略优化方法。
为达到上述目的，本发明采用以下技术方案实现：作为本发明的第一方面，提供一种基于非参数强化学习的航空发动机维护策略优化方法，包括以下步骤：
采集航空发动机历史运行数据，建立航空发动机模型；
建立策略网络和价值网络，随机选择初始系统状态s0，制定初始维护动作a0；基于所述初始系统状态s0和初始维护动作a0，与所述航空发动机模型交互，得到一组状态动作价值组：系统状态、维护动作、系统反馈成本和下一步系统状态；建立重放缓冲区，将每一步的状态动作价值组(st,at,r(st),st+1)存入所述重放缓冲区；在所述重放缓冲区进行随机采样，作为训练集；更新策略网络和价值网络；更新训练集；判断系统运行步长是否达到设置的系统训练步长，如果达到，则停止训练，得到训练完成的网络，输出航空发动机维护优化策略，否则继续在策略网络进行动作选取，重复训练至所述运行步长达到训练步长。
进一步地，所述系统状态的参数值包括发动机每个组件i对应的停滞压力停滞温度停滞焓压力比πi，工质恒压比热工质比热比γi，效率因子ηi；所述维护动作包括大修和小修。
进一步地，所述航空发动机模型包括系统状态转移模型、系统反馈模型；
所述系统状态转移模型通过贝叶斯网络对基于马尔科夫过程得到的系统状态转移概率进行拟合，其中，贝叶斯网络参数L通过最大似然估计法表示；使用期望最大化算法进行贝叶斯网络参数寻优；得到系统状态转移函数P(st+1∣st)；
所述系统反馈模型基于系统状态st通过高斯过程拟合运行成本r1(st)；根据系统状态st下策略网络决定的维护动作，将每个组件的维护动作对应的价格之和作为维修成本r2(st)；将运行成本r1(st)和维修成本r2(st)之和作为系统状态st下的系统反馈成本r(st)；
之后，通过系统状态转移函数P(st+1∣st)，得到下一步系统状态st+1。
进一步地，所述策略网络基于系统状态st，进行动作选取，通过高斯过程得到维护动作执行概率Pnet(at)：
其中，Aπ(st,ai)为系统状态st下策略网络预测的第i个维护动作ai的优势函数；
采用最大概率估计法对动作进行选择，输出系统制定的维护动作at：
at＝argmax(Pnet(at))
所述价值网络输入系统状态st，基于高斯过程输出对系统状态st的价值评估Vπ(st)。
进一步地，所述动作选取通过ε-贪婪算法选择维护动作探索或维护策略利用来制定下一步维护动作；当ε-贪婪算法采样大于ε时，执行维护动作探索，通过策略网络在高斯过程不确定性范围中进行随机采样得到优势函数，当ε-贪婪算法采样小于ε时，执行维护策略利用，通过策略网络进行高斯过程均值输出优势函数。
进一步地，采用贝叶斯数据融合先验维护经验数据与通过策略网络得到的维护动作，辅助决策指导。
进一步地，所述更新策略网络和价值网络包括以下子步骤：基于原采样点系统状态优势函数Aπ(st,at)old与新采样点系统状态优势函数Aπ(st,at)new，扩展高斯过程拟合数据集：
输入为：输出为：
其中，Aπ(st,at)为状态st下执行动作at情况下的优势函数，计算为：
Aπ(st,at)＝r(st,at)+γVπ(st+1)-Vπ(st)
式中，γ为衰减系数，r(st,at)为系统反馈成本函数；
基于协方差矩阵构建高斯过程，作为更新后的策略网络：
其中，是观测噪声的方差，I为单位矩阵；K是由高斯过程核函数计算得到的协方差矩阵：
通过时间差分法纠正所述价值评估Vπ(st)，并进行批量训练，更新价值网络。
进一步地，所述高斯过程的核函数为：
其中，x和x′为输入向量，||x-x′||2为欧氏距离的平方，l是长度尺度参数，控制函数的平滑度，是信号方差，控制核函数的整体幅度。
进一步地，所述更新训练集包括以下子步骤：通过计算负对数似然得到训练集的各个训练数据与价值网络之间的拟合程度；对所述重放缓冲区重新进行批量随机采样，通过计算负对数似然得到重采样点与价值网络之间的拟合程度；比较，将训练集中拟合程度相较高的数据更换为拟合程度相较低的数据，使模型倾向于训练与模型拟合程度较低的数据，进而提升 模型整体可靠性。
作为第二方面，本发明还提供一种基于非参数强化学习的航空发动机维护策略优化系统，包括：
航空发动机模型构建模块，用于根据采集到的航空发动机运行数据，基于贝叶斯网络与高斯过程，构建系统状态转移模型与系统反馈模型；
网络初始化模块，用于初始化策略网络与价值网络；
维护动作决策模块，用于根据当前航空发动机模型的系统状态制定维护策略，避免航空发动机出现高度恶化状态，并降低系统反馈成本；
重放缓冲模块，用于存储每一步的状态动作价值组；
模型训练模块，用于利用重放缓冲模块批量提取的状态动作价值组对策略网络与价值网络进行训练；
动态更新模块，用于动态更新训练集，将与价值网络拟合度相较高的数据更换为拟合度相较低的数据。
本发明的有益效果是，通过构建航空发动机模型，与基于高斯过程的非参数强化学习方法，可以将数据稀疏的发动机模型通过贝叶斯网络与高斯过程进行拟合，同时保证与高精度仿真的相似度。本发明在融合系统不确定性的同时可以通过动态数据更新的方式提高整体训练数据与模型的拟合程度，提高算法采样效率，本发明还通过基于先验维护经验数据融合不确定性的动作选择方式提高系统的安全性与采样效率。效果优于现有方法，且具有通用性强、使用简便等优点。
附图说明
图1是本发明所述的智能体学习过程示意图；
图2是本发明所述的航空发动机模型建立过程示意图；
图3是本发明所述的基于非参数强化学习的航空发动机维护策略优化系统分块图；
图4是航空发动机模型的拟合效果，其中，(a)为模型预测结果与真实结果的对比，(b)为反馈模型对高压涡轮、低压涡轮、高压压缩机、低压压缩机的全局敏感性分析；
图5为使用本方法的训练结果与传统决策算法的训练结果对比图，其中，(a)为算法训练收敛结果图，(b)为该算法在训练后实际应用与航空发动机模型的五次结果箱状图；
图6为本方法针对航空发动机高压涡轮、低压涡轮、高压压缩机、低压压缩机部件的具体维护策略制定方案。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描 述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
需要说明的是，在不冲突的情况下，下述的实施例及实施方式中的特征可以相互组合。
本发明的核心技术是对稀疏数据特征的航空发动机历史数据进行数学建模，构建了航空发动机模型，针对航空发动机案例的系统不确定性与稀疏数据特性，并采用非参数强化学习求解模型。非参数强化学习是利用高斯过程的系统不确定性量化与非参数特征来将航空发动机场景中的系统不确定性融入强化学习算法总流程，并解决了真实工业场景的稀疏数据集与不确定性问题，并采用贝叶斯数据融合算法将先验维护经验融入到维护策略探索过程中，提升了算法采样效率，进而解决了航空发动机系统的预测性维护问题，并实现了融合系统不确定性的航空发动机系统运维策略制定方法。思路可以概括为：
1、构建航空发动机模型。基于发动机历史运行数据，通过贝叶斯网络对航空发动机模型的状态转移概率进行建模，并通过高斯过程对系统反馈成本进行拟合，建立可以和强化学习过程交互的航空发动机模型。
2、提出基于高斯过程的非参数强化学习方法。基于高斯过程构建强化学习算法的策略函数与价值函数，基于高斯过程不确定性进行动作的探索与利用，并使用贝叶斯数据融合算法将先验维护经验数据与强化学习决策结果融合，将发动机系统的不确定性融入到强化学习过程中，以发动机子系统恶化状态为安全性评价指标，以系统反馈成本作为维护计划的经济性评价指标，对航空发动机系统运维策略进行制定，提高了系统的整体安全性与可靠性；
3、采用模型采样点动态更新方法。基于每次模型采用的训练数据与价值网络的负对数似然，计算训练数据与价值函数模型的拟合程度，并通过动态更新的方法优先选择与模型拟合程度较低的训练点进行批量训练，提高了强化学习网络对整体模型采样点的拟合程度，提高了系统的可靠性。
4、提出基于非参数强化学习的航空发动机维护策略优化系统。
本发明提供的一种基于非参数强化学习的航空发动机维护策略优化方法，如图1所示，具体包括如下步骤：
(1)航空发动机模型的建立：建立用于算法验证的航空发动机模型，所述航空发动机模型由系统状态转移模型与系统反馈模型两部分组成。其中，系统状态转移模型负责根据当前航空发动机状态判断发动机状态恶化过程，系统反馈模型负责计算每一次发动机运行与维护的总成本，作为系统奖励。如图2所示，包括以下子步骤：
(1.1)系统状态转移模型：根据当前航空发动机状态，判断发动机状态恶化过程。由于 航空发动机运行成本高，发动机真实执飞故障数据与历史运行维护保养数据较少，并且历史航空发动机运行数据大多为稀疏数据，为了拟合出每种系统状态下不同状态之间的转移概率，首先基于稀疏发动机执飞故障数据对发动机t时刻的恶化状态st(下称系统状态st)进行定义，采用贝叶斯网络对发动机执飞故障数据的状态转移概率P(st+1∣st)进行拟合。
发动机不同组件之间的状态转移过程用等熵关系和校准的多变效率来表征：第i个组件的状态参数值为停滞压力停滞温度停滞焓压力比πi，工质恒压比热为工质比热比为γi，效率因子为ηi。不同组件之间的状态参数转移方程为：
系统的状态定义是由伽马分布进行退化模拟的。退化方程为：
其中，Δη为损伤增长，α为形状参数，β为尺度参数，Γ为伽马分布。生成100次发动机运行片段，每个片段由50步恶化过程组成，并将状态离散分为5个级别，较高的值表明恶化程度不断增加。
在贝叶斯网络的拟合过程中，航空发动机的状态转移概率基于马尔科夫过程由以下公式求得：
其中S0为转移之前的状态，S1为转移之后的状态。
贝叶斯网络参数L通过最大似然估计法表示：
其中，si为一次状态转移过程中的状态转移序列的采样，h代表了网络隐藏节点。并且，本实例通过期望最大化算法，利用Jensen不等式获得对数似然的下界：
其中，q函数为si的概率分布，且满足：∑hq(h∣si)＝1与0≤q(h∣si)≤1。
(1.2)系统反馈模型：基于稀疏历史维护保养数据，对于航空发动机恶化状态转移产生的航空发动机运行维护保养成本进行拟合。航空发动机系统在运行过程中以及从一个状态退 化或修复至另一个状态的过程中都会产生系统运行维护保养成本，并且由于系统的稀疏性，会出现缺失数据等问题，因此采用高斯过程对航空发动机系统的系统反馈成本进行拟合，高斯过程作为非参数随机过程，在面对稀疏系统的数据缺失问题相对其他拟合算法有着较好的效果。
采集发动机各个组件的历史运行成本数据，将航空发动机系统状态st与运行成本r1(st)使用x(i),y(i)进行表示，高斯过程的核函数为：
该函数用于衡量输入数据的相似性，其中x和x′为输入向量，||x-x′||2为欧氏距离的平方。l是长度尺度参数，控制函数的平滑度。是信号方差，控制核函数的整体幅度。对给定数据集(X,Y)，高斯过程的边际似然为：
其中，K为协方差矩阵，n为训练数据点的数量。通过高斯过程预测出的运行成本的均值μ与方差σ为：
μ＝k(x*,x)TK-1y
σ＝k(x*,x*)-k(x*,x)TK-1k(x*,x)
其中，x*为待预测的数据点。基于高斯过程预测均值，可以得出发动机的运行成本r1(st)。
根据各个组件在该系统状态下策略网络决定的维护动作，以及该维护动作对应的维修价格(依照现行的工厂报价)，将各个组件的维修价格加总，作为发动机在系统状态st下的维修成本r2(st)。
发动机的运行成本r1(st)与维修成本r2(st)之和作为系统反馈成本r(st)。
(1.3)航空发动机系统整合与初始化：本实例提出的航空发动机系统的数学模型由系统状态转移模型与系统反馈模型组成，该模型在运行过程中接受系统的现有状态，根据系统的当前状态与维护动作来给出下一步状态与系统反馈成本。航空发动机模型在周期运行后，航空发动机系统的根据步骤(1.1)得出的系统转移函数P(st+1∣st)基于本周期状态st和采取的维护动作at判断下个周期的航空发动机状态st+1，并通过系统反馈模型计算本周期下状态st对应的发动机运行成本r1(st)与维修成本r2(st)，将运行成本与维修成本整合作为系统反馈成本r(st)。
按如上方法，即得到了航空发动机模型与初始化状态。本实例采用贝叶斯网络与高斯过程这类非参数网络进行建模，可以针对其稀疏数据集的特性进行效果较好的拟合。
(2)初始化决策系统网络并制定策略：建立并初始化算法策略函数Pnet与价值函数Vnet，并基于步骤(1)的航空发动机模型的初始状态通过策略网络与经验辅助制定维护策略。包括以下子步骤：
(2.1)基于步骤(1)中的航空发动机模型系统状态转移函数P(st+1∣st)与系统反馈R(st)，针对该发动机的维护策略制定，本实例通过建立强化学习策略网络Pnet来制定系统维护策略。在策略网络Pnet输入系统状态st，得到系统制定的维护动作执行概率P(at)，将系统状态st与系统的优势函数Aπ(st,at)使用x,y进行表示，建立高斯过程：
高斯过程的核函数为：
该函数用于衡量输入数据的相似性,其中x和x′为输入向量，||x-x′||2为欧氏距离的平方。l是长度尺度参数，控制函数的平滑度。是信号方差，控制核函数的整体幅度。对给定数据集(X,Y),高斯过程的边际似然为：
其中，K为协方差矩阵，n为训练数据点的数量。通过高斯过程预测出的系统反馈的均值μ与方差σ为：
μ＝k(x*,x)TK-1y
σ＝k(x*,x*)-k(x*,x)TK-1k(x*,x)
其中，x*为待预测的数据点。
策略网络Pnet基于系统状态，进行动作选取。动作选取是根据ε-贪婪算法选择动作探索或策略利用两种方式得到每个维护动作的优势函数，来制定下一步维护动作。ε为一个0-1之间的数，对0-1进行随机采样，若采样点大于ε，则采用动作探索，即通过策略网络在高斯过程不确定性范围中进行随机采样得到优势函数；反之，则采用策略利用，通过策略网络进行高斯过程均值输出优势函数。
动作探索是通过策略网络在高斯过程不确定性范围σ中进行随机采样得出：
σ＝k(x*,x*)-k(x*,x)TK-1k(x*,x)
动作探索可以在一定范围内实现随机动作探索，寻找更优秀的维护动作。
而策略利用则是利用通过训练得到的策略网络进行高斯过程均值输出，继承之前训练的结果。
根据高斯过程输出的每个维护动作的优势函数计算维护动作执行概率，维护动作的执行概率为：
Aπ(st,ai)即为状态st下策略网络预测的第i个维护动作ai的优势函数；
采用最大概率估计法对动作进行选择：
at＝argmax(Pnet(at))
at即为策略网络Pnet针对当前系统状态制定的维护动作执行概率进行优选后的具体维护动作，作为策略网络Pnet的输出。其中，所述维护动作在航空领域包括大修和小修。
在得出策略网络的维护动作之后，采用贝叶斯数据融合将先验维护经验数据与策略网络进行结合，结合先验维护经验数据进行指导，提高系统的安全性并且提高采样效率，加快算法收敛。
其中，Pa为策略网络的决策动作概率，Pae为先验维护经验数据的动作执行概率，w为先验维护经验对决策的影响权重。
(2.2)强化学习过程需要价值网络Vnet对当前状态进行价值评定，在一个马尔科夫过程中，价值函数Vnet的意义为当前状态s的反馈值期望总和：
Vπ(s)＝Eπ[Gt∣St＝s]
其中，Gt为系统反馈。在价值网络Vnet输入系统状态st，得到系统制定的动作价值估计值Vt。将系统状态st与系统动作的动作价值估计值Vt使用x(i),y(i)进行表示，建立高斯过程，高斯核函数与边际似然与步骤(2.1)一致，高斯过程输出为价值网络Vnet对系统状态的价值评估Vπ(s)。
(2.3)基于步骤(2.1)与步骤(2.2)生成的强化学习策略网络Pnet与价值网络Vnet，对步骤(1)中建立的航空发动机模型进行随机状态初始化，随机选择系统状态s0，并输入至策略网络Pnet，制定第一步维护动作a0。
(3)策略迭代与网络训练：基于步骤(2)制定的初始维护动作a0与初始系统状态s0，与步骤(1)建立的航空发动机模型交互，得出一组状态动作价值组；建立重放缓冲区，将每一步的状态动作价值组(st,at,R(st),st+1)存入重放缓冲区并进行采样作为训练集；更新所述策略网络Pnet与价值网络Vnet；通过负对数似然对所述训练集进行动态更新。
(3.1)基于步骤(2.3)得到的初始系统状态s0与初始维护动作a0，与航空发动机系统进行交互，得到系统的下一步状态与状态转移过程的成本。将这一过程整理为一个状态动作组(st,at,R(st),st+1)。由于策略网络Pnet与价值网络Vnet都需要批量数据进行训练，因此需 要建立一个数据重放缓冲区，用于储存每一次的状态动作组，并批量提取出来用于训练策略网络与价值网络。将该步的状态动作组存入重放缓冲区中。
(3.2)基于步骤(3.1)建立的重放缓冲区，批量提取训练数据，并根据价值网络对在策略π指导下状态的预测Vπ(st)，计算优势函数Aπ：
Aπ(st,at)＝r(st,at)+γVπ(st+1)-Vπ(st)
其中，γ为衰减系数，r(st,at)为系统反馈成本函数。
通过优势函数，基于原采样点(即上周期采样点)系统状态优势函数Aπ(st,at)old与新采样点(即本周期采样点)系统状态优势函数Aπ(st,at)new，扩展高斯过程拟合数据集，更新策略网络：
扩展后的高斯过程输入为：输出为：
Aπ(st,at)为状态st下执行动作at情况下的优势函数，扩展后的整体协方差矩阵：
基于协方差矩阵构建高斯过程，作为更新后的策略网络：
其中，K是由高斯过程核函数计算得到的协方差矩阵，是观测噪声的方差，I为单位矩阵。
通过时间差分法TD-Error对价值函数Vπ(st)进行纠正，进行批量训练，并进行高斯过程拟合：
其中，p(st+1∣st,at)为状态转移函数，r(st,at)为系统反馈成本函数。
(3.3)基于步骤(3.2)批量提取的训练数据，计算每一个训练数据与价值网络Vnet之间的负对数似然：
其中，e为批量提取的训练数据个数，σi与μi分别为第i个训练数据的价值函数预测的方差与均值。yi为使用时间差分法纠正后的系统状态价值Vπ(si)，负对数似然表示了每一个训练数据与价值函数模型之间的拟合程度。为了从重放缓冲区中选取更多与模型拟合程度交叉的训练数据进行训练，因此，本实例同时对重放缓冲区进行重新采样(批量随机采样)，重采样 点合集为e*，并计算e*的负对数似然，负对数似然越大代表采样点与模型的拟合程度较差，替换概率Pu计算了e的负对数似然与e和e*的负对数似然之和之比，作为训练数据替换概率，进行训练数据点替换。概率越小，则倾向于保留当前训练数据点e，反之则倾向于将当前训练数据点e替换为再次采样点e*，使模型倾向于训练与模型拟合程度较低的数据，进而提升模型整体可靠性。
其中e为训练使用的采样点合集，e*为再次采样的采样点合集。
(3.4)判断系统运行步长是否达到了预先设定的系统训练步长：如果达到训练步长，则停止训练；否则则继续进行动作选取，并重复步骤(3)，直至运行步长达到了训练步长。
实施例1：
在一台配备Intel(R)Xeon(R)Gold 6330 CPU中央处理器，NVidia GTX3090图形处理器及128GB内存的机器上实现本发明的实施实例。采用上述实施方式，得到了图4～6所示的结果。
针对通用公司GE90型号发动机进行航空发动机建模，拟合出发动机的状态转移矩阵与系统反馈，拟合结果如图4所示，图4(a)展示了模型预测结果与真实结果的对比，纵坐标为预测值，横坐标为真实值。圆圈点为真实数据点，实心加号点为预测点，本方法可以很精准的预测出航空发动机的状态转移过程与系统反馈，图4(b)展示了系统反馈模型对高压涡轮、低压涡轮、高压压缩机、低压压缩机的全局敏感性分析。图5(a)展示了本方法相对于基于规则的维护方法、深度价值强化学习算法、演员-评论家强化学习算法的效果，最下方的曲线为本方法的收敛结果，上方的虚线为基于规则的维护算法，非参数强化学习相对于基于规则的维护算法可以大幅度减少系统反馈成本。上方的两条收敛曲线分别为深度价值强化学习算法、演员-评论家强化学习算法。这两种算法在经历1000次训练后的收敛结果仍旧不如非参数强化学习训练100次后的结果，并且这两种对比算法在收敛过程中均体现出较强的震荡，说明这两种方法的采样效率很低，无法稳定训练，而非参数强化学习在训练过程中则相对平稳，体现出较高的采样效率从结果可以看出，非参数强化学习相较传统强化学习方法提高了一百倍以上的数据利用效率。本方法在融合系统不确定性的同时，通过动态数据更新提高整体训练数据与模型的拟合程度，提高算法采样效率。图5(b)展示了本方法与基于规则的维护方法、深度价值强化学习算法、演员-评论家强化学习算法相比在训练后实际应用5次的平均效果箱装图，最左侧为本方法的箱状图结果。非参数强化学习方法在系统反馈成本上限、下限与均值上远低于对比方法，并且演员-评论家模型还出现了决策发散的情况。可以看出本方法可以相对稳定的针对航空发动机制定维护策略图6展示了本方法针对高压涡轮、低 压涡轮、高压压缩机、低压压缩机制定的具体维护策略，圆圈为小修，倒三角为大修。结果表明，本方法可以精准的对航空发动机进行数学建模，并且正确的针对航空发动机的不同部件制定合理的维护策略。
在本实施例中还提供了一种基于非参数强化学习的航空发动机维护策略优化系统，该系统用于实现上述实施例。以下所使用的术语“模块”、“单元”等可以实现预定功能的软件和/或硬件的组合。
本实施例提供的一种基于非参数强化学习的航空发动机维护策略优化系统，如图3所示，包括：
航空发动机模型构建模块，用于根据采集到的航空发动机运行数据，基于贝叶斯网络与高斯过程，构建系统状态转移模型与系统反馈模型；
网络初始化模块，用于初始化策略网络与价值网络；
维护动作决策模块，用于根据当前航空发动机模型的系统状态制定维护策略，避免航空发动机出现高度恶化状态，并降低系统反馈成本r(st)；
重放缓冲模块，用于存储每一步的状态动作价值组，以便批量提取出来训练策略网络与价值网络；
模型训练模块，用于利用重放缓冲模块批量提取的状态动作价值组对策略网络与价值网络进行训练；
动态更新模块，用于动态更新训练集，将与价值网络拟合度相较高的数据更换为拟合度相较低的数据。
需要指出的是，上述实施例仅用于说明本发明的优势和具体步骤，而不限定本发明的使用范围。本领域的技术人员在本发明的基础上进行其他形式的模拟均属于本发明的保护范围之内。

Claims (10)

  1. 一种基于非参数强化学习的航空发动机维护策略优化方法，其特征在于，包括以下步骤：
    采集航空发动机历史运行数据，建立航空发动机模型；
    建立策略网络和价值网络，随机选择初始系统状态s0，制定初始维护动作a0；基于所述初始系统状态s0和初始维护动作a0，与所述航空发动机模型交互，得到一组状态动作价值组：系统状态、维护动作、系统反馈成本和下一步系统状态；建立重放缓冲区，将每一步的状态动作价值组(st,at,r(st),st+1)存入所述重放缓冲区；在所述重放缓冲区进行随机采样，作为训练集；更新策略网络和价值网络；更新训练集；判断系统运行步长是否达到设置的系统训练步长，如果达到，则停止训练，得到训练完成的网络，输出航空发动机维护优化策略。
  2. 根据权利要求1所述的方法，其特征在于，所述系统状态的参数值包括发动机每个组件i对应的停滞压力停滞温度停滞焓压力比πi，工质恒压比热工质比热比γi，效率因子ηi；所述维护动作包括大修和小修。
  3. 根据权利要求1所述的方法，其特征在于，所述航空发动机模型包括系统状态转移模型、系统反馈模型；
    所述系统状态转移模型通过贝叶斯网络对基于马尔科夫过程得到的系统状态转移概率进行拟合，其中，贝叶斯网络参数L通过最大似然估计法表示；使用期望最大化算法进行贝叶斯网络参数寻优；得到系统状态转移函数P(st+1∣st)；
    所述系统反馈模型基于系统状态st通过高斯过程拟合运行成本r1(st)；根据系统状态st下策略网络决定的维护动作，将每个组件的维护动作对应的价格之和作为维修成本r2(st)；将运行成本r1(st)和维修成本r2(st)之和作为系统状态st下的系统反馈成本r(st)；
    之后，通过系统状态转移函数P(st+1∣st)，得到下一步系统状态st+1。
  4. 根据权利要求1所述的方法，其特征在于，所述策略网络基于系统状态st，进行动作选取，通过高斯过程得到维护动作执行概率Pnet(at)：
    其中，Aπ(st,ai)为系统状态st下策略网络预测的第i个维护动作ai的优势函数；
    采用最大概率估计法对动作进行选择，输出系统制定的维护动作at：
    at＝argmax(Pnet(at))
    所述价值网络输入系统状态st，基于高斯过程输出对系统状态st的价值评估Vπ(st)。
  5. 根据权利要求4所述的方法，其特征在于，所述动作选取通过ε-贪婪算法选择维护动作探索或维护策略利用来制定下一步维护动作；当ε-贪婪算法采样大于ε时，执行维护动作探索，通过策略网络在高斯过程不确定性范围中进行随机采样得到优势函数，当ε-贪婪算法采样小于ε时，执行维护策略利用，通过策略网络进行高斯过程均值输出优势函数。
  6. 根据权利要求4所述的方法，其特征在于，采用贝叶斯数据融合先验维护经验数据与通过策略网络得到的维护动作，辅助决策指导。
  7. 根据权利要求1所述的方法，其特征在于，所述更新策略网络和价值网络包括以下子步骤：基于原采样点系统状态优势函数Aρ(st,at)old与新采样点系统状态优势函数Aπ(st,at)new，扩展高斯过程拟合数据集：
    输入为：输出为：
    其中，Aπ(st,at)为状态st下执行动作at情况下的优势函数，计算为：
    Aπ(st,at)＝r(st,at)+γVπ(st+1)-Vπ(st)
    式中，γ为衰减系数，r(st,at)为系统反馈成本函数；
    基于协方差矩阵构建高斯过程，作为更新后的策略网络：
    其中，是观测噪声的方差，I为单位矩阵；K是由高斯过程核函数计算得到的协方差矩阵：
    通过时间差分法纠正所述价值评估Vπ(st)，并进行批量训练，更新价值网络。
  8. 根据权利要求3-7任一项所述的方法，其特征在于，所述高斯过程的核函数为：
    其中，x和x′为输入向量，||x-x′||2为欧氏距离的平方，l是长度尺度参数，控制函数的平滑度，是信号方差，控制核函数的整体幅度。
  9. 根据权利要求1所述的方法，其特征在于，所述更新训练集包括以下子步骤：通过计算负对数似然得到训练集的各个训练数据与价值网络之间的拟合程度；对所述重放缓冲区重新进行批量随机采样，通过计算负对数似然得到重采样点与价值网络之间的拟合程度；比较，将训练集中拟合程度相较高的数据更换为拟合程度相较低的数据。
  10. 一种基于非参数强化学习的航空发动机维护策略优化系统，其特征在于，包括：
    航空发动机模型构建模块，用于根据采集到的航空发动机运行数据，基于贝叶斯网络与高斯过程，构建系统状态转移模型与系统反馈模型；
    网络初始化模块，用于初始化策略网络与价值网络；
    维护动作决策模块，用于根据当前航空发动机模型的系统状态制定维护策略，避免航空发动机出现高度恶化状态，并降低系统反馈成本；
    重放缓冲模块，用于存储每一步的状态动作价值组；
    模型训练模块，用于利用重放缓冲模块批量提取的状态动作价值组对策略网络与价值网络进行训练；
    动态更新模块，用于动态更新训练集，将与价值网络拟合度相较高的数据更换为拟合度相较低的数据。
PCT/CN2024/109213 2024-08-01 2024-08-01 一种基于非参数强化学习的航空发动机维护策略优化方法 Pending WO2025246013A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2024/109213 WO2025246013A1 (zh) 2024-08-01 2024-08-01 一种基于非参数强化学习的航空发动机维护策略优化方法
JP2025549928A JP2026521822A (ja) 2024-08-01 2024-08-01 ノンパラメトリック強化学習に基づく航空エンジンメンテナンス方策の最適化方法

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2024/109213 WO2025246013A1 (zh) 2024-08-01 2024-08-01 一种基于非参数强化学习的航空发动机维护策略优化方法

Publications (1)

Publication Number Publication Date
WO2025246013A1 true WO2025246013A1 (zh) 2025-12-04

Family

ID=97869437

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/109213 Pending WO2025246013A1 (zh) 2024-08-01 2024-08-01 一种基于非参数强化学习的航空发动机维护策略优化方法

Country Status (2)

Country Link
JP (1) JP2026521822A (zh)
WO (1) WO2025246013A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN121303242A (zh) * 2025-12-11 2026-01-09 浙江大学 面向动态动作空间的校准终身强化学习方法及系统
CN121387561A (zh) * 2025-12-20 2026-01-23 中国人民解放军联勤保障部队第九六四医院 面向异构任务的多功能无人机资源自适应分配方法
CN121607598A (zh) * 2026-02-02 2026-03-06 燕山大学 一种铝车轮压铸模温闭环调控系统及方法
CN122199880A (zh) * 2026-05-18 2026-06-12 上海营邑城市规划设计股份有限公司 基于虚拟场景视觉导航的地下管线生成方法、装置和设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090164405A1 (en) * 2007-12-21 2009-06-25 Honda Motor Co., Ltd. Online Sparse Matrix Gaussian Process Regression And Visual Applications
CN116796425A (zh) * 2023-01-17 2023-09-22 南京林业大学 一种基于sfa-gmm-bid的航空发动机动态健康监测方法
CN116843023A (zh) * 2023-06-08 2023-10-03 南京航空航天大学 一种基于深度强化学习的航空发动机旋转部件健康参数更新方法
CN116992571A (zh) * 2023-08-22 2023-11-03 华中科技大学 数模融合驱动的航空发动机剩余使用寿命预测方法及系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090164405A1 (en) * 2007-12-21 2009-06-25 Honda Motor Co., Ltd. Online Sparse Matrix Gaussian Process Regression And Visual Applications
CN116796425A (zh) * 2023-01-17 2023-09-22 南京林业大学 一种基于sfa-gmm-bid的航空发动机动态健康监测方法
CN116843023A (zh) * 2023-06-08 2023-10-03 南京航空航天大学 一种基于深度强化学习的航空发动机旋转部件健康参数更新方法
CN116992571A (zh) * 2023-08-22 2023-11-03 华中科技大学 数模融合驱动的航空发动机剩余使用寿命预测方法及系统

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN121303242A (zh) * 2025-12-11 2026-01-09 浙江大学 面向动态动作空间的校准终身强化学习方法及系统
CN121387561A (zh) * 2025-12-20 2026-01-23 中国人民解放军联勤保障部队第九六四医院 面向异构任务的多功能无人机资源自适应分配方法
CN121607598A (zh) * 2026-02-02 2026-03-06 燕山大学 一种铝车轮压铸模温闭环调控系统及方法
CN122199880A (zh) * 2026-05-18 2026-06-12 上海营邑城市规划设计股份有限公司 基于虚拟场景视觉导航的地下管线生成方法、装置和设备

Also Published As

Publication number Publication date
JP2026521822A (ja) 2026-07-02

Similar Documents

Publication Publication Date Title
Samadian et al. Application of data-driven surrogate models in structural engineering: a literature review
Bao et al. A Deep Transfer Learning Network for Structural Condition Identification with Limited Real‐World Training Data
CN114117840B (zh) 一种基于仿真和试验数据混合驱动的结构性能预测方法
CN118657404B (zh) 一种基于多模态的电网健康评估与分析方法
Tan et al. Systemic condition-based maintenance optimization under inspection uncertainties: A customized multiagent reinforcement learning approach
CN114297779B (zh) 一种飞行器气动力系数快速预测方法、系统及设备
CN116448419A (zh) 基于深度模型高维参数多目标高效寻优的零样本轴承故障诊断方法
CN120335425B (zh) 一种工商业液冷储能系统的实时监测与故障响应控制装置
Zhao et al. A novel remaining useful life prediction method based on gated attention mechanism capsule neural network
CN102222313B (zh) 基于核主成分分析的城市演化模拟元胞模型处理方法
CN121031378B (zh) 一种量化不确定性的航空发动机模型贝叶斯优化方法
CN112001110A (zh) 一种基于振动信号空间时时递归图卷积神经网络的结构损伤识别监测方法
US20250156610A1 (en) Cfd automation method for optimal flow analysis over blades using reinforcement learning, cfd flow analysis method for blades, and cfd flow analysis device for blades
CN110162895B (zh) 一种两阶段的高能效船型优化设计方法
CN121435775A (zh) 一种数字孪生驱动的桥梁全生命周期损伤预测与评估方法及系统
CN117469683A (zh) 一种锅炉受热面智能吹灰方法、装置、电子设备及介质
CN118535968A (zh) 多源数据融合下基于pso-lstm算法的公路桥梁结构损伤识别方法
CN118586234A (zh) 基于自适应半代理模型的涡轮叶片多模式模糊失效状态疲劳寿命可靠性分析方法
CN121902636A (zh) 参数的预测方法及装置、电子设备和存储介质
CN116303786B (zh) 一种基于多维数据融合算法的区块链金融大数据管理系统
CN121881203A (zh) 一种基于数字孪生的水利工程运维管理系统
Luo et al. A novel method for remaining useful life prediction of roller bearings involving the discrepancy and similarity of degradation trajectories
Wang et al. Cbla: Empowering virtual sensor nodes with zero deployment costs for shm systems
CN113947030B (zh) 基于梯度下降灰色Markov模型的装备器材需求预测方法
CN119026460B (zh) 一种基于非参数强化学习的航空发动机维护策略优化方法

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2025549928

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24941291

Country of ref document: EP

Kind code of ref document: A1