CA2966306C - Tool for validating a system for monitoring an aircraft engine - Google Patents

Abstract

The invention relates to a tool for validating a system for monitoring (3) at least one item of equipment of an aircraft engine (5), comprising furthermore: processing means (11) configured to gather observation data relating to said item of equipment, analysis means (12) configured to calculate a current value of at least one quality indicator regarding a current quantity of observation data, analysis means (12) configured to estimate the probability that said current value of the quality indicator reaches a predetermined reliability criterion, thus forming a probabilistic reliability law, and analysis means (12) configured to estimate on the basis of said probabilistic reliability law a minimum quantity of observation data on the basis of which the value of the quality indicator reaches a predetermined reliability criterion with a probability greater than a predetermined value.

Description

VALIDATION TOOL FOR AN AIRCRAFT ENGINE MONITORING SYSTEM
TECHNICAL AREA
The present invention relates to the field of systems for monitoring a engine of an aircraft and more particularly, of a tool for validating a system of monitoring of aircraft engine equipment.
PRIOR ART
Monitoring systems are used to verify proper operation various aircraft engine equipment. There is, for example, a system of monitoring to analyze motor behavior during the process ignition, another to analyze the trajectory of gases, yet another to detect clogging filters, and another to analyze oil consumption, etc.
All of these monitoring systems improve safety and aircraft engine reliability. In particular, they make it possible to avoid or limit stop in flight, reduce flight delays or cancellations, and more in particular, facilitate the engine maintenance by anticipating failures and identifying components faulty or defective.
Currently, there is a tool for designing a monitoring system based on indicators that are compared to thresholds defined by specifications. This tool is described in the Applicant's French patent application FR2957170.
The validation of a monitoring system requires carrying out tests by example on a test bench in order to collect a large amount of data. He must then deploy a lot of resources and time to carry out these tests and to more, the big one number of collected data can require a significant calculation time.
In addition, it may happen that the level of validation of a system of surveillance is different from that of another surveillance system. This can complicate the analysis data from various engine monitoring systems.
The object of the present invention is to propose a tool for validating a aircraft engine equipment monitoring system allowing to optimize the

2 amount of data needed for validation thus reducing costs and the time of development while increasing the reliability of the surveillance system.
DISCLOSURE OF THE INVENTION
The present invention is defined by a validation tool for a system of monitoring of at least one item of equipment of an aircraft engine, comprising:
- processing means configured to collect data observations relating to said equipment, - analysis means configured to calculate a current value of at least least one quality indicator on a current amount of data observation collected by the processing means, - analysis means configured to estimate the probability that said current value of the quality indicator reaches a reliability criterion predetermined, thus forming a probabilistic law of reliability estimated on a set of values of the quality indicator relating to a corresponding set of quantities of data observation, and - analysis means configured to estimate from said law probabilistic reliability a minimum amount of observational data to from which the value of the quality indicator reaches a reliability criterion predetermined with a probability greater than a predetermined value, - test means configured to evaluate a validation of said monitoring system by applying a set of quality indicators on said minimum amount of observational data relating to said equipment.
This allows you to know when to stop collecting data for the evaluation of the surveillance system and therefore allows to reduce the costs tests.
Advantageously, said predetermined value is the complement of a predefined error as acceptable.
Advantageously, the analysis means are configured to calculate the current value of a quality indicator by applying a technique of validation

3 crossed on said current quantity of observation data. the said technique of cross-validation can be selected from the following techniques:
bootstrap, K-fold, leave-one-out.
Advantageously, the set of quality indicators comprises the following indicators: false alarm rate, detection rate, rate of location.
Advantageously, the analysis means are configured to apply a regression technique on said set of values of the indicator of quality for determining an approximation function representative of said law probabilistic of reliability depending on the amount of observational data.
According to a feature of the present invention, for an indicator of quality corresponding to false alarm rates, said function of approximation in function of the quantity n of observation data is expressed by the following relationship:
b cf(n)= a+¨+ log(n) vn where a,b,c are regression constants.
Advantageously, the test means are configured to evaluate a validation of said surveillance system before its installation on an aircraft applying the set of quality indicators on a quantity of observation data collected on a test bench and/or on a fleet of aircraft engines in operation.
this allows to validate a generic monitoring system suitable for monitoring a series engine.
Advantageously, the test means are configured to continue the validation and adjustment of said monitoring system after installation on an engine standard by applying the set of quality indicators to a quantity of data observations collected in flight. This makes it possible to specialize the system of monitoring for that it is adapted to the specific use of the engine on which it is installed knowing that the behavior of the engine can depend on the missions, the journeys, the maintenance, etc.
The invention also relates to a system for monitoring at least one equipment of an aircraft engine designed by the design tool according to one any of the preceding characteristics, said system being configured to receive Datas

4 observations specific to said equipment and to deliver a result diagnosing the state of said equipment.
The invention also relates to a method for validating a system of monitoring of at least one item of equipment of an aircraft engine, comprising stages of test to assess a validation of said monitoring system by applying a set of quality indicators on a volume of relative observation data audit equipment, said method further comprising the following steps:
- collect observation data relating to said equipment, - calculating a current value of at least one quality indicator on a current amount of observational data collected by means of treatment, - estimate the probability that said current value of the indicator of quality reaches a predetermined reliability criterion, thus forming a law probabilistic of reliability estimated on a set of values of the quality indicator relative to a set corresponding amounts of observation data, and - estimating from said probabilistic reliability law a quantity minimum observational data from which the value of the quality indicator reaches a predetermined reliability criterion with a higher probability to a value predetermined, said minimum quantity of observation data corresponding audit volume of observational data intended to be used to assess the validation of said Surveillance system.
BRIEF DESCRIPTION OF THE DRAWINGS
Other features and advantages of the tool and the method according to the invention will emerge better on reading the description given below, as indicative but no limiting, with reference to the attached drawings in which:
- Fig. 1 schematically illustrates a tool for validating a system for monitoring equipment of an aircraft engine, according to a mode of realization of the invention;

WO 2016/07160

5 PCT/FR2015/052905 - Fig. 2 is a flowchart illustrating a method for determining a minimum number of observation and validation data for a system of monitoring of at least one item of equipment of an aircraft engine, according to a mode of realization of the invention;
5 - the Fig. 3 is a graph representing curves relating to a law theoretical reliability and its approximation function, according to the invention;
- Fig. 4 is a graph representing observation measurements around the curve relating to the theoretical law of reliability;
- Fig. 5 schematically illustrates a tool for validating a system for monitoring equipment of an aircraft engine, according to a first mode preferred embodiment of the invention; and - Fig. 6 schematically illustrates a tool for validating a system for monitoring equipment of an aircraft engine, according to a second mode preferred embodiment of the invention.
DETAILED DISCUSSION OF PARTICULAR EMBODIMENTS
The present invention proposes to implement a tool allowing to estimate and anticipate the quantity of data necessary to carry out the validation of a Surveillance system.
Fig. 1 schematically illustrates a validation tool 1 of a system monitoring 3 of equipment of an aircraft engine 5, according to a mode of achievement preferred of the invention.
3 monitoring system can consist of 31-34 modules and each module performs a particular task using a specific algorithm.
Such a system monitoring is described in patent application FR2957170 of the plaintiff and may include a data acquisition module 31, a normalization 32, a diagnostic or anomaly detection module 33, a module for classification 34 to identify faults, etc.

6 In execution, the data acquisition module 31 is configured to receive specific measurements or data 71 acquired by sensors on engine 5 aircraft or by an on-board computer and to deliver data or variables specific giving indications on physical or logical elements of engine equipment 5. These data 71 are extracted from the raw measurements temporal (temperatures, pressures, fuel flow, shaft rotations, etc.).
By way of example, the specific data may correspond to the delay necessary for a shaft of the motor to reach maximum acceleration after each starting the engine, the temperature gradient of the exhaust gases from the engine, the ignition time, valve opening time, speed trend, etc The normalization module 32 is configured to receive the data specific from the acquisition module 31 to standardize these data specific and to deliver standardized data independent of from context.
From these standardized data, it is now a question of diagnosing an anomaly then to deduce a specific failure and possibly the making up physics involved.
Thus, the anomaly detection module 33 is configured to receive the standardized data coming from the standardization module 32 for build a signature representative of the behavior of the engine 5 and to diagnose whether the signature reveals an anomaly. In addition, the anomaly detection module 33 is configure to deliver a score representative of the signature. According to score value by compared to a predefined threshold, the detection module 33 is configured to generate or not an alarm indicating that an anomaly is detected.
The classification module 34 is configured to deliver a measurement identification of failures. After identifying the failures in calculating by example, for each a probability of occurrence, the monitoring system 3 can use the latter to detect faulty components.
Thus, depending on the type of application, the monitoring system 3 is configured to perform several tasks that may include data acquisition, the

7 standardization of data, detection of anomalies, and possibly the classification abnormalities detected.
However, the monitoring system 3 must undergo a qualification phase Where validation before it is put into operation.
The validation tool is therefore used to verify the validation of the system of surveillance. In accordance with the invention, the validation tool 1 comprises means of processing 11, analysis means 12 and test means 13. These means 11-13 are implemented by devices that are usually found in a computer: a central processing and calculation unit, storage means, peripheral devices input as well as output devices.
The processing means 11 are configured to collect data observation 7 relating to the engine equipment 5 to be monitored. Those data observation 7 include specific measurements 71 acquired by the sensors on the aircraft engine 5 or by an on-board computer, and intended to be used by the monitoring system 3 to detect faults or anomalies. Otherwise, the data observation 7 can also include data from the tests carried out on a bench tests or recordings made on an operational engine or possibly simulation data.
Furthermore, the test means 13 are configured to evaluate the validation of the monitoring system by applying a set of KPI quality indicators (Key Performance Indicators) on a volume of observation data 7 relating to this equipment.
More particularly, the test means 13 apply an indicator or several KPI quality indicators on the result of the system of monitoring 3 to verify fault detection. This gives a score that validates the system of monitoring 3 on the volume of observation data 7.
Advantageously, the set of KPI quality indicators includes a measurement of a POD (Probability Of Detection) detection rate, a measurement of a rate false alarm PFA (Probability of False Alarm), a measure of a rate of PCL localization (Probability of Class Location), etc. Note that each of the indicators quality

8 can be used independently of another indicator or in combination depending on the type application of the surveillance system 3.
POD detection rate, also called detection power, is the probability of detecting a fault when the monitored equipment has really a mistake. It can be estimated as the ratio of the number of faults detected over the number number of faults. Thus, the POD detection rate makes it possible to validate with a high reliability the result of the monitoring system 3.
The PFA false alarm rate is defined as the probability that the monitored equipment is healthy when the monitoring system 3 detects a mistake. the PFA false alarm rate can be calculated from Bayesian rules.
Let P(detected) the marginal probability that an anomaly will be detected by the monitoring 3, and P(healthy) the marginal probability that the equipment is healthy. So the rate of false alarm PFA is the probability that the equipment is a posteriori healthy knowing that a fault is detected.
The PFA false alarm rate is a very important criterion in the field aeronautics. Indeed, the event that the equipment is healthy when the system of surveillance 3 detects a fault is a phenomenon that must be limited because that he can permanently modify the image of credibility that the user can have system monitoring 3. Thus, it is very advantageous that the false alarm rate PFA is low.
The data quality indicator is a measure of a location rate PCL
(Probability of Class Location) defined as the probability that a location is good when an anomaly of the equipment is observed.
The PCL localization rate, which is information about the quality of classification, makes it possible to validate with great efficiency the result of the system of monitoring 3. This makes it possible to correctly locate the precise element (for example, alternator, cable, harness, etc.) of equipment with an anomaly.
Generally speaking, the POD detection rate is used to detect whether the equipment has an abnormal element, and then the location rate PCL indicates the efficiency with which one can locate or identify this element.

9 At least part of the set of quality indicators is applied on the observation data 7 collected by the processing means 11 for validate the monitoring system 3.
However, the accuracy of quality indicators depends on the amount of observation data 7. Thus, it is advantageous to know from how many observation data 7 the precision of the indicators is suitable for of the predetermined quality requirements are met by these indicators.
Indeed, Fig. 2 is a flowchart illustrating a method for determining a minimum number of observation and validation data for a system of monitoring of at least one item of equipment of an aircraft engine, according to the invention.
Step El concerns the collection of a current amount of data observation 7 by the processing means 11.
At step E2, at each current quantity (or current number) of data observation 7 collected by the processing means 11, the means analysis 12 are configured to calculate a current value of at least one indicator of quality.
In other words, for each current number n of observation data 7, the analysis means 12 calculate the current value KPI(n) of an indicator of quality KPIs.
Advantageously, the current KPI(n) value of a quality indicator is calculated by applying a cross-validation technique on the quantity current n of sighting data 7.
The cross-validation technique can be selected from the techniques following: bootstrap , K-fold , and leave-one-out . It consists of generate new datasets from the main set of current size n. For every game of data, the analysis means 12 are configured to carry out an experiment i and for calculate a corresponding value of the KPI.
More specifically, the bootstrap consists of drawing with discount n observations for calibration and calculate the KPI on the initial game.
The K-fold consists of creating calibration datasets by selection of a predetermined proportion of observational data and to test on the rest. This operation is repeated K times.

The leave-one-out calibrates the application on n-1 observation data and the tests on the last one, we obtain n elementary results.
Cross-validation thus makes it possible to generate a population of values Xi =
KPI(nexperience 0 obtained for each experiment. This population of values 5 (Xi)1=1...K gives an empirical representation of the current value KPI(n).
In step E3, the analysis means 12 are configured to estimate a probability P (K PI (n) EI) = P(n) defined as being the probability that the value current KPI(n) of the quality indicator KPI reaches a criterion of reliability predetermined, the latter corresponding to a predetermined interval I.

10 Thus, the analysis means 12 iteratively form (ie, for n=
Ni, ....,N,i), a probabilistic reliability law P(n) estimated on a set of values computed currents of the random variables KPI(Ni),...,KPI(Nni) of the indicator of quality relative to a corresponding set of common quantities N1....., Data nrri observation 7.
Advantageously, the analysis means 12 are configured to apply a regression technique on the set of values of the KPI variables (Ni), KPI (N,i) of the quality indicator to determine a representative empirical function of the law probabilistic reliability P(n) as a function of the amount of data observation 7.
Furthermore, in step E4 the analysis means 12 are configured to estimate at from the probabilistic reliability law P(n) a minimal quantity Arc, (ie a number minimum) of observation data from which the value of the quality indicator KPI(n)õ,,No reaches a predetermined reliability criterion with a probability P(KPI(n)õ,No EI) greater than a predetermined value 1 ¨ s, the value predetermined 1¨s being the complement of an acceptable error s.
In other words, the analysis means 12 search for the first Arc, such that P(KP/(No) EI) > 1 ¨s.
For example, for the POD detection rate, we are looking for the first Arc, which verifies the following relationship:
P(POD(No) > 1 ¨ /3) > 1 ¨
In addition, for the PFA false alarm rate, we look for the first No who verifies the following relationship:

11 P(PFA(No) <a) > 1 ¨
1 ¨ and a represent the reliability requirements relating to the PODs and PFAs respectively.
In order to estimate the minimum quantity No, the analysis means 12 are advantageously configured to construct a curve of the probabilistic law reliability P(n) for n = Ni, ...., Nm. This curve is then extrapolated until that she crosses the ordinate axis at the predetermined value 1 ¨ s. This will give the abscissa a estimation of the minimum number Arc, sought from observation data 7.
Alternatively, in order to determine the first Arc, making it possible to reach the goal, we can construct a curve of the error e(n) representing the complement 1¨P(n) of the P(n) law and extrapolate the curve until it intersects the axis of ordered at the acceptable error value s.
It should be noted that for the false alarm rate PFA, the probabilistic law of reliability P(n) is an average of n Boolean experiments with parameter p representing the truth PFA value. Its law is therefore that of a Binomial divided by n, it converges to a normal law with mean p and variance an = p (1 ¨ p)/n of the following form :
1 f(ce-P)/year x2 1a ¨ p P(n) = e) -2 dx = ¨2 (1 + erf az V 2 on Advantageously, to simplify the method of estimating the theoretical law P (n), we use hypotheses on the evolution of this law. For instance, when is less than p, the theoretical law P(n) converges exponentially to zero.
Thus, for the PFA false alarm rate, the theoretical reliability law P(n) may be represented by an approximation function f (n) of the following form:
f(=a++c log(n) not where a, b, c are constants which can be determined by a technique of regression on the set of values PFA(N1), , PFA(Nm) of the false alarm rate PFA. We will note that the term in log(n) is used to correct the decrease in 7.1 of the function of approximation f (n) thus reducing the error of approximation and allowing to have a more estimate conforms to the theoretical P(n) law.

12 Fig. 3 is a graph representing the curves of the relative error s (n) at the theoretical reliability law P(n) relating to the PFA and its function approximation f (n) corresponding.
The theoretical curve Cl gives the appearance of s (n) = 1 ¨ P (n) for a parameter p = 4% and a reliability requirement a = 5% and the dotted curve C2 represented the approximation 1 ¨ f (n) . The two curves Cl and C2 are almost identical and by Therefore, we can use the complement of the approximation function f(n) for determine the minimum amount of observational data.
For example, if we are looking for a precision of 0.9 (ie, an error s (n) =
0.1) with a theoretical false alarm rate of p = 4% and a requirement of a =
5%, it takes at least 650 measurements as shown in the graph. In other words, for assess a monitoring validation of aircraft engine equipment 5 with a rate false alarm less than 5% and a probability or assurance of 90%, it must collect about 650 observation data.
In addition, the precision on the observation data increases with the increase in the number of data.
Indeed, Fig. 4 is a graph representing observation measurements around the curve representative of the error s(n). These measures form a tubing confidence 41 around the Cl or C2 curve showing that the precision is low at first experiments, then the tube gets thinner as the number of data increases.
Finally, in step E5, the test means 13 apply indicators of quality KPI on the minimum No. of observation data in order to assess so optimal validation of the monitoring system 3.
Fig. 5 schematically illustrates a system validation tool of monitoring of equipment of an aircraft engine, according to a first mode of preferred embodiment of the invention.
This first embodiment relates to the validation of a system of generic 3 monitoring on a 43 test bed before installation on a aircraft.
In fact, the specifications of the surveillance system 3 are derived in specifications of needs, themselves expressed in requirements and objectives.
We answer to

13 a requirement or an objective when we have a test to validate them expected performance. This test applies to at least part of all KPI quality indicators that are compared to thresholds defined by the specifications. The KPIs are calculated by validation scenarios adapted to the problems raised by requirement or goal. Scenarios are based on data observation ensuring reasonable coverage of the need.
Thus, the processing means 11 collect observation data 7 relating to the equipment to be monitored and carried out on the test bench 43. In variant, the 7 observational data is collected on aircraft testing the system of monitoring 3.
As previously described, the analysis means 12 calculate the value of each quality indicator on a current amount of data observation 7.
Then, they estimate a minimum number of observational data from of which the value of the quality indicator verifies a predetermined reliability criterion with a probability greater than a predetermined value. This minimum number of data observation ensures sufficient coverage for the quality indicators to have a sense and allows therefore, to know from when, one can stop doing tests on bank 43.
The test means 13 then evaluate the validation of the system of monitoring 3 by applying KPI quality indicators on the number minimum of observation data 7 relating to this equipment.
Fig. 6 schematically illustrates a system validation tool of monitoring of equipment of an aircraft engine, according to a second mode of preferred embodiment of the invention.
This second embodiment relates to the validation and adjustment of a 3 monitoring system after installation on a 5 series engine of a aircraft 45 applying at least part of the set of quality indicators on the amount minimum of observational data collected in flight.
Monitoring system 3 is already pre-calibrated on a measurement set observations made on a test bench 41 according to the first mode of realization or on

14 aircraft belonging to companies agreeing to contribute in the development monitoring systems.
It will be noted that the aircraft 43 on which the system of surveillance 3 will follow missions of its own, it will also have right to operations maintenance specific to the owning company's logic. Thereby, the validation takes into account the specificity of the use of the motor on which it is installed.
As previously explained, the processing means 12 collect observation data 7 relating to the equipment to be monitored carried out on the motor in operation. The analysis means 12 estimate a minimum number of data observation from which the value of a quality indicator reaches a criterion of predetermined reliability with a probability greater than a value predetermined. This allows you to know from when you can stop the validation test, but also when to resume it if the operations performed by the carrier aircraft change or if the engine configuration changes following maintenance operations.

Claims (11)

15 1. Validation tool for a monitoring system of at least one equipment of a aircraft engine, comprising test means configured to evaluate a validation of said surveillance system by applying a set indicators of quality on a volume of observation data relating to said equipment, characterized in that it includes:
- processing means configured to collect data observation relating to said equipment, - analysis means configured to calculate a current value of at least less a quality indicator on a current amount of observation data collected by means of treatment, - analysis means configured to estimate the probability that said value current of the quality indicator reaches a reliability criterion predetermined, forming thus a probabilistic law of reliability estimated on a set of values of the indicator of quality relating to a corresponding set of quantities of data observation, and - analysis means configured to estimate from said law probabilistic reliability a minimum amount of observational data from which value of the quality indicator reaches a predetermined reliability criterion with a probability greater than a predetermined value, said minimum quantity of data of observation corresponding to said volume of observation data intended for be used to assess the validation of said monitoring system. 2. Validation tool according to claim 1, characterized in that said value predetermined is the complement of an error predefined as being acceptable. 3. Validation tool according to claim 1 or 2, characterized in that the means analysis are configured to calculate the current value of an indicator of quality in Date Received/Date Received 2021-11-12 applying a cross-validation technique on said current quantity of data observation. 4. Validation tool according to claim 3, characterized in that said technique cross-validation method is selected from the following techniques:
bootstrap, K-fold, leave-one-out. 5. Validation tool according to claim 1, characterized in that all of quality indicators includes the following indicators: rate of false alarm, rate of detection, localization rate. 6. Validation tool according to claim 1, characterized in that the means analysis are configured to apply a regression technique on said together of values of the quality indicator to determine a function of approximation representative of said probabilistic reliability law according to the amount of data observation. 7. Validation tool according to claim 6, characterized in that for a quality indicator corresponding to false alarm rates, said function of approximation as a function of the quantity n of observation data is expressed by the following relationship:
b f(n) = a + =_ + c log(n) vn where a, b, c are regression constants. 8. Validation tool according to claim 1, characterized in that the means of test are configured to evaluate a validation of said monitoring system before his installation on an aircraft by applying the set of quality indicators on a quantity of observational data collected on a test bed and/or on a fleet of aircraft engines in operation.
Date Received/Date Received 2021-11-12 9. Validation tool according to claim 8, characterized in that the means of test are configured to continue the validation and adjustment of said system of monitoring after its installation on a series engine by applying all of quality indicators on a quantity of observation data collected in flight. 10. System for monitoring at least one item of equipment of an aircraft engine designed by the validation tool according to claim 1, characterized in that it is configured for receive observation data specific to said equipment and to issue a result diagnosing the state of said equipment. 11. Method for validating a monitoring system of at least one equipment of an aircraft engine, comprising test steps for evaluating a validation of said monitoring system by applying a set of quality indicators on a volume of observation data relating to said equipment, characterized in that that he further includes the following steps:
- collect observation data relating to said equipment, -calculate a current value of at least one quality indicator on a current quantity of observation data collected by the processing means, - estimate the probability that said current value of the indicator of quality reaches a predetermined reliability criterion, thus forming a law reliability probabilistic estimated on a set of values of the quality indicator relating to a together corresponding amounts of observation data, and - estimating from said probabilistic reliability law a quantity minimum of observation data from which the value of the indicator of quality reaches a predetermined reliability criterion with a probability greater than a value predetermined, said minimum quantity of observation data corresponding audit volume of observational data intended to be used to assess the validation of said Surveillance system.
Date Received/Date Received 2021-11-12