AU2019202155A1

AU2019202155A1 - Method for evaluating the conformity of a tracking system to a set of requirements and associated devices

Info

Publication number: AU2019202155A1
Application number: AU2019202155A
Authority: AU
Inventors: Nicolas Honore; Christophe Labreuche; Olivier Pignard
Original assignee: Thales SA
Current assignee: Thales SA
Priority date: 2018-03-28
Filing date: 2019-03-28
Publication date: 2019-10-17
Also published as: FR3079637B1; SG10201902788TA; EP3547159C0; CN110321521A; EP3547159B1; MX2019003648A; FR3079637A1; EP3547159A1

Abstract

The invention relates to a method for evaluating the conformity of a tracking system to a set of requirements, at least one requirement being a mandatory requirement and at least one requirement being a recommended requirement, the evaluation method being carried out by computer, the evaluation method including at least the following steps: - supervised learning of an evaluation function of the conformity to a requirement in the form of a piecewise linear function, the linear function including at least three pieces, preferably at least six pieces, in order to obtain learned evaluation functions, - application of the learned evaluation functions to the tracking system in order to obtain evaluation values, and - use of the evaluation values to obtain an overall score evaluating the conformity of the tracking system to the set of requirements. Figure 1

Description

ORIGINAL COMPLETE SPECIFICATION STANDARD PATENT

Invention Title

Method for evaluating the conformity of a tracking system to a set of requirements and associated devices

The following statement is a full description of this invention, including the best method of performing it known to me/us:1

1a

2019202155 28 Mar 2019

The present invention relates to a method for evaluating the conformity of a tracking system to a set of requirements. The present invention also relates to a computer program product and a readable information medium allowing the implementation of the method.

In the avionics field, the tracking system is a very important system, particularly given that the tracking system guarantees the absence of collisions between airplanes. The tracking system is therefore a system that must respect to set of mandatory constraints in order to be certified and recommendations to improve safety.

It is therefore desirable to be able to evaluate the compliance score of the tracking system.

To that end, the tracking system is subject to a simple evaluation consisting of evaluating each criterion in a binary manner, namely 100% if the criterion is met and 0% if the criterion is not met.

However, in case of multiple criteria corresponding to a complex mixture of mandatory and recommended constraints and with the aim of improving the tracking system, such an evaluation is often insufficient.

There is therefore a need for a method for evaluating the conformity of a tracking system to a set of requirements making it possible to obtain a finer evaluation of the conformity of the tracking system.

To that end, the present description relates to a method for evaluating the conformity of a tracking system to a set of requirements, at least one requirement being a mandatory requirement and at least one requirement being a recommended requirement, the evaluation method being carried out by computer, the evaluation method including at least the following steps: supervised learning of an evaluation function of the conformity to a requirement in the form of a piecewise linear function, the linear function including at least three pieces, preferably at least six pieces, in order to obtain learned evaluation functions, application of the learned evaluation functions to the tracking system in order to obtain evaluation values, and use of the evaluation values to obtain an overall score evaluating the conformity of the tracking system to the set of requirements.

According to specific embodiments, the method comprises one or more of the following features, considered alone or according to any technically possible combinations:

- the method includes computing an overall score for each type of requirement by aggregating evaluation functions of all of the requirements of the considered type, the overall score evaluating the conformity of the tracking system to the set of requirements being obtained by aggregating the overall scores by requirement type.

2019202155 28 Mar 2019

- the method further includes a step for supervised learning of aggregation functions of the evaluation values, the aggregation function evaluating the conformity of the tracking system to a portion of the requirements of the set of requirements corresponding to a particular property, and a usage step comprising using the evaluation values and the learned aggregation functions to obtain a value for each portion of the requirements of the set of requirements corresponding to a particular property and using the values obtained during the use as well as aggregation functions to obtain the overall score.

- in the supervised learning step, each linear function is a linear function with five pieces.

- a requirement sets a threshold to be respected, at least one of the following properties is verified:

- each linear function includes a plurality of thresholds, one of the thresholds being the threshold set by the requirement,

- each linear function includes a plurality of thresholds, the threshold having the second lowest value by the evaluation function being the threshold set by the requirement,

- each linear function includes four thresholds, the first threshold corresponding to a maximum value by the evaluation function, the second threshold corresponding to 88% of the maximum value, the third threshold corresponding to 35% of the maximum value and the fourth threshold to 0, and

- each linear function includes four thresholds, the first threshold corresponding to a maximum value by the evaluation function, the second threshold corresponding to 80% of the maximum value, the third threshold corresponding to 20% of the maximum value and the fourth threshold to 0.

- each linear function includes a plurality of thresholds and a requirement sets a threshold to be respected, at least one of the following properties being verified:

- each threshold is a linear function of the threshold set by the requirement, and

- one of the thresholds is the arithmetic average of the other two thresholds.

- the number of requirements is greater than or equal to 20.

- the tracking system provides estimates of measurements from earlier data from several sensors, each evaluation function using, as input, at least one of said estimated measurements.

2019202155 28 Mar 2019

- each estimated measurement is tainted by an error, each evaluation function also taking said error into account.

- the method further includes a step for grouping certain evaluation values together to obtain scores evaluating the conformity of the tracking system to a portion of the requirements of the set of requirements corresponding to a particular property, a property for example being the availability, continuity, integrity, time or coherence.

The present description also relates to a computer program product including a readable information medium, on which a computer program is stored comprising program instructions, the computer program being able to be loaded on a data processing unit and suitable for driving the implementation of a method as previously described when the computer program is implemented on the data processing unit.

The present description also relates to a readable information medium storing a computer program product including a readable information medium, on which a computer program is stored comprising program instructions, the computer program being able to be loaded on a data processing unit and suitable for driving the implementation of a method as previously described when the computer program is implemented on the data processing unit.

Other features and advantages of the invention will appear upon reading the following description of embodiments of the invention, provided solely as an example and done in reference to the drawings, which are:

- figure 1, a schematic view of an example system allowing the implementation of a method for evaluating the conformity of a tracking system to a set of requirements;

- figure 2, a schematic illustration of an example evaluation function; and

- figure 3, a flowchart showing the distribution of the constraints of the ESASSP standard by property.

A system 10 and a computer program product 12 are shown in figure 1. The interaction of the computer program product 12 with the system 10 makes it possible to implement a method for evaluating the conformity of a tracking system to a set of requirements.

The system 10 is a computer.

More generally, the system 10 is an electronic computer able to manipulate and/or transform data represented as electronic or physical quantities in registers of the system 10 and/or memories into other similar data corresponding to physical data in the memories, registers or other types of display, transmission or storage devices.

2019202155 28 Mar 2019

The system 10 includes a processor 14 comprising a data processing unit 16, memories 18 and an information medium reader 20. The system 10 also comprises a keyboard 22 and a display unit 24.

The computer program product 12 includes a readable information medium.

A readable information medium is a medium readable by the system 10, usually by the data processing unit 14. The readable information medium is a medium suitable for storing electronic instructions and able to be coupled with a bus of a computer system.

As an example, the readable information medium is a CD-ROM, a magnetic-optical disc, a ROM memory, a RAM memory, an EPROM memory, an EEPROM memory, a magnetic card, an optical card, a flash memory or an SSD disc.

A computer program comprising program instructions is stored on the readable information medium.

The computer program can be loaded on the data processing unit 14 and is suitable for driving the implementation of an evaluation method.

The operation of the system 10 interacting with the computer program product 12 is described in reference to figure 2, which schematically illustrates part of one example embodiment of a method for evaluating the conformity of a tracking system to a set of requirements.

At least one requirement is a mandatory requirement and at least one requirement is a recommended requirement.

By definition, a mandatory requirement is a requirement with which non-compliance causes disqualification of the tracking system. The term necessary requirement is sometimes used to designate this type of requirement.

By definition, a recommended requirement is a requirement with which noncompliance does not cause disqualification of the tracking system, but respect for which is desirable.. The term desirable requirement is sometimes used to designate this type of requirement.

According to the proposed example, the number of requirements is greater than or equal to 20.

Hereinafter, it is assumed that each requirement sets a threshold to be respected. This means that each requirement can be formulated in the form of a physical property greater than or equal to the threshold or a physical property less than or equal to the threshold.

As an example, such a set of requirements corresponds to the requirements of standard ESASSP, which is outlined in the document titled EUROCONTROL Specification

2019202155 28 Mar 2019 for ATM Surveillance System Performance (Volume 1), the ISBN of which is 978-2-87497022-1 and which was published in March 2012.

The purpose of this standard is to guarantee the required quality to avoid collisions. This quality targets a better separation between airplanes in air traffic (civilian). In order to be able to increase air traffic capacity, it is crucial to have tracking tools making it possible to guarantee an increasingly fine separation between airplanes. Two standards have been defined to guarantee two distance separations: 5 NM and 3 NM.

The ESASSP standard defines 22 constraints named from R1 to R22, i.e., 22 specific metrics. The 22 constraints are explained briefly hereinafter.

The first constraint R1 relates to the Measurement Interval for Probability of Update. The first constraint R1 is broken down into two requirements: a mandatory requirement and a recommended requirement.

The second constraint R2 relates to the Probability of update of horizontal position. The second constraint R2 is broken down into two requirements: a mandatory requirement and a recommended requirement.

The third constraint R3 relates to the ratio of missed 3D position involved in long gaps. The third constraint R3 corresponds to a mandatory requirement.

The fourth constraint R4 relates to the Horizontal position RMS error. The RMS error is often considered a primary error representative of the measuring precision. The fourth constraint R4 is broken down into two requirements: a mandatory requirement and a recommended requirement.

The fifth constraint R5 relates to a consecutive correlated error ratio. The fifth constraint R5 corresponds to a recommended requirement.

The sixth constraint R6 relates to the Max delta time in close proximity. Such a constraint corresponds to the fact that, on a radar, very close tracks that are not refreshed at the same moment are present. In the presentation to the air traffic controller, there should not be two tracks side-by-side that have distant update dates. This could lead to collision risks, i.e., serious safety problems. The sixth constraint R6 corresponds to a recommended requirement.

The seventh constraint R7 relates to the Probability of update of pressure altitude. The seventh constraint R7 corresponds to a mandatory requirement.

The eighth constraint R8 relates to the Average data age of forwarded pressure altitude. The eighth constraint R8 corresponds to a mandatory requirement.

The ninth constraint R9 relates to the Max data age of forwarded pressure altitude. The ninth constraint R9 corresponds to a mandatory requirement.

2019202155 28 Mar 2019

The tenth constraint R10 relates to the Ratio of incorrect forwarded pressure altitude. It should be noted that the tenth constraint R10 is an estimate of the error corresponding to a stray error. The tenth constraint R10 corresponds to a mandatory requirement.

The eleventh constraint R11 relates to the Pressure altitude unsigned error. It should be noted that the eleventh constraint R11 is an estimate of the error of the primary error type. The eleventh constraint R11 corresponds to a mandatory requirement.

The twelfth constraint R12 corresponds to the Delay of apparition of the emergency indicator / SPI report. The twelfth constraint R12 corresponds to a mandatory requirement.

The thirteenth constraint R13 relates to the Delay of change in Aircraft Id. The thirteenth constraint R13 corresponds to a mandatory requirement.

The fourteenth constraint R14 relates to the Probability of update of aircraft identity with correct value. The fourteenth constraint R14 is broken down into two requirements: a mandatory requirement and a recommended requirement.

The fifteenth constraint R15 relates to the Ratio of Incorrect Aircraft Identity. It should be noted that the fifteenth constraint R15 is an estimate of the error of the stray error type. The fifteenth constraint R15 corresponds to a mandatory requirement.

The sixteenth constraint R16 relates to the Rate of climb/descent RMS error. In this context, the estimated error is of the primary error type. The sixteenth constraint R16 corresponds to a mandatory requirement.

The seventeenth constraint R17 relates to the Track velocity RMS error. In this context, the estimated error is of the primary error type. The seventeenth constraint R17 corresponds to a recommended requirement.

The eighteenth constraint R18 relates to the Track velocity angle RMS error. In this context, the estimated error is of the primary error type. The eighteenth constraint R18 corresponds to a recommended requirement.

The nineteenth constraint R19 relates to the Density of uncorrelated false target reports. In this context, the estimated error is of the stray error type. The nineteenth constraint R19 corresponds to a recommended requirement.

The twentieth constraint R20 relates to the Hourly rate of false tracks close to true tracks. In this context, the estimated error is of the correlated error type. The twentieth constraint R20 corresponds to a recommended requirement.

The twenty-first constraint R21 relates to the Continuity. The twenty-first constraint R21 corresponds to a recommended requirement.

The twenty-second constraint R22 relates to manual investigations (manual analyses of the results) that must be conducted when constraints R2, R4, R12 or R13 are

2019202155 28 Mar 2019 not met. The twenty-second constraint R22 is broken down into two requirements: a mandatory requirement and a recommended requirement.

In summary, the following table 1 can be established, the presence of an X indicating the presence of an associated requirement:

Constraints	Name used in the standard	Unit	Mandatory requirement	Recommended requirement
R1	Measurement Interval for Probability of Update	s	X	X
R2	Probability of update of horizontal position	%	X	X
R3	ratio of missed 3D position involved in long gaps	%	X
R4	Horizontal position RMS error	m	X	X
R5	Consecutive correlated error ratio	%		X
R6	Max delta time in close proximity	s		X
R7	Probability of update of pressure altitude with correct value	%	X
R8	Average data age of forwarded pressure altitude	s	X
R9	Max data age of forwarded pressure altitude	s	X
R10	Ratio of incorrect forwarded pressure altitude	%	X
R11	Pressure altitude unsigned error	%	X
R12	Delay of apparition of the emergency indicator / SPI report	s	X
R13	Delay of change in Aircraft Id	s	X
R14	Probability of update of aircraft identity with correct value	%	X	X
R15	Ratio of Incorrect Aircraft Identity	%	X
R16	Rate of climb/descent RMS error	m/s		X

2019202155 28 Mar 2019

R17	Track velocity RMS error	m/s		X
R18	Track velocity angle RMS error	degrees		X
R19	Density of uncorrelated false target reports	number		X
R20	Number per hour of false tracks close to true tracks	number		X
R21	Continuity	%		X
R22			X

Table 1: Association for each constraint of its name, the associated unit and the presence of a mandatory requirement and a recommended requirement

In summary, the ESASSP standard is a set of 14 mandatory requirements and 12 recommended requirements, which represents 26 requirements in total, these 26 requirements being assessed according to 22 different metrics.

Hereinafter, it is presumed that the method is implemented to evaluate the 26 previous requirements.

The evaluation method includes a supervised learning step, an application step and a usage step.

During the supervised learning step, a function is learned for evaluating the conformity to a requirement of a tracking system in the form of a piecewise linear function.

This makes it possible to obtain learned evaluation functions.

Each evaluation function associates a value with a physical property. The value is representative of the conformity to a requirement of a tracking system.

In the illustrated method, each evaluation function is a piecewise linear function.

By definition, a piece is a linear portion of the evaluation function. The term segment is sometimes used for the term piece.

As an example, a function defined by pieces is a function including only three pieces: a first piece for which the value of the function is nil when the variable is less than 0, a second piece for which the value is the variable when the variable is between 0 and 1 and a third piece for which the value of the function is equal to 1 when the variable is greater than 1.

For the described method, each evaluation function includes at least three pieces.

Preferably, each evaluation function is a linear function including a number of pieces less than or equal to 6.

2019202155 28 Mar 2019

In the described example, each linear function is a linear function with five pieces, a first piece, a second piece, a third piece, a fourth piece and a fifth piece.

Thus, each linear function includes four thresholds (a first threshold, a second threshold, a third threshold and a fourth threshold) delimiting the pieces. More specifically, the first threshold is the interface between the first piece and the second piece, the second threshold is the interface between the second piece and the third piece, the third threshold is the interface between the third piece and the fourth piece and the fourth threshold is the interface between the fourth piece and the fifth piece.

Each threshold is a value of a physical property.

Hereinafter, as shown by figure 2, it is assumed that the thresholds are strictly decreasing, i.e., that the first threshold has a value strictly greater than the second threshold, that the second threshold has a value strictly greater than the third threshold and that the third threshold has a value strictly greater than the fourth threshold.

Furthermore, for the illustrated example, the first threshold corresponds to a maximum value for the evaluation function, the second threshold corresponds to 88% of the maximum value, the third threshold corresponds to 35% of the maximum value and the fourth threshold has a nil value.

More specifically, for an evaluation function yielding a score between 0 and 100%, this means that:

- for a physical property greater than the first threshold, the value of the evaluation function is 100%;

- for a physical property between the first threshold and the second threshold, the value of the evaluation function decreases linearly between 100 and 88%;

- for a physical property between the second threshold and the third threshold, the value of the evaluation function decreases linearly between 88 and 35%;

- for a physical property between the third threshold and the fourth threshold, the value of the evaluation function decreases linearly between 35 and 0%; and

- for a physical property less than the fourth threshold, the value of the evaluation function is equal to 0.

Alternatively, the first threshold corresponds to a maximum value for the evaluation function, the second threshold corresponds to 80% of the maximum value, the third threshold corresponds to 20% of the maximum value and the fourth threshold to 0.

2019202155 28 Mar 2019

- for a physical property between the first threshold and the second threshold, the value of the evaluation function decreases linearly between 100 and 80%;

- for a physical property between the second threshold and the third threshold, the value of the evaluation function decreases linearly between 80 and 20%;

- for a physical property between the third threshold and the fourth threshold, the value of the evaluation function decreases linearly between 20 and 0%; and

The choice of the thresholds depends on the cases.

However, to facilitate the interpretation of the evaluation functions, it is preferable for one of the thresholds to be the threshold set by the requirement associated with the evaluation function.

In the illustrated example, the second threshold is the threshold set by the requirement.

Thus, once the value of the evaluation function for the physical property is above the second threshold, this means that the requirement is fulfilled.

The evaluation function previously presented applies for mandatory or recommended requirements.

To illustrate this idea, concrete examples are outlined hereinafter in reference to the requirements of the ESASSP standard.

For the first case of a recommended requirement for which a percentage must be maximized with a recommended threshold REC provided by the ESASSP standard, the evaluation function is a linear function with five pieces including four thresholds S1, S2, S3 and S4 determined by the following formulas:

- the first threshold S1 is such that SI = 100 - (100 - REC) * 0.35;

- the second threshold S2 is such that S2 = REC;

- the fourth threshold S4 is such that S4 = 100 - (100 - REC) * 2, and

- the third threshold S3 is such that S3 = (S2 + S4)/2.

The results of practical application to the corresponding requirements of the ESASSP standard are summarized in the following table 2:

2019202155 28 Mar 2019

Constraints	S1	S2	S3	S4
R2	99	99	98.5	98
R14	99.3	98	97	96

Table 2: Evaluation function of the recommended requirements of the percentage to maximize type

For the second case of a recommended requirement for which the value must be minimized whereas the values of the metric are relatively large (typically several hundreds in the unit) with a recommended threshold REC provided by the ESASSP standard, the evaluation function is a linear function with five pieces including four thresholds S1, S2, S3 and S4 determined by the following formulas:

- the first threshold S1 is such that SI = REC * 0.35;

- the second threshold S2 is such that S2 = REC;

- the fourth threshold S4 is such that S4 = REC * 1.43, and

- the third threshold S3 is such that S3 = (S2 + S4)/2.

The results of practical application to the corresponding requirements of the ESASSP standard are summarized in the following table 3:

Constraints	S1	S2	S3	S4
R4	122.5	350	425.25	500.5
R17	1	4	4.86	5.72
R18	5	10	12.15	14.3
R19	3.5	10	12.15	14.3

Table 3: Evaluation function of the recommended requirements of the value to minimize type with relatively large values

For the third case of a recommended requirement for which a value must be minimized whereas the values of the metric are relatively small (typically 0.1) with a recommended threshold REC provided by the ESASSP standard, the evaluation function is a linear function with five pieces and including four thresholds S1, S2, S3 and S4 determined by the following formulas:

- the first threshold S1 is such that SI = REC * 0.35;

- the second threshold S2 is such that S2 = REC;

2019202155 28 Mar 2019

- the fourth threshold S4 is such that S4 = REC * 2, and

- the third threshold S3 is such that S3 = (S2 + S4)/2.

The results of practical application to the corresponding requirements of the

ESASSP standard are summarized in the following table 4:

Constraints	S1	S2	S3	S4
R5	0.0105	0.03	0.045	0.06
R6	0.105	0.3	0.45	0.6
R16	0.889	2.54	3.81	5.08

Table 4: Evaluation function of the recommended requirements of the value to minimize type with relatively small values

For the fourth case of a mandatory requirement for which a percentage must be maximized with a mandatory threshold MAND provided by the ESASSP standard, the evaluation function is a linear function with five pieces and including four thresholds S1, S2, S3 and S4 determined by the following formulas:

- the first threshold S1 is such that SI = 100 - (100 - MAND) * 0.68;

- the second threshold S2 is such that S2 = MAND;

- the fourth threshold S4 is such that S4 = 100 - (100 - MAND) * 1.02, and

- the third threshold S3 is such that S3 = (S2 + S4)/2.

The results of practical application to the corresponding requirements of the ESASSP standard are summarized in the following table 5:

Constraints	S1	S2	S3	S4
R2	97.96	97	96.97	96.94
R7	97.28	96	95.96	95.92
R11	99.932	99.9	99.899	99.838
R14	98.64	98	97.98	97.96

Table 5: Evaluation function of the recommended requirements of the percentage to maximize type

For the sixth case of a mandatory requirement for which a value must be minimized with a recommended threshold MAND provided by the ESASSP standard, the evaluation

2019202155 28 Mar 2019 function is a linear function with five pieces and including four thresholds S1, S2, S3 and S4 determined by the following formulas:

- the first threshold S1 is such that SI = MAND * 0.96;

- the second threshold S2 is such that S2 = MAND;

- the fourth threshold S4 is such that S4 = REC * 1.02, and

- the third threshold S3 is such that S3 = (S2 + S4)/2.

The results of practical application to the corresponding requirements of the

ESASSP standard are summarized in the following table 6:

Constraints	S1	S2	S3	S4
R3	0.48	0.5	0.505	0.51
R4	480	500	505	510
R8	3.84	4	4.04	4.08
R9	15.36	16	16.16	16.32
R10	0.096	0.1	0.101	0.102
R12	11.52	12	12.12	12.24
R13	23.04	24	24.24	24.48
R15	0.096	0.1	0.101	0.102

Table 6: Evaluation function of the mandatory requirements of the value to minimize type

At the end of the learning step, learned evaluation functions are thus obtained for each requirement.

During the application step, the learned evaluation functions are applied to the tracking system to be evaluated.

This makes it possible to obtain, for each evaluation function, an evaluation value specific to the tracking system to be evaluated.

The application step is in particular implemented on data supplied by the tracking system.

At the end of the application step, evaluation values are thus obtained.

During the usage step, evaluation values are used to obtain an overall score evaluating the compliance of the tracking system with the set of requirements.

The overall score depends on the evaluation values according to a more or less complex function.

In one simple embodiment, the overall score is a simple addition of the evaluation values.

2019202155 28 Mar 2019

In one preferred embodiment, the method includes computing an overall score for each type of requirement by aggregating evaluation functions of all of the requirements of the considered type, the overall score evaluating the conformity of the tracking system to the set of requirements being obtained by aggregating the overall scores by requirement type. In such an embodiment, the evaluation values corresponding to evaluation functions associated with a mandatory requirement (first type of requirement) are first aggregated to obtain a score UMand(x) (representing the overall satisfaction level of all of the mandatory requirements), the evaluation values corresponding to evaluation functions associated with a recommended requirement (second type of requirement) are aggregated to obtain a score Urb/x) (representing the overall satisfaction level of all of the recommended requirements), and lastly the two scores UMand(x) et UR_ec(x) are aggregated to obtain the overall score U(x).

The overall score U(x) is a weighting applying a first weight for UMand(x), a second weight for UR_ec(x), a third weight for the smallest score between UMand(x) and UR_ec(x), and lastly a fourth weight for the largest score between UMand(x) and UR_ec(x).

In the general case, these four weights are different. For example, the ratio between the first weight and the second weight is greater than or equal to 3.

Likewise, UMand(x) is obtained by weighting evaluation functions associated with a mandatory requirement and different minimums and maximums between the subsets of these functions.

Urb/x) is obtained by weighting evaluation functions associated with a recommended requirement and different minimums and maximums between the subsets of these functions.

According to one particular example, the overall score is given by the following function:

fi(x) ⁼ 0.805 * UMAnd(^x) + 0.105 * URE(/x) + 0.11 * min(t7_M^_WD(x), (/rec(%)) if (%) Um and (^x) Urec (^x)

The score UMand(x) can be obtained hierarchically from computations of intermediate scores as follows:

• Time. Upemps ; Mand(x) — 0.25 Ur8 ; Mand(x) + 0.25 Ur9 ; Mand(x) +0.25 Ur12 ; Mand(x) +0.25 * UR13;Mand(x) • Availability: U_Di_Spo; Mand(x) = 0.225 * U_R2;Mand(x) + 0.225 * U_R7;Mand(x) + 0.11 * max(L/r2 ; Mand(x) , Ur7 ; Mand(x}} + 0.22 ITlin(L/R2, Mand(x) , Ur14 ; Mand^X)) + 0.22 min (Dr,- ; Mand(x) , Ur14 ; Mand(x}} • False Error. UFaux ,· Mand(x) — 0.5 Urio ,· Mand(x) + 0.5 Ur15 ; Mand(x) • Core error (precision): Ucoeur; Mand(x) = 0.5 * U_R4, Mand(x) + 0.5 * Urh , Mand(x) • Integrity error. Ulnteg ,· Mand(x) — Ucoeur; Mand(x)/3+ 2 Upaux: Mand(x}/^

2019202155 28 Mar 2019 • Overall score on the mandatory requirements: UMand(x) = 0.2 * Uinteg; MandfX) + 0.4

UDispo; Mand(x) + 0.2 ITlin(L/R3; Mand(x) , UTemps; Mand(x}} + 0.1 ΓΤΊΪη(ί_//?3; Mand(x) ,

Ulnteg; Mand(x}} + 0.1 ITlin(L/R3; Mand(x) , UDispo; Mand(x)

The score UR_ec(x) can be obtained hierarchically as follows, from computations of intermediate scores:

• Core error (precision): Ucoeur;Rec(x) = 0.13 * UR4;R_ec(x) + 0.205 * URi6;Rec(x) +0.07 * Ur17 ; Rec(x) +0.465 * UR18;Rec(x) + 0.13 * ΠΊ3Χ(ίΛ?77, Rec(x) , Ur18 ; Rec(x)) • Correlated error: Ucorr;Rec(x) = 0.25 * l/R5,R_ec(x) + 0.75 * UR20;Rec(x) • Integrity error: Ui_nt;Rec(x) = 0.05 * Ucon-;Rec(x) + 0.25 * U_Coeur;Rec(x) + 0.55 * Uri_9;Rec(x) + 0.05 mίη(Ucorr, Rec(x) , Ucoeur; Rec(x}} + 0.05 ΠΊΪΠ(0/(3ολλ Rec(x) , Ur19 ; Rec(x)) + 0.05 * min(L/c oeur; Rec(x), Ur19 ; Rec(x)) • Availability: UDispo_; Rec(x) = U_R2;R_ec(x)/3 + 2 * Uri_{4 ;} Rec(x)/3 • Overall score on the recommended requirements: UR_ec(x) = 0.22 * Uoispo; Rec(x) + 0.07 * UR6;Rec(x) + 0.57 * Ulnt;Rec(x) + 0.14 * max(UDispo; Rec(x) , Ulnt;Rec(x))

In a more elaborate embodiment, the overall score is a weighting of each evaluation function, each weighting coefficient being obtained by supervised learning.

The weighting coefficients in each intermediate score computation are also obtained by supervised learning.

At the end of the usage step, an overall score is obtained for the tracking system, this overall score reflecting the compliance with the requirements by the tracking system.

The evaluation method thus makes it possible to obtain a finer evaluation of the conformity of the tracking system than a binary evaluation method presented in the introduction of the application.

Indeed, each evaluation function is differentiated in that it makes it possible to account for the deviation of the property relative to the desired property by the mandatory and recommended requirement.

Nevertheless, the evaluation function remains a relatively simple function allowing the management of a large number of criteria (here a number greater than or equal to 20).

Obtaining such an overall score allows a use for multiple applications.

This makes it possible to facilitate the certification of a tracking system.

In particular, it becomes possible to evaluate the performance of a tracking system during design.

The comparison of two tracking systems is also made easier, since the overall score is finer. Such a property is advantageously used in a method for designing a tracking system for which the optimization function would consist of maximizing the overall score. The

2019202155 28 Mar 2019 property can also be used by a tracking system designer to determine the elements to be improved in the tracking system.

The overall score also makes it possible to detect the weaknesses of an existing tracking system and to determine how the addition of one or more sensors makes it possible to improve the tracking system.

To still further facilitate the task of a tracking system designer, the method further includes a step for supervised learning of aggregation functions of the evaluation values, the aggregation function evaluating the conformity of the tracking system to a portion of the requirements of the set of requirements corresponding to a particular property, the usage step comprising using the evaluation values and the learned aggregation functions to obtain a value for each portion of the requirements of the set of requirements corresponding to a particular property and using the values obtained during the use as well as aggregation functions to obtain the overall score.

In other words, this amounts to grouping certain evaluation values together to obtain scores evaluating the conformity of the tracking system to a portion of the requirements of the set of requirements corresponding to a particular property.

A property is for example the availability, continuity, integrity, time or coherence.

As illustrated in figure 3, in the case of the ESASSP standard, all of the constraints R1 to R22 are broken down into five properties G1, G2, G3, G4 and G5.

The first property G1, called availability, groups together the requirements of the constraints R1, R2, R7 and R14.

The second property G2, called continuity, groups together the requirements of the constraints R1 and R21.

The third property G3, called integrity, groups together three classes C1, C2 and C3, which are the evaluation classes of the error. The three classes are respectively the primary errors, the correlated errors and the stray errors. The first class C1 corresponding to an evaluation of the primary errors groups together the requirements of constraints R4, R11, R16, R17 and R18. The second class C2 corresponding to an evaluation of the correlated errors groups together the requirements of constraints R5 and R20. The third class C3 corresponding to an evaluation of the stray errors groups together the requirements of constraints R10, R15 and R19.

The fourth property G4, called “time, groups together the requirements of the constraints R8, R9, R12 and R13.

The fifth property G5, called coherence, groups together only the recommended requirement of the sixth constraint R6.

2019202155 28 Mar 2019

The proposed evaluation method implies that the input physical properties are a good reflection of the field reality.

Thus, an improvement in the input physical properties results in an improvement in the relevance of the overall score.

Therefore, preferably, the tracking system provides estimates of measurements from earlier data from several sensors, each evaluation function using, as input, at least one of said estimated measurements.

According to one preferred embodiment, each estimated measurement is tainted by an error, the error being estimated using later data coming from the same sensors as reference, each evaluation function also using, as input, at least one error from said estimated measurements.

As an example, this in particular applies to the estimate of the trajectory of an aircraft by a tracking system.

Thus proposed is a method for determining the error on the estimate of the trajectory of an aircraft by the tracking system to be evaluated.

The method includes an obtaining step, a prediction step, a collection step, a reconstruction step and a determination step.

During the obtaining step, data are obtained from several sensors between a first moment t1 and a second moment t2.

The sensors are for example a radar, an ADS, a WAM.

The ADS (Automatic Dependent Surveillance), also called ADS-B (Automatic Dependent Surveillance-Broadcast) is a cooperative surveillance system for air traffic control and other related applications. An airplane equipped with ADS-B determines its position through a satellite positioning system (GPS) and periodically sends this position and other information to ground stations.

A WAM (Wide Area Multilateral ration) is airplane surveillance technology based on the principle of the time of arrival difference that is used in an airport. A WAM uses the data from several sensors to obtain the location of the airplane. For example, the WAM collects the data from several antennas on the ground to apply mathematical calculations making it possible to obtain the position of the airplane.

A RADAR (RAdio Detection And Ranging) makes it possible to detect the presence of airplanes in the sky and to determine their position. The RADAR emits electromagnetic pulses in the sky, and the detection and location of an airplane are obtained by analyzing the wave reflected by the airplane and retransmitted toward the RADAR.

In the described example, the time interval between the first moment t1 and the second moment t2 is less than or equal to 30 minutes.

2019202155 28 Mar 2019

The obtaining step thus makes it possible to obtain multiple data between the first moment t1 and the second moment t2.

During the prediction step, the tracking system predicts the trajectory of the aircraft from data obtained between the first moment t1 and the second moment t2.

A predicted trajectory for the aircraft is thus obtained.

During the collection step, the data from the sensors are collected between an initial moment to and the first moment, as well as between the second moment t2 and a third moment t3.

The initial moment tO is before the first moment t1.

The third moment t3 is after the first moment t1 and the second moment t2.

The ratio R between the time interval between the second moment t2 and the third moment t3 and the time interval between the first moment t1 and the second moment t2 is greater than or equal to 0.5.

The ratio R’ between the time interval between the initial moment tO and the first moment t1 and the time interval between the first moment t1 and the second moment t2 is greater than or equal to 0.5.

For example, the time interval between the second moment t2 and the third moment t3 is greater than or equal to 15 minutes. As a result, in the described example, the ratio R is greater than 0.5.

Likewise, the time interval between the initial moment tO and the first moment t1 is greater than or equal to 15 minutes. As a result, in the described example, the ratio R’ is greater than 0.5.

During the reconstruction step, the actual trajectory of the aircraft is reconstructed from collected data and obtained data.

The reconstruction more particularly relates to the trajectory of the aircraft between the first moment t1 and the second moment t2.

The reconstruction step for example includes an interpolation of the data by a cubic spline and an adjustment by a least-squares technique.

During the determination step, the error is obtained by comparison of the predicted trajectory and of the reconstructed trajectory.

To determine the error between the reconstructed trajectory and the predicted trajectory by the tracking system, the following steps are carried out:

- Interpolation of the reference trajectory to the update time of the track by the tracking system. This interpolation means that several physical properties characterizing the track are interpolated, the main ones being the horizontal position, the vertical position, the horizontal speed vector, the vertical speed,

2019202155 28 Mar 2019

- The error is calculated along a given axis (position, speed, etc.) by obtaining the difference between the interpolated property of the reference trajectory and the property of the predicted trajectory, and

- For an entire trajectory, the RMS (Root Mean Square) value and the MAX value are computed using the set of computed errors for a given physical property.

Such a method makes it possible to obtain a reliable error for the trajectory of the aircraft.

The method can be implemented remotely by processing data using a system as described in reference to figure 1.

According to another embodiment, the method is implemented by a controller located in the tracking system.

This for example makes it possible to verify the operation of the tracking system with a delay time to alert if an excessive error is obtained by the determining method.

The invention relates to any technically possible combination of the previously described embodiments.

Throughout this specification and the claims which follow, unless the context requires otherwise, the word comprise, and variations such as comprises and comprising, will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as an acknowledgment or admission or any form of suggestion that that prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavor to which this specification relates.

Claims

1. - A method for evaluating the conformity of a tracking system to a set of requirements, at least one requirement being a mandatory requirement and at least one requirement being a recommended requirement, the evaluation method being carried out by computer, the evaluation method including at least the following steps:

- supervised learning of an evaluation function of the conformity to a requirement in the form of a piecewise linear function, the linear function including at least three pieces, in order to obtain learned evaluation functions,

- application of the learned evaluation functions to the tracking system in order to obtain evaluation values, and

- use of the evaluation values to obtain an overall score evaluating the conformity of the tracking system to the set of requirements.

2. - The method according to claim 1, wherein the method includes computing an overall score for each type of requirement by aggregating evaluation functions of all of the requirements of the considered type, the overall score evaluating the conformity of the tracking system to the set of requirements being obtained by aggregating the overall scores by requirement type.

3. - The method according to claim 1 or 2, wherein the method further includes:

- a step for supervised learning of aggregation functions of the evaluation values, the aggregation function evaluating the conformity of the tracking system to a portion of the requirements of the set of requirements corresponding to a particular property, and a usage step comprising using the evaluation values and the learned aggregation functions to obtain a value for each portion of the requirements of the set of requirements corresponding to a particular property and using the values obtained during the use as well as aggregation functions to obtain the overall score.

4. - The method according to any one of claims 1 to 3, wherein, in the supervised learning step, each linear function is a linear function with five pieces.

5. - The method according to claim 4, wherein a requirement sets a threshold to be respected, at least one of the following properties is verified:

2019202155 28 Mar 2019

6. - The method according to any one of claims 1 to 5, wherein each linear function includes a plurality of thresholds and a requirement sets a threshold to be respected, at least one of the following properties being verified:

- one of the thresholds is the arithmetic average of the other two thresholds.

7. - The method according to any one of claims 1 to 6, wherein the number of requirements is greater than or equal to 20.

8. - The method according to any one of claims 1 to 7, wherein the tracking system provides estimates of measurements from earlier data from several sensors, each evaluation function using, as input, at least one of said estimated measurements.

9. - The method according to claim 8, wherein each estimated measurement is tainted by an error, each evaluation function also taking said error into account.

10. - The method according to any one of claims 1 to 9, wherein the method further includes a step for grouping certain evaluation values together to obtain scores evaluating the conformity of the tracking system to a portion of the requirements of the set of requirements corresponding to a particular property, a property for example being the availability, continuity, integrity, time or coherence.

11. - The method according to any one of claims 1 to 10, wherein the linear function including at least six pieces.

2019202155 28 Mar 2019

12. - A computer program product including a readable information medium, on which a computer program is stored comprising program instructions, the computer program being able to be loaded on a data processing unit and suitable for driving the implementation of a method according to any one of claims 1 to 11 when the computer program is implemented on the data processing unit.

13. - A readable information medium storing a computer program comprising program instructions, the computer program being able to be loaded on a data processing unit and suitable for driving the implementation of a method according to any one of claims 1 to 11 when the computer program is implemented on the data processing unit.