EP4055343A1

EP4055343A1 - Method and device for assisting in landing an aircraft under poor visibility conditions

Info

Publication number: EP4055343A1
Application number: EP20797523.6A
Authority: EP
Inventors: Thierry Ganille; Jean-Emmanuel HAUGEARD; Pierre-Yves Dumas
Original assignee: Thales SA
Current assignee: Thales SA
Priority date: 2019-11-07
Filing date: 2020-11-03
Publication date: 2022-09-14
Also published as: US20220373357A1; WO2021089539A1; FR3103050A1; FR3103050B1

Abstract

The invention relates to a method and a device for assisting in landing an aircraft under poor visibility conditions. The method makes it possible to receive sensor data during an approach phase toward a runway where the runway and/or an approach ramp are not visible to the pilot from the cockpit; then to determine, in the received sensor data, data of interest that are characteristic of the runway and/or of the approach ramp; then to calculate, on the basis of the data of interest, the coordinates of a target zone; and to display, on a head-up display screen, a conformal symbol representative of the target zone, the conformal symbol being displayed before the aircraft reaches the decision height in order to provide the pilot with a visual cue within which to search for the runway and/or the approach ramp.

Description

DESCRIPTION

Title of the invention: AIRCRAFT LANDING ASSISTANCE PROCESS AND DEVICE IN CONDITIONS OF DEGRADED VISIBILITY

[0001] The invention relates to the field of landing assistance systems for aircraft based on on-board imaging cameras or sensors.

[0002] The invention more specifically addresses the problem of assisting the landing of aircraft in difficult weather conditions, in particular conditions of reduced or degraded visibility, in the event of fog for example.

[0003] Aviation standards impose rules for obtaining visibility during the landing phase. These rules result in decision thresholds which refer to the altitude of the aircraft during its descent phase. By law, an aircraft pilot operating in instrument flight (IFR) must, during the landing phase, visually acquire certain references made up of elements of the approach ramp or the airstrip. , and this before an altitude or a height determined for each approach. We then speak of decision altitude (DA) or decision height (DH) which vary according to the level of equipment of the runway (ILS level CAT I or CAT II or CAT III), the type of approach ( precision, non-precision, ILS, LPV, MLS, GLS, ...) and the topographical environment around the runway (plain, mountain, obstacles, ...). Typically, on an airport equipped with a CAT I ILS on the plain, the most common case today, a decision height (DH) is 60.96 meters (200 ft) and a decision altitude (DA) of the altitude of the runway is + 60.96 meters (+ 200 ft).

[0004] During landings in degraded visibility conditions, for example due to fog, snow, rain, at each of the thresholds, the acquisition of visual cues is more difficult. If the pilot has not visually acquired the regulatory references before reaching the DA or DH, he has the obligation to interrupt the landing by performing a go-around to regain altitude and either retry the same approach, or leave for a diversion airport. A go-around is an expensive operation for an airline, and abandoned landing maneuvers represent a real problem for air traffic management and for flight planning. It is necessary estimate before take-off the ability to land at destination on the basis of weather forecasts, more or less reliable, and if necessary provide fallback solutions.

[0005] Also the problem of landing aircraft in conditions of reduced visibility has been the subject of the development of several techniques.

[0006] One of these techniques is the Instrument Landing System (ILS). The ILS system is based on several radio frequency equipment installed on the ground, at the level of the landing strip, and a compatible instrument placed on board the aircraft. The use of such a guidance system requires expensive equipment and specific pilot qualification. Moreover, it cannot be installed at all airports. This system is only present at the main airports because its cost makes it unacceptable to install it at the others. In addition, new technologies based on satellite positioning systems will probably replace ILS systems in the future.

A so-called synthetic visualization solution SVS ("Synthetic Vision System" in English) makes it possible to display a terrain and the landing runways from the position of the aircraft supplied by a GPS and from its attitude supplied by its inertial unit. However, the uncertainty on the position of the aircraft as well as the precision of the positions of the runways which are stored in the databases prohibit the use of an SVS in critical phases where the aircraft is close to the ground such as the landing and take-off. More recently, SVGS solutions ("Synthetic Vision with Guidance System" in English), adding certain controls to an SVS allow a limited reduction in landing minima (the DH decision height is reduced by 15.24 meters (50 ft) only on ILS SA CAT I approaches).

Another approach is the augmented vision technique known as EVS or EFVS ("Enhanced (Flight) Vision System" in English) based on the display on a head-up display which makes it possible to display on the pilot's primary screen an image of the environment towards the front of the aircraft which is better than natural vision. This solution uses electro-optical, infrared or radar sensors to film the airport environment during the landing of an aircraft. The principle is to use sensors that are more efficient than the pilot's eye in degraded weather conditions, and to embed the information collected by the sensors in the pilot's field of vision, through a head-up display or on the visor of the pilot 'a helmet worn by the pilot. This technique is essentially based on the use of sensors to detect the radiation from the lamps arranged along the runway and on the approach ramp. Incandescent lamps produce visible light, but they also emit in the infrared range. Sensors in the infrared range make it possible to detect this radiation and the detection range is better than that of humans in the visible range, during degraded weather conditions. An improvement in visibility therefore makes it possible to a certain extent to improve the approach phases and limit abandoned approaches. However, this technique is based on parasitic infrared radiation from the lamps present in the vicinity of the runway. For the sake of lamp durability, the current trend is to replace incandescent lamps with LED lamps. The latter have a less extensive spectrum in the infrared range. A side effect is therefore to cause technical obsolescence of EVS systems based on infrared sensors.

[0009] An alternative to infrared sensors is the obtaining of images by a radar sensor, in centimeter or millimeter band. Certain frequency bands chosen outside the peaks of water vapor absorption have very low sensitivity to severe weather conditions. Such sensors therefore make it possible to produce an image through fog, for example. However, even though these sensors have fine range resolution, they exhibit much coarser angular resolution than optical solutions. The resolution is directly related to the size of the antennas used, and it is often too coarse to obtain precise positioning of the airstrip at a sufficient distance to perform the realignment maneuvers.

The emergence of the use of active sensors, such as for example LIDAR ("Light Detection and Ranging" in English) or millimeter radars, which are capable of detecting the airstrip from further away and by almost any visibility conditions, brings much better results than passive sensors like IR cameras. However, data from such sensors does not not make it possible to provide the pilot with a clear and easily interpretable image like an IR image.

Solutions using CVS ("Combined Vision Systems") visualization systems are based on the simultaneous display of all or part of a synthetic image and of a sensor image, for example by superimposing the various images. and possibly registration of the synthetic image on a remarkable element of the sensor image, or even by overlaying the sensor image in a medallion of the synthetic image or by trimming of remarkable elements or elements of interest of the sensor image and overlay of these elements on the synthetic image.

[0012] With a current EVS / EFVS system displayed head-up, the pilot before acquiring visibility of the ramp or track expects to see them appear around the speed vector and the synthetic track. Figure 1 illustrates a landing phase piloting symbology for a head-up display of an EVS system. The conformity of symbols depends primarily on the accuracy of the aircraft attitude data. If the roll and pitch are generally known with good precision, this is not always the case with heading, in particular for aircraft equipped with an AHRS (“Attitude and Heading Reference System” in English which is a set sensors on 3 axes making it possible to define the position of an airplane in space thanks to the accelerations and the magnetic fields which they undergo) where the heading error can reach 2 to 3 degrees and not of a 1RS (" Inertial Reference System ”in English) much more precise but also much more expensive. This can cause the compliant symbology to be displayed by as many degrees. Now, the pilot, who must be concentrated on finding visual cues around the speed vector, can then detect the runway later, in particular in meteorological conditions leading to degraded visibility. In many situations, the decision height threshold can be reached before visual detection of the runway, and a go-around can be decided which could have been avoided.

[0013] Also, there is a need for an aid for the visual identification of a landing strip by the pilot. An object of the invention is to overcome the drawbacks of the known techniques by meeting the aforementioned needs by a solution for assisting the landing of aircraft, in particular an aid in visual identification for the pilot before reaching decision height. (DH) or decision altitude (DA).

To obtain the desired results, a computer-implemented method for assisting aircraft landing in conditions of degraded visibility is proposed, the method comprising at least the steps of:

- receive data from a sensor during an approach phase to a landing strip, said landing strip and / or an approach ramp not being visible to the pilot from the cockpit;

-determine in the sensor data received data of interest characteristic of said landing runway and / or said approach ramp;

-calculate from the data of interest the coordinates of a target area; and

- display on a head-up display screen a compliant symbol representative of the target area, said compliant symbol being displayed before the aircraft reaches the decision height in order to provide the pilot with a visual cue in which to look for said landing strip and / or approach ramp.

According to alternative or combined embodiments:

the data reception step consists of receiving data from a sensor on board the aircraft and looking forward, said sensor being chosen from the group of FLIR piloting type sensors, multispectral camera ,

LIDAR or millimeter radar.

the step of determining data of interest consists in executing an artificial intelligence algorithm on the received sensor data, the algorithm implementing an artificial intelligence model trained for image processing, obtained during a learning phase through deep learning.

- deep learning is based on convolutional neural networks.

- the step of calculating a target area consists in determining, from the characteristics of the sensor and the attitude of the aircraft corresponding to the sensor data, the heading and elevation coordinates of the target area.

- The method comprises after the step of calculating a target area, a step of sending the coordinates of the target area to the head-up display device.

- the coordinates of the target area correspond to two opposite corners of a rectangle surrounding the data of interest characteristic of said landing runway and / or said approach ramp, and in which the compliant symbol which is displayed is said surrounding rectangle.

- the head-up display step consists in displaying the framing symbol on a fixed head-up screen in the cockpit and / or on a head-up screen worn by the pilot.

[0017] The invention also covers a computer program product comprising code instructions making it possible to perform the steps of the method for assisting in aircraft landing, in particular in conditions of degraded visibility, as claimed, when the program is run on a computer.

[0018] The invention further covers a device for assisting the landing of an aircraft, in particular in conditions of degraded visibility, the device comprising means for implementing the steps of the method for assisting in landing d aircraft in degraded visibility conditions according to any one of the claims.

In one embodiment, the data allowing the calculation of the target area come from a first sensor, the device further comprising a second sensor capable of providing an image that can be displayed in the head-up device carried by the device. pilot, the compliant symbol calculated from the data of the first sensor being displayed on said image supplied by the second sensor.

Another object of the invention is a man-machine interface comprising means for displaying a compliant symbol obtained according to the claimed method.

Another object of the invention is a landing aid system, in particular of the SVS, SGVS, EVS, EFVS or CVS type carrying an aircraft landing aid device, in particular in conditions degraded visibility, as claimed.

[0022] The invention also addresses an aircraft comprising an aircraft landing aid device, in particular in conditions of degraded visibility, as claimed.

Other characteristics, details and advantages of the invention will emerge on reading the description given with reference to the accompanying drawings given by way of example and which represent, respectively: [0024] [Fig.1] a known landing symbology for a head-up display of an EVS system;

[0025] [Fig.2] a method of assisting the landing of an aircraft making it possible to obtain a compliant symbol for a head-up display, according to one embodiment of the invention;

[0026] [Fig.3] a head-up display of an EVS system with the display of a compliant symbol obtained by the method of the invention;

[0027] [Fig.4] a display of a symbol according to the invention on an IR image; and

[0028] [Fig.5] a general architecture of a display system for implementing the method of the invention.

[0029] FIG. 2 illustrates the steps of a method 200 for assisting the landing of an aircraft, making it possible to obtain a compliant symbol for a head-up display, according to one embodiment of the invention.

The method begins upon receipt 202 of sensor data, from a sensor on board an aircraft and looking forward. The method of the invention applies to any type of sensor, whether it is a pilot FLIR providing an IR image, a multispectral camera, a LIDAR, or a millimeter radar.

The technical problem that the invention solves is that of aid in the detection of the landing strip of an aircraft by the pilot in a sensor image or in direct vision before descending below the decision height, especially in degraded visibility conditions. The invention allows the display of a new compliant symbol generated from an automatic detection technique of the approach ramp or the landing runway in data from a sensor (sensor data). Advantageously, the symbol displayed is perfectly consistent with the outside world and indicates to the pilot the zone where the runway and / or the approach ramp will appear before they are visible to the pilot with the naked eye in direct vision. This allows the pilot to identify where to look for the expected visual references, but also to tell him when to look for them, especially if the visibility conditions are reduced, since as soon as the system begins to identify the runway and / or the ramp and display the symbol, the pilot expects to be able to identify it himself visually. In a next step (204), after receiving the sensor data, the method makes it possible to determine in the received sensor data, data of interest which are characteristic of the landing runway and / or of the ramp. 'approach. The sensor can be an IR or multispectral sensor whose image is presented to the pilot or be a second sensor of the active type, in principle more efficient than an IR sensor, such as for example a millimeter radar, but whose data is not displayable to the pilot because they are difficult to interpret.

[0033] In one embodiment, the determination of data of interest consists in implementing a conventional algorithm for detecting lines and patterns.

The Applicant's patent application FR3049744 describes an example of such a conventional detection algorithm. In one embodiment, the algorithm consists in calculating a bounding box of the elements of interest detected, in the form of a rectangle whose coordinates in pixels of two opposite corners correspond respectively to the smallest X coordinate and in Y is the smallest among the pixels belonging to the detected elements, and the largest X and Y coordinate among the pixels belonging to the detected elements. The area of the rectangle can be increased by a few percent, for example 10%, while remaining centered on the initial rectangle.

In a preferred embodiment, the step of determining data of interest consists in executing an artificial intelligence algorithm on the received sensor data, the algorithm implementing an artificial intelligence model for processing 'images trained for landing runway and approach ramp detection. The trained model is a model on board the aircraft for operational use, which was obtained during a learning phase, and in particular by deep learning by artificial neural network for runway and ramp detection. In an advantageous embodiment, the artificial neural network is a convolutional neural network (CNN for “Convolutional Neural Network”).

A classical model based on CNN (Convolutive Neural Network) can be set up for the detection and segmentation of track and ramp, for example by using a mask-RCNN (Regions with CNN features) - resNet 101 (101 layers) architecture ) [Mask R-CNN - Kaiming et al. 2017] From this model, a transfer learning (then finer learning) can be carried out to adapt to the track and ramp use case.

[0036] After a phase of learning convolutional neural network models, it is important to validate the model learned by a phase of testing on data. These data were not used during the learning phase to test the robustness of the model with respect to the variabilities it will face in an operational environment with different weather, different tracks, different light bars. Several iterations of learning and then testing may be necessary to obtain a valid and generic CNN model that meets the operational need. The validated model (i.e. its architecture and the learned hyper parameters) can be integrated into an onboard system on board an aircraft which includes at least one sensor of the same type as that used for training.

[0037] In the field of computer vision, the goal of deep learning is to model with a high level of data abstraction. In short, there are two phases: a learning phase and an inference phase. The learning phase defines and generates a trained AI model that meets the operational need. This model is then used in the operational context during the inference phase. The learning phase is therefore essential. Learning is considered efficient if it allows a predictive model to be defined that adapts well to learning data but is also able to predict well on data that was not seen during learning. If the model does not fit the training data, the model suffers from under-training. If the model adapts too well to the training data and is not able to generalize, the model suffers from overfitting.

[0038] In order to obtain the best model, the learning phase requires having collected a large database that is the most representative of the operational context and having labeled them with regard to ground truth (VT).

The ground truth or "ground truth" in English, is a reference image which represents an expected result after a segmentation operation. In the context of the invention, the ground truth of an image represents at least one track and an approach ramp as well as the visible ground. The result of a segmentation of an image is compared with the reference image or ground truth in order to evaluate the performance of the classification algorithm.

Thus from numerous labeled images, the learning phase makes it possible to define the architecture of the neural network and the associated hyper-parameters (the number of layers, the types of layers, the learning step, etc. .), then to seek by successive iteration, the best parameters (the weightings of the layers and between the layers) which best model the different labels (track / ramp). At each iteration of learning, the neural network propagates (extract / abstract characteristics specific to the objects of interest) and estimates the presence and position of the objects. From this estimate and the ground truth, the learning algorithm calculates a prediction error and backpropagation in the network in order to update the model parameters.

A training database must contain a very large number of data representing a maximum of possible situations, including for the context of the invention, different approaches on different tracks with different light ramps of approaches for different weather conditions . In order to implement a deep network learning method and learn to recognize a landing strip in the received sensor data, the database which is constituted contains a plurality of labeled or labeled datasets, where each set of labeled data corresponds to a pair (sensor data, ground truth VT). A VT ground truth for the operational context of the present invention is a description of various elements of interest to be recognized in the sensor data, including at least one runway and one approach ramp.

Returning to FIG. 2, after the step of determining data of interest in the received sensor data, the method makes it possible to calculate an area in which the approach light strip and / or the landing runway have been detected.

In one embodiment, the target area is calculated as being a rectangle surrounding the identified elements of interest. From the characteristics of the sensor and the attitude of the aircraft corresponding to the sensor data, the method allows to calculate the coordinates in heading and in elevation of two opposite corners of a surrounding rectangle.

After calculating the target area, the coordinates are sent to a head-up display device (or head-down on a head-down EVS or CVS device), carried or not, and the area is displayed in a step following (208) as a symbol conforming to the outside world, in the form of a surrounding rectangle. Figure 3 illustrates a head-up display of an EVS system with the display of a conforming symbol (302) obtained by the method of the invention.

[0045] The display of the symbol makes it possible to validate the trajectory towards the landing strip before the visual acquisition thereof by the pilot. The display of the symbol thus helps the pilot in acquiring the mandatory visual references before the DH since he knows he must search inside the rectangle.

In one embodiment, the head-up device can also display an SVS, an EVS or a CVS. In these last two EVS or CVS cases, the sensor image displayed is the one that fed the search for IΊA.

In another embodiment as illustrated in FIG. 4, the method makes it possible to display a head-up IR image in which the pilot searches for his visual references aided by a framing symbol (402), for example a rectangle originating from the detection of the landing strip by the CNN model or by any other runway and ramp detection algorithm, on data from an active sensor, for example a millimeter radar. This embodiment is particularly interesting because the active sensors have increased detection capacities compared to IR sensors in degraded visibility conditions, but the data provided by these sensors are difficult to interpret by the human eye. In this variant embodiment, the aircraft benefits from the lowering of the EFVS landing minima.

In another embodiment, the onboard sensor is a simple visible camera and its image is not presented to the pilot. Only the compliant symbol resulting from the method of the invention is presented to the pilot then providing him with an aid in the visual detection of the runway, for example during visual flights (VFR for “View Flight Rules” in English) but with reduced visibility. . This embodiment does not benefit from a reduction in landing minima. FIG. 5 illustrates a general architecture of a display system 500 making it possible to implement the method of the invention.

In a preferred implementation, a validated AI model (architecture and the learned hyper-parameters) is integrated into a system on board an aircraft which comprises at least one sensor of the same type as that used for the learning. The on-board system 500 also comprises a field database (BDT) 502, a database of elements of interest (BDEI) 504, a module for generating a synthetic view 506 in 3D towards the front of the aircraft (SVS) from the position and attitude of the aircraft received by sensors 508, and sensors 510, an analysis module 512 comprising at least one validated AI model, and a device for SVS display 514 for the aircraft crew. The display device 514, or human-machine interface, may be a head-down display (HDD), a transparent head-up display (HUD), a transparent head-mounted display (HWD), the windshield of the 'aircraft. Advantageously, the usual piloting symbology presenting the piloting parameters of the aircraft (attitude, heading, speed, altitude, vertical speed, speed vector, etc.) is superimposed on the synthetic 3D view. The analysis module 512 can be configured to correct the airstrip position shown on the SVS 506.

Thus, the present description illustrates a preferred implementation of the invention, but which is not limiting. Examples are chosen to allow a good understanding of the principles of the invention and a concrete application, but are in no way exhaustive and should allow those skilled in the art to make modifications and variants of implementation while keeping the same. principles.

The invention can be implemented from hardware and / or software elements.

It may be available as a computer program product on computer readable medium and includes code instructions for performing the steps of the methods in their various embodiments.

Claims

1. A method (200) for assisting the landing of an aircraft in conditions of degraded visibility, the method comprising at least the steps of:

- receive (202) during an approach phase to a landing runway data from a sensor, said landing runway and / or an approach ramp not being visible to the pilot from the cockpit;

- determining (204) in the received sensor data data of interest characteristic of said landing runway and / or said approach ramp;

- calculate (206) from the data of interest the coordinates of a target area; and

- display (208) on a head-up display screen a compliant symbol representative of the target zone, said compliant symbol being displayed before the aircraft reaches the decision height in order to provide the pilot with a visual cue in which to look for said landing strip and / or approach ramp.

2. The method of claim 1 wherein the step (202) of receiving sensor data consists of receiving data from a sensor on board the aircraft and looking forward, said sensor being chosen in the field. group of FLIR pilot type sensors, multispectral camera, LIDAR or millimeter radar.

3. The method of claim 1 or 2 wherein the step (204) of determining data of interest consists of executing an artificial intelligence algorithm on the received sensor data, the algorithm implementing a model of. artificial intelligence trained for image processing, obtained during a learning phase through deep learning.

4. The method of claim 3 wherein the deep learning is based on convolutional neural networks.

5. The method according to any one of claims 1 to 4 wherein the step (206) of calculating a target area consists in determining, from the characteristics of the sensor and the attitude of the aircraft corresponding to the sensor data. , heading and elevation coordinates of the target area.

The method according to any one of claims 1 to 5 further comprising after the step (206) of calculating a target area, a step of sending the coordinates of the target area to the head-up display device.

7. The method of claim 5 or 6 wherein the coordinates of the target area correspond to two opposite corners of a rectangle surrounding the data of interest characteristic of said landing runway and / or said approach ramp, and wherein the conformal symbol which is displayed is said surrounding rectangle.

8. The method according to any one of claims 1 to 7 wherein the head-up display step (208) consists in displaying the framing symbol on a fixed head-up screen of the cockpit and / or on a screen in head held high by the pilot.

9. An aircraft landing aid device (512), in particular in degraded visibility conditions, comprising means for implementing the steps of the aircraft landing aid method in visibility conditions. degraded, according to any one of claims 1 to 8.

10. The device according to claim 9 wherein the data allowing the calculation of the target area come from a first sensor, the device further comprising a second sensor capable of providing an image that can be displayed in the head-up device carried by the device. the pilot, the compliant symbol calculated from the data of the first sensor being displayed on said image supplied by the second sensor.

11. A man-machine interface comprising means for displaying a compliant symbol obtained by the method of any one of claims 1 to 8.

12. A landing aid system (500), in particular of the SVS, SGVS, EVS, EFVS or CVS type comprising an aircraft landing aid device (512), in particular in degraded visibility conditions. , according to one of claims 9 or 10.

13. An aircraft comprising an aircraft landing aid device, in particular in degraded visibility conditions according to one of claims 9 or 10.

14. A computer program comprising code instructions for executing the steps of the method for assisting aircraft landing, in particular in conditions of degraded visibility, according to any one of claims 1 to 8, when said program is executed by a processor.