CN114202706A - Perception-based autonomous landing of aircraft - Google Patents

Perception-based autonomous landing of aircraft Download PDF

Info

Publication number
CN114202706A
CN114202706A CN202110917124.6A CN202110917124A CN114202706A CN 114202706 A CN114202706 A CN 114202706A CN 202110917124 A CN202110917124 A CN 202110917124A CN 114202706 A CN114202706 A CN 114202706A
Authority
CN
China
Prior art keywords
runway
image
aircraft
mask
machine learning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110917124.6A
Other languages
Chinese (zh)
Inventor
C·比尔哈茨
M·卡洛克
E·R·缪尔
T·斯塔迪格
N·S·埃文斯
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Boeing Co
Original Assignee
Boeing Co
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Boeing Co filed Critical Boeing Co
Publication of CN114202706A publication Critical patent/CN114202706A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
    • G01S13/933—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of aircraft or spacecraft
    • G01S13/934—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of aircraft or spacecraft on airport surfaces, e.g. while taxiing
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
    • G01S17/933—Lidar systems specially adapted for specific applications for anti-collision purposes of aircraft or spacecraft
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • General Physics & Mathematics (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Electromagnetism (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Image Analysis (AREA)
  • Traffic Control Systems (AREA)

Abstract

The present application relates to awareness-based autonomous landing of an aircraft and provides a method of supporting an aircraft approaching a runway. Example embodiments relate to receiving a sequence of images captured by a camera onboard an aircraft. Example embodiments relate to applying received images to a machine learning model trained to detect runways or runway markings in the images, and generating a mask comprising pixel segments of object classes assigned to the runways or markings in the images. Example implementations may also involve applying the mask to a corner detector to detect points of interest on the mask and matching the points of interest to corresponding points on the runway or marker having known runway framed local coordinates. Example embodiments may also involve performing a perspective n-point estimate to determine a current attitude estimate of the aircraft relative to the runway or marker, and outputting the current attitude estimate for use during the final approach.

Description

Perception-based autonomous landing of aircraft
Technical Field
The subject disclosure relates generally to aircraft operation, and more particularly to autonomous detection of runways and runway markings.
Background
Safe, predictable, and reliable landings are an essential aspect of an effective aircraft-based transportation system. In many cases, a piloted aircraft can only visually land under high visibility and clear weather conditions. While external navigation systems such as Instrument Landing Systems (ILS) and Global Positioning Systems (GPS) may be used in some scenarios, some situations require visibility and cloud conditions below a specified decision height to allow the pilot to visually land. Even where external systems are available, those external systems can be subject to interruptions, inaccuracies and disturbances, which present technical challenges to efforts to automate the landing process.
The technical challenges associated with automatic landing or automatic generation of guidance during landing to assist pilots are complicated by factors such as the diversity and variability of weather and visibility conditions, the availability and feasibility of specialized equipment at different airports, and the robustness required for algorithmic methods of landing operation.
Disclosure of Invention
Example embodiments of the subject disclosure are directed to autonomous detection of runways and runway markings. Example embodiments of the subject disclosure address and overcome technical challenges associated with autonomous approach and landing operations in aircraft by: a machine learning model is used to identify features in the images of the landing environment, determine the attitude of the aircraft relative to the landing environment, and generate horizontal, vertical, and other position guidance required for landing. Some example embodiments are flexible in at least the following sense: such example embodiments may accommodate a range of passive and active sensors, including but not limited to, for example, visual imaging devices, hyperspectral imaging devices, optical radar, etc., to sense the landing environment and determine aircraft-related attitude, and may be used with various manned and unmanned aircraft of different sizes to land on visual, precise, and non-precise runways. Some example embodiments are also cost-effective in at least the following sense: such example embodiments do not require the installation of expensive ground infrastructure at a given airport.
Accordingly, the subject disclosure includes, but is not limited to, the following example embodiments.
Some example embodiments provide a method of supporting aircraft approach to a runway on an airport, the method comprising receiving a sequence of images of the airport captured by a camera onboard the aircraft approaching the runway; and for at least one image of the sequence of images, applying the image to a machine learning model trained to perform object detection and segmentation, wherein a runway or runway marker on the runway on the airport is detected in the image, and wherein a mask (mask) is generated comprising pixel segments in the image assigned to the object class of the runway or runway marker; applying the mask to a corner detector to detect points of interest on the mask and thereby detect points of interest on the runway or runway markings in the image; matching interest points on the runway or runway signs in the images with corresponding points on the runway or runway signs with known runway framed local coordinates; performing a perspective n-point estimation using the points of interest and the known runway framed local coordinates to determine a current attitude estimate of the camera and thereby the aircraft relative to the runway or runway marker; and outputting the current attitude estimate for at least one of guidance or control of the aircraft at the final approach.
In some example embodiments of the method of any preceding example embodiment, or any combination of any preceding example embodiments, the image is a color image, and the method further comprises converting the color image to a monochrome image.
In some example embodiments of the method of any preceding example embodiment, or any combination of any preceding example embodiments, the method further comprises cropping the image to reduce the field of view of the camera and enlarge only a portion of the airport where the runway or runway sign is located, before applying the image to the machine learning model.
In some example embodiments of the method of any preceding example embodiment, or any combination of any preceding example embodiments, the method further comprises accessing an earlier image in the sequence of images, and the earlier mask comprises respective pixel segments of an object category assigned to the runway or runway sign in the earlier image; and identifying a portion of the framing earlier mask in the earlier image, and thereby identifying a portion of the airport at which the runway is located, and wherein cropping the image comprises cropping the image to a portion of the image corresponding to the portion of the earlier image.
In some example embodiments of the method of any preceding example embodiment, or any combination of any preceding example embodiments, the object detection and segmentation is a one-class object detection and segmentation, wherein only a runway or runway sign is detected in the image and the mask includes only pixel segments in the image assigned to the object class of the runway or runway sign.
In some example implementations of the method of any preceding example implementation, or any combination of any preceding example implementations, the object detection and segmentation is a multi-class object detection and segmentation, wherein a runway and a runway marker are detected in the image, and the mask comprises a segment of pixels in the image assigned to the object classes of the runway and runway marker, and the point of interest comprises a point of interest on the runway and runway marker.
In some example embodiments of the method of any preceding example embodiment, or any combination of any preceding example embodiments, the method further comprises determining an operating state of the aircraft; and selecting, based on the operating state, either a first machine learning model or a second machine learning model as a machine learning model to which the image is applied, the first machine learning model being trained to perform single-class object detection and segmentation, and the second machine learning model being trained to perform multi-class object detection and segmentation.
In some example embodiments of the method of any preceding example embodiment, or any combination of any preceding example embodiments, the machine learning model further determines a confidence interval associated with detection of the runway or runway marker, and the method further comprises applying the mask to a confidence filter, the filter passing the mask to the corner detector only if the confidence interval is above a threshold confidence interval dynamically set based on an operating state of the aircraft.
In some example embodiments of the method of any preceding example embodiment, or any combination of any preceding example embodiments, performing perspective n-point estimation comprises determining a current pose estimate of the camera in camera coordinates and transforming the camera coordinates into corresponding runway framed local coordinates that are output for at least one of guidance or control of the aircraft.
In some example embodiments of the method of any preceding example embodiment, or any combination of any preceding example embodiments, performing a perspective n-point estimate further comprises determining a confidence interval associated with the current pose estimate, and the method further comprises: accessing one or more additional pose estimates and associated confidence intervals; performing sensor fusion on the current attitude estimate and the one or more additional attitude estimates using the confidence intervals and associated confidence intervals to generate an updated current attitude estimate; and outputting the updated current attitude estimate for at least one of guidance or control of the aircraft.
Some example embodiments provide an apparatus for supporting aircraft access to a runway at an airport, the apparatus comprising: a memory configured to store computer readable program code; and processing circuitry configured to access the memory and execute the computer readable program code to cause the apparatus to perform at least the method of any of the preceding example embodiments, or any combination of any of the preceding example embodiments.
Some example embodiments provide a computer-readable storage medium for supporting an aircraft to approach a runway on an airport, the computer-readable storage medium being non-transitory and having computer-readable program code stored therein that, in response to execution by processing circuitry, causes an apparatus to perform at least the method of any preceding example embodiment or any combination of any preceding example embodiments.
These and other features, aspects, and advantages of the subject disclosure will become apparent from the following detailed description, which is to be read in connection with the accompanying drawings, which are briefly described below. The subject disclosure includes any combination of two, three, four, or more features or elements set forth in this disclosure, whether or not such features or elements are expressly combined or otherwise referred to in the specific example embodiments described herein. The disclosure is intended to be read in its entirety such that any separable feature or element of the disclosure, in any of its aspects and example embodiments, should be considered combinable unless the context of the disclosure clearly dictates otherwise.
Therefore, it should be understood that this summary is provided merely for purposes of summarizing some example embodiments so as to provide a basic understanding of some aspects of the disclosure. Therefore, it should be understood that the above-described example embodiments are merely examples and should not be construed to narrow the scope or spirit of the present disclosure in any way. Other exemplary embodiments, aspects and advantages will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of some described exemplary embodiments.
Drawings
Having thus described example embodiments of the present disclosure in general terms, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and wherein:
FIG. 1 illustrates an example aircraft that may benefit from example embodiments of the subject disclosure;
FIG. 2 is a block diagram of a system for supporting aircraft access to runways at an airport in accordance with an example embodiment of the subject disclosure;
FIG. 3 is a block diagram of a system pipeline that may be used in conjunction with a system for supporting aircraft access to runways at an airport in accordance with an example embodiment of the subject disclosure;
FIG. 4A is a block diagram of an image of an airport that may be used in connection with example embodiments of the subject disclosure;
FIG. 4B is a block diagram illustrating an image of an airport of an example embodiment of the subject disclosure;
FIG. 5 is a flow chart illustrating various steps in a method for supporting aircraft access to runways at an airport according to an example embodiment of the subject disclosure; and is
Fig. 6 illustrates an apparatus according to some example embodiments of the subject disclosure.
Detailed Description
Some embodiments of the subject disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the disclosure are shown. Indeed, various embodiments of the disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these example embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. For example, unless otherwise indicated, references to things as first, second, etc. should not be construed as implying a particular order. Also, things that can be described as being above something else (unless otherwise indicated) may instead be below, and vice versa; and similarly, things described as being to the left of others may instead be to the right, and vice versa. Like reference numerals refer to like elements throughout. To the extent that the terms "includes," "including," "has," "contains," and variants thereof are used herein, such terms are intended to be inclusive in a manner similar to the term "comprising" as an open transition word without precluding any additional or other elements.
Example embodiments of the subject disclosure are directed to autonomous detection of runways (and/or heliports) and runway markings. To address and overcome technical challenges associated with autonomous approach and landing operations for aircraft, example embodiments of the subject disclosure relate to using machine learning models to identify features in landing environment images and then determine the attitude of the aircraft relative to the landing environment in order to generate horizontal, vertical, and other positional guidance required for landing.
Conventional methods using computer-generated landing maneuver assistance provide limited runway edge detection using resource-intensive, rule-based algorithms. These rule-based, resource-intensive runway detection methods lack the robustness required for many real-world implementations of aircraft and airports. In particular, some conventional approaches require prior knowledge of runway characteristics, such as the location, orientation, length, width, sign type and surface material of a given runway, and often require the installation of expensive ground equipment at the airport. Such methods are not easily generalized to new runways and may be less effective if the aircraft is redirected to a new or different airport. Furthermore, even in the event that a portion of a runway is detected, conventional approaches lack the ability to effectively guide and control the approach of aircraft to the airport using such detection.
Example embodiments of the subject disclosure combine computer vision techniques with machine learning perception techniques as follows: allowing for the identification of a different set of runways, runway markings, runway corner points, threshold markings, aiming pads, etc. under a wide range of visibility conditions. Some example embodiments accommodate a range of passive and active sensors, including but not limited to vision imaging devices, hyperspectral imaging devices, light radar, etc., for example, to sense the landing environment and determine aircraft-related attitude, and may be used with various different sized manned and unmanned aircraft to land on visual, precise, and non-precise runways.
Some example embodiments relate to convolutional neural networks as part of a multi-mode method of detecting runways and/or runway features. In some cases, example embodiments of the subject disclosure supplement visual data with optical radars and/or radar range measurements of the ground and associated obstacles. Regardless of the basic source of the visual and/or supplemental data used, example embodiments of the subject disclosure are configured to generate and provide accurate and precise pose estimation and guidance without requiring extensive prior knowledge of the runway or expensive ground infrastructure installed at a given airport.
FIG. 1 illustrates an aircraft 100 that may benefit from example embodiments of the subject disclosure. As shown, the aircraft includes a body 102 having a fuselage 104, wings 106, and a tail 108. The aircraft also includes a plurality of advanced systems 110, such as a propulsion system 112, an electrical system 114, a hydraulic system 116, and/or an environmental system 118. Any number of other systems may be included on the aircraft.
Fig. 2 illustrates a system 200 for supporting aircraft approach to a runway in accordance with an example embodiment of the subject disclosure. The system 200 may include any of a number of different subsystems (each being a separate system) for performing one or more functions or operations. As shown, in some examples, system 200 includes one or more cameras 202 (located on an aircraft and configured to produce a sequence of images 204), a pose estimation system 205, an image processing system 206, a first machine learning model 208, and a second machine learning model 210 (configured to produce one or more masks 212 that may be applied to the images), a corner detector 214, a perspective n-point system 216 (configured to generate one or more pose estimates 218), a guidance system 220, and a control system 222. The subsystems including the camera, image processing system, first machine learning model, second machine learning model, corner detector, perspective n-point system, guidance system, and control system may be co-located with or directly coupled to each other, or in some examples, various ones of the subsystems may communicate with each other across one or more computer networks 224. In some example embodiments, any of the subsystems may access and/or otherwise interact with one or more additional information sources 226, any of which may be co-located with each other or directly coupled to each other and/or with and/or directly coupled to system elements, or accessible via a computer network and/or cloud-based architecture.
Further, while shown as part of system 200, it is to be understood that any one or more of the above may function or operate as a stand-alone system, regardless of any of the other subsystems. It should also be understood that the system may include one or more additional or alternative subsystems other than those shown in fig. 2.
In example embodiments of the subject disclosure, one or more elements from system 200 may be placed on or otherwise added to an aircraft, such as aircraft 100 shown in fig. 1. In such example implementations, system 200 and any of its elements may communicate and/or otherwise interact with any of systems 110 of aircraft 100. For example, a camera 202 may be mounted to the aircraft 100 in a configuration that allows the camera to capture a sequence of images 204 of the environment in the direction of travel of the aircraft 100. In some such example embodiments, the attitude estimation system 205 is configured to incorporate the image processing system 206 and/or the perspective n-point system 216 (both of which may also be incorporated into the aircraft 100), the attitude estimation system 205 providing information to a guidance system 220 on the aircraft 100 and/or a control system 222 of the aircraft 100 to assist in landing of the aircraft 100.
In some example embodiments, the system 200 is configured to receive an image sequence 204 of an airport as the aircraft 100 approaches a runway at the airport. In some such example embodiments, the sequence of images is captured by a camera 202 mounted on the aircraft 100 proximate the runway.
Camera 202 may be any camera capable of capturing a sequence of images including, but not limited to, a visual imaging device, a hyperspectral imaging device, a light radar imaging device, a radar imaging device, and the like. In some cases, camera 202 is mounted to and/or otherwise incorporated into aircraft 100 in a configuration that allows the camera to capture a view of the environment in front of aircraft 100 in the direction of travel of aircraft 100.
In some example embodiments, for at least one image 204A in the sequence of images 204, the system 200 is configured to apply the image to a machine learning model, such as the first machine learning model 208 or the second machine learning model 210, for example, trained to perform object detection and segmentation, wherein a runway on an airport or a runway marker on the runway is detected in the image. In some example embodiments, the first machine learning model 208 and/or the second machine learning model 210 incorporate a convolutional neural network. It should be understood that any neural network or other machine learning model type may be used in example embodiments of system 200.
In some example implementations, the image 204A may be processed before, concurrently with, and/or after the image 204A is applied to one or more machine learning models 208, 210, such as by the image processing system 206. For example, in some example embodiments, image 204A is a color image, and image processing system 206 and/or another aspect of system 200 is configured to convert the color image to a monochrome image.
In some example embodiments, the image processing system 206 and/or another aspect of the system 200 is configured to further crop the image 204A to reduce the field of view of the camera 202 and enlarge only a portion of the airport where the relevant runway or relevant runway sign is located. In some such example embodiments, cropping of the image 204A is performed prior to applying the image 204A to the machine learning models 208 and/or 210.
In some example embodiments, the first machine learning model 208 and/or the second machine learning model 210 are configured to generate at least one mask 212. In some example embodiments, the mask 212 is generated such that it is configured to include a segment of pixels in the image 204A that are assigned to an object category of the relevant runway or relevant runway marker. In some example embodiments, the mask 212 is stored and may be accessed by the system 200 and/or another system for use with subsequent image sequences.
In some example embodiments of the system 200, the object detection and segmentation performed by the first machine learning model 208 or the second machine learning model 210 is a single class of object detection and segmentation, where only a runway or runway marker is detected in the image 204A. In some such example embodiments, the generated mask 212 includes only the pixel segments in the image 204A that are assigned to the object categories of the runway or runway marker.
Some example embodiments relate to instance-guided (instance-wise) multi-class object detection and segmentation. In some example implementations of the system 200, the object detection and segmentation performed by the first machine learning model 208 or the second machine learning model 210 is multi-class object detection and segmentation, wherein runways and runway markings are detected in the image 204A, and the mask 212 includes pixel segments of the object classes in the image 204A that are assigned to the runways and runway markings. In some such example embodiments, the points of interest identified in the images include points of interest on runways and runway signs.
In some example embodiments, system 200 is configured to apply mask 212 to corner detector 214. In some such example embodiments, the corner detector is configured to detect points of interest on the mask 212, and thus on runways or runway markings in the image 204A. In some example implementations, the points of interest may coincide with portions of the runway itself (such as corners, edges, or other features of the runway). In some example embodiments, the points of interest may coincide with portions of the runway markings (such as alignment pads, letters, numbers, geometry, etc. present on a given runway).
As such, in some example embodiments, the system 200 is further configured to match points of interest on the runway or runway marker in the image 204A to corresponding points on the runway or runway marker having known runway framed local coordinates. It will be appreciated that in order to provide accurate and precise information that may be used to guide or control the aircraft 100 during a landing operation, it may be necessary for the system to determine how the appearance and relative positioning of runway related elements and/or runway signage related elements in a given image correspond to the actual appearance and relative positioning of those elements on the runway itself.
In example embodiments where runway outlined local coordinates are known, those known runway outlined local coordinates may be matched to corresponding points in the image by image processing system 206 or another component of system 200. For example, local coordinates of runway framing may be pre-captured or otherwise determined for a given runway, such as by ascertaining the location of various symbols, lights, equipment, or other runway markings that remain at a given orientation relative to the runway. In some example embodiments, runway framed local coordinates are stored in a local data store and/or accessed by the system 200 from a remote data store prior to attempting a landing on a given runway. In some example implementations, information from one or more additional information sources 226 may be used in conjunction with the machine learning models 208, 210 and/or the corner detector 214 to identify points of interest in the images and/or to match points of interest in the images to corresponding points on the runway. Such additional information may include, for example, global positioning system information, compound flight point (MAP) information, Instrument Landing System (ILS) information, and the like.
As shown in fig. 2, image processing system 206 is shown incorporating at least a first machine learning model 208, a second machine learning model 210, and a corner detector 214, such that the image processing system is configured to receive image sequence 204 and process at least one of the received images by applying image(s) 204 to the machine learning model, generating mask 212, applying mask 212 to corner detector 214, and otherwise processing received image 204A. It should be understood that in some example embodiments, image processing system 206 and/or its constituent elements may be implemented individually and/or in different groupings, and additional elements may be incorporated.
Regardless of the precise configuration of the image processing system 206 and its associated elements, some example embodiments of the system 200 are also configured to perform perspective n-point estimation. In some such example embodiments, the system 200, such as through operation of the perspective n-point system 216, is configured to determine a current pose estimate 218 of the camera 202, and thus a pose estimate of the aircraft 100 relative to the runway or runway marker, using the points of interest and known runway framed local coordinates.
In some example embodiments, the perspective n-point system 216 is configured to determine the current pose estimate 218 of the camera 202 in camera coordinates and to transform the camera coordinates into corresponding runway framed local coordinates 236, the local coordinates 236 being output for at least one of guidance or control of the aircraft 100. In some such example embodiments, the runway outlined local coordinates are communicated from the perspective n-point system 216 to a guidance system 220, a control system 222, and/or another advanced system 110 of the aircraft 100.
In some example embodiments, the perspective n-point system 216 is further configured to determine a confidence interval 228 associated with the current pose estimate 218. In some example embodiments, the confidence interval may be a confidence value and/or a multidimensional confidence matrix.
In some such example embodiments, the system 200 is further configured to access one or more additional pose estimates 218A and associated confidence intervals 228A. The system 200 may then perform sensor fusion on the current pose estimate 218 and one or more additional pose estimates 218A using the confidence interval 228 and the associated confidence intervals to generate an updated current pose estimate 218B. In some such example embodiments, the system is further configured to output the updated current attitude estimate 218B for at least one of guidance or control of the aircraft 100, such as by providing the updated attitude estimate to a guidance system 220, a control system 222, or another advanced system 110 of the aircraft 100.
In some such example embodiments, system 200 is further configured to output current attitude estimate 218 for at least one of guidance or control of aircraft 100 at the time of the final approach. For example, the current attitude estimate is communicated by the perspective n-point system 216 to a guidance system 220 of the aircraft, a control system 222 of the aircraft 100, and/or another advanced system 110 of the aircraft 100, where the current attitude estimate may be used for autonomous landing operations and/or to support a pilot performing landing operations.
In some example embodiments, the system 200 is configured to iteratively process an image 204 captured by the camera 202. In some such example embodiments, the image processing system 206 and/or another aspect of the system 200 is configured to access an earlier image, such as image 204B in the sequence of images. The system may also access an earlier mask, such as mask 212A, that includes corresponding pixel segments in the earlier image that are assigned to the object category of the relevant runway or runway marker. By accessing the earlier image and the earlier mask, the system 200 may be configured to identify a portion of the framing earlier mask in the earlier image, and thereby identify the portion of the relevant airport where the runway is located. In example implementations involving cropping an image (such as image 204A), system 200 may be configured to crop an image into a portion of the image that corresponds to a portion of an earlier image.
In some example embodiments, the system 200 uses and generates additional information that may be used in conjunction with the received images 204, masks 212, and attitude estimates 218 to provide guidance and or control information to support landing operations of the aircraft 100. In some such example embodiments, the system is configured to determine an operating state 230 of the aircraft, such as a state of the propulsion system 112, the electrical system 114, the hydraulic system 116, the environmental system 118, and/or any of the advanced systems 110. In some such example embodiments, the image processing system 206 or other aspects of the system 200 are configured to select the first machine learning model 208 or the second machine learning model 208 as the machine learning model to which the image 204A is applied based on the operating state. For example, the first machine learning model 208 may be configured and trained to perform single-class object detection and segmentation, and the second machine learning model 208 may be configured and trained to perform multi-class object detection and segmentation. In some example embodiments, the operating state of the aircraft 100 is determined based on information obtained from one or more of the advanced systems 110.
In another example implementation where the system 200 uses and generates additional information, a given machine learning model, such as the first machine learning model 208 or the second machine learning model 210, for example, is further configured to determine a confidence interval 232 associated with the detection of a relevant runway or a relevant runway marker. In some such example embodiments, image processing system 206 or another aspect of system 200 is further configured to apply mask 212 to confidence filter 234, which confidence filter 234 is configured to pass mask 212 to corner detector 214 only when confidence interval 232 is above a threshold confidence interval that is dynamically set based on operating state 230 of aircraft 100.
Fig. 3 is a block diagram of a system pipeline 300 that may be used in connection with a system for supporting aircraft access to runways at airports, such as system 200, according to an example embodiment of the subject disclosure. As discussed and otherwise disclosed herein, a sequence of images, such as the sequence of images 204 captured by the camera(s) 202, is applied to one or more trained machine learning models, such as, for example, the first machine learning model 208 and the second machine learning model 210, and processed to generate information that may be used to support guidance and/or control of the aircraft 100 during landing operations. The system pipeline 300 shown in FIG. 3 presents an example method of acquiring and processing information for training machine learning models and providing information for guidance and control of aircraft.
As shown in FIG. 3, the system pipeline 300 includes a block 302 that involves building a data set that can be used to train one or more machine learning models. In some example implementations of block 302, an image 304 (which may include one or more instances of the image sequence 204) is processed to generate a set of extracted frames 306. In some example implementations, the extracted frames 306 include frames from various image sequences, and may include depictions of one or more features that one or more machine learning models are to be trained to detect. For example, the extracted frames 306 may include various views of a runway, runway markings, and/or other image elements relevant to the detection of the presence or absence of a runway or runway marking in an image. The extracted frames 306 are then annotated with one or more pose tags 307, such that the pose tags 307 and information associated with a given pose are linked to the corresponding image in the extracted frames 306.
As shown in fig. 3, the system pipeline 300 also includes a block 308 that involves using the images 304 and/or extracted frames 306 as part of a data set for deep learning and training of a relevant machine learning model (e.g., 208, 210). Any method of deep learning suitable for use with the associated machine learning model may be used in connection with the example implementation of block 308. In some example embodiments, at least one model is trained using a deep learning approach at block 310 to detect a runway and/or portions of a runway. In some example embodiments, at least one model is trained using a deep learning approach to detect one or more runway markings at block 312.
As shown in FIG. 3, the system pipeline 300 also includes a block 314 that involves attitude estimation. In some example implementations of block 314, the trained machine learning model, such as the first machine learning model 208 and/or the second machine learning model 210, for example, is configured to generate masks in a defined format that can be used to perform pose estimation and/or images processed in conjunction with the generated masks. For example, the perspective n-point system 216 may receive the mask 212 and/or a version of the image 204A processed with the mask and generate pose estimates 218 that relate relevant points of interest on the image to their associated points on the runway itself. At block 316, the attitude estimate is output in a form that may be used as a guide and/or other information that may be used by the aircraft 100 or its pilot in connection with a landing maneuver.
Fig. 4A depicts an airport 400 at which example embodiments of the subject disclosure may be deployed and which illustrates various aspects of the subject disclosure. As shown in fig. 4, an airport 400 includes a runway 402, the runway 402 configured to allow an aircraft, such as aircraft 100, to land. As shown in fig. 4A, runway 402 also includes one or more runway markings 404. It should be understood that the runway signs 404 shown in fig. 4A are non-exclusive examples of runway signs that may be on a given runway, and that other signs (such as letters, numbers, lines, curved patterns, geometric shapes, etc.) may be used for embodiments of the runway signs. Fig. 4A also shows a point 406 representing a known location on the runway 402 and/or runway markings 404. In some example embodiments, the location of point 406 is known and is represented in track-framed coordinates.
Fig. 4B illustrates an image 408 of an airport 400, according to some example embodiments of the subject disclosure. In some example embodiments, the image 408 may be one image 204A of the sequence of images 204 captured by the camera 202 on the aircraft 100 while approaching the airport 400. As shown in fig. 4B, the image 408 depicts a view of one or more of the runway 402 and the runway markings 404. Although the example image in fig. 4B depicts a view of the runway 402 in its entirety, it should be understood that in some embodiments, the image may provide a partial view of the runway 402 and/or runway markings 404, depending on the orientation of the aircraft 100 and/or cameras 202, and the conditions of the airport environment.
As shown in fig. 4B, a mask 410 (e.g., mask 212) is applied to the image 408, identifying an area in the image 408 that includes the runway 402 and/or one or more runway markings 404. A set of points of interest 412 also exist on the image, corresponding to a portion of the runway and/or runway marker in the image 408. In some example embodiments of the subject disclosure, a point of interest 412 is identified in the image 408 and then matched with a corresponding point 406 having known, runway framed coordinates.
As discussed herein and otherwise disclosed, the points of interest 412 identified in the given image 408 and the corresponding known, runway framed coordinates 406 at the relevant airport 400 are used with the perspective n-point estimate to determine the current pose estimate 218 of the camera 202, and thus the current pose estimate 218 of the aircraft 100 relative to the runway 402 or runway marker 404.
The perspective n-point estimate may be used to find a deterministic solution (deterministic solution) that maps real-world points to pixel value positions in a given image of the environment in which the real-world points are located. The camera matrix associated with a given camera (e.g., such as camera 202) may be represented as a zoom and principal point matrix a, and may be represented as listed in equation 1 below:
Figure BDA0003206031280000131
in equation 1, the nth point is considered to have real world coordinates [ X [ ]n,Yn,Zn]tAnd camera coordinate [ U ]n,Vn,Wn]t. The presence of this point also causes the ray cast to the W-1 plane to be cast at un、vnWhere they intersect. As expressed in equation 1, cx、cyIs a zoom focal length and fx、fyIs the principal point.
The transformation of the nth point from the camera coordinate system to the global coordinate system can be expressed as listed in equation 2 below:
Figure BDA0003206031280000132
in equation 2, R is the correlation basis rotation matrix and T is the translation vector. Therefore, the projection of a point in the camera coordinate system to the plane W — 1 can be expressed as listed in the following equation 3:
Figure BDA0003206031280000133
in equation 3, s is an unknown scaling factor. To produce a transformation from real world coordinates to a projection onto an image plane, equations 1, 2 and 3 can be combined and expressed as listed in equation 4 below:
Figure BDA0003206031280000134
as expressed in equation 4, the scaling factor s, the value of the rotation matrix R, and the translation vector T are unknown. However, example embodiments of the subject disclosure relate to resolving an unknown in a formula. For example, the perspective n-point system 216 may use an interactive algorithm or a programmed method. One such method involves receiving four or more mappings of pixels to real world coordinates and resolving the unknowns. In some such methods, the values returned from a given algorithm or procedure are rotation and translation vectors to move from global coordinates to camera coordinates. In an example implementation directed to estimating the position of the camera 202 (or aircraft 100) relative to a desired real world landing position (which is represented by a global coordinate system origin), the coordinate system origin of the camera must be transformed to a global coordinate point. This transformation can be expressed as listed in equation 5 below:
Figure BDA0003206031280000135
equation 5 may be adjusted to directly determine the linear positioning of the aircraft 100 (equipped with the camera 202) relative to the origin of the landing site, as represented by equation 6 below:
Figure BDA0003206031280000141
in some example embodiments, a difference matrix is constructed and the change in order of euler angles of a conventional aircraft is taken into account to determine the pose of a given aircraft 100 from the rotation matrix output. The rotation matrix from the camera coordinate system to the aircraft coordinate system representing a steady level flight can be expressed as set forth in equation 7 below:
Figure BDA0003206031280000142
in such an exemplary embodiment, the difference matrix between the output of SolvePnP and the stable horizontal flight can be expressed as set forth in equation 8 below:
Figure BDA0003206031280000143
it should be understood that the difference matrix reveals a rotation matrix that will be used to change from a steady level flight to the current attitude of the aircraft 100. In some example embodiments, the rotation matrix is still based on the camera coordinate system, while the aircraft pose is typically applied in order of 3-2-1 Euler angles (yaw, then pitch, then roll). To maintain the correct order in the formulas presented herein, the order of the Euler angles is taken to be 2-1-3 (so that [ X, Y, Z ] maps to [ W, U, V ]). The combined rotation matrix for roll φ, pitch θ, and yaw ψ may be expressed in camera coordinates as set forth in equation 9 below:
Rdiff＝Rz(φ)Rx(θ)Ry(ψ) (9)
equation 9 can be expanded and expressed as listed in equation 10 below:
Figure BDA0003206031280000144
the matrices in equation 10 may be multiplied together to establish the terms listed below in equations 11, 12, and 13 below:
equation 11:
Rdiff，21＝-sin(θ) (11)
equation 12:
Rdiff，20＝sin(ψ)cos(θ) (12)
equation 13:
Rdiff，01＝sin(ψ)cos(θ) (13)
in some example embodiments, the terms from equations 11, 12, and 13 are used to determine the pose of aircraft 100 in conventional aircraft coordinates from the difference matrix, which in turn may be used to allow system 200 to generate information for guidance and/or control of the aircraft as it approaches airport 400 and runway 402 thereon.
Fig. 5 is a flow chart illustrating various steps in a method 500 of supporting an aircraft 100 in proximity to a runway 402 at an airport 400, according to an example embodiment of the subject disclosure. As shown at block 502, the method includes receiving a sequence of images captured by an onboard camera(s). Some example implementations of block 502 involve receiving a sequence of images of an airport captured by a camera onboard an aircraft proximate to a runway. In some such example embodiments, the image processing system 206 receives a sequence of images 204 captured by the cameras 202 onboard the aircraft 100 proximate the runway 400.
As shown at block 504, the method 500 includes, for at least one image in a sequence of images, applying the image to a machine learning model. Some example implementations of block 504 involve applying the image to a machine learning model trained to perform object detection and segmentation, wherein a runway or runway marker on the runway on the airport is detected in the image, and wherein a mask is generated that includes a segment of pixels in the image that are assigned to the object class of the runway or runway marker. In some such example embodiments, the image 204A is applied by the image processing system 206 or another aspect of the system 200 to a machine learning model, such as the first machine learning model 208 or the second machine learning model 210, to perform object detection and segmentation and to generate the mask 212.
As discussed herein with respect to block 504, some example embodiments of method 500 include detecting an object and segmenting an image, as shown at block 504A. Some example implementations of block 504A involve performing object detection. For example, the first machine learning model 208 or the second machine learning model 210 may be trained and configured to detect the runway 402 or runway markings 404, and may further identify points of interest 412 in the images 204A and/or 408.
In some example embodiments, as shown at block 504B, the method 500 includes generating a mask. Some example implementations of block 504B involve performing segmentation to produce a mask. For example, the first machine learning model 208 or the second machine learning model 210 may be trained and configured to generate a mask, such as mask 212 or mask 410. In some such example embodiments, the mask includes a segment of pixels in the image that are assigned to an object category of the relevant runway or relevant runway sign. In some example embodiments, the object detection and segmentation is a single class of object detection and segmentation, wherein only a runway or runway marker is detected in the image and the mask includes only pixel segments in the image that are assigned to the object class of the runway or runway marker. In some example embodiments, the object detection and segmentation is a multi-class object detection and segmentation, wherein a runway and a runway marker are detected in the image, and the mask comprises a segment of pixels in the image assigned to object classes of the runway and runway marker, and the point of interest comprises a point of interest on the runway and runway marker.
As shown at block 506, the method 500 includes applying the mask to the corner detector. Some example implementations of block 506 involve applying the mask to a corner detector to detect points of interest on the mask, and thus detect points of interest on runways or runway markings in the image. In some such example implementations, image processing system 206 or another aspect of system 200 applies mask 212 to corner detector 214 to detect points of interest on mask 212. Referring to FIG. 4B, the point of interest may be, for example, point of interest 412 on mask 410.
As shown at block 508, the method 500 includes matching points of interest in the image with corresponding real-world points. Some example implementations of block 508 involve matching points of interest on a runway or runway marker in the image to corresponding points on the runway or runway marker with known runway framed local coordinates. For example, the image processing system 206 or another aspect of the system 200 may be configured to match the point of interest 412 in the image 408 with the corresponding point 406 on the runway 402 or runway logo 404.
As shown at block 510, the method 500 includes performing perspective n-point estimation. Some example embodiments of block 510 involve performing a perspective n-point estimate using the point of interest and known runway framed local coordinates to determine a current attitude estimate of the camera and, thus, the aircraft relative to the runway or runway marker. For example, the perspective n-point system 216 may determine the current pose estimate 218 of the camera 202 by using the point of interest 412 and known runway outlined local coordinate points 236 or 406 to ascertain the pose of the aircraft 100 relative to the runway 402 or runway markings 404.
As shown at block 512, method 500 includes outputting a current pose estimate. Some example implementations of block 512 involve outputting the current attitude estimate for use in at least one of guidance or control of the aircraft at the final approach. For example, the system 200 may be configured to cause the attitude estimate 218 to be output to a guidance system 220 (e.g., including a GUI or other display device) or a control system 222, where the attitude estimate may be used to guide or control the aircraft 100 in the eventual approach to the airport 400.
In some example embodiments, the method 500 may include one or more additional blocks. For example, in some embodiments, the method includes, as shown in block 514, converting the multi-channel image to a single-channel image. Some example embodiments of block 514 relate to the case where the image is a multi-channel image, and the method further includes converting the multi-channel image to a single-channel image. For example, image processing system 206 or another aspect of system 200 can convert image 204A from a color image to a monochrome image. In another example, the image 204A may be an image captured by a thermal camera, and the multi-channel thermal image may be converted to a single channel image.
In some example embodiments, the method 500 includes, as indicated by block 516, cropping the image. Some example implementations of block 516 involve cropping the image to reduce the field of view of the camera and zoom in on only a portion of the airport where the runway or runway marker is located, before the image is applied to the machine learning model. For example, before the image is applied to the first machine learning model 208 and/or the second machine learning model 210, the image processing system 206 or another aspect of the system 200 may crop the image 204A to reduce the field of view of the camera 202 and enlarge only a portion of the airport 400 where the runway 402 or runway signage 404 is located.
In some example embodiments involving cropping, method 500 includes, as shown at block 518, accessing an earlier image and a mask. Some example implementations of block 518 involve accessing an earlier image in the sequence of images, while the earlier mask includes a corresponding segment of pixels in the earlier image that are assigned to an object category of the runway or runway marker. For example, the image processing system 206 or another aspect of the system 200 may access an earlier image 204B from the image sequence 204 and access an earlier mask 212A, the earlier mask 212A including respective pixel segments of the object categories assigned to the runway 402 or runway markings 404 in the earlier image.
In some example embodiments involving cropping, method 500 includes, as shown at block 520, identifying a portion of the framing earlier mask in the earlier image. Some example implementations of block 520 involve identifying a portion of the framing earlier mask in the earlier image, and thereby identifying the portion of the airport where the runway is located. For example, image processing system 206 or some other portion of system 200 may be configured to identify a portion of framing earlier mask 212A in earlier image 204B, and thereby identify a portion of airport 400 at which runway 402 is located.
In some example embodiments involving cropping, method 500 includes, as shown at block 522, cropping the image to a portion corresponding to an earlier image. In some example implementations of block 522, cropping the image includes cropping the image as a portion of the image that corresponds to a portion of an earlier image. For example, image processing system 206 or another aspect of system 200 may be configured to crop image 204A into a portion of the image corresponding to a portion of earlier image 204B (such as a portion of framing earlier mask 212A).
In some example embodiments, method 500 includes, as shown at block 524, determining an operating state of the aircraft. For example, the system 200 may be configured to determine an operating state 230 of the aircraft 100. In some example embodiments, the system may receive information associated with the operating state of the aircraft from one or more advanced systems of the aircraft 110 and/or from one or more additional information sources 226.
In some example embodiments, the method 500 includes selecting either the first machine learning model or the second machine learning model, as shown in block 526. Some example implementations of block 526 involve selecting either a first machine learning model or a second machine learning model as the machine learning model to which the image is applied, the first machine learning model being trained to perform single-class object detection and segmentation, and the second machine learning model being trained to perform multi-class object detection and segmentation. For example, the image processing system 206 or another aspect of the system 200 may be configured to select the first machine learning model 208 or the second machine learning model 210 as the machine learning model to which the image 204A is applied, where the first machine learning model is trained to perform single-class object detection and segmentation and the second machine learning model is trained to perform multi-class object detection and segmentation.
In some example embodiments, the method 500 includes, as shown at block 528, determining a confidence interval for object detection. Some example embodiments of block 528 involve the machine learning model being further configured to determine a confidence interval associated with the detection of the runway or runway marker. For example, the first machine learning model 208 and/or the second machine learning model 210 may be configured to determine a confidence interval 232 associated with the detection of the runway 402 or runway markings 404.
In some example embodiments, the method 500 includes, as shown at block 530, applying the mask to a confidence filter. Some example implementations of block 530 involve applying the mask to a confidence filter that passes the mask to the corner detector only when the confidence interval is above a threshold confidence interval that is dynamically set based on the operating state of the aircraft. For example, image processing system 206 or another aspect of system 200 may be configured to apply mask 212 to confidence filter 234, which is configured to pass the mask to corner detector 214 only when confidence interval 232 is above a threshold confidence interval dynamically set by another aspect of image processing system or system 200 based on operating state 230 of aircraft 100.
In some example embodiments, method 500 includes, as shown in block 532, determining a pose estimate in camera coordinates. Some example implementations of block 532 involve performing perspective n-point estimation by determining a current pose estimate for the camera at least in camera coordinates. For example, the perspective n-point system 216 or another aspect of the system 200 may be configured to determine a current pose estimate 218 of the camera 202.
In some example embodiments, the method 500 includes, as shown in block 534, transforming the camera coordinates to runway framing coordinates. Some example implementations of block 534 involve transforming the camera coordinates to corresponding runway framed local coordinates that are output for at least one of guidance or control of the aircraft. For example, the perspective n-point system 216 or another aspect of the system 200 may be configured to transform camera coordinates to runway framing coordinates 236, which coordinates 236 are output by the perspective n-point system or another aspect of the system 200 for use by the guidance system 220, the control system 222, or another advanced system 110 of the aircraft 100.
In some example embodiments, the method 500 includes, as shown at block 536, determining a confidence interval for the pose estimate. In some example embodiments, the confidence interval may take the form of a multidimensional confidence matrix. Some example implementations of block 536 involve performing a perspective n-point estimation by at least further determining a confidence interval associated with the current pose estimate. For example, the perspective n-point system 216 or another aspect of the system 200 may be configured to determine a confidence interval 228 associated with the current pose estimate 218.
In some example embodiments, the method 500 includes, as shown at block 538, accessing additional pose estimates and confidence intervals. Some example implementations of block 538 involve accessing one or more additional pose estimates and associated confidence intervals. For example, the perspective n-point system 216 or another aspect of the system 200 may be configured to access one or more additional pose estimates 218A, and one or more associated confidence intervals 228A.
In some example embodiments, method 500 includes, as shown in block 540, performing sensor fusion. Some example implementations of block 540 involve performing sensor fusion of the current pose estimate and one or more additional pose estimates using the confidence intervals and associated confidence intervals to generate an updated current pose estimate. For example, the perspective n-point system 216 or another aspect of the system 200 may be configured to perform sensor fusion of the current pose estimate 218 and at least the additional pose estimate 218A and the confidence interval 228 and at least the associated confidence interval 228A to generate an updated pose estimate 218B.
In some example embodiments, method 500 includes, as shown at block 542, outputting the updated current pose estimate. Some example implementations of block 542 involve outputting the updated current attitude estimate for at least one of guidance or control of the aircraft. For example, the perspective n-point system 216 or another aspect of the system 200 may be configured to output an updated pose estimate 218B for use by a guidance system 220, a control system 222, or another advanced system 110 of the aircraft 100.
According to example embodiments of the subject disclosure, system 200 and its subsystems (including camera 202, pose estimation system 205, image processing system 206, first machine learning model 208, second machine learning model 210, corner detector 214, perspective n-point system 216, guidance system 220, control system 222, and confidence filter 234) may be implemented by various components. The means for implementing the system and its subsystems may comprise hardware, either alone or under the direction of one or more computer programs from a computer readable storage medium. In some examples, one or more devices may be configured to act as or otherwise implement the systems and subsystems thereof shown and described herein. In examples involving more than one device, the respective devices may be connected to or otherwise in communication with each other in many different ways, such as directly, or indirectly via a wired or wireless network, etc.
Fig. 6 illustrates an apparatus 600 according to some example embodiments of the subject disclosure. In general, the apparatus of the exemplary embodiments of the subject disclosure may comprise, include or be embodied in one or more fixed or portable electronic devices. Examples of suitable electronic devices include smart phones, tablet computers, notebook computers, desktop computers, workstation computers, server computers, and the like. The apparatus may include one or more of each of a number of components, such as, for example, processing circuitry 602 (e.g., a processor unit) connected to a memory 604 (e.g., a storage device).
The processing circuitry 602 may be comprised of one or more processors alone or in combination with one or more memories. The processing circuitry is generally any computer hardware capable of processing information such as, for example, data, computer programs, and/or other suitable electronic information. Processing circuitry consists of a collection of electronic circuits, some of which may be packaged as an integrated circuit or as a plurality of interconnected integrated circuits (integrated circuits are sometimes more commonly referred to as "chips"). The processing circuitry may be configured to execute a computer program, which may be stored on the processing circuitry or otherwise in memory 604 (of the same device or another device).
The processing circuitry 602 may be multiple processors, a multi-core processor, or some other type of processor, depending on the particular implementation. Further, the processing circuitry may be implemented using multiple heterogeneous processor systems, where a main processor and one or more secondary processors reside on a single chip. As another illustrative example, the processing circuitry may be a symmetric multi-processor system containing multiple processors of the same type. In yet another example, the processing circuitry may be embodied as or otherwise include one or more ASICs, FPGAs, and the like. Thus, while the processing circuitry is capable of executing a computer program to perform one or more functions, the processing circuitry of the various examples is also capable of performing one or more functions without the aid of a computer program. In either instance, the processing circuitry may be suitably programmed to perform functions or operations in accordance with example embodiments of the subject disclosure.
Memory 604 is generally any computer hardware capable of storing information, such as, for example, data, computer programs (e.g., computer readable program code 606), and/or other suitable information, temporarily and/or permanently. The memory may include volatile and/or nonvolatile memory, and may be fixed or removable. Examples of suitable memory include Random Access Memory (RAM), Read Only Memory (ROM), hard disk drive, flash memory, thumb drive, a removable computer diskette, an optical disk, magnetic tape, or some combination of the foregoing. Optical disks may include compact disk-read only memory (CD-ROM), compact disk-read/write (CD-R/W), DVD, and the like. In various instances, the memory may be referred to as a computer-readable storage medium. A computer-readable storage medium is a non-transitory device that can store information and is distinguished from a computer-readable transmission medium, such as an electronic transitory signal, that can carry information from one location to another. Computer-readable media as described herein may generally refer to computer-readable storage media or computer-readable transmission media.
In addition to the memory 604, the processing circuitry 602 may also be connected to one or more interfaces for displaying, transmitting, and/or receiving information. The interfaces may include a communication interface 608 (e.g., a communication unit) and/or one or more user interfaces. The communication interface may be configured to transmit and/or receive information, such as to and/or from other device(s), network(s), etc. The communication interface may be configured to transmit and/or receive information over a physical (wired) and/or wireless communication link. Examples of suitable communication interfaces include a Network Interface Controller (NIC), a wireless NIC (wnic), and so forth.
The user interface may include a display 610 and/or one or more user input interfaces 612 (e.g., input/output units). The display may be configured to present or otherwise display information to a user, suitable examples of the display including a Liquid Crystal Display (LCD), a light emitting diode display (LED), a Plasma Display Panel (PDP), and so forth. The user input interface may be wired or wireless and may be configured to receive information from a user into the apparatus, such as for processing, storage, and/or display. Suitable examples of user input interfaces include a microphone, an image or video capture device, a keyboard or keypad, a joystick, a touch-sensitive surface (separate from or integrated into a touch screen), a biometric sensor, and so forth. The user interface may further include one or more interfaces for communicating with peripheral devices, such as printers, scanners, etc.
As indicated above, program code instructions may be stored in memory and executed by processing circuitry programmed thereby to carry out the functions of the systems, subsystems, tools, and their respective elements described herein. It will be appreciated that any suitable program code instructions may be loaded onto a computer or other programmable apparatus from a computer-readable storage medium to produce a particular machine, such that the particular machine becomes a means for implementing the functions specified herein. These program code instructions may also be stored in a computer-readable storage medium that can direct a computer, processing circuitry, or other programmable apparatus to function in a particular manner, thereby making an article of manufacture or special machine. The instructions stored in the computer-readable storage medium may produce an article of manufacture, wherein the article of manufacture becomes a means for performing the functions described herein. The program code instructions may be retrieved from a computer-readable storage medium and loaded into a computer, processing circuitry, or other programmable apparatus to configure the computer, processing circuitry, or other programmable apparatus to perform operations to be performed on or by the computer, processing circuitry, or other programmable apparatus.
The retrieval, loading, and execution of program code instructions may be performed sequentially, such that one instruction is retrieved, loaded, and executed at a time. In some embodiments, the retrieving, loading, and/or executing may be performed in parallel such that multiple instructions are retrieved, loaded, and/or executed together. Execution of the program code instructions may produce a computer-implemented process such that the instructions executed by the computer, processing circuitry, or other programmable apparatus provide operations for implementing the functions described herein.
The instructions are executed by the processing circuitry or stored in the computer-readable storage medium in support of combinations of operations for performing the specified functions. In this manner, the apparatus 600 may include processing circuitry 602 and a computer-readable storage medium or memory 604 coupled to the processing circuitry, wherein the processing circuitry is configured to execute computer-readable program code 606 stored in the memory. It will also be understood that one or more functions or combinations of functions may be implemented by special purpose hardware-based computer systems and/or processing circuitry that performs the specified functions, or combinations of special purpose hardware and program code instructions.
Further, the present disclosure includes embodiments according to the following clauses:
clause 1: an apparatus for supporting aircraft access to runways at an airport, the apparatus comprising: a memory configured to store computer readable program code; and processing circuitry configured to access the memory and execute the computer readable program code to cause the apparatus to at least: receiving a sequence of images of the airport captured by a camera onboard the aircraft proximate to the runway; and for at least one image of the sequence of images, applying the image to a machine learning model trained to perform object detection and segmentation, wherein the runway or a runway marker on the runway on the airport is detected in the image, and wherein a mask is generated comprising a segment of pixels in the image assigned to an object class of the runway or the runway marker; applying the mask to a corner detector to detect points of interest on the mask and thereby detect points of interest on the runway or runway markings in the image; matching the points of interest on the runway or runway markings in the image with corresponding points on the runway or runway markings having known runway framed local coordinates; performing a perspective n-point estimation using the point of interest and the known runway framed local coordinates to determine a current attitude estimate of the camera and thereby determine a current attitude estimate of the aircraft relative to the runway or the runway marker; and outputting the current attitude estimate for use in at least one of guidance or control of the aircraft at a final approach.
Clause 2: the apparatus of clause 1, wherein the image is a multi-channel image, and the processing circuitry is configured to execute the computer readable program code to cause the apparatus to further convert the multi-channel image to a single-channel image.
Clause 3: the apparatus of any of clauses 1-2, wherein the processing circuitry is configured to execute the computer readable program code to cause the apparatus to further crop the image to reduce the field of view of the camera and enlarge only a portion of the image where the runway or runway sign is located before the image is applied to the machine learning model.
Clause 4: the apparatus of any of clauses 1-3, wherein the processing circuitry is configured to execute the computer readable program code to cause the apparatus to further at least: accessing an earlier image in the sequence of images and an earlier mask comprising respective pixel segments of the object category in the earlier image assigned to the runway or the runway marker; and identifying a portion of the earlier image that frames the earlier mask, and thereby identifying the portion of the airport at which the runway is located, and wherein causing the device to crop the image comprises causing the device to crop the image into a portion of the image that corresponds to the portion of the earlier image.
Clause 5: the apparatus of any of clauses 1-4, wherein the object detection and segmentation is at least one-class object detection and segmentation (an at least one-class object detection and segmentation), wherein only the runway or the runway marker is detected in the image, and the mask includes only the pixel segments in the image that are assigned to the object class of the runway or the runway marker.
Clause 6: the apparatus of any of clauses 1-5, wherein the object detection and segmentation is a multi-class object detection and segmentation, wherein the runway and the runway markings are detected in the image, and the mask comprises a segment of pixels in the image assigned to object classes of the runway and the runway markings, the point of interest comprising a point of interest on the runway and the runway markings.
Clause 7: the apparatus of any of clauses 1-6, wherein the processing circuitry is configured to execute the computer readable program code to cause the apparatus to further at least: determining an operating state of the aircraft; and selecting, based on the operating state, a first machine learning model or a second machine learning model as the machine learning model to which the image is applied, the first machine learning model being trained to perform at least single class object detection and segmentation, and the second machine learning model being trained to perform multi-class object detection and segmentation.
Clause 8: the apparatus of any of clauses 1-7, wherein the machine learning model is further configured to determine a confidence interval or multidimensional confidence matrix associated with detection of the runway or the runway marker, and the processing circuitry is configured to execute the computer readable program code to cause the apparatus to further apply the mask to a confidence filter configured to pass the mask to the corner detector only when the confidence interval is above a threshold confidence interval dynamically set based on an operating state of the aircraft.
Clause 9: the apparatus of any of clauses 1-8, wherein causing the apparatus to perform the perspective n-point estimation comprises causing the apparatus to determine the current pose estimate of the camera in camera coordinates and transform the camera coordinates into corresponding runway framed local coordinates, the local coordinates being output for at least one of guidance or control of the aircraft.
Clause 10: the apparatus of any of clauses 1-9, wherein causing the apparatus to perform the perspective n-point estimate further comprises causing the apparatus to determine a confidence interval associated with the current pose estimate, and the processing circuitry is configured to execute the computer readable program code to cause the apparatus to further at least: accessing one or more additional pose estimates and associated confidence intervals; performing sensor fusion of the current pose estimate and the one or more additional pose estimates using the confidence interval and the associated confidence interval to generate an updated current pose estimate; and outputting the updated current attitude estimate for at least one of guidance or control of the aircraft.
Clause 11: a method of supporting aircraft access to runways at an airport, the method comprising: receiving a sequence of images of the airport captured by a camera onboard the aircraft proximate to the runway; and for at least one image of the sequence of images, applying the image to a machine learning model trained to perform object detection and segmentation, wherein the runway or a runway marker on the runway on the airport is detected in the image, and wherein a mask is generated comprising a segment of pixels in the image assigned to an object class of the runway or the runway marker; applying the mask to a corner detector to detect points of interest on the mask and thereby detect points of interest on the runway or runway markings in the image; matching the points of interest on the runway or runway markings in the image with corresponding points on the runway or runway markings having known runway framed local coordinates; performing a perspective n-point estimation using the point of interest and the known runway framed local coordinates to determine a current attitude estimate of the camera and thereby determine a current attitude estimate of the aircraft relative to the runway or the runway marker; and outputting the current attitude estimate for use in at least one of guidance or control of the aircraft at a final approach.
Clause 12: the method of clause 11, wherein the image is a multi-channel image, and the method further comprises converting the multi-channel image to a single-channel image.
Clause 13: the method of any of clauses 11-12, further comprising cropping the image to reduce the field of view of the camera and enlarge only a portion of the image where the runway or runway marker is located prior to applying the image to the machine learning model.
Clause 14: the method of any of clauses 11-13, further comprising: accessing an earlier image in the sequence of images and an earlier mask comprising respective pixel segments of an object class in the earlier image assigned to the runway or the runway marker; and identifying a portion of the earlier image framing the earlier mask and thereby identifying the portion of the airport at which the runway is located, and wherein cropping the image comprises cropping the image to a portion of the image corresponding to the portion of the earlier image.
Clause 15: the method of any of clauses 11-14, wherein the object detection and segmentation is at least a single class of object detection and segmentation, wherein only the runway or the runway marker is detected in the image, and the mask includes only a segment of pixels in the image that are assigned to the object class of the runway or the runway marker.
Clause 16: the method of any of clauses 11-15, wherein the object detection and segmentation is a multi-class object detection and segmentation, wherein the runway and the runway markings are detected in the image, and the mask comprises pixel segments of object classes assigned to runways and runway markings in the image, the points of interest comprising points of interest on the runway and the runway markings.
Clause 17: the method of any of clauses 11-16, further comprising: determining an operating state of the aircraft; and selecting, based on the operating state, a first machine learning model or a second machine learning model as the machine learning model to which the image is applied, the first machine learning model being trained to perform at least single class object detection and segmentation, and the second machine learning model being trained to perform multi-class object detection and segmentation.
Clause 18: the method of any of clauses 11-17, wherein the machine learning model further determines a confidence interval or multidimensional confidence matrix associated with detection of the runway or the runway marker, and the method further comprises applying the mask to a confidence filter that passes the mask to the corner detector only when the confidence interval is above a threshold confidence interval that is dynamically set based on an operating state of the aircraft.
Clause 19: the method of any of clauses 11-18, wherein performing a perspective n-point estimate comprises determining the current pose estimate of the camera in camera coordinates and transforming the camera coordinates into corresponding runway framed local coordinates that are output for at least one of guidance or control of the aircraft.
Clause 20: the method of any of clauses 11-19, wherein performing the perspective n-point estimate further comprises determining a confidence interval associated with the current pose estimate, and the method further comprises: accessing one or more additional pose estimates and associated confidence intervals; performing sensor fusion of the current pose estimate and the one or more additional pose estimates using the confidence interval and the associated confidence interval to generate an updated current pose estimate; and outputting the updated current attitude estimate for at least one of guidance or control of the aircraft.
Many modifications and other embodiments of the disclosure set forth herein will come to mind to one skilled in the art to which this disclosure pertains having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the disclosure is not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, although the foregoing descriptions and the associated drawings describe example embodiments in the context of certain example combinations of elements and/or functions, it should be appreciated that different combinations of elements and/or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and/or functions than those explicitly described above are also contemplated as may be set forth in some of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims (10)

1. An apparatus (600), comprising:
a memory (604) configured to store computer readable program code (606); and
processing circuitry (602) configured to access the memory and execute the computer readable program code to cause the apparatus to at least:
receiving (502) a sequence of images (204) of an airport captured by a camera (202) onboard an aircraft approaching a runway; and for at least one image (204A) of the sequence of images,
applying (504) the image to a machine learning model (208, 210) trained to perform (504A) object detection and segmentation, wherein at least one of the runway on the airport or a runway marker (404) on the runway is detected in the image, and wherein a mask (212) is generated (504B) comprising pixel segments in the image assigned to object classes of the runway or the runway marker;
applying (506) the mask to a corner detector (214) to detect one or more points of interest (412) on the mask and thereby detect one or more points of interest (412) on the runway or runway marker in the image;
matching (508) the one or more points of interest on the runway or runway sign in the image with one or more corresponding points (406) on the airport or runway sign having known runway framing local coordinates; and
performing (510) a perspective n-point estimation using the one or more points of interest and the known runway framed local coordinates to determine a pose estimate (218) for the camera, and thereby determine a pose estimate (218) for the aircraft relative to the runway or the runway marker, wherein the pose estimate is used for at least one of guidance (220) or control (222) of the aircraft.
2. The apparatus (600) of claim 1, wherein the image (204A) is a multi-channel image, and the processing circuitry (602) is configured to execute the computer readable program code (606) to cause the apparatus to further convert (514) the multi-channel image to a single-channel image.
3. The apparatus (600) of claim 1 or 2, wherein the processing circuitry (602) is configured to execute the computer readable program code (606) to cause the apparatus to further crop (516) the image to reduce a field of view of the camera and enlarge only a portion of the image where the runway or runway sign is located before the image is applied to the machine learning model.
4. The apparatus (600) of claim 3, wherein the processing circuitry (602) is configured to execute the computer readable program code (606) to cause the apparatus to further at least:
accessing (518) an earlier image (204B) in the sequence of images (204) and an earlier mask (212A) comprising respective pixel segments of the object category in the earlier image assigned to the runway (402) or the runway marker (404); and
identifying (520) a portion of the earlier image framing the earlier mask and thereby identifying the portion of the airport where the runway is located, and
wherein causing the device to crop the image comprises causing the device to crop (522) the image to a portion of the image corresponding to the portion of the earlier image.
5. The device (600) according to claim 1 or 2, wherein the object detection and segmentation (504A) is at least a single type of object detection and segmentation, wherein only the runway (402) or the runway marker (404) is detected in the image (204A) and the mask (212) comprises only the pixel segments of the object class assigned to the runway or the runway marker in the image.
6. The apparatus (600) of claim 1 or 2, wherein the object detection and segmentation (504A) is a multi-class object detection and segmentation, wherein the runway (402) and the runway markings (404) are detected in the image (204A) and the mask (212) comprises pixel segments in the image assigned to object classes of the runway and the runway markings, the one or more points of interest (412) comprising a first point of interest on the runway and the runway markings.
7. The apparatus (600) of claim 1 or 2, wherein the processing circuitry (602) is configured to execute the computer-readable program code (606) to cause the apparatus to further at least:
determining (524) an operational state (230) of the aircraft (100); and
based on the operating state, selecting (526) either a first machine learning model (208) or a second machine learning model (210) as the machine learning model to which the image (204A) is applied, the first machine learning model being trained to perform at least single-class object detection and segmentation, and the second machine learning model being trained to perform multi-class object detection and segmentation.
8. The apparatus (600) of claim 1 or 2, wherein the machine learning model (208, 210) is further configured to determine (528) a confidence interval or multi-dimensional confidence matrix (232) associated with detection of the runway (402) or the runway marker (404), and the processing circuitry (602) is configured to execute the computer-readable program code (606) to cause the apparatus to further apply the mask (212) to a confidence filter (234), the confidence filter (234) being configured to pass the mask (212) to the corner detector (214) only when the confidence interval is above a threshold confidence interval dynamically set based on an operating state (230) of the aircraft (100).
9. The apparatus (600) of claim 1 or 2, wherein causing the apparatus to perform (510) the perspective n-point estimation comprises causing the apparatus to determine (532) the pose estimate (218) of the camera (202) in camera coordinates and to transform (534) the camera coordinates to corresponding runway framed local coordinates (236), the local coordinates (236) being output (512) for at least one of guidance (220) or control (222) of the aircraft.
10. A method (500), comprising:
receiving (502), by a system comprising a processor, a sequence of images (204) of an airport captured by a camera (202) onboard an aircraft proximate to a runway on the airport; and for at least one image (204A) of the sequence of images,
applying (504) the image to a machine learning model (208, 210) trained to perform (504A) object detection and segmentation, wherein the runway or a runway marker (404) on the runway on the airport is detected in the image, and wherein a mask (212) is generated (504B), the mask (212) comprising pixel segments in the image assigned to object classes of the runway or the runway marker;
applying the mask (506) to a corner detector (214) to detect one or more points of interest (412) on the mask and thereby detect one or more points of interest (412) on the runway or runway marker in the image;
matching (508) the one or more points of interest on the airport or runway sign in the image with corresponding points (406) on the runway or runway sign having known runway framed local coordinates;
determining an attitude estimate (218) of the aircraft relative to the runway or the runway marker based on the one or more points of interest and the known runway framed local coordinates; and
outputting (512) the attitude estimate for use in at least one of guidance (220) or control (222) of the aircraft at a final approach.
CN202110917124.6A 2020-08-28 2021-08-11 Perception-based autonomous landing of aircraft Pending CN114202706A (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202063071777P 2020-08-28 2020-08-28
US63/071,777 2020-08-28

Publications (1)

Publication Number Publication Date
CN114202706A true CN114202706A (en) 2022-03-18

Family

ID=80645910

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110917124.6A Pending CN114202706A (en) 2020-08-28 2021-08-11 Perception-based autonomous landing of aircraft

Country Status (1)

Country Link
CN (1) CN114202706A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114937007A (en) * 2022-04-29 2022-08-23 西北工业大学 Unmanned aerial vehicle visual landing runway initial line detection method based on corner features

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114937007A (en) * 2022-04-29 2022-08-23 西北工业大学 Unmanned aerial vehicle visual landing runway initial line detection method based on corner features

Similar Documents

Publication Publication Date Title
CN109324337B (en) Unmanned aerial vehicle route generation and positioning method and device and unmanned aerial vehicle
CN111326023B (en) Unmanned aerial vehicle route early warning method, device, equipment and storage medium
CN105512646B (en) A kind of data processing method, device and terminal
CN111352434B (en) Device and method for supporting aircraft approaching airport runway of airport
EP2618322B1 (en) System and method for detecting and displaying airport approach lights
Coutard et al. Visual detection and 3D model-based tracking for landing on an aircraft carrier
AU2019203567B2 (en) Geo-registering an aerial image by an object detection model using machine learning
Anitha et al. Vision based autonomous landing of an unmanned aerial vehicle
US11887329B2 (en) Moving body guidance apparatus, moving body guidance method, and computer-readable recording medium
CN114415736B (en) Multi-stage visual accurate landing method and device for unmanned aerial vehicle
EP3989034B1 (en) Automatic safe-landing-site selection for unmanned aerial systems
Oszust et al. A vision-based method for supporting autonomous aircraft landing
CN108225273A (en) A kind of real-time runway detection method based on sensor priori
Lee et al. Study on image-based ship detection for AR navigation
CN114202706A (en) Perception-based autonomous landing of aircraft
Deng et al. A binocular vision-based measuring system for UAVs autonomous aerial refueling
Koizumi et al. Development of attitude sensor using deep learning
KR102135725B1 (en) Automatic landing control device and operation method of the same
US11783575B2 (en) Perception-based autonomous landing for aircraft
US20220198703A1 (en) Determining a current pose estimate of an aircraft relative to a runway to support the aircraft on approach
Kakillioglu et al. Autonomous altitude measurement and landing area detection for indoor uav applications
Liu et al. Runway detection during approach and landing based on image fusion
Khan et al. Near Helipad Auto Guidance System for Landing and Take-off of Helicopter
More et al. Landmark Detection for Auto Landing of Quadcopter Using YOLOv5
Lei et al. Adaptive Visual-Inertial Navigation for Fixed-Wing Aircraft Landing in GPS-Denied Condition

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination