WO2023150888A1

WO2023150888A1 - System and method for firefighting and locating hotspots of a wildfire

Info

Publication number: WO2023150888A1
Application number: PCT/CA2023/050182
Authority: WO
Inventors: Robert Atwood; Sean BURT; Richard Sullivan
Original assignee: Hummingbird Drones Inc.
Priority date: 2022-02-11
Filing date: 2023-02-13
Publication date: 2023-08-17

Abstract

A computer-implemented method of locating a hotspot of a wildfire using aerial imagery, including determining a pixel location of the hotspot in an aerial image of a set of aerial images suitable for stereoscopic imagery by processing the set of aerial images in accordance with a pre-trained machine learning model, generating a three-dimensional model in response to the set of aerial images, and mapping the pixel location of the hotspot in the aerial image to a corresponding pixel location in the three-dimensional model to determine the location of the hotspot. A system for locating the hotspot, comprising a processor, and memory coupled thereto and storing instructions to cause the processor to execute the method. Computer-readable memory have stored thereon such instructions.

Description

SYSTEM AND METHOD FOR FIREFIGHTING AND LOCATING HOTSPOTS OF A WILDFIRE

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No. 63/309,499, filed on 11 February 2022.

TECHNICAL FIELD

The disclosure relates generally to automated tools for fighting of wildfires, and more particularly to systems and methods for locating hotspots of wildfires.

BACKGROUND

From a firefighting perspective, it is advantageous for firefighters, fire managers, and decision makers to have accurate information about wildfire(s) they are managing. Aerial imagery is used to survey wildfires because the sizes of areas vulnerable to wildfires are frequently very large. For example, in some existing approaches, an airplane or helicopter may fly over a vulnerable area to allow a professional to survey large areas at once, by eye. Once detected, fire crews may be informed of a general area which is affected. Ground and/or air crews may be dispatched to locate and combat the local fires. Present approaches may be slow, expensive, and/or require relatively high levels of human intervention to effectively characterize wildfires for firefighting.

U.S. Patent No. 9,878,804 B2 discloses a system for generating thermographs of an area using aerial thermal images generated or captured by athermal image sensor mounted on to an unmanned aerial vehicle. The images are contrasted, georeferenced, orthorectified, and/or stitched together to form a complete thermal image. The thermal images may be corrected using surface-based temperature measurements. An output module displays the individual or stitched thermal images to a user for evaluation.

United States Patent Publication No. US 2021/0110136 Al discloses using a convolutional neural network (CNN) to detect wildfires from aerial images generated by satellites and drones and uses this information to navigate a drone, e.g. to obtain additional drone images of the wildfire, via a reinforcement learning system. The CNN is trained using image libraries that have images with metadata indicating the location, in the respective image, of any wildfire. The system of United States Patent Publication No. US 2021/0110136 Al requires training data that accurately identifies locations of wildfires in images.

U.S. Patent No. US 9,047,515 B2 discloses a system for detecting wildfires that is based on analyzing digital images from a video camera to detect gray colored regions, and determining whether a detected gray colored region is smooth. A support vector machine (SVM) is used to determine whether the region shows smoke from a wildfire.

United States Patent Publication No. 2021/0192175 Al discloses a wildfire early detection system that uses aerial video clips of areas under surveillance by UAVs and tethered aerostats in a computer vision algorithm trained to recognize plumes of smoke that is used as a sign for fire. The computer vision algorithm is trained with an “Aerial Training Dataset”, generated using digitally simulated plumes of smoke for video clips, and uses a generative adversarial network (GAN)-type discriminator network to classify images.

Improvement over these existing approaches is desired, including by allowing better scaling (with respect to time and cost) with the size and number of wildfires, reducing human effort and/or intervention, increasing the efficiency of fighting wildfires, and/or improving the speed and accuracy of evaluating wildfires for firefighting.

Such improvements are especially important due to the increasing prevalence of wildfires around the world and the rapid growth of areas vulnerable to such wildfires.

SUMMARY

Wildfires may be effectively and efficiently combatted by mitigating hotspots — regions or locations of relatively higher temperatures that can be discerned on infrared aerial images, e.g. in some cases, such locations may be indicative of the presence of particularly intense fire (flames).

Locating hotspots in wildfires using aerial images is a challenging endeavour that would be timeconsuming and expensive. Hotspots could be manually tagged by a human operator, possibly with the help of detection software as described previously, and then communicated to ground teams to determine hotspot locations. For example, hundreds or thousands of aerial images may need to be captured by multiple aerial teams, including aerial vehicles and staff, and then be separately analyzed to survey the condition of a vulnerable area. While flying faster and at higher altitudes may allow faster sweeping over large areas that are vulnerable, better and more expensive cameras (higher resolution and capture rates) are then required to compensate for faster sweeps over an area and for lower pixel density per geographic unit. Flying faster and at higher altitudes may also increase costs associated with the aerial platform, e.g. passenger aircraft may be needed instead of relatively inexpensive unmanned aerial vehicles. In some cases, overlapping aerial images are combined to improve resolution. However, this tends to wash out any signature of fires, especially smaller ones, because of the dynamic nature of fires and heat, as well as the tendency of aerial images of large areas at practical resolutions (such as those considered here) to represent heat signatures by relatively small amounts of pixels. Existing methods may be considerably computationally expensive and inefficient, which can tax the resources of emergency response teams and cause harmful delays in locating hotspots. In surveying vulnerable areas for firefighting, speed of execution and scope of coverage are critical for saving life, limb, property including critical infrastructure, and maintaining safe delivery of essential goods and services. As such, with ever more frequent wildfires around the world, improvements in systems and methods for locating hotspots are much needed.

It is found that the location of a hotspot may be obtained in a timely, accurate, cost-effective, and computationally efficient manner from relatively lower resolution raw aerial images by configuring a processor to generate a mapping of a pixel location of a hotspot, (both) identified by processing digital raw aerial images in accordance with a pre-trained machine learning model for object detection, onto an a three-dimensional model (e.g. via an orthorectified image) generated by the processor using some subset (strict or not) of the digital raw aerial images.

A three-dimensional model, such as a digital surface model, and associated tie-points may allow geographic referencing.

Orthorectification may allow accurate alignment and georeferencing of raw aerial images even with relatively low resolution and without detailed surface models, as long as there are a sufficient number of (geographically) overlapping images to allow stereophotography. In some cases, ground control point(s) with known locations depicted in images or other types of tie-points may facilitate orthorectification. Orthorectification may generally include generating a three- dimensional model of the surface being imaged, and/or identifying tie-points.

Machine learning models, such as (fast and/or faster) region-based convolutional neural networks (R-CNN), allow for accurate detection and location of objects within images. In some cases, with an appropriately labeled training dataset, such models may be trained to detect hotspots with low resolution images, even if such images may not directly yield accurate geolocation. For example, in some cases, hotspots depicted as small as 1 pixel may be detected by processing raw aerial images via machine learning models. Thermal infrared aerial images may also contain “false positives”, i.e. non-fire or hot regions that appear as hotspots in infrared imagery due to reflected infrared radiation rather than true source infrared radiation. For examples, infrared radiation may be reflected from water and/or rocks. With sufficiently rich training datasets having false positives accurately labelled, the machine learning models may be trained to detect and eliminate such false positives from consideration as hotspots.

Processing images to improve their quality may obscure faint signatures of hotspots that would otherwise be discernible in a raw aerial image. For example, each image may have specific errors and variances associated with hardware and software components, including a speed of an aerial vehicle, Global Navigation Satellite System (GNSS) sampling and timing (e.g. of Global Positioning System-based measurements), inertial measurement units (IMU), shutter speeds, tilt angle of camera, pixel pitch of camera sensors, altitudes (e.g. by up to 10 m), and errors due to vibrations. It is found that combining images, e.g. to achieve stereoscopic vision or superresolution, or to average error in overlapping regions, may indiscriminately compound individual errors and eliminate signatures of hotspots that are critical for detecting and rapidly locating regions of high heat. Additionally, it is found that combining images may eliminate or weaken signatures of potential false positives — a certain proportion of which may be actual hotspots — and may thereby reduce the reliability of machine learning models by preventing verification of potential false positives. For example, in many cases, a higher number of potential false positives may be tolerated to ensure a lower number of undetected hotspots even though this may reduce a measure of overall accuracy of the machine learning model.

Performing object detection on relatively low resolution two-dimensional (grayscale or single channel, at least in some cases) raw aerial images may be computationally more efficient and allows faster surveying.

Performing object detection and orthorectification using raw aerial images may allow at least some parallel processing, reduce computational overhead, and may improve processor performance, e.g. lower computational times.

In some cases, separating challenges associated with accurate geolocation and challenges associated with accurate (pixel location) detection of hotspots, may allow the use of lower cost cameras deployed at higher altitudes, e.g. altitudes above 2000 ft, speeds above 55 km/h, and resolutions below 640 MP may be achieved. For example, such cameras may be lighter and may be deployed via cheaper platforms.

In some cases, computer-implemented methods disclosed herein may allow timely, efficient, automatic, and rapid detection and location of hotspots in wildfires, thereby reducing time and effort. For example, in some cases, aerial imagery resolutions may be such that hotspots the size of an adult human fist may be able to be captured in aerial imagery. Manual surveying several square kilometers of aerial footage by eye for such small hotspots, and manual geolocation of such small hotspots, is not feasible to be achieved in a timely manner.

For example, the system or a plurality of systems may be fed aerial images from an unmanned aerial vehicle and may generate wildfire location(s) that are then transmitted to a network- connected server to update a user interface to show up to date hotspot and wildfire information.

In some cases, automated firefighting may be achieved. This is particularly important in light of a shortage of firefighting personnel, including pilots and water inundation equipment operators, and the fatigue and the risk to life and limb sustained by such personnel. For example, autonomously generated hotspot locations may be transmitted via a signal to a firefighting platform, e.g. an autonomous aerial vehicle or swarm of such vehicles, for flight planning and water inundation of automatically generated hotspot locations.

In one aspect, the disclosure describes a computer-implemented method of locating a hotspot of a wildfire using aerial imagery, the method comprising: receiving data indicative of a set of aerial images suitable for stereoscopic imagery; determining a pixel location of the hotspot in an aerial image of the set of aerial images by processing the set of aerial images in accordance with a machine learning model, the machine learning model having been pre-trained using training data, the training data including data indicative of images and known hotspots in the images; generating data indicative of a three-dimensional model using the set of aerial images; and mapping the pixel location of the hotspot in the aerial image to a corresponding location in the three-dimensional model to determine a location of the hotspot.

In another aspect, the disclosure describes a system for locating a hotspot of a wildfire using aerial imagery, the system comprising: a processor; computer-readable memory coupled to the processor and storing processor-executable instructions that, when executed, configure the processor to: receive data indicative of a set of aerial images suitable for stereoscopic imagery; determine a pixel location of the hotspot in an aerial image of the set of aerial images by processing the set of aerial images in accordance with a machine learning model, the machine learning model having been pre-trained using training data, the training data including data indicative of images and known hotspots in the images; generate data indicative of a three-dimensional model using the set of aerial images; and map the pixel location of the hotspot in the aerial image to a corresponding location in the three-dimensional model to determine a location of the hotspot.

In yet another aspect, the disclosure describes anon-transitory computer-readable medium having stored thereon machine interpretable instructions which, when executed by a processor, cause the processor to perform a computer-implemented method of locating a hotspot of a wildfire using aerial imagery, the method comprising: receiving data indicative of a set of aerial images suitable for stereoscopic imagery; determining a pixel location of the hotspot in an aerial image of the set of aerial images by processing the set of aerial images in accordance with a machine learning model, the machine learning model having been pre-trained using training data, the training data including data indicative of images and known hotspots in the images; generating data indicative of a three-dimensional model using the set of aerial images; and mapping the pixel location of the hotspot in the aerial image to a corresponding pixel location in the orthorectified image to determine a location of the hotspot.

In a further aspect, the disclosure describes a computer-implemented method of locating a hotspot of a wildfire using aerial imagery, the method comprising: receiving data indicative of a set of aerial images suitable for stereoscopic imagery; determining a pixel location of the hotspot in an aerial image of the set of aerial images by processing the set of aerial images in accordance with a machine learning model, the machine learning model having been pre-trained using training data, the training data including data indicative of images and known hotspots in the images; generating an orthorectified image in response to the set of aerial images; and mapping the pixel location of the hotspot in the aerial image to a corresponding pixel location in the orthorectified image to determine a location of the hotspot.

In another further aspect, the disclosure describes a system for locating a hotspot of a wildfire using aerial imagery, the system comprising: a processor; computer-readable memory coupled to the processor and storing processor-executable instruction that, when executed, configure the processor to: receive data indicative of a set of aerial images suitable for stereoscopic imagery; determine a pixel location of the hotspot in an aerial image of the set of aerial images by processing the set of aerial images in accordance with a machine learning model, the machine learning model having been pre-trained using training data, the training data including data indicative of images and known hotspots in the images; generate an orthorectified image in response to the set of aerial images; and map the pixel location of the hotspot in the aerial image to a corresponding pixel location in the orthorectified image to determine a location of the hotspot.

In yet another further aspect, the disclosure describes a non-transitory computer-readable medium having stored thereon machine interpretable instructions which, when executed by a processor, cause to the processor to perform a computer-implemented method of locating a hotspot of a wildfire using aerial imagery, the method comprising: receiving data indicative of a set of aerial images suitable for stereoscopic imagery; determining a pixel location of the hotspot in an aerial image of the set of aerial images by processing the set of aerial images in accordance with a machine learning model, the machine learning model having been pre-trained using training data, the training data including data indicative of images and known hotspots in the images; generating an orthorectified image in response to the set of aerial images; and mapping the pixel location of the hotspot in the aerial image to a corresponding pixel location in the orthorectified image to determine a location of the hotspot.

Embodiments can include combinations of the above features.

Further details of these and other aspects of the subject matter of this application will be apparent from the detailed description included below and the drawings.

DESCRIPTION OF THE DRAWINGS

Reference is now made to the accompanying drawings, in which:

FIG. 1 is a schematic of a system for locating hotspots of a wildfire using aerial imagery, in accordance with an embodiment;

FIG. 2 is a schematic block diagram of a system for locating hotspots of a wildfire using aerial imagery, in accordance with an embodiment;

FIG. 3 shows an aerial image showing hotspots, and control points, in accordance with an embodiment;

FIG. 4 shows a set of aerial images, in accordance with an embodiment; FIG. 5 is a schematic of a workflow for generating orthorectified images and orthophotomosaics (or orthomosaics), in accordance with an embodiment;

FIG. 6 is a detailed schematic of a workflow for generating orthorectified images and orthophotomosaics (or orthomosaics), in accordance with an embodiment;

FIG. 7 is a schematic of an object detection module, in accordance with an embodiment;

FIG. 8 is a schematic of a system for locating hotspots of a wildfire using aerial imagery, in accordance with another embodiment;

FIG. 9 is a flow chart of an exemplary computer-implemented method of locating a hotspots of a wildfire using aerial imagery, in accordance with an embodiment;

FIG. 10 illustrates a block diagram of a computing device, in accordance with an embodiment of the present application;

FIG. 11 is a schematic workflow for an orthorectification module, in accordance with an embodiment; and

FIG. 12 is a flow chart of an exemplary computer-implemented method of locating a hotspot of a wildfire using aerial imagery, in accordance with an embodiment.

DETAILED DESCRIPTION

The following disclosure relates to locating hotspots in wildfires using aerial imagery. In some embodiments, systems and methods disclosed herein can facilitate more efficient, faster, and scalable approaches to locate hotspots compared to existing systems and methods.

Coordinate locations (geospatial positioning) of hotspots in active wildfires in an area may be determined using aerial images of the area, e.g. captured using a camera-equipped drone flying overhead. It is understood that aerial images may refer to images of detected (electromagnetic) radiation, such as optical images based on received light (visible spectrum) or images based on the non-visible spectrum. Nevertheless, it is found particularly advantageous for the methods disclosed herein to utilize infrared, and more particularly, thermal infrared imagery. In such embodiments, the aerial images may be infrared (IR) images (or thermal infrared images), which may show locations with relatively higher temperature (“hotspots”). Such images may potentially show other features besides regions of relatively higher temperature. It is found that such aerial images are particularly amenable for machine learning approaches discussed herein. In some cases, unlike stationary and/or relatively spatially well-defined objects, hotspots of wildfires may be difficult to detect due to rapid movement and variation, e.g. two images of a hotspot taken seconds apart may be significantly different. Such differences may be magnified when the images are taken at slightly different positions and/or angles, and under variable tree cover and other structures, since a view of the hotspot may then be easily obstructed (or not).

A direct “single-image” process of locating hotspots may be used for low altitude aerial photography. Each aerial image may be orthorectified to obtain a corresponding orthophoto corrected for topographic relief, lens distortion, and camera tilt, so that coordinates of hotspots may be identified.

For wildfires, aerial images are preferably taken from relatively high altitudes because of the relatively large size of areas that need to be surveyed. Errors may be magnified at such high altitudes, leading to significant error in estimated locations of the hotspots. Sources of error include limited resolution of topographic maps, uncertainty in camera tilt, and uncertainty in the speed and location of the drone. Errors may increase away from the center of the aerial image or other reference position thereof, where geographic location coordinates may be determined most accurately.

For improved accuracy, an ensemble of aerial photographs may be used to generate an orthorectified image, e.g. an orthophotomosaic, that is locally (at the spatial scale of a single aerial image) more accurate than a single aerial image. The coordinates determined using such an orthorectified image may be sufficiently accurate to allow the use of high-altitude aerial imagery for determining accurate coordinates of chosen locations. In some embodiments, a tolerance of less than 3 metres may be required for accurate geolocation of fires and for achieving efficient fire fighting. In some cases, the process of generating an orthophotomosaic may obscure a hotspot. For example, a hotspot may appear at a particular location in only one of several aerial images and thus may be “averaged out of the ensemble” after processing, and/or lead to an unacceptably low signal-to-noise ratio. A hotspot may appear in only one aerial image of an ensemble due to a variety of reasons, including due to the movement and dynamics of the fire itself and/or due to the physical blocking of the line-of-sight (LoS) of the IR camera, such as by tree cover, that reduces the exposure of ground-level high temperature regions from certain image capture angles.

It is proposed that hotspots may be detected directly from individual (raw) aerial images, e.g. using a supervised machine learning model for detecting and/or locating hotspots. In some embodiments, these aerial images may be mapped on to orthorectified image(s) to obtain coordinates of the hotspots (georeferencing to ground coordinates). In some embodiments, transformations associated with orthorectified image(s) may be used to transform pixel locations of hotspots to obtain coordinates of the hotspots without directly mapping on to orthorectified image(s).

The coordinates so determined may be used to cause a tangible effect, including one or more of: sending notifications to a firefighting application on user terminal, updating a firefighting map layer in the application, alerting firefighting units, e.g. drone or helicopter pilot to release fire suppressant, automatically releasing fire suppressant from a transportation/delivery mechanism, and/or creating digital mapping for drone swarm water delivery.

Aspects of various embodiments are described in relation to the figures.

FIG. 1 is a schematic of a system 100 for locating hotspots of wildfires using aerial imagery, in accordance with an embodiment.

An aerial vehicle 102 equipped with an aerial imaging system 104 may fly over an area 106 that is vulnerable to fire. For example, an area vulnerable to fire may be in or around a known fire perimeter of a wildfire. In various embodiments, the aerial vehicle 102 may fly at altitudes between 50-500 m or greater or, in some particular embodiments, between 400-1600 ft.

In various embodiments, the aerial vehicle 102 may be a fixed or rotary wing aircraft. For example, the aerial vehicle 102 may be a helicopter, or propeller aircraft. In various embodiments, the aerial vehicle 102 may be operated by a pilot situated inside the aerial vehicle 102. In some embodiments, the aerial vehicle 102 may be remotely operated, e.g. without an operator positioned inside the aerial vehicle 102.

The aerial imaging system 104 may be an infrared imaging system configured to generate two- dimensional maps of infrared radiation intensity of objects defined in a field of view 108 of the aerial imaging system 104. In some embodiments, the display field of view may be about 40°. For example, an aerial image captured by the aerial imaging system 104 may represent a (projected) area between about 100-100,000 m² or more.

In some embodiments, aerial images may have a resolution between 1260 x 1260 and 336 x 256 (e.g. 640 x 512) and may cover a region between 300-500 square metres, e.g. for a geographic resolution between about 10²-10³ pixels/m² or about 1-10 pixels/dm² In some cases, the geographic resolution may be about 10 pixels/m² or lower (between 1 and 10 pixels/m²). For example, at such resolutions, a hotspot the size of an adult human fist may be able to be captured, in principle, in aerial imagery.

The aerial imaging system 104 may include a sensor, e.g. a CMOS sensor, that is oriented directly downwards from the aerial vehicle 102. In various embodiments, the aerial imaging system 104 may sense thermal infrared radiation, and particularly in the spectral band with wavelengths between about 7.5-13.5 pm (long-wave infrared or LWIR). Advantageously, in some embodiments, the aerial imaging system 104 may be flown, and aerial images captured, at night.

In some embodiments, the aerial imaging system 104 may be a thermal infrared camera with a sensitivity (Noise Equivalent Temperature Difference or NETD) of <50 mK with a f/1.0 lens aperture (relative aperture of 1.0) and may allow sensing temperature variations between -40 °C to 150 °C or up to 550 °C (with a higher gain amplifier, e.g. if noise amplification is not an important factor), or between -25 °C to 100 °C or up to 135 °C. In some embodiments, the aerial imaging system 104 may be cooled to achieve greater sensitivities and improve reliability in the low temperature range. Advantageously, in some embodiments disclosed herein, processing raw aerial images through object detection machine learning models may facilitate more efficient and reliable sensing of temperature anomalies closer to a device sensitivity or NETD, especially at elevated temperatures.

In some embodiments, the aerial imaging system 104 may capture images and/or video at a pixel resolution of 640 x 512 or 336 x 256. In various embodiments, the aerial imaging system 104 may capture images at 30 frames per second or higher (full frame rates), or 9 frames per second. In various embodiments, the aerial imaging system 104 may include a camera having a pixel pitch between 12-17 pm.

In some embodiments, the aerial vehicle 102 may be an unmanned aerial platform, such as an aerostat or quadrotor drone controllable by a ground controller and/or configured for autonomous operation.

The aerial vehicle 102 may be in motion and the aerial imaging system 104 may be configured to capture images at a sufficiently high rate, in relation to the speed of travel of the aerial vehicle 102, such that it generates geographically overlapping images. In some embodiments, 70-90% or more of the areas of successive images may be overlapping. In some embodiments, less than 70-90% of the areas of successive images may be overlapping. For example, at an image capture rate of 10 Hz and an aerial vehicle speed configured for 80% overlap of photographs, the aerial imaging system 104 may capture five images of a particular location over a period of 0.5 seconds. In some embodiments, frequency of image capture may depend on the altitude.

The aerial vehicle 102 and/or aerial imaging system 104 may be in one-way or two-way communication, via a network 118, with a computing device 110, a server 112, a client device 114, and/or a firefighting platform 116, e.g. an aerial firefighting platform configured for water inundation.

In some embodiments, aerial images may be provided by the aerial imaging system 104 to the computing device 110, the server 112, the client device 114, and/or the firefighting platform 116 in “real-time” via the network 118 or may be provided in batches, e.g. after the conclusion of a flight by the aerial vehicle 102. For example, the aerial images may be provided prior to conclusion of a flight by the aerial vehicle 102 and/or before one or more other aerial images are captured or simultaneously therewith.

The network 118 may include a radio network, cellular network, satellite-based network, or direct one-to-one transmission.

A computing device 110 may include a processor 120, memory 122 with instructions thereon to cause the processor 120 to execute one or more methods, and a communication interface 124 for communicating with entities external to the computing device 110. The memory 122 may include transitory and non-transitory computer-readable memory.

In some embodiments, the computing device 110 may be mounted on to the aerial vehicle 102.

The client device 114 may include a personal computing device, messaging system, or other device configured to receive and possibly send communications from a user. In some embodiments, the client device 114 may have a firefighting application providing a user interface for users to receive hotspot location information and interact with a map showing the same. For example, the client device 114 may be a smartphone and/or a tablet device.

The system 100 may be configured to detect and locate a hotspot 126. For example, the system 100 may be configured to generate coordinates in a standard earth coordinate system of a location of the hotspot 126. In some embodiments, coordinates may be generated using a geographic coordinate system (GCS) and may specify latitude and longitude. In some embodiments, coordinates may be based on the World Geodetic Standard (WGS), e.g. WGS 84. The server 112 may be a network-connected server and may include communication interface(s), transitory and non-transitory memory, and one or more processors. The server 112 may store information relating to hotspot locations on a database and may be configured to provide access to users to locations of hotspots via the database. For example, the server 112 may be configured to collect information from a plurality of systems to aggregate information provided in different forms to provide a single place for a user to access up to date or real-time hotspot location information. In some embodiments, hotspot locations may be sent to the server 112 by the computing device 110 in “real-time” via the network 118. In some embodiments, hotspot locations may be batch processed.

In some embodiments, the firefighting platform 116 may include a water inundation system, e.g. a bambi or helicopter bucket, which may be triggered or actuated (automatically or after approval by a user) based on a signal sent by the computing device 110, server 112, and/or client device 114 to the firefighting platform 116 that is indicative of a location, size, and/or other characteristics of the hotspot 126. In some embodiments, such a signal may be used by the firefighting platform 116 or a pilot thereof for flight planning. For example, in some embodiments, a fully or partially autonomous aerial vehicle may generate a route to the hotspot location based on the signal to inundate the hotspot location with water. For example, this aerial vehicle may be unmanned. In some embodiments, the firefighting platform 116 may include swarms of drones working cooperatively to extinguish one or more fires associated with hotspot locations received by them via a signal from the computing device 110, server 112, and/or client device 114.

In some embodiments, the hotspot 126 may not be visible to the naked eye but may be a higher temperature area that would appear as a possible location for a fire or a hotter region thereof. For example, in some cases, the fire may increase the temperature at the top of the tree cover above the fire but may not penetrate it sufficiently to be clearly visible in an aerial image. Elevated temperatures may be depicted and/or visible in infrared images of the tree cover. For example, in other cases, fire may be present underground and/or only smoke associated with the fire may be visible by naked eye. In general, a hotspot 126 as described previously may be indicative of the presence of a fire or a hotter region thereof.

FIG. 2 is a schematic block diagram of a system 100 for locating hotspots of wildfires using aerial imagery, in accordance with an embodiment.

The system 100 may be implemented using the computing device 110. The aerial imaging system 104 may generate data 134 indicative of a set of aerial images suitable for stereoscopic imagery. The aerial images may be at least partially overlapping to provide multiangle (multi-location) views of any single object such that the images may be suitable for combining together to form a 3D image, i.e. including height data.

In some embodiments, the data may be received, e.g. separately, by an object detection module 128 and an orthorectification module 130. In some embodiments, the data 134 may be received substantially in parallel to allow faster processing and to leverage multithreading.

The object detection module 128 may be configured to determine pixel locations of hotspots in the set of aerial images represented in the data 134 by processing the set of aerial images (i.e. the data 134) in accordance with a machine learning model. The output of the object detection module 128 may include data indicative of the pixel locations along with associated aerial images where the pixel locations are to be found.

The machine learning model may be pre-trained using training data that includes data indicative of images and known hotspots in those images. For example, such data may have been generated by manually tagging images. As a further example, a masked machine learning model is trained using a training dataset having a masked layer. A mask fde is generated by pre-processing input images to produce a binary fde that includes hotspot boundaries outlining each hotspot. In some embodiments, a software tool is employed to generate the mask fde. In various embodiments, it is found advantageous to use masked R-CNN.

In various embodiments, the machine learning model may be trained in two or more separate stages. In some embodiments, in a first stage, the machine learning model may be trained using a dataset that includes a small number of hotspots for easy-data training, and in a second stage, may be trained using a dataset that includes a large number of hotspots for difficult-data training. For example, in the first stage, the machine learning model may be trained using a labelled image dataset that includes one or more aerial images that are annotated to identify and locate a small number of hotspots and/or fire. In various embodiments, labelled objects (such as hotspots) in images may be located or localized therein using bounding boxes. For example, in the second stage, the machine learning model may be trained using training data that includes data indicative of images showing a large (or large) number of known hotspots in those images. As a variation of multi-stage training, the first stage may involve training using a dataset that is not specific to aerial images, and the second stage may involve training using a dataset that includes labelled aerial images, e.g. aerial images from previous flights that are processed to identify and locate hotspots therein (if any). For example, in the first stage, the machine learning model may be trained using a labelled image dataset, such as Common Object in Context (COCO), that includes one or more non-aerial images that are annotated to identify and locate objects therein other than hotspots and/or fire. In various embodiments, labelled objects in images may be located or localized therein using bounding boxes. For example, in the second stage, the machine learning model may be trained using training data that includes data indicative of images and known hotspots in those images. In various embodiments, the machine learning models may be trained to detect water bodies (or other types of objects, such as animals, cars, or houses, that could be confused with hotspots), distinguish water bodies (or such other types of objects) from hot spots, and/or classify water bodies (or such other types of objects) separately from hotspots.

In some embodiments, the machine learning model is a variant of a Region-based Convolutional Neural Networks (R-CNN), such as faster R-CNN or faster R-CNN. In some embodiments, the machine learning model is a YOLO model (You Only Look Once model).

Pixel locations may include single pixel locations, e.g. representing a single-pixel hotspot or a centre of a hotspot, or may include a plurality of pixel locations representing where a detected hotspot is located. In some embodiments, pixel locations may be defined using a bounding box and a geometry thereof.

The orthorectification module 130 may receive the data 134 indicative of the set of aerial images to generate an orthorectified image in response to the set of aerial images.

As referred to herein “orthorectified image” refers to an image that has been synthetically generated as an image of a surface in an orthogonal parallel projection. An orthorectified image may have a constant scale such that objects may be represented accurately in relation to their ground position, thereby enabling accurate measurement of distances, angles, and areas. Orthorectification of an image may address a variety of artifacts that affect remotely sensed imagery. These include perspective distortion and field of view (FoV) effects, lens distortion, earth curvature, relief displacement, radial displacement, and scanning errors.

In various embodiments, “orthorectified image” may refer to a single orthorectified image or an orthophotomosaic formed by stitching together a plurality of orthorectified images. Such a stitching process may involve other steps, e.g. finding optimal seams for stitching and/or matching the intensity of adjacent orthorectified images. As described later, the orthorectification module 130 may generate a three-dimensional (3D) representation of the ground (a digital surface model). The orthorectified image may be a 3D image and may be generated based on the digital surface model, camera calibration parameters, and adjustment points such as GPS coordinates of the center of an image, ground control points, and image tie-points generated in overlap regions between aerial images.

In various embodiments, orthorectification may address radiometric distortion as well as geometric distortion.

In some embodiments, the orthorectification module 130 may receive data indicative of images identified as having hotspots from the object detection module 128.

In some embodiments, the orthorectification module 130, in response to receiving data indicatives of the images, may generate orthophotos localized around the detected hotspots, e.g. by forming an orthophoto using only aerial images overlapping with aerial images having hotspots. Doing so may improve computational efficiency and increase performance.

As output, the orthorectification module 130 may generate, in various embodiments, orthophoto(s) (orthorectified image(s) corresponding to individual aerial images), an orthophotomosaic that combines such orthophotos together, a three-dimensional model (such as a digital surface model), a three-dimensional model with image tie-points, and/or transformation(s), e.g. transformations associated with the orthorecified image(s).

In some embodiments, the output of orthorectification module 130 may include data indicative of a mapping between aerial images and the corresponding orthorectified images, or a corresponding portion of the orthophotomosaic. For example, the orthorectification module 130 may generate a pair of complementary data structures, wherein an indexed location in the first data structure is matched to an indexed location in the second data structure. A location [i,j] in a first data structure may be matched to a location [k,l] in a second data structure. The indices of the first data structure may be matched one-to-one to the indices of the second data structure. The first data structure may be indicative of an aerial image and the second data structure may be indicative of an orthorectified image.

In some embodiments, the orthorectification module 130 may generate data indicative of orthorectified image(s) and/or orthophotomosaic(s). In some embodiments, the orthorectification module 130 may generate data indicative of a transformation associated with an orthorectified image. For example, applying such a transformation onto an appropriate aerial image may cause at least partial orthorectification of the aerial image. In some cases, the orthorectification module 130 may not generate data indicative of the orthorectified image(s) but only data indicative of the transformation.

In some embodiments, the orthorectification module 130 may generate data indicative of a three- dimensional model of the imaged surface, i.e. a digital surface model. In some embodiments, such data may be provided with data indicative of tie-points between aerial images. Tie-points may allow correlation or association of one image to another. For example, two tie-points on an image may correspond to two tie-points on another image to substantially fix the position and orientation between the two images. If only a single tie-point on a first image corresponds to only a single tiepoint on a second image, only the position of the first image (and not orientation) relative to the second image may be fixed.

The output of the orthorectification module 130 may be received by a mapping module 132, e.g. orthorectified image(s) and/or transformation(s) to achieve orthorectification. The mapping module 132 may receive pixel locations from the object detection module 128.

The mapping module 132 may be configured to map pixel locations of hotspots in the aerial images to corresponding pixel locations in the orthorectified images to determine the location of the hotspot 126.

In various embodiments, the pixel locations of the hotspots may be mapped to the orthorectified image via the mapping provided by the orthorectification module 130, e.g. by referencing the pixel location to an index of a first data structure representing an aerial image and then evaluating the corresponding index in the second data structure representing the corresponding orthorectified image.

In some embodiments, the pixel locations of the hotspots may be mapped to the orthorectified image by generating a vector from a known reference location, e.g. known GNSS coordinates at a center of the aerial image, to the pixel location and then applying that vector to the orthorectified image, which also contains the same known GNSS coordinates. In some embodiments, the vector may be transformed to match (the transformed internal) coordinates of the orthorectified image before application thereto.

By performing such a mapping, the mapping module 132 generates a pixel location in the orthorectified image from the pixel location of a hotspot 126 in the aerial image, from which a geographic location of the hotspot may be evaluated, e.g. in terms of latitude/longitude and/or an earth-centred fixed coordinate system. In some embodiments, the geographic location may be determined by using a position vector from a known location in the orthorectified image to the pixel location in the orthorectified image. Such a position vector may be augmented (e.g. via vectorial addition) to the position vector of the known location, which may be expressed in terms of latitude/longitude and/or an earth-centric fixed coordinate system, to achieve a geographic location of the hotspot 126.

In some embodiments, the pixel locations of the hotspots 126 may be mapped to a three- dimensional model (such as a digital surface model) to determine a geographically-accurate location of the hotspot 126. In various embodiments, an image having the hotspot 126 may be associated with a portion of the three-dimensional model. Extending a ray from the pixel location to an intersection point with the three-dimensional model may allow determination of a geographic coordinate. For example, the geographic coordinates may then be determined by projecting the three-dimensional model onto a geographic coordinate system.

In some embodiments, the pixel location of the hotspot 126 may be associated with a portion of the three-dimensional model by using tie-points. In some embodiments, such tie-points may be referenced to tie-points associated with the three-dimensional model, which may be provided by the orthorectification module 130 to the mapping module 132.

In various embodiments, the location of the hotspot 126 may be transmitted to the server 112, which may be network-connected and may be configured to provide access to data indicative of hotspot locations to a plurality of users, e.g. via a user terminal operably connected to the server 112. In some embodiments, the server 112 may provide access to the data via web interface or Application Programming Interface (API).

FIG. 3 shows an aerial image 135 showing hotspots 136, and control points, in accordance with an embodiment.

In some embodiments, control points may be points with known locations in the aerial image 135. In some embodiments, no control points may be used or identified for use for orthorectification.

In some embodiments, control points may include (or refer to) a reference position 138, such as a center position, of every aerial image. Such a reference position 138 may be associated with a geographic coordinate location based on a geographic location of the aerial imaging system 104. In various embodiments, the geographic location of the aerial imaging system 104 (or aerial imagery system) relative to the aerial vehicle 102 and image (internal or pixel) coordinates of the reference position 138 may be ascertained in pre-flight calibration, e.g. ground-based calibration. The position of the aerial vehicle 102 may be determined by on-board telemetry and instrumentation, e.g. using a coordinate location received from a global satellite navigation system (GNSS) such as GPS, differential GPS (DGPS), and/or GLONASS. A known geographic location of the reference position 138 in each aerial image may then be determined by (vectorially) adding the image coordinates of the reference position, the geographic location of the aerial imaging system 104 relative to the aerial vehicle 102, and the geographic location of the aerial vehicle 102 itself.

In some embodiments, control points may include ground control points 140. Ground control points may be objects with known geographic locations that are depicted in aerial images. Ground control points may generally be immovable objects that are clearly discernible and sufficiently localizable in aerial imagery, e.g. infrastructure such as power plants or industrial plants. Ground control points may be automatically flagged in aerial images, e.g. in the object detection module 128

In some embodiments, control points and ground control points 140 may be also used as tie-points. Tie-points may include manually or automatically detected common points between aerial images. Certain control points and/or ground control points 140 may appear in multiple aerial images, which may allow such points to be used as tie-points. In some cases, control points and/or ground control points 140 may also be referenced to their absolute locations, which may be known a priori, even if such points appear in only one image. As referred to herein, tie-points in an aerial image may also refer to control points and ground control points 140 if they tie the aerial image to an absolute location and/or to another aerial image. In some embodiments, tie-points may include a point of high contrast visible in two or more aerial images.

FIG. 4 shows a set 142 of aerial images 135A-135D, in accordance with an embodiment.

The aerial image 135A corresponds to the aerial image 135 in FIG. 3.

The aerial images 135A-135C may be overlapping, according to a direction 143 of flight of the aerial vehicle 102. For example, some features (see FIG. 3; left unlabelled here for clarity) in the left-most image(s) may not be seen in the right-most image(s). FIG. 5 is a schematic of a workflow for generating orthorectified images and orthophotomosaics (or orthomosaics), in accordance with an embodiment.

In various embodiments, orthorectifying may involve addressing radiometric distortions and geometric distortions.

Radiometric distortions may include errors in conversion of (ground or surface) reflectance and/or radiation values to gray values or digital numbers in the aerial image. In various embodiments, radiometric distortions may be caused by the sun’s azimuth and elevation, atmospheric conditions, and sensor limitations. For example, performing thermal infrared aerial photography at night may reduce radiometric distortions.

Geometric distortions may include errors in scale and location in the image, e.g. which may be caused by terrain displacement, the curvature of the Earth, perspective projections and instrumentation.

In an example workflow, the set 142 of raw aerial images may be corrected for radiometric distortion at block 144, then corrected for geometric distortion at block 146, then post-processed at block 148 to generate appropriate orthorectified images 150.

For example, correcting radiometric distortions at block 144 may include calibrating images based on known parameters and image meta data.

For example, correcting geometric distortions at block 146 may include determining control points (discussed above), aligning images, and generating digital surface models representing 3D information, e.g. using point clouds. The set 142 of aerial images may be orthorectified using the digital surface model.

In some embodiments, a digital surface model may be generated using the set 142 of aerial images and a set of coordinate locations corresponding to the set 142. In various embodiments, the set of coordinate locations may be generated using positions of the aerial imaging system 104 that captured the set 142 of aerial images. The positions of the aerial imaging system may be based on a set of Global Navigation Satellite System (GNSS) coordinates associated with the aerial imaging system 104 when the aerial imaging system 104 captured the set 142 of aerial images. In some embodiments, LIDAR may be used to determine distances of one or more points and may be used in developing digital surface models, e.g. to generate tie-points. For example, a digital surface model may be generated using the set 142 of aerial images and reference positions 138 corresponding to these aerial images.

In various embodiments, a digital surface model is generated using the set 142 of aerial images and known locations of objects depicted therein, e.g. ground control points 140.

For example, post-processing the corrected images at block 148 may include stitching or otherwise combining the orthophotos together to generate an orthophotomosaic, and/or generating images in an appropriate digital format (e.g. to represent 3D information).

In some embodiments, the digital surface model may be a point cloud model generated using overlapping aerial images. For example, the digital surface model may include a list of three- dimensional position vectors representing physical locations that are determined to be part of the modelled three-dimensional surface. In some embodiments, the point cloud model may be equipped with functions for determining the shape of the surface in-between points and/or surface reconstruction functions. In some embodiments, the point cloud model may be formed using tiepoints and/or may only include points corresponding to tie-points. Such tie-points may include automatically generated tie-points. In some embodiments, a dense point cloud model may be formed, e.g. if a sufficiently high number of aerial images are provided.

In various embodiments, “digital elevation model” and “digital surface model” may refer to substantially the same class of models. Such models may be representative or indicative of the three-dimensional surface whose projection forms the aerial images, e.g. a digital surface model may include a (coarse-grained or smoothened) representation of a tree canopy. For example, such models may be distinct from “digital terrain models” that do not necessarily account for three- dimensional bodies formed on top of the underlying terrain.

FIG. 6 is a detailed schematic of a workflow for generating orthorectified images and orthophotomosaics (or orthomosaics), in accordance with an embodiment.

Step 1 may generally represent receiving raw aerial images and correcting for radiometric distortions.

Steps 2 and 3 may generally represent correcting the images for geometric distortions and generating 3D information for the image (digital models for the imaged surface). The sparse and dense point clouds may be 3D point clouds representing 3D digital surface model(s).

Step 4 may generally represent post-processing the corrected images. In various embodiments, images may be aligned by identifying common features and overlapping these to form a 3D point cloud of common points.

The points of a 3D point cloud may be connected to form a mesh, e.g. including interpolation and distortions or transformations based on the 3D mesh may be applied to each aerial image to achieve orthorectification.

Images referred to in FIG. 6 may be in any suitable format, e.g. Tag Image File Format (TIFF), as shown, JPEG, and/or R.JPEG.

FIG. 7 is a schematic of an object detection module 128, in accordance with an embodiment.

The object detection module 128 receives data indicative of the aerial image 135 and processes it in accordance with a machine learning model 152. In various embodiments, the machine learning model 152 is a region-based convolutional neural network (R-CNN or a variant thereof such as Faster R-CNN) for object detection or a YOLO (You Only Look Once) model.

In some cases, it was discovered that the multi-scale nature of hotspots and the surrounding tree cover (and/or other environmental features) poses particular challenges for object detection that are less pronounced when detecting objects in non-aerial images, e.g. pets in indoor photographs. Additionally, for firefighting purposes, it is preferable to have a false positive rather than not detect a hotspot (false negative) due to the destructive potential of a hotspot. In some embodiments, fast and faster R-CNN were found to be particularly advantageous for detection of hotspots in wildfires, including for the considerations described above. It was also found that R-CNN-based machine learning models may be particularly computationally effective for hotspot detection, e.g. faster and liable to use less computational resources.

In some embodiments, the machine learning model 152 may be implemented using a software library such as PyTorch Tensors.

In some embodiments, the input data to be processed in accordance with a machine learning model 152 is indicative of a normalized three-channel image. In some embodiments, the input data may be a single-channel image, e.g. with integer values (uint!6 type).

In various embodiments, processing in accordance with the machine learning model 152 may include modifying the image data using an image mean and standard deviation. For example, the mean of the image data may be subtracted from the image data, and the resultant value may be normalized by the standard deviation. Thereafter, the data may be recentered by adding one and dividing by two.

In various embodiments, processing the aerial image 135 in accordance with the machine learning model 152 may include generating (data indicative of) region proposals 154 for regions of the aerial image 135 where objects may be present. Thereafter, such regions may be processed in accordance with a pre-trained classifier 156, such as a convolutional neural network or support vector machine, to classify objects detected in the region proposals 154 or to provide an indication if no objects are detected in a particular region.

In some embodiments, an output of the object detection module 128 may be segmentation mask 158 indicating the regions where hotspots are detected. In some embodiments, coordinates of bounding boxes and/or pixels therein may be generated as output of the object detection module 128. In some embodiments, a confidence of hotspot detection may be generated by the object detection module 128 to attach metric of quality to a prediction.

In some embodiments, an output of the object detection module 128 may include a list of rectangular bounding boxes specified in terms of their comers, e.g. in the form [xi, y i, X2, y2] where (xi,yi) are coordinates of a first end of the diagonal and (x2, y2) are coordinates of a second end of the diagonal, a list of segmentation masks (one for each bounding box), and a confidence value between 0 and 1 (one for each bounding box).

In some embodiments, only thermal infrared images may form input channels or features to the machine learning model 152. In some embodiments, the thermal infrared images may be augmented by additional data. In some embodiments, it is found that moisture levels, elevation data, and/or altitude may be particularly effective in achieving accurate detection of hotspots.

In various embodiments, the machine learning model 152 may be a supervised machine learning model. The machine learning model 152 may be pre-trained using training data 160.

The training data 160 may include aerial images that have corresponding data indicative of pixel locations of hotspots in the aerial images, e.g. in the form of one or more of the output types of the object detection module 128 described above. This data is at least partially generated by manual labelling of hotspots in aerial images. In various embodiments, images containing no hotspots are also included in the training data 160. In some embodiments, image and data augmentation may be deployed to expand the training dataset. For example, data augmentation may be used during training. In various embodiments, images may be randomly cropped and flipped horizontally, vertically, or both.

In various embodiments, the training data 160 is configured to include images from two types of cameras or a plurality of types of cameras in a particular class, e.g. thermal infrared cameras with a similar or the same spectral band. In various embodiments, the aerial images represented in the training data 160 may be captured at altitudes varying between 50-500 m or more.

In various embodiments, the training data 160 may comprise over 100 aerial images or between 100 and 500 aerial images. The aerial images in the training data 160 may have a smaller size than an aerial image 135 used as input to the machine learning model 152. For example, in various embodiments, the aerial images in the machine learning model 152 may be 64x64, 128x128, or 256x256 pixels.

In various embodiments, once the machine learning model 152 is trained, parameters thereof are stored in memory for use without reference to the training data 160. In various embodiments, the machine learning model 152 may be periodically retrained and/or updated.

In various embodiments, during training, the learning rate of the model may start out at higher values (0.01) and progressively decrease. Gradients may be accumulated into minibatches due to the low batch size to accomodate memory constraints of the machine used for training. In some embodiments, gradients are clipped to 1.0 due to higher initial learning rate. In some embodiments, 16-bit precision is used to lower memory usage and memory bandwidth and increase speed of training. In various embodiments, Stochastic Weight Averaging may be used to improve generalization.

FIG. 8 is a schematic of a system 100 for locating hotspots using aerial imagery, in accordance with another embodiment.

In the system 100 in FIG. 8, the orthorectification module 130 may be configured to generate orthorectified images (including orthophotomosaic) where needed. This may reduce computational cost and improve computational speed (reduce latency).

In some embodiments, the object detection module 128 may detect an aerial image having a hotspot 136 located therein and may send data to the orthorectification module 130, or a module that has pre-computed orthorectified image(s) or transformations associated with such image(s), that is indicative of such a hotspot-containing aerial image. In some embodiments, the object detection module 128 may send data indicative of the pixel location of the hotspot 136 in the hotspot-containing aerial image.

In some embodiments, the orthorectification module 130 may be configured to process the set 142 of aerial images to determine or generate a strict subset 162 of aerial images based on the data indicative of the hotspot-containing aerial image and/or the pixel location of the hotspot 136. In various embodiments, after determining the strict subset 162, an orthorectified image may be generated by the orthorectification module 130 based on the strict subset 162.

In various embodiments, the strict subset 162 may include a set of geographically overlapping aerial images suitable for generating a digital surface model of a region 164 depicted as surrounding the pixel location of the hotspot 136. In some embodiments, the strict subset may be determined to include aerial images depicting an object with a known location or other control point(s). For example, in some cases, the strict subset 162 may be expanded to include additional aerial images that have control point depicted therein.

FIG. 9 is a flow chart of an exemplary computer-implemented method 900 of locating a hotspot using aerial imagery, in accordance with an embodiment.

Step 901 may include training a machine learning model using training data that includes data indicative of images and known hotspots in the images. In some embodiments, step 901 may be performed separately or may be eliminated, e.g. if a pre-trained machine learning model with known parameters is available.

Step 902 may include receiving data indicative of a set of aerial images suitable for stereoscopic imagery

Step 904 may include determining a pixel location of a hotspot in an aerial image of the set of aerial images by processing the set of aerial images in accordance with the machine learning model

Step 906 may include generating an orthorectified image in response to the set of aerial images.

In some embodiments, generating the orthorectified image may include generating a (image) transformation associated with an orthorectified image. In some embodiments, data indicative of the orthorectified image may be generated. In some embodiments, only data indicative of the transformation may be generated. Step 908 may include mapping the pixel location of the hotspot in the aerial image to a corresponding pixel location in the orthorectified image to determine a location of the hotspot.

In some embodiments, such a mapping may include transforming the pixel location of the hotspot using the transformation, e.g. without using data indicative of an orthorectified image.

Some embodiments of the method 900 may include transmitting the location of the hotspot to a network-connected server configured to provide access to data indicative of hotspot locations to a plurality of users.

In some embodiments of the method 900, generating the orthorectified image using the set of aerial images may include: generating a digital surface model using the set of aerial images and known locations of objects depicted in the set of aerial images.

In some embodiments of the method 900, the orthorectified image is an orthophotomosaic, and generating the orthorectified image using the set of aerial images may include: generating a set of orthophotos corresponding to the set of aerial images by orthorectifying images of the set of aerial images using the digital surface model; and combining the orthophotos to generate the orthophotomosaic.

In some embodiments of the method 900, the machine learning model is a region-based convolutional neural network for object detection.

In some embodiments of the method 900, the set of aerial images includes thermal infrared images with a resolution between 100 pixels/m² and 1000 pixels/m² or between 1 pixels/m² and 10 pixels/m²

In some embodiments of the method 900, generating the orthorectified image in response to the set of aerial images may include determining a strict subset of the set of aerial images based on the pixel location of the hotspot; and after determining the strict subset, generating the orthorectified image based on the strict subset.

In some embodiments of the method 900, the strict subset of the set of aerial images may include a set of geographically overlapping aerial images suitable for generating a digital surface model of a region depicted as surrounding the pixel location.

In some embodiments of the method 900, the strict subset of the set of aerial images may include a depiction of an object with a known location. In some embodiments of the method 900, generating the orthorectified image using the set of aerial images may include: generating a data indicative of elevation (e.g. digital surface model) using the set of aerial images and a set of coordinate locations corresponding to the set of aerial images.

In some embodiments of the method 900, the set of coordinate locations may be generated using positions of an aerial imaging system that captured the set of aerial images.

In some embodiments of the method 900, the positions of the aerial imaging system may be based on a set of Global Navigation Satellite System (GNSS) coordinates associated with the aerial imaging system when the aerial imaging system captured the aerial images.

In some embodiments, step 904, step 906, step 908, and/or a step of transmitting the location of the hotspot to a network-connected server may be executed after receiving a signal indicative of an event, i.e. a triggering event. For example, in some embodiments, the triggering event may refer to a successful capture of an aerial image or a certain number of aerial images, an upload of an aerial image to a server (e.g. a processing server), and/or a request generated by a user device (e.g. in response to interaction of the user with an interface or a graphical user interface). In various embodiments, a device involved in the triggering event may generate the signal. For example, the signal may be data indicative of or characterizing the event.

In some embodiments, an event trigger may cause batch processing of aerial photos.

FIG. 10 illustrates a block diagram of a computing device 1000, in accordance with an embodiment of the present application.

As an example, the system 100, the computing device 110, the server 112, or the client device 114 of FIG. 1 may be implemented using the example computing device 1000 of FIG. 10.

The computing device 1000 includes at least one processor 1002, memory 1004, at least one I/O interface 1006, and at least one network communication interface 1008.

The processor 1002 may be a microprocessor or microcontroller, a digital signal processing (DSP) processor, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, a programmable read-only memory (PROM), or combinations thereof.

The memory 1004 may include a computer memory that is located either internally or externally such as, for example, random-access memory (RAM), read-only memory (ROM), compact disc read-only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read-only memory (EPROM), and electrically-erasable programmable read-only memory (EEPROM), Ferroelectric RAM (FRAM).

The I/O interface 1006 may enable the computing device 1000 to interconnect with one or more input devices, such as a keyboard, mouse, camera, touch screen and a microphone, or with one or more output devices such as a display screen and a speaker.

The networking interface 1008 may be configured to receive and transmit data sets representative of the machine learning models, for example, to a target data storage or data structures. The target data storage or data structure may, in some embodiments, reside on a computing device or system such as a mobile device.

FIG. 11 is a schematic workflow for the orthorectification module 130, in accordance with an embodiment.

The orthorectification module 130 may receive the set 142 raw aerial images and output tie-points and/or a three-dimensional model (block 172).

In block 168, the orthorectification module 130 may identify common points, or other types of tiepoints (e.g. ground control points), in the set 142 of raw aerial images.

In block 170, the orthorectification module 130 may generate a three-dimensional model using the raw aerial images. For example, the three-dimensional model may be a digital surface model. In various embodiments, the three-dimensional model may be a sparse or dense point cloud, a mesh representing or interpolated (or extrapolated) from such point clouds, and/or transformations (or equations) representing (or indicative of) the sparse or dense point clouds or a surface interpolated (or extrapolated) therefrom. Such surfaces may be represented using mathematical functions, geometric (e.g. linear) transformations, or other types of representations (mathematical and/or digital).

In some embodiments, it is possible that the orthorectification module 130 generates the tie-points in parallel with the three-dimensional model.

FIG. 12 is a flow chart of an exemplary computer-implemented method 1200 of locating a hotspot of a wildfire using aerial imagery, in accordance with an embodiment.

Step 1201 may include training a machine learning model using training data that includes data indicative of images and known hotspots in the images. In some embodiments, step 1201 may be performed separately or may be eliminated, e.g. if a pre-trained machine learning model with known parameters is available.

Step 1202 may include receiving data indicative of a set of aerial images suitable for stereoscopic imagery

Step 1204 may include determining a pixel location of a hotspot in an aerial image of the set of aerial images by processing the set of aerial images in accordance with the machine learning model.

Step 1206 may include generating data indicative of a three-dimensional model using the set of aerial images.

Step 1208 may include mapping the pixel location of the hotspot in the aerial image to a corresponding location in the three-dimensional model to determine a location of the hotspot.

In some embodiments of the method 1200, generating data indicative of the three-dimensional model using the set of aerial images includes determining a strict subset of the set of aerial images based on the pixel location of the hotspot; and after determining the strict subset, generating data indicative of the three-dimensional model based on the strict subset.

In some embodiments of the method 1200, the step of generating data indicative of the three- dimensional model using the set of aerial images is part of a step of generating an orthorectified image in response to the set of aerial images.

In some embodiments of the method 1200, mapping the pixel location of the hotspot in the aerial image to the corresponding location in the three-dimensional model to determine the location of the hotspot includes mapping the pixel location of the hotspot in the aerial image to a corresponding pixel location in the orthorectified image.

In some embodiments of the method 1200, generating the orthorectified image in response to the set of aerial images includes determining a strict subset of the set of aerial images based on the pixel location of the hotspot; and after determining the strict subset, generating the orthorectified image based on the strict subset.

In some embodiments of the method 1200, the strict subset of the set of aerial images includes a set of geographically overlapping aerial images suitable for generating data indicative of a digital surface model of a region depicted as surrounding the pixel location. In some embodiments of the method 1200, the strict subset of the set of aerial images includes a depiction of an object with a known location.

Some embodiments of the method 1200 may include transmitting the location of the hotspot to a network-connected server configured to provide access to data indicative of hotspot locations to a plurality of users.

In some embodiments of the method 1200, generating data indicative of the three-dimensional model using the set of aerial images includes: generating data indicative of elevation using the set of aerial images and known locations of objects depicted in the set of aerial images.

In some embodiments of the method 1200, generating data indicative of the three-dimensional model using the set of aerial images includes: generating a set of orthophotos corresponding to the set of aerial images by orthorectifying images of the set of aerial images using the three- dimensional model; and combining the orthophotos to generate an orthophotomosaic.

In some embodiments of the method 1200, the machine learning model is a region-based convolutional neural network for object detection.

In some embodiments of the method 1200, the set of aerial images includes thermal infrared images with a resolution between 100 pixels/m² and 1000 pixels/m²

In some embodiments of the method 1200, the set of aerial images includes thermal infrared images with a resolution between 1 pixels/m² and 10 pixels/m².

In some embodiments of the method 1200, generating data indicative of the three-dimensional model using the set of aerial images includes: generating a digital surface model using the set of aerial images and a set of coordinate locations corresponding to the set of aerial images.

In some embodiments of the method 1200, the set of coordinate locations is generated using at least one of positions of an aerial imaging system that captured the set of aerial images or positions that are common between at least two aerial images of the set of aerial images.

In some embodiments of the method 1200, the positions of the aerial imaging system are based on a set of Global Navigation Satellite System (GNSS) coordinates associated with the aerial imaging system when the aerial imaging system captured the aerial images. The term “connected” or "coupled to" may include both direct coupling (in which two elements that are coupled to each other contact each other) and indirect coupling (in which at least one additional element is located between the two elements).

Although the embodiments have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the scope.

Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification. As one of ordinary skill in the art will readily appreciate from the disclosure, processes, machines, manufacture, compositions of matter, means, methods, or steps, presently existing or later to be developed, that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein may be utilized. Accordingly, the embodiments are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method of locating a hotspot of a wildfire using aerial imagery, the method comprising: receiving data indicative of a set of aerial images suitable for stereoscopic imagery; determining a pixel location of the hotspot in an aerial image of the set of aerial images by processing the set of aerial images in accordance with a machine learning model, the machine learning model having been pre-trained using training data, the training data including data indicative of images and known hotspots in the images; generating data indicative of a three-dimensional model using the set of aerial images; and mapping the pixel location of the hotspot in the aerial image to a corresponding location in the three-dimensional model to determine a location of the hotspot.

2. The method of claim 1, wherein generating data indicative of the three-dimensional model using the set of aerial images includes determining a strict subset of the set of aerial images based on the pixel location of the hotspot; and after determining the strict subset, generating data indicative of the three-dimensional model based on the strict subset.

3. The method of claim 1, wherein the step of generating data indicative of the three-dimensional model using the set of aerial images is part of a step of generating an orthorectified image in response to the set of aerial images.

4. The method of claim 3, wherein mapping the pixel location of the hotspot in the aerial image to the corresponding location in the three-dimensional model to determine the location of the hotspot includes mapping the pixel location of the hotspot in the aerial image to a corresponding pixel location in the orthorectified image.

5. The method of claim 3, wherein generating the orthorectified image in response to the set of aerial images includes determining a strict subset of the set of aerial images based on the pixel location of the hotspot; and after determining the strict subset, generating the orthorectified image based on the strict subset.

6. The method of any one of claims 5 or 2, wherein the strict subset of the set of aerial images includes a set of geographically overlapping aerial images suitable for generating data indicative of a digital surface model of a region depicted as surrounding the pixel location.

7. The method of any one of claims 5 or 2, wherein the strict subset of the set of aerial images includes a depiction of an object with a known location.

8. The method of claim 1, further comprising: transmitting the location of the hotspot to a network-connected server configured to provide access to data indicative of hotspot locations to a plurality of users.

9. The method of claim 1, wherein generating data indicative of the three-dimensional model using the set of aerial images includes: generating data indicative of elevation using the set of aerial images and known locations of objects depicted in the set of aerial images.

10. The method of claim 9, wherein generating data indicative of the three-dimensional model using the set of aerial images includes: generating a set of orthophotos corresponding to the set of aerial images by orthorectifying images of the set of aerial images using the three-dimensional model; and combining orthophotos of the set of orthophotos to generate an orthophotomosaic.

11. The method of claim 1, wherein the machine learning model is a region-based convolutional neural network for object detection.

12. The method of claim 1, wherein the set of aerial images includes thermal infrared images with a resolution between 100 pixels/m² and 1000 pixels/m²

13. The method of claim 1, wherein the set of aerial images includes thermal infrared images with a resolution between 1 pixels/m² and 10 pixels/m²

14. The method of claim 1 , wherein generating data indicative of the three-dimensional model using the set of aerial images includes: generating a digital surface model using the set of aerial images and a set of coordinate locations corresponding to the set of aerial images.

15. The method of claim 14, wherein the set of coordinate locations is generated using at least one of positions of an aerial imaging system that captured the set of aerial images or positions that are common between at least two aerial images of the set of aerial images.

16. The method of claim 15, wherein the positions of the aerial imaging system are based on a set of Global Navigation Satellite System (GNSS) coordinates associated with the aerial imaging system when the aerial imaging system captured the aerial images.

17. A system for locating a hotspot of a wildfire using aerial imagery, the system comprising: a processor; computer-readable memory coupled to the processor and storing processor-executable instructions that, when executed, configure the processor to: receive data indicative of a set of aerial images suitable for stereoscopic imagery; determine a pixel location of the hotspot in an aerial image of the set of aerial images by processing the set of aerial images in accordance with a machine learning model, the machine learning model having been pre-trained using training data, the training data including data indicative of images and known hotspots in the images; generate data indicative of a three-dimensional model using the set of aerial images; and map the pixel location of the hotspot in the aerial image to a corresponding location in the three-dimensional model to determine a location of the hotspot.

18. The system of claim 17, wherein the processor-executable instructions, when executed, further configure the processor to: transmit the location of the hotspot to a network-connected server configured to provide access to data indicative of hotspot locations to a plurality of users.

19. The system of claim 17, wherein to generate data indicative of the three-dimensional model using the set of aerial images includes to: determine a strict subset of the set of aerial images based on the pixel location of the hotspot; and after determination of the strict subset, generate data indicative of the three-dimensional model based on the strict subset.

20. A non-transitory computer-readable medium having stored thereon machine interpretable instructions which, when executed by a processor, cause the processor to perform a computer- implemented method of locating a hotspot of a wildfire using aerial imagery, the method comprising: receiving data indicative of a set of aerial images suitable for stereoscopic imagery; determining a pixel location of the hotspot in an aerial image of the set of aerial images by processing the set of aerial images in accordance with a machine learning model, the machine learning model having been pre-trained using training data, the training data including data indicative of images and known hotspots in the images; generating data indicative of a three-dimensional model using the set of aerial images; and mapping the pixel location of the hotspot in the aerial image to a corresponding pixel location in the three-dimensional model to determine a location of the hotspot.

21. The non-transitory computer-readable medium of claim 20, wherein the method further comprises: transmitting the location of the hotspot to a network-connected server configured to provide access to data indicative of hotspot locations to a plurality of users.

22. The non-transitory computer-readable medium of claim 20, wherein generating the three- dimensional model using the set of aerial images includes determining a strict subset of the set of aerial images based on the pixel location of the hotspot; and after determining the strict subset, generating data indicative of the three-dimensional model based on the strict subset.

23. The non-transitory computer-readable medium of claim 22, wherein the strict subset of the set of aerial images includes a set of geographically overlapping aerial images suitable for generating a digital surface model of a region depicted as surrounding the pixel location.

24. A computer-implemented method of locating a hotspot of a wildfire using aerial imagery, the method comprising: receiving data indicative of a set of aerial images suitable for stereoscopic imagery; determining a pixel location of the hotspot in an aerial image of the set of aerial images by processing the set of aerial images in accordance with a machine learning model, the machine learning model having been pre-trained using training data, the training data including data indicative of images and known hotspots in the images; generating an orthorectified image in response to the set of aerial images; and mapping the pixel location of the hotspot in the aerial image to a corresponding pixel location in the orthorectified image to determine a location of the hotspot.

25. A system for locating a hotspot of a wildfire using aerial imagery, the system comprising: a processor; and computer-readable memory coupled to the processor and storing processor-executable instructions that, when executed, configure the processor to execute the computer- implemented method of claim 24.

26. A non-transitory computer-readable medium having stored thereon machine interpretable instructions which, when executed by a processor, cause the processor to perform the computer- implemented method of claim 24.