EP4724986A1

EP4724986A1 - Method and apparatus for determining a location of a platform

Info

Publication number: EP4724986A1
Application number: EP24739242.6A
Authority: EP
Inventors: Antony Joseph Frank Lowe
Original assignee: MBDA UK Ltd
Current assignee: MBDA UK Ltd
Priority date: 2023-06-07
Filing date: 2024-06-06
Publication date: 2026-04-15
Also published as: GB202308455D0; GB202408087D0; GB2631587A; WO2024252147A1; AU2024284653A1

Abstract

There is disclosed a method for determining a location of a platform in a region. The platform comprises an event camera having an array of pixels and a memory store. Each pixel is operable to output events which represent a change in light intensity recorded at that pixel. Before platform motion in the region, a signature library is stored in the memory store. Each of the signatures has an associated patch at a known location, and each signature represents the number of events that would be captured by viewing the associated patch using the event camera whilst moving the sightline of the event camera through each of a predetermined set of intervals on a predetermined trajectory. During platform motion in the region, the platform location is determined by using the event camera to obtain a sensed signature. The sensed signature is compared to the signature library to locate the platform.

Description

METHOD AND APPARATUS FOR DETERMINING A LOCATION OF A PLATFORM

FIELD OF THE INVENTION

This invention relates to a method and apparatus for determining a location of a platform.

BACKGROUND OF THE INVENTION

A number of ways in which the location of a platform can be determined are known. Satellite-based systems, such as the global positioning system are now commonly used, but may not always be available, for example because signals from the satellites are obstructed. It is a particular problem in military applications, when it is also possible for the signals to be jammed or otherwise interrupted. Under these circumstances alternative ways of determining location must be used. Inertial management units can help, but they drift and therefore require regular location updates from other sources as well.

Image-based location determination is also known, and can be used for airborne platforms. Overflown terrain is imaged, and computationally matched to a stored database of imagery of the relevant region, or of salient features of the relevant region. For example, a database may store scale invariant feature transform (SIFT) points, or speeded up robust features (SURF) points of a relevant region. Such schemes can work well to match perceived image features to their equivalent in a database, especially if the region to be examined is relatively small. This may be the case if the location is constrained through knowledge of a prior-determined location. However, it is typically highly computationally intensive to determine a first location of a platform in a large possible search space. Significant time and/or processing power can be required; and storing the database of features can require a large amount of memory. SUMMARY OF THE INVENTION

In accordance with a first aspect of the present invention there is provided a method for determining a location of a platform in a region, the platform comprising an event camera and a memory store; the event camera having a sensor comprising an array of pixels, each pixel being operable to output events, each event being a change in light intensity recorded at a pixel; and the method comprising the steps of: a) before platform motion in the region: i. providing a signature library, the signature library comprising a number of signatures, each of the signatures having an associated patch at a known location in the region, and each of the signatures representing the number of events that would be captured by viewing the associated patch using the event camera whilst moving the sightline of the event camera through each of a predetermined set of intervals on a predetermined trajectory; ii. storing the signature library in the memory store; b) during platform motion in the region, determining the location of the platform by: i. moving the sightline of the event camera in the predetermined trajectory whilst capturing sensed event data; ii. recording the number of events captured in each of the set of predetermined intervals in the predetermined trajectory to obtain a sensed signature; and iii. comparing the sensed signature to the signature library to determine the location of the platform in the region.

An event camera could be any image sensor coupled with appropriate processing to enable it to output changes in intensity levels, rather than absolute intensity levels, from individual pixels of the image sensor. The navigation method can operate in the absence of external navigation signals, such as global positioning system signals, or when such systems are jammed. In addition the signatures provide a much simpler manner of identifying a location from image data than is possible with existing methods of image-based navigation, and require only limited storage capacity to store for large regions.

The predetermined trajectory may be a rotation of the sightline of the event camera. The predetermined trajectory may be a rotation through at least 180°. The predetermined trajectory may move the sightline in a circle without altering the camera orientation. The predetermined trajectory may be effected by nutating the camera.

The predetermined intervals may be angular intervals in the range between 0.1 ° and 10°, preferably 1 °.

The platform may be an airborne platform. For example, the platform maybe an unmanned air system, a missile, a helicopter, or other type of aircraft.

The event camera may face the ground during the recording of the sensed event data.

The step of providing a signature library may comprise defining one or more characteristic features of the signatures, and indexing the signature library against each of the characteristic features. Use of indices enables the library to be searched rapidly for matches to the sensed signature. Exemplary characteristic features can include the ratio of the minimum number of events in any one interval to the maximum number of events in any one interval; the mean of the number of events in each of the intervals; the standard deviation of the number of events in each of the intervals; the angle, relative to north, at which the interval capturing the maximum number of events lies; the angle, relative to north, at which the interval capturing the minimum number of events lies; the angle at which the weighted centroid interval lies; the angle between the interval capturing the largest number of events, and the interval capturing the second largest number of events; the angle between the interval capturing the least number of events, and the interval capturing the second least number of events. In some examples the characteristic features may be defined relative only to other features of the signature, so that it is not necessary to independently identify the orientation of the platform. The step of comparing the sensed signature to the signature library may comprise the step of determining the one or more characteristic features for the sensed signature, and identifying one or more candidate signatures in the signature library from the index of characteristic features.

The step of determining the location of the platform may comprise determining a score relating to the similarity of the sensed signature to each of the candidate signatures, and determining the location of the platform to be the location of the patch associated with the candidate signature having the score indicating the highest similarity to the sensed signature. The score may, for example, be the sum of the absolute difference between the number of events recorded in each interval of the sensed signature and the candidate signature, after normalisation by the mean value of the bins of each histogram.

The step of providing a signature library may be performed by overflying the region on a platform equipped with an event camera and an independent navigation system and directly recording the signatures. Alternatively, the step of providing a signature library may comprise the steps of: a) providing aerial imagery of the region b) dividing the aerial imagery into a plurality of image patches at each of the associated locations; c) determining the signature by simulating, for each of the image patches, the number of events that would be captured by viewing said each of the image patches using the event camera whilst moving the sightline of the event camera through each of a predetermined set of intervals on a predetermined trajectory.

According to a second aspect of the present invention there is provided a computer readable medium having stored thereon a signature, which signature comprises information extracted from data recorded by an event camera; the event camera having a sensor comprising an array of pixels and being operable to output events, each event being a change in light intensity recorded at a pixel; the signature being determined by moving an event camera in a predetermined trajectory and recording the number of events captured in each of a set of intervals of the trajectory.

According to a third aspect of the present invention there is provided apparatus for determining a location comprising an event camera, a memory store, and a processor, the event camera being arranged such that its sightline is movable; and the apparatus being configured to perform the method defined above.

The invention extends to an airborne platform comprising the apparatus defined above, wherein the event camera is arranged such that its sightline is movable relative to the platform.

The invention extends to a computer readable medium comprising instructions that when executed cause a processor to implement the method described above.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described by way of example only and with reference to the accompanying drawings, of which:

Figure 1 is a flow diagram illustrating steps of a method in accordance with an example of the invention;

Figure 2 is a schematic illustration of an aerial platform according to an example of the invention and operating the method of Figure 1 ;

Figure 3 is a schematic illustration of a patch of terrain;

Figures 4a and 4b are graphical representations of a signature of the patch of Figure 3;

Figure 5 is a flow diagram schematically illustrating steps in a method for searching a library of signatures; and

Figures 6a and 6b are schematic illustrations of a further platform in accordance with an example of the invention. DETAILED DESCRIPTION OF THE INVENTION

The embodiments described below provide an apparatus and method for determining the location of a platform moving in a region. Figure 1 is a flow chart 100 providing an overview of a method for determining the location of a platform in a region in accordance with an embodiment of the invention. At step 110, a signature library is provided. The region is divided into patches at known locations, and for each patch, a signature is defined. The signatures are constructed such that they can be sensed using an event camera onboard the platform, with the event camera being controlled so as to move its sightline in a predefined trajectory. The signatures for each patch in the region are stored in the signature library that is constructed prior to the platform operating in the region. The signatures, and the construction of the signature library, are described in further detail below.

At step 120, the signature library is stored in a memory store onboard the platform. In this way, the platform has access to the signature library whilst it is moving in the region. At step 130, the platform captures event data, using an onboard event camera, whilst it moves the sightline of the event camera in a predetermined trajectory. At step 140, the event data is processed to determine a sensed signature for the platform’s present location. At step 150, the sensed signature is compared to the signature library. If the sensed signature matches a signature in the library, at step 160, the platform can determine its present location as being that associated with the matching signature in the signature library. Various techniques, described in further detail below, can be applied if the sensed signature does not match any signatures in the library. The signature library may include signatures for closely overlapping signatures, so that the likelihood of there being no close match is reduced. Or the platform may simply move to another nearby location and attempt again to find a matching signature.

The signature of a patch is defined in terms of the number of events that would be captured by viewing the patch using an event camera, and moving the sightline of the event camera through each of a predetermined set of intervals on a predetermined trajectory. The number of events recorded in each of the set of intervals is recorded, as is explained in further detail below. The data can be conditioned, for example by normalising, to produce the signature for the patch being viewed. The conditioning of the data is used to reduce the dependency of the histograms on the actual event camera used to record the data, as well as on the operating parameters used for the camera.

Event cameras are a specific class of camera. Event camera sensors comprise a number of pixels, as would be found in standard camera sensors, but the event camera pixels do not report the intensity to which they are exposed, but the rate at which it is changing. A number of event cameras are known. One exemplary type of event camera comprises a detector with an array of pixels. An image is projected onto the array. Impinging photons are registered as an electrical signal, and the rate at which the photons are arriving is noted. Each pixel in the event camera detector also has a reference level, and a threshold. The photon arrival rate is equivalent to the scene intensity of the point corresponding to this pixel’s position on the detector array. When the photon arrival rate of a given pixel differs from the reference level by more than the threshold amount, the pixel reference level is reset to the current intensity value, and the pixel sends an output to the camera computing core to register the change at this pixel’s location. In this way, an asynchronous record can be kept, by the computing core, of the degree to which each area of the viewed scene is varying its intensity. In some cases, the pixel may alternatively or additionally send an output to the camera’s external interface.

Standard framing camera pixels have a fixed capacitive well size and accumulate charge over a whole staretime. However each pixel has a finite storage capacity. If the rate of arrival of the photons is high enough, the pixel will saturate. That saturation limits the dynamic range of such cameras. Event cameras such as that described above do not have this limitation and have a significantly higher dynamic range than standard cameras. Event camera pixels typically fire and are registered in a period of the order of a millisecond; and up to of the order of 100 million events per second can be generated. Detector arrays for event cameras of size up to around one megapixel are available.

Figure 2 is a schematic illustration of a platform 200 flying over terrain 250.

Platform 200 includes an event camera 210 that faces towards the ground and has a field of view 300 that is schematically illustrated in more detail in Figure 3. In Figure 3, the field of view of the event camera covers a patch of terrain 300 illustrated by the box drawn in large-dashed line. In the following description, the ‘upward’ direction as seen in the Figure is referred to as ‘north’, with other directions referenced accordingly. A number of exemplary features are present in the field of view: a building 260 is located in a field 270; a stream 280 runs through the south-western corner of the field; a road 285 runs across the southeastern corner of the field; and there are a number of bushes 290 in the north eastern corner of the field of view. As the platform 200 flies over the terrain, it moves the sightline of the event camera 210 in a predetermined trajectory. In this case the predetermined trajectory is selected so as to define a circle 220 on the ground. As the sightline moves, the field of view of the event camera changes. A number of intervals in the circle can be selected. In the present example, each interval is selected to be 1 ° of the sightline rotation, and there are therefore 360 intervals in the sightline rotation.

As the sightline moves through one interval, the intensity recorded by some of the pixels of the event camera will change. Where the change in intensity is greater than the threshold value for any one pixel, that pixel will record an event. Two exemplary pixels are shown schematically at reference numerals 20 and 30 in Figure 3 close to the building 260. Pixel 20 is immediately south of building 260, whilst pixel 30 is south and east of the building 260. When the sightline moves such that these pixels image ground that is incrementally further north in one interval (as indicated by the arrows in Figure 3), pixel 20 will record an event as its view changes from one of field to one of building. Pixel 30 will not record an event, as it is too far to the east of building 260 for its view to change, and it will only view the field 270. The intensity recorded at pixel 230 will therefore not change by an amount as great as the threshold value. In contrast, when the sightline moves such that these pixels image ground that is incrementally further west, neither pixel will record an event, since both still view the field 270. An event camera controlled such that its sightline moves relative to the terrain from a particular point will therefore record a different number of events in dependence on the direction in which the sightline is moved. Sightline motion in a direction perpendicular to a linear image feature will typically cause far more pixels to be triggered than sightline motion parallel to the same feature.

Figure 4a is a graphical illustration showing the number of events that would be recorded by an event camera when its sightline is moved so as to define circle 220 on the ground. The normalised number of events recorded is illustrated as distance away from the centre of the chart at the relevant angular interval. It will be seen that the illustration shows a general, background level of events at around 10% to 20% of the peak for each one degree interval. The background level may for example result from intensity variations observed within the boundaries of the features described. In addition eight relatively narrow spikes are visible, and two relatively broad peaks are visible. A group of four spikes 410 align with the north, south, east, and west directions at approximately 0°, 90°, 180°, and 270°. It will be appreciated that these spikes result from the large number of events that will be recorded when the sightline is moving perpendicular to the walls of building 260. Similarly, spikes 420, 430 result from movement of the sightline perpendicular to the edges of the field. Spikes 420, aligned approximately along the north east direction at 65°, and the south west direction, at 245°, result from movement of the sightline perpendicular to the field edge that is almost entirely within the field of view of the event camera. Spikes 430, which are perpendicular to spikes 430, result from movement of the sightline perpendicular to the field edge that is only partially in the field of view. As a result spikes 430 are relatively smaller than spikes 420. The broad peaks 440 arise from movement of the sightline perpendicular to the stream 280. These peaks are broader than spikes 410, 420, and 430 because the streams are not as straight as the edges of the field or the building. The small peaks 450 arise from movement of the sightline perpendicular to the road 285. These peaks are sharp, because the road is straight, but small, because only a short length of the road is present in patch 250. The general background level of events is due to features in the terrain such as bushes 290, which will result in a uniform number of events being recorded regardless of the direction of movement of the sightline.

In practice the spikes described above will have widths which are partially governed by the variation of angles a feature may follow, such as the stream 280. The spikes will also have finite widths because sightline motion in a range of angles can cause a pixel to traverse from one feature to another. Motion normal to feature boundaries will tend to cause more firings, so such boundaries will cause peaks, but the sharpness of those peaks is determined by a range of factors, such as the pixel trigger threshold and the radius of the circle in which the sightline is moving. It is also intended that the sharpness of the peaks may be modified by the histogram conditioning process. For example, the conditioning process may be used to sharpen the response of the system (in terms of numbers of events) which occur as the event camera sightline is moving perpendicularly to a line in the scene.

Figure 4b is an alternative representation of the data illustrated in Figure 4a. In Figure 4b, the data for the first half of the rotation of the sightline around circle 220 is illustrated in the form of a histogram. The histogram shows the number of events recorded in each of the 1 ° intervals from 0° to 180°, normalised to the range between zero and one. This data is one example of a signature that can be used to identify patch of terrain 300, and therefore the location of the platform 200, using a library of such signatures stored in a memory store on the platform 200. In this example, only the data from 0°to 180° is used because, as a result of the symmetry of the circular sightline motion, and where the nutation radius is small compared to the image size, the remaining data very nearly repeats this. This can be seen in Figure 4a. This is because it does not matter whether the intensity change at a particular pixel is an increase or a decrease, only that the absolute value of the change is greater than the threshold level.

The library of signatures for a region can be constructed in a number of ways. The patches in the region are first defined, with a specific shape, size and location. The library can be constructed using a platform having both an event camera with an appropriately movable sightline, and an independent method of determining its location, such as the global positioning system GPS. The event camera can then be used to sense the signature of each patch defined in the region, using its GPS navigation to determine its location, and navigate between patches. The parameters used to capture the event camera data can also be recorded. Such parameters might include the radius of the sightline rotation; the height of the platform above the ground when the data is captured; and the threshold value of intensity change above which an event will be captured.

As will be understood, the presence of shadows on the ground may affect the signature for a patch. This effect can be mitigated by determining and sensing signatures in the long wave infrared (LWIR) waveband. At LWIR wavelengths, the effect of sun shadows is usually very small compared with the inherent intensity variation within a scene. A further clear advantage of operation in the LWIR waveband is that navigation will be available at all times, rather than only during daylight hours. Event cameras operating in the LWIR waveband are currently not commercially available but are being developed.

The library can also be created by processing standard aerial imagery of a region. The aerial image data for each patch can be processed to determine what events would be recorded by an event camera moving its sightline in the predefined trajectory. Processing of readily available aerial imagery can be done by known algorithms, such as ‘ESIM: an Open Event Camera Simulator’, by Henri Rebecq, Daniel Gehrig, and Davide Scaramuzza, published in the Proceedings for the 2nd Conference on Robot Learning (CoRL 2018), Zurich, Switzerland.

The patches for the signatures stored in the library should be quite small, for example squares of side length of the order of tens or hundreds of metres, and preferably based on the finest available spatial resolution. Actual limits on resolution, sensitivity, and image size which would be required for the library data, will be dependent on a number of factors, including the type of event camera used on the platform, its optics, sensitivity, waveband, and the required accuracy of location estimates.

It is possible, where, for example, a platform is to operate a number of missions in different parts of a region, to generate an overall database of signatures for the region, and to create a library from the overall database for each particular mission. The particular library developed for any one mission can then be adapted, from the database, for the needs of that one mission. The database can be created using the smallest reasonably possible patches. The size may be limited by, for example, the resolution of satellite imagery, if that is being used to generate the signatures, or by the characteristics of an event camera being used to generate the signatures. A library of larger size patches can be created from smaller patches since the signature of a larger patch can be derived from the signatures of its constituent smaller patches. In a simple case, where the signature is the raw number of events recorded as the sightline of the event camera moves, the signature of the larger patch will be the sum of the signatures of its constituent patches. The library provided to the platform for any one mission can therefore comprise signatures for relatively larger patches generated from the database of smaller patches.

In some cases, the threshold level of the event camera can be actively controlled so as to ensure that the number of events recorded for each patch is roughly constant. Ensuring the number of events remains roughly constant is likely to facilitate discrimination between patches across different types of terrain, rather than selecting a single threshold level for a large area. The overall database may then record both the number of events in each bin of a histogram and the relative intensity of each of those events. The signatures for larger patches can then still be derived from the signatures for smaller patches by removing lower intensity events when summing the signatures, to ensure an approximately constant number of events.

Once constructed, the library of signatures is provided to the platform and stored in a memory store on the platform. The library can include the signatures and the characteristics defining how the signatures were obtained, such as the size of each patch, the separation of the patch centres, the shape of each patch, and the resolution of the camera obtaining the signature. The signatures will also have characteristic features, which can include the following items:

1 . The trigger threshold to which the event camera pixels are set.

2. The ratio of the minimum number of events in any one bin to the maximum number of events in a bin.

3. The mean of the number of events in each of the bins in the histogram.

4. The standard deviation of the number of events in each of the bins in the normalised histogram. 5. The angle, relative to north, at which the bin capturing the maximum number of events lies.

6. The angle, relative to north, at which the bin capturing the minimum number of events lies.

7. The angle at which the weighted centroid bin lies. The weighted centroid is defined as where N_a is the number of events captured in the interval at angle a, and the sum is over angles between 0° and 180°.

8. The angles between any pair of items 5 to 7 above.

9. The angle between the bin capturing the largest number of events, and the bin capturing the second largest number of events.

10. The ratio of the histogram standard deviation divided by the histogram mean.

The library entry for each signature can include each of these items. A sorted reference index for the library is then created, so that a signature can be retrieved from the library using one of the characteristic features. For example, it is then possible to directly retrieve from the library those signatures which have their maximum number of events recorded within a certain tolerance of 63° from north. Typically the index may include around ten characteristic features. The characteristic features may be sorted in a number of different ways so as to facilitate searching the index. The signature library stored on the platform thus includes the signatures themselves, their characteristic features, and the index.

Use of the index makes searching the library significantly quicker. For example, a number of candidate patches can be identified quickly as those best matching each of the characteristic features of the navigation patch signature as measured by the platform. A bin-by-bin comparison, can then be made to identify the actual best match, for example, by calculating the correlation coefficient between the histogram of the flown and candidate patches. The ability to use the sorted index for each metric to list all of the library patches which lie within a specified tolerance of each of the metric values for a given patch provides a means to rapidly produce a small subset of the patches in the library against which the candidate patch can be checked. In most cases multiple candidate patches will appear in the nearest match groups of more than one metric. These may be checked first, thus minimising the time to find the correct patch even further. A number of different search algorithms can be constructed that make use of this advantage.

One example of a search algorithm that can be implemented on a processor using the method is schematically illustrated as a flow diagram in Figure 5. In this example the processor has been provided with an indexed signature library. The index comprises a number of lists of the values of characteristic features for each of the signatures in the library, each value having a unique identifier. The unique identifier can be a simple reference number, or any string that serves to uniquely identify the characteristic feature and associate it with a particular patch.

A first list of the values indexes the values of the characteristic features by their unique reference number. A second list orders the values of the characteristic features from smallest to largest and provides a rank to each characteristic feature. A third list provides a list of the ranks of each characteristic value ordered against a discretised list of potential values. The discretised list is created by dividing the difference between the maximum and minimum of the characteristic values into a number of discrete values at uniform intervals. Each discrete value is then associated, in the list, with the lowest characteristic value in the library that is larger than the discrete value. By way of example, for a characteristic feature that has values between a maximum of 6.8, and a minimum of, say 1 .5, the interval may be selected to be 0.000530, so that there are 10,000 discrete values in the list. The index also includes the maximum and minimum value of each characteristic feature, and the location associated with each of the signatures in the library.

In a first step 510 a sensed signature is provided to the processor for searching against the indexed signature library. Any conditioning, such as normalisation, has been performed before the search algorithm commences. At a second step 520, a characteristic feature of the sensed signature is determined. For example, the sensed signature may be analysed to determine the ratio of the minimum number of events in any one bin to the maximum number of events in any one bin.

At a third step 530, the discretised list is interrogated. The minimum library value for the characteristic feature is subtracted from the sensed value, and this difference is divided by the interval between the discrete values on the list. The integer values immediately above and below the result provide the positions on the list of the discretised value immediately above and below the sensed values. The ranks of these values can be identified from the list of discretised values.

At a fourth step 540, the ranks obtained from the discretised list are used to interrogate the rank index so as to determine the unique identifier for the characteristic values associated with those ranks.

At a fifth step 550, the unique identifiers obtained are used to interrogate the identifier index so as to determine the actual values of the characteristic values in the library, and the positions of the patches thus identified. Once this is completed, the determined value of the characteristic feature for the sensed signature has led to the identification of two library patches having values for that characteristic feature that bracket the determined value.

At a sixth step 560, it is determined whether or not the values of all the characteristic features of the sensed signature have been determined. If there remain some features for which the values are undetermined, the process repeats steps 520 to 550. If the values of all the characteristic features of the sensed signature have been determined, steps 520 to 550 will have generated a number of candidate patches, each of which are associated with one of the library values bracketing the sensed value for a particular feature. Thus the number of candidate patches will be up to twice the number of characteristic features stored in the library for each signature. Notably it may be less than this number because some characteristic features may of course identify the same candidate patch.

At step 570, a score is calculated for each of the candidate patches. The score enables a determination to be made of which candidate patch best matches the sensed signature. The score can for example be based on how similar the values of the characteristic features for the candidate patches are to the corresponding value for the sensed patch. In this case the score may be the sum of the squares of the differences between the values of the candidate patches and the corresponding value for the sensed patch, with the candidate patch with the lowest score being identified as the closest match. Such a score could also be used to determine a weighted mean of the locations of each of the patches (the lowest scores being given the greatest weight) which could be used to identify the location of the platform.

The positions of the candidate patches can optionally also be considered in the calculation of the score. For example, the score can be weighted using the sum of the mean square distances to the other candidate patches, so as to give greater weight to patches that are closer together, and less weight to outlier patches.

Once a first location has been determined in the above manner, the candidate patches identified in steps 520 to 550 can be used to support the determination of location in subsequent iterations of the method as further sensed signatures are received. The platform’s current motion, determined for example from an inertial navigation system, can be used to predict which patches the platform may be flying over at the time the next sensed signature is received. Thus, at step 570, each candidate patch identified in steps 520 to 550 can be used in this way to determine a further predicted patch. The sensed signature can then also be compared to the predicted patches. This can increase confidence in the location determination.

In one example illustrated in Figures 6a and 6b, the platform may be a helicopter 600 equipped with a navigation apparatus 650. The navigation apparatus 650 is shown in more detail in Figure 6b and includes a downwardfacing event camera 610 operable, for example, in the LWIR waveband. When the helicopter is in level flight, therefore, the event camera will face towards the ground. Figure 6a is an illustration of the navigation apparatus 650 mounted on helicopter 600. As is shown in Figure 6b, the navigation apparatus comprises an event camera 610 and a processor 640 including a memory store 645. A thin wedge prism 620 is mounted in front of the event camera. The thin wedge deflects the sightline, indicated in dashed line 630 in Figures 6a and 6b, of the event camera through a small angle. The centre point of the event camera field of view (in other words, where the sightline 630 projects onto the ground) is therefore offset slightly from immediately beneath the helicopter. The angular deflection provided by the prism is a small fraction of the angular width of the camera’s angular field of view. Prism 620 is mounted so that it is able to rotate in a complete circle about a vertical axis (when the helicopter is in level flight) that is illustrated by the dotted line 625 in Figure 6b. As the prism rotates, the centre point of the event camera field of view will define a circle on the ground beneath the helicopter 600. If the helicopter is hovering, the circle will be centred on a point immediately beneath the helicopter. In this example, the helicopter is also equipped with an independent means for sensing its orientation. For example, it may have a magnetic compass. The helicopter in this example is also equipped with an altimeter. Before take-off, the platform operator identifies the region in which the helicopter will fly. The platform operator provides a signature library for the region and stores the library in a memory store 645 of a processor 640 on the navigation apparatus 650.

The signature library in this example comprises square patches with side length 100m. Each square has sides parallel to the north-south and east-west directions, and the signatures are defined with the north direction being represented at 0°. The resolution used to define the signatures is matched to the resolution of the event camera 610 used on the helicopter 600. The patches for which signatures are defined in the library cover the region, and in this example there is a large overlap between the patches, with the centres of neighbouring patches being separated by only 10m (or 10% of the side length of each individual patch). Because the signature for each patch requires only a small amount of memory storage, such a large degree of overlap can still be accommodated for the size of region over which a helicopter might typically fly. The overlap reduces the likelihood that the helicopter will capture a signature that does not match a signature in the signature library. As described above, the signature library includes an index as well as the defining features listed above. The signature library can be constructed well in advance of actual operation of the helicopter in the region, and may have been derived from pre-existing aerial imagery of the region, or have been determined directly by prior flights over the region with a platform equipped with both an event camera and independent navigation means, such as GPS.

Once the signature library is provided to the helicopter memory store, the helicopter can begin to fly over the region. The event camera is configured to have a field of view corresponding, on the ground, to a square of at least twice the desired patch width, at the flight altitude of the helicopter. This can be achieved by zooming the event camera, or by altering the flight altitude of the helicopter, or by a combination of zoom and altitude change. The flight altitude is likely to be at least approximately known before take-off, so that the signatures in the library can be selected to be of approximately the right size. Where the field of view is larger than the 100m patch size, any pixels of the event camera field of view outside the patch size can be ignored in subsequent processing. Whilst over the region, the prism is rotated, and a signature recorded for each rotation of the prism. The recording of a signature for each rotation is started when the sightline is offset by the nutation radius to the north, defined as 0° in the present example. The event camera trigger threshold can be set in dependence on the type of terrain in the region and the characteristics of the library of signatures stored in the database. The trigger threshold may for example be determined as a parameter prior to flight, and passed to the helicopter with the library. The rate of rotation of the prism is not critical but is kept constant, and is preferably fast to enable an entire rotation’s worth of data to be captured whilst the helicopter remains relatively stationary, so that its forward motion during the time for collection of data for one patch is small or negligible compared to the length of the patch. The actual rate of rotation used will be dependent on various factors, including for example: the angular resolution of the image, the size of the field of view of the camera, the helicopter altitude, the size of the event camera’s pixel array, its maximum pixel processing rate, and other factors. A signature is captured by recording the number of events triggered in each chosen angular interval of a single rotation of the sightline. The nominal angular interval can be 1 °. Each recorded signature is processed to determine each of the defining features listed above. Because the sightline rotates in a circle, there will be a number of pixels in a border region of the field of view that will not be present for the whole sightline rotation. A first step of the processing in the present example is therefore to ignore events captured at these border regions. The index is then searched using the above-described algorithm to identify a matching signature, and the location of the helicopter is thus determined.

The inherent error in the determination of the position of the platform from a single match should be of the order of half the separation of the patch centres. Thus, if the centres of square patches of side length 100 m are separated by 10m, a nominal position error of 5 m might be expected. Once an estimate of location has been calculated, the quality of the match can be estimated using the correlation coefficient between the measured signature and the library signature. This correlation coefficient can be used to estimate the position error from the centre of the patch associated with the library signature. A series of such matches can be used to improve the position estimate, for example in conjunction with a Kalman filter, or a linear or polynomial regression weighted by the values of the correlation coefficients.

Various modifications to the above described embodiment are possible. For example, whilst in the above it has been described that the platform is a helicopter, the location determining method will also work on other airborne platforms. Where those platforms are not capable of hovering, or moving at a sufficiently low speed, account must be taken of the platform’s forward motion. Forward motion will result in there being parts of the event camera’s field of view that trigger events as a result of the camera’s forward motion changing the field of view, rather than because of the change of sightline. In many cases, particularly if the rate of rotation of the sightline is sufficiently rapid relative to the forward motion of the platform, the effect may be only small. However, it will also be possible to remove the effects of forward motion through additional processing of the raw data from the event camera, by compensating the raw angular intervals recorded so that the relative angle of the sightline to a reference angle on the ground is used to create the signature, rather than an angle determined simply from the prism motion. Alternatively or additionally, the event camera itself could be moved in a rocking motion to compensate for the forward motion of the platform such that the locus of the sightline on the ground always does describe a circle. Further alternatively, rocking the event camera sightline from left to right at an appropriate frequency whilst the platform is in forward motion can result in the sightline creating a locus of connected semicircles. Each semicircle can be used to derive a signature. It may be possible to use various other shapes of sightline motion.

It has also been described, in the above, to move the sightline of the event camera in a circle to capture the signature. For an event camera as described, with a thin wedge rotating prism rotating in front of its sensor to deflect its sightline, the sightline will define a circle on the ground if the event camera is facing directly downwards. However, for many platforms, it is usual for sensors to be forward-looking, at least to some extent. If the event camera does not face directly downwards, the sightline will not define a circle on the ground as the prism is rotated. Some transforms should be applied to the raw data in this case so as to account for the change. In particular, each interval in the signature should be referenced to an angle which is perpendicular to the motion of the sightline on the ground, and not to the angle of rotation of the prism. For a circle, these two quantities are the same, but this is not the case for other shapes, such as an ellipse, as may be the case if the event camera is forward-looking.

Where the event camera is forward-looking, there will be a slant to its sightline. However, based on the above considerations, it would be possible to determine with reasonable accuracy, from data captured by a forward looking event camera, the signature for a patch of terrain, which would have been created when viewing it from the vertical, using appropriate angular transforms. Alternatively it may be possible to modify the event camera sightline motion to compensate for any forward-looking angle and forward motion of the platform. Such a determination may not be possible for very large slant angles, or only possible with limited accuracy. Some degree of forward-looking is however possible. Similar considerations apply if the platform is not in level flight. Since an event camera could be used as a forward looking imager in a range of potential aircraft, including unmanned air systems and missiles, it will be noted that an event camera could fulfil multiple roles for the platform, including both imaging and navigation.

It has also been described in the above to use an event camera operating in the LWIR waveband. However, it would also be possible to use event cameras operating in other parts of the spectrum, such as in the visible region or medium wave infrared (MWIR). Many event cameras are commercially available that operate in the visible part of the spectrum. However, it would be necessary to account for the effect of shadows on sensed signatures in both the visible and MWIR wavebands. Whereas, in the LWIR waveband, sun shadows are likely to have only a small effect relative to inherent intensity variation in a scene, in the visible region, sun shadows may substantially alter the signature of a patch. They can create a substantial number of lines, all lying in the same direction and thus potentially giving the maximum signal at an angle which would otherwise be an insignificant part of the signature.

This effect can be accounted for, however, during data processing. The platform can be equipped with a clock, and a sensor such as a compass to determine its heading. Processing can then determine which angular intervals will be affected by the shadows. Features which lie in those intervals can then be ignored.

The effect of shadows can also be accounted for in the signature library by including the nominal shadow angle with the signature library. The shadow angle can be calculated by using the time of data collection, the position of the patch, and the orientation of the event camera capturing the data. Notably standard aerial imagery can be obtained with metadata indicating time of image capture as well as location and orientation. Strong peaks in the library signatures caused by shadows can then be ignored during the comparison of a recorded signature to the library signature.

Where an event camera operating in the visible waveband is to be used, it may also be desirable to include an illumination source, such as a laser, so as to enable operation of the navigation apparatus at night. It will also be understood that, whilst it has been described to use a generic event camera in the above, any type of event camera could be used. A review of event cameras is provided in Gallego et al., ‘Event-Based Vision: A Survey’, published in IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 44, No. 1 , pages 154 to 180, January 2022. There may be many different capture patterns for the data. The trigger threshold for each camera pixel, for example, could be set differently for different event cameras, so as to ensure that the number of triggers does not exceed the rate at which they can be handled by the relevant processor (either a processor embedded in the camera, or a platform processor dedicated to the processing of the event camera data). The trigger threshold can also be adjusted during operation of the navigation method, for example to ensure that the number of events recorded falls within a certain range or below a certain threshold. It may also be possible to use a standard framing camera coupled with suitable processing, although this would require significant processing power in order to operate. As a result an event camera directly outputting events from pixels responsive to changes in intensity is more likely to be used for embodiments of the present invention.

It will also be possible to use different means of moving the camera sightline, in alternatives to the above-described method of rotating a thin wedge prism in front of the event camera sensor. For example, the sightline could be relayed from the scene to the sensor via a mirror, tilting the mirror surface, and rotating the mirror so that the sightline on the ground describes a circle. In this case, the tilt of the mirror may also be controllable, which may enable compensation of forward platform motion, and enable the radius of the circle described on the ground by the sightline to be controlled. Alternatively, the event camera sensor itself may be nutated so as to move the sightline.

It is further noted that, whilst in the above it has been described to compile a specific library for a specific, known region in which the platform may be flying, the signatures can be combined in such a way that a single reference library can be used for multiple missions. It is expected that the library would not require large memory store (it can be estimated that the signatures for the entire world surface, in 100m square patches, would only require up to 4 Tb of storage), and so it is possible to derive smaller libraries from a reference library. Because of the low memory storage requirement, it may also be possible to provide multiple libraries to a platform if, for example, the platform may operate at different heights during any one mission, so that different size patches or different resolution patches might be useful in the library. In practice, a platform is most likely to start navigation with at least some idea of its position. For example, a missile when launched can easily be provided with its position, and similarly the position of an aircraft when taking off will be known. The library need then cover only a much smaller area. 4Tb of storage could, for example, hold a signature library for an area of a million square kilometres with square patches of side length 8m or less; or a signature library with larger but significantly overlapping patches; or multiple signature libraries with different size or resolution patch.

It may also be noted that the signatures for neighbouring patches may not be at all similar. However, where there is some overlap, there will be some correlation between patches, and this property may be used for location determination.

Whilst it has been described in the above to use patches that are square shaped, it will be understood that the patches themselves could take any shape. For example, the patches could be circular; or polygonal. It may not be trivially possible, however, to calculate the signature of a larger circular patch from a number of smaller circular patches.

Whilst it has been described in the above to measure one signature for each rotation of the event camera sightline, it will be understood that it will be possible in practice to measure multiple signatures in one rotation. Measuring multiple signatures in one cycle may obviate the need to store the signatures of multiple patches in the signature library, since multiple, overlapping patches can be analysed in one sightline rotation so as to determine multiple signatures. An event recorded in any one part of the event cameras field of view is simply added to the histogram for all the patches overlapping at that point. Such a technique reduces the memory capacity required for the signature library but increases the required processing time. However, since each rotation of the sightline is likely to take of order one second, processing time is not constrained. In any event it is not necessary for the position of the platform to be determined within a short period from the actual measurement of the signature. In most applications it is anticipated that a delay of several seconds from measurement of the signature to determination of position would be acceptable. Finally, it should be clearly understood that any feature described above in relation to any one embodiment may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. In particular it should be noted that in all of the above embodiments, the processor hardware may be built specifically for the purpose of controlling apparatus, or, particularly if the apparatus is to be operated in a missile or other aerospace or vehicle platform, may be firmware installed on a platform processor; or the processor may be a general purpose computer with software installed to control the apparatus as described above.

Claims

1 . A method for determining a location of a platform in a region, the platform comprising an event camera and a memory store; the event camera having a sensor comprising an array of pixels, each pixel being operable to output events, each event being a change in light intensity recorded at a pixel; and the method comprising the steps of: a) before platform motion in the region: i. providing a signature library, the signature library comprising a number of signatures, each of the signatures having an associated patch at a known location in the region, and each of the signatures representing the number of events that would be captured by viewing the associated patch using the event camera whilst moving the sightline of the event camera through each of a predetermined set of intervals on a predetermined trajectory; ii. storing the signature library in the memory store; b) during platform motion in the region, determining the location of the platform by: i. moving the sightline of the event camera in the predetermined trajectory whilst capturing sensed event data; ii. recording the number of events captured in each of the set of predetermined intervals in the predetermined trajectory to obtain a sensed signature; and iii. comparing the sensed signature to the signature library to determine the location of the platform in the region.

2. A method as claimed in claim 1 , wherein the predetermined trajectory is a rotation of the sightline of the event camera.

3. A method as claimed in claim 2, wherein the predetermined trajectory is a rotation through at least 180°.

4. A method as claimed in claim 2 or claim 3 wherein the predetermined intervals are angular intervals in the range between 0.1 ° and 10°, preferably 1 °.

5. A method as claimed in any preceding claim, wherein the platform is an airborne platform.

6. A method as claimed in claim 5, wherein the event camera faces the ground during the recording of the sensed event data.

7. A method as claimed in any preceding claim, wherein the step of providing a signature library comprises defining one or more characteristic features of the signatures, and indexing the signature library against each of the characteristic features.

8. A method as claimed in claim 7, wherein the step of comparing the sensed signature to the signature library comprises the step of determining the one or more characteristic features for the sensed signature, and identifying one or more candidate signatures in the signature library from the index of characteristic features.

9. A method as claimed in claim 8, wherein the step of determining the location of the platform comprises determining a score relating to the similarity of the sensed signature to each of the candidate signatures, and determining the location of the platform to be the location of the patch associated with the candidate signature having the score indicating the highest similarity to the sensed signature.

10. A method as claimed in any preceding claim, wherein the step of providing a signature library comprises the steps of: a) providing aerial imagery of the region b) dividing the aerial imagery into a plurality of image patches at each of the associated locations; c) determining the signature by simulating, for each of the image patches, the number of events that would be captured by viewing said each of the image patches using the event camera whilst moving the sightline of the event camera through each of a predetermined set of intervals on a p red ete rm i n ed traj ecto ry .

11. A computer readable medium having stored thereon a signature, which signature comprises information extracted from data recorded by an event camera; the event camera having a sensor comprising an array of pixels and being operable to output events, each event being a change in light intensity recorded at a pixel; the signature being determined by moving an event camera in a predetermined trajectory and recording the number of events captured in each of a set of intervals of the trajectory.

12. Apparatus for determining a location comprising an event camera, a memory store, and a processor, the event camera being arranged such that its sightline is movable; and the apparatus being configured to perform the method of any one of claims 1 to 12.

13. An airborne platform comprising the apparatus of claim 12, wherein the event camera is arranged such that its sightline is movable relative to the platform.