BR112019009599A2 - systems and methods for detecting flying animals - Google Patents

Abstract

a system for detecting flying animals, the system comprising: a detection module configured for detecting flying animals; a local processing module associated with the detection module and configured to remove background noise from data collected by the detection module; an analysis module configured to receive data from the local processing module and process said data to determine the presence of flying animals and classify such flying animals by species; and an impediment configured to repel one or more flying animal species based on the classification of detected flying animals.

Description

"SYSTEMS AND METHODS FOR DETECTING FLYING ANIMALS" [0001] The present invention describes systems and methods for flying animals.

[0002] Flying animals can pose a significant threat to aircraft, wind turbines and other high-value facilities. A bird strike on an aircraft could result in the aircraft crashing with a resulting loss of life or, as a best-case scenario, it could result in damage to the aircraft that requires the aircraft to remain on the ground while it is inspected and repaired. Likewise, a bird collision on a wind turbine blade could damage the blade making the wind turbine inoperable.

[0003] In addition to potential damage to the facility or aircraft, facility operators may be fined if found responsible for the injury or death of particular species of birds or bats. It is also desirable to avoid injury or death to protected species of birds or bats from a moral and environmental perspective.

[0004] It is in this context that the present invention arose.

SUMMARY [0005] One aspect of the present invention provides a system for detecting, classifying and preventing birds and bats, the system comprising: a detection module configured to detect birds and / or bats; a local processing module associated with the detection module and configured to remove background noise from data collected by the detection module; an analysis module configured to receive data from the local processing module and process said data to determine the presence of birds and / or bats and to classify such birds and / or bats by species; and an impediment configured to repel one or more species of birds and / or bats based on the classification of detected birds and / or bats.

[0006] Local processing of data recorded for

Petition 870190064264, of 07/09/2019, p. 4/20

2/12 generating a subset of the data target reduces the transmission time for sending core data for analysis by a centralized analysis unit. The claimed configuration allows multiple data streams to be collected and processed in real time, to detect, classify and prevent birds and / or bats from a target area.

[0007] The system is capable of detecting objects flying up to 200 meters away from each detection module within a target environment. Larger objects (such as birds having a wingspan> 1 meter) can be detected up to 400 meters away from a detection module. Tests have shown that the system is capable of detecting up to 90% of birds during the day, with an accuracy of 80% of flight path prediction in a range of 100 meters from a target object. The system was also tested to show that only 2-3 false positive detections are found per day (based on an annual average).

[0008] A further aspect of the invention provides a method for detecting objects in image data in real time, the method comprising: i) capturing at least two image streams from an environment; ii) convert each associated frame of each image stream into binary form; iii) compare associated binary frames of each image stream to identify non-environmental objects; iv) apply a threshold to each table to divide data into segments; and v) determine the size and two-dimensional position of each non-environmental object.

[0009] A further aspect of the invention provides a method for classifying objects into real-time image data, the method comprising: i) combining a plurality of time-encoded image frames to derive a composite image; ii) use the composite image to determine Cartesian coordinates for one or more non-environmental objects; iii) group individual non-environmental objects; iv) apply Kalman filtration to each group of non-environmental objects; v) classify each non-environmental object by animal species; vi) predict flight paths from

Petition 870190064264, of 07/09/2019, p. 5/20

3/12 object; and vii) providing a probability of collision with a predetermined target.

[0010] An additional aspect of the Invention provides a method of predicting flight paths of birds or bats, the method comprising: I) comparing an instantaneous subset of data for a species of bird or bat identified against a stored subset of data for the same species of bird or bat; li) extracting flight path data from the stored subset of data; ill) extrapolating flight path data from instant subset data based on flight path data from the stored data subset; and iv) generate one or more flight path predictions.

[0011] A further aspect of the invention provides a system for detecting, classifying and preventing birds and / or bats, the system comprising: a detection module configured to detect flying objects entering a first detection detection zone; an analysis module to determine whether a flying object within the first detection zone is a bird or a bat; and a first impediment configured to activate following determination that a flying object within the first detection zone is a bird or a bat and that bird or bat enters a second detection zone, where the second detection zone is closer to a predetermined reference point than the first detection zone.

[0012] Such a system is optimized to detect and classify flying objects at a distance from a predetermined reference point or object of interest. By the time the object approaches the second detection zone, the system will have determined whether the flying object is an animal, that is, either a bird or a bat. If the object is determined to be an animal and the animal enters the second detection zone an impediment is activated. When determining whether the object is an animal, while the animal is in the first detection zone, false positives are reduced and the

Petition 870190064264, of 07/09/2019, p. 6/20

4/12 impedance is only activated when necessary to prevent an animal far from the predetermined reference point. By only activating the impediment when necessary, unnecessary inconvenience to neighbors close to the predetermined reference point is minimized.

[0013] The system may further comprise a third detection zone that is closer to the predetermined reference point than the first detection zone and the second detection zone.

[0014] The system can also comprise a second operable impediment element to activate upon approaching the third detection zone by a bird or bat.

[0015] The first impediment can be visual and the second impediment can be audible.

[0016] Use of a visual impediment before an audible impediment is advantageous, as a visual impediment is likely to be less intrusive to neighbors than an audible impediment. Audible impediment is activated as a last resort to prevent animals that have not been impeded by visual impairment.

[0017] The first detection zone can be 500 to 1000 meters from the predetermined reference point. The second detection zone can be 250 to 500 meters from the predetermined reference point. The third detection zone can be less than 250 meters from the predetermined reference point.

FIGURES [0018] The invention will now be described by reference to the following Figures:

[0019] Figure 1 shows a simplified view of a system according to modalities of the invention;

[0020] Figure 2 shows a camera-based system according to the modalities of the invention;

[0021] Figure 3 illustrates one embodiment of the invention

Petition 870190064264, of 07/09/2019, p. 7/20

5/12 using a Kalman filter and an artificial neural network;

[0022] Figure 4 illustrates a particular embodiment of the invention of Figures 1 and 2; and [0023] Figure 5 illustrates a system for detecting bats.

DESCRIPTION [0024] A simplified version of a system 10 according to the present invention is shown in Figure 1. The system comprises a master unit 12 with which one or more pairs of slave units 14 are in communication. Slave units 14 are arranged in pairs, with each pair defining a detection module. Each slave unit 14 comprises a motion detector 16 configured to detect and record motion in a target environment, a central processing unit (CPU) or a microprocessor 18 to process recorded data and a memory 20 to store both recorded and processed data . Each slave unit is connected to the master unit via appropriate communication 22, such as Ethernet, WiFI, Bluetooth® or GSM. Each motion detector associated with a slave unit 16 is focused on the same surveillance area around an object of interest.

[0025] In the modality illustrated in Figure 2, each motion detector 16 includes a camera that could be directional cameras, spherical cameras, thermal cameras or a combination of them. Each camera records image data for the same target area. Recorded image data is stored in memory 20 associated with each respective slave unit 14 and processed by the CPU or microprocessor 18 to convert each frame of the image data into binary format. Each binary image is divided into pixels that either have a white value or a black value. A first threshold process is used to identify whether each pixel should be set to white or black. Each unit

Petition 870190064264, of 07/09/2019, p. 8/20

6/12 slave 14 uses dedicated software to process image frames registered by the associated camera 16.

[0026] The binary image data of each slave unit 14 are compared to identify any differences between captured frames. Threshold is then applied to each frame to segment the binary image. Any constant background elements are erased and the threshold is applied again to each frame to clearly identify only that portion of each frame that is of interest. In other words, when the number of frames stored in relation to each record is n> 2 or n = 2, the absolute difference value between the frames can be calculated to ensure that only moving objects within the captured frames are retained for later analyze. The resulting image is then erased using Gaussian erasure or Gaussian smoothing to remove any discontinuities that entered the image through the first threshold process. The second threshold step is then applied to remove unwanted noise from the resulting image and to reduce the number of details that are not related to moving objects.

[0027] A contour is applied to the image after the completion of the second threshold step. The contour is analyzed by the slave unit, to determine whether the contour size measured in pixels falls between the upper and lower pixel thresholds. If the contour size is less than a lower threshold or higher than the upper threshold, the image is not yet analyzed, as the object is assumed by the system to not be a bat or a bird. Since the contour size is between the lower and upper thresholds, the location of the object is calculated using the centroid method. The centroid of an image (object contour) X is calculated by the well-known momentum method. Centroide:

f__Ao M ₀₁ íx, VI = {-----, -----} ¹ % ₀ , W [0028] Central moments of digital image f (x, y) are

Petition 870190064264, of 07/09/2019, p. 9/20

7/12 defined as:

= ΣΣ χ'χύζχ, γ) x y [0029] The calculated contour size of the object and its position are then sent to the master unit for further processing.

[0030] Master unit 12 combines data received from each of the pair of slave units 14 associated with a detection module. The combined data is analyzed to remove all non-bird / bat objects. Master unit 12 determines the contour size, speed, position and x, y, z coordinates of an object using the Cartesian 3D system. 3D coordinates are determined using the 2D image data sent from the slave unit pair 14 and analyzing the absolute difference between the position of the detected object as identified in the 2D image data. All objects identified in the 2D image data are grouped based on the mean quadratic error (mse) between positions:

1

- £ 4 ώ and their true positions are calculated using a Kalman filter applied to a plurality of frames recorded over a period of time. Where a and b in the equation define the frames for each pair of slave units 14.

[0031] The analyzed data is used to calculate a predicted flight path of each registered bird / bat object and to calculate the probability of a collision with an object of interest, that is, aircraft, wind turbines or other high-value installation . In the event that a bird / bat object is determined to be likely to collide with an object of interest or enter a predetermined target zone, the unit

Petition 870190064264, of 07/09/2019, p. 10/20

8/12 master is configured to automatically trigger an alarm to warn of the risk of a collision with birds / bats and / or implement preventive countermeasures to repel birds / bats in the distant vicinity of the object of interest.

[0032] All recorded data relating to birds / bats are stored in a centralized database along with calculated flight paths, time of day and other relevant information, such as audio recordings.

[0033] The stored data is used to classify the type of bird / bat that has been recorded by species. This classification is used to select an appropriate impediment countermeasure for the registered bird / bat type. The stored data is also used to improve the accuracy of flight path prediction and collision probability calculations by applying an appropriate algorithm to a subset of data relating to a particular bird / bat species.

[0034] To minimize errors in calculated position and to predict the flight path of a bird / bat object, an extended Kalman filter combined with a neural network is used. The following algorithm is used to predict the flight path of a bird / bat object:

[0035] Where:

x, y - Cartesian coordinates determined by cameras r, θ - polar coordinates of the detected object.

[0036] The Jacobian of the measurement equation:

+ 1) íA (íí + D + í) t i) dt 0 o

1

0 days

1J

Petition 870190064264, of 07/09/2019, p. 11/20

9/12 [0037] The updated measurement equation ϊ%

Jacobian matrix:

I _ cos (%) 0 sin ($ &) 0 ^& | _x ~ sin (0fc) A; ü cos (^) / F _fc Oi [0038] Figure 3 shows the combination of the Kalman filter with the artificial neural network. The polar coordinates determined r, θ of a series of previous positions of the object are applied to the artificial direct neural network (ANN) of twenty-five neurons in a first layer and ten in a second layer. ANN's task is to predict the bird tracking vector in the range of detection zone 24. The accuracy and precision of the bird tracking prediction depends on the number of previous points applied to the network and increases with the observation time. The neural network is additionally provided with information about the size and / or speed of the detected object. During the first week of the work of the system, neural network weights and deviations are tuned with a Levenberg-Marquardt backpropagation algorithm (trainlm) to adopt the behavior of certain sapiens that occur in a given geographical area. As shown in Figure 4, the detection zone comprises a first detection zone 24a approximately 500 to 1000 meters from an object of interest. When a flying object enters the first detection zone 24a the system determines whether the flying object is a bird or a bat, as described above, and its relative size, predicted position and flight path. As the bird or bat enters the second detection zone 24b the system activates a first impediment which can be a visual impediment, such as

Petition 870190064264, of 07/09/2019, p. 12/20

10/12 strobe lights. If the bird or bat then subsequently enters a third detection zone 24c that is less than 250 meters from the object of interest, the system activates a second impediment that can be an audible impediment, such as a siren.

[0039] Although the invention has been described with reference to slave units 14, each comprising a motion detector 16, which is a camera, it will be appreciated that other devices can be used as motion detectors, in particular radar devices or acoustic. Doppler radar, noise radar or frequency modulated continuous wave radar devices or directional microphone or ultrasound recording devices can be used in place of one or more of the cameras in a slave unit 14.

[0040] Multiple detection modules comprising a combination of cameras, radar and acoustic devices can be connected to master unit 12. In one example, a detection unit comprises two slave units 14 with each slave unit 14 comprising a stereoscopic camera to calculate the size and 2D position of an object. In another embodiment, a slave unit 14 comprises a stereoscopic camera and the other slave unit comprises a radar unit for detecting the height of an object or, at least, an acoustic device for recording bird / bat calls to assist in the classification of objects.

[0041] In order to improve the local analysis of data recorded by each slave unit 14, an environmental monitoring module can be associated with each of the slave units. The environmental monitoring module comprises a series of different sensors to monitor environmental factors, such as temperature, humidity, wind direction, wind speed, atmospheric pressure, precipitation volume, atmospheric gas content, for example. Each of these factors is taken into account by the software that processes recorded data to calculate

Petition 870190064264, of 07/09/2019, p. 13/20

11/12 more accurately the size and position of registered objects.

[0042] The ability of each slave unit 14 to communicate with the master unit is critical for real-time detection, classification and deterrence of bird / bat objects. To protect against communication failure, each slave unit 14 communicates with master unit 12 using various communication protocols including: GPS, radio / satellite, fixed-line communication, IR lighting, Ethernet or GSM, for example.

[0043] For a wind turbine application, detection modules and environmental sensors are placed on each turbine to provide protection for an entire wind farm installation. Upon detection of a flying object by a detection module, the associated data is processed to calculate the size and position of the object in two dimensions. The resulting data is sent to master unit 12 for object classification and analysis of the Cartesian position and the predicted flight path. The predicted position and flight path are used to calculate the probability of collision with the wind turbine blades. If the detected object is classified as a protected bird or bat species, the master unit either instructs the wind turbine blades to slow down or stop or issues an impediment to encourage specific bird or bat species to move away from the wind turbine.

[0044] For an airport installation, the detection modules and environmental sensors are placed around the airport, so that at least one detection module covers each segment of the air space surrounding the airport. Upon detection of a flying object by a detection module, the associated data is processed to calculate the size and position of the object in two dimensions. The resulting data is sent to master unit 12 for object classification and analysis of the Cartesian position and the predicted flight path. The predicted position and flight path are used to calculate the probability of a

Petition 870190064264, of 07/09/2019, p. 14/20

12/12 collision with an aircraft or a ground installation or the probability that the object will enter a predetermined airspace. Master unit 12 is configured to generate an alarm to alert ground control of bird / bat collision threats and implement deterrent countermeasures to repel birds / bats from the area.

[0045] In relation to a system for detecting bats as shown in Figure 5, the motion detector is a sound sensor for measuring an ultrasonic signal emitted by bats during flight. Such a system comprises a plurality of ultrasonic sensors 26 placed around each slave unit, typically spaced around a perimeter of one meter. The recorded ultrasonic signal is sampled at a frequency of 250 kHz and a 16-bit resolution. The frequency spectrum is determined by the Fourier Transform in the 100 ms time window.

[0046] Each slave unit 14 is equipped with a relevant sound sensor. The data is processed in real time by the slave unit 14 (simultaneously with the image analysis). For master unit 12, only information about particular frequencies occurring in the signal is transferred. If the bat detection is confirmed by the series of observations, the time domain signal is transferred to master unit 12 in order to calculate the actual bat position. The position is calculated based on the phase shift between the signal and the difference in arrival time.

[0047] It will also be appreciated by those skilled in the art that, although the invention has been described by way of example with reference to various modalities, it is not limited to the disclosed modalities and that alternative modalities could be constructed without departing from the scope of the invention as defined in the appended claims.

Claims (21)

1. System to detect, characterized by the fact that the system comprises: a detection module configured to detect flying animals; a local processing module associated with the detection module and configured to remove background noise from data collected by the detection module; an analysis module configured to receive data from the local processing module and process said data to determine the presence of flying animals and to classify such flying animals by species; and an impediment configured to repel one or more species of flying animals based on the classification of detected flying animals. 2. System according to claim 1, characterized by the fact that the detection module comprises at least a first motion recorder and a second motion recorder. 3. System according to claim 2, characterized by the fact that a memory is associated with each of the first motion recorder and the second motion recorder and is configured to store motion data detected by the first motion recorder and the second motion recorder. 4. System according to claim 2 or claim 3, characterized by the fact that at least one of the first motion recorder and / or the second motion recorder is a camera. 5. System according to claim 4, characterized by the fact that at least one of the first motion recorder and / or the second motion recorder is a camera Petition 870190044307, of 05/10/2019, p. 34/39 2/5 stereoscopic, directional camera, spherical camera or thermal camera. 6. System according to claim 3 or claim 4, characterized by the fact that at least one of the first motion recorder and / or the second motion recorder is a radar. 7. System according to claim 6, characterized by the fact that at least one of the first motion recorder and / or the second motion recorder is a Doppler radar, noise radar or frequency modulated continuous wave radar. 8. System according to claim 2 or claim 3, characterized by the fact that at least one of the first motion recorder and / or the second motion recorder is an acoustic device. 9. System according to claim 2 or claim 3, characterized by the fact that at least one of the first motion recorder and / or the second motion recorder is a directional microphone or an ultrasound recorder. 10. System, according to claim 3, characterized by the fact that the local processing module is configured to compare data recorded by the first motion recorder and the second motion recorder to identify constant background noise. 11. System according to claim 1, characterized by the fact that the local processing module is further configured to apply a threshold to the data recorded by the first motion recorder and the second motion recorder to segment the data and obscure background noise . 12. System, according to claim 1, characterized by the fact that the local processing module is still configured to calculate the size of an object and its position in space Petition 870190044307, of 05/10/2019, p. 35/39 3/5 two-dimensional · 13. Method for detecting objects in image data in real time, characterized by the fact that it comprises: i) capture at least two image streams from an environment; ii) convert each associated frame of each image stream into binary form; iii) compare associated binary frames of each image stream to identify non-environmental objects; iv) apply a threshold to each table to divide data into segments; and v) determine the size and the two-dimensional position of each non-environmental object. 14. Method for classifying objects in real-time image data, characterized by the fact that it comprises: I) combining a plurality of time-encoded image frames to derive a composite image; ii) use the composite image to determine Cartesian coordinates for one or more non-environmental objects; iii) group individual non-environmental objects; iv) apply Kalman filtration to each group of non-environmental objects; v) classify each non-environmental object by species of animal; vi) provide object flight paths; and vii) provide a probability of collision with a predetermined target. 15. Method according to claim 14, characterized by the fact that it comprises the additional steps of: viii) generate an alarm if the probability of collision Petition 870190044307, of 05/10/2019, p. 36/39 4/5 is greater than 80%; and ix) implement an impediment to repel classified animal species. 16. Method for predicting flight paths of birds or bats, characterized by the fact that it comprises: i) comparing an instantaneous subset of data for an identified animal species against a subset of data stored for the same animal species; li) extracting flight path data from the stored data subset; iii) extrapolating flight path data from instant subset data based on flight path data from the stored data subset; and iv) generate one or more flight path predictions. 17. Method, according to claim 16, characterized by the fact that the method also comprises the step of: v) predict the probability of the animal colliding with a target object. 18. System to detect, classify and prevent animal species, characterized by the fact that it comprises: two or more image capture devices; a local processing unit for comparing and converting image frames captured by two or more image capture devices into binary format and determining the approximate size and two-dimensional position of objects of interest; a central processing unit for determining the three-dimensional coordinates and the predicted trajectory of the objects of interest and for classifying the objects of interest based on size, appearance and other identification characteristics; and Petition 870190044307, of 05/10/2019, p. 37/39 5/5 an impedance module configured to implement impediment countermeasures according to the object classification. 19. System, according to claim 18, characterized by the fact that it also comprises a radar to calculate the height of an object. 20. System according to claim 18 or claim 19, characterized by the fact that it further comprises one or more environmental sensors for measuring environmental conditions. 21. System according to claim 20, characterized by the fact that the one or more environmental sensors comprise one or a combination of temperature sensors, humidity sensors, wind direction sensors, wind speed sensors, temperature sensors atmospheric pressure, precipitation sensors or gas analyzers.