AU2004202924A1 - Method and apparatus for determining horizon angle and displacement - Google Patents

Description

112446150 1.

AUSTRALIA

Patents Act 1990 (Cth)

COMPLETE

SPECIFICATION

MONASH UNIVERSITY Invention Title Method Apparatus for Determining Horizon Angle and Displacement The invention is described in the following statement: Blake Dawson Waldron Patent Services Level 36, Grosvenor Place 225 George Street SYDNEY NSW 2000 Telephone: 61 2 9258 6000 Fax: 61 2 9258 6999 Ref: DGC 1368 3971 112446150 2.

METHOD APPARATUS FOR DETERMINING HORIZON ANGLE DISPLACEMENT FIELD OF THE INVENTION The present invention relates broadly to a method of determining horizon angle and/or displacement. The invention also relates generally to an apparatus for determining horizon angle and/or displacement. The invention relates particularly, though not exclusively, to the use of these methods and apparatus in the flight control of small unmanned air vehicles (UAVs).

BACKGROUND OF THE INVENTION Vision processing techniques lend themselves to many autonomous navigation and control tasks, but the usually high amount of processing that needs to be done requires a reasonably powerful computer, large-scale programmable logic controllers or ASIC's.

Technological advances and increased sales volumes continue to shrink the size, weight and cost of such computers, but still the electrical power and weight constraints of small UAVs militate against complex onboard vision processing systems. The risk of loss and damage to the equipment that comes with the nature of the missions that UAVs may be called upon to perform also requires a low-cost approach. Because the payload capacity of the airborne platforms are so small the vision processing is often done on the ground, using radio telemetry to retrieve the video. Devices sensing in the infra-red spectrum, using a small number of discrete infra-red sensors, are used in UAV and aerospace applications, for stabilising aircraft and satellites. There are also devices such as mechanical, solid-state and optical rate gyros that are used in inertial guidance systems in many aircraft. Many of these devices are too large, heavy and require too much power to be useful in a UAV context, but others are well suited to UAV applications and are being used for such. Some of the inertial guidance systems are not particularly accurate, suffering from inherent drift problems.

SUMMARY OF THE INVENTION According to one aspect of the present invention there is provided a method of determining horizon angle, said method comprising the steps of: taking an image of a horizon together with adjacent sky and earth portions; 112446150 3.

defining a predetermined area of the image which captures at least part of the horizon and the adjacent sky and earth portions; determining respective centroids of the captured sky and earth portions; calculating the gradient of an imaginary centroid line connecting the sky and earth centroids; and calculating the horizon angle from an inverse of the gradient of the imaginary centroid line.

According to another aspect of the invention there is provided an apparatus for determining horizon angle, said apparatus comprising: a camera designed to take an image of a horizon together with adjacent sky and earth portions; and processing means in communication with the camera and being configured to process the image whereby a predetermined area of the image is captured and respective centroids of the captured sky and earth portions are determined and the gradient of an imaginary centroid line connecting the sky and earth centroids calculated and the horizon angle calculated from an inverse of said gradient of the imaginary centroid line.

According to a further aspect of the invention there is provided a method of determining horizon displacement, said method comprising the steps of: taking an image of a horizon together with adjacent sky and earth portions; defining a predetermined area of the image which captures at least part of the horizon and the adjacent sky and earth portions; and calculating the area of the captured earth portion and deriving the horizon displacement from said calculated earth area and a coordinate or dimension of the predetermined area.

According to yet another aspect of the invention there is provided an apparatus for determining horizon displacement, said apparatus comprising: a camera designed to take an image of a horizon together with adjacent sky and earth portions; and processing means in communication with the camera and being configured to process the image whereby a predetermined area of the image is captured and an area of the captured earth portion calculated, the processing means deriving the horizon 112446150 4.

displacement from said calculated earth area and a coordinate or dimension of the predetermined area.

Generally the predetermined area is substantially circular.

Preferably the camera is a visible light video or still camera. Alternately, the camera is suitable for use in the infra-red spectrum. More preferably the camera includes a digital interface which communicates digitally with the processing means.

Preferably the processing means includes a microprocessor together with associated software being configured to process the image and calculate the horizon angle and/or derive the horizon displacement.

Preferably the step of defining the predetermined area involves masking the image with a predetermined boundary to define the captured sky and earth portions. More preferably the predetermined boundary is a circular mask which exposes the area within the circular boundary.

Preferably the method also comprises the step of classifying pixels of the captured image into sky class and earth class pixels. More preferably said classification step involves application of a thresholding criterion wherein sky class and earth class pixels are classified according to their colour in the visible spectrum.

Preferably the step of determining the respective centroids of the captured sky and earth portions involves totalling each of x 5 and ys coordinates for the sky class pixels and dividing by the number of pixels of the captured sky, and performing a like calculation for the earth class pixels, to obtain x and y coordinates for the sky and earth centroids

(X

9 Y and X, YE). More preferably the gradient of the horizon is calculated as the inverse of the gradient of the imaginary centroid line, (XS -XE) (Ys YE) Even more preferably the horizon angle is calculated as,

(X

s (Y YE) 112446150 Preferably in determining the horizon displacement the step of calculating the area (ngnd) of the captured earth portion involves assuming: the predetermined boundary is a circle with a radius (ii) the horizon is a straight line or chord of length 2Xof the circle; and (iii) the angle (20) from the centre of the circle subtended by the chord (2A) is equal to arcsin and then considering said earth area to be equal to the area of a segment of the circle subtended by 20 minus a triangular section of height Y(being the horizon displacement) and base 2X, ngnd R1 (8 cos(8)X/R) X= -Y) ngnd IR arccos(Y/R) YV(R Y2) Preferably the horizon displacement Yis approximated by a linear equation dependent on the calculated earth area and on the radius Rof the circular mask. For example, for R= 72 the equation is: Y= -0.00837 ngnd+ It will be appreciated that this linear equation will be different for different values of R.

BRIEF DESCRIPTION OF THE FIGURES In order to achieve a better understanding of the nature of the present invention, preferred embodiments of a method and an apparatus for determining horizon angle and/or displacement will now be described, by way of example only, with reference to the accompanying drawings in which: Figure 1 is a block diagram of an experimental setup of an apparatus for determining horizon angle and /or displacement; 112446150 6.

Figure 2 is a systematic illustration of an imaginary horizon line which is perpendicular to an imaginary centroid line connecting respective centroids of sky and earth portions; Figure 3 is a binary image of a horizon derived from the experimental setup of Figure 1; Figure 4 is a graph of measured horizon angle versus real angle during a smooth 3600 roll and utilising a circular mask and the experimental setup of Figure 1; Figure 5 is a graph of earth and sky centroids measured during a smooth 360' roll utilising the circular mask and experimental setup of Figure 1 and from which the horizon angle or roll angle is calculated in Figure 4; Figure 6 is a graph of measured horizon angle versus real angle for a 3600 roll but with turbulence and using the experimental setup of Figure 1; Figure 7 is a graph of earth and sky centroids from which the measured horizon angles of Figure 6 are calculated; Figure 8 is a schematic illustration of a masked image of a horizon showing dimensions and co-ordinates from which horizon displacement is derived; and Figure 9 is a graph of horizon displacement versus calculated earth area according to the methodology and apparatus of another aspect of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT As shown in Figure 1 there is an experimental setup of an apparatus 10 for determining horizon angle and/or displacement. The apparatus 10 of this example comprises a camera in the form of a visible light video camera 12, and processing means including a microprocessor 14 and associated software.

The video camera is in this embodiment a CMUcam developed at Carnegie Mellon University and is a low-cost CMOS digital output camera having an embedded microprocessor that captures the image and performs some primitive vision processing.

The microprocessor 14 is in this embodiment a microchip PIC16F876 (referred to as simply PIC) and the associated software performs high level vision processing to apply the methodology of determining horizon angle and thus aircraft roll angle. The microprocessor or PIC 14 is at the lower end of the computational power spectrum for 112446150 7.

microprocessors available today, and was chosen for that reason, to demonstrate this aspect of the method. The small size, weight and cost of the video camera 12 and microprocessor 14 means that all of the processing can for example be performed aboard a UAV and the results of the vision processing are in a form that can be simply passed on to appropriate control software to be used for maintaining stable flight.

The experimental setup used was to place the CMUcam 12 in front of a video monitor 16 and utilising the microprocessor 14 together with its associated software to generate an image of a rotating horizon formed between grey earth and pale-blue sky portions. The CMUcam 12 communicates with or is connected to the microprocessor or PIC 14 via a 115200 Baud asynchronous serial connection. In turn the PIC 14 is connected via a second serial port to a desktop computer running the microprocessor associated software which also collects an output of the PIC 14. In this example the output from the microprocessor or PIC 14 includes measured centroid coordinates and a calculated horizon angle at a rate of about twice per second. In this example the associated software of the PIC 14 controls the camera or CMUcam 12 to configure white balance and to set the range of colours that would be classified as sky class pixels. The CMUcam 12 generates a binary image and the CMUcam 12 is polled by the PIC 14 for the image. The binary image is then processed on the PIC 14 to: 1. apply a circular mask to the binary image; 2. calculate centroids of sky and earth class pixels of the captured image; and 3. calculate the horizon angle from an imaginary line connecting the sky and earth centroids.

In this example, and in order to avoid the need for image storage memory, application of the circular mask was performed during accumulation of the sky and earth coordinate pixels. This was achieved by defining the mask in terms of the first pixel that would be accepted per row which was precalculated for an approximation to a circular mask and requires only a small amount of lookup memory. The last pixel address in the row can be calculated by symmetry and this row mask then can be applied on a row-by-row basis. In this embodiment the amount of time it takes to apply the mask and accumulate the average coordinates for the sky and earth classes is 40 microseconds per 8 bit chunk on the 20MHz PIC microprocessor 14. This was achieved using assembler language 112446150 8.

programming for the critical inner loop of the pixel decoder. It will be appreciated that a faster microprocessor with for example an optimising C compiler could perform this accumulation and calculation all in C.

In this embodiment of the invention horizon angle is measured relying on contrast between sky and earth brightness in an image taken by a camera, such as the digital video camera 12 of the experimental setup, aligned to a longitudinal axis of an aircraft such as a UAV. Image pixels are classified into one of two classes, sky or earth, using a thresholding criterion. As described earlier, a pre determined area of the image is captured with a circular mask and the average coordinates (the centroid) of each of the sky and earth classes is calculated. The angle of the horizon is then determined by developing an imaginary horizon line perpendicular to an imaginary centroid line connecting the sky and earth centroids. This angle of the horizon gives the roll angle of the camera and hence of the aircraft.

As illustrated in Figure 2 and in line with this methodology for calculating horizon angle, the inventors in a preferred example considered a horizon image in a circular view and analysed the use of centroids for horizon angle and aircraft roll derivations. It is understood that in the case of a rectangular view the asymmetry causes errors that can be compensated for to a certain degree, this can be eliminated by simply ignoring or masking out pixels that fall outside of a circular boundary.

The following definitions have been developed in the process of deriving an equation for horizon angle.

Definition 1: the sky class is defined as those pixels that belong to the part of the image that is formed from light coming from the sky.

Definition 2: the earth, or typically ground, class is defined as those pixels that belong to the part of the image that is formed from light coming from the earth or ground.

Definition 3: the centroid of a class is defined as the average coordinate of those pixels that belong to the class, determined for each axis xyby taking the sum of the x,y coordinates of each pixel in the class and then dividing each of those totals by the respective number of pixels in the respective sky and ground classes.

112446150 9.

The theorem proposed by the inventors is that a line joining centroids of the sky and ground classes will bisect the horizon at a right angle provided the horizon makes a straight line in the view. This theorem will apply regardless of the roll angle and of the pitch angle. With reference to Figure 2 and based on arguments of symmetry and an appeal to Euclid's Elements, regarding the properties of a chord of a circle, this theorem can be proved as follows.

1. If the horizon is a straight line, then it forms a chord on the circle.

2. A line bisecting the chord formed by the horizon and perpendicular to it will pass through the centre of the circle, call this line the bisector.

3. Because the bisector passes through the centre of the circle and it is perpendicular to the horizon chord, the bisector divides the sky class into two equal and symmetric areas with area B being the reflection of area A about the bisector and the same logic applies with the ground class.

4. The average coordinate of area B is therefore in a position that is the reflection of the average coordinate of area A, the reflection being about the line of the bisector.

The average coordinate of the sky is equal to half the sum of the average coordinates of A and B. This average coordinate falls half way on the vector joining the average coordinates of a A and B which is a point on the line of symmetry between them which is on the bisector and the same applies for the ground class.

6. As both the sky and ground centroids fall on the bisector and it has been shown to be perpendicular to the horizon chord, then the proposed theorem is proven.

In the preferred methodology of this aspect of the invention, this theorem is used for finding the angle of the horizon by measuring the average coordinates of the sky and ground classes. It will be appreciated that the method has no dependence on the position of the horizon within the view but rather depends on the angle of the horizon only. The measured horizon angle will not change as the horizon moves with perturbations of for example the digital camera 12 (or its platform) that do not cause a change in the relevant angle between the camera horizontal axis and the horizon angle. In other words, disturbances to for example the pitch and yaw of an aircraft that are not so extreme as to 112446150 move the horizon out of the view do not have to be explicitly compensated for in the horizon measurement. This considerably simplifies the implementation of the preferred method of the invention. From the proven theorem, an imaginary horizon line is perpendicular to an imaginary centroid line connecting sky and earth or ground centroids. The gradient of the horizon should therefore be the inverse of the gradient of this imaginary connecting centroid line. This results in an equation for m, the gradient of the horizon: (Ys From the equation for m, the angle 0 that the horizon makes to the horizontal is: 0 arctan arctan -XE -E

((YS-YE)

where Xs, Ys and XE, YE are coordinates for the sky and earth centroids, respectively.

Figure 3 is an illustration of the results of the experimental setup of Figure 1 showing a captured image after "application" of a circular mask. The calculated sky and earth centroids are also shown. It will be appreciated that a smoother mask could be used and would improve accuracy but possibly at the cost of increased computation.

Figure 4 shows the angle calculated on the micro controller or PIC 14 of this example during a smooth 360° roll and Figure 5 shows the trajectories of the earth and sky centroids during the roll. It should be noted that the first data point is marked rejected and not used in the error calculation only as a precaution, as the first measurement was found to often be anomalous due to the nature of the communications between the micro controller and the software on the PC. The RMS error of 3.9° is relatively low considering it represents close to 1% of the full range of 360°.

Figures 6 and 7 show the angles and the centroid trajectories during a 360° roll manoeuvre where a random jitter of up to ±100 pixels was applied to the synthetic horizon image to simulate disturbances of up to approximately ±10° of pitch and yaw which was as much disturbance as could be applied without having the horizon leave the view. Figure 7 shows dramatically the amount by which the centroid positions were 112446150 11.

disturbed. However, Figure 6 shows by its remarkable similarity to Figure 4 and by the similar RMS error of 3.50 just how little effect the disturbances had on the horizontal roll angle measurement.

In another aspect of the invention there is a method and apparatus for determining horizon displacement. In this embodiment the apparatus is included in the apparatus of the preceding aspect of the invention for determining horizon angle. That is, the camera 12 and microprocessor 14 of the horizon angle apparatus is utilised in this aspect of the invention for calculating the horizon displacement. The processing means including the microprocessor 14 of this aspect also includes software configured to calculate the horizon displacement.

As shown in Figure 8 the method of this aspect of the invention relies upon a masked image of the horizon including adjacent sky and earth portions with the horizon assumed to be a straight line. For ease of understanding the captured image, which in this example has been circularly masked, has been rotated so that the line of the horizon is horizontal and the ground is at the bottom of the image. The image is centred at the origin 0,0 and the radius of the masked circular image is R. The area of the earth or ground in the masked view has been designated as ngnd and the x coordinate increases toward the right and the y coordinate increases upwards. The x coordinate of the point where the horizon cuts the edge of the circular view is denoted as X, and Yrepresents the distance of the line of the horizon from the origin, or the horizon displacement.

If we define as 2Othe angle subtended by the chord of length 2Xthen: O= arcsin (1) By considering the area of the ground to be the area of a segment of the circle subtended by 20minus a triangular section of height Yand base 2X we arrive at the following: ngnd R' cos(O)X/R) (2) y= y2 (3) ngnd k? arccos(Y/R) YvJ(R /2) 112446150 12.

Unfortunately Equation 4 is not convenient to use on a small microprocessor because it can't be solved analytically for Y and the numerical solution involves a time-consuming gradient search technique. The horizon displacement Ycan be approximated by a linear equation dependent on the calculated earth area Ron the radius Rof the circular mask.

For example for R 72 we can use: Y =-0.00837ngnd+ 65 Figure 9 shows an illustration of this approximation. Note that the 3% error that this approximation introduces is reasonably small and will almost certainly be outweighed by other error sources, including the inaccuracy introduced by the assumption of a straight line horizon.

Now that preferred embodiments of the various aspects of the invention have been described in some detail it will be apparent to those skilled in the art that the method and apparatus for determining horizon angle and/or displacement have at least the following advantages: 1. The apparatus including the camera and processing means are generally of a lightweight construction and as such lend themselves to UAV and other aircraft applications; 2. The method and apparatus for determining horizon angle is relatively accurate and generally inherently able to ignore pitch and yaw of an aircraft, provided the horizon stays in camera view; 3. The method and apparatus for determining horizon angle and/or displacement imposes a relatively low computation burden on vision processing equipment and, for example, does not require a frame buffer as all the operations of classifying pixels and accumulating coordinates for determining centroids are local operations; and 4. The method and apparatus in one embodiment permits the application of a circular mask without a frame buffer and as such relatively simple vision architecture can be used and processing effected at a relatively fast rate.

112446150 13.

Those skilled in the art will appreciate that the invention described herein is susceptible to variations and modifications other than those specifically described. For example, an analog camera may be used together with an interface for connection to a programmable logic device for calculating the horizon angle and/or height. It should also be appreciated that the method and apparatus are not restricted to any particular application and for example extend to motor vehicle and terrestrial applications. In these other applications the method and apparatus may for example be used to avoid terrain. All such variations and modifications are to be considered within the scope of the present invention the nature of which is to be determined from the foregoing description.

Claims (7)

14. Claims 1. A method of determining horizon angle, said method comprising the steps of: taking an image of a horizon together with adjacent sky and earth portions; defining a predetermined area of the image which captures at least part of the horizon and the adjacent sky and earth portions; determining respective centroids of the captured sky and earth portions; calculating the gradient of an imaginary centroid line connecting the sky and earth centroids; and calculating the horizon angle from an inverse of the gradient of the imaginary centroid line. 2. A method of determining horizon angle as defined in claim 1 wherein the step of defining the predetermined area involves masking the image with a predetermined boundary to define the captured sky and earth portions. 3. A method of determining horizon angle as defined in claim 2 wherein the predetermined boundary is a circular mask which exposes the area within the circular boundary. 4. A method of determining horizon angle as defined in any one of the preceding claims also comprising the step of classifying pixels of the captured image into sky class and earth class pixels. 5. A method of determining horizon angle as defined in claim 4 wherein said classification step involves application of a thresholding criterion wherein sky class and earth class pixels are classified according to their colour in the visible spectrum. 6. A method of determining horizon angle as defined in any one of the preceding claims wherein the step of determining the respective centroids of the captured sky and earth portions involves totalling each of x and y coordinates for the sky class pixels and dividing by the number of pixels of the captured sky, and 112446150 performing a like calculation for the earth class pixels, to obtain x and y coordinates for the sky and earth centroids (x,y and x,ye). 7. A method of determining horizon angle as defined in claim 6 wherein the gradient of the horizon is calculated as the inverse of the gradient of the imaginary centroid line. 8. A method of determining horizon angle as defined in claim 7 wherein the horizon angle is calculated as arctan 9. An apparatus for determining horizon angle, said apparatus comprising: a camera designed to take an image of a horizon together with adjacent sky and earth portions; and processing means in communication with the camera and being configured to process the image whereby a predetermined area of the image is captured and respective centroids of the captured sky and earth portions are determined and the gradient of an imaginary centroid line connecting the sky and earth centroids calculated and the horizon angle calculated from an inverse of said gradient of the imaginary centroid line. An apparatus for determining horizon angle as defined in claim 9 wherein the camera is a visible light video or still camera. 11. An apparatus for determining horizon angle as defined in claim 9 wherein the camera is suitable for use in the infra-red spectrum. 12. An apparatus for determining horizon angle as defined in any one of claims 9 to 11 wherein the camera includes a digital interface which communicates digitally with the processing means. 13. An apparatus for determining horizon angle as defined in any one of the claims 9 to 12 wherein the processing means includes a microprocessor together with associated software being configured to process the image and calculate the horizon angle. 112446150

16. 14. A method of determining horizon displacement, said method comprising the steps of: taking an image of a horizon together with adjacent sky and earth portions; defining a predetermined area of the image which captures at least part of the horizon and the adjacent sky and earth portions; and calculating the area of the captured earth portion and deriving the horizon displacement from said calculated earth area and a coordinate or dimension of the predetermined area. A method of determining horizon placement as defined in claim 14 wherein the step of calculating the area (ngnd) of the captured earth portion involves assuming: the predetermined boundary is a circle with a radius (ii) the horizon is a straight line or chord of length 2Xof the circle; and (iii) the angle (20) from the centre of the circle subtended by the chord (2A) is equal to arcsin and then considering said earth area to be equal to the area of a segment of the circle subtended by 20 minus a triangular section of height Y(being the horizon displacement) and base 2X, ngnd IR (0 cos(0)X/R) X= I( -Y) ngnd RI arccos(Y/R) Yv(R Y 2 16. A method of determining horizon displacement as defined in claim 15 wherein the horizon displacement Yis approximated by a linear equation dependent on the calculated earth area and on the radius Rof the circular mask. 112446150

17. 17. A method of determining horizon displacement as defined in claim 16 wherein for R= 72 the equation is: Y= -0.00837 ngnd+

18. An apparatus for determining horizon displacement, said apparatus comprising: a camera designed to take an image of a horizon together with adjacent sky and earth portions; and processing means in communication with the camera and being configured to process the image whereby a predetermined area of the image is captured and an area of the captured earth portion calculated, the processing means deriving the horizon displacement from said calculated earth area and a coordinate or dimension of the predetermined area.

19. An apparatus for determining horizon displacement as defined in claim 18 wherein the camera is a visible light video or still camera. An apparatus for determining horizon displacement as defined in claim 18 the camera is suitable for use in the infra-red spectrum.

21. An apparatus for determining horizon displacement as defined in any one of claims 18 to 20 the camera includes a digital interface which communicates digitally with the processing means.

22. An apparatus for determining horizon displacement as defined in any one of claims 18 to 21 wherein the processing means includes a microprocessor together with associated software being configured to process the image and calculate the horizon angle and/or derive the horizon displacement. Dated: 28 June 2004 Monash University Patent Attorneys for the Applicant: BLAKE DAWSON WALDRON PATENT SERVICES