GB2127649A - Compensation for video sensor movement - Google Patents

Abstract

A method and apparatus for correcting a video input signal received from a scene scanning device so that the corrected signal allows for the spatial movement of the scanning device relative to the scene being scanned e.g. the effects of crabbing, yawing or pitching of an aircraft having a scene scanning device on board, are not relayed to the video output signal. The corrections are applied to the incoming signal before storage in a frame store whose output may be viewed or recorded. The number of lines scanned is greater than the number displayed to avoid duplication of lines. <IMAGE>

Description

SPECIFICATION Picture correction This invention relates to a method and apparatus for processing a video signal to correct for spatial inaccuracies or movements of the sensor producing the signal. For example, a vertical-looking crosstrack line scan sensor on board an aircraft scans the ground beneath the aircraft along lines transverse to the aircraft movement direction. As the aircraft moves, so the scan lines advance along the ground to build up a video signal representative of the ground covered. This signal may be displayed or recorded in a manner similar to a t.v. video signal.However, if the aircraft is moving in a crabwise attitude or if it pitches or rolls, or even if it is travelling at an inappropriate height or speed, the monitor on which the signal is eventually displayed and which will generally operate on the assumption that the original scan lines were evenly spaced along, bisected by and at right angles to the line on the ground above and along which the aircraft was or is moving, may present a distorted picture. For example, the effect of crabbing or yawing of the aircraft is to produce rhombic distortion of the picture while pitching produces successive compression and spreading of the picture.

According to the invention, there is provided picture correcting apparatus for receiving a video signal formed by a scene scanning device and for modifying said signal to allow for spatial movement and/or malpositioning of said device relative to said scene, the apparatus comprising frame store means having a data input for receiving successive digitized elements of said signal or 'pixels' and having address input means for receiving signals in dependence upon which said pixels are stored at respective addresses within the frame store and the apparatus further comprising base address supply means for supplying a base address signal for the pixels to be stored in the frame store, computer means for calculating, on the basis of positional information relating to said device and/or a craft wherein it is carried, a series of address offset values respectively associated with respective ones of a current group of said pixels, register means for storing said values calculated by said computer, and means for reading said stored values in sequence and for causing a signal comprising a combination of the read value and said base address signal to be passed to said frame store in synchronism with the associated pixel.

With the method and apparatus to be particularly described and illustrated herein, a scanned picture can be corrected in real time for skew providing the instantaneous values of error are known or may be predicted. The corrected data is stored in a frame store (or partial frame store) and may be passed on for viewing or storage with a delay of one frame maximum.

For a better understanding of the invention, reference will now be made, by way of example, to the accompanying drawings, in which: figure 1 is a simplified circuit diagram of a picture correction apparatus for use with an aircraft line scanner, figure 2 is a geometrical diagram for explaining part of the operation offigure 1.

The apparatus shown is operable for receiving a video signal V from an aircraft borne verticallooking cross-track line scanner (not shown) and, in dependence upon signals (from the aircraft navigation system for example) indicative of instant values of roll, pitch, crab, yaw, speed and height of the craft, for producing a picture signal corrected as regards these parameters. For the example shown, the line scanner is assumed to comprise say an infra-red radiation detector element or array thereof and an optical system including a rotating polygonal reflector element which, as it rotates, causes the radiation received by the detector element(s) to be that which arrives from the ground beneath the aircraft in a direction varying between the horizontal to one side of the aircraft and the horizontal at the other side.Thus, in effect the detector element(s) scans the ground from one horizon to the other along lines perpendicular to the direction of movement of the craft. The scan lines advance along the ground as the aircraft moves giving a continuous frame advancing effect. The scanner is fixed within the aircraft so that, if the aircraft rolls, the normal mid-scan position on the ground will move from directly beneath the aircraft to one or the other side thereof. Depending on the aircraft speed and the width of each scan line on the ground, which in turn depends upon the aircraft height and the convergence of the instant field of view of the scanner, the lines may overlap one another, be exactly continguous, or may be spaced apart (i.e. so that some portions of the ground are missed).If the aircraft pitch varies, then while it is actually changing, some of the scan lines will be compressed or moved further apart. If the aircraft 'yaws' or 'crabs', the direction of the lines will vary from the correct position of perpendicular to the aircraft movement direction. The effect of the above possibilities, if uncorrected, is to produce distortion in the eventual displayed picture.

Referring now to the drawing, the video signal V comprises a series of portions each representative of a respective scanning line and each comprising a series of values representative of successive picture elements or 'pixeis' along the respective line. Also received from the scanner is a line sync. signal which is here assumed to be generated at the start of each line, i.e. whenever the scanner is looking in the horizontal direction from which the scan lines start. The video and line sync. signals are received by a correction logic circuit 1 comprising a plurality of shift registers to be described separately later. The output from the logic circuit 1 is passed to a frame-store 2.On the basis of navigation information passed thereto from say the aircraft navigation system, a function computer 3 controls the logic circuit 1 and the frame store 2 to select and/or reposition pixels and lines of the video signal V whereby a corrected video signal Vo becomes available at the output of the frame store.

The method of applying the corrections is to calculate the nearest, rectangular coordinate point for the data collected and store the data in that particular pixel store in the frame store.

The correction factors are calculated by the function computer and output to the appropriate registers. Since the correction factors do not change rapidly, they can be loaded and used while the next set of correction factors are calculated.

The corrections are actually performed by the apparatus in three stages as follows:- i. Roll distortion. This is simply corrected by adjustment of the line sync. pulse in accordance with the roll angle. This correction is assumed to be performed prior to the "line select" operation.

ii. Rate/position corrections. The parameters affecting these factors are vehicle height, ground speed and pitch. They affect the number of lines (and hence pixels) along the line of track i.e.

vertically up the display.

iii. Angular corrections. These have two inputs Yaw and Crab. They both have the same effects and differ only in their time constants, Yaw is relatively rapid while Crab is slow. Crab and Yaw angles are added arithmetically to produce the angular corrections in both X and Y coordinates.

Referring now to figure 1 , the video signal V is fed via an And gate 4 to a line select unit 5 of the logic i.e. circuit 1. As mentioned earlier, the signal V comprises a series of video signal lines each corresponding to a scan of the line scanner. The function of the line select unit 5 is to pass a controlled proportion of the number of received video lines with a view to causing the eventual picture signal to contain a desired number of lines for a given distance over the ground, which number of lines are reasonably evenly spaced over this distance, despite changes in aircraft ground speed, height and pitch.

Signals G, H and P respectfully representative of the latter parameters are passed to the line select unit 5 from the aircraft navigation apparatus (not shown). The vehicle ground speed will determine how many sensor scans are taken over any length of ground. The 'pitch' of the vehicle will effectively alter the ground speed (both positively and negatively) with a fairly rapid time constant. The pitch angie, knowing the height, can be used to determine an instantaneous value of ground speed.

The number of scan lines to be displayed per unit of ground length covered is also known as are the scan rate of the sensor and the instantaneous ground speed. Therefrom there is determined the ratio of the number of lines gathered to the number of lines displayed. The number of lines gathered must always be greater than the number of lines displayed if the duplication of lines (hollow magnification) is to be avoided. In practice between 2 and 13 lines are gathered for each one displayed.

The number of lines output from the line select operation must be equal to the number of lines displayed (or an integer multiple of them).

The lines to be output are selected by finding the nearest line to the correct displayed position, i.e.

the nearest "line vertical" as follows:- In the following explanation, N is the number of lines scanned per Kilometre, n is the number of lines displayed per Kilometre, lv' is the input line number and Iv is the displayed line number.

The nearest value will change each line unless N is an integer multiple of n, then the offsets will cycle at the N/n rate.

It will not be possible to compute new values at this rate so a compromise will be made.

If the effect of pitch is not taken into account, the nearest display line to any given input line is given by:

where a is the vertical difference between the two ref.

origins.

so that input lines would only be used or passed on when

The effect of changing pitch of the aircraft can be taken into account by modifying equation 1 as follows: The displacement on the ground = h tan 8 where h is vehicle height and 8 is the instantaneous pitch angle from the horizontal.

This gives a pitch factor which is added to equation 1 to give:

and input lines are now deemed valid and hence used when Iv' satisfies the following criteria:

The displayed line number then becomes

In effect, the shift caused by the vehicle pitch (S) has been added to the line position. As will be appreciated pitch variation introduces an additional variable and it can be catered for easily only providing the angle changes relatively slowly. This argument applies also to N which is a function of the vehicle ground speed.

If the number of lines being output from the 'line select' operation is an integer multiple of the number being displayed, then a positionally more accurate picture can be built up at the expense of additional hardware. Since an additional Y offset Shift register and X Offset Shift register is required for each multiple, e.g. 7 lines 7 registers.

It may also be possible for fractional lines collected/lines displayed to be handled provided it is possible to dynamically calculate the X and Y offset values in real time.

The line data collected must be longer than that displayed to allow for the Crab and Yaw corrections.

Line length displayed Line Length collected > Cosine (Crab angle + Yaw angle) If insufficient line length is collected then it will result in blank strips at the edge of the display, for areas where data was not collected. It will not affect the positional accuracy however.

In order to correct for yaw and crab, the input pixels are displaced vertically, the amount of displacement varying with the horizontal position of the pixel, and preferably also horizontally to correct for the lateral displacement introduced by the crab and yaw. The latter corrections are only really relevant for the edges of the picture and are not as important as the vertical corrections. The values of vertical displacement for the respective pixels of a line are loaded by computer 3 into a Y-offset shift register 9. The displacement values are read from the register in sequence in synchronism with the reading out, from pixel shift register 7, of the pixels themselves. Each displacement value is added to (or subtracted from) the line number provided by a line counter 10.The content of which is incremented by the line sync pulses from the scanner, to give a corrected Y-address value for the relevant pixel.

Meanwhile, in a similar way, horizontal displacement values loaded into an X-offset shift register 10 by computer 3 are being read from that register in sequence and added to (or subtracted from) a value supplied by pixel counter 12, and incremented by a strain of pixel sync pulses fed to the apparatus, which is representative of the base or uncorrected horizontal pixel position. The resultant corrected horizontal or X-address value is fed to frame store 2 along with the corrected Y-address value and the corresponding pixel is stored at that corrected address.The displacement values are calculated by the computer 3 as follows:- Let scanned lines/km = N displayed lines/km = n input line width = k/N metres k displayed line width represents - = metres n Vertical Line count from bottom of display = Iv Horizontal pixel count from left hand edge of display = 1h Iv a 1h are integers.

Input pixels/km = M displayed pixels/km = m k Input pixel width - = metres m k displayed pixel width represents - = metres m Yaw and Crab angle = 6.

The horizontal input pixel count Ih' is counted from some arbitrary trigger point at the start of the scan. The Vertical line count Iv' is counted from an arbitrary point in the frame.

It is assumed that the input data is digitized at a rate that will give consecutive pixels just touching the previous pixel.

Note that the quantization rate could be adjusted (Linear frequency adjust) so that the input pixels are vertically in line with the output pixels. The frequency adjustment is fN = fJSin 0 where fN = New Sampling frequency fo = display sample frequency 0 = Yaw + Crab Displayed Pixel centres are:

Input Pixel centres are:

Distance between centres is:

This distance is a minimum when each of the contents of the 1 j brackets is a minimum, since only real numbers are involved.

Horizontals

(Ih +7l)mk (Ihl + 2) k Cos 8 -b = MINIMUM ('h +2l)mk: (IhI+2l)wkcose+b IF MINIMUM ZERO Ih=i{(lh 2) M Cos 0 + b 2M}k k

WHERE DENOTES NEAREST INTEGER VALUE If pixel quantization is the same vertically and horizontally then M = N m = n

1h = I N (Ih + 2) Cos û 2 + k.

Vertically

k (lvi +Q)k - a (1V' : MINIMUM (Iv + l)k (ii+1)Sin 8 - N Cos 9 lklk. 8 (lvi +71)k + a IF MINIMUM ZERO Iv +2n=I.nI+TMSrn N Cos 9 Linear magnification and equal resolution vertically and horizontally

M=N m=n In this system Iv is stepped with Iv' if there is only one line selected for one line displayed so the vertical stepping is effectively removed from the correction equation, i.e. Iv' = 0 for one line in, one line out. For more than one line in, Iv' takes on the values 0 for the first line, 1 for the second line etc.

The respective displacement values for all the input pixels from first to last in the line, i.e. the respective displacement values for all the values of input pixel position Ih' from 0 to the total number of input pixels in a line, are calculated by the computer using the functions described above and then held for as long as the corresponding value of yaw plus crab angle 0 remains extant. If 0 varies by more than a predetermined increment, a new set of values are calculated and loaded.

For expediency, it is preferred that the computer 3 should anticipate each incremental change in 8 by holding the displacement values associated with the value of 0 prior to its last incremental change and by calculating and holding the displacement values for the value of 0 which would be extant if it were to continue to change in the same direction through another increment. Then, if it does so continue to change, these newly calculated values can be immediately loaded into the offset registers. On the other hand, if the direction of change of 0 reverses and goes back to the previous incremental value, the held displacement values associated with the prior 0 value can be loaded.In this case, of course, each recalculation by the computer is not of the displacement values for the present 0 value but of those for the next expected value. The extra sets of displacement values may be held in respective further shift registers (not shown) each comprising a length or number of stages which, like that of the registers 9 and 10 equals the maximum number of pixels per line.

The width of the Y-offset register, i.e. the bit capacity of each stage thereof, and the width of each of the further vertical displacement registers if used, should be sufficient to allow for the maximum amount of vertical offset likely to be required which in turn depends upon the likely maximum value of yaw plus crab angle fl. For example, if the likely maximum value of 0 is + 250, then for a 1000 x 1000 pixel data display the maximum offset, as a number of lines, is + 1000 tan 250 or + 466. Ten bits would be required for this (giving in fact a maximum offset value range of + 512).

The line start unit 6 inhibits Pixel data and pixel sync in dependence upon Roll angle, i.e., it gets all lines to start at the same angle to the air vehicle position rather than its attitude. For example, the data and pixel sync may be inhibited until the angle is such as to give a swathe of 1 km on the ground taking into account the roll angle and the height.

Figure 2 of the drawings shows the position P of the aircraft looking along its longitudinal or roll axis. The aircraft is at height h above the ground G and it is desired that the displayed picture should be of an area extending to 500 metres from each side of the line directty beneath the aircraft. As mentioned earlier, the line sync signal is produced by the line scanner when this is looking in a direction which would be horizontal if the aircraft were flying straight and level i.e. when the scanner is looking in a direction parallel to the lateral or pitch axis Y. Meanwhile, the delayed line sync signal should occur when the scanner is looking at the first boundary of the area to be displayed, i.e. along the line L if the scanning proceeds in the direction shown by arrow A.Thus, the required delay equals the time required for the scanner rotor to move through the angle X = tan~1 (h/500) between line Land the horizontal plus (or minus) the actual angle of roll. If the angular velocity of the scanner rotor is W then the delay is (O + tan- (h/5000). W seconds where 0 is the roll angle - this is made positive if the roll is opposite the direction of scanner rotor movement and negative otherwise.

Meanwhile, the end of each line occurs after (0 + tan~1 (h/500) + 2 tan-' (500/h)). W seconds which expression reduces to (O + 7r/2 + tan~1 (500/h)). W seconds from the line sync pulse.

The pixel shift register 7 is designed to smooth out the differing rates of collection and use of the pixels. If the hardware can cope with the maximum pixel gathering rate it will not be needed. The device is essentially a F.l.F.O. (First in First Out) register. It holds the quantized video data in the form of 6 (or 7 or 8) bit pixels. A pixel is output from the register to be stored in the frame store on every Pixel sync pulse. The position it will be stored in is determined by the current X and Y address, and whether or not it is to be stored will depend upon the "Not Used" register.

Since there are more pixels gathered than displayed, no pixel will be used twice. If a pixel had to be used twice then another register containing information to use the pixel twice (or more) must be used, together with circuitry to inhibit shifting the F.l.F.O. while the X and Y Offset registers are shifted.

Associated with the register 7 is a 'Pixel not used' register 8 which is used to indicate when any particular pixel will not be used, this will occur when another pixel is better suited to the display position, or when a pixel is discarded owing to wrong positioning as in the case of rectilinearization which may also be performed in this unit.

The unit is loaded from the computer in conjunction with X and Y offset registers. The pixels are not used when any two (or more) values of input pixel 1,h give the same two values of Iv and 1h (this can happen since Iv and 1h are integers). All but one of these pixels must be discarded. In fact, the occurrence of pixels with the same values of Iv and Ih will be quite low and it would not matter too much if, on these occasions, a quite arbitrary decision were made about which one is not discarded.

Preferably, however the one kept is that whose actual values are nearest the integer values. The nearest and correct choice is given by the value of Ih+n and which gives the iowest value to the function.

for values of A =0,1,2 or-1, -2 etc. The values of Ih which give the same Iv and 1h should be discarded except the value of Ih which gives the lowest value of the function above.

As mentioned earlier, the X-offset shift register 10 is used to apply any lateral shift to a pixel to correct for the lateral displacement introduced by the Crab and Yaw, which corrections are only required at the edges of the picture and are not as important as the Y-offset corrections.

The rectilinearization corrections can also be added to this register if desired. The rectilinear corrections can be applied by computing a composite value of the Y offsets or holding the rectilinear offsets in a separate shift register and adding the two outputs.

The contents of the register are calculated from:

VALUE:Ih' - (Ih'+ 2) Cos 0 2 k } 2 for each value of Ih' up to the maximum number of pixels per line.

The width of this register need only be sufficient to cater for the maximum lateral offset of any pixel. For a 1000 pixels/line display and maximum yaw and crab angle 0 = 250 (the sign of 0 is not relevant here - +250 and 250 give exactly the same effect): Maximum offset (as a number of pixel positions) =1000(1 - Cos 25)= 94.

This will require 8 bits only.

The frame store 2 accepts data and modified X and Y addresses and stores the pixel value in the appropriate position in the same store. This position is now spatially correct.

The data may be read out from the frame store as another part is being loaded, providing the store is large enough to cater for the largest values of O then the store can be used as a rolling store. Note that it does not have to cater for + since both values are not likely to occur within one frame. If the frame is small then the offset value for unloading the store must allow for the sign of 0.

As described, the computer 3 takes the values of Yaw, Crab, height, Groundspeed and pitch, and computes which lines to select and the values of the X and Y offset registers, is also calculates which pixels are to be used. Since the parameters will not change rapidly in relation to the video data, a fairly slow computer should be able to cope with the demand. If, as is likely, the input parameters are continuous, they are quantised into steps which have a meaningful size.

The principle mentioned earlier, whereby the computer holds and/or calculates previous and likely next yaw and crab displacement values could be extended so that say the previous and likely next set of values for the line select, X and Y offset registers and the 'Not used' registers are kept available. The required set of values can then be loaded into the appropriate register as soon as the relevant parameter has undergone an incremental change and, while holding the previous set in case the direction of change reverses, the computer can calculate the set appropriate to the next incremental parameter change in the same direction.

The description given above relates particularly to apparatus for correcting the signal supplied by a line-scanner on board an aircraft (which term includes helicopters). Thus, the types of correction applied take account of the possible spatial positioning of an aircraft. It will be appreciated however that the invention is not only applicable to the described situation. Thus, it could be used for another sort of scanning sensor, for example a t.v. camera, (particularly forward-looking t.v. surveillance apparatus on an aircraft) or for a line-scanner or other kind of sensor on board a vehicle or platform other than an aircraft. Whether or not the particularly described situation is extant, it may be desirable, necessary or allowable to modify the corrections applied to the signal from those described, or possibly not all of them will be needed. For example, even for a line-scanner or an aircraft, it may not be necessary to correct for say roll and rectilinearisation since these may be corrected, perhaps preferably so, prior to the correcting apparatus.

As will be appreciated, the particular implementation of the hardware making up the apparatus is selectable and will be chosen having regard particularly to the input data ratio and the required size of the frame store 2.

Claims (3)

1. Picture correcting apparatus for receiving a video signal formed by a scene scanning device and for modifying said signal to allow for spatial movement and/or malpositioning of said device relative to said scene, the apparatus comprising frame store means having a data input for receiving successive digitized elements of said signal or 'pixels' and having address input means for receiving signals in dependence upon which said pixels are stored at respective addresses within the frame store and the apparatus further comprising base address supply means for supplying a base address signal for the pixels to be stored in the frame store, computer means for calculating, on the basis of positional information relating to said device and/or a craft wherein it is carried, a series of address off-set values respectively associated with respective ones of a current group of said pixels, register means for storing said values calculated by said computer, and means for reading said storedvalues in sequence and for causing a signal comprising a combination of the read value and said base address signal to be passed to said frame store in synchronism with the associated pixel.

2. Picture correcting apparatus substantially as hereinbefore described with reference to the accompanying drawings.

3. A picture correcting method substantially as hereinbefore described with reference to the accompanying drawings.