GB2149261A - Image processing apparatus - Google Patents

Abstract

Where the output of a particular detector element in a "staring" array has a value indicating a dead or seriously malfunctioning element it is convenient to replace it with an average value calculated from those output by the adjacent elements. A Shift Register 34 long enough to hold in predetermined locations N1-N4 the values output by all the suspect element's near neighbours (and all the ones inbetween the first and last of these in the read sequence) co-operates with means for replacing a "dead" element output value in the Register 34 with a calculated value 35 based on an average of those values in the defined neighbours' locations. Addresses of "dead" elements are determined during an initialisation stage and stored in microprocessor 36. The addresses are read out sequentially until agreement with an address in register 33 corresponding to the element being scrutinised is determined. The average value 35 is then written into the output value shift register. 34. <IMAGE>

Description

SPECIFICATION Image processing apparatus This invention concerns image processing apparatus, and relates in particular to the processing of image data derived from a detector array, especially an infra-red detector array.

In a number of fields it is now common to supplement, or even to replace, imaging systems using visible light with corresponding systems using infra-red (IR) radiation (and referred to as thermal imaging systems). Many techniques are employed in the detection of this IR energy; one uses a single detector cell (commonly a cryogenically-cooled mercury cadmium telluride device) across which a system of rotating mirrors scans the image of the viewed scene in successively vertically displaced lines, while another uses a line of such individual detector cells across which the image is scanned as a single swathe. Detector systems of this type are, however, delicate, and expensive to construct and maintain at peak efficiency, and the more modern IR imaging systems employ solid state detector area arrays which "stare" at the viewed scene (and are thus known as "staring arrays").

These arrays are analogous to those used in present-day television cameras except that they are smaller (as few as 32 elements square as opposed to one or two thousand square) and are constructed so as to be IR sensitive rather than visible light sensitive.

Unfortunately, the current level of technology is not capable of constructing IR sensitive detector arrays with real consistency and uniformity (and indeed finds some difficulty in this regard with visible light arrays). As a result, the sensitivity (the gain in output for a specified gain in input) and the output absolute value (dependent upon both the gain-and the output at some specified input level, which latter is referred to as the "offset") of the detector elements can vary very significantly over the array. Indeed, the variation is usually markedly larger than the output changes caused by the differences in radiation output of the viewed scene; this means that the raw output from the detector array is practically useless, and must first be processed in some way to provide--ultima- tely-a meaningful picture.

The problem may in the long term be solved by improvements in technology enabling detector arrays-and especially IR arra ys-to be constructed so that all the elements in the array have the same gain and offset.

For the present, however, the invention of our copending Application for UK Letters Patent No. 8329470 Publication No. ) (1/ 6768/ELL) puts forward quite a different solution, in which the output of each individual detector element is normalised (corrected to the value that an average element would give). Specifically, there are found for each detector element the values defining its gain and offset, there are then recorded for each element the two factors that will normalise these values (convert each to the mean value of the entire array), and thereafter the relevant two normalisation factors are in use applied to the output of each element, before that output is employed to form the desired image of the viewed scene, so that the output is modified to the value it would have if it had emanated from an element having the predetermined average values for gain and offset.

The method of the aforementioned invention-and specifically the normalisation calculations and the storage of the constants involved-is conveniently performed under the control of a microprocessor, and one of the tasks the microprocessor can be required to perform is that of providing "average" values for any detector elements that are malfunctioning to a significant degree. This may be effected as follows. Firstly, the normalised output value data is fed to a temporary store (preferably one or other of two such stores in sequence; first one store is filled, then the other, and thereafter the data is read out therefrom, and sent on to the VDU, from that store not presently being filled).Secondly, while in the temporary store (and before it is fed to the VDU) the data is examined, datum by datum (and thus store location by store location) to see whether it is of a value indicating a dead or seriously malfunctioning detector element-whether, say, it is above or below a threshold level indicative of a "dead" detector. Where such a datum is found then the data/store locations for the adjacent ele ments---conveniently those four immediately above, below, to the left and to the right of the faulty element-are examined, and their values averaged to give an extrapolated value for the faulty element, which value is then placed in the relevant store location.Indeed, rather than perform this examination every time it is much preferred to effect it oncc say, during the initial setting up of the apparatus, when the normalisation constants are calculated-and to store the address (the identifying location) of each of those elements that fails to provide an acceptable output.

Following this, in operation it is necessary merely to look at the output values stored for the elements adjacent each of the "dead" addresses, and after averaging their values to insert the result into the store location relevant to that of the faulty element.

These operations are software controlled-they are performed in real time by the microprocessor following its pre-programmed instructions-and they take significant amounts of time (even the preferred variation) that may become unacceptably large when the array size is itself large. The present invention suggests a hardware solution, proposing the use of a Shift Register long enough to hold in predetermined locations the values output by all the suspect element's near neighbours (and all the ones inbetween the first and last of these in the read sequence) together with means for replacing a "dead" element output value in the Register with a calculated value based on an average of those values in the defined neighbours' locations.

In one aspect, therefore the invention provides apparatus for providing a computed output value to replace the output value of a malfunctioning detector element within a matrix of elements, the computed value being derived from the output values of the element's neighbours in the matrix, which apparatus comprises:: a shift register, to which is fed in the read sequence data defining the output values of the detector elements, this shift register having sufficient cells (locations) such that when one predetermined cell holds the output value of the suspect element being investigated a last predetermined cell holds the output value of the first neighbouring element to be scanned during the read sequence while a first predetermined cell holds the output value of the last neighbouring element to be scanned during the read sequence, the output values of all the other neighbouring elements, and of the suspect element itself, being in predetermined cells between these two; comparator means for recognising when the value in the cell holding the suspect element output value is indicative of a malfunctioning element; and calculator means for computing, from the output values in the neighbouring element cells, an average output value, and for inserting that output value as a replacement into the cell holding the suspect element output value.

The detector element matrix array with which the apparatus of the invention operates may be any of those used or suggested for use in the Art for the relevant radiation (which may itself be visible or non-visible UV or IR, say-light). A typical such array sensitive to visible light is the Integrated Photomatrix Ltd IPL 2D1 (a 54 X 54 photodiode array), while an IR array is the 32 by 32 element infra-red detector CCD array available from Mullard under the designation M4680.

The inventive apparatus provides a computed value for a malfunctioning detector element upon the basis of an average value calculated from the values of the element's neighbours in the matrix. This point is discussed further hereinafter: here it suffices to say that the "average" may be any sort of average (though an arithmetical average--- summing the neighbours' output values, and dividing by the number of these values appears quite satisfactory), and that "neighbour" preferably means those elements actually adjacent the malfunctioning element (there will generally be eight such neighbours, four-above, below, left, right being "row/column" neighbours, four diagonally up left, down right and up right, down left-being "diagonal" neighbours), and of these is very preferably either the four row/column neighbours or the four diagonal neighbours, not both or any mixture thereof (because that makes the averaging arithmetic more complex).

In the apparatus of the invention the data read out from the detector element matrix is fed, in the read sequence, into a shift register (a memory device having a series of memory locations, like pigeon holes, in which data can be stored, the device being so designed and constructed that data items fed in to the location at one end are moved along, location by location, as more data is fed in thereafter, until the first data item pops out of the downstream end of the device). Shift registers are ccmmon devices in today's digital electronics art. They can be made in almost any length and width (the length refers to the number of locations cells-from end to end, while the width refers to the number of bits-binary digits, 0 or 1 cach location can hold), and need not be described in more detail herein.

The width of the shift register is naturally sufficient to hold the number of bits defining each detector element output value (a typical such width would be 1 6 bits). The length of the register is sufficient such that when one predetermined cell holds the output value of the suspect element being investigated a last predetermined cell holds the output value of the first neighbouring element to be scanned during the read sequence while a first predetermined cell holds the output value of the last neighbouring element to be scanned during the read sequence, the output values of all the other neighbouring elements, and of the suspect element itself, being in predetermined cells between these two. In other words the register holds (in reverse sequence) all the output values of all the detector elements from the first of the neighbours to be read to the last of the neighbours to be read. Each such output value is in a predetermined cell relative to that cell holding the output value of the suspect element. The first such value is a known number of cells beyond, while the last is a known number before, and the other neighbours are similarly other known numbers of cells beyond or before; the numbers are in each case the same as those defining the relative positions of the elements themselves in the array. An example of a typical situation may help to clarify the point.Imagine the element array is an orthogonal matrix composed of a number of lines X elements long, and that the selected neighbours are the four elements directly above, below, to the left and to the right of the suspect element, the read sequence of the array being from left to right starting at the top. The first neighbouring element in the read sequence is the one above the suspect element, the last is the one below; at the end of the line stating with the first element comes the one to the left, while at the beginning of the line ending with the last element comes the one to the right. Thus, including the four neighbouring elements there are X elements in the sequence before the suspect element and X elements in the sequence after so including the suspect element itself there are 2X + 1 elements, and the shift register needs to be at least 2X + 1 cells long.Continuing this example, it can readily be seen that the relative positions of the register cells holding the output values for the suspect cell's neighbours are 1 and X before and 1 and X after the suspect cell itselfor, counting from the start of the register, that the "below" (the last) neighbour is in cell 1, the "above" (the first) in cell 2X + 1, and that the two "either side" neighbours are in cells X and X + 2 respectively.

The shift register is preferably exactly long enough so that the first neighbouring element value is in the last cell and the lavt value is in the first cell (other possibilities exit, but render system control and synchronisation rather difficult).

The inventive apparatus incorporates comparator means for recognising when the value in the cell holding the suspect element output value is indicative of a malfunctioning element. It is possible to make this comparison upon the basis of the value itself if below some threshold level, say, it means that that element is "dead' '-but this mode may cause problems when the detection system is looking at a scene that does in truth give output values below the threshold level. Accordingly, it is preferred that the comparator means operates not upon the value in the cell but upon the address of the element giving rise to that value, the address being compared against the next address in a sequentiallyarranged list of the addresses of known (i.e., predetermined) malfunctioning elements.In a very much preferred aspect, therefore, the apparatus of the invention includes a second shift register holding in sequence the addresses of the detector elements whose output values are held in sequence in the first shift register. This second shift register is hereafter referred to as the address shift register, while the first shift register is referred to as the output value shift register.

Just as the output value shift register is long enough and wide enough, so is the address shift register (but see below). Thus, matching the former the latter has sufficient cells that when one predetermined cell holds the address of the suspect element being investigated a (preferably the) last predetermined cell holds the address of the first neighbouring element to be scanned during the read sequence while a (preferably the) first predetermined cell holds the address of the last neighbouring element to be scanned during the read sequence, the addresses of all the other neighbouring elements, and of the suspect element itself, being in predetermined cells between these two.

It is possible that the address shift register might have less cells than the number of elements from (and including) the first neighbour to the last neighbour, the "gaps" being filled not by cells but by means for adjusting the address in the cell preceding the gap to be the correct address when shifted into the cell succeeding the gap. However this would cause severe problems in operation, not least when using an element array involving the output of video blanking pulses at each line end, and it is as stated much preferred to employ an address shift register having a number of cells exactly matching/parallelling the output value shift register.

When using an address shift register the apparatus includes means for comparing the address in the suspect element cell therein with the address of the next known malfunctioning element. As explained in the Specification of the aforementioned Application No 8329470 (1/6768/ELL), a detection system of the kind using the apparatus of the present invention is conveniently initialized (when turned on) by a process involving looking at two temperature-spaced uniform scenes (one hot, one cold) and calculating gain and offset correction factors, and where either factor is indicative of a malfunctioning element (say, the gain is negligable) then the address of that element is stored in a list.The address comparison means employed in the present invention preferably makes use of this list; the next known malfunctioning element address is latched into a comparator, which then checks it against the address in the suspect element cell of the address shift register, and if they are the same then an average output value is computed and inserted (and the next malfunctioning element address is taken from the list).

As just referred to, the apparatus of the invention includes calculator means for computing, from the output values in the neighbouring element cells, an average output value, and for inserting that output value as a replacement into the cell holding the suspect element output value. This calculating means is activated to insert the computed value whenever the output value in the suspect element cells is identified as that of a malfunctioning element (when its address matches the present one from the list of malfunctioning element addresses, for example). Because the output value shift register cells holding the neighbouring cell values are predetermined it is a simple matter to copy their contents into an averaging module (a simple "add-and-divide-by-four" device, say) and to insert the result straight into the suspect element cell in the output value shift register.A typical device for such an add-and-divide function is a combination of six 4-bit adders (74 LS 283 ICs) and an "AND" gate, and is described in more detail with reference to the accompanying drawings.

The output values going into the output value shift register may be the output values before or after normalisation, but are preferably the latter and when the latter the values coming out of the shift register may either be used directly to form a display or may first be stored and thereafter used to form the display.

The apparatus of the invention may be used, instead of the software method there described, to deal with "dead" elements in a normalising set up as in the aforementioned Application No 8329470 (1/6768/ELL), or in the more advanced normalising set up described and claimed in the Specification of our copending Application No.......... No .. (Publica- tion No ) (1/6770/ELL).

The invention naturallv extends to an imaging system, particularly a thermal imaging system, when using an apparatus as described and claimed herein.

An embodiment of the invention is now described, though by way of illustration only, with reference to the accompanying drawings in which: Figure 1 is a schematic view of a thermal imaging system employing the method and apparatus of the invention; Figure 2 is a simplified block diagram showing the method and apparatus of the invention; Figures 3A and B are more detailed versions of parts of Figure 2; and Figure 4 is a more detailed version of part of Figure 3.

In Figure 1 there is shown a thermal imaging camera (11) viewing a scene !12) and providing a visible image of that scene in a television-type Visual Display Unit (1 3) via a processing module (14). The camera 11 employs an IR detector element array (not shown separately) and the processing module 14 normalises the output of that array, including correcting for malfunctioning elements in accordance with the invention, to enable a meaningful image to be formed on the VDU 13. The nature of the processing module 14 is shown in more detail in Figure 2.

The camera 11 contains an IR detector element array (21 in Figure 2, shown as a 32 X 32 matrix of elements) that is driven, by clock pulses originating in the system Master Clock (22), to output its contents element by element to a Gain and Offset Corrector (23) within the Processing Module 14. At the same time, and in synchronism, the Master Clock 22 causes an Address Generator (24) to provide the identifying address of the array element presently giving its output, and this address is sent to a Conversion Factor Store (25), causing it to output to the Gain and Offset Corrector 23 values for the Gain Correction Factor and Offset Correction Factor applicable to this particular element.The Corrector then normalises the element output (modifying it in accordance with first one and then the other Correction Factor), and outputs the resulting normalised value to a Dead Element Corrector (27), which replaces any values known to be those of malfunctioning elements (as stored in Dead Element Address Store 28) with average values derived from the malfunctioning elements's neighbours.

The Corrector 27 then passes each result to the Video Store (26) where it is placed in the correct location for the relevant element as determined by that element's Address (which is also fed to the Video Store from the Address Generator 24 via the Dead Element Corrector).

In due time the contents of the Video Store are read out and passed to a VDU for display.

Figure 3A represents part of the detector element array of Figure 2, while Figure 3B represents a block diagram of the Dead Element Corrector 27 in Figure 2.

Figure 3A shows the relative positions, in a 32-by-32 element array, of the presently suspect element and its nearest neighbours (Nl,N4,N2,N3 respectively above, below, left,right). Using a system similar to that described in the Specification of the aforementioned Application 8329470 (1/6768/ELL) the address of this element, if actually "dead", is found and stored in memory.

When operating the present invention the address of the first dead element due to come downstream is put in a Latch (31) connected to one side of a digital Comparator (32). The address of all the detector elments, and their output values after correction for non-uniformity of array response, are then fed respectively into an Address Shift Register (33) and an Output Value Shift Register (34), as shown, with an Adder (35) constantly averaging (by summing up and dividing by 4) the contents of the Output Value Shift Register cells 1,32,34 and 65 (which hold the values for the neighbours N1,N2,N3 and N4). When in the Address Shift Register the dead address reaches the suspect element cell (Cell No.33) the Comparator 32 recognises that both addresses are identical, and enables the output of the Adder 35 to write its "averaged" data into the Output Value Shift Register suspect cell (cell No.33). This then causes the Processor (36; or store) to write the next dead element address into the Latch 31, and hence in due course the next dead element value is corrected.

As a divide by 4 is merely a shift right of two places in binary, an Adder 35 is all that is required, the least significant two bits being simply ignored. A typical circuit for such an Adder is shown in Figure 4.

As will immediately be apparent, the system operates with only a two line delay from input to output, and simply by altering the size of the Shift Registers (the only special devices in the circuitry) any size of array can be used.

The Latch 31 at the comparator 32 is conveniently filled directly from RAM which has previously been filled with the addresses of dead elements by the microprocessor 36 or from a PROM. Access is from an Address Generator (not separately shown in Figure 3B) incremented by the output of the Comparator and reset by the System Array field drive.

Hence, when the first dead element address is found by the Comparator it increments by 1 the address to the RAM/PROM, thus obtaining the next address. When the last dead address has been read and found the address to RAM/PROM address store is incremented and an address either outside the size of the array or at the beginning of the next field is read from it (it is immaterial which as, at the end of each field, the address to the RAM/ ROM can be reset to the base address provid ing the first dead element, and hence made ready to continue for the field following).

An embodiment of the Add-and-divide-by-4 unit 35 is shown in Figure 4. It uses six standard 74 LS 283 ICs 4-bit adders with carry and operates on the four inputs N1, N2, N3 and N4 from the Output Value Shift Register 34 as follows: Two ICs (41, 42) add the N4, N3 outputs to produce a 9-bit output (8 + a carry), while two ICs (43, 44) do the same to the N2, N1 outputs. The most significant eight bits of the resulting 9 bit numbers are fed to two more ICs (45, 46+the least significant bits of the two 9 bit numbers are fed to an AND gate (47) to determine if the carry into IC 46 is set, and then discarded (a first . 2) to produce a final 9 bit result; the least significant bit of this is discarded (a second . 2), and the remainder is the required average, (A + B + C + D)/4.

Claims (7)

1. Apparatus for providing a computed output value to replace the output value of a malfunctioning detector element within a matrix of elements, the computed value being derived from the output values of the element's neighbours in the matrix, which apparatus comprises: a shift register, to which is fed in the read sequence data defining the output values of the detector elements, this shift register having sufficient cells (locations) such that when one predetermined cell holds the output value of the suspect element being investigated a last predetermined cell holds the output value of the first neighbouring element to be scanned during the read sequence while a first predetermined cell holds the output value of the last neighbouring element to be scanned during the read sequence, the output values of all the other neighbouring elements, and of the suspect element itself, being in predetermined cells between these two; comparator means for recognising when the value in the cell holding the suspect element output value is indicative of a malfunctioning element; and calculator means for computing, from the output values in the neighbouring element cells, an average output value, and for inserting that output value as a replacement into the cell holding the suspect element output value.

2. Apparatus as claimed in Claim 1, wherein the computed value is the arithmetical average of the output values of those elements actually adjacent the malfunctioning element above, below, to the left, and to the right.

3. Apparatus as claimed in either of the preceding Claims, wherein the shift register is exactly long enough so that the first neighbouring element value is in the last cell and the last value is in the first cell.

4. Apparatus as claimed in any of the preceding Claims, wherein the comparator means operates upon the address of the element giving rise to the value in the cell holding the suspect element output value, the address being compared against the next address in a sequentially-arranged list of the addresses of known malfunctioning elements, and the apparatus includes a second shift register holding in sequence the addresses of the detector elements whose output values are held in sequence in the first shift register.

5. Apparatus as claimed in Claim 4 which uses an address shift register and includes means for comparing the address in the suspect element cell therein with the address of the next known malfunctioning element, and wherein in operation the next known malfunctioning element address is latched into a comparator, which then checks it against the address in the suspect element cell of the address shift register, and if they are the same then an average output value is computed and inserted.

6. Apparatus as claimed in any of the preceding Claims and substantially as described hereinafter.

7. An imaging system, particularly a thermal imaging system, when using an apparatus as claimed in any of the preceding Claims.