WO2016128700A1

WO2016128700A1 - Processing multispectral images

Info

Publication number: WO2016128700A1
Application number: PCT/GB2015/051968
Authority: WO
Inventors: Gary John BISHOP; John Peter Oakley
Original assignee: BAE Systems PLC
Current assignee: BAE Systems PLC
Priority date: 2015-02-11
Filing date: 2015-07-07
Publication date: 2016-08-18
Anticipated expiration: 2017-08-11
Also published as: GB2535175A; GB201502221D0

Abstract

A method of processing a multispectral or hyperspectral image captured in respect of an area of interest including a variably reflective surface, said image comprising at least one visible-wavelength signal and a NIR signal, the method comprising: -performing an image matching process using an iterative robust estimation method in respect of data points derived from said at least one visible-wavelength signal so as to estimate a scaling coefficient representative of specular reflection from said surface; -applying said scaling coefficient to said NIR signal such that a specular reflection component of the resultant scaled NIR signal is substantially the same as that of said at least one visible-wavelength signal; and -subtracting said scaled NIR signal from said at least one visible- wavelength signal to generate a visible-wavelength output image signal.

Description

PROCESSING MULTISPECTRAL IMAGES

This invention relates generally to an apparatus and method for processing multispectral images and, more particularly but not necessarily exclusively, to processing of multispectral images, obtained during remote sensing of a body of water, so as to compensate for solar glint.

Remote sensing techniques are known for monitoring the sea, and other large bodies of water, and thus detecting underwater targets, hazards and activity. Such techniques tend to employ airborne spectrographic imaging systems, for collecting multispectral images representative of radiation from an area of interest. In general, it is the image data collected at the visible wavelengths that are employed to analyse a body of water in this regard.

However, specular reflection of solar radiation, or 'glint', on non-flat water surfaces can present a significant problem to an optical system viewing through the water. Glint is always present to some extent when imaging the sea surface in visible and infra-red wavelengths, and is generally regarded as a nuisance effect, which causes white bands within the visible-wavelength images that hamper visual identification of benthic features and strongly influence image classification.

Hochberg, E.J., Andrefouet, S., and Tyler, M.R., 2003, Sea surface correction of high spectral resolution Ikonos images to improve bottom mapping in near-shore environments. IEEE Transactions on Geoscience and Remote Sensing, 41 , pp. 1724-1729 proposes a method of mitigating sea glint in visible images, based on the principle that absorption of light by sea water at Near Infra Red (NIR) wavelengths is high and so almost all of the lightness in such images is due to surface glint. Thus, in the proposed method, the sun-glint component of the remotely sensed signal is removed from visible wavelength spectral bands by utilization of information from a spectral band in near-infrared (NIR, ~700-910nm); and this method is therefore applicable to imagery from sensors that include an NIR band (e.g. satellites) or suitably configured multispectral and hyper spectral imagery (e.g. from airborne sensors).

Ultimately, image pixels are adjusted to remove the glint component of the recorded signal, thereby leaving only the component derived from benthic reflectance and radiative transfer processes within the water column.

The method described by Hochberg, et al (2003) relies on two

assumptions: (1 ) that the brightness in the NIR is composed only of sun glint and a spatially 'ambient' NIR component; and (2) that the amount of sun glint in the visible bands is linearly related to the brightness in the NIR band. An apparent weakness in the proposed method is that, in order to establish the linear relationship between the NIR brightness and sun glint in the visible wavelengths, only two pixels are used, and the resultant method is therefore unduly sensitive to outlier pixels, and requires numerous and complex processing steps.

Hedley, J.D., Harborne, A.R. and Mumby, P.J., Technical note: Simple and robust removal of sun glint for mapping shallow-water benthos.

International Journal of Remote Sensing, 26(10): 2107-21 12, 2005 describes a revised version of Hochberg, et a/'s method in which the linear relationships between NIR and visible bands are established using linear regression based on a sample of image pixels. One or more regions of the image are selected where a range of sun glint is evident, but where the underlying spectral brightness would be expected to be consistent, i.e. areas of the sea surface that do not contain any surface objects. For each visible band, all of the selected pixels are included in a linear regression of NIR brightness against the visible band brightness, and the value of all pixels in each band is reduced by the product of the regression slope and the difference between the pixel NIR value and the ambient NIR value. In other words, the pixel values in each spectral band are adjusted using the results of the above-mentioned linear regression, which is specifically dependent on the visible band brightness within the spectral band to which it relates.

However, whilst the technique described by Hedley, et al provides a clear visual improvement over previous remote benthic sensing techniques, and is less sensitive to outlier pixels than the method proposed by Hochberg, et al, there is still an element of distortion in the processed images, caused by the use of regression to estimate a scaling coefficient in respect of the pixel values, because the scaling coefficient estimate is inevitably distorted by outliers and, whilst the selection of areas of the sea surface that do not contain any surface objects is intended to ameliorate this issue to a certain extent, it is not necessarily wholly effective and, in any event, adds a restriction and additional computational cost.

It is therefore an object of some aspects of the present invention to address this issue, with the aim of providing an approach that offers an improved estimate for the scaling coefficient.

In accordance with a first aspect of the present invention, there is provided a method of processing a multispectral or hyperspectral image captured in respect of an area of interest including a variably reflective surface, said image comprising at least one visible-wavelength signal and a NIR signal, the method comprising: performing an image matching process using an iterative robust estimation method in respect of data points derived from said at least one visible-wavelength signal so as to estimate a scaling coefficient representative of specular reflection from said surface; applying said scaling coefficient to said NIR signal such that a specular reflection component of the resultant scaled NIR signal is substantially the same as that of said at least one visible-wavelength signal; and subtracting said scaled NIR signal from said at least one visible- wavelength signal to generate a visible-wavelength output image signal.

The term "iterative robust estimation method" will be known to a person skilled in the art, but for the avoidance of doubt the term used herein is intended to mean any one of a number of statistical estimation methods that have the common characteristic of ignoring data which are outliers.

By using a robust estimation technique to estimate the scaling

coefficient, the above-mentioned restriction associated with regression techniques, namely the need to select areas of the image that do not contain surface objects, is obviated. ln one exemplary embodiment of the present invention, the image matching process may comprise using a Random Sample Consensus algorithm to estimate said scaling coefficient using, as said data points, brightness values at respective specified locations within said at least one visible-wavelength signal. However, other robust estimation techniques are known. For example, the trimmed mean method is a simple robust estimator that deletes a certain percentage (e.g. 10%) from each end of the data, and then computes the mean in a conventional manner. Another known method, termed Winsorizing, involves accommodating outliers by replacing them with the next highest or smallest value, as appropriate. The present invention is, therefore, not necessarily intended to be limited in this regard.

Furthermore, the inventors have identified that sea glint can provide a useful spectral reference that may be exploited, in some exemplary

embodiments of the invention, to detect and classify targets on or below the sea surface.

Thus, in an exemplary embodiment, wherein the image comprises a frame consisting of a plurality of picture elements, the method may further comprise the step of classifying the picture elements of an image frame based on intensity values of said visible-wavelength output image signal corresponding to respective picture elements.

In this case, the step of classifying may include determining if said intensity values are significantly positive or negative.

Aspects of the present invention extend to a method of processing a multispectral or hyperspectral image frame captured in respect of an area of interest including a body of water and classifying targets identified in said image, said image frame consisting of a plurality of picture elements comprising at least one visible-wavelength signal and a NIR signal, the method comprising: performing the method of claim 1 in respect of said image frame to generate a visible wavelength output image comprising one or more visible- wavelength image components representative of respective picture elements; analysing the or each visible wavelength component of a picture element to determine an intensity value thereof; and classifying a target represented within a picture element as a subsurface target if said intensity value is statistically significantly positive and as a surface target if said intensity value is statistically significantly negative; and otherwise classifying image data within said picture element as water surface

A person skilled in the art will readily understand that the term "statistically significant", which is usually referenced with respect to standard deviation (sigma) of data. Thus, in this case, if the image noise level is sigma, then values of greater than 3^*sigma would be statistically significant.

The image frame and the visible wavelength output image may include a plurality of image signals corresponding to respective visible wavelengths, and a target represented within a picture element may be classified as a sub-surface target if the intensity values of all of the image signals are significantly positive. Typical image signal-to-noise ratio (SNR) values of around 46dB correspond to pixel standard deviation of one part in 255. Thus, the threshold for determining statistical significance in this case, and in an exemplary embodiment of the invention, could be set to 3 grey levels. However, this will be dependent on application and system parameters, characteristics and constraints, and the present invention is not necessarily intended to be limited in this regard.

Aspects of the present invention extend further to apparatus for processing a multispectral or hyperspectral image captured in respect of an area of interest including a variably reflective surface, the apparatus comprising an input for receiving a multispectral or hyperspectral image comprising at least one visible-wavelength signal and a NIR signal, and at least one processor for performing the method defined above.

Aspects of the present invention extend still further to a multispectral or hyperspectral imaging and analysis apparatus, comprising: a multispectral or hyperspectral imaging device for capturing a multispectral or hyperspectral image comprising at least one visible-wavelength signal and a NIR signal; at least one processor configured to perform the method described above; and an output for displaying data representative of said visible- wavelength output image signal.

In accordance with yet another aspect of the invention, there is provided a program or plurality of programs arranged such that when executed by a computer system or one or more processors, it/they cause the computer system or the one or more processors to operate in accordance with the method described above.

This aspect of the invention may extend still further to a machine readable storage medium storing a program or at least one of the plurality of programs as defined above.

These and other aspects of the present invention will become apparent from the following specific description in which embodiments of the present invention are described, by way of examples only, and with reference to the accompanying drawing, in which: Figure 1 is a schematic flow diagram illustrating the principal steps of an image processing method according to an exemplary embodiment of the present invention.

Referring to Figure 1 of the drawings, there is illustrated a flow chart illustrating the principal steps of a method of multispectral (or hyperspectral) image processing according to an exemplary embodiment of the present invention. Thus, the method starts, at step 200, with the input of a captured image from a multispectral or hyperspectral imaging system. For the purposes of the following description, the proposed method will be described in relation to remote sensing of the sea to detect targets, but it will be appreciated that at least some aspects of the invention may be applicable to other applications, and the present invention is not necessarily intended to be limited in this regard. Thus, in this case, the multispectral images provided as the input to the image processing method may be captured using an airborne spectrographic imaging system such as a Compact Airborne Spectrograhic Imager (CASI) or the like. However, once again, the present invention is not necessarily intended to be limited in this regard, and a person skilled in the art will be aware of many types of multispectral and hyperspectral imaging systems that could be used as an alternative.

Sequential multispectral and hyperspectral imaging is an acquisition technique that involves collecting images of a target or an area of interest at different wavelengths, to compile a spectrum for each pixel. Multispectral or hyperspectral imaging systems have the ability to provide a continuous graph of the electromagnetic emission from or absorption by a sample of material across a range of the electromagnetic spectrum, and the particular output from the imaging system is dependent on the channels selected by a user, whereby the channels correspond to specified wavelength bands. Thus, for the purposes of this exemplary embodiment of the invention, it is assumed that a single visible- wavelength band (corresponding to any of the wavelengths in the range (380 - 700nm) and a single NIR band (corresponding to any of the wavelengths in the range (750 - 1400nm) have been selected, such that the input at step 200 of the method illustrated in Figure 1 of the drawings comprises a set of image frames comprising a single visible image signal and a single NIR image signal, wherein each image frame can be considered to comprise a plurality of pixels which may correspond to an imaged area of, say, 1 m².

Due to the nature of multispectral acquisition, and the resultant images captured, there will inevitably be a slight misalignment between images captured in respect of the various different wavelengths and, in order to process the image frames further, it is necessary to correctly align the visible and NIR image signals therein. Thus, at step 202, an image alignment process is performed. Any known technique can be used for this purpose, as will be apparent to a person skilled in the art, for example, a matching technique is known which is configured to identify matching pixel pairs between the signals and perform image alignment by aligning the matching pixel pairs. Next a scaling coefficient is estimated at step 204. The aim of this step is to estimate a coefficient by which the NIR signal of each image frame needs to be scaled so that the glint component in the NIR and visible-wavelength image(s) is substantially the same. In other words, a substantially linear relationship needs to be defined between the glint in the visible band(s) and the brightness in the NIR band, which linear relationship is defined by a scaling coefficient. The visible-wavelength image signal(s) are used to estimate the scaling coefficient and, in accordance with aspects of the present invention, a robust image matching technique is employed for this purpose, which does not rely on regression or correlation to establish the above-mentioned linear relationship between the glint in the visible band(s) and the NIR brightness. In an exemplary embodiment of the present invention, a RANSAC (Random

Sample Consensus) algorithm is employed to estimate ^x , given a fitting problem with (in this case) scaling coefficient ^x . Assume:

• ^x can be estimated from N data items

• there are M data items in total

(wherein a data item can be considered to be a measure of brightness in the visible image at a particular location therein) the algorithm:

(i) selects N data items at random; estimates ;

(iii) finds how many data items (K) of M fit the model with parameter vector ^x within a user given tolerance;

(iv) if K is big enough, exits with success; and

(v) repeats 1 - 4 L times, where L can be found by:

L

log(l - (/? ) and wherein p_fan is the probability that the algorithm will exit without success, and p_g is the probability that a randomly selected data item is part of a good model.

Once ^x , i.e. the scaling coefficient, has been determined by the above- described method from a particular visible image signal (or number of signals) within a frame, it is applied to the corresponding NIR image signal in that frame, at step 206. The resultant, scaled NIR image signal is referred to herein as the estimated glint component.

Next, at step 208, the scaled NIR image signal, or glint component, is subtracted from the visible image signal(s), and the resultant image signal is subjected to post processing at step 210 before being output at step 212.

It will be appreciated that the scaling coefficient estimation step 204 does not necessarily need to be performed for each image frame in a sequence. In view of the fact that the scaling coefficient is not expected to change

significantly between successive image frames, it is envisaged that the scaling coefficient could be estimated for every N frames, so as to reduce

computational load. Thus, once the scaling coefficient has been estimated for a frame, the same scaling coefficient could be applied to the next few frames, before the estimation step needs to be performed again. After the glint component has been subtracted from the visible image signal(s) within a frame, it is possible, during post processing (step 210) to classify targets according to the resulting intensity values of the visible image signal(s) that remain. Each frame is analysed, pixel by pixel, and:

• if intensities at the or all visible wavelengths are significantly

positive, the pixel is classified as a sub-surface target;

• if intensities at the or any of the visible wavelengths are significantly negative, the pixel is determined to be a surface target;

• otherwise, the pixel is classified as sea surface. The change in refractive index of sea water between visible and NIR wavelengths is negligible an, therefore, rather than being a "nuisance" factor, the glint can provide a spectral reference for the purpose of classification of targets. Surface targets are generally painted to be clearly visible to the naked eye, but such pigments generally have complex absorption spectra and it is, therefore, likely that the reflectance at an NIR wavelength will exceed that of at least one visible wavelength. Subtraction of the scaled NIR image (or "glint component") will, therefore, lead to one or more negative values in the spectral components. On the other hand, a sub-surface target will have any NIR emission strongly attenuated by sea water and, therefore, subtraction of the scaled NIR image will not give negative spectral components.

A potential limitation in respect of the above-described classification method is that certain pigments exist that mimic sub-surface objects, in the sense that the reflectance thereof at the NIR wavelength is less than any of the visible wavelengths employed. This issue can be mitigated by grouping all target pixels together to form regions, such that if any region contains negative spectral components then the target should be classified as a surface target.

Although illustrative embodiments of the invention are described in detail herein with reference to the accompanying drawings, it is to be understood that the invention is not limited to these precise embodiments.

Furthermore, it is contemplated that a particular feature described either individually or as part of an embodiment can be combined with other individually described features, or parts of embodiments, even if the other features and embodiments make no mention of the particular feature. Thus, the invention extends to such specific combinations not already described.

Still further, it will be apparent to a person skilled in the art, from the foregoing description, that modifications and variations can be made to the described embodiments without departing from the scope of the invention as claimed.

Claims

1 . A method of processing a multispectral or hyperspectral image

captured in respect of an area of interest including a variably reflective surface, said image comprising at least one visible- wavelength signal and a NIR signal, the method comprising:

- performing an image matching process using an iterative robust estimation method in respect of data points derived from said at least one visible-wavelength signal so as to estimate a scaling coefficient representative of specular reflection from said surface;

- applying said scaling coefficient to said NIR signal such that a specular reflection component of the resultant scaled NIR signal is substantially the same as that of said at least one visible- wavelength signal; and

- subtracting said scaled NIR signal from said at least one visible- wavelength signal to generate a visible-wavelength output image signal.

2. A method according to claim 1 , wherein said image matching process comprises using a Random Sample Consensus algorithm to estimate said scaling coefficient using, as said data points, brightness values at respective specified locations within said at least one visible- wavelength signal.

3. A method according to claim 1 or claim 2, wherein said image

comprises a frame consisting of a plurality of picture elements, and the method further comprises the step of classifying the picture elements of an image frame based on intensity values of said visible- wavelength output image signal corresponding to respective picture elements.

4. A method according to claim 3, wherein said step of classifying

includes determining if said intensity values are significantly positive or negative.

5. A method of processing a multispectral or hyperspectral image frame captured in respect of an area of interest including a body of water and classifying targets identified in said image, said image frame consisting of a plurality of picture elements comprising at least one visible-wavelength signal and a NIR signal, the method comprising:

- performing the method of claim 1 in respect of said image frame to generate a visible wavelength output image comprising one or more visible-wavelength image components representative of respective picture elements;

- analysing the or each visible wavelength component of a picture element to determine an intensity value thereof; and

- classifying a target represented within a picture element as a subsurface target if said intensity value is statistically significantly positive and as a surface target if said intensity value is

statistically significantly negative; and otherwise classifying image data within said picture element as water surface

6. A method according to claim 5, wherein said image frame and said visible wavelength output image includes a plurality of image signals corresponding to respective visible wavelengths, and wherein a target represented within a picture element is classified as a sub-surface target if the intensity values of all of said image signals are

significantly positive.

7. Apparatus for processing a multispectral or hyperspectral image

captured in respect of an area of interest including a variably reflective surface, the apparatus comprising an input for receiving a multispectral or hyperspectral image comprising at least one visible- wavelength signal and a NIR signal, and at least one processor for performing the method of any of claims 1 to 6.

8. A multispectral or hyperspectral imaging and analysis apparatus, comprising: - a multispectral or hyperspectral imaging device for capturing a multispectral or hyperspectral image comprising at least one visible-wavelength signal and a NIR signal;

- at least one processor configured to perform the method of any of claims 1 to 6; and

- an output for displaying data representative of said visible- wavelength output image signal.

9. A program or plurality of programs arranged such that when executed by a computer system or one or more processors, it/they cause the computer system or the one or more processors to operate in accordance with the method of any of claims 1 to 6.

10. A machine readable storage medium storing a program or at least one of the plurality of programs, according to claim 9.