WO1996018976A1 - Image processing - Google Patents
Image processing Download PDFInfo
- Publication number
- WO1996018976A1 WO1996018976A1 PCT/GB1995/002942 GB9502942W WO9618976A1 WO 1996018976 A1 WO1996018976 A1 WO 1996018976A1 GB 9502942 W GB9502942 W GB 9502942W WO 9618976 A1 WO9618976 A1 WO 9618976A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- amplitude
- signal
- measure
- window
- noise
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/262—Analysis of motion using transform domain methods, e.g. Fourier domain methods
Definitions
- This invention relates to image processing and more particularly to the correlation of images, for example to identify movement.
- the invention provides for the signal processing of video signals to provide a measure of correlation between two successive fields or other corresponding inputs. Correlation between successive video fields enables motion vectors to be identified in a video sequence.
- Phase correlation is a known method by which motion vectors can be identified in a video signal. Reference is directed, for example, to:
- the phase correlation process takes blocks of luminance information at field intervals.
- a Fast Fourier Transfer (FFT) is performed to produce amplitude and phase signals.
- the phase signal is subtracted from a one-field-delayed signal to produce a phase difference signal.
- This is recombined with the amplitude signal in an inverse FFT to produce a correlation surface.
- candidate motion vectors can be identified for subsequent allocation on a pixel-by- pixel basis.
- a noise threshold is set, usually as a result of trial and error; components below the threshold are set to zero and components above the threshold are set to unity. Usually, the resulting, noise gated spectrum is then windowed.
- the form of the window is chosen to optimise the performance of the particular form of peak hunter chosen.
- One previous suggestion is to employ a Gaussian window with a peak hunter optimised to find quadratic peaks.
- the correlation surface undergoes logarithm processing before the peak hunter; the logarithm of a Gaussian produces a quadratic.
- the prior approach is basically successful but has a number of limitations.
- noise in the correlation surface hinders the accurate identification of peaks in the correlation surface, and thus reduces the likelihood of correct motion vectors being generated.
- the present invention consists, in one aspect, in a method of image processing to provide a measure of correlation between two images, comprising the steps of defining a set of corresponding samples in each image; performing transforms on said sample sets in each image to derive separate phase and amplitude signals; deriving a phase difference between said images; noise processing the amplitude signal by removing components beneath a noise threshold; and performing an inverse transform on the phase difference signal and noise-processed amplitude signal to provide a correlation signal, characterised in that the step of noise processing the amplitude signal comprises the steps of taking a measure of the amplitude difference between said images over said set and varying said noise threshold dynamically in response to variations in said measure.
- the measure of the amplitude difference between said images over said set is an averaged value.
- the measure is a root mean square.
- said set of samples comprises a corresponding block of pixels in successive video fields.
- the present invention recognises that noise in the correlation surface can arise from large spectral differences. For example, when areas of the picture change in a manner which is not simply related to movement of an object between two fields, the associated phase difference information cannot help in identifying a motion vector. However, the associated amplitude signal - because it is large - is above any sensible noise threshold. Therefore, the "useless" phase difference information contributes fully to the correlation surface. This situation arises with regions of the picture that are revealed or obscured between the two fields of interest and also where there is motion which is exceptionally fast.
- the method of the present invention is able to discriminate between - for example - a small but significant amplitude in two generally similar fields and a large but insignificant amplitude in fields which differ widely.
- the associated phase difference can be identified with movement; in the latter case, the differences between the two input fields are so great that no high precision measurement of motion is possible.
- a second limitation of the prior art approach is related to the form of windowing. It is recognised that appropriate windowing in the frequency domain, which is equivalent to filtering in the time domain, is of considerable benefit in the subsequent flitering steps.
- Various types of window have been suggested and the particular benefit of a Gaussian window has already been mentioned. Hitherto, however, it has been assumed that the signal to be windowed is a full spectrum window that is to say a signal with significant contributions from all frequency bands in the bandwidth.
- the present invention in a different aspect, recognizes that for significant numbers of real pictures, this assumption breaks down and breaks down in a manner which can now be seen to lead to material errors in motion estimation.
- phase correlation according to the prior art tends to produce a correlation surface with spurious side peaks and undershoots which can completely mask a neighbouring "real" peak.
- picture material containing a moving one-dimensional object a similar ringing phenomenon is observed.
- the present invention consists in a method of image processing to provide a measure of correlation between two images, comprising the steps of performing a transform to derive separate phase and amplitude signals; deriving a phase difference between said images; window processing the amplitude signal by attenuating components outside a defined frequency window; and performing an inverse transform on the phase difference signal and window-processed amplitude signal to provide a correlation signal, characterised in that the step of window processing the amplitude signal comprises the steps of determining the spectral content of the amplitude signal, generating a window function dynamically in accordance with said determination, and applying the window function to the amplitude signal.
- the step of determining the spectral content of the amplitude signal comprises identifying those discrete frequencies at which significant amplitude information exists.
- a count is made of those discrete frequencies at which significant amplitude information exists.
- the amplitude function will generally have a predetermined shape - such as a Gaussian - and will vary in dimension to fit the spectral content of the signal.
- the width parameter in one embodiment, is proportional to the count of those discrete frequencies at which significant amplitude information exists.
- selecting the width of the Gaussian to include a set proportion of said discrete frequencies such as selecting the width of the Gaussian to include a set proportion of said discrete frequencies.
- Figure 1 is a block diagram illustrating a phase correlation approach according to the prior art
- Figure 2 is a block diagram illustrating a modification according to the present invention
- Figures 3 and 4 are plots illustrating a simple moving object
- Figure 5 is a frequency spectrum for Figure 4.
- Figure 6 illustrates a prior art approach to gating and windowing the spectrum of Figure 5;
- Figure 7 is a plot similar to Figure 6 illustrating an approach according to the present invention.
- Figure 8 illustrates a prior art correlation surface (depicted in one dimension) created from the amplitude terms shown in Figure 6;
- Figure 9 is a plot similar to Figure 8 illustrating a correlation surface achieved through use of the present invention;
- Figures 10 to 16 correspond respectively with Figures 3 to 9, illustrating picture material having revealed detail
- Figures 17 to 23 correspond respectively with Figures 3 to 9, illustrating picture material having an out-of-focus moving object
- Figures 24 and 25 depict a moving one dimensional object in a two dimensional system
- Figures 26 and 27 illustrate the spectrum of Figure 25, before and after coring, respectively;
- Figure 28 depicts a prior art window function
- Figure 29 illustrates the prior art correlation surface resulting from use of the window depicted in Figure 28;
- Figure 30 illustrates adaptive windowing according to the invention.
- Figure 31 illustrates the correlation surface resulting from use of the window depicted in Figure 30.
- a known phase correlation process receives blocks of luminance picture information at field intervals, at input terminal (8).
- the blocks may be 64 by 64 pixels, overlapping such that every pixel of the picture is covered by four blocks.
- the sampled block has a Fast Fourier Transform performed on it in an FFT block (10), producing separate amplitude and phase signals, on lines (12) and (14), respectively.
- the phase signal (14) passes to one input of a subtracter (16) directly, and to the other input of the subtracter through a one-field delay (18).
- the derived phase difference signal outputted from the subtracter (16) on line (20), being a measure of motion between the two fields, is taken to the phase input of an inverse FFT block (22).
- the amplitude signal (12) from the FFT block (10) is taken through a noise gate (24).
- the principle behind noise gating is to normalise any significant component to full amplitude and to set to zero any components below the noise threshold. As a practical matter, the noise threshold is set empirically.
- the noise-gated amplitude signal is then passed through a frequency domain window (26).
- the purpose of the window is to filter the correlation surface, windowing in the frequency domain being the same as filtering in the time domain.
- the shape of the window is typically Gaussian and the size is such that the window decays to zero at the edge of the (assumed complete) spectrum.
- the window is intended to maximise the amount of available band width without introducing ringing on the correlation surface. Ringing is produced by having a sharp transition in the spectrum.
- the noise-gated and windowed amplitude terms (28) are then taken to the inverse FFT block (22).
- the result of the inverse FFT is a correlation surface containing peaks, where the position of a peak represents a motion between the two fields.
- a peak hunter (30) which may for example be as described in WO-A- 94 01830, operates on the correlation surface to produce a menu of motion vectors for subsequent selection and allocation pixel-by-pixel.
- the amplitude signal (12) identified from Figure 1 is supplied directly and through one field delay (130) to an arrangement (132) serving to derive the RMS value of the inter-field spectral difference.
- This arrangement comprises a subtracter (134), a modulus block (136), a squaring block (138), a summing block (140) and a square root block (142).
- the summing and square root blocks are controlled through a block-enable signal provided at a terminal (144). In this way, the RMS inter-field spectral difference is available block-by-block.
- This difference signal forms the noise threshold for the otherwise conventional noise gate (24) identified from Figure 1.
- the output from arrangement (132) is taken as the threshold signal to noise gate (24) which through a balancing block delay (146), also receives the amplitude signal.
- the described arrangement (132), providing the RMS value of the difference between the two fields over the block is but one example of numerous arrangements capable of providing a measure of the amplitude difference between two images over the pixel block (or other set of samples).
- the noise-gated amplitude terms pass in turn to a spectral distribution measurement block (148) and an amplitude component generation block (150).
- the amplitude component generation block (150) generates components by forming the product of the noise-gated amplitude terms with a window function derived from a measurement of spectral distribution in block (148). This may be contrasted with the prior art approach of applying a fixed window function.
- the first stage of the process is to find the RMS level of the spectral difference between the two fields, measured over the total block area.
- the present invention recognises that "noise" in the correlation surface, in the sense of information which is not related to movement between the two fields, can arise from large spectral differences.
- the associated phase difference information cannot help in identifying a motion vector.
- the associated amplitude signal - because it is large - is above any sensible, fixed noise threshold applied according to the prior art. Therefore, the "useless" phase difference information contributes fully to the correlation surface. This situation arises with regions of the picture that are revealed or obscured between the two fields of interest and also where there is motion which is exceptionally fast.
- the method of the present invention is able to discriminate between - for example - a small but significant amplitude in two generally similar fields which is likely to be associated with an identifiable motion vector and a large but insignificant amplitude in fields which differ widely.
- the amplitude component used for the inverse FFT is the noise-gated and windowed input spectrum.
- the present invention generates an amplitude component which is (in this case) a Gaussian that fits the available spectrum.
- the size of the Gaussian used is related to the number of active bins. This can be achieved in a variety of ways and two examples are:-
- the size of the Gaussian in this example is purely a function of the total number of active bins.
- Figures 3 and 4 illustrate schematically a simple object moving from field 1 to field 2.
- the spectrum is shown in Figure 5; this can be from either field but is more conveniently from field 2 to avoid the use of a delay.
- the spectrum shows the amplitude value for each of the frequency bins utilised in the FFT.
- a fixed noise threshold is employed, for example at the level A shown in Figure 5.
- Figure 6 shows the effect of prior art noise gating at this level A; amplitude terms beneath the threshold are zeroed and all others are set to unity.
- Figure 6 also shows the fixed Gaussian window function extending over the expected frequency range. It can be seen that as a result of the chosen noise level, a significant number of the frequency bins within the fixed window are empty.
- the resulting correlation surface (in one dimension) is shown in Figure 8.
- the amplitude values are cored not a predetermined fixed threshold, but at the level for the block in question of the RMS field difference.
- FIGs 10 and 11 there is shown an example of a moving object with revealed detail.
- the spectrum is shown in Figure 12 and the prior art noise gating and windowing (utilising a fixed noise threshold shown at A) is depicted in Figure 13. It should be noted that a large number of the frequency bins within the window are not active.
- Figure 14 shows first the result of coring at the RMS level, depicted at B. This produces more active bins than the arrangement of Figure 14, but less active bins than in Figure 7 of the previous example; due to the revealed differences resulting in a low spectral match factor. Because there are fewer active bins, a narrower amplitude term is derived. Indeed, in comparison with Figure 13, the "window" is very much narrower. The practical effects of these distinctions are evident from comparison of the respective correlation surfaces in Figures 15 and 16, respectively.
- the non-adaptive scheme of the prior art shown in Figure 15 produces an unsatisfactory result with large under shoots and very poor signal to noise ratio.
- the adaptive scheme produces in Figure 16 a slightly softer peak, reduced under shoots and improved signal to noise ratio.
- the coring level or threshold is significantly higher; hence the correlation surface is cleaner but a bit softer.
- Figures 17 and 18 depict the moving, out-of-focus object and the incomplete nature of the spectrum is clearly apparent from Figure 19.
- the inadequacies of the prior art approach are seen most clearly in Figure 20, where a very small fraction of the frequency bins within the window are active.
- the approach of the present invention produces, as seen in Figure 21 , more active bins and a window which is tailored to fit the available spectral information.
- Figure 21 more active bins and a window which is tailored to fit the available spectral information.
- Figures 22 and 23 totally unacceptable results are seen with the non-adaptive scheme of the prior art.
- the 100% undershoots which are produced at either side of the peak could totally remove other real motion peaks which fall at the same position.
- the adaptive scheme of the present invention manages to produce one soft peak with virtually no ringing.
- the spectral match factor is high, which allows nearly all of the available spectrum through the noise gate. But there are very few bins active, hence a narrow amplitude term is selected.
- Figures 24 and 25 depict a moving bar.
- the resultant spectrum is shown in Figure 26 and after coring, in Figure 27.
- Figure 28 shows the conventional Gaussian
- Figure 30 shows the result of tailoring the window to fit the amplitude spectrum of Figure 27.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
Description
Claims
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP95941169A EP0797813A1 (en) | 1994-12-15 | 1995-12-15 | Image processing |
AU42656/96A AU4265696A (en) | 1994-12-15 | 1995-12-15 | Image processing |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
GBGB9425328.3A GB9425328D0 (en) | 1994-12-15 | 1994-12-15 | Signal processing |
GB9425328.3 | 1994-12-15 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO1996018976A1 true WO1996018976A1 (en) | 1996-06-20 |
Family
ID=10765996
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/GB1995/002942 WO1996018976A1 (en) | 1994-12-15 | 1995-12-15 | Image processing |
Country Status (4)
Country | Link |
---|---|
EP (1) | EP0797813A1 (en) |
AU (1) | AU4265696A (en) |
GB (1) | GB9425328D0 (en) |
WO (1) | WO1996018976A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1997023844A1 (en) * | 1995-12-21 | 1997-07-03 | Philips Electronics N.V. | Directional adaptive noise reduction |
CN100388217C (en) * | 2004-11-16 | 2008-05-14 | 国际商业机器公司 | Dynamic threshold scaling method and system in communication system |
CN113867438A (en) * | 2021-09-27 | 2021-12-31 | 湖南省计量检测研究院 | Method and system for measuring and controlling temperature of electric heating furnace of lubricating oil evaporation loss tester |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0480807A1 (en) * | 1990-10-09 | 1992-04-15 | Thomson-Csf | Image matching process and device using a weighted phase correlation for determining a shift |
US5329368A (en) * | 1990-08-02 | 1994-07-12 | Hughes Aircraft Company | Image tracking system and technique |
-
1994
- 1994-12-15 GB GBGB9425328.3A patent/GB9425328D0/en active Pending
-
1995
- 1995-12-15 EP EP95941169A patent/EP0797813A1/en not_active Ceased
- 1995-12-15 WO PCT/GB1995/002942 patent/WO1996018976A1/en not_active Application Discontinuation
- 1995-12-15 AU AU42656/96A patent/AU4265696A/en not_active Abandoned
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5329368A (en) * | 1990-08-02 | 1994-07-12 | Hughes Aircraft Company | Image tracking system and technique |
EP0480807A1 (en) * | 1990-10-09 | 1992-04-15 | Thomson-Csf | Image matching process and device using a weighted phase correlation for determining a shift |
Non-Patent Citations (1)
Title |
---|
JOHN (JUYANG) WENG: "Image matching using the windowed Fourier phase", INTERNATIONAL JOURNAL OF COMPUTER VISION, vol. 11, no. 3, NORWELL US, pages 211 - 236, XP000397473 * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1997023844A1 (en) * | 1995-12-21 | 1997-07-03 | Philips Electronics N.V. | Directional adaptive noise reduction |
US6049623A (en) * | 1995-12-21 | 2000-04-11 | U.S. Philips Corporation | Directional adaptive noise reduction |
CN100388217C (en) * | 2004-11-16 | 2008-05-14 | 国际商业机器公司 | Dynamic threshold scaling method and system in communication system |
CN113867438A (en) * | 2021-09-27 | 2021-12-31 | 湖南省计量检测研究院 | Method and system for measuring and controlling temperature of electric heating furnace of lubricating oil evaporation loss tester |
Also Published As
Publication number | Publication date |
---|---|
EP0797813A1 (en) | 1997-10-01 |
AU4265696A (en) | 1996-07-03 |
GB9425328D0 (en) | 1995-02-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7099518B2 (en) | Measurement of blurring in video sequences | |
KR100591470B1 (en) | Detection of transitions in video sequences | |
Ferzli et al. | No-reference objective wavelet based noise immune image sharpness metric | |
US7046307B1 (en) | Video signal noise level estimator | |
Amer et al. | Reliable and fast structure-oriented video noise estimation | |
EP0398861A2 (en) | Method for adaptively sharpening electronic images | |
JP2005176388A (en) | Noise elimination method of animation data | |
KR101034518B1 (en) | Image processing apparatus for and method of improving an image and an image display apparatus comprising the image processing apparatus | |
US8515202B2 (en) | Reduction of blocking artifacts in image decompression systems | |
Urhan et al. | Modified phase-correlation based robust hard-cut detection with application to archive film | |
EP0797813A1 (en) | Image processing | |
KR20060058703A (en) | Block artifacts detection | |
Balamurugan et al. | Source camera identification using SPN with PRNU estimation and enhancement | |
JPH07121703A (en) | Image processing method | |
Fischer et al. | Electronic still picture camera spatial frequency response measurement | |
Sari et al. | Development of denoising method for digital image in low-light condition | |
KR100213235B1 (en) | Filtering method and the low pass filter for adaptable edge low pass filter | |
Kao | Modification of the LULU operators for preservation of critical image details | |
Stentiford | A visual attention estimator applied to image subject enhancement and colour and grey level compression | |
Pan et al. | Smoothing images by a probability filter | |
Chen et al. | Image quality measurement based on statistics of activity regions | |
EP0425288B1 (en) | Movement detection apparatus | |
Borga et al. | FSED-feature selective edge detection | |
Yung et al. | New feature preserving noise removal algorithm based on the discrete cosine transform and the a prior knowledge of pixel type | |
Subramanium et al. | A Hybrid Optimum Method for Removal of Haze using Weight Map Estimation and Fusion Strategy |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AK | Designated states |
Kind code of ref document: A1 Designated state(s): AU CA JP US |
|
AL | Designated countries for regional patents |
Kind code of ref document: A1 Designated state(s): AT BE CH DE DK ES FR GB GR IE IT LU MC NL PT SE |
|
DFPE | Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101) | ||
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
WWE | Wipo information: entry into national phase |
Ref document number: 1995941169 Country of ref document: EP |
|
WWP | Wipo information: published in national office |
Ref document number: 1995941169 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: CA |
|
WWR | Wipo information: refused in national office |
Ref document number: 1995941169 Country of ref document: EP |
|
WWW | Wipo information: withdrawn in national office |
Ref document number: 1995941169 Country of ref document: EP |