GB2280812A

GB2280812A - Deblurring image data using motion vector dependent deconvolution

Info

Publication number: GB2280812A
Application number: GB9316307A
Authority: GB
Inventors: Martin Rex Dorricott; Morgan William Amos David
Original assignee: Sony United Kingdom Ltd
Current assignee: Sony Europe BV United Kingdom Branch
Priority date: 1993-08-05
Filing date: 1993-08-05
Publication date: 1995-02-08
Anticipated expiration: 2013-08-05
Also published as: GB9316307D0; GB2280812B; JP3251127B2; JPH0765163A

Abstract

The deblurring deconvolution is dependent upon the capture time T of the image and the motion vector detected for that part of the image. A multiplexer 14 selects between deblurring deconvolutions performed in parallel in dependence upon a plurality of different candidate motion vectors V1, V2, V3 and zero motion. A region processor 10 identifies boundaries between areas within the image corresponding to different objects and controls a mixer 16 to selectively output deblurred values or unprocessed raw video values so as to reduce edge effects due to the inappropriate application of deblurring deconvolution at the edges between objects. Provision of parallel deconvolutions allows the system to operate in a continuous pipelined manner. <IMAGE>

Description

IMAGE ENHANCEMENT This invention relates to the field of image enhancement. More particularly, this invention relates to the enhancement of arrays of image data captured during a capture time such that moving images are blurred.

The blurring of moving images often occurs due to the non-zero capture times of image capture devices, such as stills cameras and television cameras. Blurring can be produced by decreasing the capture time, but this brings with it its own set of problems, such as the requirement for higher light sensitivity. The problems of image blurring are made worse when the scene contains rapidly moving images (e.g. motor racing) or the capture device itself is moving relative to the image (e.g. satellite photography where long capture times are also required).

It is known that blurring due to image motion can be reduced using deconvolution techniques. The image data value at a given point within the blurred image may be considered as a convolution over the capture time of the actual instantaneous image at that point and the motion. This convolution can be reversed given values of the variables representing the capture time and the motion of the image assuming the motion can be approximated to uniform linear motion.

A mathematical treatment of this technique may be found in "Digital Image Processing" by Gonzalez & Wintz published by Addison Wesley 1987, pages 224-232. The convolution in the spatial domain can be considered as multiplication by a transfer function in the spatial frequency domain, the transfer function being capable of solution in terms of the capture time and speed of movement of the image.

Such techniques have been used to enhance images, such as still images and satellite images, where the degree of motion present is either known from the outset (e.g. a fixed satellite orbit) or may be determined by trial and error attempts at deblurring. Whilst the deblurring technique can be effective, the problems of solving the transfer function have meant its application has been limited to specialist fields and it may be considered as something of a textbook curiosity.

Viewed from one aspect this invention provides apparatus for processing an array of image data captured during a capture time such that moving images are blurred, said apparatus comprising: means for detecting an array of motion vectors associated with corresponding portions of an array of image data, each of said motion vectors representing image motion of a corresponding one of said portions between temporally spaced arrays of image data; and means for deconvolving a blur function from each of said portions in dependence upon said capture time and a corresponding motion vector for that portion detected by said means for detecting to yield an array of reduced blur image data.

The invention both recognises and exploits that the systems that have been developed for motion compensated interpolation of image data to effect standards conversion can be utilised to detect motion vectors for moving images as part of an image deblurring system of general applicability. Furthermore, differing motion vectors can be identified for differing parts of a complex moving scene and the appropriate deblurring convolution applied to the corresponding parts of the scene.

This considerably enhances the effectiveness of the technique.

In order to ease the burden of identifying the motion vectors, it is preferred that said portions comprise arrays of pixel values.

Such an approximation of associating motion vectors with a block of pixel values provides an advantageous compromise between system processing requirements and the achieved accuracy of results.

Having identified motion vectors to be associated with each of the arrays of pixel values, it is important to provide at least some flexibility as to which motion vectors are actually utilised.

Accordingly, it is preferred that said means for detecting an array of motion vectors detects a plurality of candidate motion vectors for each of said arrays of pixel values and a means for selecting selects a motion vector to be associated with each pixel value from among said candidate motion vectors.

The provision of a plurality of candidate motion vectors for each of the portions (/arrays of pixel values) from among which a particular motion vector to be used is selected on a pixel by pixel basis, provides for differences between what the pixel values within a given portion in fact represent without inflicting unduly disadvantageous processing requirements.

The deconvolution process required to produce a particular output pixel value requires contributions from a number of surrounding pixel values. In order to improve the efficiency of the system, it is desirable that the image data should be input to the system as a continuous stream that can be passed through the system without undue requirements for buffering and the like. This presents a problem in the case of deconvolution that may be based upon one of a number of candidate motion vectors. In such a circumstance, the particular deconvolution that is required may not be identified until after at least some of the pixel values upon which the result depends have passed through at least part of the apparatus.In order to deal with this, it is preferred that said means for deconvolving performs a deconvolution upon each pixel value for each of said candidate motion vectors to yield a plurality of reduced blur pixel values and a multiplexer controlled by said means for selecting switches one of said candidate reduced blur image pixel values corresponding to said selected motion vector to form part of said array of reduced blur image data.

The performing of a plurality of deconvolutions when only one will ultimately be used, may seem at first sight to disadvantageously increase the amount of hardware that will be needed to implement the system. However, when consideration is made that the processing may then be made in a continuous pipelined fashion without the need to repeatedly access the same pixel data values, then an overall advantage results from such an approach.

As mentioned previously, the ability of the system to apply different motion vectors to different parts of the image allows a scene containing multiple moving images to be dealt with. Such scenes do however create problems at the boundaries between objects within the scene. As a first stage in dealing with such problems it is desirable that the system should include means for detecting boundaries between areas within said array of image data sharing a substantially common motion vector, sharing of a substantially common motion vector within such an area being indicative of said area representing a single object (the object may comprise a plurality of different parts, but they all move together).

Having identified the boundaries between objects, the problems that could be introduced by an inappropriate application of a deblurring deconvolution at a boundary between different objects can be reduced by providing a mixer responsive to said means for detecting boundaries for mixing together deblurred image data from said means for deconvolving and unprocessed image data such that at boundaries between detected areas substantially wholly unprocessed image data is output from said mixer with an increasing proportion of deblurred image data being output from said mixer upon moving away from said boundaries.

It is preferably that the mixer should select unprocessed blurred image data at boundaries rather than risk the introduction of the potentially more disturbing artifacts due to an inappropriate deblurring deconvolution being applied.

A particularly efficient embodiment of the means for deconvolving comprises a two dimensional finite impulse response filter having a set of filter coefficients selected in dependence upon motion vector components in orthogonal directions for that portion of image data being processed.

The realisation that the blurring transfer function operates in the spatial frequency domain of the image makes the use of a finite impulse response filter for transformation back into the spatial domain particularly suitable. Furthermore, such a filter is easy to operate in a pipelined fashion whereby the data is sequentially clocked along the filter to yield a filtered output result upon each clock cycle.

It will be appreciated that the filter coefficients for the finite impulse response filter is dependent upon the orthogonal motion vector components and the capture time of the image. In order to efficiently deal with such variations, it is preferred to provide a filter coefficient store for said filter storing a set of filter coefficients for a range of possible motion vectors and capture times that are triggered into use in dependence upon detected motion vector components and the particular capture time in question.

Viewed from another aspect the invention provides a method of processing an array of image data captured during a capture time such that moving images are blurred, said method comprising the steps of: detecting an array of motion vectors associated with corresponding portions of an array of image data, each of said motion vectors representing image motion of a corresponding one of said portions between temporally spaced arrays of image data; and deconvolving a blur function from each of said portions in dependence upon said capture time and a corresponding detected motion vector for that portion to yield an array of reduced blur image data.

An embodiment of the invention will now be described, by way of example only, with reference to the accompanying drawings in which: Figure 1 illustrates an apparatus for deblurring image data; and Figure 2 illustrates a two dimensional finite impulse response filter for effecting image deblurring.

Figure 1 shows an apparatus for image deblurring. A video data stream is input to a vector selector 2 and first, second and third deconvolution means 4, 6 and 8 respectively. Four candidate motion vectors (V1, V2, V3 and the zero motion vector) for a block/portion of the image are input to the vector selector 2. It will be appreciated that the candidate motion vectors are detected using apparatus and techniques such as those described in British Published Patent Application GB-A-2 231 749 (Sony Corporation) for the purpose of motion compensated standards conversion.

One of the candidate motion vectors is the zero motion vector.

The other candidate motion vectors V1, V2 and V3 are fed to the respective deconvolution means 4, 6 and 8.

The vector selector 2 selects from among the candidate vectors for the block (Vectors/b) on a pixel by pixel to yield basis that vector (Vectors/p) which is most suitable for a particular pixel. In accordance with the known technique, this selection may be achieved by projection from one area of image pixel values to another using each of the vectors and testing to determine how well the pixel values in the target matches those source.

The particular vectors selected for each pixel (Vectors/p) are passed to a region processor 10 where an array of vectors selected for all of the pixel values is assembled and analysed as will be described later. The vector selector 2 also outputs a pixel vector choice value to a delay unit 12 through which it passes to a multiplexer 14.

The multiplexer 14 receives the outputs from the first, second and third deconvolution means 4, 6 and 8 respectively. The multiplexer 14 also receives the unprocessed video data in one channel corresponding to the zero motion candidate motion vector. The multiplexer 14 selects one of its inputs for output to a mixer 16 in dependence upon the output of the delay 12. The delay 12 is chosen to match the delays in the vector selector 2 and first, second and third deconvolution means 4, 6 and 8. The first, second and third deconvolution means 4, 6 and 8 each serve to generate deblurred pixel values as will be described below.

The region processor 10 serves to identify boundaries between objects within a scene of moving objects. This is achieved by searching through the motion vector selected for the various pixel positions within the image and identifying areas showing substantially common motion vectors within a predetermined threshold of one another.

When these areas of common motion vectors have been identified then their boundaries may be traced to represent the boundaries of differing moving objects within the scene. In synchronism with the output of the deblurred pixel value from the multiplexer 14, the region processor 10 outputs a boundary flag via a delay unit 18 to the mixer 16. This boundary flag is a multi-bit word indicating the proximity of the pixel now being output to a boundary between objects within the scene. If the boundary flag indicates that the particular pixel is at the boundary between two objects, then the mixer 16 selects the unprocessed video for output.As the boundary flag progressively indicates that a pixel is further away from a boundary between moving objects, then the mixer 16 correspondingly selects a higher proportion of the output from the multiplexer 14 representing the deblurred pixels rather than the raw unprocessed pixels. The action of the mixer 16 and the region processor 10 serves to suppress the generation of disturbing artifacts around the borders of moving objects at which the deblurring deconvolutions would not be appropriately applied.

Figure 2 illustrates one of the deconvolution means 4, 6 and 8 in more detail. The deconvolution means 4, 6 and 8 is implementing using a two dimensional finite impulse response filter 20. This filter 20 has conventional pixel and line delays between taps. The input video data is pipelined through the filter 20 in sequence to generate the output.

The filter coefficients applied by the filters 20 are stored within a coefficient store 24. The coefficient stores 24 is supplied with the capture time of the video data and a value of the horizontal vector component and vertical vector component of the pixel values currently at the centre position within the filter 20.

The coefficient store 24 outputs n horizontal coefficient values and m vertical coefficient values to the filter 20 in dependence upon the horizontal vector component, the vertical vector component and the capture time. The values of the coefficients are determined in advance for particular combinations of vector component values and capture time using the mathematical treatment referenced in the introduction.

The blurring transfer function may be represented by:

where: u = horizontal spatial frequency; v = vertical spatial frequency; T = capture time; a = (horizontal motion vector) * T; and b = (vertical motion vector) * T.

Accordingly, the deblurring transfer function may be represented by:

By substituting into this equation the spatial frequencies corresponding to the differing horizontal pixel spacings and interline spacings in the vertical direction, a set of filter coefficient values for representative values of the capture time T and the motion vector might be derived. Unfortunately, the value of H,(u.v) becomes infinite at the zero crossing that occur within HB(U,V); the first zero crossing occurs at lua+vbl = 1.

One way of dealing with this to provide approximate solutions for the filter coefficients without the zero crossing problem is to only consider up to the first zero crossing of H,(u,v) and then apply a window function. An example of a window function W(u,v) that may be used is a raised cosine: W(u,v) = 1/2(1 + cos(x(ua+vb))) ... for Jua+vbl < 1 W(u,v) = O . . . elsewhere Thus, the applied non-separable deblur function becomes:

An alternative is to use a least-mean-square (Wiener) filter as described in the mathematical treatment referenced in the introduction.

The Wiener filter given by equation 5.5-10 in this reference (where K is a constant) is:

Claims

CLAIMS 1. Apparatus for processing an array of image data captured during a capture time such that moving images are blurred, said apparatus comprising: means for detecting an array of motion vectors associated with corresponding portions of an array of image data, each of said motion vectors representing image motion of a corresponding one of said portions between temporally spaced arrays of image data; and means for deconvolving a blur function from each of said portions in dependence upon said capture time and a corresponding motion vector for that portion detected by said means for detecting to yield an array of reduced blur image data.
2. Apparatus as claimed in claim 1, wherein said portions comprise arrays of pixel values.
3. Apparatus as claimed in claims 2, wherein said means for detecting an array of motion vectors detects a plurality of candidate motion vectors for each of said arrays of pixel values and a means for selecting selects a motion vector to be associated with each pixel value from among said candidate motion vectors.
4. Apparatus as claimed in claim 3, wherein said means for deconvolving performs a deconvolution upon each pixel value for each of said candidate motion vectors to yield a plurality of reduced blur pixel values and a multiplexer controlled by said means for selecting switches one of said candidate reduced blur image pixel values corresponding to said selected motion vector to form part of said array of reduced blur image data.
5. Apparatus as claimed in any one of claims 2, 3 and 4, comprising means for detecting boundaries between areas within said array of image data sharing a substantially common motion vector, sharing of a substantially common motion vector within such an area being indicative of said area representing a single object.
6. Apparatus as claimed in claim 5, comprising a mixer responsive to said means for detecting boundaries for mixing together deblurred image data from said means for deconvolving and unprocessed image data such that at boundaries between detected areas substantially wholly unprocessed image data is output from said mixer with an increasing proportion of deblurred image data being output from said mixer upon moving away from said boundaries.
7. Apparatus as claimed in any one of the preceding claims, wherein said means for deconvolving comprises a two dimensional finite impulse response filter having a set of filter coefficients selected in dependence upon motion vector components in orthogonal directions for that portion of image data being processed.
8. Apparatus as claimed in claim 7, comprising a filter coefficient store for said finite impulse response filter storing sets of filter coefficients for a range of possible motion vector components, detected motion vector component triggering selection of a corresponding set of filter coefficients.
9. Apparatus as claimed in claim 8, wherein said filter coefficient store stores a plurality of sets of filter coefficients for each possible motion vector, each of said plurality of sets corresponding to a different capture time.
10. A method of processing an array of image data captured during a capture time such that moving images are blurred, said method comprising the steps of: detecting an array of motion vectors associated with corresponding portions of an array of image data, each of said motion vectors representing image motion of a corresponding one of said portions between temporally spaced arrays of image data; and deconvolving a blur function from each of said portions in dependence upon said capture time and a corresponding detected motion vector for that portion to yield an array of reduced blur image data.
11. Apparatus for processing arrays of image data substantially as hereinbefore described with reference to the accompanying drawings.
12. A method of processing arrays of image data substantially as hereinbefore described with reference to the accompanying drawings.