GB2449929A

GB2449929A - Hierarchical spatial resolution building processes to fill holes in an interpolated image

Info

Publication number: GB2449929A
Application number: GB0711081A
Authority: GB
Inventors: Michael James Knee
Original assignee: Snell and Wilcox Ltd
Current assignee: Snell Advanced Media Ltd
Priority date: 2007-06-08
Filing date: 2007-06-08
Publication date: 2008-12-10
Also published as: GB0711081D0; WO2008149162A3; WO2008149162A2

Abstract

The invention finds application in image interpolation processes to generate new images within an existing image sequence. Motion vectors from the existing images are used to generate an interpolated new image. However, in the case of a diverging motion vector field for example, there may be 'holes' in the new image. As an improvement on prior art techniques of filling these holes the invention performs picture building processes at a plurality of spatial resolutions. The output of the lowest spatial resolution process may be up-sampled and is input to the next highest resolution picture building process thereby combining the two picture building processes. This is repeated until the highest resolution picture building process generates the final pixel output value. The lowest spatial resolution process will fill the holes but the lower resolution in the holes would be visible as an artefact in an otherwise higher resolution image. The hierarchical filling of the holes in this manner of the invention reduces the visibility of the lower resolution in the holes.

Description

IMPROVEMENTS RELATING TO PICTURE BUILDING

FIELD OF INVENTION

This Invention concerns the interpolation of new images within a sequence of related images. Examples of sequences of images include views taken by an imaging device at different points in time; and, images of a scene taken from different viewpoints.

BACKGROUND OF THE INVENTION

There are many applications requiring new images to be interpolated within a sequence of existing images. Examples include television standards conversion, display up-conversion and reconstruction of compressed image sequences. In these applications the interpolation is temporal and the processing often makes use of motion vectors which define the change in position of portrayed objects between images. Analogous vectors can be used to assist the interpolation of a sequence of view points ("view interpolation") or other, non-temporal sequence parameters. In some applications spatial and temporal interpolation is combined; for example the BuIletTimeTM special effect used in films and computer games.

The invention to be described is equally applicable to these non-temporal and combined spatial-temporal interpolations.

The process of creating the pixel values of an interpolated intermediate image is known as "picture building". The process makes use of: pixel values in existing images adjacent to the position of the new image in the sequence; and, motion vectors derived from motion measurement between existing images of the sequence. There are two approaches to picture building.

In read-side" picture building the process is based on output pixels of the newly created image; an example is the method of UK patent application 0618323.0.

Read-side picture building is difficult because the motion vectors have been derived from the input images, and are therefore not directly related to the pixels of the new image.

In write-side" picture building the process is based on pixels of the existing images of the sequence; these pixels are projected (written) to locations in the new, interpolated images and the appropriate location is determined from the relevant motion vector for the input pixel. An example is given in EP 0 648 398.

The present invention relates to a write-side picture builder. Write-side picture building is convenient; motion vectors are normally associated with pixels in the input picture because they result from motion measurement between input pictures. Each motion vector will typically have been measured as a displacement between two adjacent input pictures and will be scaled prior to use for picture building so as to describe the expected displacement between the input picture being projected and the output picture.

In general, an arbitrary, scaled motion vector can have non-integer vertical and horizontal components and so the corresponding input pixel will be projected to an intermediate point on the output pixel grid. It may also be the case (especially in standards conversion) that the input pixels are not co-sited with output pixels.

Figure 1 shows an example, where a pixel P is projected, according to the vector (1) to a point with coordinates (x,y) which lies between four output pixel positions.

P is a pixel from an existing image and its position corresponds to its position in that image. The vector (1) defines the expected positional change between the time of the original image and the time of the new image.

If, for convenience, we place the origin of the co-ordinate system at the position of the upper left output pixel (in television the vertical co-ordinate is usually a line number so the direction of the vertical axis is downwards), then the output pixels have co-ordinates (0,0) (1,0) (0,1) (1,1); and, the projected pixel co-ordinate values x y lie in the range zero to unity.

There are various known ways of calculating the contribution of P to each of the four output pixels. One method is to weight the four contributions in proportion to the distance between (x,y) and the respective output sample opposite the sample receiving the contribution in each dimension.

The resulting contributions for the example shown in Figure 1 are given by the

following table:

Output Pixel Positi Contribution from Input Pixel P1 (co-ordinate) (0,0) Px(l-x).(1_y) (0,1) Px(1-x).y (1,0) Pxx.(l-y) (1,1) Pxx.y Iii Contributions from other input samples are accumulated at the output pixel locations to give a total value for each output pixel.

Another known way of dealing with sub-pixel vectors is to project the nearest output pixel position back to the input picture, calculating the contribution to that output pixel by interpolation of the four input pixels surrounding the back-projected point. The interpolation may be bilinear or it may use a filter of larger aperture.

One of the problems with the prior art is that, after projecting all the input pixels by any of the methods described above (including simple integer-vector projection), the output pixels may not have received equally-weighted contributions. Ideally, the weights of the contributions to each output pixel will sum to unity. However, they may sum to a value greater or less than unity, depending on the convergence or divergence of the motion-vector field. Typically this problem is solved by normalizing the contributions; i.e. dividing the accumulated weighted sum of pixel values by the sum of the weights of the projections. Normalizing contributions will not work, however, where an output pixel has received no contributions at all, for example in an area where motion vectors diverge. Such pixels are known as "holes".

Holes have to be filled with something, because every output pixel requires a value. There are known ways of filling holes. Where a hole is adjacent or very near to one or more output pixels that are not holes, it can be filled by copying or by linear or non-linear interpolation or extrapolation of the nearby pixel values.

However, those methods do not produce acceptable results where holes occupy large areas of adjacent output pixels.

Another method of filling holes is to make use of an input picture that is temporally on the other side of the output picture (i.e. a picture adjacent to the input picture in the sequence of input pictures.) If the first input picture has divergent vectors (pointing forward to the new image), leading to holes, the second one is likely to have convergent vectors (pointing backward to the new image), which means that the holes will be filled from the other side. However, this does not eliminate the problem, particularly in areas of complex motion or at scene changes.

A third method of filling holes, often used as a default when other methods are not available, is to use co-sited pixels from the input picture (i.e. without motion compensation).

These prior-art methods are not wholly satisfactory. Often a line of holes (a "crack') occurs at the edge of a moving object and the different treatment of these pixels makes them conspicuous as an image artefact. The present invention seeks to provide an improved method of picture building in which artefacts are less apparent.

SUMMARY OF THE INVENTION

The invention consists in one aspect in a method and apparatus for determining values for pixels of a new image within an existing sequence of images, the method including a plurality of picture building processes that create new pixel values from existing pixel values by making use of motion vectors derived from comparison of existing images in the said sequence, wherein the said picture building processes operate on respective sets of pixel values at a plurality of respective spatial resolutions and one or more of the said values for pixels of a new image is obtained by combining the outputs from two or more of the said picture building processes.

In one embodiment, the inputs to the said picture building processes represent a sub-band decomposition of each input image so that the sum of the said inputs is substantially equal to the respective input image.

Suitably, the said values for pixels of a new image are obtained by summing the outputs of two or more of the said picture building processes.

Advantageously, at least one of the said picture building processes operates on pixel value differences between image representations at two of the said spatial resolutions.

In a further embodiment the said values for pixels of a new image are obtained by switching between the outputs of two or more of the said picture building processes.

Advantageously, the said values for pixels of a new image are obtained by cross-fading between the outputs of two or more of the said picture building processes.

The said motion vectors derived from comparison of existing images may be derived from a hierarchical motion measurement process that simultaneously outputs motion vectors at more than one level of spatial resolution.

Advantageously, the said motion vectors are derived from a phase correlation process.

In another aspect, the present invention consists in a method of filling a hole in a new image generated within an existing sequence of images by a picture building process that creates new pixel values from existing pixel values in at least one existing image by making use of motion vectors associated with that existing image, the method comprising up-sampling the output of a lower spatial resolution picture building process conducted on the existing images, to generate a pixel value to fill the hole.

Advantageously, the method comprises up-sampling the output of at least one of a hierarchy of a successively lower spatial resolution picture building processes conducted on the existing images, to generate a pixel value to fill the hole.

Preferably, the hole is filled with a pixel value generated from the highest spatial resolution picture building process that has no hole at the relevant location.

In another aspect, the present invention consists in apparatus for filling a hole in a new image generated within an existing sequence of images by a picture building process that creates new pixel values from existing pixel values in at least one existing image by making use of motion vectors associated with that existing image, the apparatus being configured to up-sample the output of a lower spatial resolution picture building process conducted on the existing images, to generate a pixel value to fill the hole.

BRIEF DESCRIPTION OF THE DRAWINGS

An example of the invention will now be described with reference to the drawings in which: Figure 1 shows a motion compensated input pixel and its spatial relationship to four output pixels.

Figure 2 shows a block diagram of a picture building system in accordance with an embodiment of the invention.

Figure 3 shows a block diagram of a picture building system in accordance with an alternative embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

Figure 2 shows a picture building system in which pixels (2402) of an output image are derived from pixels (2401) of an input image by making use of motion-vectors (2204) associated with the image portrayed by the input pixels (2401).

The system comprises a hierarchical set of picture building processes (2000) (2100) (2200) (2300) (2400) operating at differing levels of spatial resolution in which holes are filled at each level from the built output from the immediately lower level.

In the illustrated system the input motion vectors (2204) are assumed to be at a lower level of spatial resolution than the input pixels (2401); however, they can be at any convenient resolution, typically the resolution of the motion estimator which creates them.

The top level process (2400) operates at the resolution of the input pixels (2401).

These pixels are input to a picture builder (2403) which creates output pixels by taking appropriate contributions from the input pixels (2401) according to motion vectors (2404) associated with the input pixels (2401). The motion vectors (2404) are derived from the input motion vectors (2204) by cascaded spatial upsampling processes (2205) and (2305). These upsampling processes can operate by treating the two (horizontal and vertical) components of the set of vectors for an image as two separate images. These "vector component images" are interpolated respectively to provide the required vector components at the required sample positions. This process can use any of the known two-dimensional image interpolation techniques and inherently re-scales the vectors so that their components are expressed in units of the relevant pixel pitch.

The picture builder (2403) operates in any convenient known manner but does not fill holes. However it includes a method of detecting holes; for example a system which identifies those output pixels which have received no contributions from input pixels. The output from the picture builder (2403) is passed to a switch (2406) which is controlled by a hole flag signal (2407). When a pixel corresponding to a hole is output from the picture builder (2403), the hole flag signal (2407) causes the switch (2406) to route the output (2302) of the next lower picture building system (2300) to the output (2402). As the process (2300) operates at lower spatial resolution than the process (2400), the output (2302) is up-sampled in a spatial up-sampler (2308) to the same resolution as the output from the picture builder (2403).

The input pixels (2401) are also input to the picture building process (2300) which includes a picture builder (2303) which operates in the same way as the picture builder (2403). Because this process operates at a lower spatial resolution, the input is spatially down-sampled by the down-sampler (2309). The down-sampled input is also passed to the next lower process where it is further down-sampled to the next lower spatial resolution and so on.

The picture building processes (2200) and (2100) operate in the same way as the process (2300).

Each picture builder, apart from the top level picture builder (2400), provides an output which is used to fill holes in the output of the next higher picture builder.

The lowest level process (2000) includes a hole filler (2020) which operates according to any convenient known method making use of its (down-sampled) input pixels. For example, non-motion-compensated input pixels could be used.

The hole filler (2020) fills any holes in the output from the picture builder (2003) and the result is up-converted by the spatial up-sampler (2008) to form the hole filling input to the switch (2106).

This embodiment of the invention exploits the fact that spatially down-sampled and scaled motion vectors are less likely to produce holes because the vectors are smaller, and their divergence (expressed in units of pixel-pitch) is less. This results in fewer, smaller holes. The up-sampled built pictures are therefore suitable for filling holes from higher resolution picture building operations. By arranging the processes in a hierarchy the reduced resolution in the holes is made much less apparent.

As explained above the motion vectors were assumed to be at the resolution appropriate to the process (2200), so they are down-converted (and down-scaled) for lower levels of the hierarchy and up-converted (and up-scaled) for higher levels. These conversions could be avoided if a hierarchical motion estimator were used which generates vectors at all the required resolutions as part of the motion estimation process.

In Figure 2, the down-conversion and upconversion operations use a factor of 2, though other factors may be used. The filters used for up and down sampling the pixels may be of any design taken from the known field of wavelet or sub-band image processing. Normally, the same filter designs and conversion factors would be used for horizontal and vertical conversion, though they could differ. In a typical embodiment, the down-conversion filter has coefficients of 1/4, ", in each dimension and the upconversion filter has coefficients of /2, V2 in each dim ension.

The motion vectors may also be up and down-sampled using linear filters.

However, more sophisticated, non-linear techniques may be used, for example median filtering or the method of UK Patent Application 0614567.6.

Figure 2 shows a possible hardware implementation. A software implementation may generate identical signals, but savings could be made in processing time by making use of the fact that the output of any particular level is only required at pixels where there is a hole to be filled in the level above.

The above-described embodiment provides a useful method of hole filling but it can suffer from the effects of "hard switching" between the outputs of the different levels. An alternative embodiment will now be described which overcomes that problem by generating a "soft" combination of the outputs from all the levels.

An example of the alternative embodiment is shown in Figure 3. Where elements of this system are analogous to equivalent elements of Figure 2 they have equivalent reference numerals with the initial 2 replaced by 3. The main difference between this embodiment and the system of Figure 2 is that all the signals except the bottom level are differences between appropriately sampled versions of adjacent levels of down-conversion. They thus represent successively tower bands of an additive sub-band or wavelet decomposition of the picture, each expressed on an appropriate sampling structure. There is no explicit hole filling process, except at the lowest level, and the output built picture is the sum of the outputs of all the processing levels.

For each image down-sampling stage (3309) (3209) ... (3009), there is an equivalent up-sampling stage (3310) (3210) ... (3010), which up-converts the down-sampled pixel values and feeds them to the inverting input of a subtractor (3411) (3311) ... (3111) so as to derive the input to the picture builder at the level of the respective down-sampler's input. This ensures that, if the picture builders (3403) (3303) ... (3003) were transparent, the summation of their up-converted outputs in the chain of adders (3412) (3312) ... (3112) would be identical to the input (3401). At holes the picture building process will not be transparent, but the summing together of processes at different levels of resolution reduces the resulting artefacts.

The processing at the bottom level (3000) can be identical to the process (2000) of Figure 2, apart from the need for the up-converter (3010).

In Figure 3 the input motion vectors have the top level resolution and are therefore directly input to the picture builder (3403), however other resolution or multiple inputs at different resolutions could be used, as for the system of Figure 2.

The system of Figure 3, can be implemented as hardware or software. All levels are involved in the calculation of each output pixel, and so all pixels must be processed at all levels. However, the lower levels involve fewer pixels and so this is not a serious disadvantage.

Two examples of the invention have been described and other implementations are possible. The number of resolution levels used can be greater or less than four. The conversions between the different levels of spatial resolution can use any methods from the known arts of sub-band coding or wavelet decomposition.

The reduction or increase in resolution may depend on direction and may not be a variables-separable process. The picture building process at each spatial resolution level can use any of the known methods of motion compensated interpolation. -10-

Any of the known methods of motion measurement including phase correlation and block matching can be used to derive the motion vectors; the motion measurement process may operate at any convenient spatial resolution and the vectors scaled and re-sampled where necessary to the required resolution levels.

S The switches in the system of Figure 2 may be replaced be cross-faders and the controlling hole flag signals may be made proportional to the probability that the relevant pixel describes a hole, for example by using a sum of the contribution weights for a pixel as a measure of how unlike a hole that pixel is. -11 -

Claims

1. A method of determining values for pixels of a new image within an existing sequence of images, the method including a plurality of picture building processes that create new pixel values from existing pixel values by making use of motion vectors derived from comparison of existing images in the said sequence, wherein the said picture building processes operate on respective sets of pixel values at a plurality of respective spatial resolutions and one or more of the said values for pixels of a new image is obtained by combining the outputs from two or more of the said picture building processes.

2. A method according to Claim 1 in which the inputs to the said picture building processes represent a sub-band decomposition of each input image so that the sum of the said inputs is substantially equal to the respective input image.

3. A method according to Claim 1 or Claim 2 wherein the said values for pixels of a new image are obtained by summing the outputs of two or more of the said picture building processes.

4. A method according to Claim 3 wherein at least one of the said picture building processes operates on pixel value differences between image representations at two of the said spatial resolutions.

5. A method according to Claim 1 or Claim 2 wherein the said values for pixels of a new image are obtained by switching between the outputs of two or more of the said picture building processes.

6. A method according to Claim 1 or Claim 2 wherein the said values for pixels of a new image are obtained by cross-fading between the outputs of two or more of the said picture building processes. -12-

7. A method according to any previous Claim wherein the said motion vectors derived from comparison of existing images are derived from a hierarchical motion measurement process that simultaneously outputs motion vectors at more than one level of spatial resolution.

8. A method according to any previous Claim wherein the said motion vectors are derived from a phase correlation process.

9. A program carrier carrying processor-readable instructions for carrying out the method as claimed in any preceding claim.

10. Apparatus for determining values for pixels of a new image within an existing sequence of images, the apparatus including a plurality of picture building processes that create new pixel values from existing pixel values by making use of motion vectors derived from comparison of existing images in the said sequence, wherein the said picture building processes operate on pixel values at a plurality of respective spatial resolutions and one or more of the said values for pixels of a new image is obtained by combining the outputs from two or more of the said picture building processes.

11. Apparatus according to Claim 10 in which the inputs to the said picture building processes represent a sub-band decomposition of each input image so that the sum of the said inputs is substantially equal to the respective input image.

12. Apparatus according to Claim 10 or Claim 11 wherein the said values for pixels of a new image are obtained by summing the outputs of two or more of the said picture building processes. -13-

13. Apparatus according to Claim 12 wherein at least one of the said picture building processes operates on pixel value differences between image representations at two of the said spatial resolutions.

14. Apparatus according to Claim 10 or Claim 11 wherein the said values for pixels of a new image are obtained by switching between the outputs of two or more of the said picture building processes.

15. Apparatus according to Claim 10 or Claim 11 wherein the said values for pixels of a new image are obtained by cross-fading between the outputs of two or more of the said picture building processes.

16. Apparatus according to any of Claims 10 to 15 wherein the said motion vectors derived from comparison of existing images are derived from a hierarchical motion measurement process that simultaneously outputs motion vectors at more than one level of spatial resolution.

17. Apparatus according to any of Claims 10 to 16 wherein the said motion vectors are derived from a phase correlation process.

18. A method of filling a hole in a new image generated within an existing sequence of images by a picture building process that creates new pixel values from existing pixel values in at least one existing image by making use of motion vectors associated with that existing image, the method comprising up-sampling the output of a lower spatial resolution picture building process conducted on the existing images, to generate a pixel value to fill the hole.

-14 -

19. A method according to Claim 18, comprising up-sampling the output of at least one of a hierarchy of a successively lower spatial resolution picture building processes conducted on the existing images, to generate a pixel value to fill the hole.

20. A method according to Claim 19, wherein the hole is filled with a pixel value generated from the highest spatial resolution picture building process that has no hole at the relevant location.

21. Apparatus for filling a hole in a new image generated within an existing sequence of images by a picture building process that creates new pixel values from existing pixel values in at least one existing image by making use of motion vectors associated with that existing image, the apparatus being configured to up-sample the output of a lower spatial resolution picture building process conducted on the existing images, to generate a pixel value to fill the hole.

22. A program carrier carrying processor-readable instructions for carrying out the method as claimed in Claim 18 or Claim 19.