WO2010103106A1

WO2010103106A1 - Device and method for intermediate image interpolation

Info

Publication number: WO2010103106A1
Application number: PCT/EP2010/053196
Authority: WO
Inventors: Peter Rieder; Markus Schu; Kilian Jacob
Original assignee: Trident Microsystems, Inc.
Priority date: 2009-03-12
Filing date: 2010-03-12
Publication date: 2010-09-16
Also published as: DE102009001520B4; DE102009001520A1

Abstract

A method and a device for intermediate image interpolation are described.

Description

DEVICE AND METHOD FOR INTERMEDIATE INTERPOLATION

description

The present invention relates to an apparatus and a method for interframe interpolation.

Devices and methods for interframe interpolation are known in principle and serve, for example, to generate an interpolated image signal representing an image sequence having a second, higher image frequency from an image signal which represents an image sequence with a first image frequency. Such apparatus and methods are used, for example, in television technology to generate from an image signal representing a 50Hz (half) image sequence, an image signal representing a 100Hz image sequence, or from a motion picture film signal comprising a 24Hz image sequence represents interpolating a 100 Hz image signal.

In order to be able to interpolate the intermediate images in the correct direction, it is known to subdivide the images of the image sequence to be interpolated into image blocks and to determine motion information about the individual image blocks, for example in the form of motion vectors, by comparing the image contents of successive images. This motion information is then used for the interpolation of the contents of the image blocks of the intermediate images.

The motion estimation, ie the determination of motion information on image blocks of the intermediate images to be interpolated, can be very computationally intensive, in particular if the image points of the images of the image sequence to be interpolated are assigned image information values which can originate from a large range of values. This is the case, for example, if the image sequence to be interpolated is a sequence of images with a high dynamic range. Such Image sequence is also referred to below as HDR (High Dynamic Range) image sequence. The pixels of the images of such an HDR image sequence are associated, for example, with image information in the form of data words having a data word width of 32 bits.

The object of the present invention is to provide a method for inter-frame interpolation, which requires a reduced computational effort, and to provide a device for carrying out such a method.

This object is achieved by a device according to claim 1 and by a method according to claim 2.

The apparatus according to the invention comprises: an input for supplying an input image signal representing a sequence of images, each having a number of pixels, each associated with at least one image information value; a companding unit supplied with the input image signal and configured to generate a companded image signal from the input image signal by companding the image information values; a motion estimation unit supplied with the companded image signal and configured to determine motion information about the input image signal based on the composite image signal; an interpolation unit supplied with the input image signal and the motion information and configured to generate an interpolated image signal representing an image sequence having a higher image frequency and / or images having other motion phases as compared with the image sequence represented by the input image signal.

The inter-frame interpolation method according to the invention comprises: providing an input image signal representing a sequence of images each having a number of Having pixels, each associated with at least one image information value; Generating a companded image signal from the input image signal by companding the image information values of the input image signal; Determining a motion information to the input image signal based on the companded image signal; Generating an interpolated image signal representing an image sequence having a higher frame rate and / or images with other motion phases compared to the image sequence represented by the input image signal, from the input image signal using the motion information.

A companding of the image information values is to be understood in the context of the present description as a reduction of the value range of the image information values.

This reduction of the range of values can optionally be done using a linear or a non-linear compander characteristic. In one example, the image information values of the input image signal are each represented by data words having a data word width of 32 bits, and these image information values are mapped to composite image information values, each represented by a data word having a data word image width of 8 bits. In the simplest case, such a companding takes place, for example, by dividing the data word to be compiled into a number of partial data words which each have the same or different data word widths and in that the companded data word consists of the most significant bits (MSB) of the partial data words composed.

By applying the motion estimation in this device or in this method not to the input image signal but to the computed image signal reduced in terms of its data volume, the computational outlay in connection with the motion estimation can be considerably reduced. Embodiments will be explained in more detail with reference to figures. These figures serve to illustrate the basic principle, so that only the aspects necessary for illustrating this basic principle are shown. In the figures, unless otherwise indicated, like reference numerals designate like features having the same meaning.

FIG. 1 schematically illustrates an input image sequence represented by an input image signal and an interpolated image sequence resulting from this input image sequence.

FIG. 2 illustrates an example of an apparatus for generating the interpolated image sequence from the input image sequence, which comprises a compander unit, a motion estimation unit and an interpolation unit.

FIG. 3 illustrates the operation of an example of a compander unit.

FIG. 4 illustrates a block diagram of an example of the motion estimation unit.

FIG. 5 illustrates a block diagram of an example of an interpolation unit.

FIG. 6 illustrates by means of a block diagram a further example of a device for generating an interpolated image sequence from an input image sequence.

FIG. 1 schematically shows an image sequence which is also referred to below as an input image sequence and which has temporally successive images F (il), F (i). This picture that are shown in Figure 1 as temporally successive images, where t denotes time and k denotes a discrete time variable. The individual images F (il), F (i) each have a number of pixels (pixels), to each of which at least one image information value is assigned. For example, if these images are color images, each pixel is associated with three image information values, either a red value (R), a green value (G) and a blue value (B) in an RGB representation, or a luminance value (Y). Value) and two chrominance values (U value and V value) in a YUV representation. For various reasons, it may be desirable to interpolate intermediate images at temporal positions between two successive images F (il), F (i) of the input image sequence. "Moving correctly" in this context means that these intermediate images are interpolated such that moving objects located in an image F (i-1) of the input image sequence at a first position and in the second image F (i) at a second position are located in the at least one intermediate image at a spatial position between the first and second position. The relative spatial position of this object between the first and second position corresponds to the relative temporal position of the interpolated intermediate image between the temporal positions of the first and second image F (il), F (i). In FIG. 1, two such intermediate images F (i-) (cxi-1)) and F (i- (cx2-l)) to be interpolated are shown by way of example. These intermediate images differ from the input images F (il), F (i) in terms of their motion phase α, which may in principle be between 0 and 1. For example, in the illustrated example, the input images have the motion phase α = 0 or α = 1, while the intermediate images have motion phases

own, which lie between 0 and 1.

The input image sequence may, with regard to the image information values assigned to the individual pixels, in particular a high-resolution image sequence, such. B. an HDR image sequence be. These image information values can come from a high range of values compared to conventional television images. In the case of HDR image sequences, each pixel is assigned, for example, a digital data word with a word width of 32 bits, of which 24 bits determine the color distribution and 8 bits the brightness. A digital data word for a pixel of an HDR image thus comprises: 8 bits for a red component (R); 8 bits for a green component (G); 8 bits for a blue component (G); 8 bits for a brightness-determining exponent.

For purposes of interframe interpolation, it is generally known to perform a so-called motion estimation for individual image blocks of the intermediate image to be interpolated. In this case, the intermediate image is subdivided into individual blocks which are arranged in the form of a matrix and which, for example, comprise 8 × 8 or 16 × 16 pixels. For each of these image blocks, motion information indicating each image block to be interpolated is obtained by using the image contents of which image blocks in the first and second images F (i-1), F (i), the image content of the image to be interpolated

To interpolate image block is. This movement information is present, for example, in the form of a so-called motion vector. The determination of this movement information can be done, for example, by means of a recursive motion estimation method. Such a method is basically known, so that it is possible to dispense with further explanations.

The determination of a movement information for an image block to be interpolated requires depending on the used

Estimation method a plurality of comparisons between the image contents of image blocks in the first image F (il) on the one hand and the second image F (i) on the other. Such a "block comparison" is all the more computationally intensive, the larger the value ranges to which the individual pixel values originate. A device and a method for inter-frame interpolation with the device that can be carried out with this device This computational effort can be reduced, is illustrated below with reference to FIG 2.

FIG. 2 schematically shows a device for interframe interpolation. This device has an input to

Supplying an input image signal F representing an input image sequence, such as the image sequence with the images F (i-1), F (i) illustrated with reference to FIG. This input image signal F thus contains a sequence of the pixel values which are assigned to the pixels in the individual images. This input image signal F is supplied on the one hand to a compander unit 1 and on the other hand to an interpolation unit 4. Optionally, a delay element 2 is connected between the input and the interpolation unit 4, which serves to "compensate" for a signal delay which results from the processing of the input image signal F in the companion unit 1 to be explained below. The compander unit 1 is designed to generate a companded image signal Fl from the input image signal F by companding the pixel values contained in the input image signal F. The input image sequence F comprises, for example, a sequence of data words of a word width of n bits each, and the combined image signal F1 comprises, for example, a sequence of data words with a word width of m bits, where m is smaller than n.

The companded image signal Fl represents an image sequence which has the same image frequency as the image sequence represented by the input image sequence F and whose images have the same number of pixels as the images of the input image sequence. In other words, the image sequence represented by the companded image sequence F1 results from the input image sequence by reducing the representation accuracy of the pixel values and thus the amount of data required for this purpose. In one example, it is provided that the pixel values of the input image sequence are data words of width n = 32 bits, while the pixel values of the "companded image sequence" Data words of width m = 8 bits. In this case, there is a factor of 4 reduction in the amount of data. This reduction in the amount of data can be done in any manner using both a linear and a nonlinear compander characteristic. Compander to

Companding an HDR image, for example, to an 8-bit image are generally known, so that it is possible to dispense with further explanations.

For the sake of clarity only, Figure 3 illustrates the operation of an example of a compander that compandes an HDR image with 32 bits per pixel onto an image with 8 bits per pixel. D denotes a data word of a pixel of an HDR image in FIG. Of the 32 bits of this data word, 8 bits represent a red component R of the pixel, 8 bits a green component G, 8 bits a blue component B and 8 bits a brightness-determining exponent E. For example, in a composite data word D 'with 8 bits only 2 bits still a red component R, 2 bits a green component G, 2 bits a blue component B and 2 bits a brightness determining exponent E. The conversion of the color components representing partial data words of 8 bits in the original data word on the color components representing partial data words of each 2 bits in the companded data word can be made using any linear or non-linear compander characteristic. In one example, it is provided to delete from the 8-bit-wide partial data words in each case the 6 least significant bits (LSBs, Least Significant Bits) in order to obtain the 2-bit-wide partial data words. The exponent can be companded in a corresponding manner. In this connection, it should be noted that the color components and the exponent differ in different ways, i. using different compander characteristics, can be companded.

In another example (not shown), it is provided that the data word D is removed by deleting the exponent E. pandieren. The resulting data word in this case has a word length of 24 bits. In addition to deleting the exponent representing the brightness information, it is possible to compand the red, green and blue color information to further reduce the data word width. The companding factor is dependent on the word width of the data word D 'desired in the result. The word width of the partial data words representing the colors in the companded data word D 'is, for example, between 3 bits and 6 bits, whereby the individual colors can be companded differently both with regard to the companding factor and with respect to the compander characteristic.

In another example, it is provided to compander the data word D by deleting the color components. The compiled data word corresponds in this case to the exponent E. A motion estimation in this case is performed exclusively using the brightness information represented by the exponent.

The companded image signal Fl is supplied to a motion estimation unit 3, which is designed to determine a motion information to the input image signal F based on the composite image signal Fl. This motion estimation unit 3 can be a conventional motion estimation unit, for example a motion estimation unit which determines motion information for each image block of a desired intermediate image to be interpolated by comparing individual image blocks of successive images of the "companded image sequence" represented by the companded image signal Fl. Such a block comparison comprises a comparison of the individual pixels of these blocks. In this comparison, all image information can be compared, which are assigned by the companded data words individual pixels. However, only parts of the companded data words can be compared with each other, such as only one or more of the color information or just the health information.

The motion information is determined, for example, in the form of motion vectors, wherein, for example, each image block of an image to be interpolated is assigned such a motion vector. This motion information M is supplied to the interpolation unit 4, which is designed to generate, based on the input image signal F and this motion information M, an interpolated image signal F2 representing an image sequence with interpolated images. With reference to FIG. 1, this image sequence with interpolated images can be an image sequence which comprises the input images and the interpolated intermediate images, which therefore has a higher frame rate than the input image sequence. However, this image sequence with interpolated images can also be an image sequence that no longer contains the input images and that only includes images with different motion phases than the input images.

In the method according to the invention or the device according to the invention, the computation outlay for the motion estimation is reduced since the motion estimation is not applied to the input picture sequence F but to the companded picture sequence Fl comprising a reduced data set. The motion estimation unit 3 and the interpolation unit 4 may be conventional motion estimation units and interpolation units 4.

An example of a motion estimation unit 3 is shown schematically in FIG. This motion estimation unit 3 comprises two memories 31, 32, which are designed to each store a companded picture of the picture sequence represented by the companded picture signal F1. The companded image signal Fl is supplied to a first memory 31 directly and a second memory 32 via a delay element 33 with a time delay. A delay period of the delay In this case, the term "delay" 33 corresponds to the image duration of an image, so that two images are stored at one time in the two memories 31, 32, which temporally follow one another in the companded image sequence. The movement estimation unit 3 also comprises a motion estimator 34, which accesses the two memories 31, 32 in order to compare individual picture blocks of the stored pictures. The motion estimation unit 3 may also include a motion vector memory 35 in which motion vectors used by the motion estimator 34 for motion estimation are stored. The motion information M is available at the output of the motion estimator 34. This motion information M is present, for example, in the form of motion vectors, wherein the motion estimator 34 generates a motion vector for each image block of an intermediate image to be interpolated. In one example, it is provided that no motion estimation is carried out for motion phases α = 0 and α = 1, but that the images of the input image sequence are directly taken over into the interpolated image sequence for these motion phases.

An example of an interpolation unit 4 configured to generate the interpolated image signal F2 from the input image signal F using the motion information M is shown in FIG. This interpolation unit 4 comprises two memories 41, 42 which are each capable of storing an image of the input image sequence. In this case, the input image signal F is fed directly to one of the memories 41 and to a second memory 42 via a delay element 43, which has a delay time of 1 bit. The interpolation unit 4 also comprises an interpolator 44, which accesses the two memories 41, 42 and which is designed to interpolate each image block of an intermediate image to be interpolated by using the image vector block associated with each motion vector in the image first memory 41 stored image block and an image block stored in the second memory 42 are mixed.

FIG. 6 illustrates, by means of a block diagram, a further device for inter-frame interpolation and a method which can be carried out by this device. In this device, the input image signal F is supplied directly to the motion estimation unit 3, which in this case is adapted to perform a motion estimation on the uncompensated input signal F and to generate a motion information M to the input image signal F. This motion information M is used in the interpolation unit 4 to generate the interpolated image signal F2. This interpolated image signal F2 can represent, for example, an image sequence with input images and interpolated intermediate images, as shown in FIG. This interpolated image signal is fed to a companding unit 1, which is designed to generate a companded interpolated image signal F3 from this interpolated intermediate image signal F2. The companded image signal F3 has the same frame rate as the interpolated image signal F2 but has a reduced dynamic range with respect to the pixel values of the individual pixels. The scope of the companding is adapted, for example, to the display options of a display device on which the companded interpolated image signal F3 is to be displayed. This device makes it possible, for example, to interpolate an input image sequence which is an HDR image sequence and to display it on a conventional television set. Although the application of the motion estimation to the input image sequence is computation-intensive, it offers the advantage of a more accurate motion estimation.

Claims

claims

1. Apparatus for intermediate image information, comprising:

an input for supplying an input image signal (F) representing a sequence of images (F (i-1), F (i)) each having a number of pixels to which at least one image information value is respectively assigned;

a compander unit (1) supplied with the input image signal (F) and adapted to generate from the input image signal (F) a companded image signal (Fl) by companding the image information values;

a motion estimation unit (3) to which the companded image signal (Fl) is applied and which is adapted to determine a motion information (M) to the input image signal (F) based on the companded image signal (Fl);

an interpolation unit (4) to which the input image signal (F) and the motion information (M) are supplied and which is adapted to generate an interpolated image signal (F2) representing an image sequence which is compared with that obtained by the input image signal (F ) have a higher frame rate and / or images with other motion phases.

2. A method for interframe interpolation comprising:

Providing an input image signal (F) representing a sequence of images (F (i-1), F (i)) each having a number of pixels each associated with at least one image information value;

Generating a companded image signal (Fl) from the input image signal (F) by companding the image information values of the input image signal (F); Determining motion information (M) to the input image signal (F) based on the companded image signal (Fl);

Generating an interpolated image signal (F2) representing an image sequence having a higher frame rate and / or images with other motion phases compared to the image sequence represented by the input image signal (F), from the input image signal (F) using the motion information.