WO2007129257A1

WO2007129257A1 - Controlled frame rate conversion

Info

Publication number: WO2007129257A1
Application number: PCT/IB2007/051628
Authority: WO
Inventors: Marco K. Bosma; Fedde S. Bouwman; Lambertus A. Van Eggelen
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2006-05-04
Filing date: 2007-05-02
Publication date: 2007-11-15

Abstract

The image processing apparatus (100) comprises: an image rate conversion unit (102) for conversion of the sequence of input images into a sequence of intermediate images, by means of temporal interpolation; a renderer (106) for providing the sequence of output images, on basis of consecutive intermediate images and optionally a number of input images; and a conversion rate control unit (104) which is arranged to control the image rate conversion factor value C(t) of the image rate conversion unit (102) on basis of timing information from the renderer (106).

Description

Controlled frame rate conversion

The invention relates to an image processing apparatus for providing a sequence of output images to be displayed on a display device.

The invention further relates to a method of providing a sequence of output images to be displayed on a display device. The invention further relates to a computer program product to be loaded by a computer arrangement, comprising instructions to provide a sequence of output images to be displayed on a display device, the computer arrangement comprising processing means and a memory.

There is a trend for consumer electronics (CE) and personal computer (PC) platforms to grow towards each other. Media Center PC platforms are expected to deliver a high quality of video playback. Because of the numerous video source and display formats available in PC's that have to work together, frame/image rate conversion is used. That means that by means of temporal interpolation additional images are computed to convert the sequence of input images having an input frequency into a sequence of output images with a display frequency which is different from the input frequency.

Usually, the image rate conversion factor C is not exactly set to the actual output-input ratio R , i.e. the ratio between the actual display frequency and the actual input frequency. This results in synchronization problems between the rendering of the audio and video data, e.g. lip synchronization problems. To solve that problem, images are dropped or repeatedly provided to the display device by the video renderer. However dropping and/or repeating of images results in shocks in the motion portrayal, especially in video content with a high degree of (consistent) motion. To further elaborate on this issue an example is provided.

Suppose that that the assumed display frequency is 60 Hz. The value 60 Hz may be available as a configuration parameter of the display device or graphics card. Suppose that the assumed input frequency is 24 Hz. On basis of these two values (60 Hz and 24 Hz) a predetermined fixed image rate conversion factor C of 2.5 may be derived and used by an image rate converter. That means that the image rate converter converts the sequence of input images from a video source to the display frequency of 60 Hz by creating 2.5 images for each input image. However typically, the display frequency is not exactly 60 Hz. If the display frequency is higher than 60 Hz (e.g. 60.5 Hz), the buffering means for temporarily storage of interpolated images, will be empty after a while. This will result in repeatedly providing some of the images to the display device. If the display frequency is lower than 60 Hz, the buffering means will eventually get full. To prevent that from happening, images are dropped. Typically, the dropping of images is carried out by the renderer although in some cases, the images may be dropped before they are sent to the renderer. The problem is that due to omitting or doubling images in the video output, the motion is not fluent anymore. In some configurations, the whole buffering means is flushed in case of a full buffer thus potentially causing a big shock in the video. This is especially visible if a lot of motion is present in the video.

In the example above the display frequency was not exactly 60 Hz. Another example of the problem situation occurs if the input frequency is not exactly 24 Hz, e.g.

23.97 Hz. It will be clear that problem may be even more severe if the actual input frequency and the actual display frequency both deviate from the assumed input frequency and the assumed display frequency, respectively. Further problems might occur if the input frequency is not constant but changes as function of time, e.g. because of missing images or interrupted transfer.

It is an object of the invention to provide an image processing apparatus for providing a sequence of output images to be displayed on a display device, resulting in a more fluent motion portrayal.

This object of the invention is achieved in that the image processing apparatus comprising: conversion means for conversion of a sequence of input images having an input frequency into a sequence of intermediate images, by means of temporal interpolation, the intermediate images having respective required display times; buffering means for temporally buffering the intermediate images; rendering means for providing the sequence of output images to the display device with a display frequency which is different from the input frequency, on basis of consecutive intermediate images from the buffering means; and a time difference computing means for computing a first display time difference based on a first display frequency based actual display time and a first required display time of a first one of the intermediate images in the buffering means, to control the conversion means to create a second one of the intermediate images wherein the second required display time of the second one of the intermediate images is based on the first display time difference.

The conversion means is controlled on basis of actual time information. That means that the conversion means is not set to generate a sequence of intermediate images on basis of a predetermined fixed image rate conversion factor value C , whereby the predetermined fixed image rate conversion factor value C is for example based on the ratio between the assumed display frequency and the assumed input frequency. In the image processing apparatus according to the invention, the required display times of the subsequent intermediate images are computed on basis of actual display time differences.

If the first display time difference is relatively high then the first image rate conversion factor value C(Y₁) , which was used to determine the required display time of the first intermediate image deviates relatively much from the output-input ratio. That means that the second image rate conversion factor value C(Y₂) which is needed to compute the second required display time or at least a further image rate conversion factor value C{t_n ) which is needed to compute at further required display time should be different from the first image rate conversion factor value C(Y₁ ) . In other words, the image rate conversion factor values C(Y) are dynamically adapted in order to approach the output-input ratio R .

average{C(t)\t > O} = R (1)

On basis of actual display time differences, the actual image rate conversion factor values C(Y) are computed. Typically the actual image rate conversion factor value C(Y) is not constant but fluctuates as function of time t between a maximum image rate conversion factor value C and a minimum image rate conversion factor value C Preferably, the current image rate conversion factor value C(Y) is computed recursively, i.e. on basis of a previous image rate conversion factor value C(Y -α) .

C(t) = f(C(t -a)) (2) It should be noted that the conversion means, requires values of the subsequent required display times. The availability of image rate conversion factor values is not a strict necessity in order to compute the required display times. In principle the image rate conversion factor is only a mathematical quantity, i.e. a quantity which represents a relation between the input frequency and display frequency. Computations of subsequent required display times may be based on image rate conversion factor values. Alternatively, the subsequent required display times are computed on basis of a different quantity, e.g. the estimated phase shift S_e(t) which is the inverse/reciprocal of the image rate conversion factor value.

S₆(O = 1/ C(O (3)

Typically, the sequence of output images comprises intermediate images and also a number of the input images. In an embodiment of the image processing apparatus according to the invention, the rendering means is arranged to provide the sequence of output images, comprising a number of the input images.

Preferably the image processing apparatus according to the invention is arranged to create the sequence of output images by making a trade-off between minimum display time differences of the respective intermediate images in the buffering means and a relatively high share of the input images in the sequence of output images. A relatively high share of input images in the sequence of output images, i.e. using relatively many original input images instead of interpolation of intermediate images, means a relatively low processing load and potentially a higher quality of the sequence of output images. Interpolated images are typically less sharp than original images on which the interpolated images are based. This embodiment of the image processing apparatus is arranged to use a set of convenient image rate conversion factor values to minimize the amount of (interpolated) intermediate images. The average of the actually applied image rate conversion factor values corresponds to the ratio between the actual display frequency and the actual input frequency, i.e. the average of the actually applied image rate conversion factor values is substantially equal to the output-input ratio.

Preferably, the image processing apparatus is arranged to create the sequence of output images by selecting a particular one of the input images as one of the output images if the first display time difference is below a predetermined threshold and else to interpolate the second one of the intermediate images on basis of multiple input images. In other words, a choice is made between a minimal amount of interpolation and minimal judder, respectively. In an embodiment of the image processing apparatus according to the invention, the second image rate conversion factor value is based on the image content of a number of the input images. The computation of the image rate conversion factor value may be based on a number of conditions: the ratio between the actual display frequency and the actual input frequency, i.e. the output-input ratio ; maximizing the share of input images in the sequence of output images; and minimizing the amount of judder.

If the applied image rate conversion factor value C(t) differs relatively much from the output-input ratio R , that does not necessarily result in visible artifacts. For instance, if there is hardly any or even no motion, then repetition or dropping of images is not visible. In that case the applied image rate conversion factor value C(t) may differ relatively much from the output-input ratio R . Also, if a lot of motion is present in the video content, the chance is high that no static talking faces are in the video, so lip synchronization is less important. However, if the amount of motion is moderate it is better that the applied image rate conversion factor value C(t) and the output-input ratio R are substantially mutually equal. So preferably, the image rate conversion factor value C(t) is based on estimated motion between a number of the input images.

Alternative video content information may be, scene change notifications and the consistency of motion vector fields. It is a further object of the invention to provide a method of providing a sequence of output images to be displayed on a display device, resulting in a more fluent motion portrayal.

This object of the invention is achieved in that the method comprises: conversion of a sequence of input images having an input frequency into a sequence of intermediate images, by means of temporal interpolation, the intermediate images having respective required display times; temporally buffering the intermediate images in buffering means; providing the sequence of output images to the display device with a display frequency which is different from the input frequency, on basis of consecutive intermediate images from the buffering means; and computing a first display time difference based on a first display frequency based actual display time and a first required display time of a first one of the intermediate images in the buffering means, to control the creation of a second one of the intermediate images wherein the second required display time of the second one of the intermediate images is based on the first display time difference.

It is a further object of the invention to provide a computer program product for providing a sequence of output images to be displayed on a display device, resulting in a more fluent motion portrayal.

This object of the invention is achieved in that the computer program product, after being loaded, provides said processing means with the capability to carry out: conversion of a sequence of input images having an input frequency into a sequence of intermediate images, by means of temporal interpolation, the intermediate images having respective required display times; temporally buffering the intermediate images in buffering means; providing the sequence of output images to the display device with a display frequency which is different from the input frequency, on basis of consecutive intermediate images from the buffering means; and computing a first display time difference based on a first display frequency based actual display time and a first required display time of a first one of the intermediate images in the buffering means, to control the creation of a second one of the intermediate images wherein the second required display time of the second one of the intermediate images is based on the first display time difference.

Modifications of the image processing apparatus and variations thereof may correspond to modifications and variations thereof of the method and the computer program product, being described.

These and other aspects of the image processing apparatus, the method and the computer program product, according to the invention will become apparent from and will be elucidated with respect to the implementations and embodiments described hereinafter and with reference to the accompanying drawings, wherein: Fig. 1 schematically shows an embodiment of the image processing apparatus according to the invention;

Fig. 2 schematically shows a sequence of input images and a sequence of output images; and

Fig. 3 schematically shows an image processing configuration comprising an embodiment of the image processing apparatus according to the invention and optionally comprising a display device.

Same reference numerals are used to denote similar parts throughout the Figures.

Table 1 lists a number of typical input frequencies, i.e. input rates and Table 2 lists a number of typical display frequencies, i.e. display rates.

Table 1 Typical input frequencies

Table 3 list a number of output-input ratios R , i.e. ratios between input frequency and display frequency.

Table 3 output-input ratios

The most commonly "used" display frequencies are 60, 72, 75 and 85. However, due to inaccuracy in the clock of video cards, the actual display frequencies are slightly higher or lower (e.g. 59.7 Hz or 60.4 Hz) in many cases. The display rate 59.94 is also an official standard.

Table 4 list a number of actual phase shifts A , i.e. ratio between display frequency and input frequency. The concept "phase" is used to express the position of an output image relative to two subsequent input images. E.g. phase = 0.0 corresponds to the position of a particular input image, while phase equals 1.0 corresponds to the position of the next input image.

A = MR (4)

Table 4 Actual phase shifts

Fig. 1 schematically shows an embodiment of the image processing apparatus according to the invention. The image processing apparatus is arranged to provide a sequence of the output images at its output connector 122 on basis of a sequence of input images which is provided at its input connector 114. The sequence of input images has an input frequency which is different from the output frequency of the sequence of output images. The sequence of output images is provided to a display device which is arranged to display the sequence of output images with a display frequency which is equal to the output frequency.

The sequence of output images comprises copies of a number of the input images and a number of intermediate images, which are computed by means of temporal interpolation of input images.

The image processing apparatus 100 comprises: an image rate conversion unit 102 for conversion of the sequence of input images into a sequence of intermediate images, by means of temporal interpolation; - a renderer 106 for providing the sequence of output images, on basis of consecutive intermediate images and optionally a number of input images; and a conversion rate control unit 104 which is arranged to control the image rate conversion factor value C(Y) of the image rate conversion unit 102 on basis of timing information from the renderer 106. Preferably the timing information corresponds to time differences which are based on actual display times and corresponding required display times of respective intermediate images. Preferably, a number of time differences are filtered by means of a PID filter to derive a value to control the conversion rate control unit 104.

The image rate conversion unit 102, the renderer 106 and the conversion rate control unit 104 may be implemented using one processor or multiple processors, e.g. multiple-core. Normally, these functions are performed under control of a software program product. During execution, normally the software program product is loaded into a memory, like a RAM, and executed from there. The program may be loaded from a background memory, like a ROM, hard disk, or magnetical and/or optical storage, or may be loaded via a network like Internet. Optionally an application specific integrated circuit provides the disclosed functionality.

Preferably, the working of the image rate conversion unit 102 is a specified in the article "An evolutionary architecture for motion-compensated 100 Hz television", in IEEE Transactions on Circuits and Systems for Video Technology, 1995, by G. de Haan et al. Preferably, the image rate conversion unit 102 comprises a motion estimator unit as specified in the article "True-Motion Estimation with 3-D Recursive Search Block Matching" by G. de Haan et al. in IEEE Transactions on circuits and systems for video technology, vol.3, no.5, October 1993, pages 368-379.

Preferably, the renderer 106 comprises: a memory device 110 for temporally buffering the intermediate images which have been computed by the image rate conversion unit 102. Optionally, the memory device 110 is also arranged to temporally store a number of the input images. The number of input images may be directly put in the memory device 110 or they may be transferred by the image rate conversion unit 102. Alternatively, the memory device 110 is not part of the renderer 106 but is located externally. Typically the memory device 110 behaves like a fϊrst- in- first-out buffering means, i.e. the images are put into memory device 110 at the "entrance" and fetched from the "exit"; a memory access unit 112 for fetching images from the memory device 110 for providing these images as output images at the output connector 122. The fetching takes place with the display frequency. Each of the images is accessed and fetched at a respective actual display time; and a time difference computing unit 108 for computing display time differences based on actual display times and corresponding required display times of respective intermediate images in the memory device 110. The actual display times are based on the display frequency. The computed display time differences are provided to the conversion rate control unit 104. The renderer 106 comprises a time interface connector 120 from which the renderer 106 receives the actual time. Typically the actual time will be provided by a system timer. Each of the images in the buffer, i.e. the memory device 110 comprises a timestamp representing the required display time of the image. The required display time of a particular image corresponds to the display time at which the particular image should be displayed, i.e. provided at the output connector 122. Because the renderer 106 is responsible for providing the images, the renderer 106 exactly knows when a particular image is actually provided. On basis of that "provide" trigger and the received actual time which is provided at the time interface connector 120, the time difference computing unit 108 is arranged to determine the actual display time of the particular image. On basis of the required display time as stored in the corresponding timestamp and the actual display time, the corresponding display time difference is computed. Preferably, for each of the images which is put or fetched from the memory device 110 the corresponding display time difference is computed and provided to the conversion rate control unit 104.

The working of image processing apparatus will be explained next. It is assumed that the image processing apparatus is already running for a while, meaning that memory device 110 comprises a number of intermediate images. Input images are subsequently provided to the image rate conversion unit 102. On basis of a previously computed image rate conversion factor value C(Y) the so-called estimated estimated phase shift S_e(t)is computed. For example if the image rate conversion factor value C(t) is equal to 2.0 then the estimated phase shift S_e(t) equals 0.5 (in this example the estimated phase shift is normalized between 0.0 and 1.0). Suppose that the previous input image has been passed to the memory device 110. If the estimated phase shift S_e(t) equals 0.5 then the required display time of the next image to be computed is equal to the required display time of the previous input image plus the estimated phase shift which is equal to 0.5. Consequently, the image rate conversion unit 102 will interpolate or select the next image on basis of that required display time.

As long as, the conversion rate control unit 104 provides the image rate conversion unit 102 with the image rate conversion factor value C(t) which is equal to 2.0, the image rate conversion unit 102 will provide a sequence of intermediate images of which the difference between subsequent required display times is equal to 0.5, i.e. the estimated phase shift S_e(t) of 0.5. However, if the conversion rate control unit 104 provides another image rate conversion factor value C(t) , i.e. not equal to 2.0, the image rate conversion unit 102 will act accordingly, resulting in a sequence of intermediate images with a different difference between subsequent required display times.

The conversion rate control unit 104 actually will change the image rate conversion factor value C(Y) if a computed display time difference gives reason for that, i.e. if the display time difference of another, previously computed image is not equal to zero. The conversion rate control unit 104 comprises a content control interface connector 116 by which the conversion rate control unit 104 is provided with information about the content of the input images. The amount of motion in the video content and the appearance of scene changes are examples of content in this respect. On basis of the content the conversion rate control unit 104 adapts a so-called phase tolerance T(t) . The phase tolerance relates to a maximum deviation between an optimal required display time and an actually set/applied required display time. The following example is provided to explain what is meant to phase tolerance. Suppose that a particular image should be calculated/interpolated with a phase 0.9 in order to get an optimal motion portrayal. As explained before, the estimated phase shift S_e(t) and hence the applied phase shift S_a (t) is based on the image rate conversion factor value C(t) . However there is, per definition, an input image in the sequence of input images which has a phase which is equal to 1.0. The conversion rate control unit may decide to set the applied phase shift S_a (t) equal to 1.0. The result is that there is no need for interpolation because an input image can be directly used. In this case the deviation between the optimal required display time and actually set required display time is 0.9- 1.0 = -0.1. It will be clear that that are limitations to the amount of deviation. The maximum amount of deviation is called phase tolerance T(t) . See Equation 5.

The minimum phase shift is specified in Equation 6

S_ma (t) = S_β(t) - T(t) (6)

The maximum phase shift is specified in Equation 7

S_^ (t) = S_e(t) + T(t) (7)

The conversion rate control unit 104 comprises a load control interface connector 118 by which the conversion rate control unit 104 is provided with information about the load of computing resources of the image processing apparatus. Computing resources may include the image rate conversion unit 102, the conversion rate control unit 104 and the renderer 106 or components of these units. If the load of the computing resources is relatively high, meaning that there may become a shortage, the conversion rate control unit 104 may change the image rate conversion factor value C(t) resulting in temporally more copies of input images in the sequence of output images and less interpolated intermediate images. Typically this means that the image rate conversion factor value C(t) is set to an integer value. Table 5 lists a number of situations and the consequence for the phase tolerance Table 5 Examples of phase tolerances

Preferably, the maximum phase tolerance T(t) is related to the estimated phase shift S_e(t) . For example, with a high phase tolerance T(Y) the applied phase shift S_a (t) can maximally change with 20% of the estimated phase shift S_e(t) and with a low phase tolerance T(Y) the applied phase shift S_a(t) can change with only 5% of the estimated phase shift S_e(t) .

As explained above, in many cases the actual input frequency and/or display frequency deviate from the assumed input frequency and/or display frequency, respectively. The image processing apparatus according to the invention is arranged to convert the sequence of input images into a sequence of output images by taking care of the actual input frequency and output frequency, by measuring deviation's between required display times and actual display times. The results of measurements are fed back to control the image rate conversion unit 102. However, even if the actual input frequency and display frequency are exactly equal to the assumed input frequency and display frequency, respectively, it is preferred to apply a non constant image rate conversion factor value C(t) in order to get a maximum number of input images in the sequence of output images. Table 6 illustrates the number of input images before a next input image would be used for the sequence of output images. The character "x" stands for the really high number of skipped images.

Table 6 The number of input imaj *es which are skipped

Input frequency

Display frequency

23.97 24 25 30 50 60

60 X 2 5 1 5 1

72 X 1 X 5 X 5

75 X 8 1 2 2 4

85 X X 5 X 10 X The image processing apparatus according to the invention is arranged to dynamically adapt the image rate conversion factor value in order to maximize the number of input images in the sequence of output images while still achieving a relatively smooth motion portrayal. Next a number of examples are provided to further elaborate of on the working of the image processing apparatus according to the invention. In these examples one input period is defined as 1000 units, i.e. the phase difference between a particular input images and the next input image corresponds to 1000 units. In the examples above, the period was normalized to 1.0.

Fig. 2 schematically shows a sequence of input images 10, II, 12 and a sequence of output images O0-O5. Fig. 2 illustrates the perfect conversion of an input frequency of 24 Hz into an output frequency of 60 Hz. As can be seen, one input period is defined as 1000 units. The estimated phase shift S_e(t) equal 400 units. The phase tolerance

T(t) equal 80 units. The different phases of the output images are expressed by the function mod(lOOO) relative to the input images: - output image OO has phase 0 relative to input image 10; output image Ol has phase 400 relative to input image 10; output image 02 has phase 800 relative to input image 10; output image 03 has phase 200 relative to input image II; output image 04 has phase 600 relative to input image II; and - output image 05 has phase 1000 relative to input image Il is phase 0 relative to input image 12.

Multiple methods are possible for calculating optimal phase values. The example of the method described below is based on the condition that a maximum number of input images is taken. That means that a maximum number of output samples have phase 0 (or 1000) while still meeting condition that the phases are within the limits which are based on the phase tolerance.

The method has the following steps: compute the reference phase; compute the estimated phase shift S_e(t) ; - compute the phase tolerance T(t) on basis of image content;

Check if the reference phase is close to 0 or 1000. If the difference in phase compared to 0 (or 1000) is smaller than the phase tolerance, it is possible to set the phase to zero. Stop the method. If the output phase cannot be made zero directly, plan the output phases in a way that an output phase of zero is reached in as few input periods as possible;

Calculate the minimum phase shift (Equation 6) and maximum phase shift (Equation 7);

Initialize two counters with the value of the previous output phase. Add in multiple steps the minimum phase shift to the first counter and the maximum phase delta to the second counter (accumulate). If the first counter is under a multiple of 1000 and the second counter above the same multiple of 1000, an input phase boundary has been crossed (e.g. at phase 1000, 2000 or 3000). This means that it is possible to have a zero phase at this position within the phase tolerance. Stop the loop in this case and remember the amount of steps. Then calculate the applied phase shift S_a (t) which is necessary to have a phase of zero at this number of steps. Calculate the output phases on basis of this applied phase shift S_a (t)

Phase calculation example 1 : 24 fps to 75 fps conversion (fps= frames per second) Table 7

Table 8

If the image processing apparatus would not optimize the number of input images in the sequence of output images the phases would be {140,460,780,1100,1420,1740,2060,...}. For a very long time, no input images would be shown.

However, by inspecting Table 8, in particular counter 1 (see the third row: previous phase + step* S_1111n (t) ) and counter 2 (see the fourth row: previous phase + step* 5_^ (t) ) it can be observed that at step 6 an input image is "crossed" (at phase 2000). Thus with this phase tolerance, in two input periods of time an input image can be provided as output image. The applied phase shift S_a (t) is then:

2000 - previousphase _ 2000 - 140 _ o _a (t ) — — — 310 (8)

# steps 6

In Table 10 one can see that with this applied phase shift S_a (t) , an input image can be shown at step 6, while all phases remain within the phase tolerance.

Table 9

Table 10

Phase calculation example 2: 24 fps to 75 fps conversion

If the phase tolerance would be 40 the following situation would occur.

Table 11

Table 12

Table 13

Table 14

In this example, an original input image can be "reached" within one input period.

The output phases chosen in the previous examples 1 and 2 are chosen with a constant applied phase shift S_a(t) . In most situations, the conversion rate control unit 104 has more freedom to plan the phases. E.g. the conversion rate component 104 can do curve- fitting to get smoother image rate transitions.

Phase calculation example 3: 25 fps to 50.1 fps conversion Table 15

Table 16

The image processing apparatus will make the phase zero at the second step. Therefore it will choose an applied phase shift S_a(t) of 500. (Hence the image rate conversion factor value C(t) is equal 2.0). It will continue to choose an applied phase shift S_a(t) for a period of time. This means that too few frames are generated (compared to a phase delta of 499). The buffers of the renderer will slowly become empty, thus the display time differences will decrease. This results in a smaller estimated phase shift S_e(t) . There will be a moment that the estimated phase shift S_e(t)is less than 460. This implies that it is not possible to plan an original frame at step 2 anymore.

Table 17

Table 18

The image processing apparatus will now plan an original input image in step 7. The applied phase shift S_a(t) will be 428.6. This corresponds with an image rate conversion factor value C(t) of 2.33. Because of the increase of the image rate conversion factor value C(t) (compared to rate 2), the display time differences will increase. Finally, the reference phase delta will increase again. Now, the original input image can be planned at step 2 again, resulting in applied phase shift S_a(t) of 500. It can be computed that in the situation of 25 Hz input and 50.1 Hz output the image processing apparatus will alternate between a an image rate conversion factor value C(t) of 2.0 and an image rate conversion factor value C(O of 2.33.

It should be stressed that the different frequencies, e.g. input frequency, may vary as function of time. In other words, the image processing apparatus 100 according to the invention is arranged to deal with a not constant input frequency. The conversion factor value is adapted accordingly.

Fig. 3 schematically shows an image processing configuration 200 comprising: receiving means 202 for receiving a signal representing input images; an embodiment of the image processing apparatus 100 according to the invention as described in connection with Fig. 1 and Fig. 2; and a display device 204 for displaying the output images of the image processing apparatus 200.

The signal may be a broadcast signal received via an antenna or cable but may also be a signal from a storage device like a VCR (Video Cassette Recorder) or Digital Versatile Disk (DVD). The signal is provided at the input connector 206. The image display apparatus 200 might e.g. be a PC with capabilities of viewing video. Alternatively the image processing configuration 200 does not comprise the optional display device but provides the output images to an apparatus that does comprise a display device 406. Then the image processing configuration 200 might be e.g. a set top box, a satellite-tuner, a VCR player, a DVD player or recorder. Optionally the image processing configuration 200 comprises storage means, like a hard-disk or means for storage on removable media, e.g. optical disks.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be constructed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements and by means of a suitable programmed computer. In the unit claims enumerating several means, several of these means can be embodied by one and the same item of hardware or software. The usage of the words first, second and third, etcetera do not indicate any ordering. These words are to be interpreted as names. No specific sequence of acts is intended to be required unless specifically indicated.

Claims

CLAIMS:

1. An image processing apparatus for providing a sequence of output images to be displayed on a display device, the image processing apparatus comprising: conversion means for conversion of a sequence of input images having an input frequency into a sequence of intermediate images, by means of temporal interpolation, the intermediate images having respective required display times; buffering means for temporally buffering the intermediate images; rendering means for providing the sequence of output images to the display device with a display frequency which is different from the input frequency, on basis of consecutive intermediate images from the buffering means; and - a time difference computing means for computing a first display time difference based on a first display frequency based actual display time and a first required display time of a first one of the intermediate images in the buffering means, to control the conversion means to create a second one of the intermediate images wherein the second required display time of the second one of the intermediate images is based on the first display time difference.

2. An image processing apparatus as claimed in claim 1, wherein the time difference computing means is configured to determine a second image rate conversion factor value for achieving a conversion between the input frequency and the display frequency, on basis of the first display time difference, to control the conversion means to create the second one of the intermediate images.

3. An image processing apparatus as claimed in claim 2, wherein the time difference computing means is configured to determine the second image rate conversion factor value on basis of a first image rate conversion factor value which has been used to create the first one of the intermediate images.

4. An image processing apparatus as claimed in claim 3, wherein the first image rate conversion factor value and the second image rate conversion factor value are values of a set of image rate conversion factor values which are used to compute subsequent required display times of the intermediate images, the values of the set being mutually different.

5. An image processing apparatus as claimed in any of the claims 1 to 4, wherein the rendering means is arranged to provide the sequence of output images, comprising a number of the input images.

6. An image processing apparatus as claimed in claim 5, which is arranged to create the sequence of output images by making a trade-off between minimum display time differences of the respective intermediate images in the buffering means and a relatively high share of the input images in the sequence of output images.

7. An image processing apparatus as claimed in claim 6, which is arranged to create the sequence of output images by selecting a particular one of the input images as one of the output images if the first display time difference is below a predetermined threshold and else to interpolate the second one of the intermediate images on basis of multiple input images.

8. An image processing apparatus as claimed in any of the claims 2 to 7, wherein the second image rate conversion factor value is based on the image content of a number of the input images.

9. An image processing apparatus as claimed in any of the claims 2 to 7, wherein the second image rate conversion factor value is based on estimated motion between a number of the input images.

10. A method of providing a sequence of output images to be displayed on a display device, the method comprising: conversion of a sequence of input images having an input frequency into a sequence of intermediate images, by means of temporal interpolation, the intermediate images having respective required display times; temporally buffering the intermediate images in buffering means; providing the sequence of output images to the display device with a display frequency which is different from the input frequency, on basis of consecutive intermediate images from the buffering means; and computing a first display time difference based on a first display frequency based actual display time and a first required display time of a first one of the intermediate images in the buffering means, to control the creation of a second one of the intermediate images wherein the second required display time of the second one of the intermediate images is based on the first display time difference.

11. A computer program product to be loaded by a computer arrangement, comprising instructions to provide a sequence of output images to be displayed on a display device, the computer arrangement comprising processing means and a memory, the computer program product, after being loaded, providing said processing means with the capability to carry out: conversion of a sequence of input images having an input frequency into a sequence of intermediate images, by means of temporal interpolation, the intermediate images having respective required display times; temporally buffering the intermediate images in buffering means; providing the sequence of output images to the display device with a display frequency which is different from the input frequency, on basis of consecutive intermediate images from the buffering means; and - computing a first display time difference based on a first display frequency based actual display time and a first required display time of a first one of the intermediate images in the buffering means, to control the creation of a second one of the intermediate images wherein the second required display time of the second one of the intermediate images is based on the first display time difference.