DE69524827T2 - Filter improvements - Google Patents

Description

This invention relates to displays and, more particularly, to image scaling filters for displays.

It has become very desirable to provide video displays in many formats. There are different modes for different picture formats, such as panoramic widescreen or film. There are also different video formats for computer video displays, depending on whether the video display is VGA (640 pixels on each line by 480 lines of pixels, where a pixel is a picture element) or Super VGA or XGA. There are also different television formats from standard NTSC up to 1920 pixels on each line by 1080 lines for the high definition television (HDTV) format. There are also flat panel displays and a digital mirror device (DMD), a new projection display that uses reflections from hundreds of thousands of micromirrors, each arranged over its own cell memory.

During operation of such a system it may be desirable to have a panoramic view or widescreen view or a movie screen view, which again requires some changes in the number of lines of the video signal from a source to the number of lines in the output signal.

This is achieved by a form of image scaling. To achieve these format changes, scaling filters are used. One such known filter is a bilinear interpolation filter, which suffers from the two problems of the stair-step effect (unwanted patterns) and the aperture effect or image softening (image blurring). These problems are shown in Fig. 1. Fig. 1a shows the bilinear filter kernel, while Figs. 1b and 1c show the frequency response.

The staircase problem can be understood by considering the frequency response of the bilinear filter for an example of three-to-four scaling in Fig. 1. Image scaling is basically an image filtering and re-sampling operation. If X(w) is the response representation of the original signal, the resample operation produces a scaled image whose response representation contains shifted copies of X(w) located at the new sampling frequency. If these copies are not completely separated, they can interact with each other, as shown in the shaded portion of Fig. 1b, resulting in the staircase effect.

The second problem is the aperture effect or image softening. This is caused by the fact that interpolation filters, as shown by the frequency response of bilinear interpolation in Fig. 1d, are basically low-pass filters with high frequency attenuation.

Both of these problems relate to the frequency content of the signals, and can be particularly undesirable for high-bandwidth sources.

In this context, reference is made to US-A-5 253 064 which discloses a method of controlling a video camera device having a zoom function and using a sensor having an imaging surface. To perform the interpolation in vertical and horizontal directions and also to magnify the luminance signal and the color signal data in horizontal directions, a plurality of sets of filter coefficients are generated, the sets of filter coefficients are grouped, each set corresponding to an output element of a video display, and each set of filter coefficients is scaled such that the sum of each set is equal to 1.

Furthermore, Carlbom, L: "Optimal Filter Design for Volume Reconstruction and Visualization", Visualization '93, Proceedings, IEEE Conference, October 25, 1993-October 29, 1993, pp. 54 to 61, describes a method for minimizing the maximum difference between the actual frequency response of a filter and the desired frequency response over a certain frequency band or over certain frequency bands. The Remez exchange algorithm is used to approximate the ideal frequency response with a minimum Chebyshev error. Starting from an ideal frequency response F(u) and a positive Weight W(u) is used to find a set of filter coefficients whose frequency response is H(u) such that the maximum absolute weighted error over a closed subset of 0 ≤ u ≤ 0.5 is minimized.

According to the present invention there is provided a method of forming an image scaling filter for converting one number of input lines into another number of output lines comprising the steps defined in claim 1.

By estimating a nearly ideal filter response without sharp cutoff frequencies, an improved image scaling filter is achieved. The estimated response, along with constraints such as the number of filter taps required, is then fed into a design algorithm that calculates a set of filter coefficients for scaling. The set of filter coefficients is scaled such that each set sums to 1. The correct set of input elements is multiplied by the filter coefficients and the set is summed to produce an output row.

- Fig. 1a shows the filter kernel of a bilinear interpolation filter;

- Fig. 1b and Fig. 1c show the frequency response of a bilinear interpolation filter with a 3-to-4 scaling; and

- Fig. 1d shows the aperture effect;

- Fig. 2 is an overall block diagram of a video display system;

- Fig. 3 is a flow chart of the method according to the present invention;

- Fig. 4a shows the ideal frequency response for a 3-to-4 scaling filter;

- Fig. 4b shows an optimal frequency response;

- Fig. 5 shows how the delays and coefficients are manipulated to obtain the desired line output signal in a vertical scaling function;

- Fig. 6 shows frequency domain responses of the state-of-the-art bilinear filter (Fig. 6a) and two rescaling filters with a first filter with 4 taps (Fig. 6f) and a second filter with 6 taps (Fig. 6c).

- Fig. 7a shows filter kernel coefficients for state-of-the-art bilinear filters;

- Fig. 7b shows the first filter coefficients;

- Fig. 7c shows the second filter coefficients;

- Fig. 8a and Fig. 8b show 3-to-4 scaling coefficients applied using the first filter; and

- Fig. 9 shows the scaling results of the bilinear filter (9a), the first filter (9b) and the second filter (9c) for a 3-to-4 scaling.

Referring to Figure 2, there is shown a video display system using the image scaling filters of the present invention. A video source 11, which may be from cable, VCR or other source, is applied through A/D (analog to digital) converter 13 and the digital representation of the video signal, which includes signal encoding in signal processor 14 for converting a decoded composite signal into luminance Y and chrominance C (I and Q) signals, interleaved or non-interleaved depending on the application. The output Y and C signals are applied through a scaling filter 15 which in turn is applied through a color space converter 16 and through a digital-to-analog converter (D/A converter) 16a to a CRT 17 or directly from the converter 16 to a display such as a digital mirror device (DtVID) display 18 described in a 1987 article by Larry J. Hornbeck, a scientist at Texas Instrument Inc., and described, for example, in the November issue of The Institute of Electrical and Electronic Engineers Spectrum, pages 27-31. In some cases, scaling may be done after color space conversion. In the case of a VGA input, no color space conversion is used. The video signals may be digital, in which case there is no analog-to-digital conversion. In addition, the digital input signals may be non-video digital image signals from a photographic or printing device, where the picture elements of these are to be enlarged (increased image size) or reduced (decreased image size).

Referring to Fig. 3, a flow chart is shown for the method of the present invention for designing optimal scaling filters to overcome the problems discussed in the Background of the Invention section. The first step 101 is to determine the ideal frequency response. Fig. 1b shows the ideal frequency response for 3-to-4 scaling with the thick line marked "ideal". This ideal frequency response eliminates the staircase effect while reducing the aperture effect. However, such an ideal frequency response with a sudden cutoff frequency cannot be realized in practice.

Another ideal frequency response for a 3-to-4 scaling filter is shown in Fig. 4a. The zero crossing of this signal changes for each scaling factor. However, the ideal frequency response has a response that extends to infinity and is also not realizable in practice. In accordance with the present invention, Applicant's invention first calculates the ideal frequency response for a given scaling factor and then uses this in step 102 to estimate a desired optimal frequency response. Calculating the optimal frequency response from the ideal frequency response is an iterative process. Different numbers of filter taps and smoothings are tried, with the difference between the optimal and the actual frequency response being measured in each iteration. This process is repeated until the optimal and the ideal response (within an acceptable tolerance limit). The optimum frequency response is produced by smoothing according to the number of filter taps used. Fig. 4b shows the smoothing estimate from Fig. 4a. Once this smoothing estimate is determined, it is fed to a filter design algorithm (such as the Parks-Molellan algorithm) in step 103, together with the desired number of filter taps. The Parks-Molellan algorithm is commercially available (e.g., as part of the MATLAB software package, The MathWorks, Inc., Cochituate Place, 24 Prime Parkway, Natick, MA 01760).

The Parks-McLellan FIR filter design algorithm is perhaps the most widely used and widely used FIR filter design methodology. In the signal processing toolbar, the function named Remez designs linear phase FIR filters using the Parks-Molellan algorithm. The Parks-Molellan algorithm uses the Remez exchange algorithm and Chebyshev approximation theory to design filters with optimal matches between the desired and actual frequency responses. The filters are optimal in the sense that the maximum error between the desired frequency response and the actual frequency response is minimized. Filters designed in this way exhibit equiripple behavior in their frequency response and are thus sometimes called equiripple filters.

The function

b = remez (n, f, m)

returns a row vector b containing the n + 1 coefficients of the order in the FIR filter whose frequency-magnitude properties are matched to those given by the vectors f and m. The vectors f and m give the frequency-magnitude properties of the filter:

- f is a vector of frequency points specified in the range between 0 and 1, where 1.0 corresponds to half the sampling frequency (the Nyquist frequency).

- m is a vector containing the desired magnitude response at the points specified in f. The elements of m must occur in equal pairs.

- f and m must have the same length. The length must be even.

- The first point of f must be 0 and the last point 1.

- The frequencies must be in increasing order.

- Duplicate frequency points are allowed, but Remez separates them by 0.1 if they exactly coincide. Note that frequency transitions much faster than 0.1 are undesirable as they can introduce ripple amounts in the magnitude response.

- The graphical representation (f, m) can be used to display the filter shape.

The number of filter taps directly affects the cost of the real-time implementation of the filter. The filter design algorithm calculates the set of multiplier coefficients that match the number of taps in the optimal frequency response. The coefficients are collected or binned for a given output line in step 104. The coefficient sets (step 105) are rescaled for each set so that the sum of the coefficients is equal to one (1). In step 106, the input lines are multiplied and summed to give the values on the output lines. Figure 5 shows a four-tap filter for scaling. The four input lines are multiplied by the four coefficients in the four multipliers 41 and summed in the summer 43. The four inputs are separated by the line delays 45.

Figures 6b and 6c show the frequency domain responses of two scaling filters designed using this 3-to-4 scaling technique. These filters are referred to as filters 1 and 2, respectively. Filter 1 (Fig. 6b) has 4 taps per row, while filter 2 (Fig. 6c) has 6 taps per row. In Fig. 6a, the frequency response of the bilinear filter is shown for comparison. The hatched portion of each frequency response corresponds to the staircase region.

Figure 7 shows the filter kernels for the bilinear interpolation (Figure 7a), for filter 1 (Figure 7b) and for filter 2 (Figure 7c). Referring to Figure 7a, the prior art bilinear interpolation is shown with the coefficients shown at two taps per row. There are seven coefficients, with the first and seventh coefficients at 0.25, the second and sixth coefficients at 0.50, the fourth coefficient at the peak at the coefficient at 1.0, and coefficients three and five at 0.75. In accordance with Applicant's invention shown in Figure 7b, the filter 1 with four taps per line has 15 coefficients, with the reported values for the first and fifteenth ranging from -0.0019 and a maximum value of 0.25 for the eighth coefficient. The first coefficient is aggregated with the fifth, ninth and thirteenth, the aggregated set then corresponding to a set of coefficients for a given output line. Figure 7b shows the sixteenth coefficients for four taps per line for a 3-to-4 scaling filter. The number shown in the figure is the output from the filter design algorithm. As previously discussed, Applicant's method then decodes these numbers into a format suitable for actual scaling filter implementation. Thus, the outputs of the filter design algorithm, which contain the following set of numbers [-0.0019, -0.0077, -0.0128, 0.0, 0.0494, 0.1327, 0.2152, 0.25, 0.2152, 0.1327, 0.0494, 0.0, -0.0128, -0.0077 and -0.0019], are summarized into the following sets, where each set corresponds to a set of coefficients (one coefficient for each input row) for an output row:

[-0.0019, 0.0494, 0.2152, -0.0128], [-0.0077, 0.1327, 0.1327, -0.0077], [-0.0128, 0.2152, 0.0494, -0.0019], [0.0, 0.25, 0.0].

Since there are four taps per output line, every fourth coefficient represents the coefficient for the neighboring line.

These sets of coefficients are then rescaled so that each set sums to one, with the resulting values being applied to the multiplier 41 as shown in Fig. 5 and Fig. 8.

As shown in Fig. 7c, a design filter with six taps per output line is shown with twenty-four different coefficients ranging from 0 to a maximum of 0.2 in the middle of coefficient 12. In this embodiment, there are six coefficients (six lines) for a given output line, and these coefficients are again rescaled such that the set sums to 1. These scaled coefficients are the multiplier values in Fig. 5. Fig. 8 shows the 3-to-4 scaling implementation using filter 1.

Referring to Fig. 5a, it is shown how the samples are applied to the rows in 3-to-4 scaling. If an input row such as row 0 to 0' is directly aligned with the output row, it is a 1-to-1 relationship. If not, the 4-tap filter 1 has coefficient 0.8611 from input row 1 to output row 1', and between input row -1 and output row 1' the coefficient is -7.6 x 10-3, from input row 0 to output row 1' the coefficient is 0.197, and from input row 2 to output row 1' the coefficient is -0.0512. These input row coefficients are summed to a total of 1. These are scaled by the coefficients represented by filter 1. This represents, for example, the first set for the first output line, with the second set being the next set of coefficients summarized for the second output line, the third output line being the third set of summarized coefficients, and the fourth output line being the fourth set or summary. It then repeats, with the fifth, sixth, seventh and eighth output lines being the first set or summary and the second third and then fourth sentence or the second, third and then fourth summary.

Fig. 8b also shows the 3-to-4 scaling for filter 1, where the output row 2' is taken from rows 0, 1, 2 and 3, the coefficient being -0.0308 for input row 0, the coefficient being 0.05308 for rows 1 and 2, and the coefficient being -0.0308 for row 3. All of these coefficients are rescaled to sum to 1, this value becoming the multipliers for the corresponding input row, the rows being delayed as shown in Fig. 5 as they are summed to derive the output value for row 2'. Similarly, the input signal to the output line 3' from the input lines 1, 2, 3 and 4 has the coefficients indicated as -0.0512 from line 1, 0.8611 from line 2, 0.1977 from line 3 and -7.6 · 10-3 from line 4.

The filter in Fig. 7b is:

[-0.0019, -0.0077, -0.0128, 0.0, 0.0494, 0.1327, 0.2152, 0.25, 0.2152, 0.1327, 0.0494, 0, 0, -0.0128, -0.0077, -0.0019]

This is summarized as:

A = [-0.0019, 0.0494, 0.2152, -0.0128]

B = [-0.0077, 0.1327, 0.1327, -0.0077]

C = [-0.0128, 0.2152, 0.0494, -0.0019]

D = [0.0, 0.25, 0.0]

The elements of A add up to:

-0.0019 + 0.0494 + 0.2152, -0.0128 = 0.2499,

whereby each element of A is divided,

[-0.0019/0.2499, 0.0494/0.2499, 0.2152/0.2499, -0.0128/0.2499] = [-7.6 x 10⁻³, 0.1977, 0.8611, -0.0512]

Fig. 9 shows the scaling results of the bilinear (Fig. 9a), filters 1 (Fig. 9b) and 2 (Fig. 9c) for 3-to-4 scaling on a circular zone plate. This is a standard test used to evaluate interpolation algorithms. Note the significant stair-step effect in the upper image with the bilinear interpolation in Fig. 9a, the improvement with filter 1 (Fig. 9b) and the almost complete elimination of the stair-step effect with filter 2 (Fig. 9c).

The above scaling is vertical scaling by changing the number of horizontal rows. By changing the number of pixels (picture elements) per row and using neighboring pixels, this scaling can also be applied to width scaling. Fig. 2 would then have pixel delays instead of row delays. The steps in Fig. 3 of starting with an ideal frequency response and converting to an optimal frequency response by smoothing estimation would be followed. Then, based on the number of taps per row, the filter coefficients would be determined. Then, the coefficients are summed and scaled to the value of one. The neighboring input pixels are multiplied by the coefficients and summed to obtain the new output pixel.

Claims (2)

1. A method of forming an image scaling filter for converting a number of input lines into another number of output lines, comprising the steps: Calculating an ideal frequency response (101) for a given scaling factor; estimating an optimal frequency response (102) by an iterative process in which different numbers of filter taps and filter smoothings are tried and for each iteration the difference between the optimal and the actual frequency response is measured and the process is repeated until the difference is within an acceptable tolerance; generating a set of filter coefficients (103) based on the estimated optimal frequency response and the determined number of filter taps; grouping the coefficients (104) into a set for each output line corresponding to the input line closest to the output line; Scaling the coefficients in the set so that each set sums to 1 (105); multiplying the value of the input lines closest to the nearest output line according to the scaled coefficients in the set; and Sum the multiplied input lines to obtain the output line value (106). 2. The method of claim 1, wherein the generating step includes application of the Parks-Molellan algorithm.