WO1996018264A1

WO1996018264A1 - Digital image anti-aliasing

Info

Publication number: WO1996018264A1
Application number: PCT/IB1995/001050
Authority: WO
Inventors: Vassilis Seferidis
Original assignee: Philips Electronics N.V.; Philips Norden Ab
Priority date: 1994-12-08
Filing date: 1995-11-24
Publication date: 1996-06-13
Also published as: TW287342B; GB9424808D0

Abstract

A method is provided for the display of images in the form of an image object defined in a first pixel plane overlying a background defined in a further pixel plane. In order to counteract aliasing effects at display time, the edge pixels at the boundary of object and background are displayed with a transparency value comprising contributions from the corresponding pixel in both the first and further planes. Step discontinuities in the image edge have a slope characteristic applied across them and, for each pixel crossed by the slope characteristic, the relative areas of the pixel above and below the slope are calculated. The relative contributions of the image and background pixels are then set according to this relation. The slope characteristic may be a straight line (as shown) or some other suitable function such as a parabolic curve.

Description

64 PCMB95/01050

DESCRIPTION

DIGITAL IMAGE ANTI-ALIASING

The present invention relates to the coding of composite digital images and in particular to dealing with aliasing effects at the edges of foreground features overlying another object at greater image depth or a background.

One particular area where such aliasing problems occur is in multi-plane mixers which may, for example, combine a background video image with one or more overlaid planes containing animated graphics. Such a mixer accepts various sources of video data together with their associated depth maps and produces a new picture frame based on depth and transparency values of each pixel. Various coding schemes have been proposed for coding both the depth and transparency data in order to minimise the amount of data required to be stored. In terms of transparency data, it will be appreciated that the necessary data is greatly reduced when large areas of a picture have the same transparency or translucency value due to special effects such as fog or mist, reflections or diffractions, and fading in or out. Transparency values can also be used to reduce aliasing effects by blending at the edges of overlapping shapes. A problem which occurs, however, is that for complex images, a large amount of transparency information has to be encoded along with the picture data and, within the constraints of video systems, this can present a severe problem.

Accordingly, it is an object of the present invention to provide a method for anti-aliasing which reduces or removes the need for storage of transparency values with image data and which is capable of implementation at or close to real time in generating composite images for display.

In accordance with the present invention there is provided a method for display of images in the form of an image object defined in a first pixel plane overlying a background defined in a further pixel plane, wherein those pixels falling wholly within the object are displayed with the pixel value defined in the first pixel plane, those pixels falling wholly outside the object are displayed with the pixel value defined in the further pixel plane, and the remaining edge pixels are displayed with a transparency value comprising contributions from the corresponding pixel in both the first and further planes, characterised in that the transparency values are derived by: identifying step discontinuities in the image edge; - applying a slope characteristic across each step discontinuity; for each pixel crossed by the slope characteristic, deriving the relative areas of the pixel above and below the slope; and setting the relative contributions of the foreground and background pixels accordingly. By applying transparencies, which can be generated at display time, across the slope discontinuities (the steps on an aliased edge), the effects of aliasing can be reduced without requiring the pre-calculation and storage of transparency values. As will be described below, however, one or more of the transparency values may be pre-generated and stored with the image pixel data to be read from storage when the image is displayed, the remainder of the transparency values being calculated when the image is displayed. This partial storage may be used where system constraints limit the facility for display time calculation of transparencies. The transparency values may be pre- generated and stored for those step discontinuities having a vertical extent greater than or equal to horizontal extent, that is to say for those image edges having an edge slope of 45 degrees to greater to the horizontal. The step of derivation of relative areas for a pixel may comprise integration of the slope characteristic across the pixel to produce a first area value followed by subtraction of this from the total pixel area to give a second area value. Alternatively, the derivation may comprise dividing each pixel in a ratio given by the position of that pixel and the number of pixels across the step discontinuity.

The slope characteristic may be a straight line or a curve function, such as an elliptical or parabolic curve.

Also in accordance with the present invention there is provided a video mixing apparatus operable to form an output image signal from first and second pixel plane signals and a matte signal identifying, for each pixel of the first plane, whether or not it comprises a part of an image object defined in the first plane to be displayed overlying a background defined in the second pixel plane, the apparatus comprising: an edge classification stage arranged to receive the matte signal and identify therefrom step discontinuities at the edge of the image object; first calculation means arranged to apply a predetermined slope characteristic across step discontinuities identified by the edge classification stage; second calculation means arranged to determine, for each pixel crossed by the slope characteristic, the relative areas of the pixel above and below the slope; and mixer stage coupled to receive the first and second pixel plane signals and an indication of relative areas from the second calculation means, and arranged to output a display pixel image signal in which the values of pixels falling wholly within the image object are taken from the first pixel plane signal, the values of pixels falling wholly outside the image object are taken from the second pixel plane signal, and the values of those pixels identified as edge pixels by the edge classification stage are taken from both first and second pixel planes in a ratio by the said indication from the second calculation means.

In order to improve the flexibility of operation, the above- mentioned first calculation means may be operable to apply a selected one from a plurality of stored slope characteristics (for example elliptical or parabolic as mentioned above) with the apparatus further comprising a function store holding implementation instructions for each of the different characteristics supported.

Preferred embodiments of the present invention will now be described, by way of example only, and with reference to the following drawings, in which:

Figure 1 is a block schematic diagram of a decoder apparatus including a mixer stage embodying the present invention; Figure 2 is a schematic diagram of a stage of the mixer of Figure

1 ;

Figure 3 shows a cascaded arrangement of mixer stages as in Figure 2;

Figure 4 shows an aliased image produced by a known technique; Figure 5 shows image pixels stored for use in the technique producing Figure 4;

Figure 6 illustrates edge pixel contributions over two horizontal scan lines;

Figure 7 shows the image of Figure 4 to which a first anti-aliasing technique has been applied;

Figure 8 shows the pixels stored for use in the method producing the image of Figure 7; and

Figure 9 shows the image of Figure 4 to which a second anti- aliasing technique has been applied.

Figure 1 shows a general arrangement for a multi-plane video image system coupled to the host bus 10 of a CD-i (Compact Disc interactive) player. From the host bus 10 the base case video, which may be used to form a background to the final image, is fed to a video mixer 12, together with the video synchronising signals. Also providing inputs to the video mixer are a graphics processor 14 and a digital video decoder 16. The graphics processor and digital video decoder take their inputs from the host bus 10 which also provides control signals to the mixer 12. The output of the mixer stage comprises digital signals for the final image colour components (R,G,B) together with the video synchronisation signals. Following conversion by a digital to analogue converter 18, the analogue video signal is output on connector 20.

A first stage of the video mixer 12 is shown in Figure 2 having inputs for the base case video 22 and for the video image 24 of an object to be overlaid on the background by combining in a mixer 26. Accompanying the video image 24 is a matte 28 which is simple binary representation of the object, suitably using 1 to represent those pixels which are part of the image and 0 to represent all other pixels within the video frame. In conventional video mixing, the matte signal is used by the mixer to select either pixels from the image or pixels from the base case to make up the output picture. This technique is, however, prone to producing aliased images (that is to say the well known "staircase" effect on sloping edges) due to the whole pixel nature of the transition between the image and the background.

In order to reduce the aliasing effects, an edge classification stage 30 uses the matte signal to derive a model of the edge of the object (by noting the transition from 0 to 1 in the matte) and identifies the steps in the aliased edge to a calculation circuit 32. The calculation circuit applies a function to the values of pixels forming this step, as will be described below, and passes the result of this calculation to a further calculation stage 34 which derives a transparency value to be applied to these edge pixels. These transparency values are supplied to the mixer 26 which outputs a total picture 38 in which those pixels which are purely object or purely background have their original input values, and the edge pixels have a value which contains a contribution from both in amounts determined by the calculation carried out at stages 32 and 34. Where different functions may be used to obtain the edge pixel values, a store of different function models 36 may be provided.

In order to build up a picture having several overlaid images, mixer stages as in Figure 2 are cascaded in the manner shown in Figure 3 (note that the edge classification and calculation stages of Figure 2 are omitted for reasons of clarity). In the first stage, the background image 22 is combined through use of the appropriate matte 28 with the input image 24 having the greatest depth (Z) value to generate a composite picture. This composite picture is then combined in a further mixer 26a with an object image 28a having a depth value less than that for the first image object and by use of a further matte 28a. Subsequent images 24b with associated matte 28b are added in like manner such that the final picture is built up from the background and then through the image objects in descending order of depth.

For the purposes of comparison and illustration a known anti¬ aliasing technique will now be described, as used to calculate the transparency value by averaging (low-pass filtering) the values of neighbouring pixels within a small window (for example 3 x 3 or 5 x 5 pixels) for each of the edge pixels. The anti-aliased effect may be improved if an original higher resolution image is used: for example, an original image consisting of 720 x 576 pixels may be reduced by averaging four neighbouring pixels and then thresholding the resulting values to provide a sub-sampled version of the original image with a 360 x 288 pixel resolution. An aliased picture for such a reduced resolution image is shown in Figure 4. This method produces very good quality pictures and eliminates successfully the aliasing effect even for few quantisation levels (i.e four to eight levels or two to three bits). However, there is a major drawback in that the transparency value for each edge pixel has to be coded and stored for each frame. It is clear that the method requires a large overhead even for relatively simple situations: by way of example, Figure 5 shows the pixels for which a transparency value has to be stored for the picture of Figure 4. Whilst it would, of course, be possible to employ a compression coding algorithm to reduce the amount of data stored, there would be a corresponding increase in the complexity required at the decoder.

A first embodiment of the proposed method for anti-aliasing estimates the slope (i.e. the step height of the aliased edge) from two or more successive scan lines and then calculates for each pixel the area defined by this slope and the horizontal or vertical orientation. The area under the slope, which is used to dictate the relative contributions of the foreground and background pixels, is determined by the calculation unit 32 of Figure 2. For some implementations, it may be found that a slope characteristic other than a straight line gives improved results, for example an exponential or parabolic curve. Models of such different characteristics would be stored in the function model stage 36 of Figure 2, and the calculation stage 32 would comprise an integrator that determines the area under the particular characteristic chosen. Figure 6 illustrates an implementation of this first embodiment using a straight line characteristic over two successive scan lines which differ by four pixels (that is to say a step four pixels wide appears at the edge of the aliased image). The dark shaded area of Figure 6 represents the coverage of each pixel by an object with a near horizontal top edge and thus can be used to calculate its value. Simple geometry shows that, given the difference N between the number of pixels across the two lines, the value of each pixel can be calculated by equation 1 below:

value = 25δ Equation 1

where k = 1 , 3, 5 (2N- 1 ).

A second embodiment of the method trades a degree of accuracy for the calculation against simplicity of hardware implementation by using estimation of the required step for a pixel displacement. This is simply obtained from the ratio of pixels in a line 2N, leading to equation 2 below:

value = K.step = Equation 2

where K = 0, 1 , 2, 3 .... N

Experimental results have shown that the method of both equation 1 and equation 2 work reasonably well with an expected improved performance for the equation 1 method due to its accuracy. However, considering the implementation issues, the second method is preferred for ease of hardware implementation. The main advantage of both methods is that they provide the ability to calculate the coverage of each edge pixel in real time, eliminating the need to store transparency values.

Whilst the slope estimation method described above is a general technique in the sense that it can be used for both near-horizontal and near-vertical edges, hardware and economic constraints may dictate that only a few scan lines of picture are available to the mixer at any given time. In such a case, the slope estimation can be realistically used only for near-horizontal edges. Figure 7, for example, shows the implementation of the slope estimation method on all the edges of the picture of Figure 4 but within the constraint of a maximum of two adjacent scan lines being available at any time. Here a two-bit (four level) transparency value has been used for each anti-aliased pixel. It will be noted that although the near-horizontal lines are anti-aliased satisfactorily, this is not the case for the near vertical lines which remain aliased. To overcome this problem, and to secure efficient anti¬ aliasing for pixels belonging to near vertical edges, it is preferred to store their transparency values on the disc. Although this seems to be expensive concerning the storage requirements, there are two reasons that favour this choice. Firstly, the number of pixels which will have their transparency values stored on the disc is relatively small: as the example of Figure 8 shows, the pixels of this type for the test picture of Figure 4 are greatly reduced when compared to Figure 5. It is evident that, due to their vertical orientation, a simple coding scheme can be used to store them efficiently. Moreover, it has been found that the required resolution for each of these stored transparency values need not be high with only one or two bits being sufficient for most cases.

Turning now to the calculation of the transparency values to be stored for near-vertical edges, there are several possible solutions to this. The preferred technique calculates the transparency value based on the contribution of pixels from a higher resolution (e.g 720 x 576 pixels) video frame to obtain the 360 x 288 standard format picture, with the transparency value for an edge pixel in the lower resolution picture being the average of the four pixels in the higher one.

Depending on the storage constraints, a quantisation of this value may be required. For more visually pleasing results, a low-pass filter may also be applied to the higher resolution picture prior to calculation of the transparency values. Figure 9 shows the final anti-aliased picture obtained from the application of the slope estimation method for the horizontal edges and the use of the stored transparency values in the vertical direction. Each stored value is represented with only one bit (two transparency levels). For this rather extreme case of several anti-aliased vertical edges, it has been found that almost 600 bits of transparency information must be stored in addition to the depth map. As mentioned above, it would be possible to reduce this figure by adopting a coding scheme for the vertical transparency data but with a corresponding increase in complexity in decoder architecture. From reading the present disclosure, other variations will be apparent to persons skilled in the art. Such variations may involve other features which are already known in the methods and apparatuses for editing of audio and/or video signals and component parts thereof and which may be used instead of or in addition to features already described herein. Although claims have been formulated in this application to particular combinations of features, it should be understood that the scope of the disclosure of the present application also includes any novel feature or any novel combination of features disclosed herein either implicitly or explicitly or any generalisation thereof, whether or not it relates to the same invention as presently claimed in any claim and whether or not it mitigates any or all of the same technical problems as does the present invention. The applicants hereby give notice that new claims may be formulated to such features and/or combinations of such features during the prosecution of the present application or of any further application derived therefrom.

Claims

1 . A method for display of images in the form of an image object defined in a first pixel plane overlying a background defined in a further pixel plane, wherein those pixels falling wholly within the object are displayed with the pixel value defined in the first pixel plane, those pixels falling wholly outside the object are displayed with the pixel value defined in the further pixel plane, and the remaining edge pixels are displayed with a transparency value comprising contributions from the corresponding pixel in both the first and further planes, characterised in that the transparency values are derived by: identifying step discontinuities in the image edge; applying a slope characteristic across each step discontinuity; for each pixel crossed by the slope characteristic, deriving the relative areas of the pixel above and below the slope; and setting the relative contributions of the foreground and background pixels accordingly.

2. A method as claimed in Claim 1 , wherein one or more of the transparency values is pre-generated and stored with the image pixel data to be read from storage when the image is displayed, and the remainder are calculated when the image is displayed.

3. A method as claimed in Claim 2, wherein the transparency values are pre-generated and stored for those step discontinuities having a vertical extent greater than or equal to horizontal extent.

4. A method as claimed in Claim 1 , wherein the step of derivation of relative areas for a pixel comprises integration of the slope characteristic across the pixel to produce a first area value and subtracting this from the total pixel area to give a second area value.

5. A method as claimed in Claim 1 , wherein the step of derivation of the relative areas of the pixel comprises dividing each pixel in a ratio given by the position of that pixel and the number of pixels across the step discontinuity.

6. A method as claimed in Claim 1 , wherein the slope characteristic is a straight line.

7. A method as claimed in Claim 1 , wherein the slope characteristic is a curve.

8. Video mixing apparatus operable to form an output image signal from first and second pixel plane signals and a matte signal identifying, for each pixel of the first plane, whether or not it comprises a part of an image object defined in the first plane to be displayed overlying a background defined in the second pixel plane, the apparatus comprising: an edge classification stage arranged to receive the matte signal and identify therefrom step discontinuities at the edge of the image object; first calculation means arranged to apply a predetermined slope characteristic across step discontinuities identified by the edge classification stage; second calculation means arranged to determine, for each pixel crossed by the slope characteristic, the relative areas of the pixel above and below the slope; and mixer stage coupled to receive the first and second pixel plane signals and an indication of relative areas from the second calculation means, and arranged to output a display pixel image signal in which the values of pixels falling wholly within the image object are taken from the first pixel plane signal, the values of pixels falling wholly outside the image object are taken from the second pixel plane signal, and the values of those pixels identified as edge pixels by the edge classification stage are taken from both first and second pixel planes in a ratio by the said indication from the second calculation means.

9. Apparatus as claimed in Claim 8, wherein the first calculation means is operable to apply a selected one from a plurality of stored slope characteristics, the apparatus further comprising a function store holding implementation instructions for each of said slope characteristics.