GB2455142A

GB2455142A - Pre-processing of image data by marking of macroblocks

Info

Publication number: GB2455142A
Application number: GB0723517A
Authority: GB
Inventors: Ping Wu; Alois Martin Bock; Matthew Patrick Compton
Original assignee: Tandberg Television AS
Current assignee: Ericsson Television AS
Priority date: 2007-11-30
Filing date: 2007-11-30
Publication date: 2009-06-03
Also published as: GB0723517D0; WO2009068553A1

Abstract

A method of pre-processing video image data is provided that comprises analysing the image data at macroblock level to determine an activity level for each macroblock, marking each macroblock with a pre-determined type according to the determined activity level of each macroblock, and re-marking the type of certain ones of the macroblocks dependent upon the marked activity level of neighbouring macroblocks. Each macroblock can be categorised into one of five types, these are: Type 1, low activity indicating plain texture; Type 2, more activity that type 1, but still low texture activity; Type 3, the texture activity is identical to type 2 but includes a dominant vertical or horizontal structure; Type 4, these include skin tones; and, type 5, indicated by high levels of texture activity.

Description

Marking of Images

Field of the Invention

The invention is related to video coding in general, and in particular to pre-processing image data by marking of macroblocks.

Background

The performance of a video compression process is heavily dependent on the image content of the video being compressed and poor encoding can result if typical algorithms such as MPEG-2 and MPEG-4 are not enhanced in various ways to optimise coding conditions. This is because video compression (encoding) relies upon analysis of the differences between successive frames (or fields) to derive the compressed data. Therefore, if an input video sequence typically comprised of frames or fields can be analysed in detail at a pre-processing stage, the understanding of the behaviour of the sequence gained will be extremely useful in optimising the encoder performance.

Summary of Invention

Embodiments of the invention provide a method that may be applied on subsections or blocks of images in a video data sequence and which will mark such blocks with a label, or type, such that a subsequent encoding process will be guided towards optimum performance. The type used to mark the macroblocks may be arbitrarily chosen to refer to particular features which are advantageous in knowing during the compression process.

In a first aspect of the invention, there is provided a method of pre-processing video image data, comprising analysing the image data at macroblock level to determine an activity level for each macroblock, marking each macroblock with a pre- determined type according to the determined activity level of each macroblock, re-marking the type of certain ones of the macroblocks dependent upon the marked activity level of neighbouring macroblocks.

Preferably, the method further comprises testing each macroblock for horizontal or vertical edge dominance and where dominance is found in a particular macroblock, re-marking the type of the macroblock of interest to a type indicative of a macroblock having a dominant vertical or horizontal edge.

Preferably, a test for horizontal or vertical dominance includes determining a total vertical edge energy value for each macroblock, determining a total horizontal edge energy value for each macroblock, calculating a dominance from said total vertical edge energy value and said total horizontal edge energy value.

Preferably, the step of determining a total vertical edge energy value of a macroblock of interest comprises dividing said macroblock into sub-blocks, forward DCT transforming each sub-block of the macroblock, squaring all values in a first row of samples derived from the forward DCT transform of each sub-block, except a sample at a (0,0) location, adding all squared row sample values together to provide a sub- block total vertical edge energy value (sVE) for the sub-block, and adding all sub-block total vertical edge energy values (sVEs) for all sub-blocks of the macroblock together to provide a macroblock total vertical edge energy value (yE).

Preferably, the step of determining a total horizontal edge energy comprises squaring all values in a first column of samples derived from the forward DCT transform of each sub-block, except a sample at a (0,0) location, adding all squared column sample values together to provide the sub-block total horizontal edge energy value (sHE) for sub-block, and adding all sub-block total horizontal edge energy values (sHEs) for all sub-blocks of macroblock together to provide the macroblock total horizontal edge energy value (HE).

Preferably, the step of calculating a dominance uses the equation: D=(abs(HEVE)*1 00)/(HE+VE), where if D>90 and HE>VE, horizontal edges are dominant, and If D>90 and VE>HE, vertical edges are dominant, otherwise there is no clear edge dominance.

Preferably, the method further comprises testing each macroblock for the presence of a predetermined amount of a particular tone of colour, re-marking the type of each macroblock of interest to a type indicative of a macroblock having said particular tone of colour, where the predetermined amount of said particular tone of colour is found.

Preferably, the particular tone of colour is a skin tone, indicative of the macroblock containing portions of a human character in the image data.

Preferably, the test for a predetermined amount of a particular skin tone comprises sampling each macroblock in 4:2:0 colour space video format to obtain a luminance sample, and two associated chrominance samples: a Cr sample and a Cb sample, picking rows and columns at even index locations in the luminance sample, resetting a counter to zero, looping over the picked luminance sample and the two associated chrominance samples to check for all of the following to be true.

(1) Luminance presentative Y < 240; (2) Luminance presentative Y> 30; (3) Sample Cr> 100; (4) Sample Cr < 120; (5) Sample Gb> 137; (6) Sample Cb < 160; wherein if (1) to (6) are found to be true, the counter in increased by 1; re-marking the macroblock of interest to a type indicative of a macroblock having said particular tone of colour if the counter number is greater than or equal to 32.

Preferably, the step of analysing the image data at macroblock level to determine an activity level for each macroblock comprises determining a mean luminance value for the macroblock of interest, calculating an absolute difference between a luminance value for each pixel of the macroblock of interest and the mean luminance value of the macroblock of interest, summing all calculated absolute differences and dividing the sum of absolute differences by the total number of pixels in the macroblock of interest.

Preferably, the step of marking each macroblock with a pre-determined type according to the determined activity level of each block, comprises marking as a first type if the determined activity level is less than 2, marking as a second type if the determined activity level is between 2 and 13, marking as a fifth type if the determined activity level is greater than 13.

Preferably, the type indicative of a macroblock having a dominant vertical or horizontal edge is a third type.

Preferably, a type indicative of a macroblock having a particular tone of colour is a fourth type.

Preferably, the video data comprises a stream of pictures, wherein each picture in said stream of pictures comprises a plurality of pixels, and wherein the plurality of pixels are arranged into macroblocks, and wherein the step of re-marking certain ones of the macroblocks further comprises a first number of iterations looping over all macroblocks in the picture of interest, excluding macroblocks at an edge or corner of the picture, for re-marking macroblocks with the second and fifth types of activity level, comprising an iteration for re-marking any macroblock of interest not already of the fifth type to become the fifth type if at least five of the neighbouring macroblocks are of the fifth type, and re-marking any macroblock of interest of a fifth type to become the second type if less than three of the neighbouring macroblocks are of the fifth type, an iteration for re-marking any macroblock of interest not already of the fifth type to become the fifth type if at least six of the neighbouring macroblocks are of the fifth type, and re-marking any macroblock of interest of a fifth type to become the second type if less than two of the neighbouring macroblocks are of the fifth type, an iteration for re-marking any macroblock of interest not already of the fifth type to become the fifth type if at least seven of the neighbouring macroblocks are of the fifth type, and re-marking any macroblock of interest of a fifth type to become the second type if less than one of the neighbouring macroblocks are of the fifth type.

Preferably, the step of re-marking certain ones of the macroblocks further comprises a second number of iterations looping over all macroblocks in the picture of interest, excluding macroblocks at an edge or corner of the picture, for dealing with re-marking of macroblocks with the first and second types of activity level, comprising an iteration for re-marking any macroblock of interest that is already of the first type to become the second type if at least five of the neighbouring macrob locks are not of the first type, and re-marking any macroblock of interest of the second type to become the first type if less than three of the neighbouring macroblocks are not of the first type, an iteration for re-marking any macroblock of interest that is already of the first type to become the second type if at least six of the neighbouring macroblocks are not of the first type, and re-marking any macroblock of interest of the second type to become the first type if less than two of the neighbouring macroblocks are not of the first type, an iteration for re-marking any macroblock of interest that is already of the first type to become the second type if at least seven of the neighbouring macroblocks are not of the first type, and re-marking any macroblock of interest of the second type to become the first type if less than one of the neighbouring macroblocks are not of the first type.

Preferably, the number of iterations is three.

Preferably, the step of re-marking certain ones of the macroblocks further comprises a third number of iterations looping over all macroblocks in the picture of interest, excluding macroblocks at an edge or corner of the picture, for dealing with re-marking of macroblocks with the second, third and fifth types of activity level, comprising an iteration for re-marking any macroblock of interest that is already of type three to become type two if at least five of the neighbouring macroblocks are of the type two, an iteration for re-marking any macroblock of interest that is already of type three to become type two if at least six of the neighbouring macroblocks are of type two, an iteration for re-marking any macroblock of interest that is already of type two to become the type three if at least five of the neighbouring macroblocks are of type three, an iteration for re-marking any macroblock of interest that is already of type two to become the third type if at least six of the neighbouring macroblocks are of type three, an iteration, where if the macroblock of interest is already marked as type three and none of the neighbouring macroblocks are marked type three then,if at least four neighbouring macroblocks are already marked as type five, then remark the macroblock of interest to become type five, otherwise mark as second type, otherwise do nothing.

Preferably, the step of re-marking certain ones of the macroblocks further comprises a third number of iterations looping over all macroblocks in the picture of interest, excluding macroblocks at an edge or corner of the picture, for dealing with re-marking of macroblocks with any of activity level types first to fifth, comprising an iteration where, if the macroblock of interest is already marked type five and none of the neighbouring macroblocks are already marked type five, then if at least 4 of the neighbouring macroblocks are marked type four, re-mark the macroblock of interest to become type four, otherwise remark to type two, otherwise do nothing, an iteration where, if the macroblock of interest is already marked type one and none of the neighbouring macroblocks are already marked type one, then if at least 4 of the neighbouring macroblocks are marked type four, re-mark the macroblock of interest to become type four, otherwise remark to type two otherwise do nothing, an iteration, where if at least five of the neighbouring macroblocks are type four and the macroblock if interest is type two or three, re-mark the macroblock of interest to become type four, else if five or more of the neighbouring macroblocks are type one, two or three, and the macroblock of interest is type four, then remark macroblock of interest to become type two, a further iteration, where if at least five of the neighbouring macroblocks are type four and the macroblock if interest is type two or three, re-mark the macroblock of interest to become type four, else if five or more of the neighbouring macroblocks are type one, two or three, and the macroblock of interest is type four, then remark macroblock of interest to become type two, an iteration where, if the macroblock of interest is marked type four and none of the neighbouring macroblocks are marked type four, then if at least 4 of the neighbouring macroblocks are marked type five, re-mark the macroblock of interest to become type five otherwise remark to type two, otherwise do nothing, an iteration where if the macroblock of interest is type two and none of the neighbouring macroblocks is type two, then if at least four of the neighbouring macroblocks are type four, re-mark macroblock of interest to type four, else if at least four of the neighbouring macroblocks are type five, remark the macroblock of interest to type five, else if at least four of the neighbouring macroblocks are type one, re-mark macroblock of interest to type one, else if more or equal neighbouring macroblocks are type four than type three, remark the macroblock of interest to type four, else if more or equal neighbouring macroblocks are type one than type three, re-marl the macroblock of interest to type one, otherwise re-mark the macroblock of interest to become type three, otherwise do nothing.

Preferably, the step of re-marking certain ones of the macroblocks further comprises an iteration looping over all corner macroblocks in the picture only, comprising if all three neighbouring macroblocks are marked a type that is different to the current type of the corner macroblock of interest, then if at least two neighbouring macroblocks are marked type five, re-mark the corner macroblock of interest to become type five, otherwise re-mark the corner macroblock of interest to become type two, otherwise do nothing, an iteration looping over all edge macroblocks in the picture only, comprising if all five neighbouring macroblocks are marked a type that is different to the current type of the edge macroblock of interest, then if at least three neighbouring macroblocks are marked type five, re-mark the edge macroblock of interest to become type five, otherwise re-mark the edge macroblock of interest to become type two, otherwise do nothing.

Preferably, the video data comprises a stream of pictures, wherein each picture in said stream of pictures comprises a plurality of pixels, and wherein the plurality of pixels are arranged into macroblocks.

Preferably, the picture is a frame or field.

Preferably, the method further comprises the step of outputting the marked macroblocks for use in further processing of the video data using the marks.

Preferably, each macroblock is a 16 by 16 pixel macroblock.

Preferably, a sub-block is a 8 by 8 pixel sub-block.

Preferably, the method is carried out once for each frame and once for each field in a

stream of fields and frames.

In a second aspect of the invention, there is provided an apparatus adapted to carry out the above described method.

A computer-readable medium, carrying instructions, which, when executed, causes computer logic to carry out any the above described method.

A particular type of image block commonly used in image compression is the macroblock, which is a 16x16 pixel sized block of pixels used in video encoding/decoding calculations. The following embodiments of the invention will be described in terms of such a 16x16 macroblock (hereinafter referred to as MB) as a convenient basis for the application of the method described. However, it will be apparent to the skilled practitioner in the art of video compression that embodiments of the invention may be equally applied to any arbitrary choice of block size.

Equally, sub-blocks of macroblocks will be used, where in the specific embodiment described herein, the sub-block are 8 by 8 pixel.

The invention proposes several stages of processing: firstly there is an initial marking stage followed by a re-marking stage where the marking of like image macroblocks is refined and made consistent in the local neighbourhood of each macroblock. Next the marking is further refined by deliberate searches for major features of image texture such as skin tone and strong vertical or horizontal edges.

Finally, the image boundaries, where regional refinement is limited due to the absence of neighbouring blocks, are treated. Whilst these are the major stages and image features that have been found by experiment to be beneficial, other feature extraction processes may be added if it is found that coding can be improved by seeking to detect the presence of such features in the input video stream.

Pre-processing of video data is especially advantageous if the texture of the images forming the video data is evaluated prior to compression.

Brief description of the drawings

A method of pre-processing video data will now be described, by way of example only, with reference to the accompanying drawings in which: Fig. 1 shows an illustration of a current MB of interest and the eight neighbouring MBs; Fig. 2 shows an illustration of a picture corner MB, and its neighbouring MBs; Fig. 3 shows an illustration of a picture edge MB, and its neighbouring MBs; Fig. 4 shows an illustration of how the method is applied, in a raster scan across non-corner, and non-edge MBs, according to an embodiment of the invention; Fig. 5 shows an illustration of a general (overall) marking method according to an embodiment of the invention.

Fig. 6 shows an illustration of an activity measurement method for each MB, according to an embodiment of the invention; Fig. 7 shows an exemplary application of the activity measurement method of Fig. 6 to MBs, according to an embodiment of the invention; Fig. 8 shows a more detailed illustration of the general (overall) marking method of Fig. 6; Fig. 9 shows an exemplary embodiment of a first portion of the re-marking method according to the invention, dealing with type two and type five MBs; Fig. 10 shows an exemplary embodiment of a second portion of the re-marking method according to the invention, dealing with type one and type two MBs; Fig. 11 shows an exemplary embodiment of a method according to the invention for determining whether a macroblock has a dominant vertical or horizontal edge; Fig. 12 shows an exemplary embodiment of a third portion of the re-marking method according to the invention, dealing with type two, type three and type five MBs; Fig. 13 shows an exemplary embodiment of a method according to the invention for determining whether a macroblock has a dominant particular tone of colour, and the respective type four marking process; Fig. 14 shows an exemplary embodiment of a fourth portion of the re-marking method according to the invention, dealing with any of types one to five; Fig. 15 shows an exemplary embodiment of the re-marking method according to the invention, dealing with corner macroblocks; Fig. 16 shows an exemplary embodiment of the re-marking method according to the invention, dealing with edge MBs.

Detailed description of the specific embodiment

An embodiment of the invention will now be described with reference to the accompanying drawings in which the same or similar parts or steps have been given the same reference numerals.

Block based categorisation: Typically an input video sequence of colour images will be comprised of a stream of pixels, most often produced from a raster scan. The first step in the proposed process is to organise each video picture, either a whole frame or two separate fields (top field and bottom field), into a whole number of blocks. It is usually convenient to use a set of contiguous blocks of 16x16 pixels; each 16x16 block is commonly called a macroblock (MB). The distribution of pixel values (or transformed values of such pixel values, when taken as blocks) within each macroblock will be examined and each macroblock will be categorised into a pre-defined type group.

This distribution of pixel values in each macroblock expresses the texture of the image and this texture has a significant effect on the quality of coding, according to whether the pixels vary considerably within each macroblock, which suggests large amounts of fine detail, or whether they are very similar in value, suggesting a plain area. In particular, the coding quality varies as the texture changes from macroblock to macroblock and, depending upon the coding method chosen, it will be helpful if the image texture behaviour is categorised into several types prior to coding so that a subsequent coding method is given early warning of this behaviour along with some specific statistical indication of what kind of behaviour the image has at any particular point. The image activity can be separately expressed in the luminance and chrominance domains or both in combination. It may also be expressed separately in the time domain or a suitable transform domain or both in combination.

The method operates upon macroblocks one at a time, using the markings of the neighbouring macroblocks to determine whether, and how, to change the marking of the current macroblock of interest. The macroblock of interest is moved on one step along a raster scan line (for example, see Fig 4, item 460) after the application of the method to the current macroblock of interest time.

Fig. 1 shows an illustration of a current MB of interest 10 and the eight neighbouring MBs 20, for the case where the current macroblock of interest 10 is neither a corner macroblock, nor an edge macroblock (i.e. on the boundary of the whole picture).

Figs. 2 and 3 show the cases of. the current macroblock of interest being a corner macroblock; and the current macroblock of interest being an edge macroblock, respectively.

It is advantageous to consider a total of five categories (i.e. types) of behaviour for each macroblock. It is also advantageous to form the macroblocks into groups having like behaviour in the same region of the image and to mark them with specific labels, for example: 0, 1, 2, 3 and 4, denoting a first to fifth type, respectively.

The categories of macroblock type to be employed in an embodiment of the invention are as follows: Type 1: The macroblocks in this type group only contain extremely low activity exhibiting plain texture. In video coding, such a macroblocks will more readily exhibit blockiness artefacts if the quantisation parameter value used by the encoder is too large. This type of macroblock is the weakest in terms of exploiting image masking effects and therefore needs to be well coded A Type I label therefore instructs the encoder to use the smallest value of quantisation parameter consistent with the existing conditions in the encoding loop.

Type 2: In general these macroblocks have low texture activity but more than that in type 1.

Type 3: These macroblocks also fall into Type 2 according to their activity level measures, but there is also an indication that there is a dominant vertical or horizontal structure appearing in these macroblocks. Since this could indicate the presence of edges in the texture, extra coding care can be applied accordingly.

II

Type 4: These macroblocks are characterised by the presence of skin tones. Since the human face is likely to be a focus of interest for human viewers the marking of such a macroblock type allows the encoder to treat it specially to ensure a better decoded quality compared with other regions in the same picture.

Type 5: These macroblocks are characterised by high values of activity texture. In this case, the video masking effect is at its strongest in terms of exploiting image masking effects and therefore these macroblocks need not be as well coded as those in other groups. Typically, bits can be saved in coding this type of macroblock and these saved bits may be reallocated to compensate those niacroblocks marked as Type 1,2,3, and 4.

It will be apparent to the skilled practitioner that the number of macroblock types used may be increased or decreased, according to how much detailed pre-processing is required. Also, any other labels may be used, such as 0, 1, 2, 3, and 4, to account for digital representation of the type labels.

It is advantageous that the marking described be done separately for both frame pictures and field pictures, where both frame and field picture coding options are allowed in the encoding process. Clearly more types can be added as required to analyse other features of the image.

The proposed overall method of the macroblock based texture marking comprises four process stages (shown in Fig. 5). Clearly more stages may be added as required to enhance the process if more image features are found that contribute usefully to the analysis. Each process stage has two parts: firstly an initial marking of each block with its type followed by a re-marking in which the initial marking of individual blocks may be revised according to the marking of neighbouring blocks.

This important step is added to ensure continuity over regions of the image which vary in content.

The initial marking process is carried out for all macroblocks, in a raster scan across each macroblock in the target picture. However, the first three stages of re-marking processing are carried out on only the non-edge and non-corner macroblocks. Fig. 4 shows an illustration of how the marking method described is applied first in a raster scan 450 across all macroblocks, and then how the re-marking method is applied subsequently in a raster scan for all non-corner and non-edge macroblocks only 460, bypassing the corner macroblocks 410 and the edge macroblocks 420. The corner macroblocks 410 and edge macroblocks 420 are re-marked in stage four, again in a raster scan fashion, only this time skipping between the relevant types of macroblocks.

Returning to Fig. 5, there is shown the general outline of the overall process. In more detail, the four stages of the process are as follows: First stage marking: First, the picture is analysed on a macroblock basis in a raster scan order. For each macroblock the activity level is measured. The activity measurement method 600 for each macroblock 10 is illustrated by Fig. 6. It comprises: summing all luminance values for the pixels in the macroblock of interest 610; dividing the sum of luminance values for all pixels in the macroblock of interest by the total number of pixels in the macroblock to derive a mean pixel luminance value for the macroblock of interest 620; calculating the absolute difference between each pixel luminance value and the mean pixel luminance value 630; summing all the calculated absolute differences 640; and dividing the sum of all absolute differences by the total number of pixels in the macroblock of interest 650 before ending 660 the activity level calculation sub-process.

Using the measured activity level, the macroblocks are initially marked according to their measured activity using the labelling process 700 shown in Fig. 7.

For each macroblock, the activity is measured (step 710) according to the process 600, then: If the measured activity level is less than 2 (step 710), the macroblock is initially marked as a Type 1 (step 740), otherwise, if the activity level is greater than 13 (step 730), the macroblock is initially marked as a Type 5 (step 760); if it does not fall into Type 1 or Type 5 (step 720), it will initially be marked as a Type 2 (step 750) At this stage Type 3 and Type 4 markings are not performed. Advantage accrues throughout this invention by the judicious choice of the particular threshold values.

First stage remarking: Since in video encoding a lot of individually marked Types (groups) done in isolation will bring chaos into the change of the quantisation parameters, there could potentially be a loss in coding efficiency. Therefore, re-marking methods are introduced to rationalise the results of the marking stage. The isolated individual marking means that for a macroblock marked as a particular Type (1, 2, 3, 4, 5) it is possible that a majority of its neighbouring macroblocks are marked as a different type.

Fig. 8 shows a more detailed illustration of the overall process 800, in which the separate re-marking processes are shown: for the first stage (900, 1000); second stage 1200; third stage 1400, and finally fourth stage (for the corner 1500 and edge macroblocks 1600 respectively).

The re-marking method for the first stage is carried out as two parts, as follows: (1) individual, isolated markings in an area in which the majority markings are of Type 5 will be converted into Type 5 marking if it is necessary. There are three iterations for this process and the condition to take action will be made more rigorous each time; (2) similarly isolated markings in a majority of Type 1 area are examined and will be converted into Type I markings if it is necessary. Again there are three iterations for this grouping method.

The grouping method for Stage 1 has been illustrated into two figures: part I 900 in Fig. 9 and part 111000 in Fig. 10.

In more detail, the Part 1 re-marking process 900 involves: Whilst looping over the non-corner and non-edge macroblocks 10, if 5 or more neighbouring MBs are Type 5 and the current MB is not Type 5, remark current MB as Type 5 (step 920), otherwise if 3 or fewer neighbouring MBs are Type 5 and the current MB is Type 5, remark current MB as type 2 (step 930).

The process then loops back to the beginning of the raster scan of the macrobiocks 910 and starts a second more rigorous re-marking iteration in which: if 6 or more neighbouring MBs are Type 5 and the current MB is not Type 5, remark current MB as Type 5 (step 940), otherwise if 2 or fewer neighbouring MBs are Type 5, and the current MB is Type 5, remark current MB as type 2 (step 950).

The process then loops back to the beginning of the raster scan of the macroblocks 910 and starts a third, even more rigorous, re-marking iteration in which: If 7 or more neighbouring MBs are Type 5, and the current MB is not Type 5, remark current MB as Type 5 (step 960), otherwise if I or fewer neighbouring MBs are Type and the current MB is Type 5, remark current MB as Type 2 (step 970). This part I re-marking process 900 then ends at step 980 once all non-corner and non-edge macroblocks have been analysed for re-marking (but not necessarily actually re-marked) for a third time.

The Part lire-marking process 1000, which is carried out after the Part Ire-marking process 900 is as follows: Again, whilst looping over the non-corner and non-edge macroblocks 10, if 5 or more neighbouring MBs are not Type 1 and the current MB is Type 1, remark current MB as Type 2 (step 1010), otherwise if 3 or fewer neighbouring MBs are not Type 1 and the current MB is Type 2, remark current MB as Type 1 (step 1020).

The process then loops back to the beginning of the raster scan of the macroblocks 910 and starts a second more rigorous re-marking iteration, in which: If 6 or more neighbouring MBs are not Type 1 and the current MB is Type 1, remark current MB as Type 2 (step 1030), otherwise if 2 or fewer neighbouring MBs are not Type 1, and the current MB is Type 2, remark current MB as Type 1 (step 1040).

The process then loops back to the beginning of the raster scan of the macroblocks 910 and starts a third, even more rigorous re-marking iteration, in which: If 7 or more neighbouring MBs are not Type 1 and the current MB is Type 1, remark current MB as Type 2 (step 1050), otherwise if 1 or fewer neighbouring MBs are not Type 1 and the current MB is Type 2, remark current MB as Type 1 (step 1060). This part II re-marking process 1000 then ends at step 1080 once all non-corner and non-edge macroblocks have been analysed for re-marking (but not necessarily actually re-marked) for a third time.

After the above process is complete, the Type 1 and Type 5 regions have been rationalised and their characterisations are mutually more consistent.

In general, the marking and re-marking methods (as described at every stage in this document) can be applied to frame picture macroblocks and/or field picture macroblocks. After the stage 1 process, the macroblocks are only marked as Type 1, 2, or 5. The Type 3 marking and re-marking will come at stage 2, Type 4 marking and re-marking will come at stage 3, and final re-marking at stage 4.

In video encoding, the results of both the frame picture macroblock marking and the separate top and bottom field picture macroblock marking are expected to be available simultaneously. This is because the encoder will have to make a final decision at a very late stage on whether the picture is going to be encoded as a single whole-frame picture or two separate field pictures.

Second stage marking and re-marking: In this stage, those macroblocks that are initially marked into Type 2 but actually contain dominant vertical or horizontal edge information are identified more exactly.

The method involves DCT transformation of the image. Firstly each 16x16 macroblock luminance array is separated into four 8x8 sub-blocks and a forward 8x8 DCT transform is applied to all of them. Once the information is in the DCT transform domain, the vertical and horizontal detail dominance will be measured. If dominant horizontal or vertical edges are detected then the macroblock will be re-marked as a Type 3.

Fig. 11 shows this Type 3 marking process in more detail, in which the vertical energy VE and horizontal energy HE are measured at step 1110, then the dominance is calculated at step 1120 using the equation: D(abs(HE-VE)*100)/(HE+VE); where if D>90, and HE>VE, there is horizontal dominance; if D>90, and VE>HE, there is vertical dominance; and if the result is any other value, there is no clear edge dominance. At step 1130, the presence of either vertical or horizontal dominance is tested, and if it is present, the macroblock of interest is marked Type 3 at step 1140, otherwise it is left as it is. The vertical/horizontal edge dominance testing/marking process ends 1050 after all macroblocks are tested. This process may be carried out in macroblock raster scan order over the whole image 450 (including corner and edge macroblocks), or in macroblock raster scan order over the image 460 (excluding corner and edge rnacroblocks) instead.

When this has been done, a second stage re-marking method is applied in raster scan order, excluding corner and edge macroblocks 460 Fig. 12 shows this re-marking method 1200. This time the re-marking will conduct two iterations (1210, 1220) for the current macroblock marked as Type 2 and two iterations (1230, 1240) for the current macroblock marked as Type 3, respectively. After that, a further re-marking method step 1250 is applied if the current macroblock is marked as a Type 3 but none of its 8 neighbouring macroblocks are so marked, i e rationalisation of isolated individual Type 3 markings.

In more detail, the second stage re-marking process 1200 involves: Whilst looping over the non-corner and non-edge macroblocks 10, if 5 or more neighbouring MBs are Type 2 and the current MB is Type 3, remark current MB as type 2 (step 1210).

The process then loops back to the beginning of the raster scan of the macroblocks 910 and if 6 or more neighbouring MBs are Type 2, and the current MB is Type 3, remarks current MB as Type 2 (step 1220).

The process then loops back to the beginning of the raster scan of the macroblocks 910 and if 5 or more neighbouring MBs are Type 3 and the current MB is Type 2, remarks current MB as Type 3 (step 1230).

The process then loops back to the beginning of the raster scan of the macroblocks 910 and if 6 or more neighbouring MBs are Type 3, and the current MB is Type 2, remark current MB as Type 3 (step 1240).

The process then loops back to the beginning of the raster scan of the macroblocks 910 and if current MB is marked as Type 3 and none of 8 neighbouring MBs are marked as Type 3, do the following, otherwise do nothing: Whether 4 or more neighbouring MBs are marked as Type 5: if YES, re-mark current MB as Type 5; if NO, re-mark current MB as Type 2 (step 1250).

Third stage marking and grouping: In this stage, those macroblocks that contain pixels having a particular tone of colour related texture, in particular skin tone, are detected and re-marked from their stage 1 and 2 type values. This portion of the overall process is shown in Fig. 13. All the available macroblocks may be re-examined in a macroblock raster scan order 450 (including edge and corner MBs), or in a macroblock raster scan order 460 (excluding edge and corner MBs) instead. After the initial third stage (re-) marking, the neighbouring macroblock based re-marking then occurs in raster scan order over non-corner and non-edge macroblocks only 460. If skin tone has been detected as present and dominant, the current macroblock of interest will be re-marked as Type 4. After the (re-)marking, a specific re-marking method is once again applied to these re-marked macroblocks. This is shown in Figs. 14a and b. Any isolated macroblock will be re-marked according to the marking of the majority of its neighbours.

In more detail, the third stage marking method 1300 involves: Whilst looping over the non-corner and non-edge macroblocks 10, for each macroblock at 4:2:0 video format, obtain 16x16=256 luminance samples, and chrominance: 8x8=64 Cr samples and 8x8=64 Cb samples. For luminance samples, pick representative with row and column at even index locations (e.g. luminance further sampled array as 8x8). Set or reset a counter to zero when processing a macroblock 1310.

Loop over 8x8 array, checking all luminance and two associated chrominance samples 1320: if the following conditions are all true, then the counter will be increased by 1: (1) Luminance representative Y<240 (2) Luminance representative Y>30 (3) Sample Cr>100 (4) Sample Cr<120 (5) Sample Cb>137 (6) Sample Cb<160 If the total counter number is greater than or equal to 32, the current macroblock will be remarked as Type 4 (step 1330). Once all macroblocks are tested, the process ends at step 1390.

In more detail, the third stage re-marking method 1400 involves: Whilst looping over the non-corner and non-edge macroblocks 10, if the macroblock is Type 5 and none of its neighbour is marked Type 5, do the following, otherwise do nothing: If 4 or more neighbours are Type 4, remark the current macroblock as type 4, otherwise Type 2 (step 1410).

The process then loops back to the beginning of the raster scan of the macroblocks 910 and if the macroblock is Type 1 and none of its neighbours are marked Type 1, do the following, otherwise do nothing: If 4 or more neighbours are Type 4, remark the current MB as Type 4, otherwise Type 2 (step 1420) The process then loops back to the beginning of the raster scan of the macroblocks 910 and if 5 or more neighbouring macroblocks are Type 4 and the current macroblock is Type 2 or Type 3, remark current macroblock as Type 4 (step 1430), else if 5 or more neighbouring macroblocks are Type 1, 2, or 3 and the current macroblock is Type 4, remark current macroblock as Type 2 (step 1440).

The process then loops back to the beginning of the raster scan of the macroblocks 910 and if 5 or more neighbouring macroblocks are Type 4 and the current macroblock is Type 2 or Type 3, remark current macroblock as Type 4 (step 1450), else if 5 or more neighbouring macroblocks are Type 1, 2, or 3 and the current macroblock is Type 4, remark current macroblock as Type 2 (step 1460) The process then loops back to the beginning of the raster scan of the macroblocks 910 and if a current macroblock is marked as Type 4 and none of 8 neighbouring macroblocks are marked as Type 4, do the following, otherwise do nothing: If 4 or more neighbouring macroblocks are marked as Type 5 re-mark the current macroblock as Type 5, otherwise re-mark the current macroblock as Type 2 (step 1470). Then, if a macroblock is Type 2 and none of its neighbouring macroblocks is Type 2, do the following, otherwise do nothing: if 4 or more neighbours are Type 4, remark as Type 4; else if 4 or more neighbours are Type 5, remark as Type 5; else if 4 or more neighbours are Type 1, remark as Type 1; else if more or equal neighbours are Type 4 than Type 3, remark as Type 4; else if more or equal neighbours are Type 1 than Type 3, remark as Type 1; else remark as Type 3 (step 1480). The process then ends at step 1490 once all macroblocks are dealt with.

Fourth stage grouping method: In this final stage, macroblocks grouped at the four picture corners are examined first (top-left, top-right, bottom-left and bottom-right) -see Fig 15. Then macroblocks grouped on the four picture boundaries (top horizontal boundary, bottom horizontal boundary, left vertical boundary and right vertical boundary) are dealt with -see Fig. 16.

In more detail, Fig. 15 shows the first part of the fourth stage re-marking method 1500, which involves: If all 3 neighbouring macroblocks are marked as a Type that is a different Type to the current corner macroblock, do the following, otherwise do nothing: Check whether there are 2 or more of the neighbouring MBs that are marked as Type 5: if YES, re-mark current corner macroblock as Type 5; if NO, re-mark current corner macroblock as Type 2 Meanwhile, Fig. 16 shows the second part of the fourth stage re-marking method 1600, which involves: If all 5 neighbouring macroblocks are marked as a Type that is different to the Type of the current boundary macroblock, do the following, otherwise do nothing: Check whether there are 3 or more of the neighbouring macroblocks that are marked as Type 5: if YES, re-mark current boundary macroblock as Type 5; if NO, re-mark current boundary macroblock as Type 2.

The above described method maybe carried out by any suitable hardware, and as such, the method may be embodied in a set of instructions, stored on a computer readable medium, which when loaded into a computer, or similar, causes the computer to carry out the hereinbefore described method. Equally, the method may be embodied as a specially programmed, or hardware designed, integrated circuit which operates to carry out the method on image data loaded into the said integrated circuit. The integrated circuit may be formed as part of a general purpose computing device, such as a PC, and the like, or it may be formed as part of a more specialised device, such as a games console, mobile phone, portable computer device or hardware video encoder.

It will be apparent to the skilled person that the exact order and content of the steps carried out in the method described herein may be altered according to the requirements of a particular set of execution parameters, such as speed of execution, type of information being detected, accuracy of detection, and the like. Accordingly, the claim numbering is not to be construed as a strict limitation on the ability to move steps between claims, and as such portions of dependent claims maybe utilised freely.

Claims

Claims: 1. A method of pre-processing video image data, comprising: analysing the image data at macroblock level to determine an activity level for each macroblock; marking each macroblock with a pie-determined type according to the determined activity level of each macroblock; re-marking the type of certain ones of the macroblocks dependent upon the marked activity level of neighbouring macroblocks.
2. The method of claim 1, further comprising: testing each macroblock for horizontal or vertical edge dominance and where dominance is found in a particular macroblock, re-marking the type of the macroblock of interest to a type indicative of a macroblock having a dominant vertical or horizontal edge
3. The method of claim 2, wherein a test for horizontal or vertical dominance includes: determining a total vertical edge energy value for each macroblock, determining a total horizontal edge energy value for each macroblock; calculating a dominance from said total vertical edge energy value and said total horizontal edge energy value.
4. The method of claim 3, wherein the step of determining a total vertical edge energy value of a macroblock of interest comprises dividing said macroblock into sub-blocks; forward DCT transforming each sub-block of the macroblock; squaring all values in a first row of samples derived from the forward DCT transform of each sub-block, except a sample at a (00) location; adding all squared row sample values together to provide a sub-block total vertical edge energy value (sVE) for the sub-block; and adding all sub-block total vertical edge energy values (sVEs) for all sub-blocks of the macroblock together to provide a macrob lock total vertical edge energy value (yE).
5. The method of claim 4, wherein the step of determining a total horizontal edge energy comprises: squaring all values in a first column of samples derived from the forward DCI transform of each sub-block, except a sample at a (0,0) location; adding all squared column sample values together to provide the sub-block total horizontal edge energy value (sHE) for sub-block; and adding all sub-block total horizontal edge energy values (sHEs) for all sub-blocks of macroblock together to provide the macroblock total horizontal edge energy value (HE).
6. The method of claim 5, wherein the step of calculating a dominance uses the equation: D=(abs(HEVE)*1 00)/(HE+VE), where if D>90 and HE>VE, horizontal edges are dominant, and If D>90 and VE>HE, vertical edges are dominant, otherwise there is no clear edge dominance.
7. The method of any preceding claim, further comprising: testing each macroblock for the presence of a predetermined amount of a particular tone of colour; re-marking the type of each macroblock of interest to a type indicative of a macroblock having said particular tone of colour, where the predetermined amount of said particular tone of colour is found.
8. The method of claim 4, wherein said particular tone of colour is a skin tone, indicative of the macroblock containing portions of a human character in the image data.
9. The method of claims 7 or 8, wherein the test for a predetermined amount of a particular skin tone comprises: sampling each macroblock in 4:2:0 colour space video format to obtain a luminance sample, and two associated chrominance samples: a Cr sample and a Cb sample; picking rows and columns at even index locations in the luminance sam pie; resetting a counter to zero; looping over the picked luminance sample and the two associated chrominance samples to check for all of the following to be true: (7) Luminance presentative V < 240; (8) Luminance presentative Y> 30; (9) Sample Cr> 100; (10) Sample Cr < 120; (11) Sample Cb> 137; (12) Sample Cb < 160; wherein if (1) to (6) are found to be true, the counter in increased by 1; re-marking the macroblock of interest to a type indicative of a macrobiock having said particular tone of colour if the counter number is greater than or equal to 32.
10. The method of any preceding claim, wherein the step of analysing the image data at macroblock level to determine an activity level for each macroblock comprises: determining a mean luminance value for the macroblock of interest; calculating an absolute difference between a luminance value for each pixel of the macroblock of interest and the mean luminance value of the macroblock of interest; summing all calculated absolute differences and dividing the sum of absolute differences by the total number of pixels in the macroblock of interest.
11. The method of any preceding claim, wherein the step of marking each macroblock with a pre-determined type according to the determined activity level of each block, comprises: marking as a first type if the determined activity level is less than 2; marking as a second type if the determined activity level is between 2 and 13; marking as a fifth type if the determined activity level is greater than 13.
12. The method of any of claims 2 to 11, wherein the type indicative of a macroblock having a dominant vertical or horizontal edge is a third type.
13. The methods of any of claims 7 to 12, wherein a type indicative of a macroblock having a particular tone of colour is a fourth type.
14. The method of claim 11, wherein the video data comprises a stream of pictures, wherein each picture in said stream of pictures comprises a plurality of pixels, and wherein the plurality of pixels are arranged into macroblocks, and wherein the step of re-marking certain ones of the macroblocks further comprises: a first number of iterations looping over all macroblocks in the picture of interest, excluding macroblocks at an edge or corner of the picture, for re-marking macroblocks with the second and fifth types of activity level, comprising: an iteration for re-marking any macroblock of interest not already of the fifth type to become the fifth type if at least five of the neighbouring macroblocks are of the fifth type, and re-marking any macroblock of interest of a fifth type to become the second type if less than three of the neighbouring macroblocks are of the fifth type; an iteration for re-marking any macroblock of interest not already of the fifth type to become the fifth type if at least six of the neighbouring macroblocks are of the fifth type, and re-marking any macroblock of interest of a fifth type to become the second type if less than two of the neighbouring macroblocks are of the fifth type; an iteration for re-marking any macroblock of interest not already of the fifth type to become the fifth type if at least seven of the neighbouring macroblocks are of the fifth type, and re-marking any macroblock of interest of a fifth type to become the second type if less than one of the neighbouring macroblocks are of the fifth type.
15. The method of any of claims 11 to 14, wherein the step of re-marking certain ones of the macroblocks further comprises: a second number of iterations looping over all macroblocks in the picture of interest, excluding macroblocks at an edge or corner of the picture, for dealing with re-marking of macroblocks with the first and second types of activity level, comprising: an iteration for re-marking any macroblock of interest that is already of the first type to become the second type if at least five of the neighbouring macroblocks are not of the first type, and re-marking any macroblock of interest of the second type to become the first type if less than three of the neighbouring macroblocks are not of the first type; an iteration for re-marking any macroblock of interest that is already of the first type to become the second type if at least six of the neighbouring macroblocks are not of the first type, and re-marking any macroblock of interest of the second type to become the first type if less than two of the neighbouring macroblocks are not of the first type; an iteration for re-marking any macroblock of interest that is already of the first type to become the second type if at least seven of the neighbouring macroblocks are not of the first type, and re-marking any macroblock of interest of the second type to become the first type if less than one of the neighbouring macrob locks are not of the first type.
16. The method of claim 14 or 15, wherein the number of iterations is three.
17. The method of any of claims 11 to 16, wherein the step of re-marking certain ones of the macroblocks further comprises: a third number of iterations looping over all macroblocks in the picture of interest, excluding macroblocks at an edge or corner of the picture, for dealing with re-marking of macroblocks with the second, third and fifth types of activity level, comprising: an iteration for re-marking any macroblock of interest that is already of type three to become type two if at least five of the neighbouring macroblocks are of the type two; an iteration for re-marking any macroblock of interest that is already of type three to become type two if at least six of the neighbouring macroblocks are of type two; an iteration for re-marking any macroblock of interest that is already of type two to become the type three if at least five of the neighbouring macroblocks are of type three; an iteration for re-marking any macroblock of interest that is already of type two to become the third type if at least six of the neighbouring macroblocks are of type three; an iteration, where if the macroblock of interest is already marked as type three and none of the neighbouring macroblocks are marked type three then: if at least four neighbouring macroblocks are already marked as type five, then remark the macroblock of interest to become type five, otherwise mark as second type; otherwise do nothing.
18. The method of any of claims 11 to 17, wherein the step of re-marking certain ones of the macroblocks further comprises.

a third number of iterations looping over all macroblocks in the picture of interest, excluding macroblocks at an edge or corner of the picture, for dealing with re-marking of macroblocks with any of activity level types first to fifth, comprising: an iteration where, if the macroblock of interest is already marked type five and none of the neighbouring macroblocks are already marked type five, then: if at least 4 of the neighbouring macroblocks are marked type four, re-mark the macroblock of interest to become type four, otherwise remark to type two; otherwise do nothing; an iteration where, if the macroblock of interest is already marked type one and none of the neighbouring macroblocks are already marked type one, then: if at least 4 of the neighbouring macroblocks are marked type four, re-mark the macroblock of interest to become type four, otherwise remark to type two; otherwise do nothing; an iteration, where if at least five of the neighbouring macroblocks are type four and the macroblock if interest is type two or three, re-mark the macroblock of interest to become type four, else if five or more of the neighbouring macroblocks are type one, two or three, and the macroblock of interest is type four, then remark macroblock of interest to become type two; a further iteration, where if at least five of the neighbouring macroblocks are type four and the macroblock if interest is type two or three, re-mark the macroblock of interest to become type four, else if five or more of the neighbouring macroblocks are type one, two or three, and the macroblock of interest is type four, then remark macroblock of interest to become type two; an iteration where, if the macroblock of interest is marked type four and none of the neighbouring macroblocks are marked type four, then.

if at least 4 of the neighbouring macroblocks are marked type five, re-mark the macroblock of interest to become type five otherwise remark to type two; otherwise do nothing; an iteration where if the macroblock of interest is type two and none of the neighbouring macroblocks is type two, then: if at least four of the neighbouring macroblocks are type four, re-mark macroblock of interest to type four, else if at least four of the neighbouring macroblocks are type five, remark the macroblock of interest to type five, else if at least four of the neighbouring macroblocks are type one, re-mark macroblock of interest to type one, else if more or equal neighbouring macroblocks are type four than type three, remark the macroblock of interest to type four, else if more or equal neighbouring macroblocks are type one than type three, re-marl the macroblock of interest to type one, otherwise re-mark the macroblock of interest to become type three; otherwise do nothing.
19. The method of any of claims 11 to 18, wherein the step of re-marking certain ones of the macroblocks further comprises: an iteration looping over all corner macroblocks in the picture only, comprising if all three neighbouring macroblocks are marked a type that is different to the current type of the corner macroblock of interest, then: if at least two neighbouring macroblocks are marked type five, re-mark the corner macroblock of interest to become type five, otherwise re-mark the corner macroblock of interest to become type two; otherwise do nothing; an iteration looping over all edge macroblocks in the picture only, comprising if all five neighbouring macroblocks are marked a type that is different to the current type of the edge macroblock of interest, then: if at least three neighbouring macroblocks are marked type five, re-mark the edge macroblock of interest to become type five, otherwise re-mark the edge macroblock of interest to become type two: otherwise do nothing.
20. The method of any preceding claim, wherein the video data comprises a stream of pictures, wherein each picture in said stream of pictures comprises a plurality of pixels, and wherein the plurality of pixels are arranged into macrob locks.
21. The method of claim 20, wherein the picture is a frame or field.
22. The method of any preceding claim, further comprising the step of outputting the marked macroblocks for use in further processing of the video data using the marks.
23. The method of any preceding claim, wherein each macroblock is a 16 by 16 pixel macroblock.
24. The method of any of claims 4 to 22, wherein a sub-block is a 8 by 8 pixel sub-block.
25. A method as claimed in any of claims 1 to 24, wherein the video picture comprises a frame.
26. A method as claimed in any of claims 1 to 24, wherein the video picture

comprises a field.
27. A method as claimed in claim 25 or 26, wherein the method is carried out once for each frame and once for each field in a stream of fields and frames.
28. Apparatus adapted to carry out the method of any of claims 1 to 27.
29. A computer-readable medium, carrying instructions, which, when executed, causes computer logic to carry out any of method claims 1 to 27.
30. A method as substantially hereinbefore described with reference to, and as illustrated in the accompanying drawings.
31. An apparatus adapted to carry out the method as substantially hereinbefore described with reference to, and as illustrated in the accompanying drawings