NO20170267A1 - Macro block level adaptive frame / field encoding at the macro block level for digital video content - Google Patents

Description

The present invention relates to coding and decoding of digital video content. More specifically, the present invention relates to frame mode coding and field mode coding of digital video content at the macroblock level as used in the MPEG-4 part 10 AVC/H.264 standard as a video coding standard.

Video compression is used in many current and upcoming products. It is an essential part of digital television set-top boxes (STBs), digital satellite systems (DSS), high-definition television (HDTV) decoders, digital video disc (DVD) players, video conferencing, Internet video and multimedia content, and other digital video applications. Without video compression, the digital video content can be extremely large, making it difficult or even impossible for the digital video content to be efficiently stored, transmitted or viewed.

The digital video content includes a stream of images that can be displayed as an image on a television receiver, computer screen or other electronic device capable of displaying the digital video content. An image presented in time before a particular image is in a "reverse directional relationship" to that particular image. Likewise, an image displayed after a particular image is in the "forward direction" relationship to that particular image.

Video compression is achieved in an encoding process in which each frame is encoded as either one frame or as two fields. Each frame includes a number of lines of spatial information. For example, a typical frame contains 480 horizontal lines. Each field contains half the number of lines in the frame. As an example, if the frame includes 480 horizontal lines, each field includes 240 horizontal lines. In a typical configuration, one of the fields comprises the odd numbered lines in the frame and the other field comprises the even numbered lines in the frame. The field comprising the odd number lines will be referred to as the "top" field hereinafter and in the appended claims, unless otherwise specifically stated. Likewise, the even-numbered lines are referred to as the "bottom field" in what follows here and in the accompanying claims, unless otherwise stated. The two fields can be inset together to form an inset frame.

The general idea behind video encoding is to remove data from the digital video content that is considered "non-essential". The reduced amount of data then requires less bandwidth for broadcasting or transmission. After the compressed video data has been transmitted, it must be decoded, or decompressed. In this process, the transmitted video data is processed to produce approximation data that is substituted into the video data to replace the "non-essential" data that is removed in the encoding process.

Video encoding transforms the digital video content into a compressed form that can be stored using less space and transmitted using less bandwidth than uncompressed digital video content. This is achieved by taking advantage of the temporal and spatial redundancies in the images of the video content. The digital video content may be stored on a storage medium such as a hard disk, DVD or other non-volatile storage medium.

There are a number of video coding methods that compress the digital video content. Accordingly, the video coding standards have been developed to standardize the various video coding methods so that the compressed digital video content is rendered in formats that a majority of video encoders and decoders can recognize. For example, the Motion Picture Experts Group (MPEG) and the International Telecommunication Union (ITU-T) have developed video coding standards that are widely used. Examples of these standards include MPEG-1, MPEG-2, MPEG-4, ITU-T H261 and ITU-T H263 standards.

Most modern video coding standards, such as those developed by MPEG ITU-T, are based in part on a temporal prediction algorithm with motion compensation (MC). Temporal prediction with motion compensation is used to remove temporal redundancy between successive frames in a digital video broadcast.

The temporal prediction algorithm with motion compensation usually uses one or two reference images to encode a particular image. A reference image is an image that has already been encoded. By comparing the particular image to be encoded with one of the reference images, the temporal prediction algorithm with motion compensation can take advantage of the temporal redundancy that exists between the reference image and the particular image to be encoded and then encode the image with a greater degree of compression than if the image were coded without using the temporal prediction algorithm with motion compensation. One of the reference images can be in the reverse direction in relation to the particular image to be coded. The second reference image is in the forward direction relative to the particular image to be coded.

However, as demands for higher resolution, more complex graphic content and faster transfer times increase, so does the need for better methods of video compression. To this end, a new video coding standard is currently being developed jointly by ISO and ITU-T. The new video coding standard is called MPEG-4 Advanced Video Coding (AVQ/H264 standard.

According to aspects of the invention, a method for encoding, a method for decoding, an apparatus for encoding, and an apparatus for decoding are provided as stated in the claims. In one of many possible embodiments, the present invention provides a method for encoding, decoding and bitstream generation of digital video content. The digital video content includes a stream of images which may be intra images, predicted images or bi-predicted images. Each of the images includes macro blocks which can further be divided into smaller blocks. The method involves encoding and decoding each of the macroblocks of each image in the stream of images in either frame mode or field mode.

The accompanying drawings illustrate various realizations of the present invention, and thus form part of the description. Along with the following description, the drawings demonstrate and explain the principles of the present invention. The illustrated embodiments are examples of the present invention and do not limit the scope of the invention. Figure 1 illustrates a sequence example for three types of images that can be used to implement the present invention, as defined by a video coding standard example such as the MPEG-4 part 10 AVC/H.264 standard. Figure 2 shows that each image is preferably divided into slices containing macroblocks in accordance with an embodiment of the present invention. Figure 3a shows that a macroblock can further be divided into a block size that is 16 by 8 pixels in accordance with an embodiment of the present invention. Figure 3b shows that a macroblock can further be divided into a block size that is 8 by 16 pixels in accordance with an embodiment of the present invention. Figure 3c shows that a macroblock can further be divided into a block size that is 8 by 8 pixels in accordance with an embodiment of the present invention. Figure 3d shows that a macroblock can be further divided into a block size that is 8 times

4 pixels in accordance with an embodiment of the present invention.

Figure 3e shows that a macroblock can be further divided into a block size that is 4 times

8 pixels in accordance with an embodiment of the present invention.

Figure 3f shows that a macroblock can be further divided into a block size that is 4 times

4 pixels in accordance with an embodiment of the present invention.

Figure 4 shows an image construction example using temporal prediction with motion compensation that illustrates embodiments of the present invention. Figure 5 shows that a macroblock is divided into a top field and a bottom field if it is to be coded in field mode. Figure 6a shows that a macroblock encoded in field mode can be divided into a block with a size of 16 by 8 pixels in accordance with an embodiment of the present invention. Figure 6b shows that a macroblock encoded in field mode can be divided into a block with a size of 8 by 8 pixels in accordance with an embodiment of the present invention. Figure 6c shows that a macroblock encoded in field mode can be divided into a block with a size of 4 by 8 pixels in accordance with an embodiment of the present invention. Figure 6d shows that a macroblock encoded in field mode can be divided into a block with a size of 4 by 4 pixels in accordance with an embodiment of the present invention. Figure 7 illustrates an example of a macroblock pair that can be used in AFF coding on a pair of macroblocks in accordance with an embodiment of the present invention. Figure 8 shows that a macroblock pair to be coded in field mode is first split into a 16 by 16 pixel top field block and a 16 by 16 pixel bottom field block.

Figure 9 shows two possible scanning patterns in AFF coding of macroblock pairs.

Figure 10 shows another embodiment of the present invention which extends the concept of AFF coding on a pair of macroblocks to AFF coding on a group of four or more adjacent macroblocks. Figure 11 shows some of the information included in the bitstream that holds information relating to each macroblock in a stream. Figure 12 shows a block to be encoded and its neighboring blocks and will be used to explain various preferred methods for calculating the PMV of a block in a macroblock. Figure 13 shows an alternative definition of neighboring blocks if the scanning path is a vertical scanning path. Figure 14 shows that each pixel value is predicted from the neighboring blocks' pixel values in accordance with an embodiment of the present invention.

Figure 15 shows different prediction directions for intra_4x4 coding.

Figure 16a-b illustrates that the selected intra-prediction mode (intra_pred_mode) of a 4 by 4 pixel block is highly correlated with adjacent blocks' prediction modes. Figure 17a-d shows neighboring block definitions in relation to a current macroblock pair to be coded.

For the indicated drawings, identical reference numbers indicate corresponding, but not necessarily identical, elements.

The present invention provides a method for adaptive frame/field (AFF) encoding of digital video content comprising a stream of images or slices of an image at the macroblock level. The present invention extends the concept of image-level AFF to macroblocks. In frame-level AFF encoding, each frame is encoded in a stream of frames to be encoded in either frame mode or field mode, regardless of the frame encoding mode or field encoding mode of other frames to be encoded. If the image is encoded in frame mode, the two fields that form the inset frame are co-encoded. Conversely, if the image is encoded in field mode, the two fields forming an inset frame are encoded separately. The encoder determines the type of encoding, frame mode encoding or field mode encoding, that is most beneficial for each image and selects that encoding type for the image. The exact procedure for choosing between frame mode and field mode is not critical to the present invention and will not be detailed here.

As stated above, the MPEG-4 Part 10 AVC/H.264 standard is a new standard for encoding and compressing digital video content. The documents establishing the MPEG-4 Part 10 AVC/H.264 standard are hereby incorporated by reference, which include the "Joint Final Committee Draft (JFCD) and Joint Video Specification" issued by the Joint Video Team (JVT) on August 10, 2002 (See also ITU-T recommendation H.264 and ISO/IEC 14496-10 AVC). The aforementioned JVT consists of experts from ISO MPEG and ITU-T. Due to the public nature of the MPEG-4 Part 10 AVC/H.264 standard, the present invention will not attempt to document all the existing aspects of video coding in accordance with MPEG-4 Part 10 AVC/H.264, but will instead support itself on the standard's incorporated specifications.

Although this method of AFF coding is compatible with and will be explained using the standard guidelines of MPEG-4 Part 10 AVC/H.264, it can be modified and applied to suit a particular standard or application.

Using the drawings, preferred embodiments of the present invention will now be explained.

Figure 1 illustrates a sequence example for three image types that can be used to implement the present invention, as defined by an example of a video coding standard such as, for example, the standard MPEG-4 part 10 AVC/H.264. As previously mentioned, the encoder encodes the images and the decoder decodes the images. The encoder or decoder may be a processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a coder/decoder (CODEC), a digital signal processor (DSP), or another electronic device capable of encoding the current of pictures. However, as used hereinafter and in the appended claims, the term "encoder" will be used to refer broadly to any electronic device that encodes digital video content comprising a stream of images, unless otherwise specifically stated. The term "decoder" will be used to refer broadly to any electronic device that decodes digital video content that includes a stream of images.

As shown in Figure 1, there are preferably three image types that can be used in the video coding method. Three image types are defined to provide support for random access to stored digital video content while investigating the maximum redundancy reduction using temporal prediction with motion compensation. The three image types are intra (I) images (100), predicted (P) images (102a, b) and bi-predicted (B) images (101a-d). An I-picture (100) provides an access point for random access to stored digital video content and can be encoded with only mild compression. Intra images (100) are coded without reference to reference images.

A predicted image (102a,b) is encoded using an I-image, a P-image or a B-image that has already been encoded as a reference image. The reference picture can be either in the forward or reverse temporal direction in relation to the P-picture being coded. The predicted images (102a, b) can be coded by more compression than the intraimages (100).

A bi-predicted picture (101a-d) is encoded using two time reference pictures, a forward reference picture and a reverse reference picture. The forward reference image is sometimes called a past reference image and the reverse reference image is sometimes called a future reference image. An embodiment of the present invention is one where the forward reference picture and the reverse reference picture can be in the same temporal direction in relation to the B picture to be coded. Bi-predicted images (101a-d) can be encoded with the most compression of the three image types mentioned.

The reference relationships (103) between the three image types are illustrated in figure 1. For example, the P image (102a) can be coded using the coded I image (100) as its reference image. The B-pictures (101a-d) may be encoded using the encoded I-picture (100) or the encoded P-picture (102a) as their reference pictures, as shown in Figure 1. Under the principles of an embodiment of the present invention coded B-pictures (101a-d) can also be used as reference pictures for other B-pictures to be coded. For example, the B-image (101c) in figure 1 is shown with two other B-images (101b and 101d) as its reference images.

The number and specific order of the I-images (100), B-images (101a-d) and P-images (102a, b) shown in Figure 1 are provided as an example of a configuration of images, but are not necessary to implement the present invention. Any number of I-images, B-images and P-images can be used in any order to best serve a particular application. The standard MPEG-4 part 10 AVC/H.264 does not impose any limitation on the number of B frames that must be between two reference frames, nor does it limit the number of frames between two I frames. Figure 2 shows that each image (200) is preferably divided into slices (202). A disc (202) includes a macroblock group (201). A macroblock (201) is a rectangular group of pixels. As shown in Figure 2, a preferred size for a macroblock (201) is 16 by 16 pixels. Figures 3a-f show that a macroblock can further be divided into blocks of smaller size. For example, as shown in Figures 3a-f, a macroblock can be further divided into block sizes of 16 by 8 pixels (Figure 3a, 300), 8 by 16 pixels (Figure 3b, 301), 8 by 8 pixels (Figure 3c, 302), 8 by 4 pixels (Figure 3d, 303), 4 by 8 pixels (Figure 3e, 304) or 4 by 4 pixels (Figure 3f, 305). These smaller sized blocks are preferred in some applications that make use of the motion compensated temporal prediction algorithm. Figure 4 shows an image construction example using temporal prediction with motion compensation illustrating an embodiment of the present invention. Temporal prediction with motion compensation assumes that a current image, image N (400), can be modulated locally as a translation of another image, image N-1 (401). The image N-1 (401) is the reference image for the coding of image N (400) and can be temporally forward or reverse in relation to image N (400).

As shown in Figure 4, each image is preferably divided into two slices comprising macroblocks (201a, b). Image N-l (401) contains an image (403) to be displayed in image N (400). The image (403) will be in a different temporal position in the image N (402) than is the case for the image N-1 (401) as shown in Figure 4. The image content in each macroblock (201b) for the image N (400) is predicted from the image content in each corresponding macroblock (201a) of the image N-l (401) by estimating the necessary amount of temporal movement in the image content in each macroblock (201a) of the image N-l (401) for the image (403) to move to its new temporal position (402) in image N (400). Instead of the original image (402) being encoded, it is the difference (404) between the image (402) and its prediction (403) that is actually encoded and transmitted.

For each image (402) in the image N (400), the temporal prediction can often be described using a motion vector that represents the amount of temporal movement necessary for the image (403) to move to a new temporal position in the image N (402). The motion vectors (406) used for temporal prediction with motion compensation must be coded and transmitted.

Figure 4 shows that the image (402) in the image N (400) can be represented using the difference

(404) between the image and its prediction and the associated motion vectors (406). The exact coding procedure using the motion vectors can be varied as best serves the particular application and can be easily implemented by one skilled in the art.

To understand AFF coding at the macroblock level, a brief introduction to AFF coding at the image level of a stream of images will now be given. A frame in an interleaved sequence contains two fields, namely the top field and the bottom field, which are interleaved and separated in time by a field period. The field period has a duration that corresponds to half of a frame period. In frame-level AFF coding, the two fields in an inset frame can be coded together or coded separately. If they are coded together, frame mode coding is used. In contrast, if two fields are coded separately, field mode coding is used.

On the other hand, fixed frame/field encoding encodes all the images in a stream of images in only one mode. This mode can be frame mode or it can be field mode. Frame-level AFF is preferred for fixed frame/field encoding in many applications because it allows the encoder to select the mode, frame mode or field mode, to encode each frame in the stream of frames based on the content of the digital video material. AFF coding results in better compression than is achieved with fixed frame/field coding in many applications.

One embodiment of the present invention is one where AFF coding can be performed on smaller portions of an image. This smaller part can be a macroblock, a pair of macroblocks, or a group of macroblocks. Each macroblock, macroblock pair, or group of macroblocks or slice is encoded in frame mode or field mode, regardless of how the other macroblocks in the image are encoded. AFF coding will in each of the three cases be described in detail below.

In the first case, AFF encoding is performed on a single macroblock. If the macroblock is to be coded in frame mode, the two fields in the macroblock are coded together. Once encoded as a frame, the macroblock can be further divided into smaller blocks as shown in Figures 3a-f for use in the temporal prediction algorithm with motion compensation.

However, if the macroblock is to be coded in field mode, the macroblock (500) is split into a top field (501) and a bottom field (502), as shown in Figure 5. The two fields are then coded separately. In Figure 5, the macroblock has M rows of pixels and N columns of pixels. A preferred value for N and M is 16, which makes the macroblock (500) a 16 by 16 pixel macroblock. As shown in Figure 5, every other row of pixels is shaded. The shaded areas represent rows of pixels in the top field of the macroblock (500) and the unshaded areas represent rows of pixels in the bottom field of the macroblock (500).

As shown in Figures 6a-d, a macroblock encoded in field mode can be divided into four additional blocks. It is necessary for a block to have single parity. The single parity requirement means that a block cannot include both a top field and a bottom field. Rather, it must accommodate a single parity of fields. Thus, as shown in Figures 6a-d, a field mode macroblock can be divided into blocks that are 16 by 8 pixels (Figure 6a, 600), 8 by 8 pixels (Figure 6b, 601), 4 by 8 pixels (Figure 6c, 602 ) and 4 by 4 pixels (Figure 6d, 603). Figures 6a-d show that each block contains fields with single parity.

AFF encoding on macroblock pairs will now be explained. AFF coding on macroblock pairs will sometimes be referred to as pair-based AFF coding. A comparison of the block sizes in Figures 6a-d with those in Figures 3a-f shows that a macroblock coded in field mode can be divided into fewer block patterns than is the case for a macroblock coded in frame mode. The block sizes of 16 by 16 pixels, 8 by 16 pixels, and 8 by 4 pixels are not available for a macroblock encoded in field mode due to the single parity requirement. This implies that the performance of a single block-based AFF may not be good for some sequences or applications that strongly prefer field mode coding. To guarantee field mode macroblock coding performance, in some applications it is preferred that macroblocks encoded in field mode have the same block sizes as macroblocks encoded in frame mode. This can be achieved by performing AFF encoding on pairs of macroblocks rather than on macroblocks individually.

Figure 7 illustrates an example of a pair of macroblocks (700) that can be used in AFF coding on a pair of macroblocks in accordance with an embodiment of the present invention. If the pair of macroblocks (700) is to be encoded in frame mode, the pair is encoded as two frame-based macroblocks. In each macroblock, the two fields in each of the macroblocks are coded together. Once encoded as frames, the macroblocks can be further divided into smaller blocks as shown in Figures 3a-f for use in the temporal prediction algorithm with motion compensation.

However, if the pair of macroblocks (700) is to be coded in field mode, it is first split into a top field 16 by 16 pixel block (800) and a bottom field 16 by 16 pixel block

(800), as shown in figure 8. The two fields are then coded separately. In Figure 8, each macroblock in the macroblock pair (700) has N=16 columns of pixels and M=16 rows of pixels. Thus, the dimensions of the pair of macroblocks (700) are 16 by 32 pixels. As shown in Figure 8, every other row of pixels is shaded. The shaded areas represent rows of pixels in the top field of the macroblocks and the unshaded areas represent the rows of pixels in the bottom field of the macroblocks. The top field block (800) and the bottom field block (801) can now be divided into one of the possible block sizes shown in figures 3a-f.

In accordance with one embodiment of the present invention, there are two possible scan paths in the AFF encoding of the pair of macroblocks (700). A scan path determines the order in which the macroblock pairs in an image are encoded. Figure 9 shows two possible scanning paths in AFF coding of the pairs of macroblocks (700). One of the scan paths is a horizontal scan path (900). In the horizontal scan path

(900), the macroblock pair (700) is encoded in an image (200) from left to right and from top to bottom, as shown in Figure 9. The second scan path is a vertical scan path (901). In the vertical scan path (901), the macroblock pairs (700) in an image (200) are encoded from top to bottom and from left to right, as shown in Figure 9. For frame mode coding, the topmost macroblock in a macroblock pair (700) is first encoded followed by of the bottom macro block. For field mode encoding, the top field macroblock in a macroblock pair is encoded first, followed by the bottom field macroblock.

Another embodiment of the present invention extends the AFF encoding concept on a pair of macroblocks to AFF encoding on a group of four or more adjacent macroblocks (902), as shown in Figure 10. AFF encoding on a group of macroblocks will occasionally be referred to as group-based AFF coding. The same scanning paths, horizontally (900) and vertically (901), are used in the scanning of macroblock pairs that are used in the scanning of groups of neighboring macroblocks (902). Although the example shown in Figure 10 shows a group of four macroblocks, the group may comprise more than four macroblocks.

If the group of macroblocks (902) is to be encoded in frame mode, the group is encoded as four frame-based macroblocks. In each macroblock, the two fields in each of the macroblocks are jointly coded. Once coded as frames, the macroblocks can be further divided into smaller blocks as shown in Figures 3a-f for use in the temporal prediction algorithm with motion compensation.

However, for example, if a group of four macroblocks (902) is to be encoded in field mode, it is first split into a top field block of 32 by 16 pixels and a bottom field block of 32 by 16 pixels. The two fields are then coded separately. The top field block and the bottom field block can now be split into macro blocks. Each macroblock is further divided into one of the possible block sizes shown in figures 3a-f. Because this process corresponds to that shown in Figure 8, a separate figure is not provided to illustrate this embodiment.

In AFF coding at the macroblock level, a frame/field flag bit is preferably included in a picture's bitstream to indicate the mode, frame mode or field mode, used in the coding of each macroblock. The bit stream includes information that applies to each macroblock in a stream, as shown in Figure 11. For example, the bit stream can include an image preamble (110) and information (111) about the macroblock type (113). Preferably, the frame/field flag (112) is included before each macroblock in the bitstream if AFF is performed on each individual macroblock. If AFF is performed on macroblock pairs, the frame/field flag (112) is preferably included before each macroblock pair in the bit stream. And finally, if AFF is performed on a macroblock group, the frame/field flag (112) is preferably included before each macroblock group in the bit stream. In one embodiment, the frame/field flag bit (112) is a 0 if frame mode is to be used and a 1 if field encoding is to be used. In another embodiment, the frame/field flag bit (112) is a 1 if frame mode is to be used and a 0 if field coding is to be used.

A further embodiment of the present invention involves a method for determining the block sizes into which the encoder divides a macroblock for AFF at the macroblock level. A preferred but not exclusive method for determining the ideal block size is the sum absolute difference (SAD) with or without bias or rate distortion bias (RD). For example, SAD checks the performance of the possible block sizes and selects the ideal block size based on its result. The exact procedure for using SAD with or bias or RD-bias can easily be carried out by an expert in the field.

According to an embodiment of the present invention, each frame/field-based macroblock in the macroblock-level AFF can be intracoded or intercoded. In intracoding, the macroblock is coded without temporal reference to other macroblocks. On the other hand, intercoding uses temporal prediction with motion compensation to encode the macroblocks.

If intercoding is used, a block with a size equal to 16 by 16 pixels, 16 by 8 pixels, 8 by 16 pixels or 8 by 8 pixels can have its own reference images. The block can be either a frame-based or field-based macro block. The MPEG-4 Part 10 AVC/H.264 standard allows for multiple reference frames instead of just two reference frames. The use of multiple reference images improves the performance of the motion-compensated temporal prediction algorithm by allowing the encoder to find a block in the reference image that best matches the block to be encoded. By using the block in the reference picture in the coding process which most coincides with the block to be coded, the greatest possible compression can be achieved in the coding of the picture. The reference images are stored in frame and field buffers and assigned reference frame numbers and reference field numbers based on their temporal distance from the current image being encoded. The closer the reference image is to the current image being stored, the more likely it is that the reference image will be selected. For field mode coding, the reference frames for a block can be any top field or bottom field in any of the reference frames in the frame buffers or field buffers.

Each block in a frame-based or field-based macroblock can have its own motion vectors. The motion vectors are spatially, predictively encoded. In accordance with an embodiment of the present invention, prediction motion vectors (PMV) for each block are also calculated in intercoding. The algebraic difference between a block's PMVs and its associated motion vectors is then calculated and encoded. This produces the compressed bits for motion vectors.

Reference is then made to figure 12 which is used to explain different preferred methods for calculating the PMV for a block in a macro block. A current block E, shown in Figure 12, is to be intercoded as well as its neighboring blocks A, B, C and D. I will hereafter be referred to as a current block and A, B, C and D will hereafter be referred to as neighboring blocks to E, with unless otherwise stated. The PMV of the block E is derived from its neighboring vectors' motion vectors. These neighboring blocks in the example in figure 12 are A, B, C and D. A preferred method for calculating the PMV for block E is to calculate either the meridian of the motion vectors of blocks A, B, C and D, the average of these motion vectors or the weight of these motion vectors average. Each of the blocks A through E can be in either frame mode or field mode.

Another preferred method of calculating PMV for block E is to use a yes/no method. Under the principles of the yes/no procedure, a block must be in the same frame encoding mode or field encoding mode as block E to have its motion vector included in the calculation of PMV for E. For example, if block E in Figure 12 is in frame mode, block A must also be in frame mode to have its motion vector included in the calculation of the PMV for the block E. If one of the neighboring blocks of E does not have the same encoding mode as the mode of the block E, its motion vector is not used in the calculation of the sought PMV for the block E.

The "always method" can also be used to calculate PMV for block E. In the "always method", blocks A, B, C and D are always used with the calculation of PMV for block E, regardless of whether they are according to frame coding mode or field coding mode. If E is in frame mode and the neighboring block is in field mode, the neighboring block's vertical component is multiplied by 2 before it is included in the PMV calculation for block E. If E is in field mode and a neighboring block is in frame mode, the neighboring block's vertical component is divided by 2 for it to be included in The PMV calculation for block E.

The "selective method" can also be used to calculate the PMV for block E if the macroblock has been encoded using pair-based AFF encoding or group-based AFF encoding. In the "selective method", a frame-based block has a frame-based motion vector that points to a reference frame. The block is also assigned a field-based motion vector that points to a reference field. The field-based motion vector is the frame-based motion vector of the block that has

the vertical motion vector component divided by two. The reference field number is the reference frame number multiplied by two. A field-based block has a field-based motion vector that points to a reference field. The block is also assigned a frame-based motion vector that points to a reference frame. The frame-based motion vector is the field-based motion vector of the block with the vertical motion vector component multiplied by two. The reference frame number is the reference field number divided by two.

The derivation of a block's PMV using the "selective method" will now be explained using Figure 12 as a reference. In macroblock pair-based AFF, each block in a macroblock is associated with a companion block that resides in the same geometric location within the macroblock pair's other macroblock. In Figure 12, each of the neighboring blocks (A, B, C and D) of E may or may not be in the same frame or field coding mode as block E. Therefore, the following rule applies: If E is in frame mode and a neighboring block is in frame mode , the neighboring block's true frame-based motion vector is used for E's PMV.

If E is in frame mode and a neighboring block is in field mode, the following rules apply for calculating E's PMV. If the neighboring block (for example, block A) and its companion field-based block have the same reference field, the average of the two blocks' assigned frame-based motion vectors is used for the calculation of E's PMV. The reference frame number used for the PMV calculation is the reference field number of the neighboring block divided by two. However, if the neighboring block and its companion field block have different reference fields, the neighboring block cannot be used in the calculation of PMV for E.

If E is in field mode and a neighboring block is in frame mode, the following rules apply for calculating PMV for E. If the neighboring block (for example, block A) and the accompanying frame-based block have the same reference frame, the average of the two blocks' assigned field-based motion vectors is used for the calculation of E's PMV. The reference field number used for the PMV calculation is the neighboring block's reference number multiplied by two. However, if the neighboring block and its companion field block have different reference frames, then the neighboring block cannot be used in the calculation of PMV for E.

If E is in field mode and a neighboring block is in field mode, the neighboring block's true field-based motion vector is used in the calculation of E's PMV.

An alternative preferred option can be used in the "selective method" to calculate a block's PMV. In Figure 12, each of block E's neighboring blocks (A, B, C, and D) may or may not be in the same frame or field coding mode as block E. Therefore, the following rules apply to this alternative, preferred option for the selective method: If E is in frame mode and a neighboring block is in frame mode, the neighboring block's true frame-based motion vector is used for E's PMV.

If E is in frame mode and a neighboring block is in field mode, the weighted average of the neighboring block's assigned frame-based motion vectors and its companion field-based block is used to calculate the PMV for E. The weighting factors are based on the neighboring block's reference field numbers and its companion block.

If E is in field mode and a neighboring block is in frame mode, the weighted average of the neighboring block's assigned field-based motion vectors and its companion frame-based block is used to calculate E's PMV. The weighting factors are based on the neighboring block's reference frame numbers and the companion block.

If E is in field mode and a neighboring block is in field mode, the neighboring block's true field-based motion vector is used to calculate the PMV for E.

Another preferred method for calculating a block's PMV is the "all-selective method". This method can be used in single-block AFF coding, pair-based macroblock AFF coding or group-based AFF coding. In this method, each block is assigned a horizontal and a vertical index number that represents the block's horizontal and vertical coordinates. Each block is also assigned a horizontal and vertical field coordinator. A block's horizontal field coordinate is the same as the block's horizontal coordinate. For a block in a top field macroblock, the vertical field coordinate is half the block's vertical coordinate and is assigned top field polarity. For a block in the bottom field macro block, the block's vertical field coordinate is obtained by subtracting 4 from the block's vertical coordinate and dividing the result by 2. The block is also assigned bottom field polarity. The result of assigning different field polarities to two blocks is that there are now two blocks with the same horizontal and vertical field coordinates, but with different field polarities. Thus, given the coordinates of a block, the field coordinates and its field polarity can be calculated, and vice versa.

The "alt-selective method" will now be explained in more detail using Figure 12 as a reference. The PMV for block E must be calculated. Let bx represent the horizontal size of the block E divided by 4, which is the size of a block in this example. The PMVs for E are obtained as follows, depending on whether E is in frame mode or field mode.

Let block E be in frame mode, and let (x, y) represent the horizontal coordinates

respectively the vertical coordinates for E. The neighboring blocks of E are defined as follows. A is the block whose coordinates are (x-l,y). B is the block whose coordinates are (x,y-l). D is the block whose coordinates are ((x-l,y-l). C is the block whose coordinates are (x+bx+1, y-1). If one of A, B, C, or D is in field mode, then its vertical motion vector is multiplied by 2 before it is used for prediction and its reference frame number calculated by dividing its reference field by 2.

Then let the block E be in top or bottom field mode and let (xf, yf) represent the horizontal field coordinates and the vertical field coordinates respectively for E. In this case the neighbors of E are defined as follows. A is the block whose field coordinates are (xf-1, yf) and has the same polarity as E. B is the block whose field coordinates are (xf, yf-1) and has the same polarity as E. D is the block whose field coordinates are (xf -1, yf-1) and has the same polarity as E. C is the block whose field coordinates are (xf+bx+1, yf) and has the same polarity as E. If A, B, C or D is in frame mode, then split its vertical motion vector by 2 before it is used for prediction and its reference field is calculated by multiplying its reference frame by 2.

In all of the methods outlined above for determining a block's PMV, a horizontal scan path was assumed. However, the scanning path can also be a vertical scanning path. In this case, the neighboring blocks of the relevant block E are defined as shown in Figure 13. A vertical scan path is preferred in some applications because the information in all neighboring blocks is available for the calculation of the PMV for the relevant block E.

Another embodiment of the present invention is

directional segmentation prediction. In directional segmentation prediction, 16 by 8 pixel blocks and 8 by 16 pixel blocks have rules that apply only to their PMV calculations. These rules apply in all PMV calculation methods for these block sizes. The rules will now be explained in more detail in connection with Figure 12. In each of these rules, a relevant block E will have its PMV calculation.

First, a 16 by 16 pixel block consists of an upper block and lower block. The upper block contains the upper 8 rows of 16 pixels. The lower block contains the bottom 8 rows of 16 pixels. In the following description, block E is in figure 12

a 16 by 16 pixel block. For the upper block in the 16 by 8 pixel block, block B is used to predict block E's PMV if it has the same reference image as block E. Otherwise, median prediction is used to predict block E's PMV. For the lower block in a 16 by 8 pixel block, block A is used to predict block E's PMV if it has the same reference image as block E. Otherwise, median prediction is used to predict block E's PMV.

An 8 by 16 pixel block is used in a left and right block. Both the left and right blocks are 8 by 16 pixels. In the following description, the blocks A-E in Figure 12 are 8 by 16 pixel blocks. For the left block, block A is used to predict block E's PMV if it has the same reference image as block E. Otherwise, median prediction is used to predict block E's PMV. For the right block, block C is used to predict block E's PMV if it has the reference image as block E. Or median prediction is used to predict block E's PMV.

For 16 by 8 pixel blocks and 8 by 16 blocks, A, B, or C may be in different coding modes (frame or field) than the current block E. The following rules apply to both block sizes. If E is in frame mode and A, B or C is in field mode, the reference frame number of A, B or C is calculated by dividing its reference field by 2. If E is in field mode and A, B or C is in frame mode, the reference field number of A is calculated , B or C by multiplying its frame of reference by 2.

According to another embodiment of the present invention, a macroblock in a P picture may be omitted in AFF encoding. If a macroblock is omitted, its data is not transmitted in the encoding of the image. An omitted macroblock in a P picture is reconstructed by copying the collocated macroblock in the last coded reference picture. The collocated macroblock is defined as having motion compensation using PMV as defined over or without motion vectors. The following rules apply to omitted macroblocks in a P-picture. If AFF coding is performed per macroblock, is an omitted macroblock in frame mode. If AFf encoding is performed on macroblock pairs and if both macroblocks are omitted, then they are in frame mode. However, if only one of the macroblocks in a macroblock pair is omitted, its frame or field encoding mode is the same as that of the non-omitted macroblock in the same macroblock pair. If AFF encoding is performed on a group of macroblocks and if the entire group of macroblocks is omitted, then all macroblocks are in frame mode. If at least one non-skipped macroblock occurs, then the skipped macroblocks are in the same group in the same frame or field encoding mode as the non-skipped macroblock.

An alternative procedure for omitted macroblocks is as follows. If a macroblock pair is omitted, its frame and field encoding mode follows that of its neighboring macroblock pair to the left. If the neighboring macroblock pair on the left is not available, its encoding mode follows its neighboring macroblock pair towards the top. If none of the neighboring macroblock pairs to the left or above are available, the omitted macroblock is set to frame mode.

Another embodiment of the present invention is

direct mode macro block encoding for B frames. In direct mode encoding, a B-picture has two motion vectors, forward and backward motion vectors. Each motion vector points to a reference image. Both the forward movement vector and the backward movement vector can point in the same temporal direction. For direct-mode macroblock coding in B-frames, the forward and

the backward motion vectors of a block from the collocated block in the backward reference image. The collocated block in the backward reference image can be frame-mode coded or field-mode coded. The following rules apply in direct mode macro block encoding for B-frame.

If the collocated block is in frame mode and if the current direct mode macroblock is also in frame mode, the two associated motion vectors of a block in the direct mode macroblock are computed from the collocated block. The forward reference frame is the one used by the collocated block. The reverse reference frame is the same frame where the collocated block resides.

If the collocated block is in frame mode and if the current direct mode block is in field mode, the two associated motion vectors of a block in the direct mode macroblock are calculated from the collocated block's motion vector with the vertical component divided by two. The forward reference field is the same parity field of the reference frame that was used by the collocated block. The reverse reference field is the same parity field of the reverse reference frame where the collocated block resides.

If the collocated block is in field mode and if the current direct mode macroblock is also in field mode, the associated motion vectors of a block in the direct mode macroblock are calculated from the collocated block with the same field parity. The forward reference field is the field used by the collocated block. The reverse reference field is the same field where the collocated block resides.

If the collocated block is in field mode and if the current direct mode macroblock is in frame mode, the two associated motion vectors of the block in the direct mode macroblock are calculated from the collocated block's motion vector with the vertical component multiplied by two. The forward reference frame is the frame in one of whose fields is used by the collocated block. The reverse reference field is the frame in one of whose fields the collocated block resides.

An alternative option is to force the direct mode block to be in the same frame or field encoding mode as the collocated block. In this case, the direct mode block is also in frame mode if the co-located block of a direct mode block is in frame mode. The direct mode block's two frame-based motion vectors are derived from the frame-based forward motion vector of the collocated block. The forward reference frame is used by the collocated block. The reverse reference frame is where the collocated block resides.

However, if the co-located block of a direct mode block is in field mode, the direct mode block is also in field mode. The two field-based motion vectors of the direct mode block are derived from the collocated block's field-based forward motion vector. The forward reference field is used by the collocated block. The reverse reference field is where the collocated block resides.

A macroblock in a B-picture may also be omitted in AFF encoding according to another embodiment of the present invention. An omitted macroblock in a B-picture is reconstructed as a normal direct-mode macroblock without any encoded transform coefficient information. For omitted macroblocks in a B picture, the following rules apply. If AFF coding is performed per macroblock, is an omitted macroblock either in frame mode or in the frame or field encoding mode of the collocated block in its reverse reference image. If AFF encoding is performed on macroblock pairs and if both macroblocks are omitted, then they are in frame mode or in the same frame or field encoding mode as the co-located macroblock pair in its reverse reference image. However, if only one of the macroblocks in a macroblock pair is omitted, its frame or field encoding mode is the same as that of the non-omitted macroblock in the same macroblock pair. If AFF coding is performed on a group of macroblocks and if the entire group of macroblocks is omitted, then all the macroblocks are in frame mode or in the same frame or field coding mode as the collocated group of macroblocks in the reverse reference image. If there is at least one macroblock that is not omitted, then the omitted macroblock is in the same group in the same frame or field encoding mode as the non-omitted macroblock.

As previously mentioned, a block can be intracoded. Intrablocks are spatially predictively coded. There are two possible intra-coding modes for a macroblock in macroblock-level AFF coding. The first is intra_4x4 mode and the second is intra_16xl6 mode. In both, each pixel's value is predicted using the real, constructed pixel values from neighboring blocks. By predicting pixel values, more compression can be achieved. Intra_4 x 4 mode and intra_16 x 16 mode will each be explained in more detail below.

For intra_4 x 4 mode, the predictions of the pixels in a 4 by 4 pixel block, as shown in Figure 14, are derived from its pixels to the left and above. In Figure 14, the 16 pixels in the block of 4 by 4 pixels are labeled A through P. Also, as shown in Figure 4, the neighboring pixels are A through P. The neighboring pixels are shown in capital letters. As shown in Figure 15, there are different prediction directions for intra_4x4 coding. They are vertical (0), horizontal (1) DC prediction (mode 2), diagonal down/left (3), diagonal down/right (4), vertical-left (5), horizontal-down (6), vertical- right (7) and horizontal-up (8). DC prediction averages all the neighboring pixels together to predict a particular pixel value.

For intra_16 x 16 mode, however, there are four different prediction directions. The prediction directions are also referred to as prediction modes. These prediction directions are vertical predictions (0), horizontal predictions (1), DC prediction and plane prediction. In the following, plan prediction will not be explained.

An intrablock and its neighboring blocks can be coded in frame or field mode. Intraprediction is performed on reconstructed blocks. A reconstructed block can be presented in both frame and field modes, regardless of the block's actual frame or field encoding mode. Since only the pixels of the reconstructed blocks are used for intra prediction, the following rules apply.

If a block of 4 by 4 pixels or 16 by 16 pixels is in frame mode, the neighboring pixels used in calculating the pixel value predictions are in the frame structure. If a block of 4 by 4 pixels or 16 by 16 pixels is in field mode, the neighboring pixels used in calculating the block's pixel value prediction are in the field structure with the same field parity.

The selected intra-prediction mode (intra_pred_mode) of a 4 by 4 pixel block is highly correlated with adjacent blocks' prediction modes. This is illustrated in Figure 16a.

Figure 16a shows that A and B are adjacent blocks to C. Block C's prediction mode is to be determined. Figure 16b shows the order of intra-prediction information in the bit stream. When and A and B's prediction modes are known (including the case where A or B are both off-disk) the most likely prediction mode (most likely mode) for C is given. If one of the blocks A or B is "outside" the most probable prediction mode is equal to DC prediction (mode 2). Otherwise, it is equal to the minimum of prediction modes used for blocks A and B. When an adjacent block is coded with 16 by 16 intramodes, the prediction mode is the DC prediction mode. When an adjacent block is coded a non-intra-macroblock, the prediction mode is "mode: DC prediction" in the normal case and "outside" in the limited intra-update case.

To signal a prediction mode number for a 4 by 4 block, the parameter applymost likely mode is first passed. This parameter is represented by a 1-bit code word and can assume the values 0 or 1. If the most likely mode used is equal to 1, the most likely mode is used. Otherwise, an additional parameter remainingmodeselector is sent as a 3-bit codeword, which can assume a value from 0 to 7. The codeword is a binary representation of the value remainingmodeselector. The prediction mode number is calculated as: if (remaining mode select > most likely mode)

intra prediction mode = residual mode selector,

else

intraprediction mode = remaining_mode_selector+l.

The ranking of prediction modes assigned to block C is therefore mature for most likely followed by the remaining modes in ascending order.

An embodiment of the present invention includes the following rules that apply to intra-mode prediction for an intra-prediction mode for a 4 by 4 pixel block or an intra-prediction mode for a 16 by 16 pixel block. Block C and its neighboring blocks A and B can be in frame or field mode. One of the following rules shall apply. Figures 16a-b will be used in the following explanations of the rules.

Rule 1: A or B is used as the neighboring block of C only if A or B is in the same frame/field mode as C. Otherwise, A or B is considered to be outside.

Rule 2: A and B are used as neighboring blocks to C regardless of their frame/field coding mode.

Rule 3: If C is encoded in frame mode and has coordinates (x,y), then A is the block with coordinates (x,y-l) and B is the block with coordinates x-l,y). Otherwise, if C is coded as field and has the field coordinates (xf,yf) then A is the block whose field coordinates are

(xf, yf-1) and has the same field polarity as C and B is the block whose field coordinates are (xf-l,yf) and has the same field polarity as C.

Rule 4: This rule only applies to macroblocks. In the case of decoding the prediction modes of blocks numbered 3, 6, 7, 9, 12, 13, 11, 14 and 15 as shown in Figure 16b, the neighboring blocks above and to the left are in the same macroblock as the current block. However, in the case of decoding the prediction modes of blocks numbered 1, 4 and 5, the top block (block A) is in a different macroblock pair than the current macroblock pair. In the case of decoding the prediction mode of blocks number 2, 8 and 10, the block on the left (block B) is in another macroblock pair. In the case of decoding the prediction mode of the block numbered 0, the blocks that are both left and above are in different macroblock pairs. For a macroblock in field decoding mode, the neighboring blocks of blocks numbered 0, I, 4, 5, 2, 8 and 10 shall be defined as follows: If the macroblock pair (170) above is decoded in field mode, for block numbers 0, 1 and 5 in the top field macroblock (173), the blocks numbered 10, 11, 14 and 15 respectively in the top field macroblock (173) of the macroblock pair (170) above be considered as the neighboring blocks above for the relevant macroblock pair (171) as shown in Figure 17a. For blocks numbered 0, 1, 4 and 5 in the bottom field macroblock (174), blocks numbered 10, II, 14 and 15 respectively in the bottom field MB of the macroblock pair (170) above shall be considered as the neighboring blocks above for the relevant macroblock pair (171), as it is shown in Figure 17a.

However, if the macroblock pair (170) above is decoded in frame mode, then for the blocks numbered 0, 1, 4 and 5 in the top field macroblock (173), the blocks numbered 10, 11, 14 and 15 respectively in the bottom frame macroblock (176) of the macroblock pair

(170) above be considered as the upper neighbor blocks of the relevant macroblock pair (171), as shown in figure 17b. For blocks numbered 0, 1, 4 and 5 in the bottom field macro block

(174), blocks numbered respectively 10, 11, 14 and 5 in the bottom frame macro block shall

(176) of the macroblock pair (170) above be considered as the neighboring blocks of the relevant macroblock pair (171), as shown in figure 17b.

If the macroblock pair (172) on the left is decoded in field mode, then for blocks numbered 0, 2, 8 and 10 in the top field macroblock (173), blocks numbered 5, 7, 13 and 15 respectively in the top field macroblock (173) of the macroblock pair (172) to the left be considered as the neighboring blocks to the left of the relevant macroblock pair (171), as shown in figure 17c. For blocks numbered 0, 2, 8 and 10 in the bottom field macroblock (174), blocks numbered 5, 7, 13 and 15 respectively in the bottom field macroblock (174) of the macroblock pair (172) on the left shall be considered as the neighboring blocks to the left of the relevant macroblock pair ( 171), as shown in figure 17c.

If the macroblock pair (172) on the left is decoded in frame mode, then for blocks numbered 0, 2, 8 and 10 in the top field macroblock (173), the blocks numbered 5, 7, 13 and 15 respectively in the top frame macroblock (175) of the macroblock pair (172) to the left be considered neighboring blocks to the left of the macroblock pair in question

(171), as shown in figure 17d. For blocks numbered 0, 2, 8 and 10 in the bottom field macroblock (174), blocks numbered 5, 7, 13 and 15 respectively in the bottom field macroblock (176) of the macroblock pair (172) on the left shall be considered as the neighboring blocks to the left of the relevant macroblock pair ( 171), as shown in figure 17d.

For macroblock pairs on the upper boundary of a slice, shall

the intramode prediction value used to predict a field macroblock is set to DC prediction if the macroblock pair (172) on the left is in frame coding mode.

The preceding descriptions of intracoding and intramode prediction can be extended to adaptive block transforms.

In another embodiment of the present invention, loop filtering is performed on the reconstructed blocks. A reconstructed block can be represented in either frame/field structure, regardless of the block's frame/field encoding mode. Loop (deblock) filtering is a process of weighted averaging of neighboring blocks' pixels. Figure 12 will be used to explain loop filtering. Assume that E in Figure 12 is a reconstructed block, and that A, B, C, and D are its reconstructed neighboring blocks, as shown in Figure 12, and that they are all represented in frame structure. As A, B, C, D and E can be frame or field coded, the following rules apply: Rule 1: If E is frame coded, loop filtering is performed over the pixels of E and its neighboring blocks A, B, C and D.

Rule 2: If E is field coded, loop filtering is performed over the top field pixels and bottom field pixels of E and its neighboring blocks A, B, C and D separately.

In another embodiment of the present invention, padding is performed on the reconstructed frame by repeating the boundary pixels. As the boundary blocks can be coded in frame or field mode, the following rules apply: Rule 1: The pixels on the left or right vertical line of a boundary block are repeated if necessary.

Rule 2: If a border block is in frame encoding, the pixels on the border block's top or bottom horizontal line are repeated.

Rule 3: If a border block is in field coding, the pixels on the border block's two top or two bottom horizontal lines (two fields) are repeated alternately.

In another embodiment of the present invention, two-dimensional transform coefficients are transformed into one-dimensional arrays of coefficients prior to entropy coding. The scanning path can be either zigzag or non-zigzag. The zigzag scanner is preferably for progressive sequences, but can also be used for interlace sequences with slow movements. The non-zigzag scanners are preferably for nested sequences. For macroblock AFF encoding, the following options can be used: Option 1: The zigzag scan is used for macroblocks in frame mode while the non-zigzag scanners are used for macroblocks in field encoding.

Option 2: The zigzag scan is used for macroblocks in both frame and field modes.

Option 3: Non-zigzag scanning is used for macroblocks in both frame and field modes.

In the description above, the present invention has been explained by means of embodiments to illustrate the invention. However, many modifications and variations are possible on the basis of the indications given above. On the basis of the embodiments and principles of the invention described here, experts in the field are able to utilize the invention in various embodiments and with various modifications as is best suited to the specific intended application.

Also provided is a further first method of encoding or decoding digital video content, which digital video content comprises a stream of images each of which may be intraframes, predicted frames or bi-predicted frames, each of the frames comprising macroblocks, the method comprising the steps of: selecting to encode or decoding a group of neighboring blocks in a frame mode or a field mode, arranging the groups of neighboring blocks for processing into frame macroblocks or field macroblocks in accordance with the selected of the frame mode or field mode, and encoding or decoding the frame macroblocks or field macroblocks, wherein the step of arranging the group of neighboring macroblocks for processing into frame macroblocks or field macroblocks includes the steps: - in the frame mode, to jointly process two fields in the group of neighboring macroblocks, which group of neighboring macroblocks is divided into frame macroblocks, where each of the frame macroblocks includes both top field pixels and bottom field tpixels, and - in the field mode, to separately process two fields in the group of neighboring macroblocks, which group of neighboring macroblocks is divided into field macroblocks, where each of the field macroblocks includes top field pixels or bottom field pixels, and that the frame macroblocks and field macroblocks are of the same size.

Furthermore, a further second method according to the further first method, where the frame macroblocks or field macroblocks can be divided into blocks, and each of the blocks includes 16 by 16 pixels, 16 by 8 pixels, 8 by 16 pixels, 8 by 8 pixels, 8 by 4 pixels, 4 by 8 pixels or 4 by 4 pixels.

Furthermore, a further third method according to the further second method, further including calculating prediction motion vectors for each of the blocks.

Furthermore, a further fourth method according to the further first method, further including omitting the frame macro block or the field macro block in a predicted image and not transferring data for the frame block or the field macro block that has been omitted.

Furthermore, a further fifth method according to the further first method, further comprising omitting the frame macroblock or the field macroblock in a bi-predicted image and not transferring data from the frame macroblock or the field macro block that has been omitted.

Furthermore, a further sixth method according to the further first method, further including encoding the frame macroblock or the field macroblock in direct mode in the bi-predicted images.

Furthermore, a further seventh method according to the further first method, further including predicting spatial pixel values for the frame macroblock or the field macroblock in the intra-images, the predicted images or the bi-predicted images.

Furthermore, a further eighth method according to the further first method, further comprising temporally predicting pixel values for the frame macroblock or the field macroblock in the predicted images or the bi-predicted images.

Claims (16)

1. Method for coding an image in an image sequence, characterized by: dividing said image into a number of smaller parts, in which each of said plurality of smaller portions is a pair of vertically adjacent macroblocks; and selectively encode at least one of said plurality of smaller parts in frame coding mode, and at least one of said number of smaller portions in field coding mode, in which both fields of each of said macroblocks are coded jointly if said frame coding mode is selected. 2. Method according to claim 1, further comprising: formation of said pair of macroblocks from a top field block and a bottom field block, when said pair of macroblocks is to be coded in said field coding mode. 3. Method according to claim 2, wherein said forming step comprises: taking a top part of said top field block to form a top field of a first macroblock of said pair of macroblocks; taking a bottom portion of said topfield block to form a topfield of a second macroblock of said pair of macroblocks; taking a top portion of said bottom field block to form a bottom field of a first macroblock of said pair of macroblocks; and taking a bottom portion of said bottom field block to form a bottom field of a second macroblock of said pair of macroblocks. 4. Method for decoding a coded image with a number of smaller portions from a bit stream, characterized by: decoding at least one of said number of smaller portions in frame coding mode, and at least one of said number of smaller portions in field coding mode, wherein each of said plurality of smaller portions are a pair of vertically adjacent macroblocks; and using said plurality of decoded smaller portions to construct a decoded image, in which both fields of each of said macroblocks are decoded jointly if said frame coding mode is selected. 5. Method according to claim 4, further comprising: formation of said pair of macroblocks from a top field block and a bottom field block, when said pair of macroblocks is coded in said field coding mode. 6. Method according to claim 5, wherein said forming step comprises: taking a top part of said top field block to form a top field of a first macroblock of said pair of macroblocks; taking a bottom portion of said top field block to form a top field of a second macroblock of said pair of macroblocks; taking a top portion of said bottom field block to form a bottom field of a first macroblock of said pair of macroblocks; and taking a bottom portion of the bottom field block to form a bottom field of a second macroblock of said pair of macroblocks. 7. Apparatus for encoding an image in an image sequence, characterized by: means for dividing said image into a number of smaller portions, wherein each of said plurality of smaller portions is a pair of vertically adjacent macroblocks; and means for selectively encoding at least one of said plurality of smaller portions in frame encoding mode, and at least one of said plurality of smaller portions in field encoding mode, wherein the encoding is applied to a pair of macroblocks, and wherein both fields of each of said macroblocks are coded jointly if said frame coding mode is selected. 8. Apparatus according to claim 7, further comprising: means for forming said pair of macroblocks from a top field block and a bottom field block, when the pair of macroblocks is coded in said field coding mode. 9. Apparatus according to claim 8, wherein the means for forming takes a top part of said top field block to form a top field of a first macroblock of said pair of macroblocks, and takes a bottom part of said top field block to form a top field of a second macroblock of said pair of macroblocks, and taking a top part of said bottom field block to form a bottom field of a first macroblock of said pair of macroblocks, and taking a bottom field of said bottom field block to form a bottom field of a second macroblock of said pair of macroblocks. 10. Apparatus for decoding an encoded image with a number of smaller portions from a bit stream, comprising: means for decoding at least one of said number of smaller portions in frame coding mode, and at least one of said number of smaller portions in field coding mode, wherein each of said plurality of smaller portions are a pair of vertically adjacent macroblocks; and means for using said plurality of decoded smaller portions to construct a decoded image, wherein both fields of each of said macroblocks are jointly decoded if said frame encoding mode is selected. 11. Apparatus according to claim 10, further comprising: means for forming said pair of macroblocks from a top field block and a bottom field block, when said pair of macroblocks is coded in said field coding mode. 12. Apparatus according to claim 11, wherein said means for forming takes a top part of said top field block to form a top field of a first macroblock of said pair of macroblocks, and takes a bottom part of said top field block to form a top field of a second macroblock of said pair of macroblocks, and takes a top part of said bottom field block to form a bottom field of a first macroblock of said pair of macroblocks, and takes a bottom part of said bottom field block to form a bottom field of a second macroblock of said pair of macroblocks. 13. Apparatus according to any one of claims 10 to 12, wherein at least one of said plurality of pairs of macroblocks comprises blocks including the same possible block sizes when in frame coding mode or field coding mode, and the blocks are for use temporal prediction with motion compensation. 14. Apparatus according to any one of claims 10 to 13, wherein at least one of said plurality of pairs of macroblocks comprises blocks having a block size including 16x16 pixels, 16x8 pixels, 8x16 pixels, 8x8 pixels 8x4 pixels, 4x8 pixels, or 4x4 pixels when are in field encoding mode, or when they are in frame encoding mode. 15. Apparatus according to any one of claims 10 to 14, wherein at least one block of at least one of said plurality of pairs of macro blocks is for use in temporal prediction with motion compensation having a block size including 16x16 pixels, 16x8 pixels, 8x16 pixels, 8x8 pixels, 8x4 pixels, 4x8 pixels, or 4 x 4 pixels. 16. Apparatus according to any one of claims 10 to 15, wherein at least one of said plurality of pairs of macroblocks is associated with a frame/field flag, wherein the frame/field flag indicates frame coding mode or field coding mode.