JP2001094941A - Compression image processor and compression image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a compression image processor and compression image processing method, that can correctly process the arrangement of bit stream images of every possible different sequences and minimize the number of frame memories used. SOLUTION: This compression image processor is provided with three frame memories. The 1st or 2nd frame memory stores I pictures and P pictures, the 3rd frame memory stores B pictures, and the B pictures are outputted with the highest priority. When the 3rd frame memory stores no B pictures, the oldest I or P pictures which are not yet outputted that are stored in the 1st or 2nd frame memory are outputted.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention arranges compressed data of I picture, P picture and B picture so that pictures referred to when decoding forward, backward and bidirectional predictive macroblocks temporally precede. The present invention relates to an apparatus and a method for decoding image data of each picture from a bit stream and outputting the pictures in display order.

[0002] The present invention is particularly useful when memory is an expensive resource or when image reproduction is suddenly started.

[0003]

2. Description of the Related Art An image compression standard, MPEG (Movin)
g Picture Expert Group 1 and MPEG2 are composed of three types of pictures. Each picture type serves a different purpose and is encoded with a different compression ratio.

An I-picture (intra-coded picture) is
It is encoded without referring to another picture. That is, the coding is performed only by the coding based on the information closed in one screen without using the motion compensation prediction. Therefore, while having the lowest compression ratio, it has the best image quality as compared with images of other picture types.

[0005] P picture (predictive-coded picture)
Uses one-way motion compensation prediction from a past I picture or P picture.

Also, a B picture (bi-directionally pict
ure) uses bidirectional prediction using past and future pictures. Since the B picture is coded with the least information, the picture quality is the worst.

An image sequence is usually composed of an I picture, P
Pictures and B pictures are mixed. The compression ratio and the arrangement of the I picture, P picture and B picture depend on the content of the image as well as the required data compression amount.

Each picture stored in the encoded image data file is arranged in a bit stream array.
This array is an array in which the image decoder receives and processes the encoded image data.

If the picture sequence consists only of I and P pictures, the decoder processes each picture and transmits it as a reconstructed picture to the output device in a first-in first-out format.

In some cases, the bit stream arrangement is different from the display arrangement. This is when a B picture exists. Because B pictures require information from past and future comparison pictures, the future pictures to be compared must first arrive at the decoder before the B picture can be restored.

However, since the B picture is displayed before the future picture to be compared, the picture data received by the picture decoder must be rearranged into the correct display arrangement. The future picture to be compared is output to the output device only after all consecutive B pictures have been decoded.

Next, a typical conversion from a bit stream array to a display array will be described.

The following shows the arrival order of pictures.

Bit stream picture order: I1P2B
3B4P5B6B7P8B9 ... Display picture order: I1B3B4P
2B6B7P5B9... The picture display array is obtained from the temporary comparison field in the picture layer.

[0015] 10-bit (1024) integers are sequentially assigned to pictures so as to form a display array. The tentative comparison is reset to 0 on the first picture in the picture group.

In this way, an array that must be displayed is defined. It is more useful that the image be transmitted in an array that the decoder needs to decode, rather than in a display array.

There are several ways to generate a display array independent of a tentative comparison of pictures.

A first method, as shown in US Pat. No. 5,621,464, is a method of rearranging pictures without using a temporary comparison field.

FIG. 1 shows a case where the bit stream arrangement of the image sequence is I1P2B3B4P5B6B7P8. Using four frame memories, each frame memory is a memory block that can store fully decoded pictures. The first two memories store either I or P pictures, while the remaining two memories store only B pictures. However, in this case, a delay of three picture display periods occurs before the first picture is sent to the output device.

The second method is that shown in US Pat. No. 5,398,072. However, it only states that only three frame memories are used for picture reconstruction and rearrangement.

In the third method, as a method of rearranging another picture, there is a method of shortening the delay time until the first picture is sent to the display device by using only three frame memories. This method is described in U.S. Pat.
No. 464.

Specifically, different algorithms are used to determine the frame memory that is selected to store incoming pictures and the frame memory that is selected to output rearranged pictures. However, in this method, unlike the above-described method, there is no restriction on the picture type held in each frame memory.

When the encoded image sequence is composed of only I pictures or I pictures and P pictures, the bit stream sequence is the same as the display arrangement, so that there is no need for rearrangement.

FIG. 2 shows a case of an image sequence of I1P2P3P4I5P6P7P8. The three frame memories function collectively as a first-in first-out buffer. The received first picture is written to the frame memory 1 and at the same time, empty data is read from the frame memory 2 by the display device. I
Pictures and P pictures are treated equally. In the next picture period, the second input picture is stored in the frame memory 2
And empty data is read from the frame memory 3.

Then, such a cyclic operation is continued in the entire image sequence. There is a two picture period delay before the first picture is sent to the output device.

That is, each picture stays in each frame memory for two picture periods until it is read.

The criterion for determining which of the three frame memories receives the next incoming picture data is whether or not each frame memory no longer holds valid data from which a picture has been read once. based on. The frame memory not held is a candidate frame memory for storing the next incoming picture. On the other hand, the selection of the frame memory by which the output device should acquire the next picture is based on first-in first-out. Therefore, the oldest picture in the frame memory that has not been read is displayed next.

The method can also process image sequences containing B pictures.

Since B pictures derive information from past and future comparison pictures, the reordering scheme used will be different. That approach is in contrast to an image sequence that contains only I-pictures and P-pictures (P-pictures that only compare to past pictures).

Before the B picture is reconstructed, its past and future comparison pictures must already be in another frame memory. Therefore, the past and future comparison pictures must precede the bitstream picture arrangement. However, B pictures are those that are sent to the display device before the comparative future picture.

FIG. 3 shows a mechanism for rearranging a typical image sequence in which I pictures, P pictures and B pictures are mixed. The bit stream picture arrangement is I1P2B3B4P5B6B7P8.

The first two, I-pictures and P-pictures, are handled as before. Next, the frame memory to which the B picture is to be allocated is tagged. This tag is used to indicate that the B picture in this special frame memory will be read during the next picture period. For example, if the next incoming picture is also of the B picture type, the frame memory that will hold the picture is also tagged. Then, after the first B picture is read, it is sent to the output device. FIG. 4 shows an example of picture rearrangement using this method. The image sequence used is I1P2B3P4B5P6B7P8.

The selection of a frame memory for storing incoming pictures is the same as that used for the image sequence if there are no B pictures. The B picture is always stored in the frame memory and is read out after one picture period.

Therefore, the B picture is held in the frame memory for one picture period. If the incoming picture is of type I or P, the oldest unread I or P picture in the frame memory after the B picture has been read will be selected for display.

However, this method does not disclose a case where a B-picture exceeds two consecutive B-pictures in a bit stream. Therefore, this rearrangement method is a logic used assuming a maximum of two consecutive B pictures, and it is expected that this will not work if the B picture exceeds two consecutive.

[0036]

In an encoded image bitstream containing only I or P pictures, the order of the pictures in the bitstream is the same as the order in which the pictures are displayed. No reordering is required.

However, if there is a B picture, the display arrangement is different from the bit stream picture arrangement. this is,
This is because both the past and future comparison pictures are required before the B picture is reconstructed.

Therefore, even when a B picture first appears during image reproduction, a future comparison picture of the B picture is required prior to the B picture in the encoded bit stream.

As a current solution for picture rearrangement, 4
One memory block is used. Each block is 1
One decoded picture can be sufficiently stored. The amount of memory allocated per block is determined by the size of the picture to be decoded.
If the memory is a valuable resource, it is necessary to optimize the distribution and use of the memory.

The above-described method currently used to reorder pictures relies on 2-4 times from the start of decoding until the first reconstructed picture is sent to the output device.
This results in a long delay in the picture period. The manner of rearrangement thus directly affects the delay.

Also, some of these rearrangement methods cannot correctly handle bit stream picture arrangements of any different order, and are not sufficient to handle different types of bit stream sequences. In particular, when two or more consecutive B pictures are present in the bit stream, there are some that do not work.

Accordingly, an object of the present invention is to provide a compressed image processing apparatus and a compressed image processing method capable of correctly handling bit stream picture arrangements in all different orders and minimizing the number of frame memories used. And

[0043]

In order to solve the above-mentioned problems, a compressed image processing apparatus according to the present invention converts each compressed data of an I picture, a P picture and a B picture into a predicted macroblock of each of forward, backward and both directions. An apparatus for decoding image data of each picture from a bit stream arranged such that a picture referred to at the time of decoding precedes in time and outputting the decoded picture data in a display order, comprising first and second pictures for storing an I picture or a P picture. It has a frame memory and a third frame memory for storing B pictures. Then, I pictures or P pictures are alternately stored in either the first frame memory or the second frame memory, and the oldest stored and unoutputted I picture or P picture is stored in the first or second frame memory. Output from frame memory.

The B picture stored in the third frame memory is output in preference to the I or P picture stored in the first or second frame memory.

Only when there is no unoutput B picture in the third frame memory, the oldest stored and unoutput I picture or P picture is output from the first or second frame memory.

By the above means, it is possible to suppress the delay of one picture period from the start of decoding until a completely decoded picture becomes available on the output device.

[0047]

DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below.

The present invention is a universal picture rearrangement algorithm that can handle all kinds of picture types: I picture, P picture and B picture.

Then, when the first picture period ends,
The first picture is immediately available and requires only three frame memories.

The three frame memories are Frame0, Frame
1 and Frame2. I picture only or I picture
For an image sequence consisting of only pictures and P pictures, only Frame0 and Frame1 are used. Frame2
Are reserved only for B pictures.

The following is an example of two image sequences.

Bit stream picture sequence 1: I1 I2 I3 I4 I5 Display picture sequence 1: I1 I2 I3 I4 I5 Bit stream picture sequence 2: I1 P2 P3 P4 P5 Display picture sequence 2: I1 P2 P3 P4 P5 The first picture I1 of bit stream sequence 1 is F
The second picture is stored in Frame1, while the second picture is stored in Frame1. At this time, the picture I1 of Frame0 is the oldest unsent picture, and is therefore delivered to the output device. Since Frame 0 includes the data completed at that point,
The third picture I3 is sent to Frame0.

The next plurality of I-pictures are Frame
0 and Frame1 are stored alternately. The rearranged picture data is stored in one frame memory, and at the same time, the picture data in the other frame memory is read.
The algorithm treats I and P pictures equally.

Therefore, the way of handling the picture of the image sequence 2 is the same as that of the image sequence 1. Then, the rearrangement of the pictures is not actually performed, and the display arrangement is the same as the bit stream arrangement.

Frame 2 exists only for storing B pictures. Used by image sequences containing B pictures. And the algorithm is how many consecutive B
Even a picture image sequence is handled correctly. The following shows three image sequences.

Bitstream picture sequence 3: I0P1B2B3P4B5B6 Display picture sequence 3: I0B2B3P1B5B6P4 Bitstream picture sequence 4: I0B1B2P3B4B5P6 Display picture sequence 4: B1B2I0B4B5P3B5 and I3P1B5B3P6 The method of allocating pictures is the same as described above. The reconstructed B picture is always stored in Frame2. This is because these B pictures obtain information from the past and future comparison pictures of Frame0 and Frame1.

After rearrangement, the B pictures are immediately sent to the output device within the same picture period. This allows the next B picture to occupy Frame2 in the subsequent picture period. Last B of consecutive B pictures
After the pictures have been read, the oldest unread I or P pictures of Frame0 or Frame1 are sent to the output device.

That is, the first picture can be output after the first picture period. FIG. 5 is a block diagram showing a method of this picture rearrangement.

This method is simple and effective in correctly rearranging every picture.

(Embodiment) Next, a specific embodiment of the present invention will be described with reference to FIGS.

FIG. 5 is a block diagram showing a mechanism of picture rearrangement, FIG. 6 is a diagram for explaining details of frame selection, and FIG. 7 is a flowchart showing details of rearrangement.

At the end of each picture period, the frame memory 11 is selected to hold the next picture to be rearranged. Another frame memory 11 is also chosen to release the picture data to the output device.

This frame memory selection mechanism 13
Is driven by the incoming picture type. Any combination of I picture, P picture, and B picture can be handled.

Hereinafter, an example in which the present invention is applied to an image sequence including only an I picture and a P picture will be described.

When receiving the first incoming I picture, the IP input selector 26 is connected to the point of frame 0 (20). This indicates that frame 0 (20) is a memory frame for storing I pictures. On the other hand, the IPB output selector 27 is connected to the point of frame 1 (21). Thus, the data in frame 1 (21) is sent to the output device.

As the picture periods change, the incoming picture type is examined. If the next picture is an I picture or a P picture, the position of the IP input selector 26 is connected to the point of frame 1 (21).

Thus, the second picture is frame 1
It is carried to (21). At the same time, the IPB output selector 27 is switched to frame 0 (20). In this way, the first I picture is sent to the output device. The time required for the first I picture to reach the display device from the start of decoding is only one picture period.

The logic for selecting the I / P frame input / output in frames 0 and 1 is represented by waveforms 24 and 25 and an inverter 23.

Waveform 24 indicates the frame number in which the incoming I or P picture is to be stored, and waveform 25
Indicates a frame number for sending picture data. The frames selected for input and output are always contradictory in an image sequence consisting of only I and P pictures. This logic is represented by the inverter 23. Therefore, the IP input selector 26 sets the frame 0 (2
0) and frame 1 (21),
The output selector 27 outputs a frame 1 (21) and a frame 0 (2
0). The rearrangement mechanism is as shown in Tables 1 and 2.

[0070]

[Table 1]

[0071]

[Table 2]

Subsequently, an I picture, a P picture, and a B picture
An example of application to an image sequence in which pictures are mixed will be described. When a B picture exists, frame 2
(22) is prepared to store rearranged B pictures. Unlike I and P pictures, B pictures have no input selector. All B pictures are automatically switched to frame 2 (22). Frame 0 and Frame 1 contain the past and future comparison pictures required for the generation of B pictures. At the end of the picture period, the IPB output selector is switched to frame 2 (22), and frame 2 (22) is selected for output.

For example, when using the image sequence I0P1B2B3P4, the first I0 picture encountered is stored in frame 0 (20), as indicated by the IP input selector 26. Frame 1 (2
1) is selected by the IPB output selector 27. Following P
When one picture is accepted, the input selector 26 and the output selector 27 of the frame memory are connected to the inverter logic 2.
Replaced by 3. This places the P1 picture in frame 1 (21) and sends out the first I0 picture in frame 0 (20).

The next B2 picture invalidates the IP input selector 26 and sets the IPB output selector 27 to frame 2 (22).
Switch to the position. B2 pictures are automatically stored in frame 2 (22) and sent out in the same picture period.

When the next B3 picture is accepted,
Frame 2 (22) is used again to store and send out B3 pictures.

The next P4 picture restarts the IP input selector 26. The IP input selector 26 is a frame memory for holding data sent prior to the arrival of a series of B pictures, and is connected to a frame memory having used data. Therefore, while the P1 picture in frame 1 (21) is being sent to the output device, frame 0 (2
0) will receive a P4 picture. The IPB output selector 27 always leads to the opposite frame memory with respect to the IP input selector 26.

The above description describes a rearrangement process for converting a bitstream image sequence from I0P1B2B3P4 to a display array I0B2B3P1B5.

Tables 3 to 5 show a rearrangement mechanism in another bit stream sequence including a B picture.

[0079]

[Table 3]

[0080]

[Table 4]

[0081]

[Table 5]

The rearrangement process can also be represented using a flowchart as shown in FIG.

Variable X is used to indicate a frame number. 0 to indicate frame 0 and frame 1 respectively
And the value of 1. The variable X is initialized to 0 when the picture rearrangement process starts (40) (4
1). At the beginning of each picture, a check is made to see if it is a B picture to confirm the type of each incoming picture (42).

When the picture is of type I or type P, X is used to determine a frame memory for storing the picture (43). X is also inverted to include the frame number for picture output 45 (44). The output device will obtain the picture from the frame number specified by the inverted X (45). That is,
If X is 0, it is inverted to 1.

The flow on the left side of the flowchart is executed for a B picture. B pictures are always stored in frame 2 (46) and delivered to the output device in the same picture period (47).

The rearrangement algorithm is performed by repeatedly checking the next picture type in the image sequence. The process ends (49) when the end of the bitstream comes (48).

[0087]

The present invention improves upon the process of rearranging the bitstream sequence pictures in display order, requiring only the memory required to store three pictures, and storing and outputting the same. Is controlled by a certain rule. The memory allocated for each picture depends on the size of the picture to be decoded. If the picture sequence is only an I picture and a P picture, only two frame memories instead of three are sufficient.

This efficient use of memory allows the design and production of hardware-based image decoders that use smaller memories. If the logic and memory for decoding are located on the same wafer, smaller chips can be realized.

These effects also reduce the power consumption of the image decoder. It brings the greatest benefit to portable image decoders. As a result, production costs are reduced because less wafers and other materials are used.

Regardless of the order of incoming pictures,
The delay from the start of decoding to the arrival of the first picture at the display device is one picture period, and this short delay provides a responsive operation of the image decoder.

The logic required for realizing the picture rearrangement is simple and effective.

[Brief description of the drawings]

FIG. 1 shows a conventional I1P2B using four frame memories.
Explanatory diagram showing a rearrangement method that handles a picture sequence of 3B4P5B6B7P8

FIG. 2 shows a conventional I3P2P using three frame memories.
Explanatory diagram showing a rearrangement method for handling 3P4I5P6P7P8 picture sequences

FIG. 3 shows a conventional I1P2B using three frame memories.
Explanatory diagram showing a rearrangement method that handles a picture sequence of 3B4P5B6B7P8

FIG. 4 shows I1P2B using three conventional frame memories.
Explanatory diagram showing a rearrangement method for handling 3P4B5P6B7P8 picture sequences

FIG. 5 is a block diagram showing a mechanism of picture rearrangement.

FIG. 6 is a diagram for explaining details of frame selection;

FIG. 7 is a flowchart showing details of rearrangement.

[Explanation of symbols]

 13 Frame memory selection mechanism (control means) 20 Frame 0 (first frame) 21 Frame 1 (second frame) 22 Frame 2 (third frame)

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5C053 FA27 GB08 GB38 KA04 KA08 5C059 KK08 MA00 MA05 PP05 PP06 PP07 UA05 UA35

Claims (10)

[Claims] 1. Each bit stream in which compressed data of I picture, P picture and B picture are arranged such that pictures referred to when decoding respective forward / backward / bidirectional prediction macroblocks are temporally preceding. A first and second frame memories for storing an I picture or a P picture, and a third frame memory for storing a B picture. . 2. The compressed image processing apparatus according to claim 1, further comprising a bit stream syntax decoder for identifying a picture type. 3. The compressed image processing apparatus according to claim 1, further comprising control means for alternately storing an I picture or a P picture in one of the first frame memory and the second frame memory. 4. A compressed image processing apparatus according to claim 3, further comprising control means for outputting the oldest stored and unoutputted I picture or P picture from said first or second frame memory. 5. A control means for outputting a B picture stored in the third frame memory in preference to an I or P picture stored in the first or second frame memory. The compressed image processing device according to claim 1. 6. A control for outputting the oldest stored and unoutputted I picture or P picture from the first or second frame memory only when there is no unoutputted B picture in the third frame memory. Claim 3 having means
A compressed image processing apparatus according to any one of the preceding claims. 7. A first method for storing an I picture or a P picture.
And a third frame memory for storing a B picture.
A bit stream in which compressed data of I picture, P picture, and B picture are arranged so that pictures referred to when decoding respective predicted macroblocks in forward, backward, and both directions temporally precede by using the frame memory of And a method of decoding image data of each picture from the first frame memory and outputting the decoded picture data in the order of display, wherein an I picture or a P picture is alternately stored in one of the first frame memory and the second frame memory. . 8. The compressed image processing method according to claim 7, wherein the oldest stored and unoutputted I picture or P picture is output from the first or second frame memory. 9. The system according to claim 7, wherein a B picture stored in said third frame memory is output in preference to an I or P picture stored in said first or second frame memory. Compressed image processing method. 10. A B which has not been output to said third frame memory.
Only when there is no picture, the oldest stored and unoutputted I- or P-picture is
8. The compressed image processing method according to claim 7, wherein the compressed image is output from the frame memory.