JP4594163B2 - Image coding method and image coding apparatus - Google Patents

Description

  The present invention relates to an image encoding method and an image encoding apparatus, and is particularly suitable for use in compressing image data.

  Conventionally, various methods for compressing and recording image data have been proposed. A new MPEG4 part-10: AVC (ISO / IEC 14496-10, also known as H.264) method has been proposed. (Hereinafter, this method is called H.264 method). Details of H.264 are as detailed in Non-Patent Document 1.

Here, H.264 inter prediction will be described in detail with reference to FIGS.
In the H.264 inter prediction, a plurality of pictures can be used for prediction. For this reason, two lists (“List 0” and “List 1”) are prepared to identify the reference pictures. A maximum of five reference pictures can be assigned to each list.

  In the P picture, only “List 0” is used, and forward prediction is mainly performed. In the B picture, “List 0” and “List 1” are used to perform bi-directional prediction (or only forward or backward prediction). That is, “List 0” is mainly assigned with pictures for forward prediction, and “List 1” is mainly assigned with pictures for backward prediction.

  FIG. 5 shows an example of a reference list used for encoding. Here, the ratio of I picture, P picture, and B picture will be described by taking a standard case as an example. That is, a case where an I picture has 15 frame intervals, a P picture has 3 frame intervals, and a B picture between them has 2 frame intervals will be described. In FIG. 5, reference numeral 501 denotes image data arranged in the display order. In the square of the image data 501, a picture type and a number indicating the display order are entered. For example, picture I15 is the 15th I-picture in display order, and only intra prediction is performed. The picture P18 is the 18th P picture in the display order, and only forward prediction is performed. Picture B16 is the 16th B picture in display order, and performs bi-directional prediction.

The order of encoding is different from the display order and is the order of prediction. That is, in FIG. 5, the encoding order is “I15, P18, B16, B17, P21, B19, B20,.
In FIG. 5, reference numeral 502 denotes a reference list (List 0). This reference list (List 0) 502 contains pictures that have been once encoded and decoded. For example, when inter prediction is performed on the picture P21 (the 21st P picture in the display order), a picture that has already been encoded and decoded in the reference list (List 0) 502 is referred to. In the example shown in FIG. 5, pictures P06, P09, P12, I15, and P18 are stored in the reference list 502.

In inter prediction, for each macroblock, a motion vector having an optimal prediction value is obtained from the reference picture in the reference list (List 0) 502 and encoded. The pictures in the reference list (List 0) 502 are given reference picture numbers in order and are distinguished (given separately from the numbers shown).
When the encoding of the picture P21 is completed, the picture P21 is newly decoded and added to the reference list (List 0) 502. The oldest reference picture (here, picture P06) is removed from the reference list (List 0) 502. After this, encoding is performed with pictures B19 and B20, and continues to picture P24. The state of the reference list (List 0) 502 at this time is shown in FIG.

The operation of the conventional image coding apparatus when the reference list is updated in this way will be described with reference to the flowchart of FIG.
In FIG. 7, when encoding is started, encoding processing is performed on each picture, and each picture is encoded (step S701).
Next, it is determined whether or not the encoded picture is the final picture (step S703). If the encoded picture is the final picture, the encoding ends. On the other hand, if the encoded picture is not the final picture, the reference list (List 0) is updated (step S703) and the reference list (List 1) is updated (step S704). When the update of the reference list (List 0, List 1) is completed, the process proceeds to the encoding process for the next picture (step S70 1 ).

FIG. 8 shows how the reference list changes for each picture.
In FIG. 8, pictures are encoded in order from the top. Also, the picture being encoded and the contents of reference lists (List 0 and List 1) for the picture being encoded are shown. As shown in FIG. 8, when a P picture (or I picture) is encoded, the reference list (List 0 and List 1) is updated, and the oldest picture in the reference list (List 0 and List 1) is removed. Is done. In this example, the reference list (List 1) has only one picture. This is because if the number of pictures referred for backward prediction is increased, the amount of buffer until decoding increases. In other words, this is because a reference to a backward picture that is too far away from the picture being encoded is avoided.

  In the case shown in FIG. 8, the reference list is updated after the I picture or P picture is encoded, and the reference list is not updated after the B picture is encoded. That is, the B picture is not added to the reference list.

  On the other hand, in the H.264 system, it is allowed to add a B picture to the reference list. FIG. 9 shows how the reference list changes when a B picture is added to the reference list. In this case, the reference list is updated for all the pictures, and the reference list (List 0, List 1) is updated for any picture.

Note that the inter prediction of B picture H264 system, in comparison with the conventional MPEG2 method, the degree of freedom in time between axial prediction is enlarged. That is, as shown in FIG. 10, in the display order, not only the same future and past (Forward, Backward) as before, but also refer to two pictures from the past only (Forward, Forward) or only the future (Backward, Backward). it can.

ISO / IEC 14496-10, "Advanced Video Coding"

  In the H.264 standard, the prediction accuracy of a P picture and a B picture is improved by referring to a plurality of pictures, and the degree of freedom of the B picture is also increased in the prediction direction. However, in backward prediction possible with B pictures, if the prediction object is a picture that is temporally separated, it is necessary to buffer the image data during that time, and the amount of delay during decoding increases. Therefore, in practice, as described with reference to FIG. 8, backward prediction for about one P picture has been realistic.

  As a result, the number of pictures referred to in the B picture is five from the reference list (List 0) and one from the reference list (List 1), and the pictures overlap in these two reference lists (List 0, List 1). In consideration of this, it is practically only 5 sheets. For this reason, the prediction of the B picture has not always obtained sufficient accuracy.

  Further, as described with reference to FIG. 9, when a B picture is added to the reference list (List 0, List 1), if the reference list (List 0, List 1) is updated in all the pictures, the reference list (List 0) , List 1) has more B pictures. Usually, B pictures often allow deterioration due to compression, and the accuracy of inter prediction is not necessarily high. For this reason, if there are many B pictures in the pictures in the reference list (List 0, List 1), it is difficult to perform sufficient inter prediction.

  For example, in the case shown in FIG. 9, since the picture referred to by the picture P24 is a picture in the reference list (List 0), the pictures are B16, B17, P21, B19, and B20. In this way, while there is only one P picture to be referenced, there are four B pictures to be referenced, making it difficult to perform sufficient prediction.

  The present invention has been made in view of the above-described problems, and an object thereof is to improve the accuracy of inter prediction.

An image encoding method of the present invention refers to a plurality of pictures stored as data in a memory as reference pictures, and performs an encoding step for encoding pictures constituting image data based on the reference result; A picture management step of storing a restored image of the picture encoded in the encoding step in the memory and managing it as the reference picture, wherein the image data does not perform inter prediction between pictures. A picture, a P picture that performs forward inter prediction, and a B picture that performs bidirectional inter prediction, and the picture management step is a reference that is referred to when encoding the P picture or B picture A first reference list for storing a plurality of pictures, and a reference picture that is referenced only when the B picture is encoded. A plurality of reference pictures stored in the memory are classified and managed in a second reference list for storing a plurality of imagers, and the I picture or P picture is encoded in the encoding step. If it is, the restored image of the I picture or P picture is added as a new reference picture to both the first and second reference lists, and the first and second reference lists are updated, If the B picture is encoded in the encoding step, the restored image of the B picture is added as a new reference picture only to the second reference list, and the second reference list is updated. It is characterized by performing .

An image encoding device of the present invention refers to a plurality of pictures stored as data in a memory as reference pictures, and based on the referred results, encoding means for encoding pictures constituting image data; Picture management means for storing a restored image of a picture encoded by the encoding means in the memory and managing it as the reference picture, and the image data does not perform inter prediction between pictures. A picture, a P picture that performs forward inter prediction, and a B picture that performs bidirectional inter prediction, and the picture management means is a reference that is referred to when the P picture or B picture is encoded A first reference list that stores a plurality of pictures and a plurality of reference pictures that are referenced only when the B picture is encoded. When the plurality of reference pictures stored in the memory are classified and managed in the second reference list, and the I picture or P picture is encoded by the encoding means, The restored picture of the I picture or P picture is added as a new reference picture to both the first and second reference lists, the first and second reference lists are updated, and the encoding means When the B picture is encoded, a restored image of the B picture is added as a new reference picture only to the second reference list, and the second reference list is updated. And

According to the present invention, it is possible to increase than conventional picture prediction target Lee centers prediction. Thereby, the precision of inter prediction can be improved.

Next, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating an example of a configuration of an image encoding device according to the present embodiment.
In FIG. 1, input image data is divided into macroblocks and sent to the subtraction unit 101. The subtraction unit 101 calculates a difference between the image data and the predicted value and outputs the difference to the integer DCT conversion unit 102. The integer DCT conversion unit 102 performs integer DCT conversion on the input data and outputs the result to the quantization unit 103. The quantization unit 103 quantizes the input data. One of the quantized data is sent to the entropy encoding unit 115 as difference image data. The other is subjected to inverse quantization by the inverse quantization unit 104 and then subjected to inverse integer DCT conversion by the inverse integer DCT conversion unit 105. The predicted value is added to the data subjected to the inverse integer DCT conversion by the adding unit 106. Thereby, the image is restored.

  One of the restored images is sent to the frame memory 107 for intra prediction. The other is subjected to deblocking filter processing by the deblocking filter 109 and then sent to the frame memory 110 for inter prediction. The image in the frame memory 107 for intra prediction is used for intra prediction in the intra prediction unit 108. In this intra prediction, the value of an adjacent pixel of an already encoded block is used as a prediction value in the same picture.

  An image in the frame memory 110 for inter prediction is composed of a plurality of pictures. The plurality of pictures are divided into two reference lists, “list 0” and “list 1”. A plurality of pictures divided into two lists are used for inter prediction in the inter prediction unit 111. After the inter prediction, the memory controller 113 updates the pictures in the reference list. In inter prediction performed by the inter prediction unit 111, a predicted image is determined using an optimal motion vector based on the result of motion detection performed by the motion detection unit 112 on image data of different frames.

  As a result of intra prediction and inter prediction, the selection unit 114 selects an optimal prediction. The motion vector is sent to the entropy encoding unit 115 and encoded together with the difference image data. In this way, an output bit stream is formed.

  In the present embodiment, the B picture encoded as described above is added to the reference list “List 1” used only for the B picture out of the two reference lists “list 0” and “list 1”. Yes.

Here, an example of the operation of the image coding apparatus when adding a B picture to the reference list will be described with reference to the flowchart of FIG.
When encoding is started, encoding processing is performed on each picture (step S201).
Next, it is determined whether or not the encoded picture is the final picture (step S202). If the encoded picture is the final picture, the encoding ends.

  On the other hand, if the encoded picture is not the last picture, the process proceeds to reference list update determination (steps S203 and S204). First, it is determined whether or not the encoded picture is a B picture, that is, an I picture or a P picture (step S203). If the result of this determination is an I picture or P picture, the reference list (List 0) is updated (step S204) and the reference list (List 1) is updated (step S205).

On the other hand, when the encoded picture is a B picture, only the reference list (List 1) is updated (step S205).
When the update of the reference list is completed as described above, the process proceeds to the encoding process for the next picture (step S20 1 ).

  As described above, in this embodiment, after encoding an I picture or a P picture, the reference list is updated for the reference list (List 0) and the reference list (List 1), and the encoded I picture or P picture is updated. To the list and delete old pictures. On the other hand, after encoding the B picture, the reference list is updated only for the reference list (List 1), the encoded B picture is added to the list, and the old picture is deleted. That is, in the present embodiment, the B picture is added to the reference list (List 1) used only for the B picture without adding the B picture to the reference list (List 0) used for the P picture. The number of pictures to be predicted for inter prediction when coding is increased so that the accuracy of inter prediction can be improved.

Next, a specific example of how the reference picture is updated will be described with reference to FIG. Here, a case where the ratio of I picture, P picture, and B picture is standard will be described as an example. That is, a case where an I picture has 15 frame intervals, a P picture has 3 frame intervals, and a B picture between them has 2 frame intervals will be described.
In FIG. 3, when the picture P21 is first coded, there are pictures P06, P09, P12, I15, and P18 in the reference list (List 0), and these pictures are referred to when coding. The On the other hand, the reference list (List 1) includes pictures B13, B14, P18, B16, and B17 (not used for encoding of P pictures).

  When the encoding of the picture P21 is finished, the reference picture is updated. Specifically, as a result of deleting the oldest picture P06 and newly adding the picture P21, the inside of the reference list (List 0) is updated to pictures P09, P12, I15, P18, and P21. Also, the reference list (List 1) is updated to pictures B14, P18, B16, B17, and P21 as a result of deleting the oldest picture B13 and newly adding a picture P21.

  Next, in the state of this reference list, the picture B19 is encoded. Since the picture B19 is a B picture, when the encoding of the picture B19 is completed, only the reference list (List 1) is updated. That is, while the inside of the reference list (List 0) remains unchanged, the oldest picture B14 is deleted and the picture B19 is newly added to the inside of the reference list (List 1), resulting in the picture P18. , B16, B17, P21, B19. Similarly, the picture B20 is encoded. Since the picture B20 is a B picture, only the reference list (List 1) is updated when the encoding of the picture B20 is completed. That is, while the inside of the reference list (List 0) remains as pictures P09, P12, I15, P18, and P21, the inside of the reference list (List 1) is updated to show pictures B16, B17, P21, and B19. , B20. Hereinafter, the encoding of the pictures P24, B22, B23,... And the updating of the reference list are sequentially performed.

In this embodiment, for example, the reference list (List 0) used when encoding the picture B19 includes pictures P09, P12, I15, P18, and P21, and the reference list (List 1) includes Pictures B14, P18, B16, B17, and P21 are included. Therefore, even if the overlapping pictures P18 and P21 are removed, there are 8 pictures that are referred to in inter prediction. Compared with the conventional technique shown in FIG. 10, which is normally considered, the number of pictures to be inter-predicted is larger when B pictures are encoded. For this reason, inter prediction with higher accuracy can be performed, and as a result, good image compression can be performed.

FIG. 4 shows an example of an internal configuration of a video camera to which the image coding apparatus according to the above-described embodiment is applied.
In FIG. 4, during moving image shooting and recording, an image signal captured by the CCD 1402 through the lens 1401 is converted into digital data by the A / D 1403, and brightness and color correction is performed by the camera processing unit 1404. Data output from the camera processing unit 1404 is encoded with moving image data by a moving image encoding / decoding processing unit (image processing unit) 1405, and recorded on a recording medium 1407 by a recording / playback control unit (REC / PB) 1406. Is done. Here, examples of the encoding method include the MPEG4 part-10: AVC (ISO / IEC 14496-10, also known as H.264) method described above. As described above, in the example illustrated in FIG. 4, the image encoding device according to the above-described embodiment is provided in the moving image encoding / decoding processing unit (image processing unit) 1405. The recording medium 1407 is, for example, a magnetic tape or a disk. The encoded video data can also be output to the outside through a digital interface (I / F) 1415.

The digital interface 1415 is IEEE1394 or USB (Universal Serial Bus). Data before being encoded by the moving image encoding / decoding processing unit 1405 is also sent to a display unit 1408 (for example, an LCD display panel). The user determines the shooting angle of view and composition while viewing the image displayed on the display unit 1408. The CPU 1409 controls each processing block. Further, an operation instruction input unit 1601 is connected to the CPU 1409, and a trigger button 1602 for moving image shooting and a shutter button 1603 for still image shooting are provided in the operation instruction input unit 1601. The user starts / stops moving image shooting recording by pressing a trigger button 1602 for moving image shooting. The trigger button 1602 for moving image shooting does not need to be kept pressed during shooting and recording. When the trigger button 1602 is pressed once, the recording operation is started, and when pressed again, the recording operation is stopped. When moving image shooting / recording is executed according to the operation of the trigger button 1602 for moving image shooting, the moving image encoding / decoding processing unit (image processing unit) 1405 also executes an encoding operation. During this time, the operation of the image coding apparatus as described in the above embodiment can be executed. At the time of reproduction, the recorded moving image data is read from the recording medium 1407 by the recording / reproduction control unit 1406, and the moving image encoding / decoding processing unit 1405 performs decoding while using information such as a reference relationship in the inter prediction at the time of encoding. The display image decoded by the display unit 1408 or the like can be displayed.

  Further, the user can record a still image shot by pressing a shutter button 1603 for shooting a still image on the memory card 1412. Data output from the camera processing unit 1404 is input to the still image encoding / decoding processing unit 1413 together with the moving image encoding / decoding processing unit 1405. The still image encoding / decoding processing unit 1413 performs still image data encoding processing and thumbnail generation. The encoding method is, for example, JPEG.

The encoded still image data is recorded on the memory card 1412 by the card control unit 1414. During reproduction, the recorded still image data is read from the memory card 1412 by the card control unit 1414, and the thumbnail image decoding and the main image decoding are performed by the still image encoding / decoding processing unit 1413. The CPU 1409 instructs whether to decode the thumbnail image or the main image. The encoded still image data can also be output to the outside through the digital interface 1415. The digital interface is IEEE1394 or USB.

  After the user presses the trigger button 1602 for moving image shooting to start moving image recording, if the user wants to shoot a still image in a desired scene during moving image recording, the still image shooting is performed without stopping the moving image recording. Still image data shot by pressing the shutter button 1603 can be recorded on the memory card 1412.

(Other embodiments of the present invention)
Each of the above-described embodiments is merely an example of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.
Also, a software program for realizing the functions of the above-described embodiment for an apparatus connected to the various devices or a computer in the system so as to operate various devices to realize the functions of the above-described embodiments. What was implemented by supplying the code and operating the various devices in accordance with a program stored in a computer (CPU or MPU) of the system or apparatus is also included in the scope of the present invention.

  In this case, the program code of the software itself realizes the functions of the above-described embodiments, and the program code itself and means for supplying the program code to the computer, for example, the program code are stored. The recorded medium constitutes the present invention. As a recording medium for storing the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  Further, by executing the program code supplied by the computer, not only the functions of the above-described embodiments are realized, but also the OS (operating system) or other application software in which the program code is running on the computer, etc. It goes without saying that the program code is also included in the embodiment of the present invention even when the functions of the above-described embodiment are realized in cooperation with the embodiment.

  Further, after the supplied program code is stored in the memory provided in the function expansion board of the computer or the function expansion unit connected to the computer, the CPU provided in the function expansion board or function expansion unit based on the instruction of the program code Needless to say, the present invention includes a case where the functions of the above-described embodiment are realized by performing part or all of the actual processing.

1 is a diagram illustrating an example of a configuration of an image encoding device according to an embodiment of the present invention. It is a flowchart explaining an example of operation | movement of the image coding apparatus at the time of showing embodiment of this invention and adding B picture to a reference list. It is a figure which shows embodiment of this invention and shows the specific example of a mode that a reference picture is updated. It is a figure which shows embodiment of this invention and shows an example of the internal structure of the video camera with which the image coding apparatus is applied. It is a figure which shows the prior art and shows the example of the reference list at the time of encoding the encoding of the picture P21. It is a figure which shows the prior art and shows the example of the reference list at the time of encoding the encoding of the picture P24. It is a flowchart which shows the prior art and demonstrates operation | movement of the conventional image coding apparatus when a reference list is updated. It is a figure which shows the prior art and shows the mode of a change of a reference list for every picture. It is a figure which shows the prior art and shows the mode of a reference list change when adding a B picture to a reference list. It is a figure which shows the prior art and shows the reference picture in the inter prediction of the B picture of a H264 system conceptually.

Explanation of symbols

102 Integer DCT Transformer 103 Quantizer 104 Dequantizer 105 Inverse Integer DCT Transformer 109 Deblocking Filter 110 Frame Memory 111 Inter Predictor 112 Motion Detector 113 Memory Controller 115 Entropy Encoder 114 Selector

Claims (2)

An encoding step of referring to a plurality of pictures stored as data in the memory as reference pictures and encoding pictures constituting the image data based on the reference results;
A picture management step of storing a restored image of the picture encoded by the encoding step in the memory and managing it as the reference picture;
The image data includes an I picture that does not perform inter prediction between pictures, a P picture that performs forward inter prediction, and a B picture that performs bidirectional inter prediction.
The picture management step includes a first reference list that stores a plurality of reference pictures that are referred to when the P picture or B picture is encoded, and a reference picture that is referred only when the B picture is encoded. Categorizing and managing a plurality of reference pictures stored in the memory in a second reference list for storing a plurality of images,
Further, when the I picture or P picture is encoded in the encoding step, the restored image of the I picture or P picture is used as a new reference picture in both the first and second reference lists. In addition, the first and second reference lists are updated, and when the B picture is encoded in the encoding step, the restored picture of the B picture is used as a new reference picture. An image encoding method , wherein the second reference list is updated by adding only to the second reference list . Encoding means for referring to a plurality of pictures stored as data in the memory as reference pictures, and encoding pictures constituting the image data based on the reference results;
Picture management means for storing a restored image of a picture encoded by the encoding means in the memory and managing it as the reference picture;
The image data includes an I picture that does not perform inter prediction between pictures, a P picture that performs forward inter prediction, and a B picture that performs bidirectional inter prediction.
The picture management means includes a first reference list that stores a plurality of reference pictures that are referred to when the P picture or B picture is encoded, and a reference that is referred only when the B picture is encoded. Classifying and managing a plurality of reference pictures stored in the memory in a second reference list for storing a plurality of pictures;
Further, when the I picture or P picture is encoded by the encoding means, the restored picture of the I picture or P picture is used as a new reference picture in both the first and second reference lists. In addition, when the first and second reference lists are updated and the B picture is encoded by the encoding means, the restored picture of the B picture is used as a new reference picture. An image encoding apparatus that updates the second reference list by adding only to the second reference list .

Images