JP2008245016A - Image encoding apparatus and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image encoding apparatus capable of reducing memory capacity to be used, the amount of image data to be transferred, and inter-block differential computational quantity. <P>SOLUTION: The present invention relates to an image encoding apparatus that divides a picture of a moving image into blocks of a prescribed size and refers to reference pictures stored in a reference image memory for each divided block to perform motion compensation predictive encoding of the picture of the moving image, including: a reading means for reading from the reference image memory a block of the reference pictures corresponding to a block of the pictures of the moving image; an index value calculating means for calculating an index value by comparing the block of the pictures of the moving image with the block of the reference pictures read by the reading means; and a storage means for storing the block of the pictures of the moving image in the reference image memory on the basis of the index value calculated by the index value calculating means. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image encoding device, an image encoding method, and a program for encoding moving image data.

  When compressing and encoding moving image data, a motion-compensated predictive encoding method that uses temporal correlation between images to reduce temporal redundancy and compress data is used. In recent years, H.C. As in the H.264 coding method, a motion compensated prediction coding method using a plurality of reference pictures has been used. This can increase the coding efficiency when there is a moving image sequence that does not move as a background and there is something that moves in front of it and the background is visible and hidden.

  A reference picture with higher correlation can be selected by increasing the number of reference pictures that can be referred to. However, on the other hand, extra information for identifying the reference picture is required, so increasing the number of reference pictures in the dark clouds does not necessarily contribute to an improvement in coding efficiency. In addition, it has the disadvantages of increasing buffer memory that increases by the number of reference pictures, memory access to reference pictures, and increased differential operation processing, so it is important how to construct reference pictures efficiently. It becomes.

FIG. 5 is a block diagram illustrating a configuration example of a motion prediction unit and a motion compensation unit of an image encoding device that realizes a motion compensated prediction encoding scheme using five reference pictures.
In FIG. 5, reference numeral 501 denotes an encoding target image memory in which encoding target picture data 520 is stored. Reference numerals 502, 503, 504, 505, and 506 are reference image memories that store reference image data used for predictive coding in order from the latest one.

  Reference numeral 510 denotes a motion prediction unit (ME) as a motion prediction unit that outputs a motion vector 521 based on a motion compensated prediction encoding method and inter-block difference pixel data from an encoding target image and a reference image. Reference numeral 511 denotes a motion compensation unit that performs decoding processing based on the motion vector 521 obtained from the motion prediction unit 510 and the inter-block difference pixel data, and outputs the reconstructed image data to the reference image memory 502. It is a motion compensation unit (MC). The reference image memory 502 stores both the reference image data for the encoding target image (current picture) and the reconstructed image data for the next picture encoding.

FIG. 4 is a diagram showing the contents of the memory when the four pictures from time t to time t + 3 are encoded in the image encoding device of FIG. As shown in FIG. 4, each picture is divided into blocks of a predetermined size.
In FIG. 4, the encoding target image data at time t is stored in the encoding target image memory 501 as encoding target image data 410. Corresponding reference image data is stored in the reference image memories 502, 503, 504, 505, and 506 as reference image data 420, 430, 440, 450, and 460 in ascending order of time.
After predictive coding (ME) at time t, the reference image data at the next time t + 1 is reconstructed in the reference image memories 502, 503, 504, 505, and 506.
The result of motion compensation (MC) of the encoding result at time t is stored as reference image data 421 in the reference image memory 502. Hereinafter, the reference image memories 503, 504, 505, and 506 store the reference image data stored in the reference image memories 502, 503, 504, and 505 at the time of encoding at time t. That is, the reference image data 420 is shifted to the reference image data 431, the reference image data 430 to the reference image data 441, the reference image data 440 to the reference image data 451, and the reference image data 450 to the reference image data 461. Stored. Thereafter, the reference image data stored in the reference image memory is similarly stored at times t + 1, t + 2, and t + 3.
For example, the reference image data 420 that has been in the reference image memory 502 at the time t is moved to the reference image data 464 in the reference image memory 506 at the end of the encoding at time t + 3.

  The above is the same as H. 1 is a configuration example of an image encoding device using a memory update method in a FIFO (First In First Out) format of a short-time reference picture in the H.264 encoding method. In such a method using a short-time reference picture, it is not necessary to designate a picture to be referred to on the user side because the referenceable pictures are sequentially updated in the order in which they are stored. However, a memory whose size is proportional to the number of pictures to be referenced is required, and the difference calculation processing amount between blocks of each reference picture and the encoding target picture and the memory transfer amount necessary for the calculation are also the number of reference pictures. Increase by number.

  Therefore, for example, when it is known that the reference frame is referred to for a long time without moving from the shooting frame, such as the background, if the background picture is specified and the reference picture is encoded without updating, Encoding efficiency can be improved.

FIG. 8 illustrates a motion prediction unit and a motion compensation unit of an image coding apparatus that implements a motion compensated prediction coding method using one normal reference picture updated each time and one reference picture referenced each time. It is a block diagram which shows one structural example.
In FIG. 8, reference numeral 801 denotes an encoding target image memory in which encoding target picture data 820 is stored. Reference numeral 800 denotes a reference background image memory in which reference background image data to be referred to at the time of encoding is stored. A reference image memory 802 stores reference image data.

  Reference numeral 810 denotes a motion prediction unit (ME) as a motion prediction unit that outputs a motion vector 821 based on a motion compensated prediction encoding method and an inter-block difference pixel data from an encoding target image and a reference image. 811 is a motion compensation unit that performs a decoding process based on the motion vector 821 obtained from the motion prediction unit 810 and the inter-block difference pixel data, and outputs the reconstructed image data to the reference image memory 802. It is a motion compensation unit (MC).

  FIG. 6 is a diagram illustrating the contents of the memory when the four pictures from time t to time t + 3 are encoded in the image encoding device of FIG. As shown in FIG. 6, the encoding target image data at times t, t + 1, t + 2, and t + 3 are stored in the encoding target image memory 801 as encoding target image data 610, 611, 612, and 613, respectively. .

  Corresponding reference image data is stored in the reference image memory 802 as reference image data 620, 621, 622, and 623, respectively. After the encoding of each picture, the reference image data of the next picture is reconstructed from the reference image data 620 to the reference image data 621 at each encoding time t. Similarly, at t + 1, t + 2, and t + 3, reconstruction is performed from reference image data 621 to reference image data 622, reference image data 622 to reference image data 623, and reference image data 623 to reference image data 624. On the other hand, the reference background image data stored in the reference background image memory 800 always has the same content when each picture is encoded. That is, the reference background image data 600, 601, 602, 603, and 604 are unchanged.

  The above is H. 1 is a configuration example of an image encoding device using a long-time reference picture in the H.264 encoding scheme. In such a system, when an image referred to for a long time such as a background can be specified, it is possible to improve the coding efficiency, and it is important to detect a long-time reference picture. Patent Document 1 discloses a technique that can increase the reference picture reference efficiency by using the picture as a background when the difference value of pixel data between pictures is small and using it as a reference picture.

Japanese Patent Publication No. 63-59313

However, in the motion compensated predictive coding method using a plurality of reference pictures described above, the size of the buffer memory, the memory access to the reference picture, and the difference calculation process are increased by the increase in the number of reference pictures.
Further, in the above-described method of referring to a long-time reference picture as reference background image data, it is necessary to detect a long-time reference picture, and the calculation cost for accurately performing the processing increases.
The present invention has been made in view of the above-described problems, and an object thereof is to reduce the amount of memory transfer when using a plurality of reference pictures.

  The present invention divides a moving picture picture into blocks of a predetermined size, refers to a reference picture stored in a reference picture memory for each of the divided blocks, and performs motion compensation predictive coding on the moving picture picture. An image encoding apparatus, wherein the reference picture block corresponding to the moving picture picture block is read out from the reference picture memory, and the moving picture block is updated when the reference picture is updated. Based on the index value calculated by the index value calculating means, the index value calculating means for calculating the index value by comparing the reference picture block read by the reading means and the block of the reference picture Storage means for storing the block in the reference image memory.

  According to the present invention, it is possible to reduce the memory to be used, the transfer amount of image data, and the inter-block difference calculation amount by storing only the image data of the portion that has changed between frames as the reference image data. .

  Further, for example, extra calculation costs can be reduced by using general intra determination and inter determination in inter-block difference encoding as an inter-block comparison operation for determining reference image data.

  Also, for example, reference image data update when a moving object is present in an image can be adaptively performed by using a motion vector parameter for determination of reference image data update.

  Further, for example, by using a short-time reference picture and a long-time reference picture, when updating the reference image data, only the image data of a portion that has changed between frames may be stored as the reference image data. Therefore, it is possible to reduce the memory to be used, the transfer amount of image data, and the difference calculation amount between blocks.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a block diagram illustrating a configuration example of a motion prediction unit and a motion compensation unit of the image encoding device 1 according to the present embodiment.
In FIG. 1, reference numeral 101 denotes an encoding target image memory in which an encoding target image (picture data) 120 that is a moving picture is stored. A reference image memory 102 stores reference image data used for predictive encoding.

  Reference numeral 110 denotes a motion prediction unit (ME) as a motion prediction unit that outputs a motion vector 121 based on a motion compensated prediction encoding method and inter-block difference pixel data from an encoding target image and a reference image. Reference numeral 111 denotes a motion detection unit (MM) as motion detection means for detecting the similarity between the encoding target image and the reference image. 112 is a motion compensation unit that performs a decoding process based on the motion vector 121 obtained from the motion prediction unit 110 and the inter-block difference pixel data, and outputs reconstructed image data to the reference image memory 102. Part (MC). Note that the reference image memory 102 stores both reference image data for an encoding target image (current picture) and reconstructed image data for next picture encoding.

FIG. 2 is a diagram showing temporal transitions of the encoding target image stored in the encoding target image memory 101 and the reference image stored in the reference image memory 102 of the image encoding device 1 according to the present embodiment. is there.
In FIG. 2, encoding target images 210, 211, 212, and 213 are contents stored in the encoding target image memory 101 at times t, t + 1, t + 2, and t + 3. The reference image 220 is the content of the reference image memory 102 at time t. The image 221 shows the encoding result of the encoding target image at time t and the contents of the reference image memory 102 at time t + 1. The image 222 is the encoding result of the encoding target image at time t + 1 and the contents of the reference image memory 102 at time t + 2. The image 223 is the encoding result of the encoding target image at time t + 2 and the contents of the reference image memory 102 at time t + 3. The image 224 shows the encoding result of the encoding target image at time t + 3 and the contents of the reference image memory 102 at time t + 4.

  In FIG. 2, the encoding target image and the reference image are each divided into blocks of a predetermined size (for example, 4 × 4). In FIG. 2, “I” and “P” described in images 221 to 224 are selected as encoding methods for the corresponding blocks, respectively, intra (intra-picture) encoding and inter (inter-picture) encoding. It has been shown. In the present embodiment, the motion estimation unit 111 compares the block of the encoding target image with the block of the reference image and determines that the similarity is low, the motion prediction unit 110 performs intra-picture encoding. (Intra coding). Also, the motion prediction unit 110 performs inter-picture coding (inter-coding) for blocks that the motion detection unit 111 determines to have a high degree of similarity, that is, a high correlation between blocks.

FIG. 12 is a flowchart showing an operation process in which the encoding apparatus according to the present embodiment performs encoding at time t.
First, for example, the motion detection unit 111 selects one unprocessed block from the block-divided encoding target images, and reads the block image data from the encoding target image memory 101 (step S101).
Next, for example, the motion detection unit 111 reads pixel data of a block of the reference image corresponding to the block read in step S101 from the reference image memory 102 (step S102) (reading unit).
Next, the motion detection unit 111 compares the blocks of the reference image corresponding to the encoding target image (step S103).
Next, when determining that the degree of similarity is low, the motion detection unit 111 sets a determination flag indicating that the region is used for encoding the next encoding target image (step S104) (index value calculating unit).
The motion detection unit 111 performs this process for all blocks for each block (step S105).
When updating the reference image data for the next encoding target image, for example, the motion compensation unit 112 updates only the block flagged in step S104 as a reference image (step S106) (storage means). Next, the process proceeds to the encoding process for the next encoding target image (step S107).

  In FIG. 2, the image 221 is a result of encoding with the encoding target image stored in the encoding target image memory 101 and the reference image stored in the reference image memory 102 at time t. An area 231 of the image 221 is a range of blocks that have undergone a large change between the compared blocks. That is, the inter-picture difference is large and cannot be inter-coded and is intra-coded.

  The reference image in the reference image memory 102 used at time t + 1 is updated only for the intra-coded block. That is, the block of the reference image in the area not surrounded by the area 231 is not updated. Similarly, in the encoding at time t + 1, the intra-coded area 232, the intra-coded area 233 at the time t + 2, and only the intra-coded area 234 at the time t + 3 are used for the next encoding. Updated as a reference image. The part that has not been updated is used as pixel data of a block of the previous reference image.

Thus, the present embodiment is equivalent to an encoding method using five reference image memories. More specifically, as shown in FIG. 3, for example, at time t + 3, it is equivalent to having reference images for five pictures of reference images 324, 334, 344, 354, and 364.
As described above, according to the present embodiment, by updating only a place having a large change between pictures (a place where intra coding is selected) as a reference image, a plurality of pictures are stored in the reference image memory with a small amount of memory. I can keep it.

(Second Embodiment)
FIG. 9 is a block diagram illustrating a configuration example of the motion prediction unit and the motion compensation unit of the image encoding device 2 according to the present embodiment.
In FIG. 9, reference numeral 901 denotes an encoding target image memory in which an encoding target image (picture data) 920 is stored. Reference numeral 900 denotes a long-term reference image memory for storing long-term reference image data (long-term reference picture) that is not updated for a long time and is used for predictive coding. Reference numeral 902 denotes a short-term reference image memory for storing short-term reference image data (short-term reference pictures) updated in a short time used for predictive coding.

  Reference numeral 910 denotes a motion prediction unit (ME) as a motion prediction unit that outputs a motion vector 921 based on a motion compensated prediction encoding method and inter-block difference pixel data from an encoding target image and a reference image. Reference numeral 911 denotes a motion detector (MM) as a motion detector that detects the similarity between the encoding target image and the reference image. 912 is a motion compensation unit that performs a decoding process based on the motion vector 921 obtained from the motion prediction unit 910 and the inter-block difference pixel data, and outputs the reconstructed image data to the short-term reference image memory 902. Compensation unit (MC). Note that the short-term reference image memory 902 stores both reference image data for an encoding target image (current picture) and reconstructed image data for encoding the next picture.

  FIG. 7 is a diagram illustrating temporal transition of images stored in the encoding target image memory 901, the long-term reference image memory 900, and the short-term reference image memory 902 of the image encoding device 2 according to the present embodiment. The short-term reference image memory 902 stores short-term reference images 720, 721, 722, 723, and 724 at time times t, t + 1, t + 2, and t + 3. In each image, only the blocks 731, 732, 733 and 734 for which intra coding is selected are updated at the respective times.

  Thus, according to this embodiment, the same images 700, 701, 702, 703, and 704 are stored in the long-term reference image memory 900 as long-term reference images at times t, t + 1, t + 2, and t + 3. Therefore, it is only necessary to store only the image data of the changed portion as the reference image data, and the memory to be used, the transfer amount of the image data, and the inter-block difference calculation amount can be reduced.

(Third embodiment)
FIG. 10 is a block diagram illustrating a configuration example of the motion prediction unit and the motion compensation unit of the image encoding device 3 according to the present embodiment.
In FIG. 10, reference numeral 1001 denotes an encoding target image memory in which an encoding target image (picture data) 1020 is stored. Reference image memories 1002 and 1003 store reference image data used for predictive coding.

  Reference numeral 1010 denotes a motion prediction unit (ME) as a motion prediction unit that outputs a motion vector 1021 based on a motion compensated prediction encoding method and inter-block difference pixel data from an encoding target image and a reference image. Reference numeral 1011 denotes a motion detector (MM) as a motion detector that detects the similarity between the encoding target image and the reference image. A motion compensation unit 1012 performs a decoding process based on the motion vector 1021 obtained from the motion prediction unit 110 and the inter-block difference pixel data, and outputs reconstructed image data to the reference image memory 1002. Part (MM). The reference image memories 1002 and 1003 store both the reference image data for the encoding target image (current picture) and the reconstructed image data for encoding the next picture.

  FIG. 11 is a diagram illustrating temporal transition of images stored in the encoding target memory 1001 and the reference image memories 1002 and 1003 of the image encoding device 3 according to the present embodiment. The reference image memory 1002 holds all the data in the picture with respect to the content transitions 1120, 1121, 1122, 1123, and 1124 of the previous picture. Also, in the reference image memory 1003, the content transitions 1130, 1131, 1132, 1133, and 1134 are blocks that are determined to be low in similarity by the motion detection unit stage 1011 and are intra-coded, as in the first embodiment. Update only.

  Thus, according to the present embodiment, as in the first embodiment, it is equivalent to having reference images for five pictures as shown in FIG.

(Fourth embodiment)
In the above-described first to third embodiments, only a block-by-block comparison is described for the sake of simplicity. However, in order not to recognize the minute movement of the moving object within the frame as a change in the block, the index value calculation method is adaptively weighted using the result of the motion vector obtained by encoding. May be. This motion vector takes into account spatial and temporal lengths.

  Each means constituting the image coding apparatus and each step of the image coding method in the embodiment of the present invention described above can be realized by operating a program stored in a RAM or ROM of a computer. This program, a program for controlling the image encoding device, and a computer-readable recording medium on which the program is recorded are included in the present invention.

  In addition, the present invention can be implemented as, for example, a system, apparatus, method, program, or recording medium. Specifically, the present invention may be applied to a system including a plurality of devices.

  The present invention supplies a software program for realizing the functions of the above-described embodiments directly or remotely to a system or apparatus. In addition, this includes a case where the system or the computer of the apparatus is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention. In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. Furthermore, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.

  As another method, the program read from the recording medium is first written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Then, based on the instructions of the program, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are also realized by the processing.

It is a block diagram which shows the structure of the image coding apparatus which concerns on 1st Embodiment. It is a figure which shows the image memorize | stored in the memory of the image coding apparatus which concerns on 1st Embodiment. It is a figure which shows the image memorize | stored in the memory of the image coding apparatus which concerns on 1st Embodiment. It is a figure which shows the image memorize | stored in the memory in the conventional image coding apparatus. It is a block diagram which shows the structure of the conventional image coding apparatus. It is a figure which shows the image memorize | stored in the memory in the conventional image coding apparatus. It is a figure which shows the image memorize | stored in the memory of the image coding apparatus which concerns on 2nd Embodiment. It is a block diagram which shows the structure of the conventional image coding apparatus. It is a block diagram which shows the structure of the image coding apparatus which concerns on 2nd Embodiment. It is a block diagram which shows the structure of the image coding apparatus which concerns on 3rd Embodiment. It is a figure which shows the image memorize | stored in the memory of the image coding apparatus which concerns on 3rd Embodiment. It is a flowchart which shows the processing operation of the image coding apparatus which concerns on this embodiment.

Explanation of symbols

101 encoding target image memory 102 reference memory 110 motion prediction unit 111 motion detection unit 112 motion compensation unit 121 motion vector 900 long-term reference memory 901 encoding target image memory 902 short-term reference memory 910 motion prediction unit 911 motion detection unit 912 motion compensation unit

Claims (7)

An image encoding device that divides a moving picture picture into blocks of a predetermined size and performs motion compensation prediction encoding of the moving picture picture with reference to a reference picture stored in a reference picture memory for each of the divided blocks Because
Reading means for reading out the block of the reference picture corresponding to the block of the picture of the moving image from the reference image memory;
Index value calculating means for calculating an index value by comparing a block of a picture of the moving image with a block of the reference picture read by the reading means when updating the reference picture;
An image encoding apparatus comprising: storage means for storing a block of a picture of the moving image in the reference image memory based on the index value calculated by the index value calculation means.   2. The image coding apparatus according to claim 1, wherein the index value calculation unit compares the similarity between the block of the moving picture and the block of the reference picture.   The image encoding apparatus according to claim 2, wherein a block of a picture of the moving image is encoded by an intra code or an inter code in accordance with the similarity.   The image coding apparatus according to claim 2 or 3, wherein the similarity is compared based on a spatial and temporal length of a motion vector.   The reference image memory includes a long-term reference image memory that stores a long-term reference picture that is not updated for a long time, and a short-term reference image memory that stores a short-term reference picture that is updated to a new picture in a short time. Item 5. The image encoding device according to any one of Items 1 to 4. An image coding method for dividing a moving picture picture into blocks of a predetermined size and referring to a reference picture stored in a reference picture memory for each of the divided blocks and performing motion compensation predictive coding on the moving picture picture Because
A step of reading out the block of the reference picture corresponding to the block of the picture of the moving image from the reference image memory;
An index value calculating step of calculating an index value by comparing the block of the moving picture picture with the block of the reference picture read in the reading step when the reference picture is updated;
And a storing step of storing a block of a picture of the moving image in the reference image memory based on the index value calculated in the index value calculating step. An image encoding device that divides a moving picture picture into blocks of a predetermined size and performs motion compensation prediction encoding of the moving picture picture with reference to a reference picture stored in a reference picture memory for each of the divided blocks A program for controlling
A step of reading out the block of the reference picture corresponding to the block of the picture of the moving image from the reference image memory;
An index value calculating step of calculating an index value by comparing the block of the moving picture picture with the block of the reference picture read in the reading step when the reference picture is updated;
A program for causing a computer to execute a storing step of storing a picture block of the moving image in the reference image memory based on the index value calculated in the index value calculating step.

Images