JP2007325119A - Image processing apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce memory access volume for reading reference pictures in the case of encoding image data by using inter prediction. <P>SOLUTION: In image data encoding processing using inter prediction, image data in a common retrieval range corresponding to macro blocks of a plurality of pictures to be encoded from a picture to be encoded next after updating a reference picture up to pictures to be encoded which are used for the succeeding updating of a reference picture are read out and the movement compensation prediction of respective macro block are executed in parallel by using the read image data in the common retrieval range to reduce the memory access volume for reading the reference pictures. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method for compressing image data using inter-frame correlation.

  Conventionally, various image encoding methods for compressing and recording image data have been proposed. A new MPEG4 part-10: AVC [Advanced Video Coding] (ISO / IEC 14496-10, also known as H.264) method has been proposed. Hereinafter, H.C. The H.264 image encoding method is also simply referred to as “H.264 method”.

  H. In the H.264 system, image data is encoded by combining motion compensation prediction, discrete cosine transform (DCT), quantization, entropy encoding, and the like. Motion compensation prediction is performed using image data of different frames as reference pictures.

  Also, in the configuration of an encoding circuit that encodes image data, there are many examples in which one encoding target image data is encoded in a time of several frames by dividing the encoding procedure into a pipeline. For example, Non-Patent Document 1 discloses an H.264 format that compresses and encodes image data using a seven-stage pipeline. An encoding circuit conforming to the H.264 system is described.

“Development of codec core for mobile devices compatible with both H.264 and MPEG-4”, Nikkei Electronics, Nikkei BP, September 27, 2004, no. 883, p. 123-133

  As described above, when image data is encoded using inter prediction in which motion-compensated prediction is performed with reference to image data of different frames, the image data of the reference picture is handled in addition to the image data to be encoded. The amount of data handled in the conversion process increases. H. In the H.264 system, since a plurality of reference pictures can be used, the amount of data tends to increase further.

  In the encoding circuit, if the number of reference pictures to be read from the frame memory in which the reference picture is stored to the codec unit that performs the encoding process is large, the number of data accesses between the codec unit and the frame memory becomes enormous. For this reason, measures such as increasing the clock speed used for data transfer (higher frequency) and increasing the data bus width are required. However, the design is difficult due to increased power consumption and high-speed clock. Etc. will occur.

  The present invention has been made in view of such circumstances, and an object thereof is to reduce the memory access amount for reading a reference picture when encoding image data using inter prediction.

The image processing method according to the present invention is an image processing method for performing motion compensation predictive coding using image data of different frames as a reference picture, and is composed of a plurality of reference pictures encoded at arbitrary frame intervals. A reference list in which an internal reference picture is updated by a picture obtained by decoding the reference picture, and after the reference list is updated, from the next encoding target picture, the encoding used for the update of the reference list is performed next. A sequence of a plurality of encoding target pictures up to the target picture is encoded by performing motion compensation prediction in parallel.
The image processing method according to the present invention is an image processing method for performing motion compensation prediction encoding using image data of different frames as a reference picture, and the picture to be encoded includes an I picture that does not perform motion compensation prediction, and a forward direction. There are P pictures that perform prediction and B pictures that perform bi-directional prediction. The reference pictures stored in the frame memory are encoded after the I picture, P picture, or part of the B pictures are encoded. From the next encoding target picture to the next encoding target picture used for updating the contents of the frame memory. A sequence of a plurality of encoding target pictures is encoded by performing motion compensation prediction in parallel.
An image processing apparatus according to the present invention is an image processing apparatus that uses the above-described image processing method, and includes a memory unit that stores image data of a reference picture and an encoding unit that performs an encoding process including motion compensation prediction. It is characterized by providing.
An image processing method according to the present invention is an image processing method for performing motion compensation predictive coding using image data of different frames as a reference picture, a storage step of storing the reference picture in a storage unit, and a storage unit storing the reference picture After the reference picture is updated, a common search range corresponding to macroblocks of a plurality of encoding target pictures from the next encoding target picture to the encoding target picture stored next as a reference picture in the storage means A read step of reading image data from each reference picture stored in the storage means, and a macro of the plurality of encoding target pictures using the image data of the common search range read in the read step An encoding step for performing motion-compensated prediction in parallel for the block and encoding That.
An image processing apparatus according to the present invention is an image processing apparatus that performs motion compensation predictive coding using image data of different frames as a reference picture, and stores the reference picture in the storage unit and the storage unit. After updating the reference picture, the image data in the common search range corresponding to the macroblocks of the plurality of encoding target pictures from the next encoding target picture to the encoding target picture stored next as the reference picture in the storage means Read out from each reference picture stored in the storage means, and using the image data in the common search range read out by the reading means, the motion of macroblocks of the plurality of encoding target pictures Coding means for performing compensation prediction in parallel and coding.
A program according to the present invention is a program for causing a computer to perform image processing for performing motion compensation predictive coding using image data of different frames as a reference picture, and storing the reference picture in a storage unit; After updating the reference picture stored in the storage means, it corresponds to a plurality of encoding target picture macroblocks from the next encoding target picture to the next encoding target picture stored as a reference picture in the storage means. A readout step of reading out image data of a common search range from each reference picture stored in the storage means, and using the image data of the common search range read out in the readout step, the plurality of codes Encoding in parallel with motion-compensated prediction for the macroblock of the target picture Characterized in that to execute the Goka steps on a computer.
A computer-readable recording medium according to the present invention records the above program.

  According to the present invention, when encoding is performed using a reference picture, image data in a common search range corresponding to macroblocks of a plurality of encoding target pictures is read from each reference picture, and each encoding target picture is read. Motion compensation prediction is performed in parallel for the macroblock. Thereby, the memory access amount for reading the reference picture in the encoding process can be reduced, and the power consumption required for the encoding process can be reduced.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
The encoder to which the image processing apparatus according to the first embodiment of the present invention described below is applied, converts input image data into H.264 format. The image is encoded by an image encoding method compliant with the H.264 method.

FIG. 1 is a block diagram illustrating a configuration example of an encoder 100 according to the first embodiment.
The encoder 100 includes a codec unit (encoding unit) 101 that performs an encoding process, and image memories 111, 112, and 113 that store image data. There are three types of image memories: a video memory 111 that stores input image data to be encoded, a frame memory 112 that stores reference pictures, and a stream buffer 113 that stores streams that are encoding results. Of these, the video memory 111 and the frame memory 112 store uncompressed image data, and thus require a particularly large capacity.

  The three types of image memories 111, 112, and 113 may be configured by different memory chips, or may be configured by allocating each area in one memory chip. The three types of image memories 111, 112, and 113 are configured by allocating each of the two types of image memories in one memory chip and the remaining one type of image memory with independent memory chips. You may do it.

  The codec unit 101 includes a main encoding function, and includes a scaling unit 102, a motion estimation unit 103, an entropy encoding unit 104, and a local decoding unit 105. The scaling unit 102 performs discrete cosine transform (DCT) on the encoding target image data read from the video memory 111 and further quantizes the data. The scaling unit 102 outputs the quantized encoding target image data to the entropy encoding unit 104 and the local decoding unit 105. In the processing in the scaling unit 102, when motion compensation prediction is performed, the processing result in the motion estimation unit 103 is referred to.

  The motion estimation unit 103 performs motion compensation prediction by performing motion detection based on the encoding target image data read from the video memory 111 and the reference picture read from the frame memory 112. The entropy encoding unit 104 entropy encodes the encoding target image data quantized by the scaling unit 102 and supplies the encoded image data to the stream buffer 113. The local decoding unit 105 performs decoding processing such as inverse quantization and inverse DCT transform on the encoding target image data quantized by the scaling unit 102. The local decoding unit 105 stores the image data obtained by locally decoding the encoded image data in the frame memory 112 as a reference picture.

FIG. 2 is a block diagram illustrating a functional configuration of the codec unit (encoding unit) 101.
As shown in FIG. 2, the input image data to be encoded is divided into macro blocks, a subtracter 201 obtains a difference from the predicted value, and a DCT transform unit 202 performs integer DCT transform. Further, after being quantized by the quantization unit 203, the difference image data is supplied to the entropy encoding 215 and also supplied to the inverse quantization unit 204.

  The image data supplied to the inverse quantization unit 204 is inversely quantized by the inverse quantization unit 204, and then subjected to inverse integer DCT conversion by the inverse DCT conversion unit 205, and the predicted value is added by the adder 206 to restore the image Is done. The output of the adder 206 is sent to the frame memory 207 for intra prediction and also sent to the frame memory 210 for inter prediction via a deblocking filter processing unit 209 that performs deblocking filter processing.

  An image in the frame memory 207 for intra prediction is used for intra prediction in the intra prediction unit 208. In intra prediction, the value of an adjacent pixel of an already encoded block is used as a prediction value in the same picture.

  An image in the frame memory 210 for inter prediction is composed of a plurality of pictures as will be described later, and the pictures are divided into two reference lists List0 and List1. The image in the frame memory 210 is used in the inter prediction in the inter prediction unit 211, and the internal image is updated by the memory controller 213 after the prediction. In inter prediction, the motion estimation unit 212 performs motion detection on image data having different frames, and obtains an optimal motion vector to determine a predicted image.

  As a result of the intra prediction and the inter prediction, the selection unit 214 selects an optimal prediction, and the intra prediction mode or the prediction vector is supplied to the entropy encoding unit 215 and is encoded together with the difference image data to form an output bit stream. The

Hereinafter, the general H.P. An encoding order and an encoding method in the H.264 scheme will be described.
First, H. The encoding order of image data in the H.264 system will be described with reference to FIG.
Here, the ratio of an I picture that performs only intra prediction, a P picture that performs only forward prediction, and a B picture that performs bi-directional prediction is as follows: an I picture has an interval of 15 frames, and a P picture has an interval of 3 frames. The case where is 2 frames will be described.

  FIG. 3 shows a display order, an encoding order, and an arrangement on the output bitstream relating to image data, and numbers indicating picture types and display order are entered in the squares. For example, I15 is the 15th I picture in the display order, P18 is the 18th P picture in the display order, and B16 is the 16th B picture in the display order. The order of encoding is different from the display order, and encoding is performed in the order of prediction. That is, in the example shown in FIG. 3, encoding is performed in the order of I15, P18, B16, B17, P21, B19, B20, and so on, and the output stream is arranged in this order.

  An example of the reference list in this case is shown in FIG. The reference list used for inter prediction differs depending on the type of picture to be encoded. When the picture to be encoded is a P picture, forward prediction is mainly performed using only the reference list List0. When the encoding target picture is a B picture, bi-directional prediction (or prediction only forward or backward) is performed using the reference lists List0 and List1.

  That is, the reference list List0 mainly stores reference pictures for forward prediction, and the reference list List1 mainly stores reference pictures for backward prediction. FIG. 4 shows a case where inter prediction is performed on the picture P21 which is the 21st P picture in the display order. A maximum of five reference pictures can be assigned to each reference list.

  401 in FIG. 4 is an example of a reference picture stored in the reference list (List0) when inter prediction is performed on the picture P21. In the reference list, pictures that have already been encoded and decoded are stored. In this example, pictures P06, P09, P12, I15, and P18 are stored in the reference list.

  In inter prediction, for each macroblock, a motion vector having an optimal prediction value is obtained from the reference picture in the reference list, and a difference value from the predicted block is obtained and encoded. Pictures in the reference list are distinguished by being sequentially assigned reference picture numbers (provided separately from the numbers shown). When the encoding of the picture P21 is completed, the oldest reference picture (picture P06 in this example) is removed from the reference list from the reference list, and the picture P21 is newly decoded and added to the reference list. As shown in FIG. 3, the encoding is thereafter performed with pictures B19 and B20, and further continues to picture P24.

  FIG. 5 is a diagram showing a change in the reference list for each encoding target picture. In FIG. 5, the picture being encoded and the contents of the reference lists List0 and List1 are shown from top to bottom in the order of pictures to be encoded. As shown in FIG. 5, when a P picture (or I picture) is encoded, the reference list is updated, the encoded picture is decoded and added, and the oldest picture in the reference list is removed. ing. In the example shown in FIG. 5, the reference list List1 has only one picture. However, if the back reference is increased, the buffer amount until decoding increases. This is because.

  In the example shown in FIG. 5, the reference list is updated after the I picture or P picture is encoded. In the H.264 standard, not only an I picture or a P picture but also a B picture can be added to the reference list. Further, even an I picture or a P picture can be excluded from the reference list. That is, the update of the reference list is performed for an arbitrary picture. Conversely, if the reference list is not updated, the reference list does not change. In practice, an I picture or P picture is set at a constant cycle, and the I list or P picture is often set so that the reference list is also updated periodically.

Here, the inter prediction for every macroblock is demonstrated in detail using FIG.
In FIG. 6, reference numeral 601 denotes an encoding target picture, and reference numerals 602 to 605 denote reference pictures in the reference list. H. In the inter prediction in the H.264 system, a reference picture in the reference list can be selected for each macroblock. That is, as shown in FIG. 6, the macroblock 606 in the encoding target picture 601 refers to the reference picture 602, the macroblock 607 refers to the reference picture 604, and the macroblock 608 refers to the reference picture 603. .

  H. H.264 inter prediction is performed in units of 16 × 16 pixel macroblocks, but it is also possible to perform prediction by dividing the macroblock into smaller sub-macroblocks. In this case, a plurality of optimum motion vectors obtained by inter prediction are determined for one macroblock. Hereinafter, for the sake of simplicity, a case will be described where one optimal motion vector is determined by inter prediction for a macroblock of 16 × 16 pixels without being divided into sub-macroblocks.

The search range on the macroblock and the reference picture in the inter prediction will be described with reference to FIGS.
FIG. 7 is a diagram illustrating an example of a search range of macroblocks and reference pictures for the encoding target picture B19. When the encoding target picture B19 is encoded, the reference list is composed of List0 and List1, reference pictures P09, P12, I15, and P18 are stored in the reference list List0, and reference picture P21 is stored in the reference List1. Has been.

  As shown in FIG. 7, a search range is set on each reference picture for one macroblock MBA on the encoding target picture B19. A search range S09A is set for the reference picture P09, a search range S12A is set for the reference picture P12, and a search range S15A is set for the reference picture I15. Further, the search range S18A is set for the reference picture P18, and the search range S21A is set for the reference picture P21.

  H. In the H.264 system, the size of a macroblock is 16 × 16 pixels. As for the search range, a larger range than the macro block is assigned, and is usually several times larger than the macro block, for example, a range of about 32 × 32 pixels. In addition, the position of the search range does not have to be the same for each reference picture, and may be a position that is different from a search result that has already been performed. In the macro block MBA, motion prediction is performed for each search range (S09A, S12A, S15A, S18A, S21A), and an optimal motion vector is determined.

  Similarly, FIG. 8 shows a search range of macroblocks and reference pictures for the encoding target picture B20, and FIG. 9 shows a search range of macroblocks and reference pictures for the encoding target picture P24.

  As shown in FIG. 8, when encoding the encoding target picture B20, the reference list includes List0 and List1. Search ranges S09B, S12B, S15B, S18B, and S21B are set in each reference picture for the macroblock MBB on the encoding target picture B20. Also, as shown in FIG. 9, when encoding the encoding target picture P24, the reference list is composed of only List0. Search ranges S09C, S12C, S15C, S18C, and S21C are set in each reference picture for the macroblock MBC on the encoding target picture P24.

  Here, the search range set for each reference picture in the reference list may have a different position if the encoding target picture is different. For example, in the reference picture P09 shown in FIGS. 7 to 9, the search ranges S09A, S09B, and S09C are set for the macroblocks MBA, MBB, and MBC, but they may be different ranges. Usually, a close range is often set.

A data movement when inter prediction is performed by the encoder according to the present embodiment will be described with reference to FIG.
Data of the macro block MB to be encoded in the encoding target image data is read from the video memory 1001 to the inter prediction unit 1002. Also, image data in the search range S in the reference picture is read from the frame memory 1003 into the inter prediction unit 1002.

  The inter prediction unit 1002 compares the read data of the macroblock MB and the data of the search range S with a difference error or the like, and obtains an optimal motion vector in the search range S. When the process is completed for the search range S of one reference picture, an optimal motion vector in the search range S of the next reference picture is obtained. When the processing is completed for the search range of all reference pictures, an optimal motion vector is determined for all the reference pictures for the macroblock.

  When the prediction process for one macroblock is completed, the data of the next macroblock is read from the video memory 1001 into the inter prediction unit 1002, and the image data of the next search range is read from the frame memory 1003 into the inter prediction unit 1002. It is. When motion vectors are obtained for all the macroblocks in the encoding target image data, the processing proceeds to the next image data. At this time, the reference picture in the reference list may be updated.

In summary, the flow of processing related to inter prediction will be described with reference to FIG.
FIG. 11 is a flowchart showing a flow of processing related to inter prediction.
When the inter prediction is started, the codec unit determines a macroblock for which inter prediction is performed in the encoding target picture (1101), and reads data of the macroblock from the video memory 1001 (1102). Next, the codec unit determines a reference picture in the reference list (1103), and reads image data in the search range of the determined reference picture from the frame memory (1104).

  The codec unit performs motion prediction using the read macroblock data and search range data (1105), and determines an optimal motion vector within the search range (1106). Subsequently, the next reference picture is determined, and the motion vector is obtained in the same manner. The codec section refers to all reference pictures in the reference list, and determines an optimal motion vector between the reference pictures of this macroblock (1107).

  Then, when the prediction of one macroblock is completed, the codec unit moves to motion prediction of the next macroblock. Thereafter, when the processing is completed for all the macroblocks in the encoding target picture, the inter prediction is ended for the encoding target picture.

  When performing inter prediction, the amount of data read from the frame memory may be data in the search range for performing inter prediction of all macroblocks of the encoding target picture, and does not necessarily read all data of the reference picture. Absent. However, since the search range is usually a position close to the macro block to be encoded, almost all data of the reference picture is searched. That is, the data of all reference pictures in the reference list is read from the frame memory for one picture to be encoded. In addition, the amount of data read from the frame memory at the time of inter prediction is the maximum value (upper limit) of the data amounts of all reference pictures in the frame memory.

  Here, a case where three pictures of pictures B19, B20, and P24 shown in FIG. 5 are encoded will be described as an example for pictures read from the frame memory into the codec section. In this case, the encoding target picture B19 is the next encoding target picture whose reference list is updated by the picture P21 encoded immediately before, and the encoding target picture P24 is scheduled to be added to the reference list next. It is a picture. Therefore, when the pictures B19, B20, and P24 are encoded, there is no change in the reference picture in the reference list as shown in FIG. Each picture B19, B20, and P24 is inter-predicted with reference to the reference pictures P09, P12, I15, P18, and P21 in the reference list unless the distinction between the reference lists List0 and List1 is considered.

  This is schematically shown in FIG. FIG. 12 shows that the processing proceeds with the passage of time in the time axis direction (the direction from the picture B19 to the picture P24). That is, the image on the video memory to be encoded sequentially shifts to pictures B19, B20, and P24. The encoding process of each picture needs to be completed within one frame unit. Here, one frame unit may be considered equivalent to one frame time when pipeline processing is not performed.

  First, the encoding target picture B19 is encoded. The codec unit reads the data of the macro block MBA from the video memory. Further, the codec section sequentially reads out the image data of the search ranges S09A, S12A, S15A, S18A, and S21A of the respective reference pictures P09, P12, I15, P18, and P21 from the frame memory. The codec section performs inter prediction on the search range to obtain one motion vector mvA. This process is performed for all macroblocks of the encoding target picture B19.

  Thereafter, the codec section moves to processing of the next encoding target picture B20. Similarly, for the encoding target picture B20, the codec section reads the data of the macroblock MBB and the image data of the search ranges S09B, S12B, S15B, S18B, and S21B. Then, the codec unit obtains a motion vector mvB by inter prediction in this search range.

  When the inter prediction of all the macroblocks of the encoding target picture B20 is completed, the codec unit next proceeds to processing of the encoding target picture P24. Similarly, for the encoding target picture P24, the codec section reads the data of the macroblock MBC and the image data of the search ranges S09C, S12C, S15C, S18C, and S21C, and obtains the motion vector mvC by inter prediction. When the inter prediction of all the macroblocks of the encoding target picture P24 is completed, the encoding target picture P24 is encoded. The codec unit decodes the encoded picture P24, adds it to the reference list, and updates the reference list.

  As shown in FIG. 12, when one picture to be encoded is processed, image data of five reference pictures are sequentially read from the frame memory. Therefore, when the three encoding target pictures B19, B20, and P24 shown in FIG. 12 are encoded, the image data for a total of 15 pictures is read from each of the five frames.

  If the processing flow mentioned above is put together simply, it will become the flow shown in FIG. That is, after the encoding of the encoding target picture P21 is completed, the reference list is updated, and thereafter the encoding target pictures B19, B20, and P24 are sequentially encoded. After the encoding target picture P24 is encoded, the reference list is updated. Updates are made. The encoding process for each encoding target picture is performed in the order shown in FIG. 11, sequentially performing macroblock reading, search range reading, motion prediction, vector determination, and the like, and sequentially outputs a stream.

  Here, as described above, there are some which encode the image data by pipelining the encoding procedure. Even if a plurality of pictures are processed in a plurality of pipelines, one unit time (frame time) In the inside, all the series of processes for the picture are performed. That is, when inter prediction is performed with one encoding target picture, all reference pictures in the reference list are accessed. Even if pipeline processing that extends over the frame time is performed, if the encoding process of a picture to be added to the reference list is not completed before inter prediction of the next picture is performed, the optimal motion in inter prediction is achieved. The vector cannot be determined. Therefore, even if pipeline processing is performed, each reference picture is read out in units of processing of one frame as shown in FIG. That is, when encoding three encoding target pictures, image data for a total of 15 pictures is read from the frame memory for each encoding target picture within three frame times. . As explained above, H.C. In the general encoding order and encoding method of H.264, the access amount to the frame memory is large.

  Therefore, the encoder in the first embodiment reduces the access amount for reading the reference picture from the frame memory by performing inter prediction of a plurality of encoding target pictures in parallel. For example, when three encoding target pictures B19, B20, and P24 are encoded, three encoding target pictures B19 are read only by reading five pictures instead of reading a total of fifteen reference picture image data. , B20 and P24 are inter-predicted.

An encoding procedure in the encoder according to the first embodiment will be described.
For convenience of explanation, as shown in FIG. 14, the I picture is composed of 15 frames, the P picture is composed of 3 frames, and the B picture is composed of 2 frames. An example of adding to the list will be described.

  FIG. 14 is a diagram for explaining the order of the encoding processing according to the first embodiment. FIG. 14 shows a display order, an encoding order, and an arrangement on the output bit stream related to image data.

  In the present embodiment, the codec unit performs encoding processing of three encoding target pictures in parallel as shown in FIG. For example, three encoding target pictures B19, B20, and P24 are encoded in parallel after encoding the immediately preceding three encoding target pictures B16, B17, and P21. Then, after the encoding process, the encoding target pictures B19, B20, and P24, which are in the normal encoding order, are corrected in order and streamed.

  FIG. 15 is a diagram illustrating a state of the reference list when the encoding target picture is encoded as illustrated in FIG. Immediately before encoding the encoding target pictures B19, B20, and P24, the encoding target pictures B16, B17, and P21 are encoded by the codec unit, and the encoded picture P21 is decoded and added to the reference list. The

  That is, when encoding the encoding target pictures B16, B17, and P21, the reference list List0 includes the reference pictures P06, P09, P12, I15, and P18, and the reference list List1 includes the reference picture P18. After the reference pictures in the reference list are respectively used for inter prediction of the encoding target pictures B16, B17, and P21, the oldest reference picture P06 is deleted from the reference list List0, and the oldest reference picture from the reference list List1. P18 is deleted. In addition, after encoding the encoding target pictures B16, B17, and P21, the encoded picture P21 is decoded and added to each reference list as a reference picture.

  In the encoding process of the encoding target pictures B19, B20, and P24, since the encoding target pictures B19 and B20 are B pictures, the inter predictions use the reference lists List0 and List1. That is, reference is made to a total of five reference pictures, that is, four reference pictures P09, P12, I15, and P18 in the reference list List0 and one reference picture P21 in the reference list List1. Note that the reference picture P21 in the reference list List0 overlaps with the reference list List1, and therefore need not be referred to. In addition, since the encoding target picture P24 is a P picture, only the reference list List0 is used in the inter prediction, and the five reference pictures P09, P12, I15, P18, and P21 are referred to.

  That is, there are five reference pictures P09, P12, I15, P18, and P21 that are referred to in the inter prediction related to the encoding target pictures B19, B20, and P24, and these are stored in the frame memory. . Although there is a difference whether the reference picture P21 becomes the reference list List0 or List1, there is no change in the image data itself of the picture P21, and when the encoding target pictures B19, B20, and P24 are encoded, The number and type of reference pictures are constant. The reference picture in the frame memory is updated next after encoding the encoding target picture P24 and after decoding the encoded picture P24.

  Next, with reference to FIG. 16 and FIG. 17, a search range of each reference picture when performing inter prediction in the present embodiment will be described. A case will be described as an example where inter prediction related to the macroblock MBA of the encoding target picture B19, the macroblock MBB of the encoding target picture B20, and the macroblock MBC of the encoding target picture P24 is performed in parallel.

  In the present embodiment, a common search range is assigned to the reference picture as the search range of each macroblock MBA, MBB, and MCC. For example, the search range S09 is set for the reference picture P09, the search range S12 is set for the reference picture P12, and the search range S15 is set for the reference picture I15. Further, the search range S18 is set for the reference picture P18, and the search range S21 is set for the reference picture P21.

  Here, as shown in FIG. 17, the search range set for the reference picture is set to a range including all search ranges of the macroblocks MBA, MBB, and MCC set by the conventional encoding target picture alone. FIG. 17 is a diagram illustrating an example of the search range S09 set in the reference picture P09.

  When the encoding target pictures B19, B20, and P24 are encoded independently, as described with reference to FIGS. 7 to 9, the search range is set for the reference picture P09 as follows. That is, for reference picture P09, search area S09A is set for macroblock MBA when encoding target picture B19 is encoded, and search range S09B is set for macroblock MBB when encoding target picture B20 is encoded. It had been. In addition, the search range S09C is set for the macroblock MBC when the encoding target picture P24 is encoded.

  The search range in this embodiment includes these search ranges. As shown in FIG. 17, the search range S09 includes the search ranges S09A, S09B, and S09C for the macroblocks MBA, MBB, and MBC. Set. FIG. 17 shows the case where the search ranges S09A, S09B, and S09C are set at completely different positions for the sake of easy understanding. In practice, the three search ranges S09A, S09B, and S09C are close to each other. It is often set to overlap. Therefore, the search range set for the reference picture in this embodiment is not so large compared to the size of the conventional search range (about 32 × 32). The setting of search range S09 shown in FIG. 17 is an example, and setting of search range S09 is arbitrary as long as it includes search ranges S09A, S09B, and S09C.

  Further, the search ranges of the other reference pictures P12, I15, P18, and P21 shown in FIG. 16 are set in the same manner. That is, the search range S12 of the reference picture P12 is set to a range including the search ranges S12A, S12B, and S12C, and the search range S15 of the reference picture I15 is set to a range including the search ranges S15A, S15B, and S15C. In addition, the search range S18 of the reference picture P18 is set to include the search ranges S18A, S18B, and S18C, and the search range S21 of the reference picture P21 is set to include the search ranges S21A, S21B, and S21C.

Next, with reference to FIGS. 18 and 19, the parallel processing operation of inter prediction in the first embodiment will be described.
FIG. 18 is a diagram schematically illustrating a state of parallel processing of inter prediction in the first embodiment. In FIG. 18, a case where three encoding target pictures B19, B20, and P24 are encoded is shown as an example, and in particular, a reading state of data used in inter prediction is shown. In FIG. 18, it is assumed that time elapses in the time axis direction.

  The picture is illustrated from the state immediately after the encoding target picture P21 that has been encoded immediately before is decoded and the reference list is updated. First, the codec unit reads out the data of the macro block MBA of the encoding target picture B19, the data of the macro block MBB of the encoding target picture B20, and the data of the macro block MBC of the encoding target picture P24 from the video memory. Further, the codec unit reads out from the frame memory image data in the search range S09 from the reference picture P09, image data in the search range S12 from the reference picture P12, and image data in the search range S15 from the reference picture I15. Further, the codec section reads image data in the search range S18 from the reference picture P18 and image data in the search range S21 from the reference picture P21 from the frame memory.

  Next, the codec section performs inter prediction related to the macroblock MBA. Specifically, the codec unit performs inter prediction on the search range S09A used for the macroblock MBA in the read search range S09. Further, the codec section performs inter prediction on the search range S12A used for the macroblock MBA in the search range S12 and the search range S15A used for the macroblock MBA in the search range S15. . Similarly, inter prediction is performed for each of the search range S18A used for the macroblock MBA in the search range S18 and the search range S21A used for the macroblock MBA in the search range S21. When the inter prediction in all the search ranges read from each reference picture is completed, the codec unit determines an optimal motion vector mvA for the macroblock MBA.

  Similarly, the codec unit performs inter prediction on the search range used for the macroblock MBB in the read search range for the macroblock MBB. That is, the codec section performs inter prediction on the search range S09B in the search range S09, the search range S12B in the search range S12, and the search range S15B in the search range S15. Further, the codec section performs inter prediction for the search range S18B in the search range S18 and the search range S21B in the search range S21. Then, the codec unit determines an optimal motion vector mvB for the macroblock MBB based on inter prediction in the search range of each reference picture.

  Further, the codec unit performs inter prediction on the search range used for the macroblock MBC among the read search ranges for the macroblock MBC. The codec section includes a search range S09C within the search range S09, a search range S12C within the search range S12, a search range S15C within the search range S15, a search range S18C within the search range S18, and a search range S21C within the search range S21. Perform inter prediction. The codec unit determines an optimal motion vector mvC for the macroblock MBC based on inter prediction in the search range of each reference picture.

  When the optimal motion vectors mvA, mvB, and mvC are determined for each of the macroblocks MBA, MBB, and MBC, the codec unit reads the data of the macroblock next to the encoding target pictures B19, B20, and P24 from the video memory. The codec unit reads image data in the next search range from the frame memory and performs the next inter prediction. If there is a portion that overlaps with the search range that has already been read as the next search range, only the newly required range may be read. By doing so, the amount of data read from the frame memory in the encoding process can be reduced.

  The above process is repeated to perform inter prediction for all the macroblocks of the encoding target pictures B19, B20, and P24, and when the optimal motion vector is obtained, the inter prediction for the encoding target picture is finished, and for each picture The encoding of is finished. When the encoding is completed, the codec unit outputs the encoded data to the stream buffer. At this time, the codec unit outputs data encoded in the order of pictures B19, B20, and P24, which are normal encoding orders. Further, the codec unit decodes the encoded picture P24 and adds it to the reference list, deletes the picture P21 from the reference list, and updates the reference list.

The above operation flow is shown in FIG.
After encoding the encoding target pictures B16, B17, and P21, the encoded picture P21 is decoded and the reference list is updated (1901).

  Thereafter, the codec unit reads the macro block data of each of the encoding target pictures B19, B20, and P24 from the video memory (1902), and also reads the image data of the inter prediction search range corresponding to the data from the frame memory (1903). . The codec section performs inter prediction on the macroblock using the read macroblock data and search range data (1904). Next, when the inter prediction is completed in all the search ranges read from each reference picture, the codec unit determines an optimal motion vector for the macroblock based on the result (1905). Thereafter, the codec section moves to processing of the next macro block of each encoding target picture.

  Then, the codec unit encodes the three encoding target pictures when the processing related to motion prediction is completed for all the macroblocks in the three encoding target pictures. Next, the codec unit outputs a stream in the normal encoding order, that is, in the order of pictures B19, B20, and P24 (1906, 1907, 1908). Subsequently, the codec unit decodes the encoded picture P24, adds it to the reference list, and updates the reference list (1909).

  In the above description, data is read in the order of macroblocks MBA, MBB, and MBC, and image data is read in the order of search ranges S09, S12, S15, S18, and S21. However, since the inter prediction of the encoding target pictures B19, B20, and P24 can be performed independently, the reading order of these data and the like can be processed before and after.

  For example, the data of the macro block MBA is read, the image data of the search range S09 is read to search the search range S09A, and then the image data of the search range S12 is read to search for S12A. Subsequently, the image data of the search range S15 is read to search the search range S15A, then the image data of the search range S18 is read to search the search range S18A, and then the image data of the search range S21 is read to search the search range. S21A is searched to determine a motion vector mvA.

  Subsequently, the data of the macro block MBB is read. The search range S09B within the search range S09, the search range S12B within the search range S12, the search range S15B within the search range S15, the search range S18B within the search range S18, and the search range S21B within the search range S21 are searched. Then, the motion vector mvB is determined.

  Finally, the data of the macro block MBC is read. The search range S09C within the search range S09, the search range S12C within the search range S12, the search range S15C within the search range S15, the search range S18C within the search range S18, and the search range S21C within the search range S21 are searched and moved. It is also possible to determine the vector mvC.

Alternatively, the data of the macroblock MBC may be read first, and the data may be read in the order of the macroblocks MBB and MBA.
What is necessary for the parallel processing described above is that inter prediction of macroblocks of a plurality of encoding target pictures can be processed in parallel, and reading of the search range from the frame memory is common to each reference picture during inter prediction. It is only necessary to read the search range to be performed.

  In the above description, the description has focused on performing inter prediction in parallel. However, encoding processing other than inter prediction, such as entropy encoding, can be originally processed independently between pictures. Also in this embodiment, processing can be performed in parallel.

  Further, in the above description, the case where the I picture or P picture has an interval of 3 frames and a certain period has been described as an example. Here, H. In the H.264 standard itself, it is optional to use a B picture for updating the reference list or not add an I picture or P picture to the reference list. That is, according to the present embodiment, the inter prediction of the current picture to be encoded is performed in parallel between the update of the reference list and the update of the next reference list, and the reference list update interval may be arbitrary. . Normally, as in the present embodiment, it is practical to set an I picture or P picture at a constant cycle and to set the reference list to be periodically updated by this I picture or P picture.

  As described above, in the first embodiment, a plurality of encoding target pictures (for example, the encoding target pictures B19, B20, and the like in the above-described example) from the update of the reference list to the update of the next reference list are described. The inter prediction of P24) is performed in parallel. The processing of each encoding target picture is performed in units of 3 frames when inter prediction of three encoding target pictures is performed in parallel, for example, according to the number of encoding target pictures performing inter prediction in parallel. Just finish.

  In the first embodiment, when the image data of the search range of the reference picture is read from the frame memory, the search range common to each macroblock of the encoding target picture is sequentially read. Furthermore, when performing inter prediction, the amount of data read from the frame memory is the maximum value (upper limit) for all reference pictures in the frame memory. Therefore, according to the first embodiment, for example, when inter prediction of three coding target pictures (B19, B20, P24) is performed in parallel, a maximum of five frames from the frame memory within three frame times. Just read the reference picture data for the minute. In other words, when viewed in one frame time, a maximum of 5/3 sheets of data may be read from the frame memory. This means that the amount of data read from the frame memory is greatly reduced compared to the case where five reference pictures are read for each encoding target picture during one frame time. When the number of encoding target pictures from the update of the reference list to the update of the next reference list is m, and the number of reference pictures in the reference list is p, in the first embodiment, between m picture processing It is only necessary to read data of up to p reference pictures at (m frame time). That is, the maximum access amount to the frame memory is p / m during one frame time.

  Therefore, according to the first embodiment, the encoder that encodes image data using inter prediction, the codec unit (encoding unit) that performs the encoding process, the image data to be encoded, and the reference The amount of access to the image memory that stores pictures can be greatly reduced. As a result, it is not necessary to increase the speed of the clock used for data transfer, and power consumption can be reduced.

(Second Embodiment)
Next, a second embodiment will be described.
H. MPEG1, MPEG2, MPEG4 (SP, ASP) and the like have been standardized as encoding methods for performing inter prediction from before the H.264 method. Here, MPEG2 will be briefly described as an example.

  In the MPEG2 standard, H.264 is used. Similar to the H.264 system, it is composed of an I picture that performs only intra prediction, a P picture that performs only forward prediction, and a B picture that performs bidirectional prediction. However, as shown in FIG. 20, the reference picture used for inter prediction in the MPEG2 standard is limited to only one I picture or P picture coded immediately before when a P picture is coded. In addition, when encoding a B picture, it is limited to one I picture or P picture before and after. The MPEG2 standard does not take the form of a reference list because reference pictures to be referred to are limited, but the point that the encoded image data is decoded and stored in the frame memory, and is read and inter-predicted is H. . This is the same as the H.264 system.

  Even in the MPEG2 standard, the encoding order is H.264. As in FIG. 3, the encoding order is different from the display order and encoding is performed in the order of prediction, and the encoded data is arranged in the encoding order on the output stream. Here, inter prediction in an encoding process compliant with the general MPEG2 standard will be described with reference to FIG.

  FIG. 21 is a diagram illustrating a state of inter prediction when encoding target pictures B19, B20, and P24. FIG. 21 illustrates a state after encoding the immediately previous encoding target picture P21, decoding it, and updating the contents of the frame memory with the reference picture P21. At this time, in the frame memory, a reference picture P21 and a reference picture P18 previously encoded / decoded are stored.

  Reference pictures P18 and P21 are referred to in the inter prediction of the current picture B19, reference pictures P18 and P21 are referred to in the inter prediction of the current picture B20, and reference pictures P21 are referred to in the inter prediction of the current picture P24. To do. For example, for the macro block MBA of the encoding target picture B19, the codec section reads the data of the macro block MBA and the data of the search ranges S18A and S21A of the reference pictures P18 and P21 from the video memory and the frame memory and performs inter prediction. Similarly, the codec unit reads the data of the macro block MBB and the data of the search ranges S18B and S21B of the reference pictures P18 and P21 from the video memory and the frame memory for the macro block MBB of the encoding target picture B20 and performs inter prediction. . Further, the codec section reads the data of the macro block MBC and the data of the search range S21C of the reference picture P21 from the video memory and the frame memory for the macro block MBC of the encoding target picture P24, and performs inter prediction. Then, after the encoding process of the encoding target picture P24, the encoded picture P24 is decoded and stored in the frame memory, and the contents of the frame memory are updated.

FIG. 22 shows a processing flow of the encoding procedure shown in FIG.
After the encoding of the encoding target picture P21 is completed, the contents of the frame memory are updated (2201), and then the encoding target pictures B19, B20, and P24 are sequentially encoded. After encoding of the encoding target picture P24, the frame memory is encoded. Is updated (2205). The encoding process (2202, 2203, 2204) of each encoding target picture B19, B20, P24 is H.264. Similarly to the H.264 system, macroblock reading, search range reading, motion prediction, vector determination, and the like are sequentially performed, and a stream is sequentially output.

  Here, as is clear from the above description, the conventional H.264 encoding method described in the first embodiment is also applicable to the conventional general MPEG2 standard-compliant encoding process. There is a problem similar to the encoding process in the H.264 system. The encoder to which the image processing apparatus according to the second embodiment of the present invention is applied performs inter prediction of a plurality of encoding target pictures in parallel as in the first embodiment, so that a reference picture is obtained from the frame memory. The amount of access for reading is reduced. However, the encoder according to the second embodiment encodes input image data using an image encoding method compliant with the MPEG2 standard.

  In addition, since the structure of the encoder in 2nd Embodiment is the structure similar to 1st Embodiment mentioned above, the description is abbreviate | omitted. However, in the case of MPEG2, no reference list is provided because no reference list is formed. Therefore, the contents of the frame memory are updated, not the reference list.

  An encoding process in the encoder according to the second embodiment will be described with reference to FIGS. 23 and 24. FIG. As described above, in the MPEG2 standard, reference pictures used for inter prediction in encoding processing are limited to I pictures or P pictures. In addition, the I picture or P picture is used as a reference picture by always updating the frame memory after encoding. In the following, after the picture P21 has been encoded, the picture B21 is decoded and the contents of the frame memory are updated, and then the pictures B19, B20, A case where P24 is encoded will be described as an example.

  FIG. 23 is a diagram schematically illustrating a state of parallel processing of inter prediction in the encoding processing according to the second embodiment. As shown in the figure, when encoding the encoding target pictures B19, B20, and P24, a picture P21 that has been encoded and decoded immediately before and a picture P18 that is a previous P picture are included. It is stored in the frame memory and referred to in inter prediction.

  First, the codec unit reads out the data of the macro block MBA of the encoding target picture B19, the data of the macro block MBB of the encoding target picture B20, and the data of the macro block MBC of the encoding target picture P24 from the video memory. Further, the codec section reads image data in the search range S18 from the reference picture P18 and image data in the search range S21 from the reference picture P21 from the frame memory.

  The search ranges S18 and S21 set for each reference picture are set so as to include the search ranges for the macroblocks MBA, MBB, and MBC, as in the first embodiment. Specifically, the search range S18 is set to a range including the search range S18A for the macroblock MBA and the search range S18B for the macroblock MBB. The search range S21 is set to include a search range S21A for the macroblock MBA, a search range S21B for the macroblock MBB, and a search range S21C for the macroblock MBC.

  Next, the codec unit performs inter prediction on the search range S18A used for the macroblock MBA in the read search range S18 for the macroblock MBA. Similarly, the codec unit performs inter prediction on the search range S21A used for the macroblock MBA in the search range S21. When the inter prediction in the search range read from the reference pictures P18 and P21 is completed, the codec unit determines an optimal motion vector mvA for the macroblock MBA.

  Similarly, the codec unit performs inter prediction on the search range used for the macroblock MBB in the read search range for the macroblock MBB. That is, the codec section performs inter prediction on the search range S18B in the search range S18 and the search range S21B in the search range S21, respectively, and determines an optimal motion vector mvB.

  Further, the codec unit performs inter prediction on the search range S21C used for the macroblock MBC in the read search range S21 for the macroblock MBC, and determines the motion vector mvC.

  When the optimal motion vectors mvA, mvB, and mvC are determined for each of the macroblocks MBA, MBB, and MBC, the codec unit reads the data of the macroblock next to the encoding target pictures B19, B20, and P24 from the video memory. The codec unit reads image data in the next search range from the frame memory and performs the next inter prediction. If there is a portion that overlaps with the search range that has already been read as the next search range, only the newly required range may be read. In this way, the amount of data read from the frame memory in the encoding process can be further reduced.

  The above processing is repeated to perform inter prediction for all the macroblocks of the encoding target pictures B19, B20, and P24, and when the optimal motion vector is obtained, the inter prediction for the encoding target picture is completed, The encoding of is finished. When the encoding is completed, the codec unit outputs the encoded data in the order of B19, B20, and P24, which is the normal encoding order, to the stream buffer. Further, the codec unit decodes the encoded picture P24 and stores it in the frame memory, deletes the picture P18 from the frame memory, and updates the contents of the frame memory.

The above operation flow is shown in FIG.
After encoding the encoding target pictures B16, B17, and P21, the encoded picture P21 is decoded and the contents of the frame memory are updated (2401).

  After that, the codec section reads the macroblock data of each of the encoding target pictures B19, B20, and P24 from the video memory (2402), and also reads the image data of the corresponding inter prediction search range from the frame memory (2403). . The codec section performs inter prediction on the macroblock (2404), and determines an optimal motion vector for each (2405). Thereafter, the codec section moves to processing of the next macroblock of each encoding target picture, and similarly performs processing 2402 to 2405.

  Then, the codec unit encodes the three encoding target pictures when the processing related to motion prediction is completed for all the macroblocks in the three encoding target pictures. Next, the codec unit outputs streams in the normal encoding order, that is, in the order of pictures B19, B20, and P24 (2406, 2407, 2408). Subsequently, the codec unit decodes the encoded picture P24, stores it in the frame memory, and updates the contents of the frame memory (2409).

  In the above description, data is read in the order of macroblocks MBA, MBB, and MBC, and image data is read in the order of search ranges S18 and S21. However, since the inter prediction of the encoding target pictures B19, B20, and P24 can be performed independently, the reading order of these data and the like can be processed before and after.

  In the second embodiment, a plurality of encoding target pictures (for example, encoding target picture B19) between the frame memory content update and the next frame memory content update in the encoding process compliant with the MPEG2 standard. , B20, P24) are performed in parallel. Thus, the processing of each encoding target picture is performed when, for example, three encoding target pictures B19, B20, and P24 are inter-predicted according to the number of encoding target pictures to be inter-predicted in parallel. What is necessary is just to complete within the unit of 3 frames.

  In the second embodiment, when the image data of the search range of the reference picture is read from the frame memory, the search range common to the macroblocks of the encoding target picture is sequentially read. Further, when performing inter prediction, the amount of data read from the frame memory is the maximum value (upper limit) for all reference pictures in the frame memory. Therefore, according to the second embodiment, for example, when performing inter prediction of three encoding target pictures B19, B20, and P24 in parallel, a maximum of two images from the frame memory within three frame times. The reference picture data may be read. That is, when viewed in one frame time, a maximum of 2/3 sheets of data may be read from the frame memory. This is because the amount of data read out from the frame memory is much larger than the case where, in the past, one picture was read by two if the picture to be encoded is a B picture and one by one if it is a P picture. It will be reduced to. When the number of encoding target pictures from the update of the contents of the frame memory to the update of the contents of the next frame memory is m and the number of reference pictures in the frame memory is 2, a maximum of two pictures are processed during the processing of every m pictures. It is sufficient to read the reference picture data from the frame memory. That is, in one frame time, the maximum access amount to the frame memory is 2 / m, and the codec unit (encoding unit) that performs the encoding process, the image data to be encoded, and the image that stores the reference picture The amount of access to the memory can be greatly reduced. As a result, it is not necessary to increase the speed of the clock used for data transfer, and power consumption can be reduced.

(Other embodiments of the present invention)
For realizing the functions of the above-described embodiment for a computer (CPU or MPU) in an apparatus or system connected to the various devices so as to operate various devices to realize the functions of the above-described embodiments. Supply software programs. And what was implemented by operating the said various devices according to the program stored in the computer of the system or apparatus is also contained under the category of this invention.
In this case, the software program itself realizes the functions of the above-described embodiments, and the program itself constitutes the present invention. Further, means for supplying the program to the computer, for example, a recording medium storing the program constitutes the present invention. As a recording medium for storing such a program, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.
In addition, such a program is also included in the embodiment of the present invention when the function of the above-described embodiment is realized in cooperation with an operating system running on a computer or other application software. Needless to say.
Further, after the supplied program is stored in a memory provided in a function expansion board or a function expansion unit related to the computer, a CPU or the like provided in the function expansion board or the like based on an instruction of the program may be part of the actual processing Do everything. Needless to say, the present invention includes the case where the functions of the above-described embodiments are realized by the processing.

For example, the image processing apparatuses shown in the first and second embodiments have a computer function 500 as shown in FIG. 25, and the CPU 501 performs the operations in the first and second embodiments.
As shown in FIG. 25, the computer function 500 includes a CPU 501, a ROM 502, and a RAM 503. Also, a keyboard controller (KBC) 505 of a keyboard (KB) 509 and a CRT controller (CRTC) 506 of a CRT display (CRT) 510 as a display unit are provided. Furthermore, a disk controller (DKC) 507 for a hard disk (HD) 511 and a flexible disk (FD) 512 and a network interface card (NIC) 508 are provided. These functional units 501, 502, 503, 505, 506, 507, and 508 are configured to be communicably connected to each other via a system bus 504.
The CPU 501 comprehensively controls each component connected to the system bus 504 by executing software stored in the ROM 502 or the HD 511 or software supplied from the FD 512. That is, the CPU 501 reads out a processing program for performing the above-described operation from the ROM 502, the HD 511, or the FD 512, and executes it, thereby performing control for realizing the operation in the first and second embodiments. Do. The RAM 503 functions as a main memory or work area for the CPU 501.
The KBC 505 controls an instruction input from the KB 509 or a pointing device (not shown). The CRTC 506 controls the display on the CRT 510. The DKC 507 controls access to the HD 511 and the FD 512 that store a boot program, various applications, user files, a network management program, and the processing programs in the first and second embodiments. The NIC 508 bidirectionally exchanges data with other devices on the network 513.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

It is a figure which shows the structural example of the encoder in the 1st Embodiment of this invention. It is a figure which shows the functional structure of the codec part shown in FIG. H. 2 is a diagram for explaining an encoding order of image data in the H.264 format. FIG. It is a figure which shows an example of a reference list. It is a figure which shows the mode of the change of the reference list for every encoding object picture. H. 2 is a diagram for explaining inter prediction of a macroblock in the H.264 scheme. FIG. It is a figure which shows an example of the search range of the macroblock of a encoding object picture, and a reference picture. It is a figure which shows the other example of the search range of the macroblock of a encoding object picture, and a reference picture. It is a figure which shows the other example of the search range of the macroblock of a encoding object picture, and a reference picture. It is a figure for demonstrating the motion of the data in the case of performing inter prediction with the encoder in this embodiment. H. 3 is a flowchart showing a flow of processing related to inter prediction in the H.264 scheme. H. It is a figure which shows the mode of the inter prediction in a H.264 system. H. 3 is a flowchart showing encoding processing in the H.264 format. It is a figure for demonstrating the order of the encoding process in 1st Embodiment. It is a figure which shows the mode of the reference list at the time of encoding an encoding object picture in 1st Embodiment. It is a figure for demonstrating the search range of the reference picture at the time of performing inter prediction in this embodiment. It is a figure which shows an example of the search range of the reference picture in this embodiment. It is a figure which shows typically the mode of the parallel process of the inter prediction in 1st Embodiment. It is a flowchart which shows the flow of the processing operation shown in FIG. It is a figure for demonstrating the image encoding in MPEG2 specification. It is a figure which shows the mode of the inter prediction in the encoding process based on MPEG2 specification. It is a flowchart which shows the encoding process based on MPEG2 specification shown in FIG. It is a figure which shows typically the mode of the parallel process of the inter prediction in the encoding process in 2nd Embodiment. It is a flowchart which shows the flow of the processing operation shown in FIG. It is a block diagram which shows the computer function which can implement | achieve the image processing apparatus in a present Example.

Explanation of symbols

100 encoder 101 codec section (encoding section)
DESCRIPTION OF SYMBOLS 102 Scaling part 103 Motion estimation part 104 Entropy encoding part 105 Local decoding part 111 Video memory 112 Frame memory 113 Stream buffer

Claims (15)

An image processing method for performing motion compensation predictive coding using image data of different frames as a reference picture,
A reference list comprising a plurality of reference pictures, wherein an internal reference picture is updated by a picture obtained by decoding an encoded picture at an arbitrary frame interval;
After the reference list is updated, motion-compensated prediction is performed in parallel for a series of multiple encoding target pictures from the next encoding target picture to the encoding target picture used for the next update of the reference list. An image processing method characterized by performing and encoding.   The image processing method according to claim 1, wherein the pictures to be encoded include an I picture that does not perform motion compensation prediction, a P picture that performs forward prediction, and a B picture that performs bidirectional prediction.   When there are p reference pictures referenced for motion compensation prediction in the reference list and m of the series of encoding target pictures (p and m are arbitrary natural numbers), m codes 3. The image processing method according to claim 1, wherein the amount of data read from the reference list is a maximum of p pictures in order to perform motion compensation prediction of a picture to be processed. The I picture and P picture are periodically set at predetermined intervals.
The reference picture in the reference list is periodically updated by a picture obtained by decoding the encoded picture after encoding the I picture or the P picture.
After the reference list is updated, motion compensation prediction is performed on a series of a plurality of encoding target pictures from a B picture to be encoded next to an I picture or a P picture to be used for updating the reference list next. The image processing method according to claim 2, wherein the image processing is performed in parallel.   When there are p reference pictures referred to for motion compensation prediction in the reference list and the interval between the I picture and P picture is m (p and m are arbitrary natural numbers), m pictures 5. The image processing method according to claim 4, wherein the amount of data read from the reference list is a maximum of p pictures in order to perform motion compensation prediction of the current picture to be encoded. An image processing method for performing motion compensation predictive coding using image data of different frames as a reference picture,
The pictures to be encoded include an I picture that does not perform motion compensation prediction, a P picture that performs forward prediction, and a B picture that performs bidirectional prediction.
The reference picture stored in the frame memory is updated with a picture obtained by decoding the encoded picture after encoding the I picture, the P picture or a part of the B picture,
After the contents of the frame memory are updated, motion compensated prediction is performed for a series of a plurality of encoding target pictures from the next encoding target picture to the encoding target picture used for updating the contents of the frame memory next. An image processing method characterized in that encoding is performed in parallel.   When there are two reference pictures referenced for motion compensation prediction in the frame memory and m of the series of plural encoding target pictures (m is an arbitrary natural number), m encoding targets 7. The image processing method according to claim 6, wherein the amount of data read from the frame memory for performing motion compensation prediction of a picture is a maximum of two pictures. One macroblock for which motion compensation prediction is performed for each encoding target picture is selected, and a search range common to the selected macroblock is read from each reference picture, and motion compensation prediction is performed for each search range of each macroblock. And
After motion compensated prediction of all macroblocks is completed, the next macroblock is selected from each encoding target picture, and image data is read from each reference picture according to the search range common to the next macroblock, 8. The motion compensation prediction is performed in parallel for a plurality of encoding target pictures by sequentially repeating the motion compensation prediction for the search range of each of the following macroblocks. The image processing method according to item. An image processing apparatus using the image processing method according to any one of claims 1, 4, and 6,
Memory means for storing image data of a reference picture;
An image processing apparatus comprising: an encoding unit that performs an encoding process including the motion compensation prediction. An image processing method for performing motion compensation predictive coding using image data of different frames as a reference picture,
A storage step of storing the reference picture in a storage means;
After updating the reference picture stored in the storage means, it corresponds to a plurality of encoding target picture macroblocks from the next encoding target picture to the next encoding target picture stored as a reference picture in the storage means. A step of reading out image data of a common search range from each reference picture stored in the storage unit;
A coding step for performing coding by performing motion compensation prediction in parallel for the macroblocks of the plurality of coding target pictures using the image data in the common search range read in the reading step. An image processing method.   11. The image processing method according to claim 10, wherein the common search range is a range in which each search range for each macroblock of the plurality of encoding target pictures is included.   In the encoding step, motion compensation prediction is performed using image data corresponding to the search range of the macroblock of each encoding target picture among the image data in the common search range read in the reading step. 12. The image processing method according to claim 10 or 11, wherein the image processing method is characterized in that: An image processing apparatus that performs motion compensation prediction coding using image data of different frames as a reference picture,
Storage means for storing the reference picture;
After updating the reference picture stored in the storage means, it corresponds to a plurality of encoding target picture macroblocks from the next encoding target picture to the next encoding target picture stored as a reference picture in the storage means. Reading means for reading out image data of a common search range from each reference picture stored in the storage means;
And coding means for performing motion-compensated prediction in parallel on the macroblocks of the plurality of coding target pictures using the image data in the common search range read by the reading means. An image processing apparatus. A program for causing a computer to execute image processing for performing motion compensation prediction encoding using image data of different frames as a reference picture,
A storage step of storing the reference picture in a storage means;
After updating the reference picture stored in the storage means, it corresponds to a plurality of encoding target picture macroblocks from the next encoding target picture to the next encoding target picture stored as a reference picture in the storage means. A step of reading out image data of a common search range from each reference picture stored in the storage unit;
Using the image data in the common search range read in the reading step, the computer executes a coding step of performing motion-compensated prediction in parallel for the macroblocks of the plurality of coding target pictures. Program for.   15. A computer-readable recording medium on which the program according to claim 14 is recorded.

Images