JP4095664B2 - Video encoding device - Google Patents

Description

The present invention relates to a moving image encoding apparatus for compressing and reducing the amount of moving image data.

  Some moving images are composed of frames such as 720P and others are composed of fields such as 1080i, but the present invention is composed of fields even when applied to those composed of frames. Even if it is applied to things, the purpose can be achieved, and similar actions and effects are produced. Therefore, in the following, the term “picture” is used to indicate one image constituting a moving image. A picture shall represent both a frame and a field.

  MPEG-2 and H.264 In a moving image coding system such as H.264 (also referred to as MPEG-4 AVC) or VC-1, the screen is divided into macroblocks (hereinafter referred to as MB) of 16 pixels × 16 pixels or 8 pixels × 8 pixels. Motion detection, motion compensation, frequency conversion such as DCT or integer conversion, quantization, variable length coding, and the like are performed in MB units. In these moving image encoding systems, in order to improve the encoding efficiency, the result of the encoded peripheral MB is fed back, and adaptive processing for determining the processing content of the MB being encoded (hereinafter referred to as the current MB). Is done.

FIG. 24 shows a block diagram of an MPEG-2 encoder. In MPEG-2, in order to prevent the generated code amount from fluctuating locally, the generated code amount is controlled by changing the quantization parameter of each MB. This is called rate control 92. In the rate control 92, the quantization parameter of the current MB is changed according to the code amount of the encoded MB generated as a result of the variable length encoding 91. For example, when the generated code amount of the left adjacent MB is large, the quantization parameter of the current MB is increased and the generated code amount of the current MB is decreased.

FIG. 2 shows a block diagram of an H.264 encoder. H. H.264 can compress moving images with twice the efficiency of MPEG-2. In order to achieve this high compression efficiency, H.264 employs an increase in the number of reference pictures in motion compensation 100, variable block size motion compensation, and frequency conversion in units of 4 pixels × 4 pixels (hereinafter referred to as integer conversion 93), and in addition to rate control 103, The three adaptive processes are newly introduced. Here, variable block size motion compensation means that MB is divided into sub-blocks (hereinafter referred to as SB) such as 8 pixels × 16 pixels, 8 pixels × 8 pixels, 4 pixels × 4 pixels, etc., and motion compensation is performed for each SB. To do. Hereinafter, when simply referring to a block, both MB and SB are meant.

(1) Considering not only the code amount of a prediction error but also the code amount of a reference picture and the code amount of a motion vector, a combination of a reference picture, a motion vector, and a block that minimizes the total code amount thereof is selected. For the motion vector, a predicted motion vector (hereinafter referred to as PMV) predicted from the motion vectors of the left adjacent block, the upper adjacent block, and the upper right adjacent block is obtained, and the difference between the motion vector of the PMV and the current block is encoded. . In addition, the motion vectors of the left adjacent block, the upper adjacent block, and the upper right adjacent block are also referred to when obtaining the motion vector in the spatial direct mode.

(2) A prediction value is created from the reproduced pixels of the adjacent MB, and intra prediction 97 is performed. As for the luminance signal, there are an intra prediction mode of 4 pixels × 4 pixels SB (hereinafter referred to as 4 × 4 SB) and an intra prediction mode of 16 × 16 pixel blocks. In the 4 × 4SB intra prediction mode of the luminance signal, an intra prediction value of the current 4 × 4SB is generated from the reproduction pixels of the left adjacent 4 × 4SB, the upper adjacent 4 × 4SB, and the upper right adjacent 4 × 4SB, and this intra predicted value And the difference between the current 4 × 4 SB pixels. The difference pixel is subjected to integer transformation / quantization / inverse quantization / inverse integer transformation, and an intra prediction value is added to the result, thereby generating a reproduction pixel. In the 16 × 16 pixel block intra prediction mode, an intra prediction value of the current 16 × 16 pixel block is created from the reproduction pixels of the left adjacent MB and the upper adjacent MB, and the difference between the intra prediction value and the pixel of the current 16 × 16 pixel block is created. Is encoded.

(3) A deblocking filter 98 is applied to the boundary of the 4 pixel × 4 pixel block to remove block noise.

On the other hand, a hardware moving image encoding apparatus employs a pipeline in MB units (referred to as a macroblock pipeline or an MB pipeline in the claims and the specification) in order to increase the operation rate of a circuit. (For example, see Non-Patent Document 1, pp. 126-128, FIG. 3). Further, in a hardware moving image encoding apparatus, encoding is generally performed from the left end MB toward the right end MB in a horizontal MB array (hereinafter referred to as MB row). When the encoding of the rightmost MB included in one MB row is completed, the MBs belonging to the lower MB row are encoded from the left end MB toward the right end MB. This is repeated from the top MB row of the picture to the bottom MB row of the picture. FIG. In the MB pipeline of the H.264 encoder, the manner in which the MB flows through each stage of the MB pipeline is shown. n-1, n, and n + 1 represent the (n-1) th, nth, and n + 1th MBs from the left in a certain MB row, respectively. The MB cycle is a unit of time required for processing one MB in each stage of the MB pipeline. In each stage, one MB is processed in one MB cycle.

In FIG. 26 , data necessary for encoding one MB is read from the external memory over 1 MB cycle in stage 0. Stage 1 performs coarse motion detection for one MB over 1 MB cycle. Detailed motion detection is performed in a narrow range around the coarse motion candidate motion vector detected in stage 1 over 1 MB cycle in stage 2 and intra prediction is performed. In stage 3, integer conversion, quantization, inverse quantization, and inverse integer conversion are performed on one MB over 1 MB cycle. In stage 4, variable length coding and motion compensation are performed for one MB over one MB cycle. Then, stage 1 performs rate control and deblocking filtering on one MB over 1 MB cycle, and stage 6 writes a reproduction pixel or bit stream for one MB over 1 MB cycle to the external memory. Assuming that the MB cycle is 1000 cycles, the encoding process for one MB takes 7000 cycles when the MB pipeline is not performed, whereas in the MB pipeline of FIG. 26 , one MB code is generated every 1000 cycles. The conversion ends.

  However, in order to perform adaptive processing according to the processing result of the left adjacent MB in this MB pipeline, there are the following four problems.

(1) As shown in FIG. 26 , when the rate control is finished for the (n-1) th MB, the quantization is already finished up to the (n + 1) th MB. Therefore, the quantization parameter can be changed according to the generated code amount of the (n−1) th MB for the (n + 2) th and subsequent MBs. That is, 2 MB rate control is delayed.

(2) In FIG. 26 , motion detection is executed in two stages of the MB pipeline, and intra prediction is executed simultaneously with high-precision motion vector detection in stage 2. An accurate PMV cannot be calculated unless a high-precision motion vector is determined and intra prediction or inter prediction is determined, but a high-precision motion vector and intra prediction or inter prediction are determined for the (n-1) th MB. Sometimes the motion vector detection with coarse accuracy in the stage 1 is finished for the nth MB. For this reason, in the MB pipeline of FIG. 26 , it is possible to select a motion vector in consideration of only a rough motion vector detection result for the left adjacent MB, and a high-precision motion vector and intra prediction / A motion vector cannot be selected in consideration of the inter prediction determination result. In addition, H.C. 8 pixels × 16 pixels obtained by dividing the MB at 264, 8 pixels × 8 pixels, 4 it is desirable to perform the motion vector detection in consideration of the amount of codes of the motion vectors even SB pixels × 4 pixels or the like, MB in FIG. 26 In a pipeline, it is difficult to perform motion vector detection in consideration of the code amounts of the motion vectors of the left adjacent SB, the upper adjacent SB, and the upper right adjacent SB.

(3) The 16 pixel × 16 pixel block intra prediction mode is determined by evaluating the pixel value of the left adjacent 16 pixel × 16 pixel block and the upper adjacent 16 pixel × 16 pixel block and the pixel value of the current block. Here, it is desirable that the pixels of the adjacent 16 pixel × 16 pixel block used for evaluation are reproduction pixels. The reproduced pixel is obtained by performing integer conversion, quantization, inverse quantization, and inverse integer conversion on a prediction error obtained by intra prediction or inter prediction, and adding a prediction value. However, in the MB pipeline of FIG. 26 , when the integer conversion of the (n−1) th MB is completed in stage 3 and the reproduction pixel of the (n−1) th MB is created by the motion compensation in stage 4, the n in the stage 2 The intra prediction of the second MB has been completed. For this reason, when the prediction mode determination of 16 pixel × 16 pixel block intra prediction is performed for the nth MB, the reproduction pixel of the (n−1) th MB adjacent to the left cannot be used. For this reason, in the prediction mode determination of the (n−1) th 16-pixel × 16-pixel block adjacent to the left, the pixel of the current picture being encoded must be used instead of the reproduction pixel. Similarly, H.M. In the 4 pixel × 4 pixel block intra prediction mode in H.264, it is desirable to determine the prediction mode using the reproduction pixels of the 4 pixels × 4 pixel block adjacent to the left side, the upper side, and the upper right side, but the MB pipeline of FIG. In this case, it is impossible to use the pixels of the current picture being encoded as a suboptimal measure. For this reason, the intra prediction mode determined by the MB pipeline of FIG. 26 is suboptimal.

(4) Since the same intra prediction value must be used in the encoding device and the decoding device, it must be generated from the reproduced pixels. On the other hand, in the P picture and the B picture, when the n−1th MB adjacent to the left is determined to be in the inter prediction mode, a reproduced pixel cannot be obtained until the motion compensation is completed. However, in the MB pipeline of FIG. 26 , the reproduction pixel generation in stage 4 for the (n-1) th MB and the integer conversion in stage 3 for the nth MB are executed simultaneously. For this reason, in the P picture and the B picture, it is difficult to create a prediction value using the reproduction pixel of the (n-1) th MB and perform integer conversion of the nth MB.

Also, since the amount of picture data is enormous, the picture is generally stored in a memory external to the moving picture coding apparatus. FIG. 27 is a block diagram showing a configuration of a conventional video encoding device (see, for example, p2009 of Non-Patent Document 2 and FIG. 3). The motion vector detector 106 includes an internal memory 104 and a motion detection circuit 105. In the conventional motion vector detector 106, a region where a motion vector may exist (hereinafter referred to as a reference region) is copied from the external memory 4 of the motion vector detector 106 to the internal memory 104 of the motion vector detector 106, An operation for detecting a motion vector is performed on the pixels in the internal memory 104. FIG. 28 shows an example of reference areas 109 and 110 of two adjacent MBs 107 and 108. The reference area 110 of the MB 108 substantially overlaps the reference area 109 of the MB 107, and the width of the non-overlapping area matches the number of pixels in the horizontal direction of the MB. The reference area 110 is only shifted from the reference area 109 to the right by the number of pixels in the horizontal direction of MB. Hereinafter, this shifted area is referred to as an update area 111. The conventional motion vector detector 106 uses this property to reduce the pixel transfer amount between the external memory 4 and the internal memory 104 of the motion vector detector 106. When only the update area 111 is transferred from the external memory 4 to the internal memory 104 and motion detection of the MB 108 is started, the origin of the reference area is changed from (0, 0) of the reference area 109 to (0, 0) ′ of the reference area 110. Replace with. As described above, the transfer amount from the external memory 4 to the internal memory 104 is reduced by reading only the update area 111 and updating the reference area. That is, conventionally, the amount of pixel transfer between the external memory 4 and the internal memory 104 has been reduced by utilizing the horizontal overlap of the reference areas. As a result, the conventional one-chip MPEG-2 codec LSI can encode a 1080i moving image by attaching two 32-bit DDR-SDRAMs operating at 200 MHz (for example, (See Non-Patent Document 3, pp.12-13).

However, the size of one picture according to the digital cinema standard defined by DCI (Digital Cinema Initiative) is 4096 pixels × 2160 pixels, which is four times as many as 1080i (1920 pixels × 1080 pixels). In addition, a CMOS sensor has been developed that outputs a picture having a number of pixels exceeding three times that of 1080i at a rate twice that of 1080i (60 pictures / second) (for example, see section 27.1 of Non-Patent Document 4). If the latest technology is used, it is predicted that a video camera that outputs a moving image of a digital cinema class can be realized. If the conventional technology is used, is a moving image encoding LSI provided with a 256-bit width input terminal? There is a problem that it is necessary to operate the external memory at a speed of about 800 MHz .
Hiroaki Watanabe, Hideaki Chaki, "Developed codec core for both H.264 and MPEG-4, encoded VGA video with 36mA", Nikkei Electronics, September 27, 2004, no.883, pp123-133. Toshihiro Minami and Jiro Naganuma, “Proposal of Motion Vector Detector Configuration Suitable for Telescopic Search”, Theory of Science (D-2), vol.J87-D-2, no.11, pp.2007-2024, Nov .2004. Jiro Naganuma, Hiroe Iwasaki, Takahiro Nitta, Ken Nakamura, Ken Yoshidome, Mitsuo Ogura, Yasuyuki Nakajima, Yutaka Tashiro, Takayuki Onishi, Mitsuro Ikeda, Makoto Endo, Yoshiyuki Yashima, “One-chip HDTV MPEG-2 CODEC LSI Configuration Technology— VASA ", NTT Technical Journal, vol.15, No.9, pp.12-15, September 2003. S. Yoshihara, M. Kikuchi, Y. Ito, Y. Inada, S. Kuramochi, H. Wakabayashi, M. Okano, K. Koseki, H. Kuriyama, J. Inutsuka, A. Tajima, T. Nakajima, Y. Kudoh, F. Koga, Y. Kasagi, S. Watanabe, T. Nomoto, “A 1 / 1.8-inch 6.4MPixel 60frames / s CMOS Image Sensor with Seamless Mode Change”, ISSCC2006, session 27.1, Feb. 2006.

Adaptive processing is performed according to the processing result of the left adjacent MB in the MB pipeline.

In addition, the amount of pixel transfer between the external memory and the reference area memory is greatly reduced.

The moving image encoding apparatus of the present invention starts encoding from the leftmost macroblock in the macroblock row that is a collection of macroblocks arranged in the horizontal direction from the left end to the right end of the picture, A video encoding unit that sequentially encodes adjacent macroblocks;
The video encoding process is divided into at least two processes, including at least two stages for performing the divided processes, including a macroblock pipeline including a stage for motion detection in the stages,
Selecting the macroblock rows as a group with at least two rows that are continuous in the vertical direction and having a number of rows smaller than the total number of the macroblock rows included in the entire picture;
Injecting the macroblock included in the top macroblock row in the group of macroblock rows into the macroblock pipeline;
Macros adjacent on each macroblock row in order from the macroblock included in the second macroblock row from the top to the macroblock included in the bottommost macroblock row in the group of macroblock rows. After the previous input macroblock included in the block row is input to the macroblock pipeline, it is included in each macroblock row and is located to the left of the previous input macroblock, and the previous input macroblock is adjacent to the upper right. A video encoding unit that inputs macroblocks that are not macroblocks into the macroblock pipeline;
The batch of macroblocks that are included in each macroblock row from the top macroblock row to the bottom macroblock row in the batch of macroblock rows and that are successively entered into the macroblock pipeline. A reference area that is referred to when detecting a motion vector for each macroblock that is subject to motion detection for the same number of lines as the number of macroblock lines, and the reference area has a smaller number of pixels than the total number of pixels in one picture. Memorize all included pixels,
Each macroblock which is located on the right side of the reference area and has an update area whose number of pixels in the horizontal direction is the number of pixels corresponding to the number of pixels in the horizontal direction of the macroblock is transferred from an external memory and is the target of motion detection for Exit motion detection, and a memory reference region in which the reference area is updated when performing a new motion detection for the right neighboring macroblocks of the macroblock in the motion detection target.

In the moving image encoding apparatus of the present invention, the moving image encoding process is divided into at least two processes, and at least two stages for performing the divided processes are included, and motion detection is performed in the stages. A macroblock pipeline containing the stage to perform,
The total number of macroblock rows included in the entire picture, which is a macroblock row that is a group of macroblocks arranged in a row in the horizontal direction from the left end to the right end of the picture, and is at least two rows continuous in the vertical direction. Select a smaller number of lines as a group, and for the macroblocks included in the uppermost macroblock line in the group of macroblock lines, select the macroblock for which the lower left adjacent macroblock has not yet been selected For a macroblock included in a macroblock row other than the topmost macroblock row in the group, select the macroblock in which the upper right adjacent macroblock has already been selected to select the macroblock. A macroblock included in one macroblock line is one macroblock line Macroblock selecting unit or selecting one,
For each macroblock selected by the macroblock selector, from the macroblock included in the top macroblock row to the macroblock included in the bottom macroblock row in the group of macroblock rows And sequentially setting data necessary for encoding each macroblock in the macroblock pipeline, and including an encoded data setting unit that inputs the macroblock into the macroblock pipeline,
When the encoded data setting unit sets the data necessary for encoding the macroblock included in the lowest macroblock row in the group of macroblock rows, in the macroblock pipeline, A moving picture encoding unit for the macroblock selection unit to newly select the right adjacent macroblock for each selected macroblock;
The batch of macroblocks that are included in each macroblock row from the top macroblock row to the bottom macroblock row in the batch of macroblock rows and that are successively entered into the macroblock pipeline. A reference area that is referred to when detecting a motion vector for each macroblock that is subject to motion detection for the same number of lines as the number of macroblock lines, and the reference area has a smaller number of pixels than the total number of pixels in one picture. Memorize all included pixels,
Each macroblock which is located on the right side of the reference area and has an update area whose number of pixels in the horizontal direction is the number of pixels corresponding to the number of pixels in the horizontal direction of the macroblock is transferred from an external memory and is the target of motion detection for Exit motion detection, and a memory reference region in which the reference area is updated when performing a new motion detection for the right neighboring macroblocks of the macroblock in the motion detection target.

In the moving image encoding apparatus of the present invention, the moving image encoding process is divided into at least two processes, and at least two stages for performing the divided processes are included, and motion detection is performed in the stages. A macroblock pipeline containing the stage to perform,
The total number of macroblock rows included in the entire picture, which are at least two macroblock rows that are a collection of macroblocks arranged in a row in the horizontal direction from the left end to the right end of the picture, and that are continuous in the vertical direction. Select a smaller number of lines as a group, and for the macroblocks included in the uppermost macroblock line in the group of macroblock lines, select the macroblock for which the lower left adjacent macroblock has not yet been selected For a macroblock included in a macroblock row other than the topmost macroblock row in the group, select the macroblock in which the upper right adjacent macroblock has already been selected to select the macroblock. A macroblock included in one macroblock line is one macroblock line First means for one selected or,
The individual macroblocks selected by the first means are changed from the macroblock included in the uppermost macroblock row to the macroblock included in the lowermost macroblock row in the group of macroblock rows. A second means for injecting into the macroblock pipeline in turn,
When the macroblock included in the lowest macroblock row in the group of macroblock rows is input to the macroblock pipeline by the second means, the first means is selected. A video encoding unit that newly selects the right adjacent macroblock for each macroblock;
The batch of macroblocks that are included in each macroblock row from the top macroblock row to the bottom macroblock row in the batch of macroblock rows and that are successively entered into the macroblock pipeline. A reference area that is referred to when detecting a motion vector for each macroblock that is subject to motion detection for the same number of lines as the number of macroblock lines, and the reference area has a smaller number of pixels than the total number of pixels in one picture. Memorize all included pixels,
Each macroblock which is located on the right side of the reference area and has an update area whose number of pixels in the horizontal direction is the number of pixels corresponding to the number of pixels in the horizontal direction of the macroblock is transferred from an external memory and is the target of motion detection for Exit motion detection, and a memory reference region in which the reference area is updated when performing a new motion detection for the right neighboring macroblocks of the macroblock in the motion detection target.

Preferably, the moving picture coding apparatus according to the present invention starts coding from the uppermost group of macroblock rows of a picture, and sequentially encodes the lowermost group of macroblock rows adjacent to each other.

As described above, according to the present invention, adaptive processing according to the processing result of the left adjacent MB can be performed in the MB pipeline .

In addition, the amount of pixel transfer between the external memory and the reference area memory can be greatly reduced.

  In the present invention, a plurality of MB rows that are vertically continuous are grouped, one MB is selected per MB row from the group of MB rows, and the selected plurality of MBs are one MB. Put continuously into the pipeline.

Hereinafter, a moving picture coding apparatus according to an embodiment of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is an explanatory diagram showing an example of MBs continuously input into the MB pipeline. The picture in FIG. 1 is composed of 12 MB rows from a row to l row, and each MB row is composed of 16 MBs from 0th to 15th. In the picture of FIG. 1, first, MBs included in the a-MB row, b-MB row, c-MB row, and d-MB row are encoded by one MB pipeline, and then the e-MB row, MBs included in the f-MB line, the g-MB line, and the h-MB line are encoded by one MB pipeline, and finally the i-MB line, the j-MB line, the k-MB line, and the l-MB. MB included in a row is encoded by one MB pipeline. The MB of e row 10 column, f row 8 column, g row 6 column, and h row 4 column in FIG. 1 is an example of an MB that is continuously input to one MB pipeline. The MB of f row and 8 columns is 1 MB away from the MB of e row and 10 columns in the left direction. Similarly, the MB of g row 6 column is 1 MB away from the MB of f row 8 column leftward, and the MB of h row 4 column is 1 MB away from the MB of g row 6 column leftward. These four MBs are not adjacent to each other in any of the left neighbor, top neighbor, or top right neighbor. In the following, it is said that these blocks are not adjacent to each other when the plurality of blocks are not adjacent to each other in the left adjacent, the upper adjacent, or the upper right adjacent.

  FIG. 2 shows a block diagram of a moving picture coding apparatus to which the present invention is applied. As shown in FIG. 1, the MB selection circuit 2 collects the e-MB row, the f-MB row, the g-MB row, and the h-MB row, and the MBs belonging to the e-MB row By selecting an MB that has not yet been selected and, for MBs belonging to f-MB, g-MB, and h-MB rows, by selecting an MB for which the upper right neighboring MB has already been selected, The MB of e row 10 column, f row 8 column, g row 6 column, h row 4 column is selected. The encoded data setting circuit 3 sends data necessary for encoding each MB to the MB pipeline 1 in the order of MB of e row 10 column, f row 8 column, g row 6 column, h row 4 column. Set. When data necessary for encoding an MB is set in the MB pipeline 1, the MB is input to the MB pipeline 1. When the data necessary for encoding the MB of h rows and 4 columns is set in the MB pipeline 1 and the MB of h rows and 4 columns is input to the MB pipeline 1, the MB selection circuit 2 A new MB of column, f row 9 column, g row 7 column, h row 5 column is selected.

  FIG. 3 shows a flowchart of the process of the moving picture encoding method to which the present invention is applied. e-MB line, f-MB line, g-MB line, and h-MB line are grouped, and for MBs belonging to the e-MB line, an MB for which the lower left neighboring MB is not yet selected is selected; and For MBs belonging to the f-MB row, the g-MB row, and the h-MB row, by selecting the MB for which the upper right adjacent MB has already been selected, the e row 10 column, the f row 8 column, the g row 6 An MB with 4 columns and h rows is selected (step S1). Next, data necessary for encoding each MB is set in the MB pipeline 1 in the order of MB of e row 10 column, f row 8 column, g row 6 column, h row 4 column, Input into the pipeline (steps S2 and S3). When data necessary for encoding an MB is set in the MB pipeline 1, the MB is input to the MB pipeline 1. When the data necessary for encoding the MB of h rows and 4 columns is set in the MB pipeline 1 and the MB of h rows and 4 columns is input to the MB pipeline 1, the process returns to step S1 (step S4). MBs of e row 11 column, f row 9 column, g row 7 column, h row 5 column are newly selected (step S1).

FIG. 4 shows a state in which each MB flows through each stage of the MB pipeline 1 when these four MBs are continuously input to one MB pipeline 1. Each processing contents of stage 6 from the stage 0 are the same as those in FIG. 26. FIG. 5 shows MB cycle numbers at which the MBs constituting the picture of FIG. 1 are input to the MB pipeline 1 and the stage 0 process is started. As shown in FIG. 4, four MBs of e row 10 column, f row 8 column, g row 6 column, h row 4 column are staged in order when the MB cycle numbers are 104, 105, 106, and 107, respectively. When the MB cycle numbers are 110, 111, 112, and 113, respectively, the stage 6 is reached in order and the encoding is completed. Thus, among a group of four MB rows, the MB belonging to the uppermost MB row, the MB belonging to the second MB row from the top, the MB belonging to the third MB row from the top, the fourth from the top The MBs that belong to the MB row are input to the MB pipeline 1 in the order of MBs, and then the right adjacent MBs are input to the MB pipeline 1 in the same order.

  According to the present invention, when the number of MB rows to be encoded as a group is four as in this embodiment, the processing result of the left adjacent MB can be fed back by three stages and reflected in the processing of the current MB. Therefore, as indicated by an arrow in FIG. 4, for example, the high-precision motion detection result and the intra prediction result of the f-row 7-column MB can be reflected in the coarse-precision motion detection of the f-row 8-column MB. Thus, unlike the conventional technique, the motion vector of the current MB can be selected in consideration of the high-precision motion detection result and intra prediction result of the MB adjacent to the left. Similarly, as indicated by an arrow, the result of motion compensation of the MB of f rows and 7 columns can be reflected in the intra prediction of MBs of f rows and 8 columns. Therefore, unlike the conventional technique, intra prediction of the current MB can be performed using the reconstructed pixel obtained by motion compensation of the MB adjacent to the left. Further, the rate control result of the MB of f rows and 7 columns can be reflected in the quantization of MBs of f rows and 8 columns. In this way, the quantization parameter of the current MB can be corrected according to the rate control result of the MB adjacent to the left.

  Here, in the present embodiment, the case where the MB included in the MB row adjacent to the lower side is 1 MB away from the MB in the upper MB row in the left direction is shown. However, the object of the present invention can be achieved, and the same action and effect are produced. Also, taking four MB rows that are vertically continuous as one group, one MB per MB row is selected from these four MB rows, and these four MBs are selected as one MB pipeline. However, it is sufficient that the number of MB lines in a group is two or more. For example, eight MB lines that are vertically continuous are grouped, and one of these eight MB lines is used as one group. Even if one MB is selected per MB row, and the selected 8 MBs are continuously input to one MB pipeline, the object of the present invention can be achieved, and similar operations and effects can be achieved. Produce. However, when the number of MB rows in a group is 2, the processing result of the left adjacent MB can be fed back to the current MB processing for only one stage. Therefore, in the stage configuration of FIG. Cannot be fed back to the current MB intra prediction and integer transform / quantization / inverse quantization / inverse integer transform. When the number of MB lines in a group is 2, it is necessary to change the stage configuration from that shown in FIG.

(Second Embodiment)
FIG. 6 shows an MB divided into four SBs of 8 pixels × 8 pixels (hereinafter referred to as 8 × 8SB). In the present embodiment, an example in which a motion vector that minimizes the sum of the cost of the motion vector and the cost of the prediction error is detected for four 8 × 8 SBs will be described. Generally, it is estimated that the cost of a motion vector is higher as the distance between the PMV and the motion vector is longer. Therefore, before starting motion detection for each 8 × 8 SB, the 8 × 8 SB PMV must be determined. For example, if PMV is determined based on the motion vector of the left adjacent 8 × 8SB, the upper adjacent 8 × 8SB, and the upper right adjacent 8 × 8SB, the numbers 0, 1, 2, and 3 in FIG. The motion vector for each 8 × 8 SB must be detected in order. However, since the 8 × 8SB motion vector adjacent to the upper right side does not exist when the third 8 × 8SB motion vector is detected, the third 8 × 8SB PMV is the first and second 8 × 8SB motion vectors MV1 and MV2. It is decided based on.

  FIG. 7 is an explanatory diagram showing an example of MBs continuously input into the MB pipeline. The picture in FIG. 7 starts from the a line, and there is an MB line below the r line. There are also MBs after the 16th column. In this embodiment, the i-MB line, j-MB line, k-MB line, l-MB line, m-MB line, n-MB line, o-MB line, and p-MB line are taken as a group. Select MB of i rows, 15 columns, j rows, 13 columns, k rows, 11 columns, l rows, 9 columns, m rows, 7 columns, n rows, 5 columns, o rows, 3 columns, and p rows, 1 column from a set of MB rows. An example is shown in which these eight selected MBs are continuously input into one MB pipeline.

  FIG. 8 shows how each MB flows through each stage when MBs included in 8 MB rows are continuously input to one MB pipeline in order to detect a motion vector every 8 × 8 SB. Show. FIG. 9 shows a block diagram of a motion vector detection circuit 12 to which the present invention is applied. The external memory 4 stores an image with single pixel accuracy. At stage 0, an image is read from the external memory 4. The image is reduced by the two-pixel precision image creation circuit 5 to create a two-pixel precision image and written in the two-pixel precision memory 6. While reading an image from the two-pixel accuracy memory 6 at stage 1, the two-pixel accuracy motion detection circuit 7 detects a candidate motion vector with two-pixel accuracy. Then, in stage 2, the area around the candidate vector is read from the external memory 4 and written to the single pixel precision memory 8. Here, since an external large-capacity memory is generally low-speed, one stage is allocated for transfer from the external memory 4 to the single pixel precision memory 8. While reading the area around the two-pixel accuracy candidate vector from the single-pixel accuracy memory 8, the motion vector is detected by the single-pixel accuracy motion detection circuit 9 in the stage 3. Thereafter, the single pixel precision pixel is read from the single pixel precision memory 8 at the stage 4, and the quarter pixel precision motion vector detection circuit 11 is created while creating the quarter pixel precision image by the quarter pixel precision image creation circuit 10. Is used to perform 1/4 pixel precision motion vector detection. Since the single pixel accuracy memory 8 supplies a single pixel accuracy image to the single pixel accuracy motion detection circuit 9 and the ¼ pixel accuracy image creation circuit 10, the single pixel accuracy memory 8 has a double buffer configuration or more than two reads. It must be possible multi-port memory. In FIG. 8, the surrounding of the single-pixel accuracy motion vector is directly searched at stage 4 to detect the 1 / 4-pixel accuracy motion vector. However, after detecting the half-pixel accuracy motion vector at stage 4, A ¼ pixel accuracy motion vector may be detected by searching around the pixel accuracy motion vector. Alternatively, single-pixel precision motion vector detection and half-pixel precision motion vector detection may be performed in stage 3, and quarter-pixel precision motion vector detection may be performed in stage 4.

  In stages 1 to 5, the MB cycle is divided into four SB cycles, and each 8 × 8 SB process is performed in each SB cycle. Here, the SB cycle means a unit of time required for processing one SB, and when performing processing for each SB, the MB cycle is divided into SB cycles. Hereinafter, the stage group in which the MB cycle is divided into SB cycles and processing for each SB is performed is referred to as an SB pipeline (or sub-block pipeline). Stages 1 to 5 in FIG. 8 are SB pipelines. FIG. 10 shows the detailed operation of the SB pipeline. −0, −1, −2 and −3 mean 8 × 8SB numbered 0, 1, 2, 3 in FIG. For example, i15-2 represents the second 8 × 8SB included in the MB of i rows and 15 columns. In FIG. 10, an empty stage is inserted into the stage 5, and for example, a ¼ pixel precision motion vector detected in the stage 4 for i15-0SB can be reflected in the cost calculation of the 2-pixel precision motion vector in the stage 1 of i15-1SB. I am doing so. As described above, when the MB cycle is divided into the number of SB cycles equal to the number of SBs, the processing result of the previous 8 × 8 SB in the same MB is obtained by setting a pipeline of (SB number + 1) stages. Of 8 × 8SB.

  In stage 1 to stage 5, it is possible to provide a motion vector detection circuit for each MB that operates in parallel with a motion vector detection circuit for each 8 × 8 SB. Further, in stages 3 to 5, in the peripheral region of the candidate vectors for each 8 × 8SB detected in stage 1, motion vector detection for each 8 × 8SB is performed in parallel with the motion vector detection for each 8 × 8SB, for each 16 × 8 pixels, for each 8 × It is also possible to detect a motion vector for every 16 pixels SB.

  After motion vectors are detected in stages 0 to 5, a prediction mode with the highest coding efficiency is selected from 16 × 16 intra prediction mode, 4 × 4 intra prediction mode, inter prediction mode, and the like in stage 6. The In stage 7, integer transformation, quantization, inverse quantization, and inverse integer transformation are performed. In stage 8, variable length coding and motion compensation are performed. In stage 9, rate control and deblocking filter processing are performed. In stage 10, the reproduced pixels and the encoded bit stream are written in the external memory 4.

(Third embodiment)
In this embodiment, two 1/4 pixel precision motion detection circuits 11 are provided, and the stage 3 and the stage 4 are combined to perform one 8 × 8SB 1/4 pixel precision motion detection over 2SB cycles. Indicates. Except for stage 3 and stage 4, the stage configuration is the same as that of the MB pipeline of the second embodiment. FIG. 11 shows the detailed operation of the SB pipeline, and FIG. 12 shows a block diagram of the motion detection circuit 13 for realizing this operation. By operating the two quarter-pixel precision motion detection circuits 11 with a shift of 1 SB cycle, individual 8 × 8 SB quarter-pixel precision motion detection can be performed over 2 SB cycles.

(Fourth embodiment)
In this embodiment, H.264 is used. An example in which the present invention is applied to 4 pixel × 4 pixel intra prediction (hereinafter referred to as 4 × 4 intra prediction) in H.264 will be described. FIG. 13 shows an MB divided into 16 4 pixels × 4 pixels SB (hereinafter referred to as 4 × 4 SB). 4 × 4SB is encoded in the order of 0 to 15 in FIG. 13 and converted into a bit stream. For example, when 4 × 4 intra prediction is performed for the third 4 × 4SB, the pixels in the lowermost 4 rows of the first 4 × 4SB and the pixels in the fourth rightmost 4 columns of the second 4 × 4SB are referred to. In addition, when performing 4 × 4 intra prediction for the 14th 4 × 4SB, the pixels at the bottom four rows in the 12th and thirteenth 4 × 4SB and the pixels at the rightmost four columns in the 11th 4 × 4SB are referred to. The As the pixels belonging to these adjacent 4 × 4 SBs, it is desirable to use reproduction pixels.

  FIG. 14 shows a state where each MB flows through each stage when MBs included in eight MB rows are continuously input to one MB pipeline for 4 × 4 intra prediction. Stage 4 performs 4 × 4 intra prediction using adjacent 4 × 4 SB reconstructed pixels, and stage 5 performs integer transformation, quantization, inverse integer transformation, and inverse quantization on the prediction error, and adds the intra prediction value Then, a reproduction pixel is generated. The 4 × 4 intra prediction circuit of stage 4 and the integer transform / quantization / inverse integer transform / inverse quantization circuit of stage 5 are the movements of stage 1 to stage 5 shown in the second embodiment or the third embodiment. Operates in parallel with the detection circuit.

  In this embodiment, the MB cycle is divided into 16 SB cycles in stages 4 and 5. For this reason, if the number of SBs is larger than the number of stages in the SB pipeline, and the number of SB pipelines is (SB number + 1), the number of empty stages increases, resulting in poor efficiency. FIG. 15 shows the detailed operation of the SB pipeline. −0, −1, −2,..., -15 means 4 × 4SB numbered 0, 1, 2,. For example, i15-8 represents the 8th 4 × 4SB included in the MB of i rows and 15 columns. As shown in FIG. 15, in this embodiment, by processing 4 × 4 SB included in two MBs alternately, one MB is processed over 2 MB cycles, that is, 32 SB cycles. When 4 × 4 SB belonging to one MB is subjected to 4 × 4 intra prediction, 4 × 4 SB belonging to another MB is simultaneously subjected to integer transform / quantization / inverse integer transform / inverse quantization. The processing of one MB is started every MB cycle, and the processing of another MB is finished, and matching with the other stages of the MB pipeline is performed. However, when operated in this way, as shown in FIG. 15, in the 4 × 4 intra prediction of stage 4, a deviation from the MB cycle occurs by 1 SB cycle, that is, 1/16 MB cycle every 2 MB cycles. Therefore, in FIG. 15, the effective period of stage 3 is set to 15/16 MB cycle, and the shift of 1/16 MB cycle in stage 4 is absorbed. When the stage 3 process is heavy and an invalid period cannot be provided, the valid period of the stage 6 may be set to 15/16 MB cycle, and the shift of 1/16 MB cycle may be absorbed. By operating in this way, the 4 × 4 intra prediction circuit and the 4 × 4 SB of the left adjacent, the upper adjacent, and the upper right adjacent are generated without generating a sky in the circuits that perform integer transform / quantization / inverse integer transform / inverse quantization. 4 × 4 intra prediction can be performed using the reconstructed pixels for the pixels belonging to.

  It should be noted that the MB pipeline shift can be absorbed by inserting an empty stage before or after the SB pipeline, instead of shortening the effective period of the stage before or after the SB pipeline.

  In the present embodiment, as indicated by arrows in FIG. 14, the rate control result of the left adjacent MB can be reflected in quantization performed in association with 4 × 4 intra prediction.

  Further, 16 pixel × 16 pixel intra prediction of the luminance signal and 8 pixel × 8 pixel intra prediction of the color difference signal in the 4: 2: 0 format are processed for each MB. For this reason, it is only necessary to refer to pixels belonging to the left adjacent MB and the upper adjacent MB. In this case, since the reconstructed pixel obtained by the motion compensation in stage 8 can be referred to, there is no need to perform integer transformation / quantization / inverse integer transformation / inverse quantization in stage 5 as in 4 × 4 intra prediction. When the 4 × 4 intra prediction mode is selected in the intra / inter determination in stage 6, the reproduced pixels obtained by the integer transform, quantization, inverse integer transform, and inverse quantization in stage 5 are stored. Alternatively, the stage 7 may be configured to perform integer transformation / quantization / inverse integer transformation / inverse quantization again in the stage 7.

  If the mode determination of 4 × 4 intra prediction is performed by the input pixels, the integer transformation / quantization / inverse integer transformation / inverse quantization in stage 5 can be omitted. However, the image quality is deteriorated as compared with the case where the mode is determined using the reproduction pixels.

(Fifth embodiment)
In the present embodiment, in the MB pipeline having substantially the same stage configuration as the second to fourth embodiments, the i-MB row, j-MB row, k-MB row, and l-MB row in FIG. Are selected, and MBs such as i rows 15 columns, j rows 13 columns, k rows 11 columns, l rows 9 columns, etc. are selected from these MB rows, and one of these four selected MBs is selected. An example of continuously charging to the MB pipeline is shown. In the second to fourth embodiments, since 8 MB rows are processed as a group, the processing result of each left adjacent MB can be fed back to the processing of 7 stages of MB. In this embodiment, 3 stages I can only give feedback. For this reason, when the processing result of the left adjacent MB is reflected in the processing of the current MB, there is a restriction compared to the second to fourth embodiments.

  When MBs included in four MB rows are continuously input to one MB pipeline in order to detect a motion vector for each 8 × 8 SB, each MB flows through each stage of the MB pipeline. Is shown in FIG. In this case, for example, when the intra / inter determination of the MB of j rows and 12 columns ends in stage 5, the 2-pixel precision motion detection of the MB of j rows and 13 columns ends simultaneously in stage 1. For this reason, there is a limitation that the intra / inter determination result of the left adjacent MB cannot be reflected in the two-pixel precision motion detection of the current MB. However, as indicated by arrows in FIG. 16, the intra / inter determination result of the left adjacent MB can be reflected in the stage 2 and subsequent stages. Also, the intra / inter determination result determined in stage 5 for the upper left adjacent MB, the upper adjacent MB, the upper right adjacent MB, the MB one left in the left direction, and the like can be reflected in the two-pixel accuracy motion detection in stage 1. Therefore, in stage 1, a temporary PMV is created with reference to motion vectors such as the upper left adjacent MB, the upper adjacent MB, the upper right adjacent MB, and the MB separated by one in the left direction, and all four 8 × 8 SBs are common. The two-pixel precision motion detection is performed with reference to the temporary PMV. In stage 2, the peripheral area of the candidate motion vector of 2-pixel accuracy detected in stage 1 is set in the single-pixel accuracy memory 8, a true PMV is created in stage 3, and a true PMV is created in stage 3 and stage 4. The single-pixel precision motion detection and the quarter-pixel precision motion detection are performed with reference to FIG.

  FIG. 17 shows the detailed operation of the SB pipeline in stages 3 and 4. In single-pixel precision motion detection of stage 3 and quarter-pixel precision motion detection of stage 4, by processing 8 × 8SB included in two MBs alternately, one MB takes 2 MB cycles, that is, 8SB cycles. Process. When single pixel precision motion detection is performed for 8 × 8SB belonging to one MB, ¼ pixel precision motion detection is simultaneously performed for 8 × 8SB belonging to another MB. The processing of one MB is started for each MB cycle, and the processing is performed so as to finish the processing of another MB, so as to be consistent with the other stages of the MB pipeline. Then, the effective period of stage 2 is set to 3/4 MB cycle, and the shift of 1/4 MB cycle in stage 4 is absorbed. According to this method, four 8 × 8 SBs included in one MB can be serially processed from No. 0 to No. 3. For this reason, in the single-pixel precision motion detection and the quarter-pixel precision motion detection, it is possible to perform motion detection based on a true PMV in consideration of the adjacent 8 × 8 SB motion vector.

  However, in this embodiment, evaluation of the spatial direct address is required separately. In stage 1, 2-pixel precision motion detection is performed based on the temporary PMV, and in stage 2, the peripheral area of the candidate motion vector detected in stage 1 is set in the single-pixel precision memory 8. When the temporary PMV of stage 1 and the true PMV of stage 3 are different, the motion detection areas in stages 3 and 4 do not include MB or 8 × 8 SB at the position corresponding to the spatial direct address. Therefore, it is necessary to perform processing for evaluating the MB of the position corresponding to the spatial direct address and the cost of 8 × 8 SB in parallel with stage 3 and stage 4.

  FIG. 18 shows how each MB flows through each stage of the MB pipeline when MBs included in four MB rows are continuously input to one MB pipeline for 4 × 4 intra prediction. Show. FIG. 19 shows the detailed operation of the SB pipeline in stages 3 and 4. In the 4 × 4 intra prediction in stage 4 and the integer transform / quantization / inverse quantization / inverse integer transform in stage 5, the same processing as in the SB pipeline of the fourth embodiment is performed.

  However, in this embodiment, the processing result of the left adjacent MB can be fed back to the current MB processing only for three stages. Therefore, when motion compensation is performed in stage 7, 4 × 4 intra prediction is performed using the reproduced pixels of the left adjacent MB. I can't. Therefore, there is a limitation that motion compensation must be performed at stage 6. The rate control result of stage 8 of the left adjacent MB can be reflected after stage 5 and cannot be reflected in the quantization of stage 4. However, there are cases where a large amount of bits are generated by variable length coding in stage 7 and it is necessary to perform rate control to increase the quantization parameter in stage 8. In this case, the 4 × 4 intra prediction mode is not selected in the intra / inter determination of stage 5, or when the 4 × 4 intra prediction mode is selected, the new quantization parameter is used in stage 6 It is necessary to redo integer conversion, quantization, inverse quantization, and inverse integer conversion.

  As shown in the first to fifth embodiments, in the present invention, a plurality of MB rows that are vertically continuous are grouped and one per MB row is selected from the group of MB rows. MBs are selected, and the selected MBs are continuously input to one MB pipeline. Since these MBs are in a narrow image area, the image encoding device is used to display the reproduction pixels, motion vectors, and the like of the upper adjacent MB and the upper right adjacent MB until the encoding of the MBs belonging to the adjacent MB row below ends. Even if it is held inside, the scale of the memory circuit required for the holding is small. By providing this memory circuit, except for the MB included in the uppermost MB row of the plurality of MB rows, reproduced pixels, motion vectors, etc. of the upper adjacent MB and the upper right adjacent MB for encoding the current MB Can be solved.

  In addition, the stage configuration of the MB pipeline shown in the first to fifth embodiments is an example, and other stage configurations may be used. If the stage configuration changes, the constraints in motion detection, intra prediction, and the like change even if the number of vertically continuous MB rows encoded as a group is the same.

(Sixth embodiment)
FIG. 20 is a block diagram of a video encoding apparatus having an internal memory. The moving image encoding device 23 reads all data necessary for MB encoding from the external memory 4 and stores the data in the internal memory. There are two types of internal memory: a reference area memory 14 and a buffer memory 15. The reference image to be referred to at the time of motion detection is stored in the reference area memory 14 and is supplied from the reference area memory 14 to the motion detection unit 16 and the motion compensation unit 17. Other data necessary for encoding is stored in the buffer memory 15, and includes a motion detection unit 16, a motion compensation unit 17, an intra prediction unit 18, an integer transform / quantization / inverse quantization / inverse integer transform unit 19, a variable The long encoding unit 20 and the deblocking filter unit 21 transfer data to and from the buffer memory 15. The control unit 22 performs intra / inter determination and rate control. The control unit 22 is configured by a wired logic or a microprocessor. In FIG. 20, the reference area memory 14 and the buffer memory 15 are shown as separate ones. However, if the reference area address and the buffer address are separated, one common memory may be used.

  FIG. 21 is an explanatory diagram showing a reference area when four MBs included in four vertically continuous MB rows are processed by one MB pipeline. Reference areas 28, 29, 30, and 31 are areas that are referred to when motions of MBs 24, 25, 26, and 27 are detected, respectively. The reference areas 28, 29, 30, and 31 are greatly overlapped, and the height of the non-overlapping portions in the reference areas of two MBs belonging to the vertically adjacent MB rows is equal to the number of pixels in the vertical direction of the MB. This overlapping portion becomes larger as the motion vector detection range becomes wider. In the present embodiment, all the pixels in the reference areas 28, 29, 30 and 31 are stored in the reference area memory 14. As in the prior art, in this embodiment, when encoding of the MB adjacent in the right direction is started, only the update areas 32, 33, 34, and 35 are transferred from the external memory 4 to the reference area memory 14, and the code is processed up to that point. The origin of the reference area 28, 29, 30, 31 of the left adjacent MB that has been converted is changed to a new origin for the right adjacent MB to be encoded next. However, the update area 33 for the MB 25 is only a portion not included in the reference area 28 of the MB 24. Similarly, the update areas 34 and 35 of MB26 and MB27 are only portions not included in the reference areas 29 and 30 of MB25 and MB26, respectively. Therefore, the amount of pixel transfer between the external memory 4 and the reference area memory 14 can be greatly reduced as compared with the prior art.

  FIG. 22 is an explanatory diagram showing another method of storing the reference area in the reference area memory. The reference area memory 14 stores a rectangular area 36 that includes all the reference areas 28, 29, 30, and 31. According to this method, the address generation of the reference area memory 14 is simplified. Further, since the update area 37 is also a group of rectangular areas, transfer of pixels from the external memory 4 to the reference area memory 14 is simplified.

  In the above, the case where four MBs included in four consecutive MB rows are simultaneously processed by one MB pipeline has been shown as an example. However, the method shown in the present embodiment may be performed on two or more consecutive MB rows. For example, it may be eight consecutive MB lines.

  The reference region memory 14 stores only a single pixel accuracy image, and the single pixel accuracy image read out from the reference region memory 14 is used to generate a half pixel accuracy image or ¼ inside the motion detection unit 16 and the motion compensation unit 17. It can be set as the structure which produces a pixel precision image. In this configuration, a half-pixel precision image or a quarter-pixel precision image is created in advance and stored in the external memory 4, and the half-pixel precision image or the quarter-pixel precision image is transferred from the external memory 4 to the reference area memory 14. The pixel transfer amount between the external memory 4 and the reference area memory 14 can be reduced as compared with the configuration to be performed.

  Further, there is a hierarchical search method in which a reduced image such as a two-pixel accuracy image is created, motion is first detected on the reduced image, and motion detection is performed on the high-accuracy image step by step. Even when the present invention is applied to this search method, only the single-pixel precision image is stored in the reference area memory 14, and a reduced image is created inside the motion detection unit 16 and the motion compensation unit 17. It can be configured. This configuration creates a reduced image in advance and stores it in the external memory 4, and transfers pixels between the external memory 4 and the reference region memory 14 rather than a configuration in which the reduced image is transferred from the external memory 4 to the reference region memory 14. The amount can be reduced.

  In addition, a telescopic search method has been proposed as a method for reducing the amount of calculation in motion detection. In the telescopic search method, an intermediate image between the encoding target image and the reference image is also required. In this case, the update method shown in the present embodiment can be applied to a region necessary for motion detection in an intermediate image.

  Further, only the luminance signal may be transferred from the external memory 4 to the internal memory of the image encoding device 23 by the method shown in this embodiment, or the method shown in this embodiment is applied to both the luminance signal and the color difference signal. May be. If the color difference signal is not referred to during motion detection, the color difference signal may be stored in either the reference area memory 14 or the buffer memory 15.

(Seventh embodiment)
When HDTV video is to be encoded, the motion vector range needs to be about ± 200 pixels horizontally and ± 100 pixels vertically to obtain good image quality. The amount of calculation for detecting a motion vector in this range is enormous, and a large number of cycles are required for motion detection. However, in a moving picture encoding apparatus having an MB pipeline structure, the number of cycles per MB cycle is common to all stages. Only the stage for motion detection cannot increase the number of cycles per MB cycle. Also, in the MB pipeline to which the present invention is applied, there is a limit to the number of stages where the processing result can be fed back, so it is not preferable to increase the number of stages in the MB pipeline.

  FIG. 23 shows a combination of a motion detection device 38 capable of detecting motion in a wide range and a moving image encoding device 23 including an MB pipeline to which the present invention is applied. The motion detection device 38 detects motion vector candidates in consideration of only the encoding cost of the prediction error without considering the encoding cost of the motion vector. The video encoding device 23 receives the motion vector candidate detected by the motion detection device 38 and performs motion detection in consideration of the motion vector encoding cost for the peripheral region of the motion vector candidate. That is, motion is detected without referring to the PMV over a wide range, and motion detection is performed again with reference to the PMV for the peripheral region of the detected motion vector candidate. The motion detection again in the peripheral region of the received motion vector candidate can be performed at a motion detection stage of any accuracy such as 2-pixel accuracy motion detection or single pixel accuracy motion detection. In the moving image encoding device 23, in addition to the peripheral region of the motion vector candidate, motion detection may be performed simultaneously in another region where there is a high possibility that a motion vector with the minimum cost exists, for example, the peripheral region of the PMV. Since the motion detection device 38 operates independently of the MB pipeline of the video encoding device 23, motion detection can be performed over many cycles.

  In addition, the motion detection device 38 may be configured to perform motion detection on an input image that has not been encoded, but is not a reproduced image created by the video encoding device 23. With this configuration, it is not necessary to send a reproduced image from the moving image encoding device 23 to the motion detection device 38.

  It is possible to integrate the motion detection device 38 and the moving image encoding device 23 on the same LSI. In this case, the external memory 4 connected to the video encoding device 23 is a shared memory of the motion detection device 38 and the video encoding device 23, and the motion detection device 38 performs motion detection on the reproduced image. Also good.

Also, MPEG-2 to H.264. When the already detected motion vector can be used as in the case of transcoding to H.264, the detected motion vector is directly input to the moving image encoding device 23 without using the motion detecting device 38, and this detected motion is detected. It is also possible to adopt a configuration in which motion detection is performed in consideration of the coding cost of the motion vector for the peripheral region of the vector .

It is explanatory drawing which showed the example of MB thrown continuously into MB pipeline. It is a block diagram of the moving image encoder to which the present invention is applied. It is a flowchart of a process of the moving image encoding method to which this invention is applied. FIG. 2 is an explanatory diagram showing a state in which each MB flows through each stage of the MB pipeline when the four MBs exemplified in FIG. 1 are continuously input into one MB pipeline. FIG. 2 is an explanatory diagram showing MB cycle numbers at which the MBs constituting the picture of FIG. 1 are input to the MB pipeline and stage 0 processing is started. It is explanatory drawing which showed MB divided | segmented into 4 SB of 8 pixels x 8 pixels. It is explanatory drawing which showed the example of MB thrown continuously into MB pipeline. In order to detect a motion vector for every 8 pixels × 8 pixels SB, when MBs included in 8 MB rows are continuously input to one MB pipeline, each MB flows through each stage. It is explanatory drawing shown. It is a block diagram of a motion vector detection circuit to which the present invention is applied. It is explanatory drawing which showed the detailed operation | movement of SB pipeline for detecting the motion vector for every 8 pixel x 8 pixel SB. It is explanatory drawing which showed the detailed operation | movement of SB pipeline for detecting the motion vector for every 8 pixel x 8 pixel SB. It is a block diagram of a motion vector detection circuit to which the present invention is applied. It is explanatory drawing which showed MB divided | segmented into 16 4 pixels x 4 pixels SB. An explanatory diagram showing how each MB flows through each stage when MBs included in 8 MB rows are continuously input to one MB pipeline for 4 pixel × 4 pixel intra prediction. is there. It is explanatory drawing which showed the detailed operation | movement of SB pipeline for 4 pixel x 4 pixel intra prediction. When MBs included in four MB rows are continuously input to one MB pipeline in order to detect a motion vector for each 8 × 8 SB, each MB flows through each stage of the MB pipeline. It is explanatory drawing which showed. It is explanatory drawing which showed the detailed operation | movement of SB pipeline for detecting the motion vector for every 8 pixel x 8 pixel SB. An explanatory diagram showing how each MB flows through each stage when MBs included in four MB rows are continuously input to one MB pipeline for 4 pixel × 4 pixel intra prediction. is there. It is explanatory drawing which showed the detailed operation | movement of SB pipeline for 4 pixel x 4 pixel intra prediction. It is a block diagram of the moving image encoder which has an internal memory. It is explanatory drawing which showed the reference area | region in the case of processing 4 MB contained in 4 vertical MB lines by one MB pipeline. It is explanatory drawing which showed another method which memorize | stores a reference area in the memory for reference areas. It is explanatory drawing which showed the combination of the motion detection apparatus which can detect a motion in a wide range, and the moving image encoding apparatus containing MB pipeline to which this invention was applied. It is a block diagram of an MPEG-2 encoder. H. 2 is a block diagram of an H.264 encoder. FIG. Conventional H.264. 2 is an explanatory diagram showing a state in which an MB flows through each stage of the MB pipeline in an MB pipeline of an H.264 encoder. FIG. It is the block diagram which showed the structure of the conventional moving image encoder. It is explanatory drawing which showed the example of the reference area of two adjacent MB.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1,80 ... MB pipeline, 2 ... MB selection circuit, 3, 83 ... Encoded data setting circuit, 14 ... Memory for reference area, 15 ... Buffer memory, 20 ... Variable length encoding part, 23, 40, 41 ... Moving picture encoding device, 24, 25, 26, 27 ... MB, 28, 29, 30, 31, 36 ... Reference area, 32, 33, 34, 35, 37 ... Update area, 28, 29, 30, 34 Rectangular area including all, 39 ... internal memory

Claims (4)

In a macroblock row, which is a collection of macroblocks arranged in a row in the horizontal direction from the left end to the right end of a picture, start encoding from the leftmost macroblock, and sequentially encode the macroblocks adjacent to the right A moving image encoding unit ,
The video encoding process is divided into at least two processes, including at least two stages for performing the divided processes, including a macroblock pipeline including a stage for motion detection in the stages,
Selecting the macroblock rows as a group with at least two rows that are continuous in the vertical direction and having a number of rows smaller than the total number of the macroblock rows included in the entire picture;
Injecting the macroblock included in the top macroblock row in the group of macroblock rows into the macroblock pipeline;
Macros adjacent on each macroblock row in order from the macroblock included in the second macroblock row from the top to the macroblock included in the bottommost macroblock row in the group of macroblock rows. After the previous input macroblock included in the block row is input to the macroblock pipeline, it is included in each macroblock row and is located to the left of the previous input macroblock, and the previous input macroblock is adjacent to the upper right. A video encoding unit that inputs macroblocks that are not macroblocks into the macroblock pipeline;
The batch of macroblocks that are included in each macroblock row from the top macroblock row to the bottom macroblock row in the batch of macroblock rows and that are successively entered into the macroblock pipeline. A reference area that is referred to when detecting a motion vector for each macroblock that is subject to motion detection for the same number of lines as the number of macroblock lines, and the reference area has a smaller number of pixels than the total number of pixels in one picture. Memorize all included pixels,
Each macroblock which is located on the right side of the reference area and has an update area whose number of pixels in the horizontal direction is the number of pixels corresponding to the number of pixels in the horizontal direction of the macroblock is transferred from an external memory and is the target of motion detection for Exit motion detection, video coding apparatus and a memory reference region in which the reference area is updated when performing a new motion detection for the right neighboring macroblocks of the macroblock in the motion detection of the target . A macroblock pipeline in which the process of encoding a moving image is divided into at least two processes, at least two stages for performing the divided processes are included, and a stage for performing motion detection is included in the stages; ,
The total number of macroblock rows included in the entire picture, which is a macroblock row that is a group of macroblocks arranged in a row in the horizontal direction from the left end to the right end of the picture, and is at least two rows continuous in the vertical direction. Select a smaller number of lines as a group, and for the macroblocks included in the uppermost macroblock line in the group of macroblock lines, select the macroblock for which the lower left adjacent macroblock has not yet been selected For a macroblock included in a macroblock row other than the topmost macroblock row in the group, select the macroblock in which the upper right adjacent macroblock has already been selected to select the macroblock. A macroblock included in one macroblock line is one macroblock line Macroblock selecting unit or selecting one,
For each macroblock selected by the macroblock selector, from the macroblock included in the top macroblock row to the macroblock included in the bottom macroblock row in the group of macroblock rows And sequentially setting data necessary for encoding each macroblock in the macroblock pipeline, and including an encoded data setting unit that inputs the macroblock into the macroblock pipeline,
When the encoded data setting unit sets the data necessary for encoding the macroblock included in the lowest macroblock row in the group of macroblock rows, in the macroblock pipeline, A moving picture encoding unit for the macroblock selection unit to newly select the right adjacent macroblock for each selected macroblock;
The batch of macroblocks that are included in each macroblock row from the top macroblock row to the bottom macroblock row in the batch of macroblock rows and that are successively entered into the macroblock pipeline. A reference area that is referred to when detecting a motion vector for each macroblock that is subject to motion detection for the same number of lines as the number of macroblock lines, and the reference area has a smaller number of pixels than the total number of pixels in one picture. Memorize all included pixels,
Each macroblock which is located on the right side of the reference area and has an update area whose number of pixels in the horizontal direction is the number of pixels corresponding to the number of pixels in the horizontal direction of the macroblock is transferred from an external memory and is the target of motion detection for Exit motion detection, video coding apparatus and a memory reference region in which the reference area is updated when performing a new motion detection for the right neighboring macroblocks of the macroblock in the motion detection of the target . A video encoding process is divided into at least two processes, a macroblock pipeline including at least two stages for performing the divided processes, and a stage for performing motion detection in the stages;
The total number of macroblock rows included in the entire picture, which are at least two macroblock rows that are a collection of macroblocks arranged in a row in the horizontal direction from the left end to the right end of the picture, and that are continuous in the vertical direction. Select a smaller number of lines as a group, and for the macroblocks included in the uppermost macroblock line in the group of macroblock lines, select the macroblock for which the lower left adjacent macroblock has not yet been selected For a macroblock included in a macroblock row other than the topmost macroblock row in the group, select the macroblock in which the upper right adjacent macroblock has already been selected to select the macroblock. A macroblock included in one macroblock line is one macroblock line First means for one selected or,
The individual macroblocks selected by the first means are changed from the macroblock included in the uppermost macroblock row to the macroblock included in the lowermost macroblock row in the group of macroblock rows. A second means for injecting into the macroblock pipeline in turn,
When the macroblock included in the lowest macroblock row in the group of macroblock rows is input to the macroblock pipeline by the second means, the first means is selected. A video encoding unit that newly selects the right adjacent macroblock for each macroblock;
The batch of macroblocks that are included in each macroblock row from the top macroblock row to the bottom macroblock row in the batch of macroblock rows and that are successively entered into the macroblock pipeline. A reference area that is referred to when detecting a motion vector for each macroblock that is subject to motion detection for the same number of lines as the number of macroblock lines, and the reference area has a smaller number of pixels than the total number of pixels in one picture. Memorize all included pixels,
Each macroblock which is located on the right side of the reference area and has an update area whose number of pixels in the horizontal direction is the number of pixels corresponding to the number of pixels in the horizontal direction of the macroblock is transferred from an external memory and is the target of motion detection for Exit motion detection, video coding apparatus and a memory reference region in which the reference area is updated when performing a new motion detection for the right neighboring macroblocks of the macroblock in the motion detection of the target .   The encoding is started from the group of macroblock rows at the top of a picture, and the group of macroblock rows adjacent to the bottom is sequentially encoded. The moving image encoding apparatus described.

Images