JP5265984B2 - Image encoding apparatus and decoding apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image encoder and an image decoder which rarely delay and have the fewer number of wiring. <P>SOLUTION: When four image encoding processing parts are provided, respective regions where one image is divided into four are assigned to the respective image encoding processing parts in the direction different from that of raster scanning when a macroblock of one image is disposed two-dimensionally. When a macroblock at the left end or right end of respective regions is processed, the image encoding processing part in charge of the next area is waited for to complete necessary processing of the macroblock, data of adjacent macroblock data are used to process a target macroblock. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to an encoding device and a decoding device according to an image compression standard such as MPEG or H.264.

  With the widespread use of high-definition broadcasting and large-capacity optical discs, it has become common to handle HD (High Definition) TV images (1920x1080). In the future, the use of UHD (Ultra High Definition) TV images (7680x4320, 3840x2160) of larger screens is being studied, and further improvement in processing performance of MPEG decoding / encoding devices is desired. To improve the processing performance, “improvement of operating frequency”, “parallelization of processing”, and the like can be considered. However, improvement of the operating frequency requires improvement of semiconductor process technology and a circuit capable of high-speed operation. On the other hand, with regard to parallel processing, MPEG decoding processing requires prediction processing between frames (pictures) and variable length decoding processing, so it is necessary to process code-compressed bitstreams in order from the front. Therefore, it is difficult to parallelize simple processes, and various ideas have been made for parallelization.

   As a prior art parallelization method, (1) considering the motion vector range, the processing start timing of the division unit is controlled to be after the processing of the reference image area required for inter-frame prediction is completed, A method of performing parallelization between frames (Patent Document 1), (2) A method of performing parallelization within a frame in units of slices using independence of data within a frame, such as a slice in MPEG2 (Patent Document 1) 2), (3) A method of performing parallel processing within a frame in units of macroblock lines by controlling the macroblock processing to start after completion of processing of adjacent macroblocks required by each macroblock (Patent Document 3) and the like have been proposed.

  The method of controlling the inter-frame prediction dependency in (1) and performing parallel processing between frames has the following problems when trying to apply to a large screen such as a UHDTV image. In this method, the processing delay from input to output increases as the number of decoders / encoders performing parallel processing increases. An H.264 decoder / encoder that handles a large screen such as UHDTV has a problem that processing delay increases because the number of necessary decoding / encoding processing blocks increases due to an increase in necessary processing capacity.

FIG. 21 to FIG. 23 are explanatory diagrams of a method for performing parallel control between frames by controlling so as to keep the dependency relationship between the conventional inter-frame predictions.
FIG. 21 shows the screen division and the assigned area of the core and the memory, FIG. 22 shows the operation timing, and FIG. 23 shows the block configuration.

21 to 23 are configuration examples in the case of using four decoding processing blocks and four memory modules.
As shown in FIG. 21, the vertical direction of a screen having 2160 pixel lines is divided into four. One macro block line is composed of 16 pixel lines, Memory 0 is macro block lines 0 to 33, Memory 1 is macro block lines 34 to 67, Memory 2 is macro block lines 68 to 101, and Memory 3 is macro block. The lines 102 to 134 are shared and stored.

  As shown in FIG. 22, each Decoder Core is in charge of processing of the entire picture. Decoder Core0 takes charge of the I0 picture and processes the macrolines 0 to 33 of the I0 picture in the first 1/60 second. When this processing is completed, Decoder Core1 in charge of the P3 picture starts processing the macroblock lines 0 to 33 of the P3 picture, and performs processing up to the macroblock line 33 in 1/60 second. After that, Decoder Core2 in charge of B1 picture starts processing of macroblock lines 0 to 33 of B1 picture, and when processing of macroblock line 33 is finished after 1/60 second, Decoder Core3 in charge of B2 picture is set to B2 Processing of macroblock lines 0 to 33 of the picture is started. In this way, each Decoder Core starts processing each picture while being delayed by 1/60 second, and thereafter performs processing in parallel.

In FIG. 23, four Decoder Cores are provided, and four Memory are also provided. And the switch 10 for switching the signal line in the meantime is provided.
In such a configuration, each Decoder Core (decoding processing block) is in charge of assigned frame processing, and its processing time is four times as long as one frame time. More encoders / decoders are required depending on the required image size and frame rate, and the processing delay is proportional to the degree of parallelism. Depending on the field in which the encoder / decoder is used (such as communication between a broadcasting station in the broadcasting field and a relay site), such a large delay cannot be allowed.

24 to 26 are explanatory diagrams of a conventional method for performing parallelization in a frame in units of slices.
24 shows the Decoder Core area, FIG. 25 shows the operation timing, and FIG. 26 shows the block configuration.

  In FIG. 24, two Decoder Cores are provided and are in charge of macroblock lines 0 to 33 and macroblock lines 34 to 67, respectively. Three memories are provided, and are in charge of macroblock lines 0 to 23, macroblock lines 24 to 43, and macroblock lines 44 to 67, respectively.

  As shown in FIG. 25, Decoder Core0 is always in charge of macroblock lines 0 to 33 of each picture, and Decoder Core1 is always in charge of macroblock lines 34 to 67 of each picture. The operation start timings of Decoder Core 0 and 1 are the same, and each assigned macro block line is processed in 1/30 second.

  In FIG. 26, two Decoder Cores are provided, and three Memory are provided. Switches 11-1 to 11-3 are provided to switch the signal line between the Decoder Core and Memory.

  This method uses the fact that the MPEG2 standard has a unique code called a slice header for each macroblock line. In this method, since the processing start timing of the Decoder Core is the same, the encoding / decoding processing time for one frame is the same as the one frame time, and the problem like the method (1) does not occur. But there is another problem. In recent image compression standards such as H.264, such a unique code does not necessarily exist in a frame, and there is a dependency relationship with an adjacent macroblock in the decoding process. For this reason, parallel processing equivalent to the slice level cannot be performed, and it cannot be applied to recent image compression standards represented by H.264.

  The solution to both of the above problems is the technique (3) of controlling the macroblock adjacent information reference dependency so that the macroblock lines are parallelized. Although this method can be used for recent image compression standards including H.264, there are the following problems when trying to apply it to large screens such as UHDTV images. In the H.264 decoder / encoder that handles a large screen like UHDTV, the number of necessary decoding / encoding processing blocks increases due to the increase in required processing capacity, and also regarding the memory module that stores images, In order to secure the necessary memory capacity and the bandwidth between the memory module and the decoding / encoding processing block, a plurality of memory modules are required.

FIG. 27 to FIG. 29 are explanatory diagrams of conventional techniques for performing control so as to protect the dependency relationship of adjacent information reference of macroblocks and performing parallelization between macroblock lines.
FIG. 27 shows the screen division and assigned area of Decoder Core and Memory, FIG. 28 shows the operation timing, and FIG. 29 shows the block configuration.

27 to 29 are configuration examples in the case of using four decoding processing blocks and four memory modules.
As shown in FIG. 27, the four memories are responsible for macroblock lines 0 to 33, macroblock lines 34 to 67, macroblock lines 68 to 101, and macroblock lines 102 to 134, respectively. Each of the four Decoder Cores is responsible for processing each macroblock line by skipping four macroblock lines. In FIG. 27, the macro block line in charge of Decoder Core 3 is shown shaded.

  As shown in FIG. 28, each Decoder Core starts processing its own macro block line after the processing of the MB referenced by the MB is completed by the Decode Core in charge of the upper MB. The processing time for one picture is 1/60 second. Each Decoder Core is in charge of fixed macroblock line processing in all picture processing.

As shown in FIG. 29, four Decoder Cores and four memories are provided, and a switch 12 is provided for switching signal lines between them.
In such a configuration, each decoding processing block is in charge of the assigned macro block line processing, and each memory stores the respective areas divided into four frames. Need to access. For this reason, a 4-to-4 connection is required between the decoding processing block and the memory module, which causes a problem that the memory bus configuration becomes complicated and enlarged. Here, for the sake of simplicity, each of the four areas divided by the horizontal line is allocated to the frame memory, but this problem occurs regardless of the allocation. This problem also existed in the method (1).
JP 2006-14113 A Japanese Patent No. 2863096 JP 2008-67026 A

Therefore, in an image compression standard decoder / encoder device using inter-frame prediction such as MPEG and H.264 that uses multiple decoding / encoding processing blocks and multiple memory modules. It is necessary to be able to be applied to necessary applications. It is necessary to realize a complicated and enlarged memory bus configuration between a decoding / encoding processing block and a memory module, and to reduce a circuit scale / layout difficulty.

  An object of the present invention is to provide an image encoding device and an image decoding device with low delay and a small number of wires.

An image encoding apparatus according to the present invention includes a plurality of storage means for dividing and storing image data and a plurality of image encoding means for performing image encoding processing in an image encoding apparatus for encoding image data in units of blocks. And when the image data is viewed as a two-dimensional array, the connection means for connecting the plurality of storage means and the plurality of image encoding means to the storage means that the image encoding means needs to access, Dividing the image data by the number of the image encoding means in a direction different from the direction in which the image encoding in units of blocks proceeds, assigning each divided image data to the plurality of image encoding means, Control means for causing the image coding means to perform image coding processing in parallel.

  An image decoding apparatus according to the present invention includes an image decoding apparatus that decodes image data, variable-length decoding means that performs variable-length decoding on variable-length encoded image data, and image decoding in units of blocks for variable-length decoded image data. The image decoding means accesses the plurality of image decoding means for performing the processing, the plurality of storage means for dividing and storing the image data subjected to the image decoding process, the plurality of storage means and the plurality of image decoding means. The connecting means for connecting to the necessary storage means, and the image decoding means in a direction different from the direction in which the image decoding processing performed in units of blocks is performed when the image data is viewed as a two-dimensional array. And control means for allocating the divided image data to the plurality of image decoding means and causing each image decoding means to perform image decoding processing in parallel.

  According to the present invention, it is possible to provide an image encoding device and an image decoding device with a low delay and a small number of wires.

  This embodiment shows a parallel processing configuration in the case where a plurality of decoding / encoding processing blocks and a plurality of memory modules are used in an apparatus that requires high processing capability such as decoding / encoding of a large screen such as a UHDTV image. .

In the present embodiment, the moving image encoding apparatus is as follows.
-It corresponds to the moving picture coding system which divides | segments an image into blocks and performs processing.
(1) One or more variable-length encoding units (2) Multiple image encoding processing units (3) Multiple memory modules for storing local decoded images output from (2) (4) A data selection function that exists between the image encoding processing unit and the memory module.
Each image encoding processing unit performs encoding processing of an area divided by a vertical line (a direction different from the direction of raster scanning or the order of processing of macroblocks when picture data is arranged two-dimensionally) Responsible for processing 1 picture in parallel at the same time.

  As a physical configuration of the image encoding processing unit and the memory module, there are a configuration in which all the image encoding processing units are arranged in one LSI and a configuration in which the image encoding processing unit is divided into a plurality of LSIs. In addition, there is a configuration in which the memory module is an external memory and a configuration in which the memory module is an on-chip memory (such as embedded DRAM). The buffer for storing the encoded intermediate data may be arranged as an internal buffer of the LSI, or may be arranged as a location area of a memory externally attached to the LSI.

Means for passing adjacent macroblock information between image encoding processing units.
The means for storing the adjacent macroblock information may be arranged as an internal memory in the LSI, or may be arranged as an area of a memory externally attached to the LSI. In addition, one (one image encoding processing unit-1) may be arranged between adjacent image encoding processing units, or one unit for managing adjacent macroblock information of the entire encoding apparatus is arranged. May be.

  A buffer for storing the encoded intermediate data output from the image encoding processing unit and its data management function are provided, and this data management function indicates the storage position of each area in the picture of the encoded intermediate data in the buffer. It has a function of recording a pointer and distributes and inputs encoded intermediate data to each of the variable length encoding processing units.

The intermediate data buffer may be arranged as an internal buffer in the LSI, or may be arranged as an area of a memory externally attached to the LSI.
Each image coding processing unit has means for notifying position information in a picture of a macroblock being coded, and controls waiting and restarting of the macroblock coding processing according to an instruction from the control unit.

  The control unit controls the start of processing of a certain macroblock from the position information in the picture notified from each image encoding processing unit after completion of the processing of the macroblock that outputs adjacent information necessary for the macroblock. . At this time, the unit of standby and restart of each image encoding processing unit does not necessarily have to be a macroblock unit, and may be a standby or restart of a larger unit, for example, a unit of 5 macroblocks.

  A memory access distribution unit is provided between the image encoding processing unit and the memory module, and the memory access distribution unit performs access from the image encoding processing unit according to a memory map, either one of a plurality of memory modules, or both It has a function to distribute to.

  When a plurality of image coding processing units are physically divided into a plurality of modules, any one of image data, adjacent macroblock information, coding intermediate data, and macroblock coding position information between modules It has a function to exchange data.

The moving picture decoding apparatus is as follows.
-It corresponds to the moving picture coding system which divides | segments an image into blocks and performs processing.
・ Has the following items.

(1) One or more variable length decoding units (2) A plurality of image decoding processing units (3) A plurality of memory modules for storing the decoded images output from (2) (4) Between the image decoding processing unit and the memory modules Data selection function that exists ・ Each image decoding processing unit is an area where a picture is divided by a vertical line (a direction different from the direction of raster scanning or the order of processing of macroblocks when picture data is arranged two-dimensionally) In parallel, one picture is processed in parallel.

Also provided is a decoding apparatus corresponding to the above-described encoding apparatus. The configuration of the decoding apparatus is different only in the order of data processing, and the basic configuration of the decoding apparatus is the same as that of the encoding apparatus.
Macroblock processing such as H.264 / VC-1 requires information on adjacent macroblocks on the left, top left, top, and top right, so macroblock processing starts after processing of these adjacent macroblocks is complete It is necessary to.

  In Japanese Patent Laid-Open No. 2008-67026, the screen is divided into macroblock line units, and a separate image encoding processing unit is assigned to each to perform parallel processing within the screen while maintaining the dependency of adjacent information. It was. Similarly, the present embodiment provides parallel processing within a screen while maintaining the dependency relationship of adjacent information, but the processing areas handled by each image encoding processing unit are different.

1 and 2 are diagrams for explaining the operation of the embodiment of the present invention.
In FIG. 1A, for example, the processing of the macroblock line 1 of the image encoding processing unit 1 is considered. The left adjacent macroblock and the upper left adjacent macroblock in the left end macroblock are within the processing range of the image encoding processing unit 0. However, if the processing of the macroblock line 1 of the image encoding processing unit 0 is completed, Macroblock processing is complete. The upper right macroblock of the rightmost macroblock is the processing range of the image coding processing unit 2, but if the processing of the leftmost macroblock of the macroblock line 0 of the image coding processing unit 2 is completed, this adjacent macroblock Processing is complete. Other adjacent macroblocks are within the processing range of their own (image encoding processing unit 1), and the image encoding processing unit 1 processes the processing range in the order of macroblocks defined in the standard. Macroblock processing is complete. Therefore, if the processing of the adjacent macroblock is completed, the processing of the processing target macroblock can be started. As shown in FIG. 1B, each of the image encoding processing units 0 to 3 processes the macroblocks in its own assigned range in the order in which processing is possible.

  In a general moving picture coding system, the maximum value of a motion vector is determined. Since the image encoding processing unit 0 has a region of 1/4 from the left of the image as the processing range, the region of the local decoded image referred to in the encoding process of the macroblock in this region is the range of the 1/4 left of this image And the area of the maximum width in the horizontal direction of the motion vector are combined (FIG. 2). The maximum horizontal motion vector value is less than or equal to the value specified in the standard, but in actual encoders, motion prediction processing in a smaller range is often performed, and the area of the locally decoded image referred to is the adjacent image. In many cases, it remains in the processing area of the encoding processing unit.

FIG. 3 is a diagram illustrating the deblocking filter process.
As for the write access, it is necessary to write to the left area of the own process for the deblocking filter process in H.264. FIG. 3 shows the left-side writing area in the deblocking filter process. In the deblocking filter process, in order to perform the filter process on the block boundary, the read / write process of the adjacent pixels of the processing macroblock is required. When attention is paid to the left adjacency, in the H.264 standard, the data of the left adjoining three pixels may change due to the deblocking filter process, and thus the write process of the left adjoining three pixels is required.

4 and 5 show a comparison of the number of connection paths between the image encoding processing unit and each memory module (Memory) in the conventional technique and the technique of the present invention.
Here, the processing area (encoding area and storage area) in charge of each image encoding processing unit and each Memory has a one-to-one correspondence, and the encoding area in charge of each image encoding processing unit is required. It is assumed that the reference image area to be stored is stored in the corresponding Memory and the preceding and following Memory (that is, the maximum vector range does not exceed the range of 1 Memory storage area). The number of connection paths in the configuration of the image encoding processing unit 4 / Memory4 and the image encoding processing unit 8 / Memory8 is as shown in the table for the conventional method and this embodiment.

FIG. 6 shows the positional relationship of adjacent macroblock information required for each macroblock for encoding processing.

  For example, when processing a macroblock in H.264, information on adjacent A (left), B (upper), C (upper right), and D (upper left) is required. Here, the adjacent macroblock information includes macroblock type information, motion vector information, pixel values, and the like in each macroblock.

  Since the macroblock at the left end of the processing region of a certain image encoding processing unit uses the adjacent information of the left and upper left macroblocks, it needs to be received from the image encoding processing unit processing the left region. Further, since the right upper macro block information is used in the macro block at the right end of the processing region, it is necessary to receive this from the image encoding processing unit processing the right region.

7 and 8 show examples of adjacent macroblock information memories.
In this example, as shown in FIG. 7, one adjacent macroblock memory is arranged between two image coding processing units. When the processing of the rightmost macroblock of MB-line0 (macrobook at position a in FIG. 8B) of the image encoding processing unit is completed, the macroblock that needs to be saved as adjacent macroblock information Is stored in the adjacent macroblock information memory 0. This information is referred to when the image encoding processing unit 1 starts processing the leftmost macroblock of MB-line0 (the macrobook at the position d in FIG. 8B), but the leftmost macroblock of MB-line1 ( Since it is also used in the macro block at the position e in FIG. 8B, it must not be erased. When the processing of the macro block d is completed, information on the macro block is stored in the macro block information memory 0. When the information of d is referred to in the processing of the macroblock b, it is not used in the processing range of the image encoding processing unit 0 thereafter, and may be deleted from the adjacent macroblock information memory 0. Therefore, it is overwritten by the macroblock information of b. The same applies to the following. When the image encoding processing unit 0 processes the macro block at the position c in FIG. 8B, the macro block at the position e is necessary. Are stored in the adjacent macroblock information memory 0. Further, when the image encoding processing unit 1 processes the macroblock at the position of f, the macroblocks at the positions of b and c are necessary, which are stored in the adjacent macroblock information memory 0. Yes. FIG. 8A is a timing chart showing the above operation.

  By such control, adjacent macroblock information is transmitted between image encoding processing units that process adjacent regions, and parallel processing becomes possible. Each adjacent macroblock information memory may have a capacity for holding information of two macroblocks.

  In general, the variable-length encoding processing needs to be performed in order from the beginning of the stream, and even if parallel processing is performed, it is parallel processing by dividing from a portion where a unique code exists. In standards such as H.264 where slice division is not essential, unique codes always exist in units of pictures. Therefore, when supporting general streams, variable-length encoding processing must be performed in units of pictures. It becomes. On the other hand, the image encoding processing unit of the present embodiment divides a picture into regions and takes charge of the encoding processing of each region. In the configuration of the present embodiment, an encoding intermediate data buffer is provided between the variable length encoding processing unit and the image encoding processing unit, and the storage position pointer of the encoding intermediate data is recorded, so that the image encoding processing unit Realization of parallel processing.

9 to 11 are explanatory diagrams of the intermediate data buffer.
9 is a block configuration, FIG. 10 is an operation timing, and FIG. 11 is a relationship between an intermediate data buffer and a pointer. FIG. 9 to FIG. 11 are examples in the case of four variable-length encoding processing units and four image encoding processing units.

  As shown in FIG. 9, the outputs of the four image encoding processing units (Decoder Core) 0 to 3 are stored in the intermediate data buffer 15 and read by the four variable length encoding processing units 0 to 3. Variable length coding.

  The image encoding processing units 0 to 3 encode the macroblock lines in the area in charge from above and store the intermediate data in the intermediate data buffer 15. At this time, each of the image encoding processing units 0 to 3 writes the intermediate data in the assigned area in the order of processing. As shown in FIG. 10, the operations of the image encoding processing units 0 to 3 are completely parallel, and the intermediate data obtained by the parallel processing is stored in the intermediate data buffer 15. As shown in FIG. 11, the start pointer for each intermediate data of a macroblock line is recorded as PTR_l_m_n, where l is a picture number, m is a macroblock line, and n is an image encoding processing unit number. The variable length coding processing units 0 to 3 perform processing in units of pictures, and perform processing for pictures 0 to 3, respectively. The operations of the variable-length encoding processing units 0 to 3 can start processing only after the data to be processed is stored in the intermediate data buffer 15. Therefore, as shown in FIG. 10, the processes of the variable-length encoding processing units 0 to 3 are parallel processes each shifted by a predetermined timing. The variable length coding processing unit 0 reads out the intermediate data in the order in which the coding processing should be performed, such as PTR_0_0_0, PTR_0_0_1, PTR_0_0_2, PTR_0_0_3, PTR_0_1_0, PTR_0_1_2, PTR_0_1_3,. The same applies to other encoding processing units. In this way, the image encoding processing unit can perform parallel processing within a picture, and the variable length encoding processing unit can perform parallel processing for each picture.

  Each image encoding processing unit notifies the position information of the macro block being processed to the control unit, and the control unit knows the progress of encoding, thereby generating adjacent macro block information necessary for encoding in each macro block. I can understand that. If each image encoding processing unit waits and resumes processing in units of macroblocks according to instructions from the control unit, decoding starts after the necessary adjacent macroblock information is prepared in units of macroblocks. Control becomes possible.

  In the processing of the moving image encoding device, memory access such as reading a reference image is normally memory access in a rectangular area, but when storing image data of one picture for each area in a plurality of memories, in the boundary area, One rectangular access request may span two memories. Therefore, it is necessary to divide the rectangular access request into two according to the memory map.

  FIG. 12 shows an example of access in units of rectangular areas. Access to the area A is access only to Memory0, access to the area B is access to only Memory1, but the area C is stored in Memory0 on the left side and Memory1 on the right side. There is a need.

  FIG. 13 shows a configuration example of the distribution module. In the case of read access, the request distribution part determines which memory is accessed or whether the request needs to be divided. In the case of one memory access, read access to that memory needs to be divided. If there is, perform the read access necessary for each Memory. For the read data from each memory, the read data is distributed to the image encoding processing unit that issued the request according to the request from which image encoding processing unit in the read data distribution units 20-3 and 20-4. return it.

  In the case of write access, the request distribution units 20-1 and 20-2 determine which memory is to be accessed or whether the request needs to be divided, and write data from the image encoding processing unit. In the case of receiving and memory access on one side, write access to that memory, and if it is necessary to divide, write access is performed while distributing the write data received from the image encoding processing unit to each memory.

  In the encoder of the above embodiment, a plurality of image encoding processing units may be physically divided into a plurality of modules. In this case, one module is provided with a plurality of modules by having a function of transferring any one of image data and adjacent macroblock information, encoded intermediate data, and macroblock encoded position information between modules. Configure. In this way, even when a single encoding device is configured with a plurality of modules, it is possible to reduce the inter-module IF and the data transfer amount by reducing the connection path between the memory and the image encoding processing unit.

  When this embodiment is applied to a decoder / encoder such as H.264, a control signal for synchronization control between image coding / decoding processing units, a signal for passing adjacent macroblock information, and the like may be required as described above. However, the increase in the circuit scale required for these functions is small compared to the circuit scale reduction due to the memory bus reduction effect. Since the memory bus between the image encoding / decoding processor and the memory usually requires a high data transfer rate, the bus width is increased (64 bits, etc.), so the bus select signal increases as the number of connection paths increases. However, since the additional circuit required in the present embodiment is a control signal and data with a low transfer rate, the circuit scale can be small. In this embodiment, since the number of connection paths of a memory bus having a wide bus width is greatly reduced compared to the prior art, even if control signals, data paths, and the like necessary for this embodiment are added, the whole The circuit scale is reduced compared to the conventional one.

  Further, even if the number of image coding / decoding processing units and the number of memories increase, the connection relationship is limited (in the conventional technology, all image coding / decoding processing units need to access all memories, but in this embodiment, (Each image encoding / decoding processor only needs to access two or three memories at the maximum), so that the relationship between the I / O terminal and the image encoding / decoding processor arrangement in the LSI can be simplified. This can greatly reduce the layout difficulty.

14 and 15 are diagrams for explaining an encoder device according to the present embodiment.
FIG. 14 shows a block configuration of an apparatus in which the present embodiment is applied to a 4Kx2K (3840x2160) H.264 encoder in four parallel manners.

  Each of the variable length coding processing units 25-1 to 25-4 and the image coding processing units 26-1 to 26-4 has four each, and the variable length coding processing units 25-1 to 25-4 The unit parallel processing and image encoding processing units 26-1 to 26-4 are responsible for the respective areas obtained by dividing the picture into four areas as shown in FIG. 15, and the four memories corresponding to the data in the respective areas. The memory map stored in. Note that the horizontal vector range in this example is ± 512, and the region referred to by the region processed by each image encoding processing unit 26-1 to 26-4 does not exceed the region processed by the left and right cores. To do. Each of the variable length encoding units 25-1 to 25-4 performs variable length encoding processing (CAVLC or CABAC or the like) according to the H.264 standard, and the image encoding processing units 26-1 to 26-4. Performs image coding processing (motion vector search, inter prediction, intra prediction, orthogonal transform, quantization, deblocking filter processing, etc.) to the H.264 standard. Between the variable length coding units 25-1 to 25-4 and the image coding processing units 26-1 to 26-4, an intermediate data buffer 27 for storing intermediate data and between each image coding unit. Are adjacent macroblock information buffers 28-1 to 28-3 for passing adjacent macroblock information, a memory control unit 29 having a function of distributing memory access between the image encoding processing unit and the memory, and each image An image encoding processing unit activation control unit 24 that receives processing macroblock position information from the encoding processing units 26-1 to 26-4 and performs activation control of each image encoding processing unit.

FIG. 16 shows the timing of the parallel operation according to the present embodiment.
The intermediate data buffer, the adjacent macroblock information buffer, the memory control unit, and the core activation control unit perform the above-described roles during the parallel encoding process, and enable the parallel operation of the present invention in the H.264 decoding process. .

  As illustrated in FIG. 16, the image encoding processing units 0 to 3 perform processing in parallel at different timings. First, when picture processing is activated, the image encoding processing unit 0 performs processing of the macro block line (MB-line) 0 of the assigned region. In this macro block line processing, the processing of each macro block is activated in turn. When the image encoding processing unit 0 finishes the processing of the macro block line 0, the process proceeds to the processing of the macro block line 1. At the same time, the image encoding processing unit 1 performs the processing of the macro block line 0 in the assigned region. Start. Similarly, the image encoding processing units 2 and 3 also start processing. When all the image encoding processing units have finished processing all the macroblock lines, the picture processing is completed.

  In the present embodiment, the following control is an example of control that is performed after completion of generation of necessary neighboring macroblock information at the encoding processing start timing of each macroblock in each image encoding processing unit. .

17 and 18 are control flows according to the embodiment of the present invention.
FIG. 17 shows a flow in which the overall control unit controls the image processing unit to perform one picture processing, and FIG. 18 shows a processing flow in one picture processing in each encoding processing unit.

In FIG. 17, in step S <b> 10, the image encoding processing unit is activated, waits until the processing of one picture is completed in the image encoding processing unit, and the processing ends.
In FIG. 18, in step S20, it is determined whether or not the next macroblock to be processed is the leftmost macroblock in the assigned area. If the determination in step S20 is yes, in step S21, it is determined whether or not the image encoding processing unit in charge of the leftmost region is about to perform processing. If the determination in step S21 is yes, the process proceeds to step S30. If the determination in step S21 is No, it is determined in step S22 whether the left adjacent and upper left adjacent macroblock information exists in the adjacent information memory between the left image encoding processing unit and No. In this case, the process waits until these pieces of information are stored in the adjacent information memory. If the determination in step S22 is Yes, the macroblock is processed in step S23, and the processing result is stored as adjacent macroblock information in the adjacent information memory with the left image encoding processing unit in step S24. Then, the process proceeds to step S31.

  If the determination in step S20 is No, it is determined in step S25 whether or not the next macroblock to be processed is the rightmost macroblock in the assigned area. If the determination in step S25 is No, the process proceeds to step S30.

  If the determination in step S25 is Yes, in step S26, it is determined whether or not the image encoding processing apparatus in charge of the rightmost region is about to perform processing. If the determination in step S26 is yes, the process proceeds to step S30, the macroblock is processed, and the process proceeds to step S31. If the determination in step S26 is No, it is determined in step S27 whether the upper right adjacent macroblock information exists in the adjacent information memory with the right image encoding processing unit. If the determination in step S27 is No, the process waits until the information is stored in the adjacent information memory. If the determination in step S27 is Yes, the macroblock is processed in step S28, and the processing result is stored as adjacent macroblock information in the adjacent information memory with the right image encoding processing unit in step S29. The process proceeds to step S31.

In step S31, it is determined whether or not the processing of the assigned macroblock in the picture is completed. If No, the process returns to step S20, and if Yes, the process ends.
FIG. 19 shows an example of a chip configuration when the encoder / decoder configuration of the present embodiment is applied to an LSI.

  In the LSI implementation, various functions and IF are expected to be implemented, but here, only the image input IF and the stream output IF, which are essential for the encoding process, are illustrated. FIG. 19A shows an example of a one-chip configuration, and FIG. 19B shows an example of a two-chip configuration.

  In the one-chip configuration of FIG. 19A, four image encoding / decoding processing units are mounted on one chip, and four memories 0 to 3 for storing images are connected. In this figure, the stream storage buffer and the intermediate data storage buffer are constituted by a separate memory 30. This memory 30 may be prepared separately from the four memories 0 to 3 for storing images, or an area is secured in a part of the four memories 0 to 3 used for storing images as the same memory. May be used.

  In the two-chip configuration of FIG. 19B, an interface 31 that performs data transfer is provided between two memories so that stream data, intermediate data, adjacent macroblock information, image data, and the like can be transferred. In this way, even when the image encoding / decoding processing unit is divided into a plurality of LSIs and the memory connection destination is divided into a plurality of LSIs, the data transfer is transferred and defined as a whole in this embodiment. When the parallel processing is performed, there is no problem even if it is divided into any number of chips.

FIG. 20 shows a configuration diagram when this embodiment is applied to an H.264 decoder.
FIG. 20 shows an apparatus in which 4Kx2K (3840x2160) H.264 decoders are applied in parallel. Each of the variable length decoding processing units 40-1 to 40-4 and the four image decoding processing units 41-1 to 41-4 includes four variable length decoding processing units 40-1 to 40-4. The processing and image decoding processing units 41-1 to 41-4 are responsible for the respective areas obtained by dividing the picture into four areas according to the present embodiment, and the data in the respective areas correspond to the four memories 0 to 3 corresponding thereto. The stored memory map. These assignments are the same as in the encoder embodiment. Each of the variable length decoding processing units 40-1 to 40-4 performs variable length decoding processing (CAVLC or CABAC or the like) according to the H.264 standard, and the image decoding processing units 41-1 to 41-4 Performs processing necessary for H.264 image decoding (inter prediction, intra prediction, inverse orthogonal transform, inverse quantization, deblocking filter processing, etc.). Further, an intermediate data buffer 42 for storing intermediate data and the image decoding processing units 41-1 to 41-41 are provided between the variable length decoding processing units 40-1 to 40-4 and the image decoding processing units 41-1 to 41-4. -4 between adjacent macroblock information buffers 43-1 to 43-3 for transferring adjacent macroblock information, between the image decoding processing units 41-1 to 41-4 and the memories 0 to 3, the memory A memory control unit 44 having a function of distributing access and an image decoding processing unit activation control unit 45 that receives processing macroblock position information from each of the image decoding processing units 41-1 to 1-4 and performs activation control of each core. Have. The method of the parallel operation of the image decoding processing unit in the decoder process is the same as the parallel operation of the image encoding processing unit in the encoding process, and the description thereof will be omitted.

  The intermediate data buffer, adjacent macroblock information buffer, memory control unit, and core activation control unit play a role similar to that described above in the decoding process, and perform the parallel operation of the present invention in the H.264 decoding process. Make it possible.

In addition to the above embodiments, the following supplementary notes are disclosed.
(Appendix 1)
In an image encoding device that encodes image data in units of blocks,
A plurality of storage means for dividing and storing the image data;
A plurality of image encoding means for performing image encoding processing;
Connection means for connecting the plurality of storage means and the plurality of image encoding means;
When the image data is viewed as a two-dimensional array, the image data is divided by the number of the image encoding means in a direction different from the direction in which image encoding progresses in units of blocks, Control means for allocating image data to the plurality of image encoding means and causing each image encoding means to perform image encoding processing in parallel;
An image encoding device comprising:
(Appendix 2)
Further comprising an adjacent unit block information storage means for storing information on adjacent unit blocks of the unit block of the image data to be processed;
2. The image encoding apparatus according to appendix 1, wherein the image encoding unit starts the processing of the unit block to be processed after information on adjacent unit blocks necessary for the processing is gathered.
(Appendix 3)
The image according to claim 1, further comprising buffer means for storing intermediate data as a processing result of the image encoding means, and having a pointer indicating a storage position for each area in the image data. Encoding device.
(Appendix 4)
The plurality of image encoding means are physically divided and arranged in a plurality of modules,
The image encoding apparatus according to claim 1, further comprising an interface unit for communicating information necessary for processing image data between the modules.
(Appendix 5)
The image encoding means acquires position information in the image data of a unit block being encoded, and performs standby and restart of encoding processing of the unit block to be processed based on the position information. The image encoding apparatus according to appendix 1.
(Appendix 6)
The image encoding apparatus according to appendix 1, wherein the direction in which the image encoding process proceeds is a raster scan direction.
(Appendix 7)
The image coding apparatus according to appendix 2, wherein the adjacent unit block information storage unit is provided for each set of the two image coding units.
(Appendix 8)
In an image decoding device for decoding image data,
Variable length decoding means for variable length decoding of variable length encoded image data;
A plurality of image decoding means for performing image decoding processing in block units on the variable length decoded image data;
A plurality of storage means for dividing and storing the image data subjected to the image decoding process;
Connection means for connecting the plurality of storage means and the plurality of image decoding means;
When the image data is viewed as a two-dimensional array, the image data is divided by the number of the image decoding means in a direction different from the direction in which the image decoding process performed in units of blocks is performed. Control means for allocating the image decoding means to the plurality of image decoding means, and causing each image decoding means to perform image decoding processing in parallel,
An image decoding apparatus comprising:
(Appendix 9)
Further comprising an adjacent unit block information storage means for storing information on adjacent unit blocks of the unit block of the image data to be processed;
9. The image decoding apparatus according to appendix 8, wherein the image decoding unit starts the processing of the processing target unit block after information on adjacent unit blocks necessary for the processing is gathered.
(Appendix 10)
9. The image decoding according to appendix 8, further comprising buffer means for storing intermediate data output from the variable length decoding means and for recording a pointer indicating a storage position for each area in the image data of the intermediate data. apparatus.
(Appendix 11)
The plurality of image decoding means can be physically divided into a plurality of modules and arranged,
The image decoding apparatus according to appendix 8, further comprising interface means for communicating information necessary for processing image data between the modules.
(Appendix 12)
The image decoding means acquires position information in the image data of the unit block being decoded, and waits and restarts decoding processing of the unit block to be processed based on the position information. The image decoding apparatus described in 1.
(Appendix 13)
The image decoding apparatus according to appendix 8, wherein the direction in which the image decoding process proceeds is a raster scan direction.
(Appendix 14)
The image decoding apparatus according to appendix 9, wherein the adjacent unit block information storage unit is provided for each set of the two image decoding units.

It is FIG. (1) explaining operation | movement of embodiment of this invention. It is FIG. (2) explaining operation | movement of embodiment of this invention. It is a figure explaining a deblocking filter process. It is FIG. (1) which shows the comparison of the number of connection paths between the image encoding process part and each memory module (Memory) by the conventional method and this invention method. It is FIG. (2) which shows the comparison of the number of connection paths between the image encoding process part and each memory module (Memory) by the conventional method and this invention method. It is a figure which shows the positional relationship of the adjacent macroblock information which each macroblock requires for an encoding process. FIG. 3 is a first diagram illustrating an example of an adjacent macroblock information memory; FIG. 6 is a second diagram illustrating an example of an adjacent macroblock information memory; It is explanatory drawing (the 1) of an intermediate data buffer. It is explanatory drawing (the 2) of an intermediate data buffer. It is explanatory drawing (the 3) of an intermediate data buffer. It is a figure which shows the example of an access of a rectangular area unit. It is a figure which shows the structural example of a distribution module. It is FIG. (1) explaining the encoder apparatus according to this embodiment. It is FIG. (2) explaining the encoder apparatus according to this embodiment. It is a figure which shows the timing of the parallel operation | movement according to this embodiment. It is the control flow (the 1) according to the embodiment of the present invention. It is a control flow (the 2) according to embodiment of this invention. It is an example of a chip configuration when the encoder / decoder configuration of the present embodiment is applied to an LSI. The block diagram at the time of applying this embodiment to an H.264 decoder is shown. It is explanatory drawing (the 1) of the method which controls so that the dependence relationship of the conventional inter-frame prediction may be protected, and performs parallel between frames. It is explanatory drawing (the 2) of the technique which controls so that the dependence relationship of the conventional inter-frame prediction may be protected, and performs parallel between frames. It is explanatory drawing (the 3) of the method which controls so that the dependence relationship of the conventional inter-frame prediction may be protected, and performs parallel between frames. It is explanatory drawing (the 1) of the method of performing the parallelization in a frame in the conventional slice unit. It is explanatory drawing (the 2) of the method of performing the parallelization in a frame in the conventional slice unit. It is explanatory drawing (the 3) of the method of performing the parallelization in a frame in the conventional slice unit. It is explanatory drawing (the 1) of the technique which controls so that the dependence relationship of the adjacent information reference of the conventional macroblock may be protected, and parallelizes between macroblock lines. It is explanatory drawing (the 2) of the technique which controls so that the dependence relationship of the adjacent information reference of the conventional macroblock may be protected, and parallelizes between macroblock lines. It is explanatory drawing (the 3) of the technique which controls so that the dependence relationship of the adjacent information reference of the conventional macroblock may be protected, and performs parallel between macroblock lines.

Explanation of symbols

10, 11-1 to 11-3, 12 switch 15 Intermediate data buffer 20-1, 20-2 Request / write data distribution unit 20-3, 20-4 Read data distribution unit 24 Image encoding processing unit activation control unit 25 -1 to 25-4 Variable length coding processing unit 26-1 to 26-4 Image coding processing unit 27 Intermediate data buffer 28-1 to 28-4 Adjacent macroblock information buffer 29 Memory control unit 30 Memory 31 Communication interface 40 -1 to 40-4 Variable length decoding processing unit 41-1 to 41-4 Image decoding processing unit 42 Intermediate data buffer 43-1 to 43-3 Adjacent macroblock information buffer 44 Memory control unit 45 Image decoding processing unit activation control unit

Claims (4)

In an image encoding device that encodes image data in units of blocks,
A plurality of storage means for dividing and storing the image data;
A plurality of image encoding means for performing image encoding processing;
When the image data is viewed as a two-dimensional array, the image data is divided by the number of the image encoding means in a direction different from the direction in which image encoding progresses in units of blocks, Assign each image data to the plurality of image encoding means, stores the divided image data when encoding, the storage unit adjacent to the storing means and the corresponding storage means corresponding to said image coding means Control means for causing each image encoding means to perform image encoding processing in parallel, based on the reference result of the image area ,
Connection means for connecting each of the image encoding means only to storage means for storing an image area to be referred to when encoding the divided image data ;
Further comprising an adjacent unit block information storage means for storing information on adjacent unit blocks of the unit block of the image data to be processed;
The image encoding device is characterized in that the processing of the unit block to be processed is started after information on adjacent unit blocks necessary for the processing is gathered .   The image encoding apparatus according to claim 1, wherein a direction in which the image encoding process proceeds is a raster scan direction. In an image decoding device for decoding image data,
Variable length decoding means for variable length decoding of variable length encoded image data;
A plurality of image decoding means for performing image decoding processing in block units on the variable length decoded image data;
A plurality of storage means for dividing and storing the image data subjected to the image decoding process ;
When the image data is viewed as a two-dimensional array, the image data is divided by the number of the image decoding means in a direction different from the direction in which the image decoding process performed in units of blocks is performed. respectively assigned to the plurality of image decoding means, wherein when decoding the divided image data, storage means and stored in said storage means adjacent the corresponding storage means image corresponding to the image decoding unit Control means for causing each image decoding means to perform image decoding processing in parallel based on the reference result of the area ;
Connection means for connecting each of the image decoding means only to storage means for storing an image area referred to when the divided image data is decoded ;
Further comprising an adjacent unit block information storage means for storing information on adjacent unit blocks of the unit block of the image data to be processed;
The image decoding device is characterized in that the processing of the unit block to be processed is started after information on adjacent unit blocks necessary for the processing is gathered . The image decoding apparatus according to claim 3 , wherein a direction in which the image decoding process proceeds is a raster scan direction.