JP4697419B2 - Video coding system reduces frame memory access by working memory direct link - Google Patents

Description

  The present invention relates to a hardware architecture used in a video encoding system, and more particularly to a more effective frame memory access of the video encoding system.

  In recent years, computer systems have been applied in various fields.

  However, in these new various fields, such as the video field, the demand for storage areas capable of holding all application data as well as high processing capacity has increased.

  As a solution to meet this requirement, the entire memory was divided according to the level of the hierarchy.

  The memory divided according to the difference in the hierarchical level is one that is configured near the arithmetic unit and can be accessed quickly but has a small capacity, while the other is configured far from the arithmetic unit and has a large capacity. Memory with slow access time.

  This type of system is a video encoding system, and its configuration is shown in FIG.

  In the video encoding system shown in FIG. 1, two different types of memories are configured. One memory is an on-chip working memory that can be accessed quickly, and is configured near the arithmetic unit of the processor unit. On the other hand, the other off-chip frame memory is used as a large capacity data memory.

  In the video encoding system, most of the video data is held in the frame memory, while a part of the video data to be processed immediately is transferred to the working memory.

  This is because a large amount of temporary data generated by the arithmetic unit during processing is stored in a work memory with fast access and does not need to be stored in a frame memory with slow access, which often leads to a reduction in processing time. Because.

  Of course, the data in the working memory must always be exchanged with newly generated data, and for this purpose the processing in the arithmetic unit must be stopped.

  In a wide variety of applications, speed can be increased by configuring a multi-port working memory. Alternatively, as shown in FIG. 2, data access is made by configuring two different working memories, one for external data transferred from the frame memory and the other for internal data accessed from the arithmetic unit. It is possible to further increase the speed. In such a case, data transfer and data processing can be performed in parallel.

  The path is routed through a switch in order for all processing units to access all working memory and provide data.

  In the embodiment shown in FIG. 2, a switch for switching between the two states is necessary. In one state, the frame memory transfers data to the working memory 0, and the arithmetic unit accesses the working memory 1. The other state is the opposite.

  However, in many algorithms, temporary data is not used directly again, but rather later.

  On the other hand, when the position where the memory switch is configured is changed, this temporary data cannot be accessed from the arithmetic unit.

  FIG. 3 shows the task scheduling operation of the example algorithm where data must be transferred between both working memory bodies, and in order to solve the above problem, the data must be stored in frame memory. Must recover later.

  First of all, the arithmetic unit must interrupt the execution of its processing once.

  Secondly, the switch position must be changed and data must be transferred from the working memory to the frame memory.

  Thirdly, the position of the memory switch must change its state back in order to load data from the frame memory into the other working memory that will be used next by the arithmetic unit.

  Finally, the interrupt of the arithmetic unit is taken, and both units are changing the working memory body by changing the position of the memory switch once more.

  Due to the slow access time to the frame memory, the compute unit process execution stops and for many applications this can be a useful part of the overall processing time.

Srinivasan et al., In-loop deblockingfilter ・ US patent US 2005/0013494 A1, Jan. 2005. Gomila et al., Deblocking filterconditioned on pixel brightness ・ US patent US 2003/0206664 A1, Nov. 2003. Sankaran, loop deblocking filtering of block coded video in a very long instruction word processor ・ US patent US 2005/0117653 A1, Jun. 2005. Draft ITU-T Recommendation and Final Draft International Standard of Joint Video Specification (ITU-T Rec. H.264 / ISO / IEC14496-10 AVC), Mar. 2003. T. Wiegand, GJ Sullivan, G. Bjntegaard, and A. Luthra, overview of the H.264 / AVC video coding standard, ・ IEEETrans. Circuits Syst.Video Technol., Vol. 13, pp. 560 ・ 76, July 2003 . P. List, A. Joch, J. Lainema, G. Bjntegaard, and M. Karczewicz, adaptive deblockingfilter, IEEE Trans. Circuits Syst. Video Technol., Vol. 13, pp. 614-19, July 2003. Miao Sima, YuanhuaZhou, and Wei Zhang, an Efficient Architecture for Adaptive Deblocking Filter of H.264 / AVC Video Coding, IEEETransactions on Consumer Electronics, Vol. 50, No. 1, FEBRUARY 2004. Yu-Wen Huang, To-Wei Chen, Bing-Yu Hsieh, Tu-ChihWang, Te-Hao Chang, and Liang-GeeChen, architecture Design For Deblocking Filter in H.264 / JVT / AVC ・ Multimedia and Expo, 2003. ICME '' 03.Proceedings. 2003 International Conference on, vol. 1, pp. 693 ・ 96,6-9 July 2003. G. Q. Zheng and L. Yu, an efficient architecture design for deblocking loopfilter ・ Picture Coding Symposium, 2004. Vivek Venkatraman, Shoba Krishnan, and Nam Ling, architecture for De-BlockingFilter in H.264, ・ presented and published in Proceedings ofthe Picture Coding Symposium 2004 (PCS), San Francisco, California, USA, December 15 ・ 17, 2004. S.-C. Chang, W.-H. Peng, S.-H. Wang and T. Chiang, "A Platform based Bus-interleaved Architecture for Deblocking Filter in H.264 / MPEG-4 AVC," IEEETrans. On Consumer Electronics, Feb. 2005.

  In the prior art shown in FIG. 2, the data must be copied once to a slow-access frame memory, resulting in a slow data transfer between two working memories.

  FIG. 3 illustrates task scheduling operations of an example embodiment application in the prior art.

  As shown in FIG. 3, in order to transfer data between two working memories, the data is first stored from one working memory to the frame memory, and then the data from the frame memory to the other working memory. Must be loaded. As described above, the two data transfers must be performed through the frame memory having a slow access.

  The present invention has been made to solve the above problems.

A first invention for solving the above problem is a video encoding system,
Off-chip memory for storing video data;
A plurality of on-chip memory for temporarily storing the image data,
A deblocking filter unit for computing video data of the on-chip memory ;
A switching device that switches connection between the off-chip memory, the deblocking filter unit, and the on-chip memory , and a direct connection between the on-chip memories is established ;
When video data in the on-chip memory is stored in another on-chip memory after being processed by the deblocking filter unit, the transfer between these on-chip memories is directly established in the switching device. Done with a connection,
In order to deblock filter the macro block of the on-chip memory, video data stored in a predetermined macro block is transferred to another on-chip memory using the direct connection established in the switching device. replicated imaging encoding system according to claim Rukoto.

  The above-mentioned problems caused by slow access times in the frame memory can be eliminated by the method as claimed in claim 1.

  As an example with two working memories, the basic architecture from FIG. 2 has been expanded in this way.

  The results are shown in FIG.

  Here, the switch is extended by an additional data link that allows a direct data connection between both working memory entities.

  Further, as can be seen from FIG. 5, the data transfer of data to be transferred between the work memories does not depend on the delay of the access time to the frame memory, but only the access time to the work memory.

  For the embodiment as claimed in claim 2, the deblocking filter for H.264 video coding was selected as shown in FIG.

  Here, the deblocking filter in the deblocking filter accelerator and the data connection from inside the frame memory are guided to the whole switching to the two working memories.

  When an algorithm for a deblocking filter macro block is chosen, the size of the working memory is one working memory with all the data needed to process the filter operation on one macro block at a time It can be selected in that way.

  However, as a result of the macro block being based on the algorithm, part of the data of two macro blocks adjacent to each other across the line overlap.

  With this new switch it is possible to transfer this overlapping data directly between both working memory entities.

  This speeds up all filtering methods equivalent to data transfer where the frame memory is included in the data transfer process.

  Using the present invention, first, the number of transfers between memories can be reduced from 2 to 1, and secondly, the transfer time can be reduced. This is because there is no need to transfer to a slow-access frame memory.

  Third, the data bandwidth required on the external bus connected to the off-chip frame memory can be reduced.

  The problem of the present invention is that in the prior art shown in FIG. 2, data must be copied once to a slow-access frame memory, resulting in a slow data transfer between two working memories.

  FIG. 3 illustrates task scheduling operations of an example embodiment application in the prior art.

  As shown in Figure 3, to transfer data between both working memories, first store the data from one working memory into the frame memory, then load the data from the frame memory into the other working memory Must. As described above, the two data transfers must be performed through the frame memory having a slow access.

  By using the present invention, the above problem can be solved.

  To solve the above problem, the multiplexer inside the memory switch must be expanded in order to establish a direct link between the two working memories, as shown in FIG.

  With this new data link implemented inside the switch, the data that must be transferred between the two working memories can be transferred directly between them, without having to go through the frame memory.

  As a result, first, the number of transfers between memories can be reduced from 2 to 1, and secondly, the transfer time can be reduced. This is because there is no need to transfer to a slow-access frame memory. Third, the data bandwidth required on the external bus connected to the off-chip frame memory can be reduced.

  FIG. 5 shows a new task schedule operation in the embodiment shown in FIG.

  The time during which the arithmetic unit is stopped can be reduced by a time obtained by subtracting the time of direct transfer between the working memories from the transfer time of the frame memory twice.

  As another embodiment, a configuration equipped with a deblocking filter will be described.

  In Patent Documents 1, 2, and 3, there are other deblocking algorithms, whereas the configuration shown in FIG. 6 is the deblocking filter of Non-Patent Documents 1, 2, and 3 for H.264 video coding. Refer to

  In contrast to the architecture described in [4], which requires a frame buffer and larger system latency to filter the underlying frame, working memory is all necessary at one time. The deblocking filter accelerator implements a deblocking filter on which the macroblock is based, since it must only have a size that holds the data in a process-one macroblock.

  As another embodiment, as described in any one of Non-Patent Documents 5, 6 and 7, a multi-port working memory capable of simultaneously executing data transfer and processing is used, or Non-Patent Document Use tight task coupling instead of loose task coupling on the working memory by loading data directly from the frame memory to the arithmetic unit without buffering elsewhere, as in FIG.

  However, this is done at the expense of the high processing time dependence of the slow-access frame memory.

  The configuration described here is used in contrast to multi-port working memory to reduce frame memory access by being present in two single-port working memories.

  This is done by processing the data stored in one working memory and another working memory that loads the next macro block from the frame memory to store the data of the previously processed macro block. By using it, this is done to allow data transfer and data processing to be performed simultaneously.

  In order to process the deblocking filter, the surrounding data from the next adjacent macro block of the next macro block image data to be processed in the arithmetic unit is required.

  These are data from the top and left macroblocks.

  The required top macro block data is already in the frame memory and can be pre-installed from it, while the required data from the left macro block is not available in the frame memory Cannot be pre-installed in parallel with the filter calculation of the deblocking filter unit.

  Instead, these data must be transferred between working memories after a filter calculation of previously processed macroblocks.

  A further different embodiment is shown in FIG. This embodiment is described in claim 3.

  In FIG. 7, the macro block data in the working memory 0 is processed, and the macro block data in the working memory 1 is processed next.

  The shaded area in Figure 7 has 20 lines for each 20 pixels, the data for the next macroblock of 16 lines x 16 pixels, and the last 4 of the top macroblock. Data used in the working memory including 4 lines × 16 pixels data from the line and 16 lines × 4 pixels data from the last 4 stages of the left macroblock.

  The texture area having a size 1/6 of the use area is an overlapping portion here. The texture area is newly calculated in the working memory 0 and used in the working memory 1.

  Due to the new inserted connection of claim 2 inside the switch, this area can be transferred directly between both fast-accessing working memories without passing through the frame memory.

  This increases the speed of data transfer. The first reason is that the transfer operation between memories is halved. Second, it does not include a frame memory that has a slow access time.

  In addition, this measurement has a positive effect on overall system performance. The reason is that the occupation of external buses is decreasing.

FIG. 1 is a schematic diagram of a typical video coding system comprising a main processor and an accelerator having a single-port working memory accessible from different units via a switch. FIG. 2 is a schematic diagram of a typical video encoding system comprising a main processor and an accelerator having two single-port working memories each accessible via a switch from a different unit. FIG. 3 is a schematic diagram of the task schedule operation possibility when the video encoding system shown in FIG. 2 has to transfer data between both work memory bodies before the arithmetic unit continues execution processing. FIG. 4 is a schematic diagram of a typical video encoding system including a main processor and an accelerator having two single-port working memories that can be accessed via switches that connect the working memories from different units. FIG. 5 is a schematic diagram of the task schedule operation possibility when the video encoding system shown in FIG. 4 has to transfer data between both work memory bodies before the arithmetic unit continues the execution process. FIG. 6 shows an H.264 video code having a main processor and a deblocking filter accelerator having two single-port working memories that can be accessed from the deblocking filter unit via a switch that connects the working memories. 1 is a schematic diagram of a system. FIG. 7 is a schematic diagram of claim 3 illustrating the advantages of claim 2 having the necessary macro block data transfer between both working memories when performing a deblocking filter based on the method of the macro block.

Explanation of symbols

0 Working memory 1 Working memory

Claims (1)

A video encoding system,
Off-chip memory for storing video data;
A plurality of on-chip memory for temporarily storing the image data,
A deblocking filter unit for computing video data of the on-chip memory ;
A switching device that switches connection between the off-chip memory, the deblocking filter unit, and the on-chip memory , and a direct connection between the on-chip memories is established ;
When video data in the on-chip memory is stored in another on-chip memory after being processed by the deblocking filter unit, the transfer between these on-chip memories is directly established in the switching device. Done with a connection,
In order to deblock filter the macro block of the on-chip memory, video data stored in a predetermined macro block is transferred to another on-chip memory using the direct connection established in the switching device. replicated imaging encoding system according to claim Rukoto.

Images