KR20110122658A - Video decoder and method for video decoding using multi-thread - Google Patents

Abstract

PURPOSE: A method and apparatus for decoding video based on multi thread are provided to improve decoding performance and satisfy data dependency according to H.264/AVC standard. CONSTITUTION: A process module(200) performs a first process of a fourth area by generating a (4-1)-th thread(241). The process module performs a third process of a second area by generating a (4-2)-th thread(242). The process module performs a second process of a third area by generating a (4-3)-th thread(243). The request of the memory is decreased by reusing a memory bank(310,320,330) in the video decoding based on the multi thread.

Description

Video decoder and method for video decoding using multi-thread}

The present invention relates to a multi-threaded video decoder and decoding method.

As information and communication technology including the Internet is developed, not only text and voice but also video communication are increasing. Conventional text-based communication methods are not enough to satisfy various needs of consumers, and accordingly, multimedia services that can accommodate various types of information such as text, video, and music are increasing. The multimedia data has a huge amount and requires a large storage medium and a wide bandwidth in transmission. Therefore, in order to transmit multimedia data including text, video, and audio, it is essential to use a compression coding technique.

The basic principle of compressing data is to eliminate redundancy in the data. Among these data, in particular, video data has a very large capacity, and thus, the importance of efficient compression is more important than other types of multimedia data.

The basic principle of video compression is spatial redundancy, such as the repetition of the same color or object within a picture (frame), temporal repetition of the same note in an audio frame, or when there is little change in adjacent frames in a video frame. Data is compressed through a method of eliminating perceptual duplication in consideration of duplication, or the insensitivity of human visual and perceptual ability to high frequencies. In a general video coding method, temporal redundancy is eliminated by temporal filtering based on motion compensation, and spatial redundancy is removed by spatial transform.

In the past, such video coding or decoding was generally performed by a single-core processor. However, in recent years, the use of multi-core processors has been increased in the field that consumes a lot of system resources such as video coding / decoding as the powerful multi-core processors have become popular.

The technical problem to be solved by the present invention is to provide a multi-threaded video decoding method with improved processing performance.

Another technical problem to be solved by the present invention is to provide a multi-threaded video decoder with improved processing performance.

Technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

An aspect of the multi-threaded video decoding method of the present invention for achieving the above technical problem is to divide a video frame into first to N-th regions, and generate a first-first thread in a first step The first process is performed on the first region, and in the second step, the second region is generated by performing the second process on the first region, and the second-2 thread is generated, and the second region is generated on the second region. Perform a first process, perform a third process for the first region by creating a 3-1 thread in a third step, generate a third-2 thread, and perform a second process for the second region, Creating a third-3 thread to perform a first process on the third region;

One aspect of the multi-threaded video decoder of the present invention for achieving the above another technical problem, an input module for receiving a video frame including the first to N-th region, and generating a plurality of threads for the video frame And a processing module for performing decoding processing, wherein the processing module generates a first-first thread in a first step to perform a first processing on the first region, and generates a second-first thread in a second step. To perform a second process on the first region, generate a 2-2 thread to perform the first process on the second region, and in the third step, generate a 3-1 thread on the first region. The third process is performed, a second thread is generated to perform the second process on the second area, and the third-3 thread is generated to perform the first process on the third area.

Specific details of other embodiments are included in the detailed description and the drawings.

According to the multi-threaded video decoder and decoding method according to an embodiment of the present invention, it is possible to improve decoding processing performance while satisfying data dependency according to the H.264 / AVC standard. In addition, by reusing the memory bank in the decoding process, it is possible to use the memory efficiently.

1 is a video decoding flowchart according to the H.264 / AVC standard.
2 is a diagram illustrating inter-macroblock dependencies in video decoding according to the H.264 / AVC standard.
3 to 6 are diagrams for describing a multi-thread based video decoding method according to an embodiment of the present invention.
7 and 8 are diagrams for describing the performance of a multi-threaded video decoding method according to an embodiment of the present invention.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various forms, and only the present embodiments are intended to complete the disclosure of the present invention, and the general knowledge in the art to which the present invention pertains. It is provided to fully convey the scope of the invention to those skilled in the art, and the present invention is defined only by the scope of the claims.

The size and relative size of the components shown in the drawings may be exaggerated for clarity of explanation. Like reference numerals refer to like elements throughout the specification, and "and / or" includes each and every combination of one or more of the mentioned items.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used in a sense that can be commonly understood by those skilled in the art. In addition, the terms defined in the commonly used dictionaries are not ideally or excessively interpreted unless they are specifically defined clearly.

Hereinafter, a multi-threaded video decoding method according to an embodiment of the present invention will be described with reference to FIGS. 1 to 6.

Hereinafter, the present invention will be described with an example of decoding according to the H.264 / AVC standard, but the present invention is not limited thereto. In addition, in the present specification, although the video frame includes a plurality of macro blocks (hereinafter referred to as MB), the description is given as an example, but the present invention is not limited thereto.

First, a workflow of decoding a video frame encoded according to the H.264 / AVC standard will be described with reference to FIG. 1.

1 is a video decoding flowchart according to the H.264 / AVC standard.

Referring to FIG. 1, when a bit stream constituting a video frame is provided, it is entropy decoded (hereinafter, referred to as ED) 10. In detail, in H.264 / AVC, the bit stream may be input in units of Network Adaptation Layer (NAL) to generate coefficients in the ED 10. Next, coefficients generated after the entropy decoding operation are received after the process of inverse quantization (hereinafter referred to as IQ) 20 and inverse transformation (hereinafter referred to as IT) 30. Can be generated).

Meanwhile, the coefficients generated after the entropy decoding operation may be used to generate a plurality of prediction data. Intra prediction (hereinafter referred to as IP) 40 performs intra prediction using spatial redundancy, and performs motion compensation ( motion compensation (hereinafter referred to as MC) 50 may perform prediction between screens using temporal redundancy. Such a prediction task may be applied with IP or MC according to the type of MB.

Blocks created after IP and MC may be merged with residual data created after IQ / IT (60). Since the decoded image may have a blocking phenomenon in which the boundary between blocks is clearly visible, a deblocking filter (hereinafter referred to as DF) 70 may be applied to remove the block boundary. . Here, the DF of H.264 / AVC may use an adaptive method that is applied while controlling the weight value.

Next, referring to FIG. 2, data dependency between MBs in the decoding method according to the H.264 / AVC standard will be described.

2 is a diagram illustrating inter-macroblock dependencies in video decoding according to the H.264 / AVC standard.

Referring to FIG. 2, in order to process IP, MC, and DF of MB currently to be processed, respectively, IP, MC, and DF of neighboring MBs must be processed. Specifically, in order to process the IP and MC of the MB to be processed currently, the IP and MC of the MB (1, 2, 3) on the upper left, upper and right sides of the MB (Current MB) to be processed and the MB (4) on the left Must be processed. In order to process the DF of the MB to be processed, the DF of the MB (2) on the upper side and the MB (4) on the left side of the MB (Current MB) to be processed must be processed. This results in the necessity of checking whether or not dependent MBs are processed in processing each MB, which may adversely affect the processing speed. This will be described later in more detail.

Next, a multi-threaded video decoding method according to an embodiment of the present invention will be described with reference to FIGS. 3 to 6.

3 to 6 are diagrams for describing a multi-thread based video decoding method according to an embodiment of the present invention.

First, referring to FIG. 3, a video frame is divided into first to Nth regions. As described above, the video frame may be a video frame encoded according to the H.264 / AVC standard.

Each of the first to Nth regions may include a plurality of MBs 110. 3 to 6 together, in the multi-threaded video decoding method according to an embodiment of the present invention, the video frame 100 is divided into first to fourth regions each including 16 MB. However, the present invention is not limited thereto. If necessary, the number of divided regions can be much larger. In addition, although the number of MBs 110 included in the first to fourth regions is the same as that of 16, the number of MBs 110 may be different. Accordingly, the first to fourth regions may include different numbers of MBs 110. 3 to 6, the video frame 100 is shown to be divided into four in the horizontal direction as shown, the present invention is not limited thereto.

Referring to FIG. 3 again, in a first step (Stage 1), a first-first thread 211 is generated to perform a first process on the first area. Wherein the first treatment may be ED. After the first-first thread 211 performs all ED for the first region, the result may be stored in the first memory bank 310. This will be described later, but may be for another thread to use the ED result for the first area for other processing.

Next, referring to FIG. 4, in a second step (Stage 2), a 2-1 th thread 221 is created to perform a second process on the first region, and a 2-2 th thread 222 is generated to generate a second thread. The first process is performed on the two areas. Here, the first process may be ED, and the second process may be MC + IQ / IP or IP + IQ / IT. In this case, one of MC and IP may be performed according to the type of MB.

The 2-1 th thread 221 uses the result of ED processing on the first area by the 1-1 thread (211 of FIG. 3) stored in the first memory bank 310. IQ / IP or IP + IQ / IT processing can be performed. The processed result may be stored in the first memory bank 310 again. The second-2 thread 222 may perform a first process on the second area, and store the processing result in the second memory bank 320.

Next, referring to FIG. 5, in the third step (Stage 3), the third-first thread 231 is generated to perform a third process on the first region, and the third-second thread 232 is generated to generate the third-first thread. The second process is performed on the second region, and the third-3 thread 233 is generated to perform the first process on the third region. Here, the first process may be ED, the second process may be MC + IQ / IP or IP + IQ / IT, and the third process may be DF.

The 3-1 th thread 231 processes the result of MC + IQ / IP or IP + IQ / IT processing of the first region by the 2-1 th thread (221 of FIG. 4) stored in the first memory bank 310. DF processing for the first area can be performed. The processed result may be stored in the first memory bank 310 again. The third-second thread 232 uses the result of ED processing of the second area by the second-second thread (222 in FIG. 4) stored in the second memory bank 320. IQ / IP or IP + IQ / IT processing may be performed, and the processing result may be stored in the second memory bank 320 again. In the multi-threaded video decoding method according to an embodiment of the present invention, the MC + IQ / IP or IP + IQ / IT processing for the second region is performed by the MC + IQ / IP or IP + IQ / for the first region. Since it is performed after the IT processing is performed, it can be seen that the data dependency according to the H.264 / AVC standard shown in FIG. 2 is satisfied. Finally, the third-3 thread 233 may perform a first process on the third area and store the processing result in the third memory bank 330.

Next, referring to FIG. 6, in a fourth step (Stage 4), a fourth-first thread 241 is created to perform a first process on a fourth region, and a fourth-second thread 242 is generated to generate a first-first thread. The third process is performed on the second region, and the fourth-3 thread 243 is generated to perform the second process on the third region. As before, here, the first process may be ED, the second process may be MC + IQ / IP or IP + IQ / IT, and the third process may be DF.

Referring to FIG. 6, the first region may be a decoded region in which all of ED, MC + IQ / IP, IP + IQ / IT, and DF processes are completed and decoded. On the other hand, the 4-2 thread 242 is processed by the 3-2 thread (232 of FIG. 5) stored in the second memory bank 320 MC + IQ / IP or IP + IQ / IT for the second area Using the result, DF processing for the second area can be performed. In addition, the performed processing result may be stored in the second memory bank 320 again. The 4-3 thread 243 is the MC + for the third area by using the result of ED processing of the third area by the third-3 thread (233 in FIG. 4) stored in the third memory bank 330. IQ / IP or IP + IQ / IT processing may be performed, and the processing result may be stored in the third memory bank 330 again. Similarly, since MC + IQ / IP or IP + IQ / IT processing for the third region is performed after MC + IQ / IP or IP + IQ / IT processing for the second region is performed in FIG. It can be seen that it satisfies the data dependency according to one H.264 / AVC standard. Finally, the 4-1 thread 241 may perform ED processing on the fourth region and store the processing result in the first memory bank 310. Here, the first memory bank 310 stores the decoded data for the first region when decoding of the first region is performed, and is used for decoding the fourth region after the decoding of the first region is completed. It may be a memory bank that is reused. That is, in the multi-threaded video decoding method according to an embodiment of the present invention, by reusing the memory bank as described above, the demand for a rapidly required memory may be reduced as the high resolution video frame is processed.

The process after the above-described fourth step can be sufficiently inferred as the description of the first to fourth steps described above, and thus a detailed description thereof will be omitted. If the decoding process continues in this order, it is possible to complete the decoding of the first to fourth regions.

Next, the processing performance improvement of the multi-threaded video decoding method according to an embodiment of the present invention will be described with reference to FIGS. 7 and 8.

Table 1 below shows the decoding times of various images (rush hour to shields) using the H.264 / AVC decoder before the parallelization method is applied. And, Table 2 shows a frame when decoding for various images (rush hour ~ shields) using a task level parallelism method, a data level parallelism method and a multi-threaded video decoding method according to an embodiment of the present invention. Is a table showing the minimum treatment time, and FIG. 7 is a graph related to Table 1. FIG.

Finally, Table 3 is a table showing the average decoding time per frame when using the same method as Table 2, respectively, Figure 8 is a graph of Table 2.

Here, a typical pipeline technique (A) is used as the task level parallelization method, and a 2Dwave technique (B) is used as the data level parallelization method. Here, the 2Dwave technique uses a technique using two threads (B-1) and a technique using three threads (B-2), respectively. Also, each image (rush hour ~ shields) was encoded using JM-v16, and the encoding environment was based on H.264 / AVC baseline profile provided by JM-v16. The operating system was developed on Linux ubuntu 9.10, kernel version 2.6.31, and used Intel quad core i5 processor. The compiler used gcc v4.4.1 and the parallelization method used OpenMP.

Basic ED (AVG) MC + IQ / IT
IP + IQ / IT (AVG) DF (AVG) AVG MIN Sequence (㎲) (%) (㎲) (%) (㎲) (%) (㎲) (㎲) rush_hour FHD, 1920X1088 3929 24% 6567 41% 5699 35% 16195 14746 blue_sky FHD, 1920X1088 4184 27% 5863 38% 5514 35% 15561 13694 pedestrain_area FHD, 1920X1088 4295 24% 6800 39% 6440 37% 17535 13498 sunflower FHD, 1920X1088 3905 29% 6301 47% 3130 23% 13336 11034 park_run HD, 1280X720 5150 40% 3926 31% 3714 29% 12790 10687 mobcal HD, 1280X720 2310 28% 3458 42% 2564 31% 8332 3882 stockholm HD, 1280X720 2029 28% 2993 42% 2190 30% 7212 6640 shields HD, 1280X720 2211 30% 2997 40% 2210 30% 7418 5707

Sequence A B-1 B-2 C MIN
(㎲) MIN
(%) MIN
(㎲) MIN
(%) MIN
(㎲) MIN
(%) MIN
(㎲) MIN
(%) rush_hour FHD, 1920X1088 11873 19% 12105 18% 9998 32% 6993 53% blue_sky FHD, 1920X1088 11018 20% 11470 16% 9558 30% 6735 51% pedestrain_area FHD, 1920X1088 12126 10% 10666 21% 9604 29% 8431 38% sunflower FHD, 1920X1088 9481 14% 9616 13% 8507 23% 6718 39% park_run HD, 1280X720 7813 27% 8482 21% 7441 30% 5580 48% mobcal HD, 1280X720 2655 32% 3154 19% 2678 31% 2172 44% stockholm HD, 1280X720 4996 25% 5395 19% 4563 31% 3702 44% shields HD, 1280X720 5046 12% 4689 18% 4160 27% 3528 38%

Sequence A B-1 B-2 C AVG
(㎲) AVG
(%) AVG
(㎲) AVG
(%) AVG
(㎲) AVG
(%) AVG
(㎲) AVG
(%) rush_hour FHD, 1920X1088 14309 12% 13390 17% 12130 25% 9236 43% blue_sky FHD, 1920X1088 13914 11% 13159 15% 11780 24% 8477 46% pedestrain_area FHD, 1920X1088 15784 10% 14264 19% 13498 23% 10875 38% sunflower FHD, 1920X1088 12023 10% 11572 13% 10580 21% 8822 34% park_run HD, 1280X720 11025 14% 10709 16% 10045 21% 6994 45% mobcal HD, 1280X720 7403 11% 6938 17% 6387 23% 5458 34% stockholm HD, 1280X720 5865 19% 6126 15% 5713 21% 4802 33% shields HD, 1280X720 6978 6% 6209 16% 5637 24% 4950 33%

First, referring to Table 1, it can be seen that the decoding time differs according to each image (rush hour to shields), and that one image also has a different processing time depending on the processing function of the decoder.

Referring to Table 2 and FIG. 7, it can be seen that the pipelined parallelization method (A) improved the minimum processing time by 12 to 32% based on the processing time of the H.264 / AVC decoder before applying the parallelization method. The 2Dwave technique using two threads (B-1) has been improved by 13 ~ 21%. On the other hand, it can be seen that the 2D wave method (B-2) using three threads improves the performance by 23 to 32%, and the video decoding method C according to the embodiment of the present invention improves the performance by 38 to 53%. It can be seen that.

Referring to Table 3 and FIG. 8, it can be seen that the pipelined parallelization method (A) improved the average processing time by 6 to 19% based on the processing time of the H.264 / AVC decoder before applying the parallelization method. In addition, the 2D wave technique (B-1) using two threads is improved by 13 ~ 19%. On the other hand, it can be seen that the 2Dwave method (B-2) using three threads improves the performance by 21 to 25%, and the video decoding method C according to the embodiment of the present invention improves the performance by 33 to 46%. It can be seen that.

As discussed above, the pipelined parallelization method (A) requires continuous synchronization between threads because the processing speed between MBs differs according to MB type, and processing speeds vary by task according to the work flow. Therefore, the pipeline parallelization method (A) can be regarded as a delay due to thread synchronization. On the other hand, in the 2Dwave method (B), since ED is not parallelized, sequential processing of ED is a bottleneck for improving performance. On the other hand, the video decoding method (C) according to an embodiment of the present invention can divide the video frame into N regions and perform decoding in a relatively fast time by minimizing the synchronization of threads, thereby improving decoding processing performance. Can be.

Next, a multi-threaded video decoder according to an embodiment of the present invention will be described with reference to FIGS. 3 to 6. Since the multi-threaded video decoder according to an embodiment of the present invention operates substantially the same as the multi-threaded video decoding method according to the above-described embodiment of the present invention, the descriptions duplicated above Omit it.

3 to 6, a multi-threaded video decoder according to an embodiment of the present invention may include an input module (not shown), a processing module 200, and first to third memory banks 310 to 330. It may include.

The input module (not shown) may be a module that receives the video frame 100 including the first to Nth regions.

In the multi-threaded video decoding method according to an embodiment of the present invention described above, the processing module 200 generates threads 211 to 241 for each step and provides a video frame 100 provided to an input module (not shown). The module may be a module configured to perform decoding processing on the memory and to store the processing results in the first to third memory banks 310 to 330.

The first to third memory banks 310 to 330 may be memory banks that perform the same functions as in the multi-threaded video decoding method according to an embodiment of the present invention described above.

Such a multi-threaded decoder according to an embodiment of the present invention can improve decoding processing performance while satisfying data dependency according to the H.264 / AVC standard. On the other hand, by reusing the memory banks 310 to 330, efficient use of the memory can also be achieved.

Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above embodiments but may be manufactured in various forms, and having ordinary skill in the art to which the present invention pertains. It will be understood by those skilled in the art that the present invention may be embodied in other specific forms without changing the technical spirit or essential features of the present invention. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

100: video frame 110: macro block
200: processing module 211 to 241: thread
310, 320, 330: memory bank

Claims (1)

Divide the video frame into first to N-th regions each comprising a plurality of macro blocks,
In a first step, a first process is performed on the first area by using a 1-1 thread.
In a second step, a second process different from the first process is performed on the first area using the 2-1 thread, and the first process is performed on the second area using a 2-2 thread. Including performing,
The video frame comprises a video frame encoded according to the H.264 / AVC standard,
The multi-threaded video decoding method of which the number of macro blocks included in the first to Nth regions is the same.

Images