KR101425620B1 - Method and apparatus for video decoding based on a multi-core processor - Google Patents

Abstract

The present invention relates to a method and apparatus for more effectively using system resources in a video decoding process requiring a large amount of computation in a multicore processor environment.

A multicore processor apparatus according to an exemplary embodiment of the present invention includes a video decoder module including functional modules for performing video decoding, a memory for storing an input bitstream and loading the functional modules, And a plurality of cores for performing video decoding on the input bitstream by using the plurality of cores, wherein when an idle time occurs in the first core among the cores during the video decoding, And a second core having a residual operation related to video decoding is assigned to the first core to perform some operations among the remaining operations to reduce the idle time.

Video coding, multicore processor, symbol decoding, motion compensation, dependency

Description

TECHNICAL FIELD The present invention relates to a video decoding method and apparatus based on a multi-core processor,

The present invention relates to a video decoding technique, and more particularly, to a method and apparatus for more effectively using system resources in a video decoding process requiring a large amount of computation in a multicore processor environment.

As information and communication technologies including the Internet are developed, not only text and voice but also video communication are increasing. Conventional character - oriented communication methods are not sufficient to satisfy various needs of consumers. Accordingly, multimedia services capable of accommodating various types of information such as text, images, and music are increasing. The amount of multimedia data is so large that it needs a large capacity storage medium and requires a wide bandwidth in transmission. Therefore, it is necessary to use a compression coding technique to transmit multimedia data including characters, images, and audio.

The basic principle of compressing data is the process of eliminating the redundancy of data. Among these data, in particular, since the video data has a very large capacity, efficient compression is more important than other kinds of multimedia data.

The basic principles of video compression are spatial redundancy such that the same color or object is repeated in one picture (frame), temporal redundancy such as a case in which adjacent frames in the moving picture frame hardly change, And a method of eliminating perceptual redundancy in consideration of redundancy or insensitivity to high frequency of human visual and perceptual ability. In a general video coding method, temporal redundancy is removed by temporal filtering based on motion compensation, and spatial redundancy is removed by a spatial transform.

Conventionally, such a video coding or decoding operation is generally performed by a single-core processor. However, in recent years, as a multi-core processor with a strong performance has been popularized, the utilization of multi-core processors is increasing in the field of consuming system resources such as video coding / decoding.

A multicore processor refers to an integrated circuit that combines more than two cores to provide more power, reduce power consumption, and handle multiple tasks more efficiently at the same time. Multicore processors are often compared to when two or more independent processors are installed in a computer. However, in the case of multicore processors, the advantage is that the two processors are actually plugged into a single socket, which makes the connections between the processors faster. Theoretically, a dual-core processor should double in performance, but in fact, a dual-core processor is about 1.5 times better in performance than a single-core processor.

Currently, the growth of multicore processor-related industries is accelerating because single-core processors are believed to reach near-physical limits in terms of complexity and speed. There are companies such as AMD, ARM, and Intel that are producing or related to multicore processor products, which are predicting that multicore processors will dominate most of the future processor market and are spurring the development.

Conventional techniques for performing video decoding using such a multicore processor include a functional partitioning method and a data partitioning method.

1 and 2 are views for explaining the functional division method. As shown in FIG. 1, in general, for video decoding, a processor may perform data reading, preprocessing / initialization, entropy decoding (hereinafter referred to as ED), inverse quantization And various details such as inverse transform (hereinafter referred to as IT), intra prediction (hereinafter referred to as IP), motion compensation (hereinafter referred to as MC), deblocking, Function.

In the functional partitioning method, a plurality of cores constituting one processor are each determined to take only predetermined functions. For example, core # 2 only performs an entropy decoding (ED) function, and core # 4 only performs a deblock function. When the multiple cores are functionally divided as described above, an imbalance occurs between the arithmetic operations 21, 22, 23, and 24, which are performed by the cores, as shown in FIG. Particularly, core # 3 acts as a critical path due to a relatively heavy load, which causes degradation of the overall performance of the processor. As described above, although the functional partitioning method is easy to implement, parallel processing is difficult because each core processes the divided functions at the same time, and the whole performance of the processor is not utilized.

3 is a diagram for explaining the data division method. As shown in FIG. 3, the data division method is a method of dividing one picture 30 into a plurality of areas and assigning them to respective cores. For example, one picture is divided into quadrants of the same size, and quadrants are processed by corresponding cores.

Thus, the data partitioning scheme ensures high parallelism in simple data processing. However, if there is a dependency between data processing processes, the implementation becomes complicated and there is a disadvantage that the performance is drastically degraded because an additional task (prediction of the relationship between the division size of the data and the operation load) is required. Also, each core of a multicore processor must have full functionality for video decoding, making it inefficient for use of system resources (eg, local storage).

In particular, the H.264 standard, which is widely used in recent years, has a higher computational complexity and higher dependency on functions than decoders of other standard standards. Thus, the performance of a multicore processor can not be exhibited in the conventional methods as described above.

An object of the present invention is to improve video decoding performance by sharing independent tasks that do not depend on each other among multicores in performing video decoding using a multicore processor.

The technical objects of the present invention are not limited to the technical matters mentioned above, and other technical subjects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a multi-core processor including: a video decoder module including functional modules for performing video decoding; A memory for storing an input bit stream and loading the function modules; And a plurality of cores for performing video decoding on the input bitstream using the functional modules, wherein during the video decoding, the first core among the cores If so, a second core with remaining work on video decoding allocates some of the remaining work to the first core to reduce the idle time.

According to another aspect of the present invention, there is provided a video decoding method based on a multicore processor, comprising: loading function modules for storing an input bitstream and performing video decoding; Generating jobs using the input bitstream and function modules, and queuing the generated jobs in a buffer to distribute the jobs to the cores according to a corresponding function; Performing video decoding on the input bitstream using the functional modules, wherein the video decoding is performed by a multicore processor having a plurality of cores; And when a idle time occurs in a first one of the cores during the video decoding, a second core having a residual operation related to video decoding allocates a part of the operations from the remaining operations to the first core .

According to the present invention, performance of video decoding can be improved by providing load balancing between cores in a multicore processor environment.

In addition, according to the present invention, a functional module of video decoding can be dynamically allocated on a core-by-core basis since dependencies existing between major operations are considered.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

4 and 5 are diagrams for explaining the concept of the dynamic load balancing technique according to the present invention. 4 is a diagram illustrating a process in which a multi-core processor performs video decoding in the related art. Here, the core # 1 and the core # 2 only handle entropy decoding (ED), inverse quantization (IQ) and inverse transform (IT) Core # 4 is only responsible for the diblock.

In this case, the core # 3, the operation is completed at t _1, core # 1 and core # 2 in t ₂ is complete, but the core # 3 until the operation is completed at t ₃ is the next picture (or the number of macro Blocks) and is in an idle state. As a result, it does not fully utilize the overall performance of multicore processors.

FIG. 5 shows an example in which the dynamic load balancing method according to the present invention is applied to the environment of FIG. If a functional partitioning scheme is applied to a multi-core processor as shown in FIG. 4, idle time occurs as described above. However, if idle time occurs in the first core, Distribution can eliminate or reduce this idle time.

Core # 4 informs Core # 3 when it is idle at t-- ₁ . Then, Core # 3 allocates some of its remaining operations to Core # 4. Of course, Core # 3 can assign some tasks to Core # 1 and Core # 2 as well as Core # 4. Thus, if the overloading of a particular core is dynamically balanced to the other cores, then all cores at t ₄ can complete their work simultaneously.

6 is a diagram illustrating a video decoding process according to an embodiment of the present invention.

The symbol decoder 61 performs lossless decoding on the input bitstream, and obtains motion vectors and texture data. The lossless decoding includes Huffman decoding, arithmetic decoding, and variable length decoding. In general, the motion vector for a particular macroblock depends on the motion vector of the neighboring macroblock. That is, the motion vector of the specific macroblock can not be obtained without obtaining the motion vector of the neighboring macroblock. If the data having dependency is processed in different cores, parallel processing becomes impossible. Therefore, in the dynamic load balancing of the load of the present invention, it is preferable that data having mutual dependency is processed in one core . ≪ / RTI >

The obtained texture data is provided to an inverse quantization unit 62, and the obtained motion vector is provided to a motion compensation unit 65. [

On the other hand, the dequantizer 62 dequantizes the texture data supplied from the symbol decoder 61. The dequantization process is a process of recovering a value matched with the index generated in the quantization process using the quantization table used in the quantization process.

The inverse transform unit 63 performs an inverse transform on the inverse quantized result. Specific methods of such inverse transform include inverse DCT transform and inverse wavelet transform. The result of the inverse transformation, that is, the reconstructed high-frequency image, is supplied to the adder 66.

The motion compensation unit 65 performs motion compensation on at least one reference frame (which has been restored and stored in the buffer 64) using the motion vector for the current macroblock provided from the symbol decoder 61, . In the case where such motion compensation is performed in units of 1/2 pixel or 1/4 pixel, a very large amount of calculation is required in the interpolation process for generating the prediction image. In addition, when motion compensation is performed using two reference frames, an average of motion compensated macroblocks is calculated. At this time, there is a dependency between the macroblocks. That is, these macroblocks need to be processed in a single core.

The adder 66 adds the high-frequency image provided from the inverse transformer 63 and the generated predictive image to reconstruct the image of the current macroblock. The deblocking unit 67 removes block artifacts of the reconstructed image by applying a diblock filter to the reconstructed image. Generally, since the reconstructed image is processed in units of macroblocks, noise is generated at the boundary of macroblocks, which is called block artifact. This block artifact tends to increase as the compression rate of video data increases. The reconstructed image obtained through the diblock filter is temporarily stored in the buffer 64 and used for restoration of other images.

On the other hand, not all macroblocks are restored through motion compensation. In some macroblocks, they are coded through intra prediction (IP) (this is referred to as intra macroblock). Intra prediction is a technique for restoring a current macroblock using images of adjacent macroblocks in the same frame. In this case, since the current macroblock has a dependency with the other macroblock, it needs to be processed in a single core.

7 is a block diagram of an overall system according to an embodiment of the present invention. Such as a TV, a set top box, a desktop, a laptop computer, a palmtop computer, a personal digital assistant (PDA), a video or image storage device (e.g., a video cassette recorder (VCR), a digital video recorder (DVR) Etc.). In addition, the system may be a combination of the above devices, or the device may be included as part of another device. The system may include at least one video source 71, at least one input / output device 72, a multicore processor 110, a memory 120, and a display device 73.

Video source 71 may represent a TV receiver, VCR, or other video storage device. The source 71 may be connected to the server 71 by using the Internet, a wide area network (WAN), a local area network (LAN), a terrestrial broadcast system, a cable network, a satellite communication network, Lt; RTI ID = 0.0 > video < / RTI > In addition, the source may be a combination of the above networks, or indicating that the network is included as part of another network. The video source 71 also means a path for obtaining video data, but also means a bit stream itself compressed by a predetermined video compression algorithm.

The input / output device 72, the multicore processor 110, and the memory 120 communicate via the communication medium 76. The communication medium 76 may be a communication bus, a communication network, or one or more internal connection circuits. The input video data received from the source 71 may be processed by the multicore processor 110 in accordance with one or more software programs stored in the memory 120. The multi-core processor 110 refers to an integrated circuit having two or more cores combined to perform more efficiently and at the same time, with more performance and lower power consumption.

 And may be executed by the multicore processor 110 to produce output video provided to the display device 73. The display device 73 may be a liquid crystal display (LCD), a light-emitting diode (LED), an organic light-emitting diode (OLED), a plasma display panel (PDP), or any other type of image And can be implemented as display means.

The software program stored in the memory 120 includes a video decoder module for performing the decoding process shown in FIG. The video decoder may be stored in the memory 120, read from a storage medium such as a CD-ROM or a floppy disk, or downloaded from a predetermined server through various networks. May be replaced by hardware circuitry by the software, or may be replaced by a combination of software and hardware circuitry. In addition, the memory 120 includes a buffer or a queue for temporarily storing data before being processed.

8 is a block diagram illustrating the configuration of a multicore processor device 100 that provides dynamic load balancing, in accordance with one embodiment of the present invention.

The multicore processor apparatus 100 may include a multicore processor 110, a memory 120, a buffer 130, and a video decoder module 140. The multi-core processor device 100 basically uses the functional division method as shown in FIG. That is, the plurality of cores are primarily responsible for only predetermined functions. However, in order to eliminate or reduce the idle time, each of the cores notifies the other cores of the completion of the work they are in charge of, and the other cores allocate some of the work in progress to the completed core .

The video decoder module 140 is video decoding software for performing the video decoding process as shown in FIG. The video decoder module 140 may comprise functional modules such as a symbol decoder, an inverse quantization unit, an inverse transform unit, and a motion compensation unit. The video decoder module may be video decoding software according to a predefined video coding / decoding standard, such as MPEG-2, MPEG-4, H.264.

The memory 120 stores the input bitstream and loads functional modules of the video decoder module 140. [ The bitstream is compressed video data at a video encoder (not shown). The memory 120 may be a nonvolatile memory device such as ROM, PROM, EPROM, EEPROM, flash memory, or a volatile memory device such as a RAM, a hard disk, Disk, or any other form known in the art.

The buffer 130 is a storage for temporarily storing data of tasks or image blocks that the multicore processor 110 has to process. The buffer 130 may be implemented as part of the memory 120 or may be implemented as a separate storage unit from the memory 120. [

The multicore processor 110 is composed of at least two or more cores. In the embodiment of FIG. 8, although the multicore processor 110 is composed of three cores, it is needless to say that the multicore processor 110 may be implemented by two or more cores. The first core 111 is responsible for symbol decoding, and the second core 112 is for decoding symbols. In this case, the first core 111 and the second core 112 are divided into three functions, i.e., symbol decoding, inverse characterization and inverse transformation, And the third core 113 is responsible for inverse quantization and inverse transform (IQ / IT).

The first core 111 reads the functional module and the bit stream that are loaded into the memory 120 and queues the work to the buffer 130. The minimum unit of work queued in the buffer 130 is a sub-block that constitutes a macroblock and to which a motion vector is assigned. The cores 111, 112, and 113 perform operations corresponding to functions assigned to the buffers 130, respectively, according to the functional division method. However, when a core becomes idle, it is necessary to allocate some of the operations of the core in operation to the idle core.

Figure 9 below is a sequence diagram illustrating a concrete example of implementing dynamic load balancing between cores.

The first core 111 transmits a control message Do_MC (N) to the second core 112 to instruct the second core 112 to perform the motion compensation operation (S2). The number N represents the number of image blocks (at least one macro block or a sub-block in the present invention) to be processed stored in the buffer 130. Then, the first core 111 transmits a control message (Do_IQ / IT) instructing the third core 113 to perform inverse quantization and inverse transformation (S4).

Hereinafter, each of the cores performs the respective tasks. That is, the first core 111 performs symbol decoding (S6), the second core 112 performs motion compensation (S8), and the third core 113 performs dequantization and inverse transform S10). However, if the symbol decoding in S6 is the symbol decoding process for the next image block in the current image block, the symbol decoding for the current image block may be already completed have. The first core 111 queues the jobs to be performed by the other cores in the buffer 130 upon completion of symbol decoding. In this case, the first core 111 may queue all the jobs in a single buffer, but it is also possible to separately create a separate buffer for each core, and separate and queue the jobs required for each core according to the function.

When the inverse quantization and inverse transformation operations in the third core 113 are completed while the respective cores are working according to the functional division method, the third core 113 corresponds to the first core 111 And transmits a control message (IQ / IT_Done) indicating that the operation is completed to the first core 111 (S12). Thereby, the third core 113 becomes idle.

The third core 113 transmits a signal (SendSignal (IQ / IT_Done)) informing the second core that the inverse quantization and inverse transformation are completed to the second core 112 (S14).

The second core 112 divides a part of the work p in the current remaining work S16 and sends a signal SendSignal (Do_MC (p)) to inform the third core 113 to perform a part of the work It is possible to arbitrarily determine how much work the second core 112 has allocated to the other cores among the remaining jobs. However, the remaining work amount can be arbitrarily set to (the number of idle cores + 1) For example, if the amount of work related to the motion compensation to be performed is N, that is, if the performance of the core is N, If the amount of work completed by the second core 112 is m, the second core 112 can allocate 1/2 of Nm, which is the remaining work amount, to the third core 113.

Thereafter, the second core 112 and the third core 113 respectively perform the assigned motion compensation operations (S20, S22). At this time, each of the cores 112 and 113 extracts and executes a job assigned to itself in the job queued in the buffer 130. For this purpose, the core may set a check bit in advance for jobs to be performed among the queued jobs.

The third core 113 transmits a signal (SendSignal (MC_Done)) notifying the second core 112 to the second core 112 (S24).

Then, when the motion compensation operation assigned to the second core 112 is completed, the second core 112 transmits a control message (MC_Done) indicating that the entire motion compensation operation is completed to the first core 111 (S26).

In the above, an example in which dynamic load balancing according to an embodiment of the present invention is applied to the existing functional division method has been described. However, there is no problem if all tasks queued in the buffer 130 are independent tasks that do not have dependencies on each other. However, if there is a job having a dependency among jobs queued in the buffer 130, the process of FIG. 9 needs to be slightly modified.

First, the first core 111 extracts a job having a dependency when the job is queued in the buffer 130 and displays a separate check bit, or jobs having interdependency and no dependency are queued in a separate buffer do. For example, it is assumed that the third core 113 is in the idle state and the queued jobs that the second core 112 should assume are as shown in FIG. At this time, the tasks 3 to 5 are interdependent tasks, so it is preferable that they are processed in one core. Accordingly, the second core 112 can allocate tasks 6 to 9 and tasks 1 and 2, which have interdependencies, from among a total of 12 remaining tasks, and assign the remaining six tasks to the third core 113 . Alternatively, conversely, the former may be assigned to the third core 113, and the latter may be assigned to itself.

In this way, even when tasks having interdependencies are allocated to one core, the remaining independent tasks can be appropriately allocated to balance the load as a whole.

As described above, the multicore processor apparatus 100 according to the present invention applies functional division and dynamic load balancing to minimize the idle time that occurs in some cores. However, not all cores may concurrently work on the same video block at the same time. 11 is a diagram illustrating an embodiment of the present invention in terms of a pipeline.

In FIG. 11, operations performed on the same image block are shaded. That is, when core # 1 completes symbol decoding on the current image block in interval 1, core # 2 queues a plurality of jobs into the buffer based on the decoded symbols in interval 2. The core # 3 performs the motion compensation operation on the current image block in the interval 3, confirms that the idle time has occurred in the core # 2, and allocates some motion compensation operations to the core # 2.

When the motion compensation is completed after the interval 3 has elapsed, the core # 1 performs processes such as inverse quantization (IQ), inverse change (IT), and intra prediction (IP) on the current image block at a specific time in the interval 4 . Finally, the core # 4 performs a deblocking process on the current image block in the interval 5 to restore one image block (first image block) by removing block artifacts of the current image block. When the interval 6 has elapsed due to the similar pipeline structure, another image block (second image block) is reconstructed.

Experimental results of performing video decoding according to the conventional functional division method using a multicore processor and experimental results of performing video decoding according to the dynamic load allocation method according to the present invention are summarized in Table 1 below.

Core # 1 Core # 2 Conventional technology 13.45 ms / frame 39.95 ms / frame Invention 26.40 ms / frame 27.01 ms / frame

In the experiment, the core # 2 performs only motion compensation and the core # 1 performs operations other than motion compensation. For the motion compensation, a motion compensation technique such as Quarter Pixel motion compensation, which has a large computational load, is used. In Table 1, according to the related art, when Core # 2 completes the operation, idle time of 26.5 ms occurs in Core # 1, but according to the present invention, only idle time of 0.61 ms occurs in Core # 1 .

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. It is therefore to be understood that the embodiments described above are in all respects illustrative and not restrictive.

1 is a view for explaining a work flow according to a conventional functional division method.

2 is a diagram showing a load imbalance generated in a conventional functional division method.

3 is a diagram for explaining a conventional data division method.

FIG. 4 is a diagram illustrating a process in which a multi-core processor performs video decoding.

FIG. 5 is a diagram showing an example in which dynamic load balancing of a load according to the present invention is applied in the environment of FIG. 4; FIG.

6 is a diagram illustrating a video decoding process according to an embodiment of the present invention.

7 is a block diagram of an overall system according to an embodiment of the present invention.

8 is a block diagram illustrating the configuration of a multicore processor device that provides dynamic load balancing, in accordance with one embodiment of the present invention.

Figure 9 is a sequence diagram illustrating a specific example of implementing dynamic load balancing between cores.

FIG. 10 is a diagram showing an example in which queued jobs in a buffer include tasks that are dependent and independent; and FIG.

11 is a view for explaining an embodiment of the present invention in a pipeline concept;

 (Reference Numerals for Main Parts of the Drawings)

61: Symbol decoder 62: Inverse quantization unit

63: Inverse transform unit 65: Motion compensation unit

66: adder 67:

71: Video source 72: Input / output device

73: Display device 100: Multicore processor device

110: multicore processor 120: memory

130: buffer 140: video decoder module

Claims (19)

A video decoder module composed of functional modules for performing video decoding; A memory for storing an input bit stream and loading the function modules; And And a multicore processor configured by a plurality of cores for performing video decoding on the input bitstream using the function modules, Wherein when the idle time occurs in the first core among the cores during the video decoding, the first core transmits a signal indicating that the idle time has occurred to the second core, Wherein the second core having the idle time divided by the number of cores in which the idle time has occurred is divided by the number of cores in which the idle time has occurred, Processor device. The method according to claim 1, Wherein the functional module is a functional module of the H.264 standard. 2. The method of claim 1, A symbol decoding module, an inverse quantization module, an inverse transform module, and a motion compensation module. 2. The system of claim 1, wherein the multicore processor Further comprising a third core for queuing the generated jobs in a buffer to generate jobs using the input bitstream and function modules and to distribute the jobs to the cores according to a corresponding function, Device. 5. The method of claim 4, wherein the third core And separates the jobs having dependencies among the jobs and queues them into the buffer or the separate buffer. delete delete 2. The method of claim 1, wherein the first core And transmits a signal to the second core indicating that the operation is completed, when the assigned task is completed. The method according to claim 1, Wherein the interdependent tasks among the remaining operations are allotted to one core. The method according to claim 1, Wherein the residual operation belongs to a motion compensation operation. Loading functional modules for storing the input bitstream and performing video decoding; Creating jobs using the input bitstream and function modules, and queuing the generated jobs in a buffer to distribute the jobs to cores according to a corresponding function; Performing video decoding on the input bitstream using the functional modules, wherein the video decoding is performed by a multicore processor having a plurality of cores; And When the idle time occurs in the first core among the cores during the video decoding, transmitting a signal indicating that the idle time has occurred to the second core; And And a second core having a residual operation relating to video decoding divided by the number of cores in which the idle time has occurred by the number of cores in which the idle time has occurred, to a core where the second core and the idle time have occurred A video decoding method based on a multicore processor. 12. The method of claim 11, Wherein said functional module is a functional module of H.264 standard. 12. The method of claim 11, A video decoding method based on a multicore processor including a symbol decoding module, an inverse quantization module, an inverse transform module, and a motion compensation module. 14. The method of claim 13, wherein the step of queuing Dividing jobs having dependencies among the jobs and queuing them into the buffer or a separate buffer. delete delete 12. The method of claim 11, And when the assigned task is completed, transmitting a signal to the second core indicating that the task is completed. 12. The method of claim 11, Wherein the interdependent tasks among the remaining tasks are all assigned to one core. 12. The method of claim 11, Wherein the residual operation belongs to a motion compensation operation.

Images