KR20240033618A - Video decoding device, operation method thereof, display device, and video system - Google Patents

Abstract

A video decoding device, a method of operating the same, a display device, and a video system are disclosed. A video decoding device according to the technical idea of the present disclosure includes a plurality of hardware blocks that decode and process a bitstream in a pipeline unit, and are located between the plurality of hardware blocks and process the bitstream in a pipeline unit by each of the plurality of hardware blocks. It includes a decoder including a plurality of pipeline memories that store unit data for each pipeline stage, and a memory that stores image data decoded by the decoder.

Description

Video decoding device, method of operation thereof, display device, and video system {VIDEO DECODING DEVICE, OPERATION METHOD THEREOF, DISPLAY DEVICE, AND VIDEO SYSTEM}

The technical idea of the present disclosure relates to an electronic device, and more specifically, the present disclosure relates to a video decoding device for performing a decoding operation on a pipeline basis, an operating method thereof, a display device, and a video system.

When video or sound is converted to digital data, the amount of digital data converted is quite large. Uncompressed digital data takes up a lot of storage space. Therefore, a technology for compressing the amount of digital data and decompressing the compressed digital data has become necessary.

Recently, according to new video compression standards (VVC (Versatile Video Coding)), for example, VVC (Versatile Video Coding), the processing units of blocks have become more diverse, and motion estimation and compensation methods have become much more complex than conventional technologies, so the requirements for motion compensation are increased. The amount of memory access has increased significantly compared to before. Therefore, an efficient memory access method is required in a video decoder.

The technical idea of the present disclosure provides a video decoding device that performs a decoding operation using a relatively small pipeline memory, a method of operating the same, a display device, and a video system.

A video decoding device according to the technical idea of the present disclosure includes a plurality of hardware blocks that decode a bitstream in a pipeline unit, and are located between the plurality of hardware blocks and operate the pipeline by each of the plurality of hardware blocks. A decoder including a plurality of pipeline memories that store processed unit data for each pipeline stage; and a memory that stores image data decoded by the decoder. The unit data includes the current coding unit (CU: Current CU size information indicating the size of the coding unit (hereinafter “CU”), next CU position information indicating the position of the next CU of the current CU to which each of the unit pixel groups belongs, and scan type of the unit pixel group. Contains the scan type information indicated.

In addition, a method of operating a video decoding device according to the technical idea of the present disclosure includes, in a first pipeline stage, a first hardware block, and first unit data in which a bitstream is divided into pipeline units in the first pipeline stage. Processing and storing first unit data in a first pipeline memory; In the second pipeline stage following the first pipeline stage, the first hardware block processes the second unit data divided into pipeline units in the second pipeline stage, and stores the second unit data in the first pipeline memory. Save to; And at the same time, the second hardware block reads the first unit data stored in the first pipeline memory and processes the first unit data.

In addition, a video system according to the technical idea of the present disclosure includes a video encoding device that provides a bitstream including pipeline area unit information regarding the pipeline area unit; and a video decoding device that decodes the bitstream in pipeline units and outputs decoded video data. A video decoding device includes a plurality of hardware blocks that process unit data divided in pipeline units; and a plurality of pipeline memories having a memory size corresponding to pipeline area unit information to store unit data.

In addition, a display device according to the technical idea of the present disclosure includes a plurality of hardware blocks that decode a bitstream in a pipeline unit, and a plurality of hardware blocks located between the plurality of hardware blocks and provided from each of the plurality of hardware blocks. A decoder including a plurality of pipeline memories that store unit data corresponding to a pipeline area unit for each pipeline stage; a memory that stores image data decoded by a decoder; and a display module that displays an image according to the decoded image data.

According to the technical idea of the present disclosure, by performing a decoding operation in relatively small pipeline units regardless of the video compression method or tree splitting mode, there is an effect of reducing the size and storage capacity of the pipeline memory.

The effects that can be obtained from the embodiments of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be explained to those skilled in the art from the following description. Can be clearly derived and understood. That is, unintended effects resulting from implementing the embodiments of the present disclosure may also be derived by a person skilled in the art from the embodiments of the present disclosure.

1 is a block diagram of a video system according to example embodiments of the present disclosure.
2 is a block diagram of a decoder according to example embodiments of the present disclosure.
3A to 3C are diagrams for explaining a pipeline memory according to example embodiments of the present disclosure.
Figure 4 is a block diagram of a video decoding device according to example embodiments of the present disclosure.
FIG. 5 is a diagram for explaining a decoding processing operation on a pipeline basis according to example embodiments of the present disclosure.
FIG. 6 is a diagram for explaining a coding tree unit and a coding unit according to example embodiments of the present disclosure.
7A to 7E are diagrams for explaining a tree splitting mode according to exemplary embodiments of the present disclosure.
FIGS. 8A to 8E are diagrams for explaining a pipeline area unit in the tree split mode shown in FIGS. 7A to 7E.
FIG. 9 is a diagram illustrating a coding unit and a pipeline area unit in a Versatile Video Coding (VVC) Codec according to example embodiments of the present disclosure.
10A to 10D are diagrams for explaining an operation of processing unit data on a pipeline basis according to example embodiments of the present disclosure.
FIG. 11 is a diagram for explaining the structure of a bitstream according to example embodiments of the present disclosure.
FIG. 12 is a diagram for explaining pipeline area unit number information according to example embodiments of the present disclosure.
FIG. 13 is a diagram for explaining pipeline area unit mode information according to example embodiments of the present disclosure.
FIG. 14 is a flowchart illustrating a method of operating a video decoding device according to example embodiments of the present disclosure.
FIG. 15 is a flowchart illustrating a method of processing unit data on a pipeline basis according to example embodiments of the present disclosure.
Figure 16 is a block diagram of a display device according to example embodiments of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

1 is a block diagram of a video system according to example embodiments of the present disclosure.

Referring to FIG. 1, the video system 10 may include a video encoding device 100 and a video decoding device 200.

The video encoding device 100 may encode video data and transmit an encoded bitstream (BS). Here, encoding may include compression and transcoding. The video encoding device 100 may be implemented with electronic devices such as a computer equipped with a camera capable of taking pictures, a mobile phone, glasses, or a watch. In this case, the video encoding device 100 can encode video data generated by shooting. Alternatively, the video encoding device 100 may receive video data from a server or another electronic device and encode the received video data. Video expressed by video data may be classified as an image, picture, etc.

The video encoding device 100 can encode video data. The video encoding device 100 may compress video data to reduce the capacity of the video data. The video encoding device 100 may quantize converted video data. The video encoding device 100 may generate a bitstream (BS) using quantized video data. The video encoding device 100 may transmit a bitstream (BS) to the video decoding device 200 through wired or wireless communication.

In some embodiments, the bitstream (BS) may include pipeline area unit information regarding the pipeline area unit. An embodiment of the bitstream (BS) will be described later with reference to FIG. 11.

The video decoding device 200 may decode the encoded bitstream (BS) and provide an image to the user. The video decoding device 200 can restore video data by decoding the bitstream (BS). The video decoding device 200 may output decoded video data (DPD). Decoded image data (DPD) may be provided to a device that can display images (eg, a display device).

The video decoding device 200 may include a decoder 210 and a memory 220.

The decoder 210 may restore video data by decompressing the bitstream (BS). The decoder 210 can extract transform coefficients using the received bitstream (BS) and restore video data through inverse transformation and quantization processes on the extracted transform coefficients. Transformation coefficients may be coefficients included in the bitstream (BS) converted into two-dimensional form.

In some embodiments, the decoder 210 may decode the bitstream (BS) on a pipeline basis and output decoded video data (DPD). Here, data decoded on a pipeline basis may be referred to as unit data. Unit data of the present disclosure may correspond to a pipeline area unit. The pipeline area unit may be an area that corresponds to a pipeline unit of a preset size or includes a preset number of unit pixel groups. One or more of these pipeline area units may be included within one CTU. A description of the pipeline area unit is described later with reference to FIGS. 8A to 8E.

Decoder 210 may be implemented by one or more processors. Additionally, the decoder 210 may be a type of program built into the processor.

The memory 220 may store decoded image data (DPD). The memory 220 may store reference image data used by the decoder 210 to compensate for motion. Memory 220 may store data representing reference pixel values used in decoder 210.

The memory 220 can store data and programs that support various functions of the video decoding device 200, can store input/output data, and can store a plurality of application programs running on the video decoding device 200. Or an application), data and commands for operation of the video decoding device 200 may be stored.

The memory 220 stores at least one type of flash memory type, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), and magnetic memory. May include media.

2 is a block diagram of a decoder according to example embodiments of the present disclosure.

Referring to FIG. 2, the decoder 210 may include a plurality of hardware blocks 211, 213, 215, and 217 and a plurality of pipeline memories 212 and 214.

A plurality of hardware blocks 211, 213, 215, and 217 may decode unit data divided on a pipeline basis. Decoding processing may be processing unit data according to the unique function of each hardware block. The plurality of hardware blocks 211, 213, 215, and 217 may be hardware elements in which each block performs a unique function. The number of hardware blocks 211, 213, 215, and 217 may be n. n may be an integer of 2 or more.

Each of the plurality of hardware blocks 211, 213, 215, and 217 may store (or write) processed unit data in the corresponding pipeline memory. In addition, each of the plurality of hardware blocks 211, 213, 215, and 217 may read unit data stored in the corresponding pipeline memory and decode the read unit data. To this end, each of the plurality of hardware blocks 211, 213, 215, and 217 may include an engine performing a unique function and direct memory access (DMA) to access each pipeline memory. there is.

The plurality of pipeline memories 212 and 214 may store unit data processed by the plurality of hardware blocks 211, 213, 215, and 217. The plurality of pipeline memories 212 and 214 may output unit data processed by the plurality of hardware blocks 211, 213, 215, and 217. In some embodiments, at a specific pipeline stage, the plurality of pipeline memories 212 and 214 may store specific unit data while simultaneously outputting other already stored unit data. The pipeline stage may refer to an operation period, an operation phase, etc., during which the decoder 210 performs a decoding operation on a pipeline basis.

A pipeline unit may be a unit of unit data input or output from each pipeline memory. A pipeline unit may be referred to as a Virtual Pipeline Data Unit (VPDU).

There may be m plurality of pipeline memories 212 and 214. m may be an integer less than or equal to n. The plurality of pipeline memories 212 and 214 may be located (or arranged) between the plurality of hardware blocks 211, 213, 215, and 217. For example, the first pipeline memory 212 may be placed between the first hardware block 211 and the second hardware block 213. The second pipeline memory 214 may be disposed between the second hardware block 213 and the third hardware block 215. A pipeline memory (not shown) may also be placed between the n-1th hardware block (not shown) and the nth hardware block 217. In this way, at least one pipeline memory may be placed between two different hardware blocks.

In some embodiments, the plurality of pipeline memories 212 and 214 may have a memory size corresponding to pipeline region unit information included in the bitstream BS to store unit data. For example, pipeline area unit information is of a pipeline area unit corresponding to 32*32 pixels (or referred to as 32*32 pixels) within one coding tree unit. When including size, each of the plurality of pipeline memories 212 and 214 may have a memory size corresponding to 32*32 pixels. However, it is not limited to this.

In some embodiments, the plurality of pipeline memories 212 and 214 may store unit data processed on a pipeline basis by each of the plurality of hardware blocks 211, 213, 215, and 217 for each pipeline stage. there is.

In some embodiments, the plurality of pipeline memories 212 and 214 store unit data corresponding to the pipeline area unit provided from each of the plurality of hardware blocks 211, 213, 215, and 217 to the pipeline stage. You can save each time.

The plurality of pipeline memories 212 and 214 may be implemented as volatile memories such as random access memory (RAM) and static random access memory (SRAM), but are not limited thereto.

As described above, by performing a decoding operation in relatively small pipeline units regardless of the video compression method or tree splitting mode, there is an effect of reducing the size and storage capacity of the pipeline memory.

3A to 3C are diagrams for explaining a pipeline memory according to example embodiments of the present disclosure. Specifically, FIG. 3A is a diagram showing a pipeline memory operating in a specific pipeline stage, FIG. 3B is a diagram showing a pipeline memory operating in the next pipeline stage of a specific pipeline stage, and FIG. 3C is a diagram showing a pipeline memory operating in a specific pipeline stage. This is a diagram showing the pipeline memory operating in the next pipeline stage of the pipeline stage.

Referring to FIG. 3A, pipeline memory 300 may be located between different hardware blocks. For example, with reference to FIG. 2 , the pipeline memory 300 may be a first pipeline memory 212 disposed between the first hardware block 211 and the second hardware block 213. However, it is not limited to this.

In some embodiments, pipeline memory 300 may include a first memory 310 and a second memory 320 . The first memory 310 and the second memory 320 may store unit data. The first memory 310 and the second memory 320 may output stored unit data.

In some embodiments, in a specific pipeline stage, a hardware block disposed at the front of the pipeline memory 300 may process the first unit data UD 1. In addition, the pipeline memory 300 may receive first unit data UD 1 from a hardware block disposed at the front of the pipeline memory 300. For example, assume that the pipeline memory 300 is the first pipeline memory 212. In the first pipeline stage, the pipeline memory 300 may receive first unit data UD 1 processed by the first hardware block 211 . The first hardware block 211 may store first unit data UD 1 in the first memory 310 . In one embodiment, the second memory 320 may be empty. When the second memory 320 is empty, a hardware block (eg, the second hardware block 213) placed at the rear of the pipeline memory 300 may wait. However, it is not limited to this, and in another embodiment, the unit data processed by the hardware block (e.g., the first hardware block 211) placed at the front of the pipeline memory 300 in the previous stage is stored in the second memory. It may also be stored at (320).

Referring to FIG. 3B, in the next pipeline stage of a specific pipeline stage, a hardware block placed at the front of the pipeline memory 300 may process the second unit data (UD 2). The pipeline memory 300 may receive second unit data (UD 2). For example, assume that the pipeline memory 300 is the first pipeline memory 212. In the second pipeline stage, the pipeline memory 300 may receive second unit data UD 2 processed by the first hardware block 211 . The first hardware block 211 may store second unit data UD 2 in the second memory 320 . Meanwhile, a hardware block disposed at the rear of the pipeline memory 300 can read unit data stored in the pipeline memory 300 in a previous pipeline stage. For example, assume that the pipeline memory 300 is the first pipeline memory 212. In the second pipeline stage, the second hardware block 213 may read the first unit data (UD 1) stored in the first memory 310 and process the first unit data (UD 1). In one embodiment, an operation of storing the second unit data UD 2 in the second memory 320 and an operation of outputting the first unit data UD 1 from the first memory 310 may be performed simultaneously. Specifically, for example, the first hardware block 211 processes the second unit data (UD 2) and stores the second unit data (UD 2) in the second memory 320, and at the same time, the second hardware block 213 can simultaneously process the first unit data (UD 1).

Referring to FIG. 3C, in a pipeline stage following the pipeline stage of FIG. 3B, a hardware block disposed at the front of the pipeline memory 300 may process third unit data UD 3. The pipeline memory 300 may receive third unit data (UD 3). For example, assume that the pipeline memory 300 is the first pipeline memory 212. In the third pipeline stage, the pipeline memory 300 may receive third unit data UD 3 processed by the first hardware block 211 . The first hardware block 211 may store third unit data UD 3 in the first memory 310 . In this case, the first unit data UD 1 stored in the first memory 310 in the pipeline stage of FIG. 3A may be deleted. That is, the third unit data UD 3 may be overwritten in the first memory 310 . Meanwhile, a hardware block (e.g., second hardware block 213) disposed at the rear of the pipeline memory 300 reads the second unit data (UD 2) stored in the second memory 320 and reads the second unit data (UD 2) can be processed. At this time, the operation of storing the third unit data (UD 3) in the first memory 310 and the operation of outputting the first unit data (UD 2) from the second memory 320 may be performed simultaneously.

'First', 'Second', and 'Third' described above in FIGS. 3A to 3C are used to distinguish configurations and terms, and are referred to as 'First', 'Second', and 'Third'. The order, etc. is not limited.

Figure 4 is a block diagram of a video decoding device according to example embodiments of the present disclosure.

Referring to FIG. 4, the video decoding device 400 may correspond to the video decoding device 200 shown in FIG. 1. The video decoding device 400 includes a bitstream parsing block 411, an IQT block 412, a motion vector block 413, an intra mode block 414, a motion compensation block 415, an inter prediction block 416, and an intra It may include a prediction and recon block 417, a loop filter block 418, first to ninth pipeline memories 421 to 429, and a memory 430.

Bitstream parsing block 411, IQT block 412, motion vector block 413, intra mode block 414, motion compensation block 415, inter prediction block 416, intra prediction and recon block 417. , and the loop filter block 418 may be included in the plurality of hardware blocks 211, 213, 215, and 217 described above with reference to FIG. 2.

The bitstream parsing block 411 can parse the bitstream (BS). The bitstream parsing block 411 can parse the bitstream (BS) for each syntax. Parsing may include Context Adaptive Variable Length Decoding (CAVLD) and Context Adaptive Binary Arithmetic Decoding (CABAD). The bitstream parsing block 411 may be referred to as an entropy decoding block.

The bitstream parsing block 411 can obtain various information included in the bitstream (BS) by performing entropy decoding on the bitstream (BS). Various information included in the bitstream (BS) includes, for example, quantized coefficients, decoded coding parameters, inter prediction parameters, intra prediction parameters, transformation parameters, quantization parameters, loop filter parameters, and/or other syntax. can do. Inter prediction parameters may include, for example, a reference picture index and a motion vector. Intra prediction parameters may include, for example, an intra prediction mode or index.

In some embodiments, the bitstream parsing block 411 parses the bitstream (BS) on a pipeline basis and stores the parsed results in the first pipeline memory 421 and the second pipeline memory 422, respectively. You can. Unit data processed by the bitstream parsing block 411 may be stored in the first pipeline memory 421 and the second pipeline memory 422. The first pipeline memory 421 may be placed between the bitstream parsing block 411 and the IQT block 412. The second pipeline memory 422 may be placed between the bitstream parsing block 411 and the motion vector block 413.

The IQT block 412 may dequantize the parsed bitstream (BS) and perform inverse transformation on the dequantized result. In some embodiments, the IQT block 412 may perform inverse quantization and inverse transformation on the bitstream (BS) parsed on a pipeline basis. In some embodiments, the IQT block 412 may read unit data stored in the first pipeline memory 421. IQT block 412 may be referred to as an inverse quantization and inverse transform block. In some embodiments, the IQT block 412 may store the results of inverse quantization and inverse transformation in the third pipeline memory 423. Unit data processed by the IQT block 412 may be stored in the third pipeline memory 423. The third pipeline memory 423 may be placed between the IQT block 412 and the intra prediction and recon block 417.

The motion vector block 413 may obtain a motion vector based on unit data obtained from the second pipeline memory 422. In some embodiments, the motion vector block 413 may process unit data divided on a pipeline basis to obtain a motion vector. In some embodiments, the motion vector block 413 may store unit data related to the motion vector in the fourth pipeline memory 424 and the fifth pipeline memory 425, respectively. The fourth pipeline memory 424 may be placed between the motion vector block 413 and the intra mode block 414. The fifth pipeline memory 425 may be placed between the motion vector block 413 and the motion compensation block 415.

The intra mode block 414 may determine prediction information for a video block of the current video slice based on the motion vectors. And, the intra mode block 414 can generate prediction blocks for the current video block being decoded using prediction information. For example, intra mode block 414 may use some of the syntaxes to determine the prediction mode (e.g., intra prediction or inter prediction) used to code the video blocks of a video slice. Additionally, the intra mode block 414 may use some of the syntaxes to determine the inter prediction slice type (e.g., B slice, P slice, or GPB slice). Additionally, the intra mode block 414 may use some of the syntaxes to determine configuration information for one or more of the reference picture lists for the slice. Additionally, intra mode block 414 can use some of the syntaxes to determine motion vectors for each inter encoded video block in a slice. Additionally, intra mode block 414 can use some of the syntaxes to determine the inter prediction state for each inter coded video block in a slice. Additionally, intra mode block 414 can use some of the syntaxes to determine other information for decoding video blocks in the current video slice.

In some embodiments, the intra mode block 414 may process unit data divided by pipeline and store the unit data in the sixth pipeline memory 426. The sixth pipeline memory 426 may be placed between the intra mode block 414 and the intra prediction and recon block 417.

The motion compensation block 415 may generate a predicted image by motion-compensating at least one reference image using a motion vector based on unit data stored in the fifth pipeline memory 425. Since the size of the reference image data relative to the reference image is relatively large, the reference image data may be stored in the memory 430. A motion vector may indicate location information of the most similar area when motion prediction is performed on the current coding unit from a reference image (or reference frame). In some embodiments, the motion compensation block 415 may store unit data corresponding to the predicted image in the seventh pipeline memory 427. The seventh pipeline memory 427 may be placed between the motion compensation block 415 and the inter prediction block 416.

The inter prediction block 416 may perform inter prediction based on unit data stored in the seventh pipeline memory 427. As a result of performing inter prediction, a prediction block for a coding unit may be generated. Here, inter prediction may refer to predicting a coding unit from data of a previously coded picture. In some embodiments, the inter prediction block 416 may store unit data corresponding to the result of inter prediction in the eighth pipeline memory 428. The eighth pipeline memory 428 may be placed between the inter prediction block 416 and the intra prediction and recon block 417.

The intra prediction and recon block 417 performs intra prediction based on a plurality of unit data stored in the third, sixth, and eighth pipeline memories 423, 426, and 428, and the third, sixth, and And the samples may be reconstructed (or restored) by adding a plurality of unit data or sample values stored in the eighth pipeline memories 423, 426, and 428. Intra prediction generally refers to predicting a coding unit from previously coded data of the same picture. In some embodiments, the intra prediction and recon block 417 may store unit data corresponding to the reconstruction result in the ninth pipeline memory 429. The ninth pipeline memory 429 may be placed between the intra prediction and recon block 417 and the loop filter block 418.

The loop filter block 418 may include a deblocking filter (not shown) that reduces blocking phenomenon that may occur when restoring the bitstream (BS). If the flag of the loop filter block 418 is activated, the loop filter block 418 may store decoded image data in the memory 430 after filtering. If the flag of the loop filter block 418 is not activated, the loop filter block 418 may store the restored image data in the memory 430 immediately after inter prediction or intra mode prediction. The loop filter block 418 may be referred to as an in-loop filter.

The memory 430 may store decoded image data and reference image data. The memory 430 may be referred to as a Decoded Picture Buffer (DPB).

The first to ninth pipeline memories 421 to 429 may be included in the plurality of pipeline memories 212 and 214 described above with reference to FIG. 2 .

In some embodiments, the first to ninth pipeline memories 421 to 429 may store current CU size information, next CU location information, and scan type information. The current CU size information may be data indicating the size of the current coding unit to which each unit pixel group corresponding to a pipeline area unit belongs among a plurality of unit pixel groups included in the coding tree unit. A description of the unit pixel group will be provided later with reference to FIGS. 7A to 7E. The next CU location information may be data indicating the location of the next coding unit of the current coding unit to which each of the unit pixel groups corresponding to the pipeline area unit belongs. Scan type information may be data indicating the scan type of a unit pixel group. For example, the scan type may be z-scan. As another example, the scan type may be raster-scan. However, it is not limited to this. In some embodiments, the first pipeline memory 421 and the second pipeline memory 422 may further store data representing decoded information. The third pipeline memory 423 may further store data representing residuals. The fourth to sixth pipeline memories 424, 425, and 426 may further store data representing motion vectors and intra modes. The seventh to ninth pipeline memories 427, 428, and 429 may further store data representing pixel values. In addition, the seventh pipeline memory 427 may further store data representing reference pixel values (or reference images or reference frames).

FIG. 5 is a diagram illustrating a decoding processing operation on a pipeline basis according to exemplary embodiments of the present disclosure.

4 and 5, in the first pipeline stage (PSTG 1), the bitstream parsing block 411 processes the first unit data (UD 1) and generates the first unit data (UD 1). It can be stored in the first pipeline memory 421 and the second pipeline memory 422. Meanwhile, the IQT block 412, motion vector block 413, intra mode block 414, motion compensation block 415, inter prediction block 416, intra prediction and recon block 417, and loop filter block ( 418) can wait.

In the second pipeline stage (PSTG 2), the bitstream parsing block 411 processes the second unit data (UD 2), and the second unit data (UD 2) is stored in the first pipeline memory 421 and the second It can be stored in the pipeline memory 422. The IQT block 412 and the motion vector block 413 each read the first unit data (UD 1) stored in the first pipeline memory 421 and the second pipeline memory 422, and read the first unit data (UD 1) 1) may be processed, and the first unit data UD 1 may be stored in the third to fifth pipeline memories 423, 424, and 425, respectively. Meanwhile, the intra mode block 414, motion compensation block 415, inter prediction block 416, intra prediction and recon block 417, and loop filter block 418 may wait.

In the third pipeline stage (PSTG 3), the bitstream parsing block 411 processes the third unit data (UD 3), and the third unit data (UD 3) is stored in the first pipeline memory 421 and the second It can be stored in the pipeline memory 422. The IQT block 412 and the motion vector block 413 each read the second unit data (UD 2) stored in the first pipeline memory 421 and the second pipeline memory 422, and read the second unit data (UD) 2) may be processed, and the second unit data UD 2 may be stored in the third to fifth pipeline memories 423, 424, and 425, respectively. The intra mode block 414 and the motion compensation block 415 each read the first unit data (UD 1) stored in the third to fifth pipeline memories 423, 424, and 425, and read the first unit data (UD 1) 1) may be processed, and the first unit data UD 1 may be stored in the sixth and seventh pipeline memories 426 and 427, respectively. Meanwhile, the inter prediction block 416, intra prediction and recon block 417, and loop filter block 418 may wait.

In the fourth pipeline stage (PSTG 4), the bitstream parsing block 411 processes the fourth unit data (UD 4), and stores the fourth unit data (UD 4) in the first pipeline memory 421 and the fourth unit data (UD 4). 2 It can be stored in the pipeline memory 422. The IQT block 412 and the motion vector block 413 each process third unit data (UD 3) provided from the first pipeline memory 421 and the second pipeline memory 422, and third unit data (UD 3) UD 3) can be stored in the third to fifth pipeline memories 423, 424, and 425, respectively. The intra mode block 414 and the motion compensation block 415 each process the second unit data (UD 2) provided from the third to fifth pipeline memories 423, 424, and 425, and second unit data (UD 2) UD 2) can be stored in the sixth and seventh pipeline memories 426 and 427, respectively. Likewise, the inter prediction block 416 and the intra prediction and recon block 417 may process the first unit data (UD 1). Meanwhile, the and loop filter blocks 418 may wait.

Likewise, in the fifth pipeline stage (PSTG 5), the bitstream parsing block 411 may process the fifth unit data (UD 5). Each of the IQT block 412 and the motion vector block 413 can process fourth unit data (UD 4). Each of the intra mode block 414 and the motion compensation block 415 may process third unit data (UD 3). The inter prediction block 416 and the intra prediction and recon block 417 may process the second unit data (UD 2). The loop filter block 418 may process the first unit data (UD 1).

In the manner described above, in the following pipeline stages of the sixth pipeline stage (PSTG 6), the seventh pipeline stage (PSTG 7), and the seventh pipeline stage (PSTG 7), hardware blocks (e.g., 411 ~ 418) can operate.

FIG. 6 is a diagram illustrating a coding tree unit (CTU) and a coding unit (CU) (hereinafter “CU”) according to example embodiments of the present disclosure.

Referring to FIG. 6, a picture (PICTURE) may be divided into a plurality of CTUs. In a picture (PICTURE), the size of the area corresponding to each CTU may be constant. However, it is not limited to this. For example, one CTU may contain 64*64 pixels. For another example, one CTU may contain 128*128 pixels. However, it is not limited to this.

One CTU may include multiple CUs. The size and shape of the area corresponding to each CU in a picture may vary. Accordingly, the number of pixels included in each CU may be the same or different for each CU.

7A to 7E are diagrams for explaining a tree splitting mode according to exemplary embodiments of the present disclosure.

Referring to FIGS. 7A to 7E, at least one CU may be included in one CTU depending on the tree splitting mode. Different CUs can be distinguished as thick lines shown in FIGS. 7A to 7E. Different CUs may include the same or different numbers of unit pixel groups (UPXG). In FIGS. 7A to 7E, it is assumed that one CTU includes 64*64 pixels, and one unit pixel group (UPXG) includes 4*4 pixels.

According to the tree splitting mode shown in FIG. 7A, one CTU may include first to fifth CUs (CU 1, CU 2, CU 3, CU 4, and CU 5). The number of unit pixel groups (UPXG) included in the first CU (CU 1) is the number of unit pixel groups (UPXG) included in the third CU (CU 3) and the units included in the fifth CU (CU 5) It may be equal to the number of pixel groups (UPXG). For example, the number of unit pixel groups (UPXG) included in each of the first CU (CU 1), the third CU (CU 3), and the fifth CU (CU 5) is 16*4, and each unit pixel group Since the number of pixels included in (UPXG) is 4*4, the number of pixels included in each of the first CU (CU 1), the third CU (CU 3), and the fifth CU (CU 5) is 64*16. You can. The number of unit pixel groups (UPXG) included in the second CU (CU 2) may be the same as the number of unit pixel groups (UPXG) included in the fourth CU (CU 4). For example, the number of unit pixel groups (UPXG) included in each of the second CU (CU 2) and the fourth CU (CU 4) is 16 * 2, and the second CU (CU 2) and the fourth CU (CU 4) The number of pixels included in each may be 64*8.

According to the tree splitting mode shown in FIG. 7B, one CTU may include first to third CUs (CU 1, CU 2, and CU 3). The number of unit pixel groups (UPXG) included in the first CU (CU 1) may be the same as the number of unit pixel groups (UPXG) included in the third CU (CU 3). For example, the number of unit pixel groups (UPXG) included in each of the first CU (CU 1) and the third CU (CU 3) is 4*16, and the number of unit pixel groups (UPXG) included in each of the first CU (CU 1) and the third CU (CU 3) is 4*16. 3) The number of pixels included in each may be 16*64. The number of unit pixel groups (UPXG) included in the second CU (CU 2) may be 8*16, and the number of pixels included in the second CU (CU 2) may be 32*64.

According to the tree splitting mode shown in FIG. 7C, one CTU may include 5 CUs, and according to the tree splitting mode shown in FIG. 7D, one CTU may include 11 CUs, and FIG. 7E According to the tree splitting mode shown in , one CTU may include 14 CUs.

FIGS. 8A to 8E are diagrams for explaining a pipeline area unit in the tree split mode shown in FIGS. 7A to 7E.

Referring to FIGS. 8A to 8E, a plurality of pipeline area units may be included in one CTU regardless of the tree splitting mode. The pipeline area unit may include unit pixel groups (UPXG) corresponding to a pipeline unit of a preset size. For example, the number of unit pixel groups included in one pipeline area unit may be 64, and the number of pixels included in one pipeline area unit may be 1,024 (or 32*32). However, it is not limited to this. The pipeline area unit may correspond to the memory size of the plurality of pipeline memories 212 and 214 and the first to ninth pipeline memories 421 to 429. Meanwhile, each unit data may be data to be decoded for one pipe area unit.

The number of pipeline area units may be, for example, four, but is not limited thereto. Hereinafter, for convenience of explanation, it is assumed that the number of pipeline area units is 4. The first to fourth pipeline area units (PAU 1, PAU 2, PAU 3, and PAU 4) may be divided as shown in FIGS. 8A to 8E within one CTU.

FIG. 9 is a diagram illustrating a coding unit and a pipeline area unit in a Versatile Video Coding (VVC) Codec according to example embodiments of the present disclosure.

Referring to FIG. 9, VVC, one of the video codecs, may be a recently developed video compression standard. According to the tree splitting mode of VVC, CU units can be further divided as shown in FIG. 9. In the case of "SPLIT_TT_VER" and "SPLIT_TT_HOR" as tree splitting modes of VVC, one CTU may be divided into an odd number of CUs.

10A to 10D are diagrams for explaining an operation of processing unit data on a pipeline basis according to example embodiments of the present disclosure.

It is assumed that the tree splitting method shown in FIGS. 10A to 10D is the tree splitting method shown in FIGS. 7A and 8A. Additionally, it is assumed that the pipeline area units shown in FIGS. 10A to 10D are the first to fourth pipeline area units (PAU 1, PAU 2, PAU 3, and PAU 4) shown in FIG. 8A. Also, it is assumed that the plurality of buffers (BUFFERS) shown in FIGS. 10A to 10D are included in the first pipeline memory 212. And, each of the embodiments shown in FIGS. 10A to 10D includes a first hardware block 211, a first pipeline memory 212, and a first hardware block 211 in each of the first to fifth pipeline stages (PSTG 1 to PSTG 5). and the second hardware block 213. It is assumed that the scan method shown in FIGS. 10A to 10D is z-scan.

Referring to FIGS. 10A to 10D, each pipeline memory may include a plurality of buffers (BUFFERS). For example, with reference to FIG. 2 , each of the pipeline memories 212 and 214 may include a plurality of buffers (BUFFERS). For example, with reference to FIG. 4 , each of the first to ninth pipeline memories 421 to 429 may include a plurality of buffers (BUFFERS). Each buffer may correspond to one address. One address may correspond to a unit pixel group (UPXG). That is, each buffer can store various information about a unit pixel group (UPXG). For example, each of the plurality of buffers (BUFFERS) contains current CU size information (CURRENT CU SIZE), next CU position information (NEXT CU POSITION), scan type information (SCAN TYPE), and other data (OTHER INFO). You can save it. Current CU size information (CURRENT CU SIZE) may include information about the size of the CU to which each unit pixel group (UPXG) corresponding to the pipeline area unit belongs. Next CU position information (NEXT CU POSITION) may include information about the position of the next CU of the current CU to which each unit pixel group (UPXG) corresponding to the pipeline area unit belongs. Scan type information (SCAN TYPE) may include information about the scan type of the unit pixel group (UPXG). Other data (OTHER INFO) may include information about reference pixel values (or reference images, reference frames, etc.), motion vectors, pixel values, prediction modes, etc. Other data (OTHER INFO) may vary for each of the first to ninth pipeline memories 421 to 429.

Referring to FIG. 10A, a processing operation for the first pipeline area unit (PAU 1) may be performed in the first pipeline stage (PSTG 1). Unit data (eg, first unit data UD 1) for the first pipeline area unit PAU 1 may be processed by the first hardware block 211. Additionally, unit data for the first pipeline area unit (PAU 1) may be stored in a plurality of buffers (BUFFERS). Specifically, data for unit pixel groups (UPXG) having numbers indicated in the CTU shown in FIG. 10A may be stored in buffers having respective addresses. For example, information about the unit pixel group (UPXG) indicated as “0” in FIG. 10A may be stored in a buffer with address “0”. The first unit data UD1 is the size of the current CU to which the unit pixel groups UPXG corresponding to the first pipeline area unit PAU 1 (or included in the first pipeline area unit PAU 1) belong. It may include first current CU size information indicating, first next CU position information indicating the position of the next CU, and scan type information.

Referring to FIG. 10B, a processing operation for the second pipeline area unit (PAU 2) may be performed in the second pipeline stage (PSTG 2). Unit data (e.g., second unit data UD 2) for the second pipeline area unit PAU 2 is processed by the first hardware block 211 and stored in a plurality of buffers BUFFERS. You can. The second unit data UD 2 is the current CU to which the unit pixel groups UPXG corresponding to the second pipeline area unit PAU 2 (or included in the second pipeline area unit PAU 2) belong. It may include second current CU size information indicating the size, second next CU position information indicating the position of the next CU, and scan type information.

Referring to FIG. 10C, unit data (e.g., third unit data UD 3) for the third pipeline area unit (PAU 3) is processed in the third pipeline stage (PSTG 3) and stored in a plurality of buffers. It can be stored in BUFFERS. The third unit data UD 3 may include third current CU size information, third next CU position information, and scan type information.

Likewise, referring to FIG. 10D, unit data (e.g., fourth unit data UD 4) for the fourth pipeline area unit PAU 4 is processed in the fourth pipeline stage PSTG 4 to generate a plurality of It can be stored in BUFFERS. The fourth unit data UD 4 may include fourth current CU size information, fourth next CU position information, and scan type information.

FIG. 11 is a diagram for explaining the structure of a bitstream according to example embodiments of the present disclosure.

1 and 11, the bitstream (BS) generated by the video encoding device 100 includes sequence header information (SEQUENCE HEADER), picture header information (PICTURE HEADER), and picture information (PICTURE DATA). can do. Picture header information (PICTURE HEADER) and picture information (PICTURE DATA) may be repeated within the bitstream (BS). Since sequence header information (SEQUENCE HEADER), picture header information (PICTURE HEADER), and picture information (PICTURE DATA) are information defined by the video codec standard, their description may be omitted.

In some embodiments, sequence header information (SEQUENCE HEADER) includes vertical size information (VERTICAL SIZE), horizon size information (HORIZENTAL SIZE), bit per pixel information (BIT PER PIXEL), chroma format information (CHROMA FORMAT), and CTU size. It may include information (CTU SIZE) and pipeline area unit number information (PAU_NUM_IN_CTU). Vertical size information (VERTICAL SIZE), horizon size information (HORIZENTAL SIZE), bit per pixel information (BIT PER PIXEL), chroma format information (CHROMA FORMAT), and CTU size information (CTU SIZE) are information defined by the video codec standard. Therefore, the explanation can be omitted. Pipeline area unit number information (PAU_NUM_IN_CTU) may be data indicating the number of pipeline area units included in one CTU. For example, pipeline area unit number information (PAU_NUM_IN_CTU) may indicate the number of pipeline area units (eg, “2”, “4”, “8”, etc.) within one CTU. Embodiments of the pipeline area unit number information (PAU_NUM_IN_CTU) will be described later with reference to FIG. 12.

In some embodiments, picture header information (PICTURE HEADER) includes intra picture flag information (INTRA PICTURE FLAG), picture Quantization Parameter (QP) information (PICTURE QP), deblock filter enable information (DEBLOCK FILTER ENABLE), and pipe May include line area unit mode information (PAU_MODE). INTRA PICTURE FLAG, picture QP (Quantization Parameter) information (PICTURE QP), and deblock filter enable information (DEBLOCK FILTER ENABLE) are information defined by the video codec standard, so their description is omitted. It can be. Pipeline area unit mode information (PAU_MODE) may be data indicating a method of dividing pipeline area units in one CTU. Embodiments of the pipeline area unit mode information (PAU_MODE) will be described later with reference to FIG. 13.

In some embodiments, picture information (PICTURE DATA) may include tile header information (TILE HEADER) and tile information (TILE DATA). Tile header information (TILE HEADER) and tile information (TILE DATA) may be repeated within picture information (PICTURE DATA). In one embodiment, tile header information (TILE HEADER) may include tile type information (TILE TYPE), tile QP information (TILE QP), tile size information (TILE SIZE), and pipeline area unit mode information (PAU_MODE). You can.

In some embodiments, when the bitstream (BS) includes pipeline area unit information, for example, pipeline area unit number information (PAU_NUM_IN_CTU) and pipeline area unit mode information (PAU_MODE), the decoder 210 The size of each of the pipeline memories included in may correspond to pipeline area unit information. In one embodiment, the size of each pipeline memory depends on the maximum number of pipeline area units, the maximum size of pipeline area units within one CTU, and/or the maximum number of pipeline area units contained in one CU. can be responded to.

In other embodiments, pipeline area unit mode information (PAU_MODE) may be included in a field related to a picture, tile, or CTU.

Although not shown, the bitstream (BS) may further include pipeline area size information indicating the size of the pipeline area unit (e.g., "0:16x16 pixels", "1: 32x32 pixels", etc.) within one CTU. It may be possible.

Although not shown, the bitstream (BS) may further include sequence parameter information, picture parameter information, slice information, transform information, etc., each including syntax used in each layer according to the hierarchical structure of the image.

FIG. 12 is a diagram for explaining pipeline area unit number information according to example embodiments of the present disclosure.

Referring to Figures 11 and 12, the value (VALUE OF PAU_NUM_IN_CTU) of the pipeline area unit number information (PAU_NUM_IN_CTU) can be expressed as a 2-bit parameter. For example, the value (VALUE OF PAU_NUM_IN_CTU) of the pipeline area unit number information (PAU_NUM_IN_CTU) may be “0x00”, “0x01”, “0x02”, or “0x03”. However, it is not limited to this.

If the value (VALUE OF PAU_NUM_IN_CTU) of the pipeline area unit number information (PAU_NUM_IN_CTU) is "0x00", the number of pipeline area units (THE NUMBER OF PAU IN CTU) included in a single CTU may be 1. This may mean that the number of CTUs and the number of PAUs are the same, or that one CTU and one PAU correspond one-to-one.

If the value (VALUE OF PAU_NUM_IN_CTU) of the pipeline area unit number information (PAU_NUM_IN_CTU) is "0x01", the number of pipeline area units (THE NUMBER OF PAU IN CTU) included in a single CTU may be 2. This may mean that one CTU includes two PAUs. That is, one CTU can be divided into two PAUs with the same area.

If the value (VALUE OF PAU_NUM_IN_CTU) of the pipeline area unit number information (PAU_NUM_IN_CTU) is "0x02", the number of pipeline area units (THE NUMBER OF PAU IN CTU) included in a single CTU may be 4. This may mean that one CTU includes 4 PAUs. In other words, one CTU can be divided into four PAUs with the same area.

If the value (VALUE OF PAU_NUM_IN_CTU) of the pipeline area unit number information (PAU_NUM_IN_CTU) is "0x03", the number of pipeline area units (THE NUMBER OF PAU IN CTU) included in a single CTU may be 8. This may mean that one CTU includes 8 PAUs. That is, one CTU can be divided into eight PAUs with the same area.

FIG. 13 is a diagram for explaining pipeline area unit mode information according to example embodiments of the present disclosure.

Referring to FIG. 13, pipeline area unit mode information (PAU_MODE) may indicate a method of dividing pipeline area units in one CTU. In some embodiments, pipeline area unit mode information (PAU_MODE) may refer to a method of dividing one CTU into PAUs having a specific type when the size of the CU is larger than the size of the PAU.

For example, with reference to FIG. 13, pipeline area unit mode information (PAU_MODE) may include quad mode (QUAD), horizon mode (HOR), vertical mode (VER), and other modes (ELSE). However, it is not limited to this.

Quad mode (QUAD) may be a mode that divides one CTU into square pipeline area units. For example, four pipeline area units may be included within one CTU, and the shape of each pipeline area unit may be square.

Horizon mode (HOR) may be a mode that divides the one CTU into rectangular pipeline area units with a relatively long horizontal length. For example, two or more pipeline area units may be included within one CTU, and the shape of each pipeline area unit may be a rectangle with a width longer than the length.

Vertical mode (VER) may be a mode that divides one CTU into rectangular pipeline area units with a relatively long vertical length. For example, two or more pipeline area units may be included within one CTU, and the shape of each pipeline area unit may be a rectangle with a length longer than width.

The other mode (ELSE) may be a mode selected as quad mode (QUAD), horizon mode (HOR), or vertical mode (VER) depending on the type of CU. For example, if the horizontal length of the CU is the same as the vertical length of the CU, that is, if the shape of the CU is square, the other mode (ELSE) may be quad mode (QUAD) or horizon mode (HOR). For another example, if the horizontal length of the CU is longer than the vertical length of the CU, the other mode (ELSE) may be the vertical mode (VER). For another example, if the vertical length of the CU is longer than the horizontal length of the CU, the other mode (ELSE) may be the horizon mode (HOR).

FIG. 14 is a flowchart illustrating a method of operating a video decoding device according to example embodiments of the present disclosure.

Referring to FIG. 14, a method of operating a video decoding device may include a first pipeline stage (S100) and a second pipeline stage (S200).

In the first pipeline stage (S100), the first hardware block processes first unit data in which the bitstream is divided into pipeline units in the first pipeline stage, and stores the first unit data in the first pipeline memory. A saving step is performed (S1000). Step S1000 may be the same as with reference to FIG. 3A.

In the second pipeline stage (S200) following the first pipeline stage (S100), the first hardware block processes the second unit data divided into pipeline units in the second pipeline stage to form the second unit. A step of storing data in the first pipeline memory is performed (S2000). Simultaneously with step S2000, the second hardware block reads the first unit data stored in the first pipeline memory, processes the first unit data, and stores the first unit data in the second pipeline memory. (S2100). Step S2000 and step S2100 may be the same as with reference to FIG. 3B.

In some embodiments, in the first pipeline stage S100, simultaneously with step S1000, the second hardware block reads unit data processed in the pipeline stage prior to the first pipeline stage from the first pipeline memory. , steps for processing unit data are performed. In one embodiment, the unit data being processed may be stored in a second pipeline memory. In another embodiment, in the second pipeline stage S200, simultaneously with steps S2000 and S2100, a third hardware block further performs the step of reading unit data stored in the second pipeline memory and processing the unit data. do.

In some embodiments, the method of operating the video decoding device may further include a third pipeline stage following the second pipeline stage (S200). In the third pipeline stage, the first hardware block processes third unit data divided into pipeline units in the bitstream and stores the third unit data in the first pipeline memory, and at the same time, the second a hardware block reading second unit data stored in a first pipeline memory, processing the second unit data, and storing the second unit data in the second pipeline memory, and at the same time, a third hardware block comprising: A step of reading first unit data stored in the second pipeline memory and processing the first unit data is performed.

As described above, by performing a decoding operation in relatively small pipeline units regardless of the video compression method or tree splitting mode, there is an effect of reducing the size and storage capacity of the pipeline memory.

FIG. 15 is a flowchart illustrating a method of processing unit data on a pipeline basis according to example embodiments of the present disclosure.

Referring to FIG. 15, the method of processing unit data in pipeline units shown in FIG. 15 includes hardware blocks as described above with reference to FIGS. 2, 3A, 3B, 3C, 4, and 5. It can correspond to the operation method of the pipeline memories.

In step S10, a coding tree block (CTB) begins.

In step S20, a CU included in one CTB is started.

In step S30, the hardware block reads the input buffer, processes unit data, and writes to the output buffer. Here, the input buffer may be a buffer of pipeline memory located at the front of the hardware block. The output buffer may be a buffer in the pipeline memory located at the rear of the hardware block.

In step S40, the buffer address is incremented. Increasing the buffer address may mean that the buffer address number increases. Referring to FIG. 10A, the address number may increase from 1 to 2. However, it is not limited to this.

In step S50, it is determined whether the ratio between the buffer address and the pipeline size is 0. Here, the pipeline size may be the size of the pipeline memory, that is, the storage capacity of the pipeline memory.

If the ratio between the buffer address and the pipeline size is not 0 (S50, no), step S30 is performed. If the ratio between the buffer address and the pipeline size is 0 (S50, yes), then in step S60, the sub-pipeline is completed. A sub-pipeline may be a pipeline included within one CU.

In step S70, it is determined whether the CU is at the end. If it is not the end of the CU (S70, no), step S30 is performed. If the end of the CU is correct (S70, yes), in step S80, it is determined whether it is the end of the CTB. If it is not the end of the CTB (S80, no), step S20 is performed. If the CTB is correct (S80, yes), in step S90, the CTB is completed.

Figure 16 is a block diagram of a display device according to example embodiments of the present disclosure.

Referring to FIG. 16, the display device 1400 is a portable communication terminal (mobile phone), a smart phone, a tablet personal computer (PC), a wearable device, a healthcare device, or an IoT (internet of things) device. It may be included in a mobile system such as. However, the display device 1400 is not limited to the above. The display device 1400 may be included in a personal computer, laptop computer, server, media player, or automotive device such as navigation.

The display device 1400 may include a processor 1410, a decoder 1420, a memory 1430, and a display module 1440.

The processor 1410 may control the overall operation of the display device 1400, and more specifically, the operation of other components forming the display device 1400. Such a processor 1410 may be implemented as a general-purpose processor, a dedicated processor, or an application processor.

Processor 1410 may include one or more CPU cores 1110. The processor 1410 may further include at least one controller for controlling the decoder 1420, the memory 1430, and the display module 1440.

The processor 1410 may further include an accelerator, which is a dedicated circuit for high-speed data computation, such as artificial intelligence (AI) data computation. Such accelerators may include a graphics processing unit (GPU), a neural processing unit (NPU), and/or a data processing unit (DPU), etc., and may be a separate chip (physically independent from other components of the processor 1410). It can also be implemented as a chip.

The decoder 1420 and memory 1430 may correspond to the decoder 210 and memory 220 shown in FIG. 1.

The display module 1440 may function as an output device that outputs visual information to each user. The display module 1440 may display an image according to image data decoded by the decoder 1420.

It is obvious to those skilled in the art that the structure of the present disclosure can be modified or changed in various ways without departing from the scope or technical spirit of the present disclosure. In view of the foregoing, it is believed that the present disclosure includes modifications and modifications of the present disclosure if such modifications and variations fall within the scope of the following claims and equivalents.

As described above, exemplary embodiments have been disclosed in the drawings and specifications. Although embodiments have been described in this specification using specific terms, this is only used for the purpose of explaining the technical idea of the present disclosure and is not used to limit the meaning or scope of the present disclosure as set forth in the claims. . Therefore, those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true technical protection scope of the present disclosure should be determined by the technical spirit of the attached patent claims.

Claims (20)

A plurality of hardware blocks for decoding and processing a bitstream on a pipeline basis, and unit data located between the plurality of hardware blocks and processed on a pipeline basis by each of the plurality of hardware blocks through a pipeline. A decoder including a plurality of pipeline memories stored for each stage; and
Includes a memory for storing image data decoded by the decoder,
The unit data is,
The current coding unit (CU: Coding Unit, hereinafter) to which each unit pixel group corresponding to a pipeline area unit belongs among a plurality of unit pixel groups included in a coding tree unit (CTU: Coding Tree Unit, hereinafter “CTU”). Current CU size information indicating the size of the unit pixel group, next CU position information indicating the position of the next CU of the current CU to which each of the unit pixel groups belongs, and scan type indicating the scan type of the unit pixel group. A video decoding device containing information. According to claim 1,
The plurality of hardware blocks are,
Comprising a first hardware block and a second hardware block,
The plurality of pipeline memories are,
A video decoding device comprising a first pipeline memory that stores first unit data processed by the first hardware block and stores second unit data to be provided to the second hardware block. According to clause 2,
The first hardware block processes the first unit data and stores the first unit data in the first pipeline memory, and at the same time, the second hardware block processes the second unit data. , video decoding device. According to clause 2,
The first pipeline memory is,
a first memory storing the first unit data; and
A video decoding device comprising a second memory storing the second unit data. According to clause 4,
In a first pipeline stage, the first hardware block stores the first unit data in the first memory, and the second hardware block reads the second unit data from the second memory,
In a second pipeline stage following the first pipeline stage, the first hardware block stores new unit data in the second memory, and the second hardware block stores the first unit data from the first memory. A video decoding device, characterized in that reading. According to claim 1,
The plurality of hardware blocks are,
Comprising a first hardware block and a second hardware block,
The first hardware block is,
A video decoding device, characterized in that storing the unit data in the bitstream in a pipeline memory between the first hardware block and the second hardware block among the plurality of pipeline memories. According to claim 1,
The plurality of pipeline memories each
A video decoding device comprising a plurality of buffers having a plurality of buffer addresses corresponding to unit pixel groups included in the pipeline area unit. According to claim 1,
The plurality of hardware blocks are,
A bitstream parsing block that parses the bitstream;
An IQT block that dequantizes the parsed bitstream and performs inverse transformation on the dequantized result;
a motion vector block that generates a motion vector based on the parsed bitstream;
an intra-mode block that generates a prediction block based on the motion vector;
a motion compensation block that performs motion compensation based on the motion vector;
an inter prediction block that performs inter prediction based on the motion vector;
an intra prediction and recon block that performs intra prediction based on the motion vector, the prediction block, and a result of the inter prediction; and
A video decoding device comprising a loop filter block that stores the decoded image data in the memory. According to clause 8,
The plurality of pipeline memories are,
a first pipeline memory disposed between the bitstream parsing block and the dequantization block;
a second pipeline memory disposed between the bitstream parsing block and the motion vector block;
a third pipeline memory disposed between the dequantization block and the intra prediction and recon block;
a fourth pipeline memory disposed between the motion vector block and the intra mode block;
a fifth pipeline memory disposed between the motion vector block and the motion compensation block;
a sixth pipeline memory disposed between the intra mode block and the intra prediction and recon block;
a seventh pipeline memory disposed between the motion compensation block and the inter prediction block;
an eighth pipeline memory disposed between the inter prediction block and the intra prediction and recon block; and
A video decoding device comprising a ninth pipeline memory disposed between the intra prediction and recon block and the filter block. In a method of operating a video decoding device including a plurality of hardware blocks and a plurality of pipeline memories,
In the first pipeline stage,
Processing, by a first hardware block, first unit data in which the bitstream is divided into pipeline units in the first pipeline stage, and storing the first unit data in a first pipeline memory;
In a second pipeline stage following the first pipeline stage,
Processing, by the first hardware block, second unit data divided into pipeline units in the second pipeline stage and storing the second unit data in the first pipeline memory; and
At the same time, a second hardware block reads the first unit data stored in the first pipeline memory and processes the first unit data;
A method of operating a video decoding device, including. According to claim 10,
In the first pipeline stage,
At the same time, the second hardware block reads unit data processed in a pipeline stage before the first pipeline stage from the first pipeline memory and processes the unit data;
A method of operating a video decoding device, further comprising: According to claim 11,
In the first pipeline stage,
The step of processing the unit data is,
Store the unit data in a second pipeline memory,
In the second pipeline stage,
The step of processing the first unit data is,
Store the first unit data in the second pipeline memory,
In the second pipeline stage,
At the same time, a third hardware block reads the unit data stored in the second pipeline memory and processes the unit data;
A method of operating a video decoding device, further comprising: According to claim 12,
In a third pipeline stage following the second pipeline stage,
Processing, by the first hardware block, third unit data divided into pipeline units from the bitstream and storing the third unit data in the first pipeline memory;
At the same time, the second hardware block reads the second unit data stored in the first pipeline memory, processes the second unit data, and stores the second unit data in the second pipeline memory. ; and
At the same time, the third hardware block reads the first unit data stored in the second pipeline memory and processes the first unit data;
A method of operating a video decoding device, further comprising: According to claim 10,
In the second pipeline stage,
The step of processing the first unit data is,
Store the first unit data in a second pipeline memory,
In a third pipeline stage following the second pipeline stage,
Processing, by the first hardware block, third unit data divided into pipeline units in the third pipeline stage and storing the third unit data in the first pipeline memory;
At the same time, the second hardware block reads the second unit data stored in the first pipeline memory, processes the second unit data, and stores the second unit data in the second pipeline memory. ; and
At the same time, a third hardware block reads the first unit data stored in the second pipeline memory and processes the first unit data; According to claim 10,
The first unit data is,
The current coding unit (CU: Coding Unit) to which each of the unit pixel groups corresponding to the first pipeline area unit belongs among a plurality of unit pixel groups included in the Coding Tree Unit (CTU) (hereinafter “CTU”). , hereinafter “CU”), first current CU size information indicating the size of the current CU, and first next indicating the location of the next CU of the current CU to which each of the unit pixel groups corresponding to the first pipeline area unit belongs. Contains CU position information and scan type information indicating the scan type of the unit pixel group,
The second unit data is,
Second current CU size information indicating the size of the current CU to which each of the unit pixel groups corresponding to a second pipeline area unit different from the first pipeline area unit among the plurality of unit pixel groups belongs, the second pipe An operation of a video decoding device, characterized in that it includes second next CU position information indicating the position of the next CU of the current CU to which each of the corresponding unit pixel groups corresponding to the line area unit belongs, and the scan type information. method. a video encoding device that provides a bitstream including pipeline area unit information regarding the pipeline area unit; and
A video decoding device that decodes the bitstream in pipeline units and outputs decoded video data,
The video decoding device,
a plurality of hardware blocks that process unit data divided into pipeline units; and
A video system comprising a plurality of pipeline memories having a memory size corresponding to the pipeline area unit information for storing the unit data. According to claim 16,
The pipeline area unit information is,
A video system comprising pipeline area unit number information indicating the number of pipeline area units included in one Coding Tree Unit (CTU) (hereinafter “CTU”). According to claim 17,
The pipeline area unit information is,
The video system further includes pipeline area unit mode information indicating a method of dividing the pipeline area unit in the one CTU. According to clause 18,
The pipeline area unit mode information is,
Quad mode, which divides the one CTU into square pipeline area units, Horizon mode, which divides the one CTU into rectangular pipeline area units with a relatively long horizontal length, and Horizon mode, which divides the one CTU into rectangular pipeline area units with a relatively long vertical length A video system comprising a vertical mode divided into rectangular pipelined area units having . According to claim 16,
The unit data is,
Current CU size information indicating the size of the current CU to which each of the unit pixel groups corresponding to the pipeline area unit belongs among a plurality of unit pixel groups included in one CTU, and the current CU to which each of the unit pixel groups belongs. A video system comprising next CU position information indicating the position of the next CU, and scan type information indicating the scan type of a unit pixel group.

Images