RU2773431C1 - Method and apparatus for processing a video signal based on a history-based prediction of the motion vector - Google Patents

Abstract

FIELD: data encoding.

SUBSTANCE: invention relates to the field of encoding for performing inter-frame prediction using a history-based prediction of the motion vector. Proposed is a method for processing video signals based on an inter-frame prediction, including the stages of: configuring a list of candidates for merging based on the spatial adjacent block and the temporal adjacent block of the current block; adding a history-based candidate for merging of the current block to the list of candidates for merging; obtaining a merge index indicating the candidate for merging, used to make an inter-frame prediction of the current block, in the list of candidates for merging; forming a sample of prediction of the current block based on the information about the movement of the candidate for merging, indicated by the merge index; and updating the list of history-based candidates for merging based on the information about the movement.

EFFECT: provided limitation on the redundancy check in order to increase the efficiency of prediction when adding an HMVP candidate to the list of candidates for merging (or the list of AMVP candidates).

14 cl, 42 dwg

Description

Technical field

[0001] An embodiment of the invention relates to a method and apparatus for processing a video signal based on inter-picture prediction, and more specifically, to a method and apparatus for performing inter-picture prediction using history-based motion vector prediction.

State of the art

[0002] Compression coding means a series of signal processing techniques for transmitting digitally represented information over a communication link, or techniques for storing information in a form suitable for an information storage medium. A medium including picture, image, audio, etc. may be the target of compression encoding, and in particular, a technique for performing compression encoding on a picture is referred to as video image compression.

[0003] Next generation video content is expected to have the characteristics of high spatial resolution, high frame rate, and high dimensionality of scene representation. Processing such content will result in a dramatic increase in memory footprint, memory access speed, and processing power.

[0004] Accordingly, the development of an encoding tool for efficient processing of next generation video content is required.

The essence of the invention

Technical problem

[0005] An object of an embodiment of the invention is to propose redundancy check constraints to improve prediction performance when an HMVP candidate is added to a merge candidate list (or AMVP candidate list).

[0006] Another object of an embodiment of the invention is to provide a method for determining the size of an HMVP table in order to efficiently apply an HVMV candidate to a merge candidate list (AMVP candidate list).

[0007] The technical objectives to be achieved by the embodiment of the invention are not limited to the above technical objectives, and other technical objectives not described above can be clearly understood by a person skilled in the relevant field of technology to which the invention pertains from the following description.

Technical solution

[0008] In an aspect of an embodiment of the invention, a method for processing a video signal based on history-based motion vector prediction may comprise: configuring a merge candidate list based on a spatial neighbor block and a temporal neighbor block of the current block; adding a history-based merge candidate of the current block to the list of merge candidates; obtaining a merge index indicating a merge candidate that is used for inter-picture prediction of the current block in the merge candidate list, generating a prediction sample of the current block based on motion information of the merge candidate indicated by the merge index; and updating the list of history-based merge candidates based on the motion information, wherein the history-based merge candidate is added to the list of merge candidates when it has different motion information than that of the predetermined merge candidate of candidates. for the merger included in the list of candidates for the merger.

[0009] In an embodiment, the size of the history-based merge candidate list may be determined based on the maximum number of merge candidates in the merge candidate list.

[0010] In an embodiment, the size of the history-based merge candidate list may be defined as a value equal to 1, which is subtracted from the maximum number of merge candidates in the merge candidate list.

[0011] In an embodiment, the size of the list of history-based merge candidates may be defined as 5.

[0012] In an embodiment, a history-based merge candidate may be added to a list of merge candidates when the history-based merge candidate includes movement information other than the movement information of a specific number of predetermined merge candidates among the candidates. for the merger included in the list of candidates for the merger.

[0013] In an embodiment, a history-based merge candidate may be added to a list of merge candidates when the history-based merge candidate includes motion information other than motion information of specific spatial merge candidates included in the list. merger candidates.

[0014] In an embodiment, a history-based merge candidate may be derived from a predetermined number of candidates in a list of history-based merge candidates.

[0015] In another aspect of the embodiment, the video signal processing apparatus based on inter-picture prediction may comprise: a memory configured to store video signals; and a processor integrated with the memory, the processor being configured to: configure a merge candidate list based on a spatial neighbor block and a temporal neighbor block of the current block; adding a history-based merge candidate of the current block to the list of merge candidates; obtaining a merge index indicating a merge candidate that is used for inter-picture prediction of the current block in the merge candidate list; generating a prediction sample of the current block based on the motion information of the merge candidate indicated by the merge index; and updating the list of history-based merge candidates based on the motion information, wherein the history-based merge candidate is added to the list of merge candidates when it has different motion information than that of the predetermined merge candidate of candidates. for the merger included in the list of candidates for the merger.

Technical results

[0016] According to an embodiment of the invention, the complexity according to the redundancy check can be reduced and the compression efficiency can be improved by limiting the redundancy check to be added to the merge candidate list (or AMVP candidate list).

[0017] In accordance with an embodiment of the invention, the memory load associated with storing the HMVP table can be reduced by determining the size of the HMVP table.

[0018] The results that can be obtained in the invention are not limited to the above results, and other technical results not described above can be clearly understood by a person skilled in the relevant field of technology to which the invention pertains from the following description.

Description of drawings

[0019] The accompanying drawings, which are included as part of the detailed description in order to assist in understanding the invention, provide embodiments of the invention and describe the technical characteristics of the invention along with a detailed description.

[0020] FIG. 1 illustrates an example of a video coding system in accordance with an embodiment of the invention.

[0021] FIG. 2 is an embodiment to which the invention is applied, and is a schematic block diagram of an encoding apparatus for encoding a video signal/image signal.

[0022] FIG. 3 is an embodiment to which the invention is applied, and is a schematic block diagram of a decoding apparatus for decoding a video/image signal.

[0023] FIG. 4 shows an example of a block diagram of a content streaming system according to an embodiment of the invention.

[0024] FIG. 5 shows an example of a block diagram of a video signal processing device according to an embodiment of the invention.

[0025] FIG. 6 is an example of a block split structure according to an embodiment of the invention, FIG. 6a to 6b respectively illustrate examples of block split structures according to quadtree (QT), binary tree (BT), ternary tree (TT), and asymmetric tree (AT).

[0026] FIG. 7 and 8 are, respectively, an inter-prediction-based video/image coding procedure and an inter-prediction means in an encoding apparatus according to an embodiment of the invention.

[0027] FIG. 9 and 10 are, respectively, an inter-prediction-based video/image decoding procedure and an inter-prediction means in an encoder according to an embodiment of the invention.

[0028] FIG. 11 illustrates an example of generating spatial merge candidates for the current block.

[0029] FIG. 12 is a flowchart illustrating a method for configuring a merge candidate list according to an embodiment to which the invention is applied.

[0030] FIG. 13 is a flowchart illustrating a method for configuring a prediction candidate list (MVP candidate list) according to an embodiment to which the invention is applied.

[0031] FIG. 14 illustrates an example of motion patterns in accordance with an embodiment of the invention.

[0032] FIG. 15 illustrates an example of a motion vector of a control point for affine motion prediction in accordance with an embodiment of the invention.

[0033] FIG. 16 illustrates an example of a motion vector of each subblock of a block to which affine motion prediction has been applied in accordance with an embodiment of the invention.

[0034] FIG. 17 illustrates an example of adjacent blocks that are used for affine motion prediction in affine merge mode in accordance with an embodiment of the invention.

[0035] FIG. 18 illustrates an example of a block on which affine motion prediction is performed using neighboring blocks to which affine motion prediction has been applied in accordance with an embodiment of the invention.

[0036] FIG. 19 is a diagram for describing a method for generating a merge candidate list using an adjacent affine coding block according to an embodiment of the invention.

[0037] FIG. 20 and 21 are diagrams for describing a method for creating an affine merge candidate list using an affine prediction encoded neighbor block according to an embodiment of the invention.

[0038] FIG. 22 illustrates an example of adjacent blocks that are used for affine motion prediction in affine interframe mode in accordance with an embodiment of the invention.

[0039] FIG. 23 illustrates an example of adjacent blocks that are used for affine motion prediction in affine interframe mode in accordance with an embodiment of the invention.

[0040] FIG. 24 and 25 are diagrams illustrating a method for extracting a motion vector candidate using motion information of an adjacent block in an affine interframe mode, in accordance with an embodiment of the invention.

[0041] FIG. 26 illustrates an example of a subunit motion vector affine field in accordance with an embodiment of the invention.

[0042] FIG. 27 is a flowchart for describing an HMVP storage method according to an embodiment of the invention.

[0043] FIG. 28 is a diagram for describing an HMVP table that operates according to an unrestricted FIFO queue discipline according to an embodiment of the invention.

[0044] FIG. 29 is a diagram for describing an HMVP table that operates according to a constrained FIFO queue discipline in accordance with an embodiment of the invention.

[0045] FIG. 30 is a diagram illustrating an HMVP LUT and a long term HMVP LUT according to an embodiment of the invention.

[0046] FIG. 31 is a diagram illustrating an example of a method for updating an HMVP LUT according to an embodiment of the invention.

[0047] FIG. 32 is a diagram illustrating a method for limiting the number of HMVP candidates, i. e. clipping test purposes, in accordance with an embodiment of the invention.

[0048] FIG. 33 is a flowchart illustrating an example of a method for performing a clipping test according to an embodiment of the invention.

[0049] FIG. 34 is a diagram for describing a method for extracting an H-STMVP candidate using motion vectors that refer to different reference pictures according to an embodiment of the invention.

[0050] FIG. 35 is a diagram illustrating a block location for retrieving a legacy HMVP affine candidate, in accordance with an embodiment of the invention.

[0051] FIG. 36 is a diagram illustrating a merge affine list or an AMVP affine list according to an embodiment of the invention.

[0052] FIG. 37 is a flowchart illustrating a video signal processing method based on history-based motion vector prediction according to an embodiment to which the invention is applied.

[0053] FIG. 38 is a diagram schematically showing an example of a service system including a digital device.

[0054] FIG. 39 is a block diagram illustrating a digital apparatus according to an embodiment.

[0055] FIG. 40 is a configuration block diagram illustrating another embodiment of a digital device.

[0056] FIG. 41 is a block diagram illustrating a digital apparatus according to another embodiment.

[0057] FIG. 42 is a block diagram illustrating the detailed configuration of the control unit in FIG. from 39 to 41.

[0058] FIG. 43 is a diagram illustrating an example in which the screen of the digital device displays the main image and the sub-image at the same time, according to the embodiment.

Embodiment of the invention

[0059] In the following, preferred embodiments of the invention will be described with reference to the accompanying drawings. The description which will be presented below with the aid of the accompanying drawings is for describing exemplary embodiments of the invention, and is not intended to describe a single embodiment in which the invention may be implemented. The description below includes specific details in order to provide a correct understanding of the invention. However, it should be understood that the invention may be practiced without specific details for those skilled in the art. In some cases, in order to avoid obscurity in the technical concept of the invention, structures or devices that are commonly known may be omitted, or may be depicted as a block diagram focusing on the basic functions of the structures or devices.

[0060] In some cases, in order to avoid obscuring the technical concept of the invention, structures or devices that are commonly known may be omitted, or may be depicted as a block diagram focused on the basic functions of the structures or devices.

[0061] In addition, although general concepts that are widely used at present are chosen as concepts in the invention as far as possible, a concept that is arbitrarily chosen by the applicant is used in a specific case. Since in such a case the meaning of the concept will be clearly described in the relevant part of the description, it should be understood that the invention will not simply be interpreted in accordance with the concepts that are only used in the description of the invention, but the meaning of the concepts should be clarified.

[0062] Specific terminologies that are used in the description below can be provided to help in understanding the invention. In addition, the specific terminology may be modified in other forms within the scope of the technical concept of the invention. For example, signal, data, sample, picture, slice, tile, frame, block, etc. may be appropriately substituted and interpreted in each encoding process.

[0063] Hereinafter, in this specification, a "processing unit" means a unit in which an encoding/decoding process such as prediction, transformation, and/or quantization is performed. A processing unit can be interpreted as having a value of one for the luma component and one for the chrominance component. For example, a processing unit may correspond to a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), or a transformation unit (TU).

[0064] In addition, the processing unit can be interpreted as a unit for the luma component or a unit for the chrominance component. For example, a processing unit may correspond to a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB) for a luminance component. Alternatively, a processing unit may correspond to a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB) for a chrominance component. In addition, the invention is not limited to this, and the processing unit can be interpreted as a value including a unit for the luminance component and a unit for the chrominance component.

[0065] In addition, the processing unit is essentially not limited to a square block and can be built in the form of a polygon with three or more vertices.

[0066] In addition, hereinafter, pixel, picture element, coefficient (transformation coefficient or transformation coefficient after first-order transformation), etc. generally called a sample (sample, sample). In addition, using a sample may mean using a pixel value, a picture element value, a transform factor, or the like.

[0067]

[0068] FIG. 1 illustrates an example of a video coding system in accordance with an embodiment of the invention.

[0069] The video coding system may include a source device 10 and a receiving device 20. The source device 10 may transmit the encoded video/image information or data to the receiving device 20 in a file or streaming format via a storage medium or a network.

[0070] Source device 10 may include a video source 11, an encoder 12, and a transmitter 13. Receiver device 20 may include a receiver 21, a decoder 22, and a renderer 23. The source device may be referred to as a video/image encoding device, and the receiving device may be referred to as a video/image decoding device. The transmitter 13 may be included in the encoder 12 . The receiver 21 may be included in the device 22 decoding. The renderer may include a display, and the display may be configured as a separate device or an external component.

[0071] A video source may acquire video/image data through a video/image capture, synthesis, or generation process. The video source may include a video/image capture device and/or a video/image generation device. The video/image capture device may include, for example, one or more cameras, a video/image archive including previously captured video/images, and the like. The video/image generating apparatus may include, for example, a computer, a tablet, and a smartphone, and may electronically generate video/image data. For example, the virtual video/image data may be generated by a computer or the like, and in this case, the video/image capturing process may be replaced by the associated data generation process.

[0072] The encoder 12 may encode the input video/image. Encoder 12 may perform a number of procedures such as prediction, transformation, and quantization for compression and coding efficiency. Encoded data (encoded video information/picture information) may be output in the form of a bitstream.

[0073] The transmitter 13 may transmit the encoded video/picture information or data that is output in the form of a bitstream to the receiver of the receiving device via a digital storage medium or a network in a file or streaming format. The digital storage media may include various storage media such as Universal Serial Bus, secure digital SD card, CD, DVD, Blu-ray, HDD, SSD. The transmitter 13 may include an element for generating a media file by a predetermined file format, and may include an element for transmission via a broadcast/communication network. The receiver 21 may extract the bit stream and transmit it to the decoder 22 .

[0074] The decoder 22 can decode the video/image data by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operations of the encoder 12.

[0075] The renderer 23 may render the decoded video/image. The video/image after rendering can be displayed by the display.

[0076]

[0077] FIG. 2 is an embodiment to which the invention is applied, and is a schematic block diagram of an encoding apparatus for encoding a video signal/image signal.

[0078] Referring to FIG. 2, the encoder 100 may be configured to include an image division means 110, a subtractor 115, a transform 120, a quantizer 130, an inverse quantizer 140, an inverse transform 150, an adder 155, a filter 160, a memory 170, an inter-picture prediction means 180, means 185 intra prediction and entropy encoder 190. The inter prediction engine 180 and the intra prediction engine 185 may be collectively referred to as a predictor. In other words, the prediction engine may include an inter prediction engine 180 and an intra prediction engine 185 . Transformer 120, quantizer 130, inverse quantizer 140, and inverse transform 150 may be included in the residual processor. The residual processor may further include a subtractor 115. In one embodiment, image division means 110, subtractor 115, transform 120, quantizer 130, inverse quantizer 140, inverse transform 150, adder 155, filter 160, inter frame prediction means 180, means 185 intra-prediction and entropy encoder 190 may be implemented as a single piece of hardware (eg, encoder or processor). In addition, the memory 170 may be implemented as a hardware component (eg, a memory or a digital storage medium) in the embodiment, and may include a decoded picture buffer (DPB).

[0079] The image dividing means 110 may divide the input image (or picture or frame) that is input to the encoding device 100 into one or more processing units. For example, a processing unit may be referred to as a coding unit (CU). In this case, a coding unit may be recursively split from a coding tree unit (CTU) or a largest coding unit (LCU) based on a binary tree quadtree (QTBT) structure. For example, one coding unit may be split into multiple deeper coding units based on a quadtree structure and/or a binary tree structure. In this case, for example, a quadtree structure may be applied first, and then a binary tree structure may be applied. Alternatively, a binary tree structure may be applied first. The coding procedure according to the invention can be performed based on a final coding unit that is no longer split. In this case, the largest coding unit can be directly used as the final coding unit based on the coding efficiency according to the image characteristic, or the coding unit can be recursively split into coding units of greater depth, if necessary. Accordingly, the optimal size coding unit can be used as the final coding unit. Here, the encoding procedure may include a procedure such as prediction, transformation, or restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, each of the prediction unit and the transformation unit may be divided or partitioned from each resulting coding unit. The prediction unit may be the unit for predicting the sample, and the transform unit may be the unit from which the transform coefficient is extracted and/or the unit in which the residual signal is extracted from the transform coefficient.

[0080] The unit can be used interchangeably with a block or zone as the case may be. In general, an M×N block may indicate a set of samples configured with M columns and N rows, or a set of transform coefficients. In general, a sample may indicate a pixel or pixel value, or may indicate only a luminance component pixel/pixel value, or only a chroma component pixel/pixel value. In a sample, one picture (or image) can be used as a concept corresponding to a pixel or image element.

[0081] The encoding apparatus 100 can generate a residual signal (residual block or residual sample array) by subtracting the prediction signal (predicted prediction block or prediction sample array) that is output by the inter prediction means 180 or intra prediction means 185 from the input image signal (original block or an array of source samples). The generated residual signal is transmitted to the converter 120. Here, as illustrated, the block in which the prediction signal (prediction block or prediction sample array) is subtracted from the input image signal (source block or source sample array) in the encoding device 100 may be called a subtractor 115 The predictor may perform prediction for a target processing block (hereinafter referred to as the current block), and may generate a predicted block including prediction samples for the current block. The predictor may determine whether intra-prediction or inter-prediction is applied in the current block or CU. The predictor may generate various pieces of prediction information, such as prediction mode information, as will be described later in the description of each prediction mode, and may transmit the information to the entropy encoder 190. The prediction information may be encoded in the entropy encoder 190 and may be output to bitstream form.

[0082] The intra prediction engine 185 may predict the current block with reference to samples in the current picture. The referenced samples may be adjacent to the current block, or may be at a distance from the current block, depending on the prediction mode. In intra prediction, the prediction modes may include a plurality of non-angle modes and a plurality of angle modes. The non-angular mode may include a DC mode and a flat mode, for example. The angle mode may include 33 angle prediction modes or 65 angle prediction modes, for example, depending on the fine prediction direction degree. In this case, angle prediction modes greater than or less than 33 angle prediction modes or 65 angle prediction modes may be used depending on the configuration, for example. The intra prediction means 185 may determine the prediction mode that is applied to the current block using the prediction mode that is applied to the adjacent block.

[0083] The inter prediction engine 180 may derive a predicted block for the current block based on a reference block (reference sample array) indicated by a motion vector in the reference picture. Here, in order to reduce the amount of motion information that is transmitted in the inter prediction mode, the motion information may be predicted as a unit of a block, subblock, or sample based on the correlation of the motion information between the adjacent block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction information (L0 prediction, L1 prediction, Bi prediction). In the case of inter prediction, the adjacent block may include a spatial adjacent block in the current picture and a temporal adjacent block in a reference picture. The reference picture including the reference block and the reference picture including the temporal neighbor block may be the same or may be different. The temporary neighbor block may be referred to by a name that corresponds to a co-located reference block or a co-located CU (colCU). A reference picture including a temporal neighbor block may be referred to as a co-located picture (ColPic). For example, the inter-picture predictor 180 may generate a list of motion information candidates for neighboring blocks, and may generate information indicating which candidate is used to extract the motion vector and/or reference picture index of the current block. Inter prediction may be performed based on various prediction modes. For example, in the case of the skip mode and the merge mode, the inter prediction engine 180 may use the motion information of the neighboring block as the motion information of the current block. In the case of the skip mode, unlike the merge mode, the residual signal may not be transmitted. In the case of a motion information prediction (MVP) mode, a motion vector of an adjacent block may be indicated by motion vector difference signaling.

[0084] The prediction signal generated by the inter prediction engine 180 or the intra prediction engine 185 may be used to generate a recovered signal or a residual signal.

[0085] Converter 120 may generate transform coefficients by applying a transform circuit to the residual signal. For example, the transform circuit may include at least one of discrete cosine transform (DCT), discrete sine transform (DST), Korunen-Loev transform (KLT), graph-based transform (GBT), or conditional non-linear transform (CNT). In this case, GBT means a transformation obtained from a graph if the relationship information between pixels is represented as a graph. CNT means a transformation obtained based on a prediction signal generated using all previously reconstructed pixels. In addition, the transformation process can be applied to equal-sized square-shaped pixel blocks, or can be applied to variable-sized non-square-shaped blocks of pixels.

[0086] The quantizer 130 may quantize the transform coefficients and provide them to the entropy encoder 190. The entropy encoder 190 may encode the quantized signal (information regarding the quantized transform coefficients) and output it in the form of a bitstream. The information regarding the quantized transform coefficients may be referred to as residual information. The quantizer 130 may reorder the quantized transform coefficients in block form into a one-dimensional vector form based on the scan sequence of the coefficients, and may generate information regarding the quantized transform coefficients based on the quantized transform coefficients in one-dimensional vector form. Entropy encoder 190 may perform various coding techniques such as exponential Golomb coding, context dependent variable length adaptive coding (CAVLC), and context dependent adaptive binary arithmetic coding (CABAC). Entropy encoder 190 may encode information (eg, syntax element values) that is needed to reconstruct the video/image, in addition to the quantized transform coefficients, together or separately. Encoded information (eg, encoded video/image information) may be transmitted or stored in a network abstraction layer (NAL) unit in the form of a bitstream. The bit stream may be transmitted over a network or may be stored on a digital storage medium. In this case, the network may include a broadcast network and/or a communication network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A transmitter (not illustrated) that transmits a signal that is output by the entropy encoder 190 and/or a store (not illustrated) for storing the signal may be an internal/external element of the encoder 100, or the transmitter may be an element of the entropy encoder 190.

[0087] The quantized transform coefficients that are output by quantizer 130 may be used to generate a prediction signal. For example, the residual signal can be recovered by applying inverse quantization and inverse transform to the quantized transform coefficients by inverse quantizer 140 and inverse transform 150 in a loop. The adder 155 may add the reconstructed residual signal to the prediction signal output by the inter prediction means 180 or the intra prediction means 185, so that a reconstructed signal (a reconstructed picture, a reconstructed block, or an array of reconstructed samples) can be generated. The predicted block may be used as the reconstructed block if there is no remainder for the target processing block, as is the case when the skip mode was applied. The adder 155 may be referred to as a restorer or a restore block generator. The generated reconstructed signal may be used for intra-frame prediction of the next target processing block of the current picture, and may be used for inter-picture prediction of the next picture through filtering, as will be described later.

[0088] Filter 160 can improve subjective/objective picture quality by applying filtering to the reconstructed signal. For example, filter 160 may generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture. The modified reconstructed picture may be stored in the DPB 170. Various filtering methods may include deblocking filtering, adaptive sample offset, adaptive loop filter, and two-sided filter, for example. The filter 160 may generate various pieces of information for filtering, as will be described later in the description of each filtering method, and may transmit them to the entropy encoder 190. The filtering information may be encoded by the entropy encoder 190 and output in the form of a bitstream.

[0089] The modified reconstructed picture that is transmitted to the DPB 170 may be used as a reference picture in the inter prediction engine 180. The encoding device can avoid prediction mismatch in the encoding device 100 and the decoding device, and improve encoding efficiency if inter-picture prediction is applied.

[0090] The DPB 170 may store the modified reconstructed picture in order to use the modified reconstructed picture as a reference picture in the inter prediction engine 180 .

[0091]

[0092] FIG. 3 is an embodiment to which the invention is applied, and is a schematic block diagram of a decoding apparatus for decoding a video/image signal.

[0093] Referring to FIG. 3, the decoder 200 may be configured to include an entropy decoder 210, an inverse quantizer 220, an inverse transform 230, an adder 235, a filter 240, a memory 250, an inter prediction engine 260, and an intra prediction engine 265. The inter prediction engine 260 and the intra prediction engine 265 may be collectively referred to as a predictor. Those. the predictor may include an inter prediction engine 260 and an intra prediction engine 265 . Inverse quantizer 220 and inverse transform 230 may be collectively referred to as a residual processor. Those. the residual processor may include an inverse quantizer 220 and an inverse transform 230. An entropy decoder 210, an inverse quantizer 220, an inverse transform 230, an adder 235, a filter 240, an inter-predictor 260, and an intra-predictor 265 may be implemented as a single piece of hardware. (eg, decoder or processor) according to the embodiment. In addition, the memory 170 may be implemented as a hardware component (eg, a memory or a digital storage medium) in the embodiment, and may include a decoded picture buffer (DPB).

[0094] When a bitstream including video information/image information is input, the decoding apparatus 200 may reconstruct the image according to the video/image information processing process in the encoding apparatus in FIG. 2. For example, the decoding device 200 may perform decoding using a processing unit that is applied in the encoding device. Accordingly, the processing unit for decoding may be a coding unit, for example. The coding unit may be split from the coding tree unit or the largest coding unit depending on the quadtree structure and/or binary tree structure. In addition, the reconstructed image signal that is decoded and output by the decoding apparatus 200 can be reproduced by the playback apparatus.

[0095] The decoder 200 may receive a signal that is output by the encoder in FIG. 1 in the form of a bitstream. The received signal may be decoded by the entropy decoder 210. For example, the entropy decoder 210 may extract information (eg, video/image information) for image reconstruction (or picture reconstruction) by bitstream analysis. For example, entropy decoder 210 may decode information in the bitstream based on an encoding method such as Exponential Golomb coding, CAVLC, or CABAC, and may output a syntax element value for image reconstruction or quantized values of transform coefficients with respect to the residual. More specifically, in the CABAC entropy decoding method, a bin corresponding to each syntax element may be received from a bitstream, a context model may be determined using decoding syntax target element information and decoding information of a neighboring or target decoding block or character/bin information, decoded in the previous step, the occurrence probability of the bin can be predicted based on the determined context model, and the symbol corresponding to the value of each syntax element can be generated by performing arithmetic decoding on the bin. Here, in the CABAC entropy decoding method, after the context model is determined, the context model can be updated by using the symbol/bin information decoded for the next symbol/bin context model. Prediction information from the information decoded in the entropy decoder 2110 may be provided to a prediction engine (inter prediction engine 260 and intra prediction engine 265). Parameter information associated with the residual value that has been entropy decoded in the entropy decoder 210, i. e. quantized transform coefficients may be input to the inverse quantizer 220. In addition, information regarding filtering from the information decoded by the entropy decoder 210 may be provided to the filter 240. Meanwhile, a receiver (not illustrated) that receives a signal that is output by the encoder, may additionally be made as an internal/external element of the decoding device 200 or the receiver may be an element of the entropy decoder 210.

[0096] An inverse quantizer 220 may inversely quantize the quantized transform coefficients and output the transform coefficients. The inverse quantizer 220 may reorder the quantized transform coefficients into a two-dimensional block form. In this case, the reordering may be performed based on the coefficient scan sequence performed in the encoder. An inverse quantizer 220 may perform inverse quantization on the quantized transform coefficients using a quantization parameter (eg, quantization step size information) and may obtain the transform coefficients.

[0097] The inverse transform 230 can output a residual signal (residual block or array of residual samples) by applying an inverse transform to the transform coefficients.

[0098] The predictor may perform prediction on the current block and may generate a predicted block including prediction samples for the current block. The predictor may determine whether intra-prediction or inter-prediction is applied to the current block based on prediction-related information output by the entropy decoder 210, and may determine the detailed mode of intra-/inter-prediction.

[0099] The intra prediction engine 265 may predict the current block with reference to samples in the current picture. The referenced samples may be adjacent to the current block or may be at a distance from the current block depending on the prediction mode. In intra-prediction, the prediction modes may include a plurality of non-angle modes and a plurality of angle modes. The intra prediction engine 265 may determine the prediction mode that is applied to the current block by using the prediction mode that is applied to the adjacent block.

[0100] The inter prediction engine 260 may derive a predicted block for the current block based on a reference block (reference sample array) indicated by a motion vector in the reference picture. Here, in order to reduce the amount of motion information that is transmitted in the inter prediction mode, the motion information may be predicted as a unit of a block, subblock, or sample based on the correlation of the motion information between the adjacent block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction information (L0 prediction, L1 prediction, Bi prediction). In the case of inter prediction, the adjacent block may include a spatial adjacent block in the current picture and a temporal adjacent block in a reference picture. For example, the inter frame prediction means 260 may configure a candidate list for motion information of adjacent blocks, and may derive a motion vector and/or a reference picture index of the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes. The prediction information may include information indicating the inter prediction mode of the current block.

[0101] The adder 235 may generate a reconstructed signal (reconstructed picture, reconstructed block, or reconstructed sample array) by adding the obtained residual signal to the prediction signal (predicted prediction block or array of prediction samples) that is output by the inter prediction engine 260 or the intra prediction engine 265. The predicted block may be used as the reconstructed block if there is no remainder for the target processing block, as is the case when the skip mode was applied.

[0102] The adder 235 may be referred to as a restorer or a restored block generator. The generated reconstructed signal may be used for intra-picture prediction of the next processing target in the current picture, and may be used for inter-picture prediction of the next picture through filtering, as will be described later.

[0103] Filter 240 can improve subjective/objective picture quality by applying filtering to the reconstructed signal. For example, filter 240 may generate a modified reconstructed picture by applying various filtering techniques to the reconstructed picture, and may pass the modified reconstructed picture to DPB 250. Various filtering techniques may include deblocking filtering, adaptive sample offset SAO, adaptive loop filter ALF, and two-sided filter, for example.

[0104] The reconstructed (modified) picture that is transmitted to the DPB 250 may be used as a reference picture in the inter-picture predictor 260.

[0105] In the invention, the embodiments described in the filter 160, the inter prediction engine 180, and the intra prediction engine 185 of the encoding apparatus 100 can be applied to the filter 240, the inter prediction engine 260, and the intra prediction engine 265 of the decoding apparatus 200, respectively, identically or in accordance.

[0106]

[0107] FIG. 4 shows an example of a block diagram of a content streaming system according to an embodiment of the invention.

[0108] A content streaming system to which the invention is applied may largely include an encoding server 410, a streaming server 420, a web server 430, a media storage 440, a user device 450, and a media input device 460.

[0109] The encoding server 410 may compress content that is input from multimedia input devices such as a smartphone, camera, camcorder, etc. into digital data to form a bitstream and transmit it to the streaming server 420. As another example, when multimedia input devices 460 such as a smartphone, camera, or camcorder directly generate a bitstream, the encoding server 410 may be omitted.

[0110] The bitstream may be generated by an encoding method or a bitstream generation method to which the invention is applied, and the streaming server 420 may temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0111] The streaming server 420 transmits media data to the user's device 450 at the user's request through the web server 430, and the web server 430 serves as an intermediary to inform the user which service is present. When the user requests the desired service through the web server 430, the web server 430 delivers the request to the streaming server 420, and the streaming server 420 transmits the media data to the user. At this time, the content streaming system may include a separate control server, in which case the control server serves to manage commands/responses between devices in the content streaming system.

[0112] The server 420 streaming may receive content from the store 440 media and/or server 410 encoding. For example, the server 420 streaming may receive content in real time from the server 410 encoding. In this case, in order to provide a continuous streaming service, the streaming server 420 may store the bitstream for a predetermined time.

[0113] For example, user device 450 may include a mobile phone, smartphone, laptop computer, digital broadcast terminal, PDA personal digital assistant, PMP portable media player, navigation terminal, flat PC, tablet PC, ultrabook, wearable device ( such as smart wristwatch), smart glasses, HMD display, digital TV, desktop computer and digital signage.

[0114] Each server in the content streaming system may operate as a distributed server, in which case the data received from each server may be processed in a distributed manner.

[0115]

[0116] FIG. 5 shows an example of a block diagram of a video signal processing device according to an embodiment of the invention. The video signal processor may correspond to the encoder 100 in FIG. 2 or decoder 200 in FIG. 3.

[0117] A video signal processing apparatus 500 according to an embodiment of the invention may include a memory 520 for storing a video signal, and a processor 510 for processing a video signal, while being combined with the memory 520.

[0118] The processor 510 according to an embodiment of the invention may be implemented as at least one processing circuit for processing a video signal, and may process the video signal by executing instructions for encoding or decoding the video signal. Those. processor 510 may encode the original video signal or decode the encoded video signal by executing the encoding or decoding methods described below.

[0119]

[0120] FIG. 6 is an example of a block split structure according to an embodiment of the invention, FIG. 6a to 6d illustrate examples of block split structures according to quadtree (QT), binary tree (BT), ternary tree (TT), and asymmetric tree (AT).

[0121] When encoding video, one block may be split based on QT. Also, one sub-unit split using QT can be further recursively split using QT. A leaf block that is no longer split according to QT may be split by at least one of BT, TT or AT. BT can have two types of crushing: horizontal BT (2N×N, 2N×N) and vertical BT (N×2N, N×2N). TT can have two types of crushing: horizontal TT (2N×1/2N, 2N×N, 2N×1/2N) and vertical TT (1/2N×2N, N×2N, 1/2N×2N). AT can have four types of crushing: horizontal top AT (2N×1/2N, 2N×3/2N), horizontal bottom AT (2N×3/2N, 2N×1/2N), vertical left AT (1/2N×2N , 3/2N×2N), vertical right AT (3/2N×2N, 1/2N×2N). Each of BT, TT, AT can be further subdivided recursively using BT, TT, AT.

[0122] FIG. 6A shows an exemplary QT partition. Block A can be split into four sub-blocks A0, A1, A2 and A3 by means of QT. Subblock A1 can be split into four subblocks B0, B1, B2 and B3 again by QT.

[0123] FIG. 6B shows an example of splitting BT. Block B3, which is no longer split by QT, can be split into vertical BT (C0, C1) or horizontal BT (D0, D1). Like block C0, each subblock can be further recursively split into a horizontal BT (E0, E1) or a vertical BT (F0, F1).

[0124] FIG. 6C shows an example of splitting TT. Block B3, which is no longer split by QT, can be split into vertical TT (C0, C1, C3) or horizontal TT (D0, D1, D2). Like block C1, each subblock can be further recursively split in the form of a horizontal TT (E0, E1, E2) or a vertical TT (F0, F1, F2).

[0125] FIG. 6D shows an example of splitting AT. Block B3, which is no longer split by QT, can be split into vertical AT (C0, C1) or horizontal AT (D0, D1). Like block C1, each subblock can be further recursively split in the form of horizontal AT (E0, E1) or vertical TT (F0, F1).

[0126] Meanwhile, BT, TT and AT partitions can be combined. For example, a subblock split by BT may be split by TT or AT. In addition, a subblock split by TT may be split by BT or AT. A subblock split by AT may be split by BT or TT. For example, after a horizontal BT split, each sub-block may be split into a vertical BT, or after a vertical BT split, each sub-block may be split into a horizontal BT. The two types of crushing methods have different crushing order, but the final crushing shape is the same.

[0127] In addition, when a block is split, the order in which the block is searched may be determined differently. In general, the search operation is performed from left to right and from top to bottom. The block search may mean the order in which each subblock is further split, or the encoding order of the corresponding subblocks when the block is no longer split, or the search order when the subblocks refer to information of other neighboring blocks.

[0128]

[0129] FIG. 7 and 8 are, respectively, an inter-prediction-based video/image coding procedure and an inter-prediction means in an encoding apparatus according to an embodiment of the invention.

[0130] The encoder 100 performs inter prediction of the current block (S710). Encoder 100 may extract the inter prediction mode and motion information of the current block, and may generate prediction samples of the current block. In this case, the procedures for determining the inter prediction mode, extracting motion information, and generating a prediction sample may be performed simultaneously, and any of the procedures may be performed before the other procedure. For example, the inter prediction means 180 of the encoding apparatus 100 may include a prediction mode determination unit 181, a motion information extraction unit 182, or a prediction sample extraction unit 183. The prediction mode determination unit 181 may determine the prediction mode for the current block. The motion information extractor 182 can extract the motion information of the current block. The prediction sample extractor 183 may extract the prediction samples of the current block. For example, the inter-picture predictor 180 of the encoding apparatus 100 may search for a predetermined area (search area) in reference pictures with respect to a block similar to the current block by motion estimation, and may extract a reference block with a minimum difference or a predetermined reference difference or less with respect to the current block. The inter frame prediction means 180 may extract a reference picture index indicating a reference picture in which the reference block is located based on the reference block, and may derive a motion vector based on a position difference between the reference block and the current block. Encoder 100 may determine the mode that is applied to the current block from various prediction modes. The encoder may compare RD costs for different prediction modes, and may determine the optimal prediction mode for the current block.

[0131] For example, if a skip mode or a merge mode is applied to the current block, then the encoder 100 may configure a merge candidate list, which will be described later, and may extract a reference block with a minimum difference or a specified reference difference or less with respect to the current block among the supporting blocks specified by the merge candidates that are included in the list of merge candidates. In this case, a merge candidate associated with the retrieved reference block may be selected. Merge index information indicating a selected merge candidate may be generated and signaled to the decoder 200 . The movement information of the current block can be extracted using the movement information of the selected merge candidate.

[0132] For example, if the (A)MVP mode is applied to the current block, then the encoder may configure the (A)MVP candidate list, which will be described later, and may use the motion vector of a motion vector predictor (mvp) candidate selected among the candidates on the mvp included in the list of (A)MVP candidates as the mvp of the current block. Here, for example, a motion vector indicating a reference block extracted by motion estimation may be used as the motion vector of the current block. The mvp candidate including the motion vector with the smallest difference with respect to the motion vector of the current block among the mvp candidates may become the selected mvp candidate. Motion vector difference (MVD) can be extracted, i. e. the difference obtained by subtracting mvp from the motion vector of the current block. In this case, the MVD information may be signaled to the decoder 100 . In addition, if the (A)MVP mode is applied, the reference picture index value may be configured as reference picture index information and may be separately signaled to the decoding apparatus.

[0133] The encoder 100 may extract residual samples based on the prediction samples (S720). The encoder 100 can extract the residual samples by comparing the original samples of the current block and the prediction samples.

[0134] The encoder 100 encodes image information including prediction information and residual information (S730). The encoding device may output the encoded image information in the form of a bitstream. The prediction information may include information regarding prediction mode information (eg, skip flag, merge flag, or mode index) and motion information as prediction procedure related information. The traffic information related information may include candidate selection information (eg, merge index, mvp flag, or mvp index), i. e. information to extract the motion vector. In addition, the information related to the motion information may include information regarding MVD and/or information about the index of the reference picture. In addition, the information associated with the motion information may include information indicating whether L0 prediction, L1 prediction, or bi-prediction is applied. The residual information is information regarding residual samples. The residual information may include information regarding quantized transform coefficients for the residual samples.

[0135] The output bitstream may be stored on a (digital) storage medium and transmitted to the decoder, or may be transmitted to the decoder via a network.

[0136] Meanwhile, as described above, the encoding device can generate a reconstructed picture (including reconstructed samples and a reconstructed block) based on reference samples and residual samples. This serves to retrieve, in the encoder 100, the same prediction results as those performed in the decoder 200. Accordingly, the encoding efficiency can be improved. Accordingly, the encoding apparatus 100 may store the reconstructed picture (or the reconstructed samples and the reconstructed block) in memory, and may use the reconstructed picture as a reference picture for inter prediction. As described above, the in-loop filtering procedure can be additionally applied to the reconstructed picture.

[0137]

[0138] FIG. 9 and 10 are, respectively, an inter-prediction-based video/image decoding procedure and an inter-prediction means in a decoding apparatus according to an embodiment of the invention.

[0139] The device 200 decoding can perform an operation corresponding to the operation that is performed in the device 100 encoding. The decoding apparatus 200 may perform prediction on the current block based on the received prediction information, and may extract prediction samples.

[0140] In particular, the decoding apparatus 200 may determine the prediction mode for the current block based on the received prediction information (S910). The decoding apparatus 200 can determine which inter prediction mode is applied to the current block based on the prediction mode information in the prediction information.

[0141] For example, the decoder 200 may determine whether a merge mode or (A)MVP mode is applied to the current block based on the merge flag. Alternatively, the decoding apparatus 200 may select one of various inter prediction mode candidates based on the mode index. Candidates for an inter prediction mode may include a skip mode, a merge mode, and/or an (A)MVP mode, and may include various inter prediction modes, which will be described later.

[0142] The decoding apparatus 200 extracts motion information of the current block based on the determined inter prediction mode (S920). For example, if a skip mode or a merge mode is applied to the current block, then the decoder 200 may configure a merge candidate list, which will be described later, and select one of the merge candidates included in the merge candidate list. The selection of a merge candidate may be performed based on the merge index. The motion information of the current block can be extracted from the motion information of the selected merge candidate. The motion information of the selected merge candidate can be used as the motion information of the current block.

[0143] As another example, if the (A)MVP mode is applied to the current block, then the decoder 200 may configure an (A)MVP candidate list, which will be described later, and may use a motion vector of a motion vector predictor (mvp) candidate , selected among the mvp candidates included in the list of (A)MVP candidates, as the mvp of the current block. The selection may be made based on selection information (mvp flag or mvp index). In this case, the decoding apparatus 200 can extract the MVD of the current block based on the information regarding the MVD. The decoder may derive a motion vector of the current block based on the mvp of the current block and the MVD. In addition, the decoding apparatus can derive the reference picture index of the current block based on the reference picture index information. The picture indicated by the index of the reference picture in the reference picture list with respect to the current block may be retrieved as the reference picture referred to for inter prediction of the current block.

[0144] Meanwhile, as will be described later, the movement information of the current block can be retrieved without configuring the candidate list. In this case, the motion information of the current block can be extracted according to the procedure disclosed in the prediction mode to be described later. In this case, a candidate list configuration such as that described above may be omitted.

[0145] The decoding apparatus 200 may generate prediction samples for the current block based on motion information of the current block (S930). In this case, the decoding apparatus 200 may derive a reference picture based on the index of the reference picture of the current block, and may retrieve prediction samples of the current block indicated with respect to the reference picture by the motion vector of the current block. Here, as will be described later, the prediction sample filtering procedure may be further performed on some or all of the prediction samples of the current block, as appropriate.

[0146] For example, the inter-picture predictor 260 of the decoding apparatus 200 may include a prediction mode determination unit 261, a motion information extractor 262, and a prediction sample extractor 263. The decoding apparatus 200 may determine the prediction mode of the current block based on the prediction mode information received from the prediction mode determination unit 261, may extract the motion information (motion vector and/or reference picture index) of the current block based on information related to the motion information received from the motion information extractor 262 . The prediction sample extractor 263 may extract the prediction samples of the current block.

[0147] The decoder 200 generates residual samples for the current block based on the received residual information (S940). The decoding apparatus 200 may generate reconstructed samples for the current block based on the prediction samples and residual samples, and may generate a reconstructed picture based on the reconstructed samples (S950). Then, as described above, the in-loop filtering procedure can be further applied to the reconstructed picture.

[0148] As described above, the inter prediction procedure may include an inter prediction mode determination step, a motion information extraction step according to the determined prediction mode, and a prediction execution step (prediction sample generation) based on the extracted motion information.

[0149]

[0150] Determining the inter-prediction mode

[0151] Various modes of inter-picture prediction can be used to predict the current block in the picture. For example, various modes such as merge mode, skip mode, MVP mode, and affine mode can be used. Decoder side motion vector refinement mode (DMVR), adaptive motion vector resolution (AMVR) mode, etc. can be additionally used as additional modes. The affine mode may be referred to as the affine motion prediction mode. The MVP mode may be referred to as the advanced motion vector prediction (AMVP) mode.

[0152] Prediction mode information indicating the inter prediction mode of the current block may be signaled from the encoding device to the decoding device. The prediction mode information may be included in the bitstream and received by the decoder. The prediction mode information may include index information indicating one of several candidate modes. Alternatively, the inter prediction mode may be indicated by hierarchical flag information signaling. Here, the prediction mode information may include one or more flags. For example, a flag may be further signaled to indicate whether a skip mode is applied, by signaling a skip flag to indicate whether a merge mode is applied, by signaling a merge flag if a skip mode is not applied, and to indicate that an MVP mode is applied. , if merge mode is not applied or for additional indication. The affine mode may be signaled as an independent mode and may be signaled as a merge mode dependent or MVP mode. For example, the affine mode may be configured as one of a merge candidate list or an MVP candidate list, as will be described later.

[0153]

[0154] Extracting motion information according to the inter-frame prediction mode

[0155] Encoder 100 or decoder 200 may perform inter-picture prediction using current block information. The encoder 100 can extract the optimal motion information for the current block in accordance with the motion estimation procedure. For example, the encoder 100 may search for a reference block with a similar correlation using the source block in the source picture for the current block in fractional pixel units within a certain search range in the reference picture. Accordingly, the encoding device can extract motion information. The similarity of the block can be extracted based on the difference between the phase-based values of the samples. For example, block similarity can be calculated based on the SAD (Sum of Absolute Differences) between the current block (or current block template) and a reference block (or reference block template). In this case, the motion information can be extracted based on the reference block with the lowest SAD within the search area. The extracted motion information can be signaled to a decoder using several methods based on the inter-prediction mode.

[0156]

[0157] Merge mode and skip mode

[0158] If the merge mode is applied, the motion information of the current prediction block is not directly transmitted, and the motion information of the current prediction block is extracted using the motion information of the neighboring prediction block. Accordingly, the encoder 100 can indicate motion information of the current prediction block by transmitting flag information to notify that the merge mode has been used and a merge index to notify which adjacent prediction block has been used.

[0159] The encoder 100 must search for a merge candidate block used to extract motion information of the current prediction block in order to perform the merge mode. For example, a maximum of 5 merge candidate blocks may be used, but the invention is not limited to this. In addition, the maximum number of merge candidate blocks can be conveyed in the slice header, and the invention is not limited to this. After searching for merge candidate blocks, the encoder 100 may generate a merge candidate list, and may select the lowest cost merge candidate block among the merge candidate blocks as the final merge candidate block.

[0160] An embodiment of the invention provides various embodiments of merge candidate blocks creating a list of merge candidates.

[0161] The merge candidate list may use 5 merge candidate blocks, for example. For example, 4 spatial merge candidates and 1 temporal merge candidate may be used.

[0162]

[0163] FIG. 11 illustrates an example of generating spatial merge candidates for the current block.

[0164] Referring to FIG. 11, at least one of left neighbor A1, lower left neighbor A2, upper right neighbor B0, upper neighbor B1, and upper left neighbor B2 can be used to predict the current block. The merge candidate list for the current block may be configured based on the procedure shown in FIG. 12.

[0165]

[0166] FIG. 12 is a flowchart illustrating a method for configuring a merge candidate list according to an embodiment to which the invention is applied.

[0167] An encoder (encoder 100 or decoder 200) searches for spatial neighboring blocks of the current block and inserts the extracted spatial merge candidates into the merge candidate list (S1210). For example, spatial neighbors may include a bottom left corner neighbor, a left neighbor, a top right corner neighbor, a top neighbor, and a top left corner neighbor of the current block. Here, this is an example, and additional neighbor blocks such as a right neighbor block, a bottom neighbor block, and a bottom right neighbor block, in addition to the spatial neighbor blocks, can be further used as the spatial neighbor blocks. The encoder may detect available blocks by searching for spatial neighboring blocks based on priority, and may extract motion information of the detected blocks as spatial merge candidates. For example, the encoder 100 or the decoder 200 may search for the 5 blocks illustrated in FIG. 11 in sequence A1, B1, B0, A0, and B2, and can configure a merge candidate list by sequentially indexing the available candidates.

[0168] The encoder searches for a temporary neighbor block of the current block, and inserts the extracted temporary merge candidate into the merge candidate list (S1220). The temporary neighbor block may be located in the reference picture, i. e. a picture different from the current picture, which is located next to the current block. The reference picture in which the temporal adjacent block is located may be referred to as a co-located picture or a col picture. The search for a temporary neighbor block can be performed in the sequence of the bottom right corner neighbor block and the bottom right center block of the co-located block of the current block in the col-picture. Meanwhile, if motion data compression is applied, specific motion information can be stored in the col picture as representative motion information for each given storage unit. In this case, it is not necessary to store motion information for all blocks in a given storage unit, and thus a motion data compression result can be obtained. In this case, the specified storage unit may be predetermined as 16×16 sample unit or 8×8 sample unit, for example, or size information for the specified storage unit may be signaled from the encoder 100 to the decoder 200. If motion data compression is applied, then the motion information of the temporal neighbor block may be replaced by representative motion information of the given storage unit in which the temporal neighbor block resides. Those. in this case, in an implementation aspect, after an arithmetic right shift is performed by a given value based on the coordinates (upper left sample positions) of the temporal neighbor block, rather than the prediction block in which the coordinates of the temporal neighbor block are located, the temporal merge candidate may be derived based on the motion information of the prediction block that covers the location with an arithmetic shift to the left. For example, if the specified storage unit corresponds to 2n×2n sample units, assuming that the coordinates of the temporal neighbor block correspond to (xTnb, yTnb), then the motion information of the prediction block located at ((xTnb>>n)<<n), (yTnb >>n)<<n)), i.e. modified location can be used for an interim merger candidate. In particular, for example, if a given storage unit corresponds to a 16×16 sample unit, assuming that the coordinates of the temporal neighboring block correspond to (xTnb, yTnb), then the motion information of the prediction block located at ((xTnb>>4)<<4) , (yTnb>>4)<<4)), i.e. modified location can be used for an interim merger candidate. Alternatively, for example, if the given storage unit corresponds to an 8×8 sample unit, assuming that the coordinates of the temporal neighbor block correspond to (xTnb, yTnb), then the motion information of the prediction block located at ((xTnb>>3)<<3 ), (yTnb>>3)<<3)), i.e. modified location can be used for an interim merger candidate.

[0169] The encoder may check whether the current number of merge candidates is less than the maximum number of merge candidates (S1230). The maximum number of merge candidates may be predetermined or may be signaled from the encoder 100 to the decoder 200. For example, the encoder 100 may generate information regarding the maximum number of merge candidates, may encode the information, and may transmit the information to the decoder 200 in the form of a bitstream. If the maximum number of merge candidates is full, the process of adding a candidate may fail.

[0170] If, as a result of checking, the current number of merge candidates is less than the maximum number of merge candidates, then the encoder inserts the added merge candidate into the merge candidate list (S1240). The added merge candidate may include ATMVP (Adaptive Motion Vector Timing Prediction), a combined bi-predictive merge candidate (if the slice type of the current slice is type B), and/or a null vector merge candidate, for example.

[0171]

[0172] FIG. 13 is a flowchart illustrating a method for configuring a prediction candidate list (MVP candidate list) according to an embodiment to which the invention is applied.

[0173] If a motion vector prediction (MVP) mode is applied, then a motion vector predictor (mvp) candidate list may be generated based on the motion vector of the reconstructed spatial neighbor block (e.g., the neighbor block described in FIG. 11) and/or vector motion corresponding to the temporary neighboring block (or Col block). That is, the motion vector of the reconstructed spatial neighbor block and/or the motion vector of the temporal neighbor block may be used as a motion vector predictor candidate. The prediction information may include selection information (eg, an MVP flag or an MVP index) indicating an optimal motion vector predictor candidate selected from among the motion vector predictor candidates included in the list. Here, the predictor may select a motion vector predictor of the current block, among the motion vector predictor candidates included in the motion vector candidate list, using the selection information. The encoder 100 predictor may calculate a motion vector difference (MVD) between the motion vector of the current block and the motion vector predictor, may encode the MVD, and may output the encoded MVD in the form of a bitstream. Those. The MVD may be calculated as a value obtained by subtracting the motion vector predictor from the motion vector of the current block. In this case, the predictor of the decoding apparatus can obtain the motion vector difference included in the prediction information, and can extract the motion vector of the current block by adding the motion vector difference and the motion vector predictor. The predictor of the decoding apparatus may obtain or derive a reference picture index indicating the reference picture from the prediction information. For example, the motion vector predictor candidate list may be configured as illustrated in FIG. 13.

[0174] Referring to FIG. 13, the encoder searches for a motion vector prediction spatial block candidate and inserts it into the prediction candidate list (S1310). For example, the encoder may search for neighboring blocks according to a predetermined search order, and add neighboring block information that satisfies a condition for a spatial block candidate to a prediction candidate list (MVP candidate list).

[0175] After generating the spatial block candidate list, the encoding device compares the number of spatial candidates included in the prediction candidate list with a predetermined reference number (eg, 2) (S1320). If the number of spatial candidates included in the prediction candidate list is greater than or equal to the reference number (eg, 2), the encoder may finish generating the prediction candidate list.

[0176] But, if the number of spatial candidates included in the prediction candidate list is less than the reference number (for example, 2), the encoder searches for a candidate temporal block and inserts it into the prediction candidate list (S1330), and when the temporal block -candidate not available adds motion vector zero to the list of prediction candidates (S1340).

[0177]

[0178] Prediction Sample Generation

[0179] The predicted block for the current block may be extracted based on motion information extracted in accordance with the prediction mode. The predicted block may include prediction samples (array of prediction samples) of the current block. When the motion vector of the current block indicates a fractional unit of the sample, an interpolation procedure may be performed, and thereby prediction samples of the current block can be extracted based on the reference samples in the fractional unit of the sample in the reference picture. When affine inter prediction is applied to the current block, prediction samples may be generated based on a motion vector in a sample/subblock unit. When bi-directional prediction is applied, the resulting prediction samples may be extracted by weighted (according to phase) sums of prediction samples extracted based on the first prediction direction (e.g., L0 prediction) and prediction samples extracted based on the second prediction direction. The reconstructed samples and reconstructed pictures may be generated based on the extracted prediction samples, and as described above, a procedure such as in-loop filtering may subsequently be performed.

[0180]

[0181] Affine Motion Prediction

[0182] FIG. 14 illustrates an example of motion patterns according to an embodiment.

[0183] In a conventional image compression technology (eg, High Efficiency Video Coding (HEVC)), one motion vector is used to represent the motion of a coding block. Although the optimal motion of a block unit can be represented using a method using one motion vector per block, it may not actually be the optimal motion of each picture element. Accordingly, if the optimal motion vector is determined in a picture element unit, the coding efficiency can be improved. Accordingly, an embodiment of the invention describes a motion prediction method for encoding or decoding a video signal using a multi-motion model. Specifically, a motion vector may be represented in each picture element unit of a block or sub-block unit using the motion vector at control points 2 to 4. The prediction scheme using the motion vector of a plurality of control points may be referred to as affine motion prediction or affine prediction.

[0184] An affine motion model according to an embodiment of the invention may be 4 motion models such as those illustrated in FIG. 14. An affine motion model for representing three motions (parallel translation, scaling and rotation) among the motions that can be represented by an affine motion model is called a similarity (or simplified) affine motion model. In describing the embodiments, the affine motion similarity (or simplified) model is mainly described for the convenience of description, but the invention is not limited to this.

[0185]

[0186] FIG. 15 illustrates an example of a motion vector of a control point for affine motion prediction in accordance with an embodiment of the invention.

[0187] As shown in FIG. 15, in affine motion prediction, a motion vector of a picture element (or sub-block) location included in a block can be determined using a pair of two control point motion vectors (CPMV) v_0 and v_1. Here, the set of motion vectors may be referred to as an affine motion vector field (MVF). In this case, the affine field of motion vectors can be determined using Equation 1.

[0188] [Equation 1]

[0189] In Equation 1, v_0(v_0={v_0x, v_0y}) indicates the motion vector CPMV0 at the first control point of the upper left location of the current block 1500. v_1(v_1={v_1x, v_1y}) indicates the motion vector CPMV1 at the second control point the top right location of the current block 1500. In addition, w indicates the width of the current block 1500. v(v={v_x, v_y}) indicates the motion vector at location {x, y}. The motion vector of a subblock unit (or picture element) can be extracted using Equation 1. In one embodiment, the motion vector precision can be rounded to 1/16 precision.

[0190]

[0191] FIG. 16 illustrates an example of a motion vector for each subblock of a block to which affine motion prediction has been applied in accordance with an embodiment of the invention.

[0192] Referring to FIG. 16, in the process of encoding or decoding, an affine field of motion vectors may be defined in a picture element unit or a block unit. Those. in affine motion prediction, the motion vector of the current block may be extracted in a picture element unit or a sub-block unit.

[0193] If an affine field of motion vectors is defined in a picture element unit, then a motion vector can be derived based on each picture element value. In the case of a block unit, the motion vector of the corresponding block can be obtained based on the value of the central picture element of the block. This document assumes that the affine motion vector field (MVF) is defined in a 4x4 block unit as in FIG. 16. In this case, it is for the convenience of description, and the embodiment of the invention is not limited to this. Fig. 16 illustrates an example of a case where a coding block is configured with 16x16 samples and a motion vector affine field (MVF) is defined in a 4x4 size block unit.

[0194] Affine motion prediction may include affine merge mode (or AF_MERGE) and affine interframe mode (or AF_INTER). The AF_INTER mode may include an AF_4_INTER mode using a motion model based on 4 parameters and an AF_6_INTER mode using a motion model based on 6 parameters.

[0195]

[0196] Affine Merge Mode

[0197] AF_MERGE determines a control point motion vector (CPMV) based on an affine motion model of a neighboring block encoded as affine motion prediction. The adjacent block affine-coded in the search sequence can be used for AF_MERGE. When one or more adjacent blocks are encoded as affine motion prediction, the current block may be encoded as AF_MERGE.

[0198] i.e. if affine merge mode is applied, the CPMVs of the current block can be extracted using the CPMVs of the adjacent block. In this case, the CPMVs of the neighbor block can be used as the CPMV of the current block without any change, and the CPMVs of the neighbor block can be modified based on the size of the neighbor block and the size of the current block, and can be used as the CPMV of the current block.

[0199] FIG. 17 illustrates an example of adjacent blocks that are used for affine motion prediction in affine merge mode in accordance with an embodiment of the invention.

[0200] In the affine merge mode (AF_MERGE), the encoder may perform encoding according to the following process.

[0201] Step-1: Scan the adjacent blocks A to E 1710, 1720, 1730, 1740, and 1750 of the current coding block 1700 in the alphabet sequence, and determine the block first encoded in accordance with the affine prediction mode based on the scan sequence as the block -candidate affine merge (AF_MERGE)

[0202] Step-2: Determine an affine motion model using the control point motion vector (CPMV) of the determined candidate block

[0203] Step-3: Determine the control point motion vector (CPMV) of the current block 1700 based on the affine motion model of the candidate block, and determine the MVF of the current block 1700

[0204]

[0205] FIG. 18 illustrates an example of a block on which affine motion prediction is performed using neighboring blocks to which affine motion prediction has been applied in accordance with an embodiment of the invention.

[0206] For example, as in FIG. 18, if block 1820A has been encoded according to the affine mode, then after block 1820A is determined as a candidate block, an affine motion model can be extracted using Control Point Motion Vectors (CPMV) (e.g., v2 and v3 ) of block 1820 A, and control point motion vectors (CPMV) v0 and v1 of the current block 1800 can be determined. The affine motion vector field (MVF) of the current block 1800 can be determined based on the control point motion vectors (CPMV) of the current block 1800, and encoding can be done.

[0207]

[0208] FIG. 19 is a diagram for describing a method for generating a merge candidate list using an adjacent affine coding block according to an embodiment of the invention.

[0209] Referring to FIG. 19, if a CPMV pair is determined using an affine merge candidate, then candidates such as those illustrated in FIG. 19. In FIG. 19, it is assumed that the candidate list scan sequence has been set to A, B, C, D, and E. Here, the invention is not limited to this, and various sequences can be preset.

[0210] In an embodiment, if the number of candidates (hereinafter referred to as affine candidates) that are encoded according to the affine mode (or affine prediction) available in adjacent blocks (i.e., A, B, C, D, E) is 0, then the current block's affine merge mode can be skipped. If the number of available affine candidates is one (eg, A), then the motion model of the corresponding candidate can be used to extract the motion vectors of the control point (CPMV_0 and CPMV_1) of the current block. In this case, an index indicating the corresponding candidate may not be required (or encoded). If the number of available affine candidates is two or more, then two candidates in the scan sequence may be configured as a candidate list for AF_MERGE. In this case, candidate selection information such as an index indicating the candidate that is selected in the candidate list may be signaled. The selection information may be phage or index information, and may be referred to as AF_MERGE_flag or AF_merge_idx.

[0211] In an embodiment of the invention, motion compensation for the current block may be performed based on the sub-block size. In this case, the subblock size of the affine block (ie the current block) is retrieved. If each of the sub-block width and height is greater than 4 chroma samples, then a motion vector for each sub-block can be extracted and DCT-IF based motion compensation (1/16 pixel for luma and 1/32 for chroma) can be performed on the sub-block . If not, then motion compensation based on the enhanced bilinear interpolation filter can be performed on all affine blocks.

[0212] In an embodiment, if the merge/skip flag is true and both the width and height of the CU are greater than or equal to 8, at the CU level, an affine flag is signaled via the bitstream indicating whether affine merge mode is used. When a CU is encoded as AF_MERGE, a merge candidate index with a maximum value of '5' is signaled to declare that a motion information candidate is used for the CU in the affine merge candidate list.

[0213]

[0214] FIG. 20 and 21 are diagrams for describing a method for creating an affine merge candidate list using an affine prediction encoded neighbor block according to an embodiment of the invention.

[0215] Referring to FIG. 20, the affine merge candidate list is configured through the following steps.

[0216] 1) Inserting a model-based affine candidate

[0217] A model-based affine candidate means that the candidate is derived from a valid adjacent reconstructed block encoded according to the affine mode. As illustrated in FIG. 20, the scan sequence for a candidate block corresponds from left A, top B, top right C, and bottom left D to top left E.

[0218] If the neighboring lower left block A is encoded in affine mode with 6 parameters, then motion vectors (v_4, v_5, v_6) of the upper left corner, upper right corner and lower left corner of the CU including block A are obtained. Vectors the motions (v_0, v_1, v_2) of the upper left corner of the current block are calculated based on the motion vectors (v_4, v_5 and v_6) according to a 6-parameter affine model.

[0219] If the neighboring lower left block A is affine-coded with 4 parameters, then the motion vectors (v_4, v_5) of the upper left corner and upper right corner of the CU including block A are obtained. Motion vectors (v_0, v_1) the upper left corner of the current block are calculated based on the motion vectors (v_4, v_5) according to the 4-parameter affine model.

[0220] 2) Insertion of control point-based affine candidates

[0221] Referring to FIG. 20, management point-based candidate means that the candidate is configured by combining neighboring control point motion information.

[0222] The motion information of the control points is first extracted from the assigned spatial neighbor blocks and the temporal neighbor block illustrated in FIG. 20. CP_k (k=1, 2, 3, 4) indicates the k-th control point. In addition, A, B, C, D, E, F, and G are spatial locations for CP_k (k=1, 2, 3) prediction, and H is a temporal location for CP4 prediction.

[0223] The coordinates of CP_1, CP_2, CP_3 and CP_4 are (0, 0), (W, 0), (H, 0) and (W, H), respectively. In this case, W and H are the width and height of the current block.

[0224] The motion information of each control point is obtained based on the next priority.

[0225] As for CP_1, the check priority corresponds to A _→ B _→ C and A is used if A is available. If not available, and if B is available, then B is used. If both A and B are not available, then C is used. If all three candidates are not available, traffic information of CP_1 cannot be obtained.

[0226] As for CP_2, the check priority corresponds to E _→ D.

[0227] As for CP_3, the check priority is G _→ F.

[0228] As for CP_4, H.

[0229] Second, combinations of control points are used to configure the motion model.

[0230] The motion vectors of the two control points are needed to calculate the transformation parameters in the 4-parameter affine model. Two control points can be selected from one of the following 6 combinations {CP_1, CP_4}, {CP_2, CP_3}, {CP_1, CP_2}, {CP_2, CP_4}, {CP_1, CP_3} and {CP_3, CP_4}. For example, to use control points CP_1 and CP_2 when creating an affine motion model with 4 parameters, they are marked as "affine (CP_1, CP_2)".

[0231] The motion vectors of the three control points are needed to calculate the transformation parameters in the 6-parameter affine model. Three control points can be selected from one of the following 4 combinations {CP_1, CP_2, CP_4}, {CP_1, CP_2, CP_3}, {CP_2, CP_3, CP_4} and {CP_1, CP_3, CP_4}. For example, to use control points CP_1, CP_2 and CP_3 when creating an affine motion model with 6 parameters, they are marked as "affine (CP_1, CP_2, CP_3)".

[0232] In addition, in an embodiment of the invention, if an affine merge candidate is present in the affine merge mode, then this can always be considered a 6-parameter affine mode.

[0233]

[0234] Affine Interframe Mode

[0235] FIG. 22 illustrates an example of neighboring blocks that are used for affine prediction mode in affine interframe mode in accordance with an embodiment of the invention.

[0236] Referring to FIG. 22, affine motion prediction may include affine merge mode (or AF_MERGE) and affine interframe mode (or AF_INTER). In the affine interframe mode (AF_INTER), after the 2-prediction of control point motion vectors (CPMVP) and CPMV is determined, a control point motion vector difference (CPMVD) corresponding to the difference can be transmitted from an encoder to a decoder. The detailed affine interframe mode (AF_INTER) encoding process may be exactly as described below.

[0237] Step-1: A candidate for a pair of two CPMVPs is determined

[0238] Step 1.1: Determine a maximum of 12 CPMVP candidate combinations (refer to Equation 2)

[0239] [Equation 2]

[0240] In Equation 2, v_0 indicates the motion vector CPMV0 at the upper left control point 2210 of the current block 2200. v_1 indicates the motion vector CPMV1 at the upper right control point 2211 of the current block 2200. v_2 indicates the motion vector CPMV2 at the control point 2212 on the lower left side of the current block 2200. v_A indicates the motion vector of the neighboring block 2220 A, which is in the upper left neighborhood of the upper left control point 2210 of the current block 2200. v_B indicates the motion vector of the neighboring block 2222 B, which is in the upper left neighborhood of the upper left control point 2210 of the current block 2200. v_C indicates a motion vector of an adjacent block 2224 C that is adjacent to the left of the top left control point 2210 of the current block 2200. v_D indicates a motion vector of an adjacent block 2226 D that is adjacent to the top right of the top right control point 2211 of the current block 2200 .v_E indicates the motion vector of the neighboring block 2228 E, which is found are adjacent to the top right of the upper right control point 2211 of the current block 2200. v_F indicates the motion vector of the adjacent block 2230 F, which is adjacent to the left of the lower left control point 2212 of the current block 2200. v_G indicates the motion vector of the neighboring G block 2232, which is located adjacent to the left of the lower left control point 2212 of the current block 2200.

[0241] Step-1.2: CPMVP candidate combinations are sorted based on a value with a small difference value (DV) and the top two candidates are used (refer to Equation 3 below).

[0242] [Equation 3]

[0243] v_0x indicates the x-axis element of the motion vector (V0 or CPMV0) at the upper left control point 2210 of the current block 2200. v_1x indicates the x-axis element of the motion vector (V1 or CPMV1) at the upper right control point 2211 of the current block 2200. v_2x indicates the element x axis of the motion vector (V_2 or CPMV_2) at the lower left control point 2212 of the current block 2200. v_0y indicates the y axis element of the motion vector (V_0 or CPMV_0) at the upper left control point 2210 of the current block 2200. v_1y indicates the y axis element of the motion vector (V_1 or CPMV_1) at the top right control point 2211 of the current block 2200. v_2y indicates the motion vector y-axis element (V_2 or CPMV_2_ at the bottom left control point 2212 of the current block 2200. w indicates the width of the current block 2200. h indicates the height of the current block 2200.

[0244] Step-2: Use the AMVP candidate list when the control point motion vector predictor pair candidate (CPMVP) is less than 2

[0245] Step-3: Determine the motion vector predictor of the control point (CPMVP) of each of the two candidates, and optimally select the candidate with the lowest value by comparing the costs of RD and CPMV

[0246] Step-4: Transmit index corresponding to optimal candidate and control point motion vector difference (CPMVD)

[0247] In an embodiment of the invention, a process for creating a CPMVP candidate in AF_INTER is provided. Identical to AMVP, the number of candidates is 2 and an index indicating the location in the list of candidates is signaled.

[0248] The process of creating a list of candidates for CPMVP is as follows.

[0249] 1) Whether neighboring blocks are encoded as affine motion prediction is checked by scanning neighboring blocks. If the scanned blocks are encoded as affine prediction, a pair of motion vectors of the current block is derived from the affine motion model of the scanned neighbor block until the number of candidates becomes 2.

[0250] 2) If the number of candidates is less than 2, a candidate configuration process is performed. Furthermore, in an embodiment of the invention, a 4-parameter affine interframe mode (with 2 control points) is used to predict the zoom and rotate motion pattern and content. As illustrated in FIG. 15, the block's affine motion field is described by two control point motion vectors.

[0251] The motion vector field (MVF) of the block is described by Equation 1 above.

[0252] In the conventional technology, the advanced motion vector prediction (AMVP) mode is necessary for scanning the motion vector prediction index (MVP) and motion vector difference (MVD). When the AMVP mode is applied to the invention, an affine flag (affine_flag) is signaled to indicate whether affine prediction is used. If affine prediction is applied, the syntaxes inter_dir, ref_idx, mvp_index, and two MVDs (mvd_x and mvd_y) are signaled. A list of candidates for an affine MVP pair is formed, which includes two affine MVP pairs. The signaled mvp_index is used to select one of two MVP affine pairs. An affine MVP pair is formed by two types of affine MVP candidates. One is a spatially inherited affine candidate and the other is a corner-derived affine candidate. If neighboring CUs are encoded in affine mode, then spatial legacy affine candidates can be generated. The affine motion model of the neighboring affine-coded block is used to form the motion vectors of the 2-control point MVP pair. The MVs of a 2-management MVP pair of a spatial legacy affine candidate are derived using the following equations.

[0253] [Equation 4]

V0x=VB0x + (VB2_x - VB0x) * (posCurCU_Y - posRefCU_Y) / RefCU_height + (VB1x - VB0x) * (posCurCU_X - posRefCU_X) / RefCU_width

[0254] [Equation 5]

V0y=VB0y + (VB2_y - VB0y) * (posCurCU_Y - posRefCU_Y) / RefCU_height + (VB1y - VB0y) * (posCurCU_X - posRefCU_X) / RefCU_width

[0255] If V_B0, V_B1, and V_B2 can be replaced by the top left MV, top right MV, and bottom left MV of a given reference/adjacent CU, then (posCurCU_X, posCurCU_Y) is the location of the top left sample of the current CU for the top left sample of the frame. (posRefCU_X, posRefCU_Y) is the top left sample location of the reference/neighbor CU for the top left sample of the frame.

[0256] [Equation 6]

V1x=VB0x + (VB1x - VB0x) * CU_width/RefCU_width

[0257] [Equation 7]

V1y=VB0y + (VB1y - VB0y) * CU_width/RefCU_width

[0258] FIG. 23 illustrates an example of adjacent blocks that are used for affine motion prediction in affine interframe mode in accordance with an embodiment of the invention.

[0259] Referring to FIG. 23, when the number of MVP pairs is less than 2, the affine candidate extracted from the corner is used. As illustrated in FIG. 23, neighboring motion vectors are used to extract the affine MVP pair. For the first affine candidate extracted from the corner, the first available MV in set A (A0, A1 and A2) and the first available MV in set B (B0 and B1) are used to configure the first pair of MVPs. As for the second affine candidate extracted from the corner, the first available MV in set A and the first available MV in set C (C0 and C1) are used to compute the MV of the upper right control point. The first available MV in set A and the computed MV of the upper right control point are the second pair of MVPs.

[0260] In an embodiment of the invention, two candidate sets including two (three) candidates {mv_0, mv_1} ({mv_0, mv_1, mv_2) are used to predict two (three) control points of the affine motion model. The given motion vector differences (mvd_0, mvd_1, mvd_2) and control points are calculated using the following equations.

[0261] [Equation 8]

[0262] FIG. 24 and 25 are diagrams illustrating a method for extracting a motion vector candidate using motion information of an adjacent block in affine interframe mode, in accordance with an embodiment of the invention.

[0263] In an affine candidate list, affine motion propagates from spatial neighboring blocks (extrapolated affine candidates), and the affine candidate list is augmented by a combination of motion vectors from spatial neighboring blocks (virtual affine candidates). Candidate sets are set up as follows:

[0264] 1. A maximum of two different sets of MV affine predictors are derived from the affine motion of neighboring blocks. Neighbor blocks A0, A1, B0, B1 and B2 are checked as illustrated in FIG. 24. If an adjacent block is encoded by an affine motion model and the corresponding reference frame is the same as the reference frame of the current block, then two control points (for a 4-parameter affine model) of the current block or three control points (for a 6-parameter affine model) parameters) of the current block are extracted from the affine model of neighboring blocks.

[0265] 2. FIG. 25 illustrates neighboring blocks that are used to form a set of virtual affine candidates. Neighbor MVs are divided into three groups: S_0={mv_A, mv_B, mv_C}, S_1={mv_D, mv_E} and S_2={mv_F, mv_G}. mv_0 is the first MV that refers to the same reference picture as the current block in S0. mv_2 is the first MV that refers to the same reference picture as the current block in S1.

[0266] If mv_0 and mv_1 are given, then mv_2 can be extracted by Equation 9 below.

[0267] [Equation 9]

[0268] In Equation 9, the size of the current block corresponds to W×H.

[0269] If only mv_0 and mv_2 are given, then mv_1 can be extracted by Equation 10 below.

[0270] [Equation 10]

[0271] In an embodiment of the invention, inter-frame affine prediction may be performed in accordance with the following sequence.

[0272] Input: affine motion parameters, reference picture samples

[0273] Output: CU prediction block

[0274] Processor

[0275] Retrieve the sub-block size of the affine block

[0276] - If both the width and height of the subblock are greater than 4 luma samples,

[0277] -- As for each sub-block,

[0278] - Extract the motion vector of the subblock

[0279] - Perform motion compensation (1/16 pixel unit for luma and 1/32 pixel unit for chrominance) based on DCT-IF over subblocks (call)

[0280] - If not, then perform (call) compensation based on the improved bilinear interpolation filter over all affine blocks

[0281] In addition, in an embodiment of the invention, if the merge/skip flag is false and the width and height of the CU is greater than or equal to 8, then an affine flag is signaled to indicate whether affine interframe mode will be used at the CU level. If the CU is encoded in affine interframe mode, then a model flag is signaled to indicate whether a 4-parameter or 6-parameter affine model is applied to the CU. If the model flag is true, then the AF_6_INTER mode (affine model with 6 parameters) is applied and the MVDs are analyzed. If not, then the AF_4_INTER mode (affine model with 4 parameters) is applied and two MVDs are analyzed.

[0282] In the AF_4_INTER mode, like the affine merge mode, pairs of motion vectors extrapolated from neighboring blocks encoded by the affine mode are generated and preferably inserted into the candidate list.

[0283] Then, if the candidate list size is less than 4, then candidates with motion vector pair {(v_0,v_1)|v0={v_A, v_B, v_c},v_1={v_D, v_E}} are generated using neighboring blocks. As illustrated in FIG. 25, v_0 is selected from the motion vectors of blocks A, B, and C. The motion vector from the neighboring block is scaled based on the relationship between the reference list, POC for neighbor block reference, POC for current CU reference, and current CU. In addition, the approach of selecting v_1 from neighboring blocks D and E is similar. When the candidate list is greater than 4, the candidates are first sorted based on the relatedness of neighboring motion vectors (similar to two motion vectors in a candidate pair), and the first 4 candidates are kept.

[0284] If the number of candidate lists is less than 4, then the list is populated with a pair of motion vectors by duplicating AMVP candidates.

[0285] In the AF_6_INTER mode, similarly to the affine merge mode, triplets of motion vectors (affine motion vector triplets) are generated that are extrapolated from neighboring blocks encoded in the affine merge mode and preferably inserted into the candidate list.

[0286] Thereafter, when the candidate list size is less than 4, the candidates including the motion vector triplets {(v_0, v_1, v_2)| v0={v_A, v_B, v_c}, v1={v_D, v_E}, v2={v_G, v_H}} are formed using neighboring blocks. As illustrated in FIG. 25, v_0 is selected from the motion vectors of block A, B, or C. The motion vector from the neighboring block is scaled based on the relationship between the reference list, POC for neighbor block reference, POC for current CU reference, and current CU. In addition, the approach for selecting v_1 from adjacent blocks D and E is similar to selecting v_2 from F and G. When the list of candidates is greater than 4, the candidates are sorted based on the relatedness of adjacent motion vectors (similar to two motion vectors in three candidates), and the first 4 candidates are saved.

[0287] When the number of candidate lists is less than 4, the list may be populated with triplets of motion vectors configured by duplicating the respective AMVP candidates.

[0288] After the CPMV of the current CU is retrieved, the MVF of the current CU is generated in accordance with Equation 11 for a 4-parameter affine model, and generated in accordance with Equation 12 for a 6-parameter affine model, based on the number of affine parameters.

[0289] [Equation 11]

[0290] [Equation 12]

[0291] In this case, the sub-block size M×N is extracted in Equation 13, and MvPre is the motion vector part precision (1/16).

[0292] [Equation 13]

[0293] After being extracted by Equation 12, M and N must be adjusted downwards, if necessary, in order to make them a divisor of w and h. When M or N is less than 8, WIF is applied. If not, then affine motion compensation is applied based on the subblock.

[0294] FIG. 26 illustrates an example of an affine field of motion vectors of a subunit unit in accordance with an embodiment of the invention.

[0295] Referring to FIG. 26, in order to extract the motion vector of each M×N sub-block, the motion vector of the center sample of each sub-block, such as that illustrated in FIG. 26 is calculated according to Equation 11 or Equation 12, and is rounded down to 1/16th. SHVC upsampling interpolation filters are used to generate a prediction of each sub-block using the extracted motion vector.

[0296] SHVC upsampling interpolation filters with exactly the same filter length and normalization factor as HEVC motion compensation interpolation filters can be used as motion compensation interpolation filters for additional fractional image unit positions. The motion vector accuracy of the chrominance component is 1/32 of the sample. Additional interpolation filters for 1/32 image unit locations are extracted using the averages at two adjacent 1/16 image unit locations.

[0297] The AF_MERGE mode can be selected at the encoder side using the same general merge mode selection method. Preferably, a list of candidates is generated and a minimum RD cost is selected among the candidates for comparison with the RD costs of other inter-frame modes. The result of the comparison is to determine whether AF_MERGE is applied or not.

[0298] With respect to the AF_4_INTER mode, an RD cost check is used to determine whether a motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) of the current CU. Once the CPMVP of the current affine CU is determined, the affine motion estimate is applied and a control point motion vector (CPMV) is obtained. Accordingly, the difference between CPMV and CPMVP is determined.

[0299] On the encoder side, the AF_6_INTER mode is only identified when the AF_MERGE or AF_4_INTER mode is determined as the optimal mode in the previous mode selection stage.

[0300] In an embodiment, the affine interframe (affine AMVP) mode may be performed as follows:

[0301] 1) AFFINE_MERGE_IMPROVE: Instead of searching for the first neighbor block in affine mode, the improvement is to find the neighbor block with the maximum coding unit size as an affine merge candidate.

[0302] 2) AFFINE_AMVL_IMPROVE: Adjacent blocks in affine mode are added to the affine AMVP candidate list similar to the general AMVP procedure.

[0303] The detailed process for generating an affine list of AMVP candidates is as follows.

[0304] First, it is identified whether the bottom left adjacent block uses an affine motion model and has the same reference index as the current reference index. If no neighbor box is present, the left neighbor box is identified using the same method. If the neighbor box is not present, it is identified whether the bottom left neighbor box uses an affine motion model and has a different reference index. If a neighboring block is present, the scaled affine motion vector is added to the reference picture list. If no neighbor box is present, the left neighbor box is identified using the same method.

[0305] Second, the top right neighbor, the top neighbor, and the top left neighbor are identified using the same method.

[0306] After the processes, if two candidates are found, then the AMVP affine candidate list generation process ends. If no two candidates are found, the original operation within the JEM software is performed to generate affine lists of AMVP candidates.

[0307] 3) AFFINE_SIX_PARAM: In addition to the 4-parameter affine motion model, the 6-parameter affine motion model is added as an additional model.

[0308] A 6-parameter affine motion model is derived by Equation 14.

[0309] [Equation 14]

[0310] Three motion vectors at the top left location MV_0, the top right location MV_1, and the bottom left location MV_2 are needed to determine the model because 6 parameters are present in the motion model. The three motion vectors can be determined using a method similar to the two motion vector method in the 4-parameter affine motion model. An affine model merge is always set as a 6-parameter affine motion model.

[0311] 4) AFFINE_CLIP_REMOVE: Removes motion vector constraints for all affine motion vectors. The motion compensation process is designed to manage the motion vector constraints themselves.

[0312]

[0313] Affine motion model

[0314] As described above, various affine motion models may be used or considered in affine interframe prediction. For example, an affine motion model could be four motions as in FIG. 14. An affine motion model capable of representing three motions (parallel translation, scaling and rotation), among the motions that can be represented by an affine motion model, may be referred to as a similarity (or simplified) affine motion model. The number of CPMVs and/or the method of extracting the MV of a sample/sub block unit from the current block may be different depending on which one of the affine motion models is used.

[0315] In an embodiment of the invention, adaptive motion models with four or six parameters are used. In AF_INTER, a 6-parameter motion model is offered in addition to the 4-parameter motion model that is present in JEM. An affine motion model with 6 parameters is described similarly to Equation 15.

[0316] [Equation 15]

x' = a*x+b*y+c

y' = d * x+e * y+f

[0317] In this case, the coefficients a, b, c, d, e, and f are affine motion parameters. (x, y) and (x',y') are coordinates at pixel locations before and after the affine motion model transformation. In video coding, in order to use the affine motion model, if CPMV0, CPMV1 and CPMV2 are MVs for CP0 (upper left), CP1 (upper right) and CP2 (lower left), then Equation 16 can be described as follows.

[0318] [Equation 16]

[0319] Here, CPMV_0={v_0x, v_0y}, CPMV_1={v_1x, v_1y}, CPMV_2={v_2x, v_2y}, and w and h are the width and height of each coding block. Equation 16 is the motion vector field (MVF) of the block.

[0320] A flag is parsed at the CU level to indicate whether a 4-parameter or 6-parameter affine motion model is used when neighboring blocks are encoded as affine prediction. If a neighboring block encoded as affine prediction is not present, then the flag is omitted and a 4-parameter model is used for affine prediction. In other words, a 6-parameter model is considered under the condition that one or more neighboring blocks are encoded in an affine motion model. With respect to the number of CPMVDs, each of two or three CPMVDs are signaled against 4-parameter and 6-parameter affine motion models.

[0321] In addition, in an embodiment of the invention, motion vector refinement with pattern matching can be used. In sample-matched motion vector extraction (PMMVD, hereinafter abbreviated as PMVD in the JEM encoder description) of JEM, the decoder needs to evaluate some motion vectors (MVs) in order to determine an initial MV candidate for CU layer search. When searching at the sub-CU level, some MV candidates are added in addition to the optimal MV of the CU level. The decoder needs to evaluate such MV candidates in order to search for the optimal MV. This requires a lot of memory bandwidth. The proposed pattern-matched motion vector refinement (PMVR) uses the JEM concepts of pattern matching and two-way matching in PMVD. When a skip mode or merge mode is selected to indicate whether a PMVR is available, one PMVR_flag is signaled. In order to significantly reduce the memory bandwidth requirement compared to PMVD, a list of MV candidates is formed. If PMVR is applied, then the initial MV candidate index is explicitly signaled.

[0322] A candidate list is generated using a merge candidate list generation process, but sub-CU merge candidates such as affine candidates and ATMVP candidates are excluded. For two-way matching, only mono-prediction MV candidates are included. Bi-prediction MV candidates are divided into two mono-prediction MV candidates. In addition, similar MV candidates (with MV differences less than a predetermined threshold value) are also removed. For CU-level searches, diamond search MV refinement is performed starting from the signaled MV candidate.

[0323] Searching at the sub-CU level is only available in merge mode with two-way matching. The search window for searching at the sub-CU level for all sub-CUs is exactly the same as the search window for searching at the CU level. Accordingly, no additional bandwidth is required when searching at the sub-CU level.

[0324] In order to refine the MVP in the mode, pattern matching is also used. In AMVP mode, two MVPs are generated using the HEVC MVP generation process, and one MVP index is signaled to select one of the two MVPs. The selected MVP is further refined using pattern matching in PMVR. If adaptive motion vector resolution (AMVR) is applied, then the MVP is rounded to the appropriate precision before template matching refinement. This refinement process is referred to as Motion Vector Predictor with Sample Matching (PMVPR) refinement. In the remainder of this document, PMVR includes pattern-matching PMVR, bi-directional matching PMVR, and PMVPR unless specifically defined otherwise.

[0325] In order to reduce the memory bandwidth requirement, PMVR is unavailable for 4x4, 4x8 and 8x4 CUs. In order to reduce the additional memory bandwidth required, the search range {pattern matching, bidirectional matching} for a CU area of 64 can be reduced to {±2,±4}. The search range {pattern matching, bidirectional matching} for a CU area greater than 64 may be reduced to {±6, ±8}. Compared to the worst case in HEVC, the required memory bandwidth was reduced from 45.9x in PMVD in JEM-7.0 to 3.1x in PMVR using all the methods described in the PMVR section of this document.

[0326]

[0327] General history-based motion vector prediction (HMVP)

[0328] In general, the image compression technology uses the exploitation of spatial and temporal redundancy as two main schemes. For example, both High Efficiency Video Coding (HEVC) and VVC use two motion compensation schemes based on interframe coding. One is motion fusion and the other is advanced motion vector prediction (AMVP). In order to improve the two prediction modes, various modifications have been discussed. Various modifications include increasing the number of candidates to search for more spatially expanded candidates and testing temporary candidates at non-traditional locations. The two schemes include initially creating a list using the available candidates, minimizing the rate-distortion (RD) cost, and signaling the selected candidate in the bitstream.

[0329] In particular, in modern image compression technology, HMVP is discussed, in which motion information of a previously encoded block is stored, and the stored motion information is used to predict the motion of a subsequently encoded block. Such an HMVP may be added to a merge list (or merge candidate list) or an AMVP list (or AMVP candidate list).

[0330] The decoder maintains a lookup table (LUT) in a first-come-first-served (FIFO) system (or method) for HMVP. In the invention, the LUT is not limited by its name, and may be referred to as a table, HMVP table, HMVP candidate table, buffer, HMVP buffer, HMVP candidate buffer, HMVP list, or HMVP candidate list. In particular, when a non-affine prediction unit (PU) (or coding unit (CU)) is decoded, the corresponding motion information is stored in the LUT. The decoder performs decoding on the next PU. Here, the stored motion information may include motion vectors, reference index information, and mode information in the x (horizontal) and y (vertical) directions.

[0331] The decoder maintains a LUT that stores motion information of a progressively decoded non-affine candidate. The LUT size may be limited to predefined S candidates. In one embodiment, the LUT may be reset at the beginning of a slice, the beginning of a CTU row, or the beginning of a CTU.

[0332] HMVP can be applied to both merge mode and AMVP mode. A merge list can have B candidates, and an AMVP list can have two candidates. In a conventional image compression technology, a merge list is configured from the following candidates: i) a spatial candidate, ii) a temporal candidate, iii) a bidirectional prediction (Bi-Pred) candidate, iv) a zero motion candidate. Recently, a method for further considering advanced motion vector prediction (ATMVP) as a candidate has been discussed. For example, an ATMVP candidate may be inserted into the merge list before the temporary candidate. Merge list candidates are added to the merge list until they reach the maximum size of the merge list. The duplicate candidate may not be added to the merge list. Two candidates can be inserted into the AMVP list. For example, a first candidate of two candidates may be selected from available spatial candidates, and a second candidate may be selected from temporal candidates. If the list is empty, a null motion vector candidate may be added.

[0333] The HMVP is applied based on the FIFO order in which candidates exit the LUT identically to their entry order.

[0334] In one embodiment, when the HMVP is applied to a merge list configuration, the HMVP candidate may be inserted (or added) to the third list location as follows:

[0335] 1. Spatial Candidate

[0336] 2. Provisional Candidate

[0337] 3. Up to S HMVP Candidates for LUT

[0338] 4. United Candidate Bi-Pred

[0339] 5. Motion Vector Candidate

[0340] In one embodiment, when an HMVP is applied to an AMVP list configuration, the HMVP may be inserted in a third location after the temporary candidate as follows:

[0341] 1. Spatial Candidate

[0342] 2. Provisional Candidate

[0343] 3. Up to K HMVP Candidates

[0344] 4. Zero Motion Vector Candidate

[0345] FIG. 27 is a flowchart for describing an HMVP storage method according to an embodiment of the invention.

[0346] Referring to FIG. 27, the decoder decodes the current PU (or CU) (S2701).

[0347] The decoder checks if the current PU is a non-affine encoded block (S2702). In order to facilitate the use of HMVP candidates, if the current PU is an affine mode encoded block, the decoder does not store the movement information of the current PU in a table.

[0348] If the current PU is a block encoded in a non-affine mode, the decoder stores (or updates) the movement information of the current PU in a table (S2703).

[0349] In an embodiment of the invention, the HMVP table can be updated using two methods, i. ways i) FIFO without restrictions ii) FIFO with restrictions. In the first case, redundant motion information may be present, but no clipping process is applied. This helps reduce the complexity of the entire process. On the other hand, in the latter case, the clipping process is applied and no redundant motion information is present. This is described with reference to the next figure.

[0350] FIG. 28 is a diagram for describing an HMVP table that operates according to an unrestricted FIFO queue discipline according to an embodiment of the invention.

[0351] Referring to FIG. 28, the candidate that is added to the table is added to the end (right) of the table. In contrast, the candidate that is released from the table according to the FIFO method is positioned at the front (left side, oldest candidate) of the table.

[0352] If the table is not completely filled with the maximum number of predefined candidates with index L-1 (ie, end), new candidates are added without deleting the candidate. In contrast, if the table is already completely filled, i.e. if the maximum number of tables is satisfied, then the candidate that is located in the front, i.e. the oldest in the table is removed and a new candidate is added.

[0353] FIG. 29 is a diagram for describing an HMVP table that operates according to a constrained FIFO queue discipline in accordance with an embodiment of the invention.

[0354] Referring to FIG. 29, in the case where a restricted FIFO is used, pruning is performed if the addition of a new candidate causes any redundancy (ie, the new candidate includes redundant motion information). In an embodiment, if a candidate with redundant motion information is present in the table, then the redundant candidate in the table is removed and the current candidate's motion information can be added.

[0355]

[0356] With regard to HMVP candidates, in many cases, the most recent historical MVs may overlap motion information of a spatial candidate (or spatial neighbor candidate). Accordingly, the present embodiment proposes a method for setting the order of adding candidates differently from the order of HMVP LUT indexes when an HMVP candidate is added to a merge list or AMVP.

[0357] According to an embodiment of the invention, the candidate list can be efficiently configured by adaptively adjusting the HMVP candidate. Accordingly, the number of signaling bins that are used for binarization can be reduced and coding efficiency can be improved. Those. an HMVP candidate that is added to a merge list or an AMVP list may not be limited to an index in the HMVP list. As an embodiment, the following Table 1 illustrates a method for changing the order in which an HMVP candidate is added to a merge list or AMVP.

[0358] [Table 1]

HMVP LUT index (0 denotes the most recent historical MV) Order to be added to the Merge or AMVP list 0 2 one 0 2 one 3 3 .. ..

[0359] Referring to Table 1 as described above, there is a good chance that the most recently inserted HMVP candidate may have the same motion information as the spatial candidate. Accordingly, the addition order of the HMVP candidate can be predetermined regardless of the HMVP index in view of the probability.

[0360] In addition, in one embodiment, the encoder or decoder may add the HMVP candidate to the merge list or AMVP list, starting with the HMVP candidate that starts with the nth candidate in the list. Table 2 below illustrates the modified order in which a candidate is added to the merge list or AMVP.

[0361] [Table 2]

HMVP LUT index (0 denotes the most recent historical MV) Order to be added to the Merge or AMVP list 0 one 0 2 one 3 2 .. ..

[0362] Referring to Table 2, the HMVP candidate may be added to the merge list or the AMVP list from the second index.

[0363] In one embodiment, information regarding the order in which the HMVP candidate is added to a table (LUT) may be signaled from an encoder to a decoder. For example, such queuing information may be conveyed via High Level Syntax (HLS). The high-level syntax may be a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, an coding unit, and/or other appropriate syntax data header, for example.

[0364] Table 3 illustrates a high-level syntax structure to which the method of the invention can be applied.

[0365] [Table 3]

high_level_parameter_set(){ Description … set_HMVP_order_flag u(1) …

[0366] Referring to Table 3, set_HMVP_order_flag equal to 1 indicates that set_HMVP_order_flag is present in the slice header in non-IDR pictures in CVS. set_HMVP_order_flag equal to 0 indicates that set_HMVP_order_flag is not present in slice headers and adaptive HMVP is not used in VCS.

[0367] Table 4 illustrates a slice segment header syntax structure to which the method of the invention can be applied.

[0368] [Table 4]

slice_segment_header(){ Description … if(set_HMVP_order_flag) slice_HMVP_idx u(1) … } …

[0369] Referring to Table 4, slice_HMVP_idx means an index for the sequence of candidates used. For example, a slice_HMVP_idx of 0 may represent a basic HMVP sequence such as 0, 1, 2, 3. Similarly, an index value of 1 may be used to represent an HMVP sequence of 3, 2, 1, 0.

[0370]

[0371] In an embodiment of the invention, in addition to the HMVP LUT, a method is provided for using a long-term list for motion prediction. Accordingly, the number of supported HMVP candidates can be increased. In an embodiment, 2 HMVP tables may be considered. In this case, one can be used to store generic HMVP candidates, and the other can be used as a long-term list that stores candidates that require further support.

[0372] The following illustrates methods for resetting and creating a long-term list (or long-term HMVP list).

[0373] - After the first CTU of a CTU string is decoded, one or more historical MVs of the subsequent CTU may be added to the long-term HMVP LUT. Such a long-term HMVP LUT may not be used or updated until the next CTU line.

[0374] - At the start of the next CTU line, the long-term HMVP LUT can be used to reset the overall HMVP LUT. The reason for this is that the HMVP candidates of the CTU at the beginning of a CTU row may be more related than the historical MV at the end of the previous CTU row.

[0375] -- The above process can be repeated.

[0376] FIG. 30 is a diagram illustrating an HMVP LUT and a long term HMVP LUT according to an embodiment of the invention.

[0377] Referring to FIG. 30, the encoder or decoder may include two LUTs for storing HMVP candidates. One may be an HMVP LUT (either a general HMVP LUT or a short term HMVP LUT) and the other may be a long term HMVP LUT. When an HMVP candidate is added to both the merge list and the AMVP list, it may be added from an HMVP LUT or a long-term LUT, as illustrated in FIG. thirty.

[0378] The use of a long-term LUT can be signaled by using a new syntax element. In an embodiment, the syntax element may be signaled by a high-level syntax. For example, a syntax element may be present in a sequence parameter set, picture parameter set, slice header, coding tree unit, coding unit, and/or other appropriate syntax data header.

[0379] In an embodiment of the invention, a method is provided that considers flexibility for decoding when adding an HMVP candidate to an HMVP LUT. The encoder/decoder may consider the decision criterion for one or more PU (or CU) characteristics when adding an HMVP candidate to the HMVP LUT.

[0380] In an embodiment, the encoder/decoder may consider the following content when adding the HMVP candidate to the table. The encoder/decoder may add a table candidate based on characteristics such as the mode (eg, merge mode, affine mode, or AMVP mode) of the PU and/or block size, alone or in combination. In one embodiment, in addition to the characteristics, other additional characteristics may be considered. For example, the type of merge (eg, spatial candidate or temporal candidate) in which an HMVP LUT update is considered, whether it is a sub-PU, etc., may be taken into account as a criterion for selecting a candidate. A selection criterion may be defined to reduce redundancy with previous history (or previous HMVP). For example, if a PU is encoded in merge mode and the type of merge is spatial merge, then the decoder may not update the HMVP LUT with motion information of the corresponding PU.

[0381] FIG. 31 is a diagram illustrating an example of a method for updating an HMVP LUT according to an embodiment of the invention.

[0382] Referring to FIG. 31, the encoder/decoder obtains motion information of the encoded candidate (S3101).

[0383] The encoder/decoder judges whether to update the LUT with candidate motion information based on a predetermined decision criterion (S3102). As described above, the decision criterion may include characteristics associated with one or more of a candidate mode (eg, merge mode, affine mode, or AMVP mode), candidate block size, and/or candidate merge type.

[0384] The encoder/decoder updates the LUT based on the decision criterion (S4303). Those. if the candidate satisfies the predetermined decision condition, then the encoder/decoder may add candidate motion information to the LUT.

[0385]

[0386] In an embodiment of the invention, there are proposed restrictions regarding the redundancy check for adding an HMVP candidate to the merge list (or AMVP list). Redundancy check constraints can be defined or implemented in various ways.

[0387] In one embodiment, the encoder/decoder may limit the number of pruning checks for the first specific number of candidates in the merge list. As an embodiment, the encoder/decoder may limit the number of clipping checks for candidates from the first merge list candidate to a particular candidate. For example, the encoder/decoder may perform a pruning process on candidates from the first candidate of the merge list to a specific candidate. And the HMVP candidate, i.e. the target of the clipping test, can be limited to a predetermined number.

[0388] In addition, in one embodiment, the encoder/decoder may limit the pruning check by performing a pruning check on a particular merge candidate type in the merge list. For example, the encoder/decoder may only perform a clipping check against a spatial merge list candidate when adding an HMVP candidate. Alternatively, for example, the encoder/decoder may perform a clipping check on only some of the spatial merge list candidates when adding the HMVP candidate. Some of the spatial candidates may be predefined. For example, the predefined spatial candidates may be at least one of a left neighbor spatial candidate and/or a top neighbor spatial candidate. Or, for example, when adding an HMVP candidate, the encoder/decoder may perform a clipping check on only a part of the spatial candidates of the merge list, and the part of the spatial candidates may be predefined as left and top sides. An embodiment of the invention is not limited to the above examples, and other types may be combined and limited as the target of the clipping test.

[0389] FIG. 32 is a diagram illustrating a method for limiting the number of HMVP candidates, i. e. clipping test purposes, in accordance with an embodiment of the invention.

[0390] Referring to FIG. 32, in an embodiment of the invention, the number of HMVP candidates, i. e. the target of the clipping check may be limited to M. The encoder/decoder may check for redundancy of motion information among the first M HMVP LUT candidates and merge candidates in the merge list when creating the merge list using HMVP candidates.

[0391] Alternatively, the encoder/decoder may check for redundancy of motion information among the first M HMVP LUT candidates and the current decoded PU when adding motion information of the decoded processing unit (eg, PU) to the HMVP LUT.

[0392] FIG. 33 is a flowchart illustrating an example of a method for performing a clipping test according to an embodiment of the invention.

[0393] Referring to FIG. 33, the encoder/decoder obtains motion information of the decoded candidate and determines (or decodes) the number of clipping checks (S3301, S3302). The number of clipping checks may be predetermined at the encoder/decoder in accordance with the method described above (eg, as described in FIG. 32). The encoder/decoder performs a clipping test based on the determined number of clipping tests (S3303).

[0394] In one embodiment, as in Table 4 and Table 5, information that is associated with a clipping check can be signaled through a high-level syntax. In this case, the syntax element that is passed from the encoder to the decoder may be signaled by a high-level syntax to indicate the number of clipping checks. The high-level syntax may be included in a sequence parameter set, picture parameter set, slice header, coding tree unit, coding unit, and/or other appropriate syntax data header, for example.

[0395]

[0396] In an embodiment of the invention, an efficient method for selecting an HMVP candidate is provided. When a historical motion vector candidate (ie, HMVP candidate) is inserted into the merge list (or AMVP list), then a pruning check can be performed such that the HMVP candidate does not overlap the existing merge list. At this time, in order to perform cumulative redundancy checks between a redundancy list of size M and a historical LUT of size N, checks are required (M-1)×N times.

[0397] Accordingly, in an embodiment of the invention, the number of HMVP candidates may depend on the merger candidate. For example, the number of HMVP candidates may depend on the number of spatial candidates that are present in the merge list. Alternatively, for example, the number of HMVP candidates may depend on the number of spatial candidates and temporal candidates that are present in the merge list.

[0398] If there is another merge candidate in the merge list, then the number of HMVP candidates on which a pruning check will be performed based on a particular criterion (or rule) based on the number of merge candidates and/or the number of HVVP merge list may be reduced. Accordingly, the number of redundancy checks in the worst case can be reduced.

[0399] For example, in the case of a merge list of size (or length) of 6, if the merge list is full, then the merge list may include a maximum of 5 spatial or other merge candidates. In order to insert an HMVP candidate into 6 HMVP lists, 30 redundancy checks may be required in the worst case.

[0400] In one embodiment, an example related to restrictions on the number of HMVPs that should be clipping test targets is illustrated in Equation 17 and Table 5.

[0401] [Equation 17]

if (existing_candidates >= 3)

number_hist_to_check=7 - existing_candidates

[0402] [Table 5]

# of an existing candidate # of an existing candidate to test # historical MV to check # checks one one 6 6 2 2 6 12 3 3 four 12 four four 3 12 5 5 2 ten

[0403] Referring to Table 5, the number of redundancy checks for adding an HMVP in the worst case can be reduced by a factor of 12 rather than a factor of 30 by limiting the number of HMVPs, i.e. clipping test targets, up to 2.

[0404]

[0405] In an embodiment of the invention, a method for creating a merge list using history-based space-time motion vector prediction (H-STMVP) is provided. H-STMVP indicates a candidate extracted as the average of two history-based spatial MVPs and TMVPs. Two spatial HMVPs can be obtained from the HMVP buffer. The TMVP can be obtained from the current merge list. In this case, the spatial candidate may be a candidate obtained from the last 2 encoded MVs in the decoding sequence up to the current block.

[0406] For example, the last encoded MV (referred to as MV_L in the invention), MV, i. e. the latter to the second (referred to as MV_(L-1) in the invention), and MV_TMVP may be used to generate an H-STMVP candidate to be inserted into the merge list.

[0407] If all three candidates can be used, then the MVs that are added to the merge list can be calculated by Equation 18 below.

[0408] [Equation 18]

[0409] In one embodiment, if only two of the three candidates are available, then only two candidates may be averaged to form an H-STMVP. Similarly, if only one candidate is available, then one candidate can be used. If no available candidate is present, then the H-STMVP may not be used to configure the merge list.

[0410] In an embodiment of the invention, a method for obtaining an H-STMVP candidate motion vector using a method other than Equation 18 is provided.

[0411] For example, instead of averaging three or more candidates at the same time, first averaging the spatial candidates and then averaging the two candidates again using the result of the averaging may be computationally simpler. An example of this is illustrated in the following equations.

[0412] [Equation 19]

[0413] Alternatively, the average value can be obtained as follows.

[0414] [Equation 20]

[0415] [Equation 21]

[0416] [Equation 22]

[0417] The encoder/decoder may first average the two candidates as in Equations 19 through 21, and then may average the resulting values using the third candidate. Alternatively, the encoder/decoder may assign a higher importance/weight to the candidate, ie. MV_L, by applying the shift operation twice, as in Equation 22. The mean value can be extracted without a division operation, but only through a shift operation, using Equations 19 through 22.

[0418]

[0419] In an embodiment of the invention, a method is provided that uses a given number (n) of spatial candidates instead of two history-based spatial candidates in H-STMVP extraction. The n candidates need not be essentially consecutive decoding sequences. n candidates can be chosen arbitrarily or according to some rule.

[0420] Accordingly, Equation 18 can be represented using a more general method, similar to Equation 23 below.

[0421] [Equation 23]

[0422] In another embodiment, assuming 5 spatial candidates are used, the effect of increasing spatial candidates on generating an H-STMVP candidate can be minimized, and the spatial candidate and temporal candidate can be appropriately combined by improving the weighting factor that is applied to the temporal candidate.

[0423] Accordingly, to this end, after the spatial candidates are averaged together using Equation 24, the above goal can be achieved by averaging the MV-TMVP using the result of the averaging.

[0424] [Equation 24]

[0425]

[0426] In an embodiment of the invention, a method is provided for adding a weight (or weight factor) to a motion vector candidate that is used to derive an H-STMVP. In this case, the weighting factor may be determined empirically, or may be determined based on the temporal distance to the fixed reference frame, or may be determined based on the location in the historical table. For example, a new candidate may have heavier weights than a previous candidate.

[0427] i.e. in the present embodiment, Equation 18 may be represented similarly to Equation 25 below.

[0428] [Equation 25]

[0429] In this case, the weight coefficient may have the same value or a non-uniformly distributed value.

[0430]

[0431] In an embodiment of the invention, a motion vector scaling method is provided that is used to extract an H-STMVP candidate as one reference picture.

[0432] FIG. 34 is a diagram for describing a method for extracting an H-STMVP candidate using motion vectors that refer to different reference pictures according to an embodiment of the invention.

[0433] Referring to FIG. 34, candidates MV_L, MV_L-1, and MV_TMVP are assumed to refer to (or point to) respective different reference pictures. Those. Fig. 34 illustrates that the candidates that are used to generate the H-STMVP candidates may have different reference indices and, as a result, have different reference frames.

[0434] The average of Equations 18 through 25 may not be equal to the result value because a frame with a closer reference frame may substantially have a greater effect on the H-STMVP motion vector. Accordingly, a method is provided for scaling all motion vectors in one reference frame for equal comparison and merging.

[0435] In this case, the encoder can determine which one frame, performed as part of the RD optimization, is the most suitable for use as a reference frame. In an embodiment, the selected reference frame may be signaled in the slice header similar to the TMVP array index that is present in the slice header. For example, the reference frame to be used may be generated using a fixed rule. Alternatively, for example, the list may be scaled as the first available reference frame from L0, or the list may be scaled based on the current picture's sequence number value.

[0436] In one embodiment, to achieve the above goal, an encoder may send information for a single fixed picture to a decoder using high-level syntax (HLS), which may be part of a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, and/ or some other data header. For example, a high-level syntax structure such as Table 6 and/or Table 7 below may be defined.

[0437] [Table 6]

high_level_parameter_set(){ Description … set_HSTMVP_ref_pic_flag u(1) …

[0438] Referring to Table 6, set_HSTMVP_ref_pic_flag equal to 1 indicates that set_HSTMVP_idx is present in the header of a non-IDR picture slice in CVS. set_HSTMVP_ref_pic_flag equal to 0 indicates that set_HSTMVP_idx is not present in the slice header.

[0439] [Table 7]

slice_segment_header(){ Description … if(set_HSTMVP_ref_pic_flag) slice_HSTMVP_idx u(1) … } …

[0440] Referring to Table 7, slice_HMVP_idx denotes a reference index. In one embodiment, the reference index may be selected with respect to the list L0.

[0441]

[0442] In embodiments of the invention, more detailed embodiments will be described in relation to the embodiments described above. In particular, a method is provided for indirectly using an affine HMVP candidate by using position and dimensional information in order to calculate or extract the CPMV of the current block. In the invention, the extracted CPMV may be referred to as a legacy HMVP affine candidate. A legacy HMVP Affinity Candidate in accordance with an embodiment of the invention may be used in the above described process for generating a Merge Affinity List and/or an AMVP Affinity List.

[0443] FIG. 35 is a diagram illustrating a block location for retrieving a legacy HMVP affine candidate, in accordance with an embodiment of the invention.

[0444] Referring to FIG. 35, the CPMV of the current block 3501 may be extracted based on the location and size of an affine HMVP candidate in a manner similar to the method for extracting a common legacy CPMV from adjacent blocks. Those. the encoder/decoder may derive motion vectors of the control points of the current block 3501 based on position and size information (eg, width and height) of the reference block 3502, i. affine HMVP candidate.

[0445] As an option, the CPMV of the current block's legacy affine HMVP can be extracted by using Equations 26 and 27 below.

[0446] [Equation 26]

V0x=VB0x + (VB2_x - VB0x ) * ( posCurCU_Y - posRefCU_Y ) / RefCU_height + (VB1x - VB0x ) * (posCurCU_X - posRefCU_X) / RefCU_width

[0447] [Equation 27]

V0y=VB0y + (VB2_y - VB0y ) * (posCurCU_Y - posRefCU_Y) / RefCU_height + (VB1y - VB0y ) * (posCurCU_X - posRefCU_X) / RefCU_width

[0448] In Equations 26 and 27, posCurCU_Y is the vertical coordinate value of the top left sample of the current block 3501, and posRefCU_Y is the vertical coordinate value of the top left sample of the reference block 3502. posCurCU_X is the horizontal coordinate value of the top left sample of the current block 3501, and posRefCU_X is the value of the horizontal coordinate of the upper left sample of the reference block 3502. RefCU_height is the height of the reference block 3502, and RefCU_width is the width of the reference block 3502.

[0449]

[0450] In one embodiment of the invention, when an HMVP affine candidate (immediate or legacy HMVP) is added, constraints can be added to select an HMVP affine candidate that can be used to generate an affine merge list or an affine AMVP list.

[0451] As an example, an HMVP affine candidate may be added to an affine merge list or an AMVP affine list only when the HMVP affine candidate is adjacent to the current block.

[0452] As another example, an HMVP affine candidate may be added to a merge affine list or an AMVP affine list only when the HMVP affine candidate is located (or exists) within a specific distance from the current block. For example, the specific distance may be a predetermined pixel distance. The encoder/decoder may determine if the HMVP affine candidate is located within a predetermined specific distance to determine if the HMVP affine candidate is available.

[0453] As another example, an HMVP affine candidate may be added to a merge affine list or an AMVP affine list only when the HMVP affine candidate is located (or exists) at a specific location based on the current block. For example, when an HMVP affine candidate is present at a particular location, it may be the case that the HMVP affine candidate is the left or top neighbor block for the current block.

[0454] With respect to an affine HMVP LUT that has N elements, the verification process described above for all elements or the first M elements can be performed until the merge list or AMVP is complete, or until a preliminary a certain number of HMVP candidates.

[0455]

[0456] In one embodiment of the invention, a method is provided in which an HMVP affine candidate is used to replace a legacy affine candidate that is already present in a merge affine list and/or an AMVP affine list.

[0457] FIG. 36 is a diagram illustrating a merge affine list or an AMVP affine list according to an embodiment of the invention.

[0458] Referring to FIG. 36, the encoder/decoder may replace a legacy candidate present in an existing merge affine list or AMVP affine list with a legacy HMVP affine candidate. Those. when the sub-block-based merge mode is applied to the current block, the encoder/decoder may generate a sub-block-based merge candidate list using the legacy affine candidate and the already generated affine candidate, extract the legacy HMVP affine candidate, and replace at least one legacy affine candidate included into the subunit-based merge candidate list inherited by the HMVP affine candidate.

[0459] In addition, in one embodiment of the invention, an affine HMVP lookup table (LUT) may be initialized at the beginning of a slice, CTU row, or CTU. By doing this, the efficiency of parallel processing can be improved.

[0460]

[0461] Further, in the following embodiments, a method for reducing the number of worst clipping checks from HMVP is proposed.

[0462] In an embodiment, when an HMVP candidate is added to the merge list, the number of pruning checks may be determined based on the number of available candidates in the merge list and the number of HMVP candidates that can be added to the merge list. Further, when describing embodiments of the invention, for convenience of description, the variables are defined as follows.

[0463] - NST: number of available (or existing) candidates in the merge list

[0464] - NHMVP: number of HMVP candidates in the table (i.e. size of the HMVP table)

[0465] - NmrgToBeAdded: number of HMVP candidates added to merge list

[0466] - NHMVPChecked: The number of HMVP candidates that are clipped checked.

[0467] - Nmax_hmvp_prunning: number of worst case pruning checks required to add an HMVP candidate to the merge list

[0468] In one embodiment of the invention, the HMVP candidate may be added to the merge list in accordance with the following conditions.

[0469] - First condition: When the LUT is previously clipped (i.e., there are no identical mv among HMVP LUT candidates)

[0470] - Second condition: When the HMVP LUT table size is 6

[0471] Third condition: When the maximum number of available (or existing) merge candidates to add an HMVP candidate to the merge list is 4. That is, this occurs when the number of merge candidates in the merge list is less than the value obtained by subtracting 1 from the maximum merge list size (or maximum number of merge candidates). For example, the maximum merge list size may be 6, and if the number of currently available merge candidates is less than 5, then an HMVP candidate may be added (or inserted). In other words, an HMVP candidate can only be added up to index 5 of the merge list.

[0472] When an HMVP candidate is added to the merge list (ie, becomes a merge candidate), each HMVP candidate may require a pruning check to remove a duplicate among the merge candidates. According to the existing image compression technique, the worst number of clipping checks required to add an HMVP to the merge list can be calculated as shown in Table 8 below.

[0473] [Table 8]

_{NST mrgToBeAdded} N _{-HMVPChecked Nmax_hmvp_prunning} 0 5 6 0 one four 6 four 2 3 6 7 3 2 6 9 four one 6 ten

[0474] Referring to Table 8, in accordance with the existing image compression technique, a clipping check can be performed on six HMVP candidates in the HMVP table (or HMVP list, HMVP candidate list).

[0475] In particular, 1) when there is one candidate in the merge list, 4 HMVP candidates can be added to the merge list. Then, cutoff checks for the six HMVP candidates can be performed. In this case, the number of worst pruning checks can be 4. 2) When there are two candidates in the merge list, 3 HMVP candidates can be added to the merge list. Then, cutoff checks for the six HMVP candidates can be performed. In this case, the number of worst pruning checks can be 7. 3) When there are three candidates in the merge list, 2 HMVP candidates can be added to the merge list. Then, cutoff checks for the six HMVP candidates can be performed. In this case, the number of worst cutoff checks can be 9. 4) When there are four candidates in the merge list, 1 HMVP candidate can be added to the merge list. Then, cutoff checks for the six HMVP candidates can be performed. In this case, the number of worst clipping checks could be 10.

[0476] In an embodiment of the invention, a method is provided for reducing the number of worst clipping checks described above. If there are many merge candidates in the merge list, then since the HMVP encoding result will decrease as merge candidates (i.e., non-HMVP candidates) increase, the number of HMVP candidates to be truncated may need to be reduced. Accordingly, in an embodiment of the invention, the encoder/decoder may set the number of HMVP candidates to be checked (NHMVPChecked) to the number of available HMVP candidates to be added (NmrgToBeAdded) in order to reduce the number of worse clipping checks. In this case, the number of worst clipping tests can be calculated as shown in Table 9 below.

[0477] [Table 9]

_{NST mrgToBeAdded} N _{-HMVPChecked Nmax_hmvp_prunning} 0 5 5 0 one four four four 2 3 3 6 3 2 2 6 four one one four

[0478] Referring to Table 9, the number of worst clipping tests for HMVP can be reduced from 10 to 6 compared to conventional image compression techniques.

[0479] Referring to Table 9, in an embodiment, 1) when there is one candidate in the merge list, 4 HMVP candidates can be added to the merge list. Then, clipping checks for the four HMVP candidates can be performed. In this case, the number of worst pruning checks can be 4. 2) When there are two candidates in the merge list, 3 HMVP candidates can be added to the merge list. Then, cutoff checks for the three HMVP candidates can be performed. In this case, the number of worst pruning checks can be 6. 3) When there are three candidates in the merge list, 2 HMVP candidates can be added to the merge list. Then, clipping checks for the two HMVP candidates can be performed. In this case, the number of worst pruning checks can be 6. 4) When there are four candidates in the merge list, 1 HMVP candidate can be added to the merge list. Then, clipping checks can be performed for one HMVP candidate. In this case, the number of worst clipping checks could be 4.

[0480] In an embodiment of the invention, in order to reduce the number of worst clipping checks, the encoder/decoder may set the number of HMVP candidates against which clipping checks should be performed to be exactly the same as the sum of the number of available HMVP candidates (NmrgToBeAdded ) to be added, and K. Here, K is a predefined constant value. As an example, when K is equal to 1, the worst number of clipping tests can be calculated as shown in Table 10 below.

[0481] [Table 10]

_{NST mrgToBeAdded} N _{-HMVPChecked Nmax_hmvp_prunning} 0 5 6 0 one four 5 four 2 3 four 7 3 2 3 eight four one 2 7

[0482] Referring to Table 10, in an embodiment, 1) when there is one candidate in the merge list, 4 HMVP candidates can be added to the merge list. Then, cutoff checks for the five HMVP candidates can be performed. In this case, the number of worst pruning checks can be 4. 2) When there are two candidates in the merge list, 3 HMVP candidates can be added to the merge list. Then, clipping checks for the four HMVP candidates can be performed. In this case, the number of worst pruning checks can be 7. 3) When there are three candidates in the merge list, 2 HMVP candidates can be added to the merge list. Then, cutoff checks for the three HMVP candidates can be performed. In this case, the number of worst pruning checks can be 8. 4) When there are four candidates in the merge list, 1 HMVP candidate can be added to the merge list. Then, clipping checks can be performed on the two HMVP candidates. In this case, the number of worst clipping checks could be 7.

[0483] In an embodiment of the invention, in order to reduce worst case clipping checks, the number of HMVP candidates to be checked (NHMVPChecked) can be determined as in Equation 28 below.

[0484] [Equation 28]

[0485] In Equation 28, C is a predetermined constant value. If C is 2, then the worst number of clipping tests can be calculated as shown in Table 11 below.

[0486] [Table 11]

_{NST mrgToBeAdded} N _{-HMVPChecked Nmax_hmvp_prunning} 0 5 6 0 one four 6 four 2 3 6 7 3 2 four 9 four one 2 7

[0487] Referring to Table 11, in an embodiment, 1) when there is one candidate in the merge list, 4 HMVP candidates can be added to the merge list. Then, cutoff checks for the six HMVP candidates can be performed. In this case, the number of worst pruning checks can be 4. 2) When there are two candidates in the merge list, 3 HMVP candidates can be added to the merge list. Then, cutoff checks for the six HMVP candidates can be performed. In this case, the number of worst pruning checks can be 7. 3) When there are three candidates in the merge list, 2 HMVP candidates can be added to the merge list. Then, clipping checks for the four HMVP candidates can be performed. In this case, the number of worst cutoff checks can be 9. 4) When there are four candidates in the merge list, 1 HMVP candidate can be added to the merge list. Then, clipping checks for the two HMVP candidates can be performed. In this case, the number of worst clipping checks could be 7.

[0488]

[0489] When an HMVP is added to the merge list and/or AMVP list, pruning is again required to prevent duplication with candidates existing in the pruning list and/or AMVP list. If the HMVP LUT has already been clipped by a limited FIFO operation as described in FIG. 29 above, when inserting (or adding) an HMVP candidate to the merge list, a comparison (or pruning check) between the HMVP candidates may not be required. For this reason, the number of clipping checks can be reduced by using a non-limiting FIFO table, as described in FIG. 28 above. This is because pruning checks between HMVP candidates are necessary when inserting HMVP candidates into the merge list. As described above, in the invention, the HMVP LUT is not limited by its name, and may be referred to as LUT, table, HMVP table, HMVP candidate table, buffer, HMVP buffer, HMVP candidate buffer, HMVP list, HMVP candidate list, HMVP candidate list, list history-based merger candidates, etc.

[0490] In one embodiment of the invention, the HMVP lookup table (LUT) size can be determined in view of the HMVP candidate insertion process for creating a merge list and/or an AMVP list. In particular, HMVP candidates can be added up to a predetermined size of the merge list. For example, when the maximum merge list size is defined as 6, HMVP cannot be the 6th candidate. If 5 merge candidates are available (or present) in the merge list, then the HMVP candidate cannot be added. In this case, the sixth candidate may be selected from among the non-HMVP candidates (or in accordance with another method). Therefore, in view of the HMVP candidate insertion process described above, in one embodiment of the invention, the following HMVP LUT size selection method is provided.

[0491] As an option, the size of the HMVP LUT table can be defined or set to (MaxNumMergeCand-K). Here, MaxNumMergeCand indicates the maximum merge candidate list (or the maximum number of candidates included in the merge candidate list or the maximum number of merge candidates), and at this time, MaxNumMergeCand may be defined as 6. K is a predefined constant. For example, K might be 1 and the HMVP LUT size might be 5.

[0492] In accordance with an embodiment of the invention, by limiting the HMVP to MaxNumMergeCand-K (as described above, for example, K corresponds to 1), the worst number of clipping checks (or worst case) can be reduced by adding the HMVP to the merge list and/or HMV table. (Limited FIFO operation previously described in FIG. 29). In addition, according to an embodiment of the invention, the memory for storing the HMVP LUT can be reduced.

[0493] In an embodiment of the invention, an update process, such as the embodiment described below, may be applied to the HMVP motion candidate table considering the size of the HMVP table described above. The following embodiments are examples of the HMVP table update process, and the embodiments of the invention are not limited to this.

[0494]

[0495] HMVP table update process

[0496] First, the input of the HMVP table update process can be defined as follows.

[0497] - Motion vectors mvL0 and mvL1

[0498] - Reference indices refIdxL0 and refIdxL1

[0499] - Prediction list usage flags predFlagL0 and preFlagL1

[0500] The output of this update process may be a modified HMVP candidate list array. In this process, mvCand is a variable indicating a motion vector candidate with motion vectors, reference indices, and prediction list usage flags.

[0501] This update process can be performed in the following steps.

[0502] 1. The variable identicalCandExist is set to false, and the variable tempIdx is set to 0. Here, identicalCandExist is a variable indicating whether the same motion information is present in the HMVP candidate list, and tempIdx is a variable indicating the index of the HMVP candidate , in the HMVP candidate list, with the same motion information as the current motion vector.

[0503] 2. If HMVPCandNum is greater than 0, then for each HMVPIdx with HMVPIdx=0..HMVPCandNum-1, the following steps may be applied, as long as the variable identicalCandExist is true. Here, HMVPCandNum is the number of HMVP candidates in the HMVP candidate list, and HMVPIdx is an index assigned to HMVP candidates in the HMVP candidate list.

[0504] - If mvCand has the same motion vector and same reference index as HMVPCandList[HMVPIdx] (i.e., an HMVP candidate with HMVPIdx in the HMVP candidate list), then identicalCandExist may be set to true, and tempIdx can be set to HMVPIdx.

[0505] 3. And, the list of HMVP candidates can be updated in accordance with the following steps.

[0506] (1) If identicalCandExist is true or HMVPCandNum is MaxNumMergeCand-K, then the following steps may be applied. Here, MaxNumMergeCand is a variable indicating the size of the merge list (or merge candidate list) (or the maximum number of merge list candidates, the maximum number of merge candidates), and K is an arbitrary constant. In an embodiment, K may be predetermined. Further, in an embodiment, K may be set to 1 and MaxNumMergeCand-K may be 5 when the value of MaxNumMergeCand is determined to be 6. Also, in an embodiment, MaxNumMergeCand-K may be set to 5. For example, if identicalCandExist matches the value true or HMVPCandNum equals 5, then the following steps can be applied.

[0507] - For each index idx with idx = (tempIdx+1) .. (HMVPCandNum-1), HMVPCandList[idx-1] can be set to HMVPCandList[idx]. Those. the HMVP candidate index with the index after tempIdx may be set to a value that is decremented by 1.

[0508] - HMVPCandList [HMVPCandNum-1] can be set to mvCand.

[0509] (2) Otherwise (ie, if identicalCandExist is false and HMVPCandNum is less than MaxNumMergeCand-K), the following steps may be applied. As described above, in an embodiment, MaxNumMergeCand-K may be set to 5. For example, otherwise (ie, identicalCandExist is false and HMVPCandNum is less than 5), the following steps may be applied.

[0510] - HMVPCandList[HMVPCandNum++] can be set by mvCand.

[0511] As an example, this update process may be called when the current slice corresponds to a P or B slice. At this time, the HMVPCandNum variable is set to 0, and the HMVPCandList variable can be defined as an array of elements from MaxNumMergeCand-K. As described above, in the embodiment, MaxNumMergeCand-K may be set to 5. For example, at this time, the HMVPCandNum variable may be set to 0 and the HMVPCandList variable may be defined as an array of elements of 5.

[0512] Next, another example of the HMVP table update process will be described.

[0513] First, the update process input can be determined as follows.

[0514] - Motion vectors mvL0 and mvL1

[0515] - Reference indices refIdxL0 and refIdxL1

[0516] - Prediction list usage flags predFlagL0 and preFlagL1

[0517] The output of this update process may be a modified HMVP candidate list array. In this process, mvCand is a variable indicating a motion vector candidate with motion vectors, reference indices, and prediction list usage flags.

[0518] This update process can be performed in the following steps.

[0519] 1. For each HMVPIdx with HMVPIdx=0..HMVPCandNum-1, the following steps may be applied in order until sameCand is true. Here, sameCand is a variable indicating whether the same traffic information is present in the HMVP candidate list.

[0520] - If mvCand has the same motion vector and the same reference index as HMVPCandList[HMVPIdx], then sameCand is set to true.

[0521] - Otherwise, sameCand is set to false.

[0522] - HMVPIdx++ (i.e. HMVPIdx is incremented by 1)

[0523] 2. The tempIdx variable is set to HMVPCandNum.

[0524] 3. If sameCand is true or HMVPCandNum is MaxNumMergeCand-K, then for each tempIdx index with tempIdx = (sameCand? HMVPIdx: 1)..HMVPCandNum-1, HMVPCandList[tempIdx] is copied or set to HMVPCandList[tempIdx- one]. K is an arbitrary constant. In an embodiment, K may be predetermined. Further, in an embodiment, K may be set to 1 and MaxNumMergeCand-K may be 5 when the value of MaxNumMergeCand is determined to be 6. Also, in an embodiment, MaxNumMergeCand-K may be set to 5.

[0525] 4. mvCand is copied to HMVPCandList[tempIdx].

[0526] 5. If HMVPCandNum is less than MaxNumMergeCand-K, then HMVPCandNum is incremented by one. As described above, in an embodiment, MaxNumMergeCand-K may be set to 5.

[0527] The above embodiments of the invention can be divided and described for the convenience of description, but the invention is not limited to this. Those. the embodiments described above may be performed independently, or one or more of the embodiments may be combined and performed.

[0528] FIG. 37 is a flowchart illustrating a video signal processing method based on history-based motion vector prediction according to an embodiment to which the invention is applied.

[0529] Referring to FIG. 37, the decoder is mainly described for the convenience of description, but the invention is not limited to this. The method for processing a video signal based on history-based motion vector prediction in accordance with an embodiment of the invention may be identically performed in an encoder and a decoder.

[0530] In a video signal processing method based on history-based motion vector prediction, a decoder configures a merge candidate list based on neighboring blocks for the current block (S3701).

[0531] The decoder adds a history-based merge candidate for the current block to the merge candidate list (S3702).

[0532] The decoder obtains an index indicating a merge candidate, which is used for inter-picture prediction of the current block in the merge candidate list (S3703).

[0533] The decoder generates prediction samples of the current block based on the motion information of the merge candidate indicated by the merge index (S3704).

[0534] The decoder updates the history-based merge candidate list based on the motion information (S3705).

[0535] As described above, as an option, a history-based merge candidate can be added to the list of merge candidates if the history-based merge candidate has motion information that does not overlap with that of the predetermined candidate. for the merger, among the candidates for the merger included in the list of candidates for the merger.

[0536] As described above, as an embodiment, the history-based merge candidate list may be determined to have a size that is determined based on the maximum number of merge candidates of the merge candidate list.

[0537] As described above, as an embodiment, the history-based merge candidate list may be determined to have a size as a value obtained by subtracting 1 from the maximum number of merge candidates of the merge candidate list.

[0538] As described above, as an embodiment, the size of the history-based merge candidate list can be defined as 5.

[0539] As described above, as an embodiment, a history-based merge candidate may be added to the list of merge candidates if the history-based merge candidate has motion information that does not overlap with that of a predetermined specific the number of merger candidates, among the merger candidates included in the list of merger candidates.

[0540] As described above, as an option, a history-based merge candidate can be added to the list of merge candidates if the history-based merge candidate has motion information that does not overlap with that of a particular spatial candidate. for the merger included in the list of candidates for the merger.

[0541] As described above, as an embodiment, a history-based merge candidate may be extracted from a predetermined number of candidates in a list of history-based merge candidates.

[0542] As described above, as an option, the decoder may add a motion vector of zero to the merge list when the merge list does not satisfy the maximum amount of the merge list.

[0543] The embodiments described herein may be embodied and executed in a processor, microprocessor, controller, or chip. For example, the functional units illustrated in the drawings may be implemented and executed in a computer, processor, microprocessor, controller, or chip.

[0544] In addition, the decoder and encoder to which the invention is applied can be included in a multimedia broadcast transmission/reception device, a mobile communication terminal, a home theater video device, a digital cinema video device, a security camera, a video dialogue device, a mode communication device. video communication, mobile streaming device, storage media, video camera, video-on-demand (VoD) service device, over-the-network technology (OTT) video device, Internet streaming service device, three-dimensional (3D) ) video, video telephony device and medical video device, and can be used to process video signals and data signals. For example, an OTT video device may include a game console, a Blu-ray player, an Internet access TV, a home theater system, a smartphone, a tablet PC, and a Digital Video Recorder (DVR).

[0545] In addition, the processing method to which the present invention is applied may be created in the form of a computer executable program and may be stored in a computer-readable recording medium. The multimedia data with the data structure according to the invention can also be stored on a computer-readable recording medium. Computer-readable recording media includes all types of storage devices that store computer-readable data. The computer-readable recording medium may include a Blu-ray Disc (BD), Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical storage, for example. In addition, the computer-readable recording medium includes storage media implemented in the form of carriers (eg, transmission over the Internet). In addition, the bitstream generated using the encoding method may be stored in a computer-readable recording medium or may be transmitted via wired and wireless communication networks.

[0546] In addition, an embodiment of the invention may be implemented as a computer program product using program code. The program code may be executed by a computer in accordance with an embodiment of the invention. The program code may be stored on a medium that can be read by a computer.

[0547] A decoding device and an encoding device to which the invention applies may be included in a digital device. The term "digital device" includes all digital devices capable of performing at least one of transmitting, receiving, processing, and outputting, for example, data, content, and services. Here, the processing of data, content, services, etc. by means of a digital device, includes the operation of encoding and/or decoding data, content, service, and the like. The digital device is paired or connected (hereinafter referred to as ‘pairing’) with other digital devices, external servers, etc. via a wired/wireless network to send and receive data, and convert them if necessary.

[0548] The digital device includes, for example, a fixed device (or fixed device) such as network TV, HBBTV (Hybrid Terrestrial Broadband TV), smart TV, IPTV (Internet Protocol Television), PC, etc. and a mobile device (or portable device) such as a PDA (Personal Digital Assistant), smartphone, tablet PC, laptop. In the invention described later, for convenience, FIG. 39 shows and describes digital TV as an embodiment for a digital device, and FIG. 40 shows and describes a mobile device as an embodiment of a digital device.

[0549] Meanwhile, the term "wired/wireless network" described herein refers to a communication network that supports various communication standards or protocols for interconnecting and/or transmitting and receiving data between digital devices or between digital devices and external servers. These wired/wireless networks may include both current and future supported communication networks and communication protocols therefor, and may be formed by a wired communication standard or protocols such as USB (Universal Serial Bus), CVBS (Composite Video Signal), Component, S-Video (Analog), DVI (Digital Video Interface), HDMI (High Definition Multimedia Interface), RGB, D-SUB and the like, and are formed by communication standards for wireless connection such as Bluetooth, Radio Frequency Identification ( RFID), Infrared Data Communications Association (IrDA), Ultra Wideband (UWB), ZigBee, Digital Home Network Alliance (DLNA), Wireless LAN (WLAN) (Wi-Fi), Wibro (Wireless Broadband), WiMAX (Worldwide Interoperable Broadband Wireless Access), HSDPA (High Speed Downlink Packet Access), LTE (Long Term Evolution), Wi-Fi Direc t (Direct), etc.

[0550] Further, in the case of simply referring to a digital device in the invention, this may mean a fixed device or a mobile device, or include both types depending on the context.

[0551] Meanwhile, the digital device is, for example, an intelligent device that supports a broadcast reception function, a computer function, and at least one external input, and can support e-mail, web browsing, banking, a game, and an application via the wired/wireless network described above. In addition, the digital device may include an interface for supporting at least one input or control means (hereinafter referred to as input means) such as a manual input device, a touch screen, and a remote control. The digital device may use a generic general purpose operating system (OS). For example, a digital device can add, remove, modify, and update various applications in a general purpose OS kernel, and can configure and provide a more user-friendly environment through them.

[0552] Meanwhile, the external input described in the invention includes external input devices, i. e. all input means or digital devices that are wired/wireless connected to the digital devices described above and transmit/receive associated data through it. Here, the external input may include all devices such as a high-definition multimedia interface (HDMI), a game device such as a Play Station or X-Box, a smartphone, a tablet PC, a printer, or a smart TV.

[0553] And the term "server" described in the invention includes a client, i. all digital devices or systems that feed data to the digital devices described above is called a processor. Examples of such a server include a portal server that provides a web page or web content, an advertising server that provides advertising data, a content server that provides content, and a social networking service (SNS) server, a service server, or a manufacturer's server provided by manufacturer, etc.

[0554] Also, "channel" as described herein means a path, means, etc. for transmitting and receiving data, and may be exemplified by a broadcast channel. Here, the broadcast channel is expressed in terms of a physical channel, a virtual channel, and a logical channel according to digital broadcast activation. The broadcast channel may be referred to as a broadcast network. As described above, a broadcast channel refers to a channel for providing or accessing broadcast content provided by a broadcast station, and the broadcast content is mainly based on real-time broadcasting and is also called a live channel. However, recently, the environment for broadcasting has become more versatile, and non-real-time broadcasting is also activated in addition to real-time broadcasting. It can also be understood as a concept denoting the entire channel. Thus, the live channel can be understood as a concept meaning not only a real-time broadcast, but also an entire broadcast channel, including a non-real-time broadcast in some cases.

[0555] In the invention, "random channel" is further defined in relation to a channel other than the broadcast channel described above. An arbitrary channel may provide a service guide such as an electronic program guide (EPG) along with a broadcast channel. Service guide, GUI (Graphical User Interface) or OSD (On Screen Display) screen can be configured/provided with only an arbitrary channel.

[0556] Meanwhile, unlike a broadcast channel with a predetermined channel number between transceivers, a random channel is a channel that is randomly allocated in a receiver, and a channel number that does not mostly overlap with channel numbers for expressing broadcast channels is allocated to a random channel. For example, when a specific broadcast channel is configured, the receiver receives the broadcast signal and transmits the broadcast content and its signaling information via the configured channel. Here, the receiver parses the channel information from the signaling information and configures the channel explorer, EPG and the like based on the parsed channel information and provides it to the user. When the user makes a channel change request via the input means, the receiver responds accordingly.

[0557] As described above, since the broadcast channel is prearranged between the transmitting and receiving terminals, when a random channel is allocated for an overlapped broadcast channel, it may cause user confusion or confusion may occur, so it is preferable not to allocate duplicates as described above. . Meanwhile, even if the random channel number does not overlap with the broadcast channel number as described above, there is still confusion in the user's channel browsing process, and it is necessary to allocate the random channel number accordingly. This is because an arbitrary channel according to the invention can also be implemented to be accessed in response to a user's request to switch the channel via the input means in the same manner as a normal broadcast channel. Accordingly, the arbitrary channel number can be determined and displayed in a form in which the characters are written in parallel, such as arbitrary channel-1, arbitrary channel-2 and the like, instead of the numerical form similar to the broadcast channel, for the convenience of user access to the arbitrary channel and for convenience of distinguishing or distinguishing from the broadcast channel number. In this case, although the arbitrary channel number display can be implemented in the form in which symbols are written as in arbitrary channel-1, the receiver can recognize and implement the arbitrary channel in numerical form as the broadcast channel number internally. Also, an arbitrary channel number may be provided in the form of a number, such as a broadcast channel, and it is also possible to determine and display channel numbers in various ways that may differ from broadcast channels, such as video channel-1, title-1, and video-1.

[0558] The digital device executes a web browser for a web service and provides various types of web pages to a user. Here, the web page includes a web page including video content, and in the invention, the video is processed separately or independently from the web page. And, the separated video may be allocated to an arbitrary channel as described above, and provided by a service guide or the like, and output according to a channel switch request in a service guide or broadcast channel browsing process. In addition, for services such as broadcast content, games and applications, in addition to web services, predefined content, images, audio, elements, etc. are processed separately from broadcast content, games, and applications themselves, and an arbitrary channel number may be assigned to their playback, processing, and the like, and may be implemented as described above.

[0559] FIG. 38 is a diagram schematically showing an example of a service system including a digital device.

[0560] Service systems including digital devices include CP content provider 3810, SP service provider 3820, NP network operator 3830, and HNED home network end user 1840 (Consumer). Here, HNED 3840 is, for example, client 3800, i. e. digital device. The content provider 3810 creates and provides various content. As shown in FIG. 38, such content provider 3810 may be exemplified by a terrestrial broadcaster, cable SO (System Operator) or MSO (Multiple SO), satellite broadcaster, various Internet broadcasters, Private CPs, and the like. Meanwhile, the content provider 3810 provides various applications in addition to the broadcast content.

[0561] The service provider 3820 provides the content provided by the content provider 3810 as the HNED service package 3840 by packaging the content. For example, service provider 3820 in FIG. 38 packages the first terrestrial broadcast, the second terrestrial broadcast, cable MSO, satellite broadcast, various Internet broadcasts, applications, and so on. and provides them to HNED 3840.

[0562] The service provider 3820 provides services to the client 3800 in a unicast or multicast manner. And, the service provider 3820 can transmit data to multiple pre-registered clients 3800 at the same time, and a protocol called IGMP Internet Group Administration Protocol can be used for this.

[0563] The content provider 3810 described above and the service provider 3820 may be the same or the same entity. For example, content that is provided by the content provider 3810 may be packaged into a service and provided to the HNED 3840 such that the functions of the service provider 3820 are performed together or vice versa.

[0564] The network operator 3830 provides a network for data exchange between the content provider 3810 and/or the service provider 3820 and the client 3800.

[0565] The client 3800 may create a home network for transmitting and receiving data.

[0566] Meanwhile, the content provider 3810 and/or service provider 3820 in the service system may use conditional access or a content protector to protect transmitted content. In this case, the client 3800 may use a processing tool such as CableCARD (POD: Deployment Point), DCAS (Downloadable CAS), etc. in response to content restriction or protection.

[0567] In addition, the client 3800 may also use a bi-directional service over a network (or communication network). In this case, the client 3800 may act as a content provider to some extent, and the existing service provider 3820 may receive it and pass it back to another client.

[0568] FIG. 39 is a block diagram illustrating a digital apparatus according to an embodiment. Here, Fig. 39, for example, may correspond to client 3800 in FIG. 38 and refer to the digital device described above.

[0569] The digital device 3900 includes a network interface 3901, a TCP/IP manager 3902, a service delivery manager 3903, an SI decoder 3904, a demultiplexer 3905 (demux), an audio decoder 3906, a video decoder 3907, a display module 3908 (display A /V and OSD), service control manager 3909, service discovery manager 3910, SI and metadata database 3911, metadata manager 3912, service manager 3913, UI manager 3914, and the like.

[0570] The network interface 3901 receives or transmits IP packets through the network. Those. the network interface 3901 receives services, content, and the like from a service provider 3820 via the network.

[0571] The TCP/IP manager 3902 participates in the transmission of packets between source and destination of IP packets received by the digital device 3900 and IP packets transmitted by the digital device 3900. And the TCP/IP manager 3902 classifies the received packets according to the appropriate protocol. and outputs the classified packets to the service delivery manager 3905, the service discovery manager 3910, the service control manager 3909, the metadata manager 3912, and so on. The service delivery manager 3903 is responsible for managing the received service data. For example, the service delivery manager 3903 may use RTP/RTCP when managing real-time streaming data. When real-time streaming data is transmitted using RTP, the service delivery manager 3903 parses the received data packet according to the RTP and transmits it to the demultiplexer 3905 or stores it in the SI and metadata database 3911 under the control of the service manager 3913. And, the service delivery manager 3903 uses RTCP to feed back network reception information to the server that provides the service. The demultiplexer 3905 demultiplexes the received packets into audio, video, and SI system information data, and passes them to the audio/video decoder 3906/3907 and the SI decoder 3904, respectively.

[0572] The SI decoder 3904 decodes system information such as program specific information PSI, program and system information protocol PSIP, and DVB-SI digital video broadcast service information.

[0573] And the SI decoder 3904 stores the decoded service information in the SI database 3911 and metadata, for example. The service information that is stored in this way can be read and used by an appropriate configuration, such as a user request.

[0574] The audio/video decoder 3906/3907 decodes each audio and video data demultiplexed by the demultiplexer 3905. Thus, the decoded audio and video data are provided to the user by the display unit 3908.

[0575] The application manager may include, for example, a UI manager 3914 and a service manager 3913. The implementation application administration tool administers the overall status of the 3900 digital device, provides a user interface, and administers other administration tools.

[0576] The UI manager 3914 provides a GUI to a user using an OSD or the like, and receives a key input from the user to perform a device operation according to the input. For example, when the UI manager 3914 receives a key input regarding channel selection from the user, the UI manager 3914 sends a key input signal to the service manager 3913.

[0577] The service manager 3913 manages the management associated with the service, such as the service delivery manager 3903, the service discovery manager 3910, the service control manager 3909, and the metadata manager 3912.

[0578] And the service manager 3913 creates a channel map and selects a channel using the channel map according to the key input received from the user interface manager 3914. And, the service manager 3913 receives the channel service information from the SI decoder 3904, and sets the selected channel audio/video packet identifier PID in the demultiplexer 3905. The PID set in this way is used in the demultiplexing process described above. Because of this, the demultiplexer 3905 filters the audio data, video data, and SI data using the PID.

[0579] The service discoverer 3910 provides the information necessary to select a service provider providing a service. When the channel selection signal is received from the service manager 3913, the service discovery manager 3910 locates the service using the information.

[0580] The service control manager 3909 is responsible for selecting and managing services. For example, the service control manager 3909 uses IGMP or RTSP or the like when a user selects a live broadcast service such as a conventional broadcast method, and uses RTSP to select and control the service when selecting a service such as VOD (video over request). The RTSP protocol may provide a mode of quick access to media objects in relation to real-time streaming. And, the service control manager 3909 can initiate and manage a session through the IMS gateway 3950 using the IMS IP multimedia subsystem and the SIP session initiation protocol. The protocol is one embodiment, and other protocols may be used in accordance with the implementation examples.

[0581] The metadata manager 3912 manages the metadata associated with the service and stores the metadata in the SI and metadata database 3911.

[0582] The SI and metadata database 3911 stores information about the service decoded by the SI decoder 3904, metadata managed by the metadata manager 3912, and information necessary for selecting a service provider, which is provided by the service discovery manager 3910. And the SI data and metadata base 3911 can store customization data and the like for the system.

[0583] The SI and metadata database 3911 may be implemented using NVRAM non-volatile RAM, flash memory, or the like.

[0584] Meanwhile, the IMS gateway 3950 is a gateway that assembles the functions necessary to access the IMS-based IPTV service.

[0585] FIG. 40 is a configuration block diagram illustrating another embodiment of a digital device. In particular, Fig. 40 illustrates a block diagram of a mobile device as another embodiment of a digital device.

[0586] Referring to FIG. 40, the mobile device 4000 may include a wireless communication unit 4010, an A/V audio/video input unit 4020, a user input unit 4030, a touch unit 4040, an output unit 4050, a memory 4060, an interface unit 4070, a control unit 4080, and a supply unit 4090 nutrition. The components shown in Fig. 40 are not integral, so that a mobile device may be implemented with more or fewer components.

[0587] The wireless communication unit 4010 may include one or more modules that provide wireless communication between the mobile device 4000 and the wireless communication system, or between the mobile device and the network in which the mobile device resides. For example, the wireless communication unit 4010 may include a broadcast receiving module 4011, a mobile communication module 4012, a wireless Internet module 4013, a short range communication module 4014, and a location information module 4015.

[0588] The broadcast receiving module 4011 receives a broadcast signal and/or broadcast-related information from an external broadcast administration server via a broadcast channel. Here, the broadcast channel may include a satellite channel and a terrestrial channel. The broadcast administration server may mean a server that generates and transmits broadcast signals and/or broadcast-related information, or a server that receives previously generated broadcast signals and/or broadcast-related information and transmits them to the terminal. The broadcast signal may include a TV broadcast signal, a radio broadcast signal, and a broadcast data signal, and may also include a combined signal in which the broadcast data signal is combined with a TV broadcast signal or a radio broadcast signal.

[0589] Broadcast related information may mean information related to a broadcast channel, broadcast program, or broadcast service provider. Broadcast-related information may also be provided via a mobile communication network. In this case, it can be received by the mobile communication module 4012.

[0590] Broadcast-related information may be present in various forms, such as an electronic program guide or an electronic ESG service guide.

[0591] The broadcast receiving module 4011 can receive a digital broadcast signal using a digital broadcast system such as, for example, ATSC, DVB-T terrestrial digital video broadcast, DVB-S satellite, MediaFLO Multimedia Only Direct Link, DVB-H portable, terrestrial broadcasting with integrated ISDB-T digital services and the like. Of course, the broadcast receiving module 4011 can be configured to be suitable for other broadcast systems, as well as the digital broadcast system described above.

[0592] The broadcast signal and/or broadcast-related information received via the broadcast information receiving module 4011 may be stored in the memory 4060.

[0593] The mobile communication module 4012 transmits and receives a radio signal to and from at least one of a base station, an external terminal, and a server in a mobile communication network. The wireless signal may include various types of data according to the transmission and reception of a voice signal, a video call signal, or a text/multimedia message.

[0594] The wireless Internet module 4013 includes a module for wireless access to the Internet, and may be built-in or external to the mobile device 4000. Wireless LAN (WLAN) (Wi-Fi), wireless broadband Wibro connectivity, worldwide WiMAX broadband wireless access compatibility, and HSDPA high-speed downlink packet access.

[0595] Short range communication module 4014 refers to a module for short range communication. Short range Bluetooth, RFID (Radio Frequency Identification), Infrared (IrDA), UWB (Ultra Wideband), ZigBee, RS-232, RS-485, etc. can be used.

[0596] The location information module 4015 is a module for obtaining location information of the mobile device 4000, and may use the global positioning system module as an example.

[0597] The A/V input unit 4020 is used to input an audio signal and/or a video signal, and may include a camera 4021, a microphone 4022, and the like. The camera 4021 processes image frames such as still images or moving pictures acquired by the image sensor in the video call mode or shooting mode. The processed image frame may be displayed in a display unit 4051 .

[0598] The image frames processed by the camera 4021 may be stored in the memory 4060 or transmitted outside via the wireless communication unit 4010. Two or more 4021 cameras may be provided depending on the usage environment.

[0599] The microphone 4022 receives an external audio signal through the microphone in call mode or recording mode, voice recognition mode, etc., and processes it as electrical voice data. The processed voice data can be converted and output in a form that can be transmitted to the mobile base station via the mobile communication module 4012 in a call mode. The microphone 4022 can be implemented with various noise reduction algorithms to remove noise that is generated during reception of an external audio signal.

[0600] The user input block 4030 generates input data for the user to control the operation of the terminal. The user input unit 4030 may be a keypad, dome switch, touch pad (pressure/capacitive type), jog wheel, jet switch, or the like.

[0601] The sensor unit 4040 displays the current state of the mobile device 4000, such as the open/closed state of the mobile device 4000, the location of the mobile device 4000, the presence or absence of contact with the user, the orientation of the mobile device, and the acceleration/deceleration of the mobile device. It registers and generates a touch signal to control the operation of the mobile device 4000. For example, when the mobile device 4000 moves or tilts, the position or tilt of the mobile device may be detected. And it can be registered whether power is supplied to the power supply unit 4090 or whether external devices are connected to the interface unit 4070. Meanwhile, the sensor unit 4040 may include a proximity sensor 4041 including NFC near field communication.

[0602] An output unit 4050 is used to generate an output related to vision, hearing, or tactile sensation, and may include a display unit 4051, an audio output unit 4052, an alert unit 4053, and a tactile unit 4054.

[0603] The display unit 4051 displays or outputs information processed by the mobile device 4000. For example, when the mobile device is in a call, a UI or a GUI associated with the call is displayed. When the mobile device 4000 is in video calling mode or shooting mode, the captured and/or received video, UI or GUI is displayed.

[0604] The display unit 4051 may include an LCD, a TFT LCD, an OLED, and a flexible display, and a 3D display.

[0605] Some of these displays may be of a transparent type or a translucent type so that what is outside can be seen through them. This may be referred to as a transparent display and a typical example of a transparent display is a transparent OLED TOLED. The rear structure of the display unit 4051 may also be configured as a light-transmitting structure. With this structure, the user can see objects behind the terminal body through the area occupied by the display unit 4051 of the terminal body.

[0606] Two or more display units 4051 may be present depending on the implementation form of the mobile device 4000. For example, a plurality of display units may be located at a distance from one surface or located as an integral part in the mobile device 4000, or may be respectively located on different surfaces.

[0607] When the display unit 4051 and the touch operation detecting sensor (hereinafter referred to as 'touch sensor') form a mutual layer structure (hereinafter referred to as 'touch screen'), the display unit 4051 can be used as an input device in addition to the output device. The touch screen may be in the form of, for example, a touch film, a touch sheet, or a touch pad.

[0608] The touch sensor may be configured to convert a change in pressure that is applied to a particular area of the display unit 4051, or a capacitance that is generated in a particular area of the display unit 4051, into an electrical input signal. The touch sensor may be configured to detect not only the position and zone where the touch is made, but also the pressure at the time of the touch.

[0609] If a touch input is made through the touch sensor, appropriate signals are sent to the touch controller. The touch controller processes the signals and then transmits the corresponding data to the controller 4080. Accordingly, the control unit 4080 can know which area of the display unit 4051 has been touched and the like.

[0610] The proximity sensor 4041 may be located in the inner area of the mobile device, surrounded by the touch screen or near the touch screen. A proximity sensor refers to a sensor that detects the presence or absence of an object that is approaching a predetermined detection surface, or an object in the vicinity, using mechanical force or infrared light without mechanical contact. Proximity sensors have a longer life and higher utilization than contact sensors.

[0611] Examples of the proximity sensor include a through-type photoelectric sensor, a directional reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high-frequency oscillation type proximity sensor, a capacitive-type proximity sensor, a magnetic-type proximity sensor, and an infrared proximity sensor. When the touch screen is capacitive, it is capable of detecting the approach of a pointer due to a change in the electric field in accordance with the approach of the pointer. In this case, the touch screen (touch sensor) can be classified as a proximity sensor.

[0612] Hereinafter, for convenience of description, the action that causes recognition of the position of the pointer on the touch screen without the pointer touching the touch screen is referred to as "touch by proximity", and the action of actually touching the touch screen pointer is referred to as "touch by contact". . The location on the touch screen that is touched by pointer zoom means the location where the pointer is aligned vertically with the touch screen when the pointer is closely touched.

[0613] The proximity sensor detects a touch-by-proximity and a touch-by-proximity sample (e.g., touch-by-proximity distance, touch-by-proximity direction, touch-by-proximity speed, touch-by-proximity time, touch-by-proximity position, touch-by-proximity movement state, etc.). e.) Information corresponding to the registered zoom touch operation and zoom touch sample can be displayed on the touch screen.

[0614] The audio output module 4052 can output audio data received from the wireless communication unit 4010 in a call mode, call signal reception or recording mode, voice recognition mode, broadcast reception mode or the like, or audio data that is stored in the memory 4060. Output module 4052 audio may also output audio signals related to functions (eg, call receive sound, message receive sound, etc.) that are performed in the mobile device 4000. The audio output module 4052 may include a receiver, a speaker, and a buzzer.

[0615] The notification module 4053 outputs a signal to notify the occurrence of an event in the mobile device 4000. Examples of events that are generated in the mobile device include call signal reception, message reception, key input signal, and touch input signal. The alert unit 4053 may output a signal for notifying the occurrence of an event through other forms, such as vibration, in addition to a video signal or an audio signal.

[0616] The video signal or audio signal can also be output through the display unit 4051 or the audio output module 4052, so that the display unit 4051 and the audio output module 4052 can be classified as part of the notification module 4053.

[0617] The haptic module 4054 generates various tactile effects that can be felt by the user. Vibration is a typical example of a haptic effect that is generated by the 4054 haptic module. The intensity and pattern of the vibration that is generated by the 4054 haptic module is controllable. For example, different vibrations can be synthesized and output or output sequentially.

[0618] In addition to vibration, the tactile module can generate various tactile effects, including irritation effects, such as an arrangement of pins that move vertically with respect to the skin contact surface, force of expiration or suction force of air through expiration or suction, contact with the skin surface, contact with electrodes, electrostatic force and the like, and the effect of reproducing a feeling of cold or heat using an element capable of absorbing heat or generating heat.

[0619] The tactile module 4054 can not only deliver a tactile effect through direct contact, but also implement it so that the user can feel the tactile effect through muscular senses, such as fingers or hands. Two or more tactile modules 4054 may be provided in accordance with an aspect of mobile device 4000 configuration.

[0620] The memory 4060 may store a program for operating the control unit 4080, and may temporarily store input/output data (eg, phone book, message, still image, video, etc.). The memory 4060 can also store data associated with various vibration and sound samples that are output when a touch is input through the touch screen.

[0621] The memory 4060 may include a storage medium of at least one of a flash memory type, a hard disk type, a micro multimedia card type, a card type memory (e.g., SD or XD memory, etc.), memory random-access RAM, static random-access memory SRAM, read-only memory ROM, electrically erasable programmable read-only memory EEPROM, programmable read-only memory PROM, magnetic memory, optical disk. The mobile device 4000 may operate in connection with a web storage that performs the function of storing the memory 4060 on the Internet.

[0622] The interface unit 4070 serves as a gateway for all external devices connected to the mobile device 4000. The interface unit 4070 receives data from an external device, receives power, and transmits data to each component within the mobile device 4000, or provides data communication, within the mobile device 4000 to an external device. For example, a wired/wireless headphone port, an external charger port, a wired/wireless data port, a memory card port, a port for connecting with devices equipped with an ID module, an audio input/output port, a video I/O port, a headset port, and the same may be included in the 4070 interface block.

[0623] An identity module is a chip that stores various information for authenticating rights to use the mobile device 4000, and may include a user identity module UIM, a subscriber identity module SIM, a universal subscriber identity module USIM, and the like. A device equipped with an identification module (hereinafter referred to as an 'identification device') may be manufactured in a smart card format. Because of this, the identification device can be connected to the terminal 4000 via a port.

[0624] When the mobile terminal 4000 connects to an external pole, the interface unit 4070 may become a passage through which power from the pole is supplied to the mobile terminal 4000, or a passage through which various command signals are input from the pole by a user. Various command signals or power input from the pedestal may be processed as a signal to recognize that the mobile terminal is properly mounted on the pedestal.

[0625] The control unit 4080 typically controls the entire operation of the mobile device. For example, it manages and processes related to voice calls, data communications, video calls, and the like. The control unit 4080 may include a multimedia module 4081 for playing multimedia. The multimedia module 4081 may be implemented in the control unit 4080 or may be implemented separately from the control unit 4080. The control unit 4080, in particular the multimedia module 4081, may include an encoder 100 and/or a decoder 200 as described above.

[0626] The control unit 4080 may perform a sample recognition process capable of recognizing handwriting input or picture drawing input that is performed on the touch screen as symbols and images, respectively.

[0627] The power supply unit 4090 receives external power and internal power under the control of the control unit 4080, and supplies the power required to operate each component.

[0628] The various embodiments described herein may be implemented on a computer or similar machine-readable recording medium using, for example, software, firmware, or a combination thereof.

[0629] In accordance with hardware implementation, the embodiments described herein may be implemented using at least one of domain-specific ASICs, DSPs, digital signal processors DSPDs, programmable logic PLDs, field programmable gate arrays FPGA, processors, controllers, microcontrollers, microprocessors and electrical blocks to perform functions. In some cases, the embodiments described herein may be implemented by the control unit 4080 itself.

[0630] According to software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules can perform one or more of the functions and operations described in this document. The software code may be implemented in a software application written in a suitable programming language. Here, the software code is stored in the memory 4060 and can be executed by the control unit 4080.

[0631] FIG. 41 is a block diagram illustrating a digital apparatus according to another embodiment.

[0632] Another example of the digital device 4100 may include a broadcast receiving unit 4105, an external device interface unit 4156, a storage unit 4140, a user input interface unit 4150, a control unit 4170, a display unit 4180, an audio output unit 4185, a supply unit 4190 power supply and photographic unit (not shown). Here, the broadcast receiving unit 4105 may include at least one tuner 4110, a demodulation unit 4120, and a network interface unit 4130. However, depending on the case, the broadcast receiving unit 4105 may include a tuner 4110 and a demodulator 4120, but may not include a network interface unit 4130, and vice versa. Also, although not shown, the broadcast receiving unit 4105 may be equipped with a multiplexer to multiplex the signal tuned to by the tuner 4110 and then demodulated by the demodulation unit 4120 and the signal received via the network interface unit 4130. Also, although not shown, the broadcast receiving unit 4105 may be equipped with a demultiplexer to demultiplex a multiplexed signal, a demodulated signal, or a signal that has passed through the network interface unit 4130.

[0633] The tuner 4110 receives the RF broadcast signal by tuning to a channel selected by the user or all of the previously stored channels among the RF broadcast signals received via the antenna. And the tuner 4110 converts the received RF broadcast signal into an IF signal or a baseband signal.

[0634] For example, if the received RF broadcast signal is a digital broadcast signal, then it is converted to a DIF digital IF signal, and if it is an analog broadcast signal, then it is converted to an analog CVBS/SIF baseband video or audio signal. Those. the 4110 tuner can process both digital broadcast signals and analog broadcast signals. The analog video or CVBS/SIF baseband audio signal that is output from the 4110 tuner can be directly input to the 4170 controller.

[0635] The tuner 4110 may receive a single carrier wideband RF signal in accordance with the ATSC scheme or a multi-carrier RF broadcast signal in accordance with the DVB digital video broadcasting scheme.

[0636] Meanwhile, the tuner 4110 can sequentially tune in and receive the RF broadcast signals of all the broadcast signals stored by the channel storage function among the RF broadcast signals received by the antenna, and convert them into an intermediate frequency signal or a baseband signal.

[0637] The demodulation block 4120 receives and demodulates the digital IF signal DIF converted by the tuner 4110. For example, when the digital IF signal that is output from the tuner 4110 corresponds to the ATSC system, the demodulation block 4120 performs 8-partially suppressed sideband demodulation 8- VSB for example. Also, demodulation block 4120 may perform channel decoding. To this end, the demodulation section 4120 may include a trellis decoder, a de-interleaver, a Reed-Solomon decoder, and the like, and perform trellis decoding, de-interleaving, and Reed-Solomon decoding.

[0638] For example, when the digital IF signal that is output from the tuner 4110 corresponds to the DVB method, then the demodulation section 4120 performs COFDMA coded modulation demodulations, for example. In addition, demodulation block 4120 may perform channel decoding. To this end, the demodulation section 4120 may include a convolutional decoder, a deinterleaver, and a Reed-Solomon decoder, and perform convolutional decoding, deinterleaving, and Reed-Solomon decoding.

[0639] The demodulation unit 4120 may output a streaming TS signal after demodulation and channel decoding. Here, the streaming signal may be a signal in which video, audio, and data are multiplexed. For example, the streaming signal may be an MPEG-2 TS in which MPEG-2 video, Dolby AC-3 audio, and the like are multiplexed. In particular, MPEG-2 TS may include a header of 4 bytes and a payload of 184 bytes.

[0640] Meanwhile, the above-described demodulation block 4120 may be provided separately according to the ATSC method and the DVB method. Those. the digital apparatus may separately include an ATSC demodulator and a DVB demodulator.

[0641] The streaming signal that is output from the demodulator 4120 may be input to the controller 4170. The control unit 4170 may control demultiplexing, video/audio signal processing and the like, and control the image that is output through the display unit 4180 and the audio that is output through block 4185 audio output.

[0642] The external device interface unit 4156 provides an environment in which various external devices communicate with the digital device 4100. To this end, the external device interface unit 4135 may include an A/V input/output unit (not shown) or a wireless communication unit (not shown).

[0643] The external device interface unit 4156 can be wired/wirelessly connected to external devices such as digital versatile disc (DVD), Blu-ray, gaming devices, cameras, camcorders, computers (laptops, tablets), smartphones, Bluetooth devices and cloud. The external device interface unit 4156 transmits a video signal, an audio signal, or a data signal (including an image) that is input from the outside, by means of a connected external device, to the digital device controller 4170. The control unit 4170 may control the processed video signal, audio signal, or data signal to be output to the connected external device. To this end, the external device interface unit 4156 may further include an A/V input/output unit (not shown) or a wireless communication unit (not shown).

[0644] The A/V input/output unit may include a USB connector, a CVBS composite video connector, a component connector, an S-video (analog) connector, and a DVI (Digital Video Interface) connector, an HDMI (high-definition multimedia interface) connector, a RGB, D-SUB connector, and the like, to input video and audio signals from an external device to the 4100 digital device.

[0645] The wireless communication unit may perform short-range wireless communications with other electronic devices. The digital device 4100 can be networked with other electronic devices in accordance with a communication protocol such as Bluetooth, RFID, IrDA, UWB, ZigBee, DLNA, etc. .d.

[0646] And, the external device interface unit 4156 may be connected to at least one of the various set-top boxes and various connectors described above and perform I/O operations with the set-top box.

[0647] Meanwhile, the external device interface unit 4156 may receive an application or application list in an adjacent external device and transmit it to the control unit 4170 or the storage unit 4140.

[0648] The network interface unit 4130 provides an interface for connecting the digital device 4100 to a wired/wireless network, including the Internet. The network interface unit 4130 may include, for example, an Ethernet connector or the like for connecting to a wired network, and use communication standards such as, for example, wireless LAN (WLAN) (Wi-Fi), wireless broadband (Wibro), worldwide broadband wireless access (WiMAX) and High Speed Downlink Packet Access (HSDPA) compatibility for connecting to a wireless network.

[0649] The network interface unit 4130 can transmit or receive data with other users or other digital devices via a connected network or another network connected to a connected network. In particular, some content data that is stored in the digital device 4100 may be transferred to another user registered in advance in the digital device 4100, or to a selected user or a selected digital device among other digital devices.

[0650] Meanwhile, the network interface unit 4130 can access a predetermined web page via a connected network or another network connected to a connected network. Those. it is possible to connect to a predetermined web page via the network and transmit or receive data with the corresponding server. In addition, content or data provided by a content provider or network operator may be received. Those. it is possible to receive content such as a movie, advertisement, game, VOD, broadcast signal, and related information that is provided by a content provider or a network operator via the network. In addition, you can receive update information and update firmware files provided by the network operator. The network interface unit 4130 may also send data to the Internet or to a content provider or network operator.

[0651] Also, the network interface unit 4130 can select and receive the desired application from applications that are open to the public via the network.

[0652] The storage unit 4140 may store a program for processing and managing each signal in the controller 4170, or may store an image after signal processing, an audio signal, or a data signal.

[0653] Also, the storage unit 4140 may perform a function for temporarily storing an image, audio, or data signal that is input from an external device interface unit 4135 or a network interface unit 4130. The storage unit 4140 may store information associated with a predetermined broadcast channel through a channel memory function.

[0654] The storage unit 4140 may store an application or a list of applications that are input from an external device interface unit 4135 or a network interface unit 4130.

[0655] Also, the storage unit 4140 can store various platforms, which are described later.

[0656] The storage unit 4140 may include at least one storage medium among, for example, a flash memory type, a hard disk type, a multimedia micro card type, and a memory card type (for example, SD or XD memory, etc. .), RAM and ROM (EEPROM, etc.). The digital device 4100 can play back and provide content files (video files, still image files, music files, document files, application files, etc.) stored in the storage unit 4140 to a user.

[0657] FIG. 41 illustrates an embodiment in which the storage unit 4140 is provided separately from the control unit 4170, but the scope of the invention is not limited to this. Those. storage unit 4140 may be included in control unit 4170.

[0658] The user input interface unit 4150 transmits a signal that is input by the user to the control unit 4170, or a signal from the control unit 4170 to the user.

[0659] For example, the user input interface unit 4150 may receive a control signal associated with power on/off controls, channel selection, and screen settings from the remote control device 4200 in accordance with various communication methods such as an RF communication method and an infrared IR method. communication, and process it. Or, the user input interface unit 4150 may process the control signal from the control unit 4170 to be transmitted to the remote control unit 4200.

[0660] And, the user input interface unit 4150 can transmit a control signal that is input via a local key (not shown) such as a power key, a channel key, a volume key, and a value set key to the controller 4170.

[0661] The user input interface unit 4150 may transmit a control signal that is input from a registration unit (not shown) that registers a user gesture to a control unit 4170, or transmit a signal from the control unit 4170 to a registration unit (not shown). Here, the registration unit (not shown) may include a touch sensor, a voice sensor, a position sensor, and a motion sensor.

[0662] The controller 4170 may demultiplex the stream that is input through the tuner 4110, demodulator 4120, or external device interface 4156, or process the demultiplexed signals to generate and output a signal for video or audio output. The control unit 4170 may include an encoder and/or a decoder as described above.

[0663] The image signal processed by the controller 4170 may be input to the display unit 4180 and displayed as an image corresponding to the image signal. Also, the image signal processed by the control unit 4170 may be input to an external output device via the external device interface unit 4156.

[0664] The audio signal processed by the control unit 4170 may be audio, which is output to the audio output unit 4185. Also, the audio signal processed by the controller 4170 may be input to an external output device via the external device interface block 4156.

[0665] Although not illustrated in FIG. 41, the control unit 4170 may include a demultiplexer, an image processor, and the like.

[0666] The control unit 4170 may control the entire operation of the digital device 4100. For example, the control unit 4170 may control the tuner 4110 to tune into an RF broadcast corresponding to a user-selected channel or a previously stored channel.

[0667] The control unit 4170 may control the digital device 4100 through a user command that is input through the user input interface unit 4150 or an internal program. In particular, it is possible to access the network and download the desired application or application list to the digital device 4100.

[0668] For example, the control unit 4170 controls the tuner 4110 to output the signal of the channel selected according to the predetermined channel selection command received through the user input interface unit 4150, and processes the video, audio or data signals of the selected channel. The control unit 4170 outputs the channel information, etc., selected by the user, through the display unit 4180 or the audio output unit 4185 along with the processed image signal or audio signal.

[0669] As another example, in accordance with the image playback command of the external device received through the user input interface unit 4150, the control unit 4170 can output a video signal or an audio signal that is input through the external device interface unit 4135 from an external device (for example, a camera or video camera), through the display unit 4180 or the audio output unit 4185.

[0670] Meanwhile, the control unit 4170 may control the display unit 4180 to display an image. For example, the control unit 4170 may control the display unit 4180 to display a broadcast image that is input through the tuner 4110, an external input image that is input through the external device interface unit 4156, and an image that is input through the network interface unit, or an image that is stored in block 4140 storage. At this time, the image that is displayed on the display unit 4180 may be a still image or a video, and may be a 2D video or a 3D video.

[0671] Also, the control unit 4170 may control playback of the content. At this time, the content may be content that is stored in the digital device 4100, or received broadcast content, or external input content that is input from outside. The content may be at least one of a broadcast image, an external input image, an audio file, a still image, a linked web screen, and a document file.

[0672] When moving to the application view item, the control unit 4170 can control the display of a list of applications or applications that are located in the digital device 4100 or that can be downloaded from an external network.

[0673] The control unit 4170 can manage the installation and operation of an application downloaded from an external network, along with various user interfaces. And the control unit 4170 can control the image, which is associated with the application to be executed, to be displayed in the display unit 4180 by user selection.

[0674] Although not shown in the figure, it is also possible to optionally equip a channel view processing unit to generate a thumbnail image corresponding to a channel signal or an external input signal.

[0675] The channel view processing unit may receive the TS stream signal that is output from the demodulation unit 4120 or the stream signal that is output from the external device interface unit 4156, extract an image from the input stream signal, and generate a thumbnail image.

[0676] The generated thumbnail image may be input to the control unit 4170 as is or as encoded. Also, the generated thumbnail image may be encoded in the form of a stream and input to the control unit 4170. The control unit 4170 may display a list of thumbnails including a plurality of thumbnail images on the display unit 4180 using input thumbnail images. The thumbnail images in a given thumbnail list can be updated sequentially or simultaneously. Accordingly, the user can easily understand the contents of the plurality of broadcast channels.

[0677] The display unit 4180 converts image signals, data signals, OSD signals and the like processed by the controller 4170, or image signals and data signals received from the external device interface unit 4156 into R, G, and B signals, respectively, and generates a signal arousal.

[0678] The display unit 4180 may be a PDP, LCD, OLED, flexible display, 3D display, or the like.

[0679] The display unit 4180 may be configured as a touch screen and used as an input device in addition to an output device.

[0680] The audio output unit 4185 receives a signal processed by the controller 4170, such as a stereo signal, a 3.1 channel signal, or a 5.1 channel signal, and outputs it as audio. The audio output unit 4185 may be implemented as various types of speakers.

[0681] Meanwhile, in order to register the user's gesture as described above, a registration unit (not shown) with at least one of a touch sensor, a voice sensor, a position sensor, and a motion sensor can be further provided in the digital device 4100. The signal detected by the registration unit (not shown) may be transmitted to the control unit 4170 via the user input interface unit 4150.

[0682] Additionally, a photographing unit (not shown) may be provided for photographing the user. Information of the image photographed by the photographing unit (not shown) may be input to the control unit 4170.

[0683] The control unit 4170 may detect the user's gesture alone or in combination with an image captured by the photographing unit (not shown) or a signal registered by the recording unit (not shown).

[0684] The power supply unit 4190 supplies power to the entire digital device 4100.

[0685] Specifically, the power supply unit 4190 can supply power to the control unit 4170, which may be implemented as a SOC system-on-chip, the display unit 4180 for displaying an image, and the audio output unit 4185 for audio output.

[0686] To this end, the power supply unit 4190 may include a converter (not shown) that converts AC power to DC power. For example, when the display unit 4180 is implemented as a liquid crystal panel with a plurality of backlights, a PWM inverter (not shown) for dimming or dimming may be further provided.

[0687] The remote control device 4200 transmits the user input to the user input interface unit 4150. To this end, the remote control device 4200 may use Bluetooth, RF (radio frequency) communication, infrared IR communication, UWB (Ultra Wide Band), ZigBee (ZigBee) method, and so on.

[0688] In addition, the remote control device 4200 may receive an image, audio, or data signal that is output from the user input interface unit 4150, display it on the remote control device 4200, or output voice or vibration.

[0689] The digital device 4100 described above may be a digital broadcast receiver capable of processing a fixed or mobile ATSC type or a DVB type digital broadcast signal.

[0690] In addition, the digital device in accordance with the invention may omit some components or additionally include components not illustrated, if necessary. As described above, the digital device may not have a tuner and demodulator, and may also receive and play content through a network interface unit or an external device interface unit.

[0691] FIG. 42 is a block diagram illustrating the detailed configuration of the control unit in FIG. from 39 to 41.

[0692] An example control unit may include a demultiplexing unit 4210, an image processing unit 4220, an OSD generation unit 4240, a mixer 4250, an FRC converter 4255, and a formatter 4260. In addition, although not illustrated, the control unit may further include a voice processing unit and a data processing unit.

[0693] The demultiplexer 4210 demultiplexes the input stream. For example, the demultiplexer 4210 may demultiplex the input MPEG-2 TS into a video signal, an audio signal, and a data signal. Here, the streaming signal that is input to the demultiplexer 4210 may be a streaming signal that is outputted from a tuner or demodulator or an external device interface.

[0694] The image processing unit 4220 performs image processing of the demultiplexed image signal. To this end, the image processing unit 4220 may include a video decoder 4225 and a scaler 4235.

[0695] The video decoder 4225 decodes the demultiplexed video signal, and the scaler 4235 performs resolution scaling of the decoded video signal to be output to the display unit.

[0696] The video decoder 4225 may support various standards. For example, the video decoder 4225 performs the function of an MPEG-2 decoder when the video signal is encoded in MPEG-2 standard, and the function of an H.264 decoder when the video signal is encoded by DMB or H.264 digital media broadcasting.

[0697] The video signal decoded by the video processing unit 4220 is input to the mixer 4250.

[0698] The OSD generating unit 4240 generates OSD data according to user input or independently. For example, the OSD generation unit 4220 generates data for displaying various data on the screen of the display unit 4180 in graphic or text form based on the control signal of the user input interface unit. The generated OSD data includes various data such as a user interface screen of a digital device, various menu screens, widgets, thumbnails, and viewing rate information.

[0699] An OSD generating unit 4240 may generate data for displaying subtitles of broadcast images or broadcast information based on the EPG.

[0700] The mixer 4250 mixes the OSD data generated by the OSD generation unit 4240 and the image signal processed by the video processing unit and provides them to the formatter 4260. Since the decoded video signal and OSD data are mixed, the OSD is displayed superimposed on broadcast video or external input video.

[0701] Converter 4255 frame rate FRC converts the frame rate of the output video. For example, the frame rate converter 4255 may convert an input 60Hz image frame rate to a frame rate of, for example, 120Hz or 240Hz, depending on the output frequency of the display unit. As described above, various methods may exist in the method for converting the frame rate. For example, when the frame rate converter 4255 converts the frame rate from 60Hz to 120Hz, the same first frame is inserted between the first frame and the second frame, or a third frame predicted from the first frame and the second frame is inserted between the first frame and the second frame. As another example, when the frame rate converter 4255 converts the frame rate from 60Hz to 240Hz, then three identical frames or predicted frames may be inserted between existing frames. If a separate frame conversion is not performed, the frame rate conversion block 4255 may be bypassed.

[0702] The formatter 4260 changes the output of the frame rate converter 4255 to match the output format with the display unit. For example, formatter 4260 may output R, G, and B data signals, and the three R, G, and B data signals may be output as low voltage differential signaling LVDS or mini-LVDS. Also, when the output of the frame rate converter 4255 is a 3D video signal, the formatter 4260 can support a 3D service via the display unit by configuring the output as a 3D format according to the output format of the display unit.

[0703] An audio processing unit (not shown) in the control unit may perform audio processing on the demultiplexed audio signal. An audio processing unit (not shown) may support various audio formats. For example, even when an audio signal is encoded in formats such as MPEG-2, MPEG-4, AAC, HE-AAC, AC-3, BSAC, the audio processing unit may be provided with a decoder that matches the formats.

[0704] Also, an audio processing unit (not shown) in the control unit may process base control, treble control, volume control, and the like.

[0705] A data processing unit (not shown) in the control unit may perform data processing on the demultiplexed data signal. For example, the data processing unit may decode the demultiplexed data signal as it is encoded. Here, the encoded data signal may be EPG information including broadcast information such as start time and end time of the broadcast of the broadcast program on each channel.

[0706] Meanwhile, the above-described digital apparatus is an example according to the invention, and each component may be integrated, added, or omitted depending on the specification of the actual digital apparatus. Those. if necessary, two or more components may be combined into one component, or one component may be subdivided into two or more components. Also, the function that is performed in each block serves to describe an embodiment of the invention, and the specific operation or apparatus does not limit the scope of the invention.

[0707] The digital device may be an image signal processing device that performs signal processing of an image stored in the device or an input image. As another example, an image signal processing apparatus, an STB set-top box, a DVD player, a Blu-ray player, a gaming device, a computer, and the like can be further exemplified, from which the display unit 4180 and the audio output unit 4185 shown in FIG. . 41.

[0708] FIG. 43 is a diagram illustrating an example in which the screen of the digital device displays the main image and the sub-image at the same time, according to the embodiment.

[0709] A digital device according to an embodiment may simultaneously display the main image 4310 and the sub-image or sub-image 4320 on the screen 4300. The main image 4310 may be referred to as the first image, and the sub-image 4320 may be referred to as the second image. The main picture 4310 and the sub picture 4320 may include, but are not limited to, video, still picture, EPG, GUI, OSD, and the like. The main image 4310 may mean an image that is relatively smaller than the electronic device screen 4300 while being simultaneously displayed on the electronic device screen 4300 along with the sub image 4320, and may be referred to as a picture-in-picture PIP. On FIG. 43, the main image 4310 is displayed at the top left of the screen 4300 of the digital device, but the location where the main image 4310 is displayed is not limited to this, and the main image 4310 can be displayed at any location within the screen 4300 of the digital device.

[0710] The main picture 4310 and the sub picture 4320 may be directly or indirectly related to each other. As an example, the main image 4310 may be a streaming video, and the sub-image 4320 may be a GUI that sequentially displays thumbnails of video elements including streaming video-like information. As another example, the main picture 4310 may be a broadcast picture and the sub picture 4320 may be an EPG. As another example, the main image 4310 may be a broadcast image and the sub image 4320 may be a GUI. Examples of the main image 4310 and the sub image are not limited to this.

[0711] In one embodiment, the main image 4310 is a broadcast image received via the broadcast channel, and the sub image 4320 may be information that is associated with the broadcast image received via the broadcast channel. Information associated with a broadcast image received via a broadcast channel may include, for example, EPG information including an integrated channel schedule and detailed broadcast program information, and broadcast program overview information, for example.

[0712] In another embodiment, the main image 4310 is a broadcast image received via a broadcast channel, and the sub image 4320 may be an image that is generated based on information previously stored in the digital device. An image generated based on information previously stored in the digital device may include, for example, EPG basic user interface UI, basic channel information, image resolution manipulation UI, and sleep reservation UI, and is not limited to this.

[0713] In another embodiment, the main image 4310 is a broadcast image received via a broadcast channel, and the sub image 4320 may be information associated with a broadcast image received via a network. The information associated with the broadcast image received via the network may be, for example, information obtained via a search engine based on the network. In particular, for example, information related to the character currently displayed on the main image 4310 can be obtained through a web-based search engine.

[0714] However, the example is not limited to this, and information related to the broadcast image received via the network can be obtained by using, for example, an AI system. In particular, for example, the estimated location on the map of the place currently displayed on the main image 4310 can be obtained by using network-based deep learning, and the digital device can receive information about the estimated location on the map of the place that is currently displayed. on the main image 4310, via the network.

[0715] The digital device according to the embodiment can receive at least one of the image information of the main image 4310 and the image information of the sub image 4320 from outside. The image information of the main image 4310 may include, for example, a broadcast signal received via a broadcast channel, source code information of the main image 4310, and IP packet (Internet Protocol Packet) information of the main image 4310 received via the network, but is not limited to this. Similarly, the image information of the sub-image 4320 includes, for example, a broadcast signal received via a broadcast channel, source code information of the sub-image 4320, IP packet information of the sub-image 4320 received via a network, etc., but is not limited to . The digital device may decode and use the image information of the main image 4310 received externally or the image information of the sub-image 4320. However, in some cases, the digital device may store the image information of the main image 4310 or the image information of the sub-image 4320 internally.

[0716] The digital device may display the main image 4310 and the sub image 4320 on the screen 4300 of the digital device based on the image information of the main image 4310 and the information associated with the sub image 4320.

[0717] In one example, the digital device decoder 200 includes a main picture decoder and a sub picture decoder, and the main picture decoder and the sub picture decoder can respectively decode main picture image information 4310 and sub picture image information 4320. The renderer includes a main video renderer (first renderer) and an auxiliary video renderer (second renderer). The main picture renderer may display the main picture 4310 in the first area of the screen 4300 of the digital device based on the information decoded by the main picture decoder, and the sub picture renderer may cause the sub picture 4320 to be displayed in the second area of the screen 4300 of the digital device based on the information decoded by the sub picture decoder. Images.

[0718] In another example, the digital device decoding apparatus 200 may decode the image information of the main image 4310 and the image information of the sub image. Based on the information decoded by the decoder 200, the renderer may process the main image 4310 and the sub-image 4320 together to simultaneously display on the screen 4300 of the digital device.

[0719] i.e. according to this document, it is possible to provide a method for processing an image service in a digital device. The image service processing method may include: receiving image information, decoding a (main) image based on the image information, rendering or displaying a decoded image in a first zone on a display, and rendering or displaying a sub image in a second zone on a display. In this case, the step in which the first picture is decoded may follow the decoding procedure in the decoding apparatus 200 according to FIG. 3 described above. For example, as described above, decoding the first picture may include extracting prediction samples for the current block based on inter- or intra-prediction, extracting residual samples for the current block based on received residual information, and generating reconstructed samples based on prediction samples and/or residual samples. Additionally, the step of decoding the first picture may include a step of performing an in-loop filtering procedure on the reconstructed picture including the reconstructed samples.

[0720] For example, the auxiliary image may be an electronic program guide, an OSD, or a GUI. For example, image information may be received via the broadcast network, and information regarding the sub-image may be received via the broadcast network. For example, image information may be received via a communication network, and information regarding a sub-image may be received via a communication network. For example, image information may be received via a broadcast network, and sub-image information may be received via a communication network. For example, the image information may be received via a broadcast network or a communication network, and information regarding the sub-image may be stored in a storage medium in the digital device.

[0721] In the above embodiments, the elements and characteristics of the invention have been combined in a specific form. Each of the elements or characteristics may be considered as optional, unless expressly described otherwise. Each of the elements or characteristics may be implemented in a form that does not need to be combined with other elements or characteristics. In addition, some of the elements and/or characteristics may be combined to form an embodiment of the invention. The sequence of operations described in the embodiments of the invention may be changed. Some of the elements or characteristics of an embodiment may be included in another embodiment or may be replaced by corresponding elements or characteristics of another embodiment. Obviously, an embodiment can be created by combining claims that do not have an explicit citation relationship in the claims and can be included as a new claim by post-filing changes.

[0722] An embodiment in accordance with the invention may be implemented by various means, such as hardware, firmware, software, or a combination thereof. If implemented in hardware, an embodiment of the invention may be implemented using one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processors (DSPDs), programmable logic devices (PLDs), programmable gate arrays (FPGA), processors, controllers, microcontrollers, microprocessors, etc.

[0723] In the case of implementation through firmware or software, an embodiment of the invention may be implemented in the form of a module, procedure, or function to perform the above functions or operations. The software code may be stored in memory and actuated by a processor. The memory may be located within or outside the processor and may communicate with the processor through a variety of known means.

[0724] Those skilled in the art will appreciate that the invention may be embodied in other specific forms without departing from the inherent characteristics of the invention. Accordingly, the detailed description should not be construed as limiting, but should be construed as illustrative in all respects. The scope of the invention is to be determined by a reasonable analysis of the appended claims, and all changes within the equivalent range of the invention are included within the scope of the invention.

Industrial Applicability

[0725] The above preferred embodiments of the invention have been disclosed for the purpose of illustration, and various other embodiments may be improved, modified, replaced, and added to by those skilled in the art without departing from the technical spirit and scope of the invention disclosed in the appended claims.

Claims (39)

1. A method for processing a video signal based on a history-based motion vector prediction, comprising the steps of: configuring a merge candidate list based on a spatial neighbor block and a temporal neighbor block of the current block; adding a history-based motion vector predictor (HMVP) candidate of the current block to the merge candidate list; and generating a prediction sample of the current block based on the motion information of the merge candidate included in the merge candidate list, wherein the HMVP candidate list is updated based on the movement information of the merger candidate, and wherein the HMVP candidate is added to the merge candidate list based on the fact that the HMVP candidate has motion information different from the motion information of predetermined spatial merge candidates included in the merge candidate list. 2. The method of claim 1, wherein the maximum HMVP candidate list size is less than the maximum merge candidate list size. 3. The method of claim 2, wherein the maximum HMVP candidate list size is less than the maximum merge candidate list size by 1. 4. The method of claim 2, wherein the maximum HMVP candidate list size is defined as 5. 5. The method of claim 1, wherein the number of predetermined spatial merge candidates is determined to be 2. 6. The method of claim 1, wherein the predetermined spatial merge candidates include a left neighbor spatial merge candidate and a top neighbor spatial merge candidate of the current block. 7. Device for processing video signals based on interframe prediction, comprising: a memory configured to store video signals; and processor combined with memory wherein the processor is configured to: configuring a merge candidate list based on a spatial neighbor block and a temporal neighbor block of the current block; adding a history-based motion vector predictor (HMVP) candidate of the current block to a merge candidate list; and generating a prediction sample of the current block based on the movement information of the merge candidate included in the merge candidate list, wherein the HMVP candidate list is updated based on the movement information of the merger candidate, and wherein the HMVP candidate is added to the merge candidate list based on the fact that the HMVP candidate has motion information different from the motion information of predetermined spatial merge candidates included in the merge candidate list. 8. The apparatus of claim 7, wherein the maximum size of the HMVP candidate list is less than the maximum size of the merge candidate list. 9. The apparatus of claim 8, wherein the maximum size of the HMVP candidate list is less than the maximum size of the merge candidate list by 1. 10. The apparatus of claim 8, wherein the maximum HMVP candidate list size is defined as 5. 11. The apparatus of claim 7, wherein the number of predetermined spatial fusion candidates is determined to be 2. 12. The apparatus of claim 7, wherein the predetermined spatial merge candidates include a left neighbor spatial merge candidate and a top neighbor spatial merge candidate of the current block. 13. A method for processing a video signal based on a history-based motion vector prediction, comprising the steps of: configuring a merge candidate list based on a spatial neighbor block and a temporal neighbor block of the current block; adding a history-based motion vector predictor (HMVP) candidate of the current block to the merge candidate list; generating a prediction sample of the current block based on the motion information of the merge candidate included in the merge candidate list; and encoding a merge index indicating a merge candidate used to generate a prediction sample, wherein the list of HMVP candidates is updated based on the movement information of the merger candidate, wherein the HMVP candidate is added to the merge candidate list based on the fact that the HMVP candidate has motion information different from the motion information of predetermined spatial merge candidates included in the merge candidate list. 14. A non-transitory decoder-readable storage medium for storing a bitstream generated by the method, the method comprising: configuring a merge candidate list based on a spatial neighbor block and a temporal neighbor block of the current block; adding a history-based motion vector predictor (HMVP) candidate of the current block to the merge candidate list; generating a prediction sample of the current block based on the motion information of the merge candidate included in the merge candidate list; and encoding a merge index indicating a merge candidate used to generate a prediction sample into a bitstream, wherein the list of HMVP candidates is updated based on the movement information of the merger candidate, wherein the HMVP candidate is added to the merge candidate list based on the fact that the HMVP candidate has motion information different from the motion information of predetermined spatial merge candidates included in the merge candidate list.

Images