RU2807594C2 - Method and device for processing video signal based on history-based motion vector prediction - Google Patents

Abstract

FIELD: encoding.

SUBSTANCE: invention relates to the encoding for performing inter-frame prediction using history-based motion vector prediction. Devices for encoding and decoding images are proposed. The devices perform video signal processing based on interframe prediction and comprise the steps of: configuring a list of merge candidates based on a spatial neighbour block and a temporal neighbour block of the current block; adding the current block based on the history of the merge candidate to the list of merge candidates; obtaining a merge index indicating a merge candidate that is used for inter-frame prediction of the current block in the merge candidate list; generating a prediction sample of the current block based on the motion information of the merge candidate indicated by the merge index; and updating the list of history-based merger candidates based on the traffic information.

EFFECT: providing a constraint on redundancy checking to improve prediction performance when adding an HMVP candidate to a merge candidate list (or AMVP candidate list).

13 cl, 43 dwg, 11 tbl

Description

Field of technology

[0001] An embodiment of the invention relates to a method and apparatus for processing a video signal based on inter-frame prediction, and more particularly, to a method and apparatus for performing inter-frame prediction using history-based motion vector prediction.

State of the art

[0002] Compression coding refers to a series of signal processing techniques for transmitting digitally represented information over a communications link or techniques for storing information in a form suitable for storage media. A medium including picture, image, audio, etc. may be the target of compression encoding, and in particular, a technique for performing compression encoding on a picture is referred to as video compression.

[0003] Next generation video content is expected to have the characteristics of high spatial resolution, high frame rate, and high scene representation dimension. Processing such content will result in dramatic increases in memory capacity, memory access speed, and processing power.

[0004] Accordingly, development of an encoding tool to efficiently process next generation video content is required.

The essence of the invention

Technical problem

[0005] The purpose of an embodiment of the invention is to provide redundancy check constraints to improve prediction performance when adding an HMVP candidate to a merge candidate list (or AMVP candidate list).

[0006] Another object of an embodiment of the invention is to provide a method for determining the size of an HMVP table in order to efficiently apply an HVMV candidate to a merge candidate list (AMVP candidate list).

[0007] The technical objectives to be achieved by the embodiment of the invention are not limited to the above-mentioned technical objectives, and other technical objectives not described above can be clearly understood by one skilled in the relevant art from the following description.

Technical solution

[0008] In an aspect of an embodiment of the invention, a method of processing a video signal based on history-based motion vector prediction may comprise the steps of: configuring a merge candidate list based on a spatial neighbor block and a temporal neighbor block of the current block; adding a history-based merge candidate of the current block to the list of merge candidates; obtaining a merge index indicating a merge candidate, which is used for inter-frame prediction of the current block, in the list of merge candidates, generating a prediction sample of the current block based on the motion information of the merge candidate indicated by the merge index; and updating the list of history-based merge candidates based on the motion information, wherein the history-based merge candidate is added to the list of merge candidates when it has motion information different from that of the predetermined merge candidate of the candidates. for merger included in the list of merger candidates.

[0009] In an embodiment, the size of the history-based merge candidate list may be determined based on the maximum number of merge candidates in the merge candidate list.

[0010] In an embodiment, the size of the history-based merge candidate list can be defined as a value equal to 1, which is subtracted from the maximum number of merge candidates in the merge candidate list.

[0011] In an embodiment, the size of the history-based merge candidate list may be defined as 5.

[0012] In an embodiment, a history-based merger candidate may be added to the list of merger candidates when the history-based merger candidate includes movement information other than the movement information of a specific number of predetermined merger candidates among the candidates for merger included in the list of merger candidates.

[0013] In an embodiment, a history-based merge candidate may be added to the list of merge candidates when the history-based merge candidate includes motion information different from the motion information of specific spatial merge candidates included in the list merger candidates.

[0014] In an embodiment, the history-based merge candidate may be retrieved from a predetermined number of candidates in the list of history-based merge candidates.

[0015] In another aspect of the embodiment, a video signal processing device based on inter-frame prediction may comprise: a memory configured to store video signals; and a processor integrated with the memory, wherein the processor is configured to: configure a list of merge candidates based on a spatial neighbor block and a temporal neighbor block of the current block; adding the history-based merge candidate of the current block to the list of merge candidates; obtaining a merge index indicating a merge candidate that is used for inter-frame prediction of the current block in the merge candidate list; generating a prediction sample of the current block based on information about the movement of the merger candidate indicated by the merger index; and updating the list of history-based merger candidates based on the motion information, wherein the history-based merger candidate is added to the list of merger candidates when it has motion information different from that of the predetermined merger candidate of the candidates. for merger included in the list of merger candidates.

Technical results

[0016] According to an embodiment of the invention, the complexity according to the redundancy check can be reduced and the compression efficiency can be increased by limiting the redundancy check to be added to the merge candidate list (or AMVP candidate list).

[0017] In accordance with an embodiment of the invention, the memory load associated with storing the HMVP table can be reduced by determining the size of the HMVP table.

[0018] The results that can be obtained in the invention are not limited to the above-mentioned results, and other technical results not described above can be clearly understood by a person skilled in the relevant field of technology to which the invention relates from the following description.

Description of drawings

[0019] The accompanying drawings, which are included as part of the detailed description to assist in understanding the invention, provide embodiments of the invention and describe the technical characteristics of the invention along with the detailed description.

[0020] FIG. 1 illustrates an example of a video encoding system in accordance with an embodiment of the invention.

[0021] FIG. 2 is an embodiment to which the invention is applied, and is a schematic block diagram of an encoding apparatus for encoding a video signal/image signal.

[0022] FIG. 3 is an embodiment to which the invention is applied, and is a circuit block diagram of a decoding apparatus for decoding a video/image signal.

[0023] FIG. 4 shows an example of a block diagram of a content streaming system in accordance with an embodiment of the invention.

[0024] FIG. 5 shows an example of a block diagram of a video signal processing apparatus according to an embodiment of the invention.

[0025] FIG. 6 is an example of a block crushing structure according to an embodiment of the invention, FIG. 6a to 6b respectively illustrate examples of block splitting structures according to quadtree (QT), binary tree (BT), ternary tree (TT) and asymmetric tree (AT).

[0026] FIG. 7 and 8 are respectively an inter-frame prediction-based video/image encoding procedure and an inter-frame prediction means in an encoding apparatus according to an embodiment of the invention.

[0027] FIG. 9 and 10 are respectively an inter-frame prediction based video/image decoding procedure and an inter-frame prediction means in an encoding apparatus according to an embodiment of the invention.

[0028] FIG. 11 illustrates an example of creating spatial merge candidates for the current block.

[0029] FIG. 12 is a flowchart illustrating a method for configuring a merge candidate list in accordance with an embodiment to which the invention is applied.

[0030] FIG. 13 is a flowchart illustrating a method for configuring a prediction candidate list (MVP candidate list) in accordance with an embodiment to which the invention is applied.

[0031] FIG. 14 illustrates an example of motion patterns in accordance with an embodiment of the invention.

[0032] FIG. 15 illustrates an example of a control point motion vector for affine motion prediction in accordance with an embodiment of the invention.

[0033] FIG. 16 illustrates an example of the motion vector of each sub-block of a block to which affine motion prediction has been applied in accordance with an embodiment of the invention.

[0034] FIG. 17 illustrates an example of neighboring blocks that are used for affine motion prediction in an affine fusion mode in accordance with an embodiment of the invention.

[0035] FIG. 18 illustrates an example of a block on which affine motion prediction is performed using neighboring blocks to which affine motion prediction has been applied in accordance with an embodiment of the invention.

[0036] FIG. 19 is a diagram for describing a method for generating a merge candidate list using an adjacent affine encoding block in accordance with an embodiment of the invention.

[0037] FIG. 20 and 21 are diagrams for describing a method for creating an affine merge candidate list using a neighbor block encoded by affine prediction in accordance with an embodiment of the invention.

[0038] FIG. 22 illustrates an example of adjacent blocks that are used for affine motion prediction in an affine interframe mode in accordance with an embodiment of the invention.

[0039] FIG. 23 illustrates an example of adjacent blocks that are used for affine motion prediction in an affine interframe mode in accordance with an embodiment of the invention.

[0040] FIG. 24 and 25 are diagrams illustrating a method for extracting a candidate motion vector using motion information of a neighboring block in an affine interframe mode according to an embodiment of the invention.

[0041] FIG. 26 illustrates an example of a subblock unit motion vector affine field in accordance with an embodiment of the invention.

[0042] FIG. 27 is a flowchart for describing a method for storing HMVP in accordance with an embodiment of the invention.

[0043] FIG. 28 is a diagram for describing an HMVP table that operates according to a FIFO queue discipline without restrictions in accordance with an embodiment of the invention.

[0044] FIG. 29 is a diagram for describing an HMVP table that operates according to a constrained FIFO queue discipline in accordance with an embodiment of the invention.

[0045] FIG. 30 is a diagram illustrating an HMVP LUT and a long-term HMVP LUT in accordance with an embodiment of the invention.

[0046] FIG. 31 is a diagram illustrating an example of a method for updating an HMVP LUT in accordance with an embodiment of the invention.

[0047] FIG. 32 is a diagram illustrating a method for limiting the number of HMVP candidates, i.e. the purpose of checking the clipping, in accordance with an embodiment of the invention.

[0048] FIG. 33 is a flowchart illustrating an example of a method for performing a clipping check in accordance with an embodiment of the invention.

[0049] FIG. 34 is a diagram for describing a method for extracting an H-STMVP candidate using motion vectors that refer to different reference pictures in accordance with an embodiment of the invention.

[0050] FIG. 35 is a diagram illustrating the location of a block for extracting a legacy HMVP affinity candidate in accordance with an embodiment of the invention.

[0051] FIG. 36 is a diagram illustrating an affine merge list or an AMVP affine list in accordance with an embodiment of the invention.

[0052] FIG. 37 is a flow chart illustrating a method of processing a video signal based on history-based motion vector prediction in accordance with an embodiment to which the invention is applied.

[0053] FIG. 38 is a diagram schematically showing an example of a service system including a digital device.

[0054] FIG. 39 is a block diagram illustrating a digital device according to an embodiment.

[0055] FIG. 40 is a configuration block diagram illustrating another embodiment of a digital device.

[0056] FIG. 41 is a block diagram illustrating a digital device according to another embodiment.

[0057] FIG. 42 is a block diagram illustrating a detailed configuration of the control unit in FIG. from 39 to 41.

[0058] FIG. 43 is a diagram illustrating an example in which a digital device screen displays a main image and a sub-image simultaneously, according to an embodiment.

Embodiment of the invention

[0059] Next, preferred embodiments of the invention will be described by reference to the accompanying drawings. The description which will be presented below with the help of the accompanying drawings serves to describe exemplary embodiments of the invention, and is not intended to describe the only embodiment in which the invention may be implemented. The description below includes specific details in order to provide a proper understanding of the invention. However, it should be understood that the invention may be practiced without specific details by those skilled in the art. In some cases, in order to avoid obscurity in the technical concept of the invention, structures or devices that are generally known may be omitted, or may be depicted as a block diagram focusing on the basic functions of the structures or devices.

[0060] In some cases, in order to avoid obscurity in the technical concept of the invention, structures or devices that are generally known may be omitted, or may be depicted as a block diagram focusing on the basic functions of the structures or devices.

[0061] In addition, although general concepts that are widely used at present are selected as concepts in the invention as far as possible, a concept that is arbitrarily chosen by the applicant is used in a specific case. Since in such a case the meaning of the concept will be clearly described in the relevant part of the description, it should be understood that the invention will not be simply interpreted in accordance with the concepts that are only used in the description of the invention, but the meaning of the concepts must be clarified.

[0062] Specific terminologies that are used in the description below may be provided to assist in understanding the invention. Moreover, the specific terminology may be modified in other forms within the scope of the technical concept of the invention. For example, signal, data, sample, picture, slice, tile, frame, block, etc. can be appropriately replaced and interpreted in each encoding process.

[0063] Hereinafter, in this specification, a “processing unit” means a unit in which an encoding/decoding process such as prediction, transform and/or quantization is performed. A processing unit can be interpreted as having a value of one for the luma component and one for the chrominance component. For example, a processing unit may correspond to a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), or a transformation unit (TU).

[0064] In addition, a processing unit can be interpreted as a unit for a luma component or a unit for a chrominance component. For example, a processing unit may correspond to a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB) for the luminance component. Alternatively, the processing unit may correspond to a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB) for the chrominance component. Moreover, the invention is not limited to this, and a processing unit can be interpreted as a value including a unit for a luma component and a unit for a chrominance component.

[0065] Moreover, the processing unit is not essentially limited to a square block and can be constructed in the shape of a polygon with three or more vertices.

[0066] In addition, hereinafter, in this specification, a pixel, picture element, coefficient (transformation coefficient or transformation coefficient after first-order transformation), etc. generally called a sample. Additionally, using a sample may mean using a pixel value, a picture element value, a transform factor, or the like.

[0067]

[0068] FIG. 1 illustrates an example of a video encoding system in accordance with an embodiment of the invention.

[0069] The video encoding system may include a source device 10 and a receiving device 20. The source device 10 may transmit encoded video/image information or data to the receiving device 20 in a file or streaming format via a storage medium or a network.

[0070] The source device 10 may include a video source 11, an encoder 12, and a transmitter 13. The receiving device 20 may include a receiver 21, a decoding device 22, and a renderer 23. The source device may be referred to as a video/image encoder, and the receiving device may be referred to as a video/image decoding device. The transmitter 13 may be included in the encoding device 12. The receiver 21 may be included in the decoding device 22. The renderer may include a display, and the display may be configured as a separate device or an external component.

[0071] The video source may obtain video/image data through a video/image capture, synthesis, or shaping process. The video source may include a video/image capture device and/or a video/image generation device. The video/image capturing device may include, for example, one or more cameras, a video/image archive including previously captured video/images, and the like. The video/image generating apparatus may include, for example, a computer, a tablet, and a smartphone, and may electronically generate video/image data. For example, the virtual video/image data may be generated by a computer or the like, and in this case, the video/image capturing process may be replaced by a related data generation process.

[0072] The encoding device 12 may encode the input video/image. The encoding device 12 may perform a number of procedures such as prediction, transform and quantization for compression and encoding efficiency. The encoded data (encoded video information/image information) may be output in the form of a bit stream.

[0073] The transmitter 13 may transmit encoded video/image information or data that is output in the form of a bitstream to a receiver of a receiving device via a digital storage medium or a network in a file or streaming format. Digital storage media may include various storage media such as universal serial bus, secure digital SD memory card, compact disc CD, digital video disc DVD, Blu-ray, hard disk drive HDD, solid state drive SSD. The transmitter 13 may include an element for generating a multimedia file through a predetermined file format, and may include an element for transmission via a broadcast/communications network. The receiver 21 may extract the bitstream and transmit it to the decoding device 22.

[0074] The decoding apparatus 22 can decode video/image data by performing a series of procedures such as inverse quantization, inverse transform, and prediction corresponding to the operations of the encoding apparatus 12.

[0075] The renderer 23 may render the decoded video/image. The video/image after rendering can be displayed via the display.

[0076]

[0077] FIG. 2 is an embodiment to which the invention is applied, and is a schematic block diagram of an encoding apparatus for encoding a video signal/image signal.

[0078] Referring to FIG. 2, the encoding device 100 may be configured to include an image division means 110, a subtractor 115, a converter 120, a quantizer 130, an inverse quantizer 140, an inverse converter 150, an adder 155, a filter 160, a memory 170, an interframe prediction means 180, 185 intra-frame prediction and entropy encoder 190. The inter-frame prediction engine 180 and the intra-frame prediction engine 185 may be collectively referred to as a predictor. In other words, the predictor may include an inter-frame predictor 180 and an intra-frame predictor 185. Converter 120, quantizer 130, inverse quantizer 140, and inverse converter 150 may be included in the remainder processor. The remainder processor may further include a subtractor 115. In one embodiment, an image divider 110, a subtractor 115, a converter 120, a quantizer 130, an inverse quantizer 140, an inverse converter 150, an adder 155, a filter 160, an interframe predictor 180, and an interframe predictor 185. intra-frame prediction and entropy encoder 190 may be implemented as a single hardware component (eg, an encoder or processor). In addition, memory 170 may be implemented as a hardware component (eg, memory or digital storage medium) in an embodiment, and may include a decoded picture buffer (DPB).

[0079] The image division means 110 may divide the input image (or picture or frame) that is input to the encoding device 100 into one or more processing units. For example, a processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively split from a coding tree unit (CTU) or a largest coding unit (LCU) based on a quadtree binary tree (QTBT) structure. For example, one encoding unit may be divided into multiple encoding units of greater depth based on a quadtree structure and/or a binary tree structure. In this case, for example, the quadtree structure may be applied first, and then the binary tree structure may be applied. Alternatively, a binary tree structure can be applied first. The encoding procedure in accordance with the invention can be performed on the basis of a final encoding unit that is no longer subdivided. In this case, the largest encoding unit can be directly used as the final encoding unit based on the encoding efficiency according to the characteristic of the image, or the encoding unit can be recursively divided into coding units of greater depth as necessary. Accordingly, the optimal size coding unit can be used as the final coding unit. Here, the encoding procedure may include a procedure such as prediction, transformation or reconstruction, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, each of the prediction units and the transformation units may be divided or split from each resulting encoding unit. The prediction unit may be a unit for predicting a sample, and the transform unit may be a unit from which a transform coefficient is extracted and/or a unit in which a residual signal is extracted from the transform coefficient.

[0080] The unit may be used interchangeably with a block or zone as appropriate. In general, an M×N block may indicate a set of samples configured with M columns and N rows, or a set of transform coefficients. In general, a sample may indicate a pixel or pixel value, or may indicate only a pixel/pixel value of a luma component, or only a pixel/pixel value of a chrominance component. In a sample, one picture (or image) can be used as a concept corresponding to a pixel or image element.

[0081] The encoding apparatus 100 may generate a residual signal (residual block or residual sample array) by subtracting a prediction signal (predicted block or prediction sample array) that is output by inter-frame prediction means 180 or intra-frame prediction means 185 from the input image signal (original block or an array of original samples). The generated residual signal is transmitted to the converter 120. Here, as illustrated, the block in which the prediction signal (prediction block or prediction sample array) is subtracted from the input image signal (source block or source sample array) in the encoder 100 may be called a subtractor 115 The prediction means may perform prediction for a target processing block (hereinafter referred to as the current block), and may generate a predicted block including prediction samples for the current block. The predictor may determine whether intra-prediction or inter-prediction is applied in the current block or CU. The prediction means may generate various pieces of information regarding the prediction, such as information about the prediction mode, as will be described later in the description of each prediction mode, and may transmit the information to the entropy encoder 190. The information regarding the prediction may be encoded in the entropy encoder 190 and may be output to bit stream form.

[0082] The intra-frame predictor 185 may predict the current block with reference to samples in the current picture. The referenced samples may be located adjacent to the current block or may be located at a distance from the current block depending on the prediction mode. In intra-frame prediction, prediction modes may include a plurality of non-angular modes and a plurality of angular modes. Non-angle mode may include DC mode and flat mode, for example. The angular mode may include 33 angular prediction modes or 65 angular prediction modes, for example, depending on the precise degree of the prediction direction. Here, angular prediction modes more or less than 33 angular prediction modes or 65 angular prediction modes may be used depending on the configuration, for example. The intra-frame prediction engine 185 may determine a prediction mode that is applied to the current block using a prediction mode that is applied to an adjacent block.

[0083] The inter-frame prediction engine 180 may extract a predicted block for a current block based on a reference block (reference sample array) indicated by a motion vector in the reference picture. Here, in order to reduce the amount of motion information that is transmitted in the inter-frame prediction mode, the motion information can be predicted as a unit of a block, sub-block or sample based on the correlation of the motion information between a neighboring block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter-prediction direction information (L0 prediction, L1 prediction, Bi prediction). In the case of inter-frame prediction, the neighbor block may include a spatial neighbor block in the current picture and a temporal neighbor block in the reference picture. The reference picture including the reference block and the reference picture including the temporary adjacent block may be the same or may be different. A temporary neighbor block may be referred to by a name that corresponds to a co-located reference block or co-located CU (colCU). A reference picture including a temporary neighbor block may be referred to as a co-located picture (ColPic). For example, the inter-frame prediction engine 180 may generate a list of candidates for motion information of neighboring blocks, and may generate information indicating which candidate is used to extract the motion vector and/or reference picture index of the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in the case of the skip mode and the merge mode, the interframe prediction engine 180 may use the motion information of a neighboring block as the motion information of the current block. In the case of skip mode, unlike merge mode, the residual signal may not be transmitted. In the case of a motion information prediction (MVP) mode, the motion vector of an adjacent block may be indicated by signaling a motion vector difference.

[0084] The prediction signal generated by the inter-frame predictor 180 or the intra-frame predictor 185 can be used to generate a reconstructed signal or a residual signal.

[0085] Converter 120 can generate transform coefficients by applying a transform circuit to the residual signal. For example, the transform scheme may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Korunen-Loev transform (KLT), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). Here, GBT means the transformation obtained from a graph if the relation information between pixels is represented as a graph. CNT means the transform obtained from the prediction signal generated using all the previously reconstructed pixels. Additionally, the transformation process can be applied to uniformly sized blocks of square shaped pixels, or can be applied to variable sized blocks of non-square shaped ones.

[0086] Quantizer 130 may quantize the transform coefficients and transmit them to entropy encoder 190. Entropy encoder 190 may encode the quantized signal (information regarding the quantized transform coefficients) and output it in the form of a bitstream. Information regarding the quantized transform coefficients may be called residual information. The quantizer 130 can reorder the quantized transform coefficients in block form into a one-dimensional vector form based on a coefficient scanning sequence, and can generate information regarding the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. Entropy encoder 190 can perform various encoding techniques, such as exponential Golomb coding, context-sensitive adaptive variable length coding (CAVLC), and context-sensitive adaptive binary arithmetic coding (CABAC). Entropy encoder 190 may encode information (eg, syntax element values) that is needed to reconstruct the video/image, in addition to the quantized transform coefficients, together or separately. Encoded information (eg, encoded video/image information) may be transmitted or stored in a network abstraction layer (NAL) unit in the form of a bit stream. The bitstream may be transmitted over a network or may be stored on a digital storage medium. Here, the network may include a broadcast network and/or a communications network. The digital storage media may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD and SSD. A transmitter (not illustrated) that transmits a signal that is output by the entropy encoder 190 and/or a store (not illustrated) for storing the signal may be an internal/external element of the encoding device 100, or the transmitter may be an element of the entropy encoder 190.

[0087] The quantized transform coefficients that are output by quantizer 130 can be used to generate a prediction signal. For example, the residual signal can be recovered by applying inverse quantization and inverse transform to the quantized transform coefficients via an inverse quantizer 140 and an inverse transformer 150 in a loop. The adder 155 may add the reconstructed residual signal with a prediction signal that is output by the inter-picture predictor 180 or the intra-picture predictor 185 so that a reconstructed signal (a reconstructed picture, a reconstructed block, or an array of reconstructed samples) can be generated. The predicted block can be used as a reconstructed block if there is no remainder for the target processing block, as is the case when the skip mode was used. The adder 155 may be referred to as a restorer or a restorer generator. The generated reconstructed signal can be used for intra-frame prediction of the next target processing block of the current picture, and can be used for inter-frame prediction of the next picture by filtering, as will be described later.

[0088] Filter 160 may improve subjective/objective picture quality by applying filtering to the reconstructed signal. For example, filter 160 may generate a modified reconstructed picture by applying various filtering techniques to the reconstructed picture. The modified reconstructed picture can be stored in the DPB 170. Various filtering methods may include deblocking filtering, adaptive sample shifting, adaptive contour filter and bidirectional filter, for example. Filter 160 may generate various pieces of filtering information, as will be described later in the description of each filtering method, and may transmit them to entropy encoder 190. The filtering information may be encoded by entropy encoder 190 and output in the form of a bitstream.

[0089] The modified reconstructed picture that is transmitted to the DPB 170 can be used as a reference picture in the inter-frame predictor 180. The encoding apparatus can prevent prediction mismatch between the encoding apparatus 100 and the decoding apparatus and improve encoding efficiency if inter-frame prediction is applied.

[0090] The DPB 170 may store the modified reconstructed picture in order to use the modified reconstructed picture as a reference picture in the inter-frame predictor 180.

[0091]

[0092] FIG. 3 is an embodiment to which the invention is applied, and is a circuit block diagram of a decoding apparatus for decoding a video/image signal.

[0093] Referring to FIG. 3, the decoding apparatus 200 may be configured to include an entropy decoder 210, an inverse quantizer 220, an inverse converter 230, an adder 235, a filter 240, a memory 250, an inter-frame predictor 260, and an intra-frame predictor 265. The inter-frame predictor 260 and the intra-frame predictor 265 may be collectively referred to as a predictor. Those. the predictor may include an inter-frame predictor 260 and an intra-frame predictor 265. The inverse quantizer 220 and the inverse transformer 230 may be collectively referred to as a remainder processor. Those. the remainder processor may include an inverse quantizer 220 and an inverse transformer 230. The entropy decoder 210, the inverse quantizer 220, the inverse transformer 230, the adder 235, the filter 240, the inter-frame predictor 260, and the intra-frame predictor 265 may be implemented as a single hardware component. (eg, a decoder or processor) in accordance with an embodiment. In addition, memory 170 may be implemented as a hardware component (eg, memory or digital storage medium) in an embodiment and may include a decoded picture buffer (DPB).

[0094] When a bitstream including video information/image information is input, the decoding apparatus 200 can reconstruct the image according to the processing process of the video information/image information in the encoding apparatus in FIG. 2. For example, the decoding apparatus 200 may perform decoding using a processing unit that is applied in the encoding apparatus. Accordingly, the processing unit for decoding may be an encoding unit, for example. A coding unit may be split from a coding tree unit or a largest coding unit depending on the quadtree structure and/or binary tree structure. In addition, the reconstructed image signal that is decoded and output by the decoding device 200 can be reproduced by the playback device.

[0095] The decoding apparatus 200 may receive a signal that is output by the encoding apparatus in FIG. 1, in the form of a bit stream. The received signal may be decoded by the entropy decoder 210. For example, the entropy decoder 210 may extract information (eg, video information/image information) for image reconstruction (or picture reconstruction) by analyzing the bit stream. For example, entropy decoder 210 may decode information in the bit stream based on an encoding method such as exponential Golomb coding, CAVLC, or CABAC, and may output the value of a syntax element for image restoration or quantized values of transform coefficients with respect to the remainder. More specifically, in the CABAC entropy decoding method, a bin corresponding to each syntax element can be received from a bitstream, a context model can be determined using information about the target decoding syntax element and decoding information of an adjacent or target decoding block or symbol/bin information. decoded in the previous step, the occurrence probability of a bin can be predicted based on a certain context model, and a symbol corresponding to the meaning of each syntax element can be generated by performing arithmetic decoding on the bin. Here, in the CABAC entropy decoding method, after a context model is determined, the context model can be updated by using the symbol/bin information decoded for the context model of the next symbol/bin. Information regarding prediction from information decoded in the entropy decoder 2110 may be provided to the prediction means (inter-frame prediction means 260 and intra-frame prediction means 265). Parameter information associated with the residual value on which entropy decoding has been performed in the entropy decoder 210, i.e. quantized transform coefficients may be input to the inverse quantizer 220. Moreover, information regarding filtering from the information decoded by the entropy decoder 210 may be provided to the filter 240. Meanwhile, a receiver (not illustrated) that receives a signal that is output by the encoding device may be additionally implemented as an internal/external element of the decoding device 200 or the receiver can be an element of the entropy decoder 210.

[0096] Inverse quantizer 220 may inversely quantize the quantized transform coefficients and output the transform coefficients. Inverse quantizer 220 may reorder the quantized transform coefficients into a two-dimensional block form. In this case, reordering can be performed based on a coefficient scanning sequence performed in the encoder. Inverse quantizer 220 may perform inverse quantization on the quantized transform coefficients using a quantization parameter (eg, quantization step size information) and may obtain the transform coefficients.

[0097] The inverse transform 230 may output a residual signal (a residual block or an array of residual samples) by applying an inverse transform to the transform coefficients.

[0098] The predictor may perform prediction of the current block and may generate a predicted block including prediction samples for the current block. The predictor can determine whether intra-prediction or inter-prediction is applied to the current block based on the prediction information that is output by the entropy decoder 210, and can determine a detailed intra/inter-prediction mode.

[0099] The intra-frame predictor 265 may predict the current block with reference to samples in the current picture. The referenced samples may be located adjacent to the current block or may be located at a distance from the current block depending on the prediction mode. In intra-frame prediction, prediction modes may include a plurality of non-angular modes and a plurality of angular modes. The intra-frame prediction engine 265 may determine the prediction mode that is applied to the current block by using the prediction mode that is applied to the adjacent block.

[0100] The inter-frame predictor 260 may extract a predicted block for the current block based on a reference block (reference sample array) indicated by a motion vector in the reference picture. Here, in order to reduce the amount of motion information that is transmitted in the inter-frame prediction mode, the motion information can be predicted as a unit of a block, sub-block or sample based on the correlation of the motion information between a neighboring block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter-prediction direction information (L0 prediction, L1 prediction, Bi prediction). In the case of inter-frame prediction, the neighbor block may include a spatial neighbor block in the current picture and a temporal neighbor block in the reference picture. For example, the inter-frame prediction engine 260 may configure a list of candidates for motion information of neighboring blocks, and may extract a motion vector and/or reference picture index of the current block based on the received candidate selection information. Inter-frame prediction can be performed based on various prediction modes. The prediction information may include information indicating an inter-prediction mode of the current block.

[0101] The adder 235 may generate a reconstructed signal (reconstructed picture, reconstructed block, or reconstructed sample array) by adding the resulting residual signal with a prediction signal (predicted block or prediction sample array) that is output by inter-frame predictor 260 or intra-frame predictor 265. The predicted block can be used as a reconstructed block if there is no remainder for the target processing block, as is the case when the skip mode was used.

[0102] The adder 235 may be referred to as a restorer or a reconstructed block generator. The generated reconstructed signal can be used for intra-frame prediction of the next processing target block in the current picture, and can be used for inter-frame prediction of the next picture by filtering, as will be described later.

[0103] Filter 240 may improve subjective/objective picture quality by applying filtering to the reconstructed signal. For example, filter 240 may generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and may transmit the modified reconstructed picture to DPB 250. Various filtering methods may include deblocking filtering, adaptive sample offset SAO, adaptive loop filter ALF, and a two-way filter. For example.

[0104] The reconstructed (modified) picture that is transmitted to the DPB 250 can be used as a reference picture in the inter-frame predictor 260.

[0105] In the invention, the embodiments described in the filter 160, inter-predictor 180 and intra-predictor 185 of the encoding device 100 can be applied to the filter 240, inter-predictor 260 and intra-predictor 265 of the decoding device 200, respectively, identically or in accordance.

[0106]

[0107] FIG. 4 shows an example of a block diagram of a content streaming system in accordance with an embodiment of the invention.

[0108] A content streaming system to which the invention applies may largely include an encoding server 410, a streaming server 420, a web server 430, a media storage 440, a user device 450, and a media input device 460.

[0109] The encoding server 410 may compress content that is input from multimedia input devices such as a smartphone, camera, camcorder, etc. into digital data to form a bitstream and transmit it to the streaming server 420. As another example, when multimedia input devices 460 such as a smartphone, camera, or video camera directly generate the bitstream, the encoding server 410 may be omitted.

[0110] The bitstream may be generated by an encoding method or a bitstream generation method to which the invention is applied, and the streaming server 420 may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0111] The streaming server 420 transmits media data to the user device 450 upon the user's request through the web server 430, and the web server 430 serves as an intermediary to inform the user what service is present. When a user requests a desired service through the web server 430, the web server 430 delivers the request to the streaming server 420, and the streaming server 420 transmits the media data to the user. At this time, the content streaming system may include a separate control server, in which case the control server serves to manage commands/responses between devices in the content streaming system.

[0112] The streaming server 420 may receive content from the media storage 440 and/or the encoding server 410. For example, the streaming server 420 may receive content in real time from the encoding server 410. Here, in order to provide a continuous streaming service, the streaming server 420 may store the bit stream for a predetermined time.

[0113] For example, the user device 450 may include a mobile phone, a smartphone, a laptop computer, a digital broadcast terminal, a personal digital assistant PDA, a portable media player PMP, a navigation terminal, a flat PC, a tablet PC, an ultrabook, a wearable device ( e.g. smart wristwatch), smart glasses, head-mounted display (HMD), digital TV, desktop computer and digital signage.

[0114] Each server in a content streaming system may operate as a distributed server, in which case data received from each server may be processed in a distributed manner.

[0115]

[0116] FIG. 5 shows an example of a block diagram of a video signal processing apparatus according to an embodiment of the invention. The video signal processing apparatus may correspond to the encoding apparatus 100 in FIG. 2 or decoder 200 in FIG. 3.

[0117] A video signal processing apparatus 500 in accordance with an embodiment of the invention may include a memory 520 for storing a video signal, and a processor 510 for processing a video signal, while being combined with the memory 520.

[0118] The processor 510 in accordance with an embodiment of the invention may be configured as at least one processing circuit for processing a video signal, and may process the video signal by executing instructions for encoding or decoding the video signal. Those. processor 510 may encode the original video signal or decode the encoded video signal by executing the encoding or decoding methods described below.

[0119]

[0120] FIG. 6 is an example of a block crushing structure according to an embodiment of the invention, FIG. 6a to 6d illustrate examples of block splitting structures according to quadtree (QT), binary tree (BT), ternary tree (TT), and asymmetric tree (AT).

[0121] When encoding video, one block may be split based on QT. Also, one sub-block split by QT can be further split recursively using QT. A sheet block that is no longer crushable according to QT may be crushed by at least one of BT, TT or AT. BT can have two types of crushing: horizontal BT (2N×N, 2N×N) and vertical BT (N×2N, N×2N). TT can have two types of crushing: horizontal TT (2N×1/2N, 2N×N, 2N×1/2N) and vertical TT (1/2N×2N, N×2N, 1/2N×2N). AT can have four crushing types: horizontal top AT (2N×1/2N, 2N×3/2N), horizontal bottom AT (2N×3/2N, 2N×1/2N), vertical left AT (1/2N×2N , 3/2N×2N), vertical right AT (3/2N×2N, 1/2N×2N). Each of BT, TT, AT can be further recursively split using BT, TT, AT.

[0122] FIG. 6A shows an example QT partition. Block A can be divided into four sub-blocks A0, A1, A2 and A3 by QT. Subblock A1 can be divided into four subblocks B0, B1, B2 and B3 again by QT.

[0123] FIG. 6B shows an example of BT partitioning. Block B3, which is no longer split by QT, can be split into a vertical BT (C0, C1) or a horizontal BT (D0, D1). Like block C0, each subblock can be further recursively subdivided into the form of horizontal BT (E0, E1) or vertical BT (F0, F1).

[0124] FIG. 6C shows an example of TT partitioning. Block B3, which is no longer split by QT, can be split into a vertical TT (C0, C1, C3) or a horizontal TT (D0, D1, D2). Like block C1, each subblock can be further recursively subdivided into the form of a horizontal TT (E0, E1, E2) or a vertical TT (F0, F1, F2).

[0125] FIG. 6D shows an example of AT partitioning. Block B3, which is no longer split by QT, can be split into a vertical AT (C0, C1) or a horizontal AT (D0, D1). Like block C1, each subblock can be further recursively subdivided into the form of horizontal AT (E0, E1) or vertical TT (F0, F1).

[0126] Meanwhile, BT, TT and AT partitions can be combined. For example, a subblock fragmented by BT may be fragmented by TT or AT. In addition, a subblock fragmented by TT may be fragmented by BT or AT. A subblock split by AT may be split by BT or TT. For example, after horizontal BT splitting, each sub-block can be split into a vertical BT, or after vertical BT splitting, each sub-block can be split into a horizontal BT. The two types of crushing methods have different crushing order, but the final crushing form is the same.

[0127] In addition, when a block is split, the order in which the block is searched may be determined in different ways. In general, the search operation is performed from left to right and top to bottom. Block searching can mean a order of determining whether to further split each subblock, or a coding order of corresponding subblocks when the block is no longer split, or a search order where subblocks refer to information from other adjacent blocks.

[0128]

[0129] FIG. 7 and 8 are respectively an inter-frame prediction-based video/image encoding procedure and an inter-frame prediction means in an encoding apparatus according to an embodiment of the invention.

[0130] The encoding device 100 performs inter-prediction of the current block (S710). The encoding device 100 can extract inter-prediction mode and motion information of the current block, and can generate prediction samples of the current block. In this case, the procedures for determining the inter-frame prediction mode, extracting motion information and generating a prediction sample can be performed simultaneously, and any of the procedures can be performed before the other procedure. For example, the interframe prediction means 180 of the encoding device 100 may include a prediction mode determining unit 181, a motion information extracting unit 182, or a prediction sample extracting unit 183. The prediction mode determining unit 181 may determine the prediction mode for the current block. The motion information extracting unit 182 can extract motion information of the current block. The prediction sample extractor 183 may extract prediction samples of the current block. For example, the inter-picture predictor 180 of the encoding device 100 can search for a predetermined area (search area) in reference pictures for a block similar to the current block by motion estimation, and can extract a reference block with a minimum difference or a difference of a predetermined reference or less with respect to the current block. The inter-picture predictor 180 may extract a reference picture index indicating a reference picture in which a reference block is located based on the reference block, and may extract a motion vector based on a difference in location between the reference block and the current block. The encoding device 100 may determine a mode that is applied to the current block from various prediction modes. The encoder may compare RD costs for different prediction modes, and may determine the optimal prediction mode for the current block.

[0131] For example, if a skip mode or a merge mode is applied to a current block, the encoder 100 may configure a list of merge candidates that will be described later, and may extract a reference block with a minimum difference or a difference of a given reference or less with respect to the current block among the reference blocks specified by merge candidates that are included in the list of merge candidates. In this case, a merge candidate associated with the extracted reference block may be selected. Merge index information indicating the selected merge candidate may be generated and signaled to the decoding device 200. The movement information of the current block can be extracted using the movement information of the selected merge candidate.

[0132] For example, if the (A)MVP mode is applied to the current block, then the encoder may configure a list of (A)MVP candidates that will be described later, and may use the motion vector of a motion vector predictor (mvp) candidate selected among the candidates to mvp included in the (A)MVP candidate list as the mvp of the current block. Here, for example, a motion vector indicating a reference block extracted by motion estimation can be used as a motion vector of the current block. The mvp candidate including the motion vector with the smallest difference with respect to the motion vector of the current block among the mvp candidates may become the selected mvp candidate. Motion vector difference (MVD) can be extracted, i.e. the difference obtained by subtracting mvp from the current block's motion vector. In this case, the MVD information may be signaled to the decoding device 100. In addition, if the (A)MVP mode is used, the reference picture index value can be configured as the reference picture index information and can be separately signaled to the decoding device.

[0133] The encoding device 100 may extract residual samples based on the prediction samples (S720). The encoder 100 may extract residual samples by comparing the original samples of the current block and the prediction samples.

[0134] The encoding device 100 encodes image information including prediction information and residual information (S730). The encoding apparatus may output encoded image information in the form of a bit stream. The prediction information may include information regarding prediction mode information (eg, a skip flag, a merge flag, or a mode index) and motion information as information associated with a prediction procedure. Information associated with the motion information may include candidate selection information (eg, merge index, mvp flag, or mvp index), i.e. information to extract the motion vector. In addition, information related to the motion information may include information regarding the MVD and/or information about the index of the reference picture. In addition, information associated with the motion information may include information indicating whether L0 prediction, L1 prediction, or bi-prediction is applied. Residual information is information regarding residual samples. The residual information may include information regarding quantized transform coefficients for the residual samples.

[0135] The output bit stream may be stored on a (digital) storage medium and transmitted to a decoding device, or may be transmitted to the decoding device via a network.

[0136] Meanwhile, as described above, the encoding device can generate a reconstructed picture (including reconstructed samples and a reconstructed block) based on the reference samples and residual samples. This serves to extract, in the encoding device 100, the same prediction results as those performed in the decoding device 200. Accordingly, coding efficiency can be improved. Accordingly, the encoding apparatus 100 may store the reconstructed picture (or the reconstructed samples and the reconstructed block) in memory, and may use the reconstructed picture as a reference picture for inter-frame prediction. As described above, the in-loop filtering procedure can be additionally applied to the reconstructed image.

[0137]

[0138] FIG. 9 and 10 are respectively an inter-frame prediction-based video/image decoding procedure and an inter-frame prediction means in a decoding apparatus according to an embodiment of the invention.

[0139] The decoding device 200 may perform an operation corresponding to an operation that is performed in the encoding device 100. The decoding apparatus 200 may perform prediction regarding the current block based on the received prediction information, and may extract prediction samples.

[0140] Specifically, the decoding apparatus 200 may determine the prediction mode for the current block based on the received prediction information (S910). The decoding apparatus 200 can determine which inter-frame prediction mode is applied to the current block based on the prediction mode information in the prediction information.

[0141] For example, decoding apparatus 200 may determine whether a merge mode or an (A)MVP mode is applied to the current block based on the merge flag. Alternatively, decoding apparatus 200 may select one of various inter-prediction mode candidates based on the mode index. Inter prediction mode candidates may include a skip mode, a merge mode, and/or an (A)MVP mode, and may include various inter prediction modes that will be described later.

[0142] The decoding apparatus 200 extracts motion information of the current block based on the determined inter-frame prediction mode (S920). For example, if a skip mode or a merge mode is applied to the current block, the decoding apparatus 200 may configure a merge candidate list, which will be described later, and select one of the merge candidates included in the merge candidate list. The selection of a merger candidate can be done based on the merger index. The movement information of the current block can be extracted from the movement information of the selected merge candidate. The movement information of the selected merge candidate can be used as the movement information of the current block.

[0143] As another example, if the (A)MVP mode is applied to the current block, then the decoding apparatus 200 may configure a list of (A)MVP candidates that will be described later, and may use the motion vector of a motion vector predictor (mvp) candidate. , selected from among the mvp candidates included in the (A)MVP candidate list, as the mvp of the current block. The selection can be made based on the selection information (mvp flag or mvp index). In this case, the decoding apparatus 200 may extract the MVD of the current block based on the information regarding the MVD. The decoding apparatus may extract the motion vector of the current block based on the mvp of the current block and the MVD. In addition, the decoding apparatus may extract the reference picture index of the current block based on the reference picture index information. The picture indicated by the reference picture index in the reference picture list with respect to the current block may be retrieved as a reference picture referred to for inter-picture prediction of the current block.

[0144] Meanwhile, as will be described later, movement information of the current block can be retrieved without configuring the candidate list. In this case, information about the movement of the current block can be extracted in accordance with the procedure disclosed in the prediction mode, which will be described later. In this case, a candidate list configuration such as the one described above may be omitted.

[0145] The decoding apparatus 200 may generate prediction samples for the current block based on motion information of the current block (S930). Here, the decoding apparatus 200 can extract the reference picture based on the reference picture index of the current block, and can extract the prediction samples of the current block indicated with respect to the reference picture by the motion vector of the current block. In this case, as will be described later, the prediction sample filtering procedure may be further performed on some or all of the prediction samples of the current block according to circumstances.

[0146] For example, the interframe predictor 260 of the decoding device 200 may include a prediction mode determining unit 261, a motion information extracting unit 262, and a prediction sample extracting unit 263. The decoding device 200 can determine the prediction mode of the current block based on the prediction mode information received from the prediction mode determining unit 261, can extract motion information (motion vector and/or reference picture index) of the current block based on information related to the motion information , received from the motion information extraction unit 262. The prediction sample extractor 263 may extract prediction samples of the current block.

[0147] The decoding apparatus 200 generates residual samples for the current block based on the received residual information (S940). The decoding apparatus 200 can generate reconstructed samples for the current block based on the prediction samples and residual samples, and can generate a reconstructed picture based on the reconstructed samples (S950). Then, as described above, the in-loop filtering procedure can be further applied to the reconstructed image.

[0148] As described above, the inter-prediction procedure may include a step of determining an inter-prediction mode, a step of extracting motion information according to a determined prediction mode, and a step of executing the prediction (generating a prediction sample) based on the extracted motion information.

[0149]

[0150] Inter-prediction mode determination

[0151] Various inter-frame prediction modes can be used to predict the current block in a picture. For example, various modes such as merge mode, skip mode, MVP mode and affine mode can be used. Decoder side motion vector refinement (DMVR) mode, adaptive motion vector resolution (AMVR) mode, etc. can be additionally used as additional modes. The affine mode may be referred to as the affine motion prediction mode. The MVP mode may be referred to as the Advanced Motion Vector Prediction (AMVP) mode.

[0152] Prediction mode information indicating the inter-prediction mode of the current block may be signaled from the encoding device to the decoding device. The prediction mode information may be included in the bit stream and received by the decoding device. The prediction mode information may include index information indicating one of several candidate modes. Alternatively, the inter-prediction mode may be indicated by hierarchical flag information signaling. Here, the prediction mode information may include one or more flags. For example, the flag may be further signaled to indicate whether the skip mode is applied, by signaling a skip flag to indicate whether the merge mode is applied, by signaling a merge flag if the skip mode is not applied, and to indicate that the MVP mode is applied. , if merge mode is not applied or for additional indication. The affine mode can be signaled as an independent mode and can be signaled as a mode dependent on a merge mode or an MVP mode. For example, the affine mode may be configured as one of a merge candidate list or an MVP candidate list, as will be described later.

[0153]

[0154] Extracting motion information according to inter-frame prediction mode

[0155] The encoding apparatus 100 or the decoding apparatus 200 may perform inter-frame prediction using information of the current block. The encoding device 100 can extract optimal motion information for the current block according to a motion estimation procedure. For example, encoder 100 may search for a reference block with a similar correlation using the source block in the source picture for the current block in fractional pixel units within a certain search range in the reference picture. Accordingly, the encoding device can extract motion information. Block similarity can be extracted based on the difference between the phase-based values of the samples. For example, block similarity can be calculated based on the SAD (Sum of Absolute Differences) between the current block (or current block template) and the reference block (or reference block template). In this case, motion information can be extracted based on the reference block with the smallest SAD within the search area. The extracted motion information may be signaled to the decoder using several methods based on the inter-frame prediction mode.

[0156]

[0157] Merge Mode and Skip Mode

[0158] If the merge mode is applied, the motion information of the current prediction block is not directly transmitted, and the motion information of the current prediction block is extracted using the motion information of the adjacent prediction block. Accordingly, the encoding device 100 may indicate movement information of the current prediction block by transmitting flag information to notify that a merging mode has been used and a merging index to notify which neighboring prediction block has been used.

[0159] The encoding apparatus 100 must search for a merging candidate block used to extract motion information of the current prediction block in order to perform a merging mode. For example, a maximum of 5 merge candidate blocks may be used, but the invention is not limited to this. In addition, a maximum number of merge candidate blocks can be carried in the slice header, and the invention is not limited to this. After searching for merge candidate blocks, the encoder 100 may generate a merge candidate list, and may select the merge candidate block with the lowest cost among the merge candidate blocks as the final merge candidate block.

[0160] An embodiment of the invention provides various embodiments of merge candidate blocks creating a list of merge candidates.

[0161] The merge candidate list may use 5 merge candidate blocks, for example. For example, 4 spatial merge candidates and 1 temporal merge candidate could be used.

[0162]

[0163] FIG. 11 illustrates an example of creating spatial merge candidates for the current block.

[0164] Referring to FIG. 11, at least one of the left neighbor block A1, the bottom left neighbor block A2, the top right neighbor block B0, the top neighbor block B1, and the top left neighbor B2 can be used to predict the current block. The merge candidate list for the current block can be configured based on the procedure shown in FIG. 12.

[0165]

[0166] FIG. 12 is a flowchart illustrating a method for configuring a merge candidate list in accordance with an embodiment to which the invention is applied.

[0167] The encoding device (encoding device 100 or decoding device 200) searches for spatial neighbors of the current block and inserts the retrieved spatial merge candidates into the merge candidate list (S1210). For example, spatial neighbors may include a lower left corner neighbor, a left neighbor, an upper right corner neighbor, an upper neighbor, and a top left corner neighbor of the current block. Here, this is an example, and additional neighboring blocks such as a right neighboring block, a lower neighboring block and a lower right neighboring block, in addition to spatial neighboring blocks, can be further used as spatial neighboring blocks. The encoder can detect available blocks by searching for spatial neighboring blocks based on priority, and can extract motion information of the detected blocks as spatial candidates for merging. For example, the encoding device 100 or the decoding device 200 may search the 5 blocks illustrated in FIG. 11 in the sequence A1, B1, B0, A0 and B2, and can configure the list of merge candidates by sequentially indexing the available candidates.

[0168] The encoding device searches for a temporary neighbor block of the current block and inserts the retrieved temporary merge candidate into the merge candidate list (S1220). The temporary neighboring block may be located in the reference picture, i.e. a picture different from the current picture in which the current block is located. The reference picture in which the temporary adjacent block is located may be called a co-located picture or col picture. The search for a temporary neighbor block can be carried out in the sequence of the bottom right corner neighbor block and the bottom right center block of the co-located block of the current block in the col picture. Meanwhile, if motion data compression is applied, specific motion information can be stored in a col picture as representative motion information for each given storage unit. In this case, it is not necessary to store motion information for all blocks in a given storage unit, and thus a motion data compression result can be obtained. Here, the predetermined storage unit may be predetermined as a 16×16 sample unit or an 8×8 sample unit, for example, or size information for the predetermined storage unit may be signaled from the encoding device 100 to the decoding device 200. If motion data compression is applied, the motion information of the temporary neighbor block may be replaced by representative motion information of a given storage unit in which the temporary neighbor block is located. Those. in this case, in an implementation aspect, after an arithmetic right shift is performed by a specified value based on the coordinates (upper left sample positions) of the temporary neighbor block, rather than the prediction block in which the coordinates of the temporary neighbor block are located, the temporary merge candidate may be extracted based on the motion information of a prediction block that covers an arithmetic left-shifted location. For example, if a given storage unit corresponds to a 2n×2n sample unit, assuming that the coordinates of the temporary neighboring block correspond to (xTnb, yTnb), then the motion information of the prediction block located at ((xTnb>>n)<<n), (yTnb >>n)<<n)), i.e. modified location, can be used for a temporary merge candidate. In particular, for example, if a given storage unit corresponds to a 16x16 sample unit, assuming that the coordinates of the temporary neighboring block correspond to (xTnb, yTnb), then the motion information of the prediction block located at ((xTnb>>4)<<4) , (yTnb>>4)<<4)), i.e. modified location, can be used for a temporary merge candidate. Alternatively, for example, if a given storage unit corresponds to an 8x8 sample unit, assuming that the coordinates of the temporary neighboring block correspond to (xTnb, yTnb), then the motion information of the prediction block located at ((xTnb>>3)<<3 ), (yTnb>>3)<<3)), i.e. modified location, can be used for a temporary merge candidate.

[0169] The encoding device may check whether the current number of merge candidates is less than the maximum number of merge candidates (S1230). The maximum number of merge candidates may be predetermined or may be signaled from the encoding apparatus 100 to the decoding apparatus 200. For example, the encoding apparatus 100 may generate information regarding the maximum number of merge candidates, may encode the information, and may transmit the information to the decoding apparatus 200 in the form of a bit stream. If the maximum number of merge candidates is full, the process of adding a candidate may not occur.

[0170] If, as a result of the check, the current number of merge candidates is less than the maximum number of merge candidates, the encoding device inserts the added merge candidate into the list of merge candidates (S1240). The added merge candidate may include an ATMVP (Adaptive Temporal Motion Vector Prediction), a combined bi-predictive merge candidate (if the slice type of the current slice is type B), and/or a null vector merge candidate, for example.

[0171]

[0172] FIG. 13 is a flowchart illustrating a method for configuring a prediction candidate list (MVP candidate list) in accordance with an embodiment to which the invention is applied.

[0173] If a motion vector prediction (MVP) mode is used, a motion vector predictor (mvp) candidate list may be generated based on the motion vector of the reconstructed spatial neighbor block (e.g., the neighbor block described in FIG. 11) and/or the vector movement corresponding to a temporary adjacent block (or Col block). That is, the motion vector of the reconstructed spatial neighbor block and/or the motion vector of the temporal neighbor block can be used as a motion vector predictor candidate. The prediction information may include selection information (eg, an MVP flag or an MVP index) indicating the optimal motion vector predictor candidate selected among the motion vector predictor candidates included in the list. Here, the prediction means may select a motion vector predictor of the current block among motion vector predictor candidates included in the motion vector candidate list using the selection information. The predictor of the encoding device 100 may calculate a motion vector difference (MVD) between the motion vector of the current block and the motion vector predictor, may encode the MVD, and may output the encoded MVD in the form of a bitstream. Those. MVD can be calculated as the value obtained by subtracting the motion vector predictor from the motion vector of the current block. Here, the predictor means of the decoding device can obtain the motion vector difference included in the prediction information, and can extract the motion vector of the current block by adding the motion vector difference and the motion vector predictor. The prediction means of the decoding device may obtain or extract a reference picture index indicating the reference picture from the prediction information. For example, the motion vector predictor candidate list may be configured as illustrated in FIG. 13.

[0174] Referring to FIG. 13, the encoding device searches for a spatial block candidate for motion vector prediction and inserts it into the prediction candidate list (S1310). For example, the encoding apparatus may search neighboring blocks according to a predetermined search order, and add information of a neighboring block that satisfies the condition for a spatial block candidate to a prediction candidate list (MVP candidate list).

[0175] After generating the list of spatial candidate blocks, the encoding device compares the number of spatial candidates included in the prediction candidate list with a preset reference number (eg, 2) (S1320). If the number of spatial candidates included in the prediction candidate list is greater than or equal to the reference number (eg, 2), the encoder may finish creating the prediction candidate list.

[0176] But, if the number of spatial candidates included in the prediction candidate list is less than the reference number (for example, 2), the encoding device searches for a candidate time block and inserts it into the prediction candidate list (S1330), and when the time block -candidate not available, adds a zero motion vector to the list of prediction candidates (S1340).

[0177]

[0178] Prediction sample generation

[0179] A predicted block for a current block may be extracted based on motion information extracted according to the prediction mode. The predicted block may include prediction samples (an array of prediction samples) of the current block. When the motion vector of the current block indicates the fractional unit of the sample, an interpolation procedure can be performed, and through this, prediction samples of the current block can be extracted based on the reference samples in the fractional unit of the sample in the reference picture. When affine inter-frame prediction is applied to the current block, prediction samples can be generated based on the motion vector in the sample/sub-block unit. When bi-directional prediction is applied, the resulting prediction samples can be extracted by weighted (according to phase) sums of the prediction samples extracted based on the first prediction direction (eg, L0 prediction) and the prediction samples extracted based on the second prediction direction. Reconstructed samples and reconstructed pictures can be generated based on the extracted prediction samples, and as described above, a procedure such as in-loop filtering can subsequently be performed.

[0180]

[0181] Affine motion prediction

[0182] FIG. 14 illustrates an example of motion patterns in accordance with an embodiment.

[0183] In conventional image compression technology (eg, high efficiency video coding (HEVC)), a single motion vector is used to represent the motion of a coding block. Although the optimal motion of a block unit can be represented using a method using one motion vector for each block, this may not actually be the optimal motion of each picture element. Accordingly, if the optimal motion vector is determined in a picture element unit, encoding efficiency can be improved. Accordingly, an embodiment of the invention describes a method for predicting motion encoding or decoding a video signal using a multi-motion model. Specifically, a motion vector may be represented in each picture element unit of a block or sub-block unit using the motion vector at control points 2 to 4. A prediction scheme using the motion vector of a plurality of control points may be referred to as affine motion prediction or affine prediction.

[0184] The affine motion model according to an embodiment of the invention may be 4 motion models such as those illustrated in FIG. 14. An affine motion model to represent three motions (parallel translation, scaling and rotation) among the motions that can be represented by an affine motion model is called a similarity (or simplified) affine motion model. When describing embodiments, the affine similarity motion model (or simplified one) is mainly described for convenience of description, but the invention is not limited to this.

[0185]

[0186] FIG. 15 illustrates an example of a control point motion vector for affine motion prediction in accordance with an embodiment of the invention.

[0187] As shown in FIG. 15, in affine motion prediction, the motion vector of a picture element (or sub-block) location included in a block can be determined using a pair of two control point motion vectors (CPMVs) v_0 and v_1. In this case, the set of motion vectors may be referred to as an affine motion vector field (MVF). In this case, the affine field of motion vectors can be defined using Equation 1.

[0188] [Equation 1]

[0189] In Equation 1, v_0(v_0={v_0x, v_0y}) indicates the motion vector CPMV0 at the first control point of the upper left location of the current block 1500. v_1(v_1={v_1x, v_1y}) indicates the motion vector CPMV1 at the second control point the top-right location of the current block 1500. Additionally, w indicates the width of the current block 1500. v(v={v_x, v_y}) indicates the motion vector at location {x, y}. The motion vector of a subblock unit (or picture element) can be extracted using Equation 1. In one embodiment, the motion vector precision can be rounded to a precision of 1/16.

[0190]

[0191] FIG. 16 illustrates an example of a motion vector for each sub-block of a block to which affine motion prediction has been applied in accordance with an embodiment of the invention.

[0192] Referring to FIG. 16, in the encoding or decoding process, an affine field of motion vectors may be defined in a picture element unit or a block unit. Those. in affine motion prediction, the motion vector of the current block can be extracted in a picture element unit or a subblock unit.

[0193] If the affine field of motion vectors is defined in the unit of a picture element, then the motion vector can be obtained based on each value of the picture element. In the case of a block unit, the motion vector of the corresponding block can be obtained based on the value of the central picture element of the block. Herein, it is assumed that an affine motion vector field (MVF) is defined in a 4x4 block unit as in FIG. 16. In this case, this is for convenience of description, and the embodiment of the invention is not limited to this. Fig. 16 illustrates an example of a case where a coding block is configured with 16x16 samples and an affine motion vector field (MVF) is defined in a block unit of size 4x4.

[0194] Affine motion prediction may include an affine merge mode (or AF_MERGE) and an affine interframe mode (or AF_INTER). The AF_INTER mode may include an AF_4_INTER mode using a 4-parameter motion model and an AF_6_INTER mode using a 6-parameter motion model.

[0195]

[0196] Affine Merge Mode

[0197] AF_MERGE determines a control point motion vector (CPMV) based on an affine motion model of a neighboring block encoded as an affine motion prediction. An adjacent block affine-coded in the search sequence can be used for AF_MERGE. When one or more neighboring blocks are encoded as an affine motion prediction, the current block may be encoded as AF_MERGE.

[0198] That is if affine merge mode is applied, the CPMVs of the current block can be extracted using the CPMVs of the neighboring block. Here, the CPMVs of the neighboring block can be used as the CPMVs of the current block without any modification, and the CPMVs of the neighboring block can be modified based on the size of the neighboring block and the size of the current block, and can be used as the CPMVs of the current block.

[0199] FIG. 17 illustrates an example of neighboring blocks that are used for affine motion prediction in an affine fusion mode in accordance with an embodiment of the invention.

[0200] In affine merge mode (AF_MERGE), the encoder can perform encoding according to the following process.

[0201] Step-1: Scan adjacent blocks A to E 1710, 1720, 1730, 1740 and 1750 of the current encoding block 1700 in an alphabet sequence, and determine the block first encoded in accordance with the affine prediction mode based on the scanning sequence as the block -affine fusion candidate (AF_MERGE)

[0202] Step-2: Determine an affine motion model using the control point motion vector (CPMV) of the determined candidate block

[0203] Step-3: Determine the control point motion vector (CPMV) of the current block 1700 based on the affine motion model of the candidate block and determine the MVF of the current block 1700

[0204]

[0205] FIG. 18 illustrates an example of a block on which affine motion prediction is performed using neighboring blocks to which affine motion prediction has been applied in accordance with an embodiment of the invention.

[0206] For example, as in FIG. 18, if block 1820 A has been encoded in accordance with the affine mode, then after block 1820 A is determined as a candidate block, an affine motion model can be extracted using control point motion vectors (CPMVs) (eg, v2 and v3 ) of block 1820 A, and the control point motion vectors (CPMV) v0 and v1 of the current block 1800 can be determined. An affine motion vector field (MVF) of the current block 1800 can be determined based on the control point motion vectors (CPMV) of the current block 1800, and coding can be done.

[0207]

[0208] FIG. 19 is a diagram for describing a method for generating a merge candidate list using an adjacent affine encoding block in accordance with an embodiment of the invention.

[0209] Referring to FIG. 19, if a CPMV pair is determined using an affinity merge candidate, then candidates such as those illustrated in FIG. 19. In FIG. 19, it is assumed that the scanning sequence of the candidate list has been set to A, B, C, D and E. In this case, the invention is not limited to this, and various sequences can be preset.

[0210] In an embodiment, if the number of candidates (hereinafter referred to as affine candidates) that are encoded according to an affine mode (or affine prediction) are available in adjacent blocks (i.e., A, B, C, D, E) matches 0, then the affine merge mode of the current block can be skipped. If the number of available affine candidates matches one (e.g., A), then the motion model of the corresponding candidate can be used to extract the control point motion vectors (CPMV_0 and CPMV_1) of the current block. In this case, the index indicating the corresponding candidate may not be required (or encoded). If the number of available affine candidates is two or more, then two candidates in the scan sequence can be configured as a candidate list for AF_MERGE. Here, candidate selection information such as an index indicating a candidate that is selected from the candidate list may be signaled. The selection information may be phage or index information, and may be referred to as AF_MERGE_flag or AF_merge_idx.

[0211] In an embodiment of the invention, motion compensation for the current block may be performed based on the size of the sub-block. In this case, the subblock size of the affine block (i.e. the current block) is retrieved. If each of the subblock width and height is greater than 4 chroma samples, then the motion vector for each subblock can be extracted, and DCT-IF-based motion compensation (1/16 pixel for luma and 1/32 for chrominance) can be performed on the subblock . If not, then motion compensation based on an improved bilinear interpolation filter can be performed on all affine blocks.

[0212] In an embodiment of the invention, if the merge/skip flag is true and both the width and height of the CU are greater than or equal to 8, at the CU level, the affine flag is signaled via a bitstream indicating whether the affine merge mode is used. When a CU is encoded as AF_MERGE, the merge candidate index with a maximum value of '5' is signaled to announce that the motion information candidate is used for the CU in the affine merge candidate list.

[0213]

[0214] FIG. 20 and 21 are diagrams for describing a method for creating an affine merge candidate list using a neighbor block encoded by affine prediction in accordance with an embodiment of the invention.

[0215] Referring to FIG. 20, the affine merge candidate list is configured through the following steps.

[0216] 1) Inserting a model-based affine candidate

[0217] A model-based affine candidate means that the candidate is extracted from a valid neighboring reconstructed block encoded according to an affine mode. As illustrated in FIG. 20, the scan sequence for a candidate block corresponds from left A, top B, top right C and bottom left D to top left E.

[0218] If the adjacent lower left block A is 6-parameter affine mode encoded, then the motion vectors (v_4, v_5, v_6) of the upper left corner, upper right corner and lower left corner of the CU including block A are obtained. Vectors the motions (v_0, v_1, v_2) of the top left corner of the current block are calculated based on the motion vectors (v_4, v_5 and v_6) according to a 6-parameter affine model.

[0219] If the adjacent lower left block A is 4-parameter affine mode encoded, then the upper left corner and upper right corner motion vectors (v_4, v_5) of the CU including block A are obtained. Motion vectors (v_0, v_1) the upper left corner of the current block are calculated based on the motion vectors (v_4, v_5) in accordance with the affine model with 4 parameters.

[0220] 2) Inserting control point-based affine candidates

[0221] Referring to FIG. 20, a control point-based candidate means that the candidate is configured by combining adjacent motion information of control points.

[0222] Control point movement information is first extracted from the assigned spatial neighbor blocks and the temporal neighbor block illustrated in FIG. 20. CP_k (k=1, 2, 3, 4) indicates the k-th control point. In addition, A, B, C, D, E, F and G are the spatial locations for the prediction of CP_k (k=1, 2, 3), and H is the temporal location for the prediction of CP4.

[0223] The coordinates of CP_1, CP_2, CP_3 and CP_4 are (0, 0), (W, 0), (H, 0) and (W, H), respectively. In this case, W and H are the width and height of the current block.

[0224] The movement information of each control point is obtained based on the next priority.

[0225] As for CP_1, the test priority is A→B→C and A is used if A is available. If unavailable and if B is available, then B is used. If both A and B are unavailable, then C is used. If all three candidates are unavailable, CP_1 traffic information cannot be obtained.

[0226] As for CP_2, the test priority is E→D.

[0227] As for CP_3, the check priority is G→F.

[0228] Regarding CP_4, H is used.

[0229] Second, combinations of control points are used to configure the motion pattern.

[0230] The motion vectors of the two control points are needed to calculate the transformation parameters in the 4-parameter affine model. The two control points can be selected from one of the following 6 combinations {CP_1, CP_4}, {CP_2, CP_3}, {CP_1, CP_2}, {CP_2, CP_4}, {CP_1, CP_3} and {CP_3, CP_4}. For example, to use control points CP_1 and CP_2 when creating a 4-parameter affine motion model, they are labeled "affine (CP_1, CP_2)".

[0231] The motion vectors of the three control points are needed to calculate the transformation parameters in the 6-parameter affine model. The three control points can be selected from one of the following 4 combinations {CP_1, CP_2, CP_4}, {CP_1, CP_2, CP_3}, {CP_2, CP_3, CP_4} and {CP_1, CP_3, CP_4}. For example, to use control points CP_1, CP_2 and CP_3 when creating a 6-parameter affine motion model, they are labeled as “affine (CP_1, CP_2, CP_3)”.

[0232] Moreover, in an embodiment of the invention, if an affine merge candidate is present in an affine merge mode, then it may always be considered a 6-parameter affine mode.

[0233]

[0234] Affine Interframe Mode

[0235] FIG. 22 illustrates an example of adjacent blocks that are used for affine prediction mode in affine interframe mode in accordance with an embodiment of the invention.

[0236] Referring to FIG. 22, affine motion prediction may include an affine merging mode (or AF_MERGE) and an affine interframe mode (or AF_INTER). In the affine interframe mode (AF_INTER), after the 2-prediction of control point motion vectors (CPMVP) and CPMV are determined, a control point motion vector difference (CPMVD) corresponding to the difference can be transmitted from the encoder to the decoder. The detailed affine interframe mode (AF_INTER) encoding process may be exactly as described below.

[0237] Step-1: Determine a candidate for a pair of two CPMVPs

[0238] Step 1.1: Determine a maximum of 12 CPMVP candidate combinations (refer to Equation 2)

[0239] [Equation 2]

[0240] In Equation 2, v_0 indicates the motion vector CPMV0 at the upper left control point 2210 of the current block 2200. v_1 indicates the motion vector CPMV1 at the upper right control point 2211 of the current block 2200. v_2 indicates the motion vector CPMV2 at the control point 2212 on the lower left side of the current block 2200. v_A indicates the motion vector of an adjacent block 2220 A that is adjacent to the top left control point 2210 of the current block 2200. v_B indicates the motion vector of an adjacent block 2222 B that is adjacent to the top left control point 2210 of the current block 2200. v_C indicates the motion vector of the adjacent block 2224 C, which is adjacent to the left of the upper left control point 2210 of the current block 2200. v_D indicates the motion vector of the adjacent block 2226 D, which is adjacent to the upper right control point 2211 of the current block 2200 v_E indicates the motion vector of an adjacent E block 2228 that is adjacent to the upper right control point 2211 of the current block 2200. v_F indicates the motion vector of an adjacent F block 2230 that is adjacent to the lower left control point 2212 of the current block 2200. v_G indicates the motion vector of the adjacent block 2232 G, which is adjacent to the left of the lower left control point 2212 of the current block 2200.

[0241] Step-1.2: Sort the candidate CPMVP combinations based on the value with a small difference value (DV) and use the two best candidates (refer to Equation 3 below).

[0242] [Equation 3]

[0243] v_0x indicates the motion vector x-axis element (V0 or CPMV0) at the upper left control point 2210 of the current block 2200. v_1x indicates the motion vector x-axis element (V1 or CPMV1) at the upper right control point 2211 of the current block 2200. v_2x indicates the element the x-axis motion vector (V_2 or CPMV_2) at the lower left control point 2212 of the current block 2200. v_0y indicates the y-axis element of the motion vector (V_0 or CPMV_0) at the upper left control point 2210 of the current block 2200. v_1y indicates the y-axis element of the motion vector (V_1 or CPMV_1) at the upper right control point 2211 of the current block 2200. v_2y indicates the y-axis element of the motion vector (V_2 or CPMV_2_ at the lower left control point 2212 of the current block 2200. w indicates the width of the current block 2200. h indicates the height of the current block 2200.

[0244] Step-2: Use the AMVP candidate list when the control point motion vector predictor pair (CPMVP) candidate is less than 2

[0245] Step-3: Determine the control point motion vector predictor (CPMVP) of each of the two candidates and optimally select the candidate with the smallest value by comparing the costs of RD and CPMV

[0246] Step-4: Transmit the index corresponding to the optimal candidate and the control point motion vector difference (CPMVD)

[0247] In an embodiment of the invention, a process for creating a CPMVP candidate in AF_INTER is provided. Identical to AMVP, the number of candidates is set to 2 and an index indicating the location in the candidate list is signaled.

[0248] The process for creating a CPMVP candidate list is as follows.

[0249] 1) Whether neighboring blocks are encoded as affine motion prediction is checked by scanning neighboring blocks. If the scanned blocks are encoded as affine prediction, a pair of motion vectors of the current block is extracted by the affine motion model of the scanned neighboring block until the number of candidates becomes 2.

[0250] 2) If the number of candidates is less than 2, the candidate configuration process is performed. Moreover, in an embodiment of the invention, a 4-parameter (2-control point) affine interframe mode is used to predict the zoom in/out motion pattern and rotation and content. As illustrated in FIG. 15, the affine field of block motion is described by two motion vectors of the control point.

[0251] The motion vector field (MVF) of a block is described by Equation 1 above.

[0252] In conventional technology, an advanced motion vector prediction (AMVP) mode is required to scan the motion vector prediction (MVP) index and motion vector difference (MVD). When AMVP mode is applied to an invention, an affine flag (affine_flag) is signaled to indicate whether affine prediction is used. If affine prediction is applied, the syntaxes inter_dir, ref_idx, mvp_index, and two MVDs (mvd_x and mvd_y) are signaled. A list of candidates for the MVP affine pair is generated, which includes two MVP affine pairs. The signaled mvp_index is used to select one of two MVP affine pairs. An MVP affine pair is formed by two types of affine MVP candidates. One is a spatial inherited affine candidate, and the other is a corner-extracted affine candidate. If neighboring CUs are encoded in affine mode, then spatial inherited affine candidates can be generated. The affine motion model of the adjacent affine-coded block is used to generate the motion vectors of a pair of MVPs with 2 control points. The MVs of MVP pairs with 2 spatial legacy affine candidate control points are extracted using the following equations.

[0253] [Equation 4]

V0x=VB0x + (VB2_x - VB0x) * (posCurCU_Y - posRefCU_Y) / RefCU_height + (VB1x - VB0x) * (posCurCU_X - posRefCU_X) / RefCU_width

[0254] [Equation 5]

V0y=VB0y + (VB2_y - VB0y) * (posCurCU_Y - posRefCU_Y) / RefCU_height + (VB1y - VB0y) * (posCurCU_X - posRefCU_X) / RefCU_width

[0255] If V_B0, V_B1 and V_B2 can be replaced by the top left MV, top right MV and bottom left MV of a given reference/neighboring CU, then (posCurCU_X, posCurCU_Y) are the location of the top left sample of the current CU for the top left sample of the frame. (posRefCU_X, posRefCU_Y) is the location of the top left sample of the reference/neighboring CU for the top left sample of the frame.

[0256] [Equation 6]

V1x=VB0x + (VB1x - VB0x) * CU_width/RefCU_width

[0257] [Equation 7]

V1y=VB0y + (VB1y - VB0y) * CU_width/RefCU_width

[0258] FIG. 23 illustrates an example of adjacent blocks that are used for affine motion prediction in an affine interframe mode in accordance with an embodiment of the invention.

[0259] Referring to FIG. 23, when the number of MVP pairs is less than 2, the affine candidate extracted from the corner is used. As illustrated in FIG. 23, the adjacent motion vectors are used to extract the affine pair MVP. Regarding the first affine candidate extracted from the corner, the first available MV in set A (A0, A1 and A2) and the first available MV in set B (B0 and B1) are used to configure the first pair of MVPs. As for the second corner-extracted affine candidate, the first available MV in set A and the first available MV in set C (C0 and C1) are used to calculate the MV of the top-right control point. The first available MV in set A and the computed MV of the top right control point are the second MVP pair.

[0260] In an embodiment of the invention, two candidate sets including two (three) candidates {mv_0, mv_1} ({mv_0, mv_1, mv_2) are used to predict two (three) control points of the affine motion model. The specified motion vector differences (mvd_0, mvd_1, mvd_2) and control points are calculated using the following equations.

[0261] [Equation 8]

[0262] FIG. 24 and 25 are diagrams illustrating a method for extracting a motion vector candidate using motion information of a neighboring block in an affine interframe mode in accordance with an embodiment of the invention.

[0263] In an affine candidate list, affine motion is propagated from spatial neighboring blocks (extrapolated affine candidates), and the affine candidate list is augmented with a combination of motion vectors from spatial neighboring blocks (virtual affine candidates). Candidate sets are established as follows:

[0264] 1. At most two different sets of affine MV predictors are extracted from the affine motion of neighboring blocks. Neighboring blocks A0, A1, B0, B1 and B2 are checked as illustrated in FIG. 24. If an adjacent block is encoded by an affine motion model and the corresponding reference frame is the same as the reference frame of the current block, then two control points (for a 4-parameter affine model) of the current block or three control points (for a 6-parameter affine model) parameters) of the current block are extracted from the affine model of neighboring blocks.

[0265] 2. FIG. 25 illustrates the adjacent blocks that are used to form a set of virtual affine candidates. Neighboring MVs are divided into three groups: S_0={mv_A, mv_B, mv_C}, S_1={mv_D, mv_E} and S_2={mv_F, mv_G}. mv_0 is the first MV that references the same reference picture as the one of the current block in S0. mv_2 is the first MV that references the same reference picture as the one of the current block in S1.

[0266] If mv_0 and mv_1 are given, then mv_2 can be retrieved by Equation 9 below.

[0267] [Equation 9]

[0268] In Equation 9, the size of the current block corresponds to W×H.

[0269] If only mv_0 and mv_2 are specified, then mv_1 can be extracted by Equation 10 below.

[0270] [Equation 10]

[0271] In an embodiment of the invention, affine inter-frame prediction can be performed in accordance with the following sequence.

[0272] Input: affine motion parameters, reference picture samples

[0273] Output: CU prediction block

[0274] Processor

[0275] Retrieves the subblock size of the affine block

[0276] - If both the width and height of the subblock are greater than 4 luminance samples,

[0277] -- For each subblock,

[0278] - Extract the motion vector of the subblock

[0279] - Perform motion compensation (1/16 image unit for luma and 1/32 image unit for chrominance) based on DCT-IF over subblocks (called)

[0280] - If not, then compensation is performed (called) based on the improved bilinear interpolation filter on all affine blocks

[0281] Moreover, in an embodiment of the invention, if the merge/skip flag is false and the CU width and height are greater than or equal to 8, then the affine flag is signaled to indicate whether the affine interframe mode at the CU level will be used. If the CU is encoded in affine interframe mode, then a model flag is signaled to indicate whether a 4-parameter or 6-parameter affine model is applied to the CU. If the model flag is true, then the AF_6_INTER mode (affine model with 6 parameters) is applied and the MVDs are analyzed. If not, then AF_4_INTER mode (affine model with 4 parameters) is applied and two MVDs are analyzed.

[0282] In the AF_4_INTER mode, like the affine merge mode, pairs of motion vectors extrapolated from adjacent blocks encoded by the affine mode are generated and preferably inserted into the candidate list.

[0283] Then, if the candidate list size is less than 4, then candidates with motion vector pair {(v_0,v_1)|v0={v_A, v_B, v_c},v_1={v_D, v_E}} are generated using adjacent blocks. As illustrated in FIG. 25, v_0 is selected from the motion vectors of blocks A, B, and C. The motion vector from the adjacent block is scaled based on the relationship between the reference list, the POC for the reference to the adjacent block, the POC for the reference to the current CU, and the current CU. Moreover, the approach method for selecting v_1 from neighboring blocks D and E is similar. When the candidate list is larger than 4, the candidates are first sorted based on the relatedness of adjacent motion vectors (similar to two motion vectors in a candidate pair), and the first 4 candidates are kept.

[0284] If the number of candidate lists is less than 4, then the list is populated with a pair of motion vectors by duplicating AMVP candidates.

[0285] In the AF_6_INTER mode, similar to the affine merge mode, motion vector triplets (affine motion vector triplets) are generated which are extrapolated from adjacent blocks encoded in the affine merge mode and preferentially inserted into the candidate list.

[0286] Thereafter, when the size of the candidate list is less than 4, the candidates including motion vector triplets {(v_0, v_1, v_2)| v0={v_A, v_B, v_c}, v1={v_D, v_E}, v2={v_G, v_H}} are formed using neighboring blocks. As illustrated in FIG. 25, v_0 is selected from the motion vectors of block A, B, or C. The motion vector from the adjacent block is scaled based on the relationship between the reference list, the POC for the reference to the adjacent block, the POC for the reference to the current CU, and the current CU. Moreover, the approach for selecting v_1 from neighboring blocks D and E is similar to selecting v_2 from F and G. When the candidate list is larger than 4, the candidates are sorted based on the relatedness of adjacent motion vectors (similar to two motion vectors in three candidates), and the first 4 candidates are saved.

[0287] When the number of candidate lists is less than 4, the list can be populated with motion vector triplets configured by duplicating the corresponding AMVP candidates.

[0288] After the CPMV of the current CU is extracted, the MVF of the current CU is generated in accordance with Equation 11 for a 4-parameter affine model and is generated in accordance with Equation 12 for a 6-parameter affine model, based on the number of affine parameters.

[0289] [Equation 11]

[0290] [Equation 12]

[0291] In this case, the subblock size M×N is extracted in Equation 13 and MvPre is the precision of the motion vector portion (1/16).

[0292] [Equation 13]

[0293] Once extracted by Equation 12, M and N must be adjusted downward, if necessary, to make them a divisor of w and h. When M or N is less than 8, WIF is applied. If not, then affine motion compensation is applied based on the subblock.

[0294] FIG. 26 illustrates an example of an affine field of motion vectors of a subblock unit in accordance with an embodiment of the invention.

[0295] Referring to FIG. 26, in order to extract the motion vector of each M×N sub-block, the motion vector of the center sample of each sub-block such as that illustrated in FIG. 26, calculated according to Equation 11 or Equation 12, and rounded to 1/16ths. SHVC upsampling interpolation filters are used to generate a prediction of each sub-block using the extracted motion vector.

[0296] SHVC upsampling interpolation filters with exactly the same filter length and normalization factor as HEVC motion compensation interpolation filters can be used as motion compensation interpolation filters for additional fractional image unit positions. The motion vector precision of the chrominance component is 1/32 of a sample. Additional interpolation filters for 1/32 unit locations are extracted using the average values at two adjacent 1/16 unit locations.

[0297] The AF_MERGE mode can be selected on the encoder side using the same general merge mode selection method. Preferably, a list of candidates is generated and a minimum RD cost is selected among the candidates for comparison with the RD costs of other interframe modes. The result of the comparison is to determine whether AF_MERGE is applied or not.

[0298] With respect to the AF_4_INTER mode, the RD cost check is used to determine whether a motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) of the current CU. Once the CPMVP of the current affine CU is determined, the affine motion estimate is applied and the control point motion vector (CPMV) is obtained. Accordingly, the difference between CPMV and CPMVP is determined.

[0299] On the encoder side, the AF_6_INTER mode is identified only when the AF_MERGE or AF_4_INTER mode is determined as the optimal mode in the previous mode selection stage.

[0300] In an embodiment, affine interframe (affine AMVP) mode can be implemented as follows:

[0301] 1) AFFINE_MERGE_IMPROVE: Instead of searching for the first neighboring block in affine mode, the improvement is to search for the neighboring block with the maximum encoding unit size as an affine merge candidate.

[0302] 2) AFFINE_AMVL_IMPROVE: Adjacent blocks in affine mode are added to the affine AMVP candidate list similar to the general AMVP procedure.

[0303] The detailed process for generating the AMVP affine candidate list is as follows.

[0304] First, it is identified whether the neighboring block on the bottom left uses an affine motion model and has the same reference index as the current reference index. If a neighboring block is not present, the left neighboring block is identified using the same method. If a neighbor block is not present, it is identified whether the bottom left neighbor block uses an affine motion model and has a different reference index. If an adjacent block is present, the scaled affine motion vector is added to the list of reference pictures. If a neighboring block is not present, the left neighboring block is identified using the same method.

[0305] Secondly, the upper right neighboring block, the upper neighboring block and the upper left neighboring block are identified using the same method.

[0306] After the processes, if two candidates are found, then the AMVP candidate affine list generation process is completed. If two candidates are not found, an initial operation within the JEM software is performed to generate AMVP affine candidate lists.

[0307] 3) AFFINE_SIX_PARAM: In addition to the 4-parameter affine motion model, the 6-parameter affine motion model is added as an additional model.

[0308] A 6-parameter affine motion model is extracted via Equation 14.

[0309] [Equation 14]

[0310] Three motion vectors at the top left location MV_0, top right location MV_1 and bottom left location MV_2 are needed to define the model because 6 parameters are present in the motion model. Three motion vectors can be determined using a method similar to that of two motion vectors in a 4-parameter affine motion model. Affine model fusion is always set as a 6-parameter affine motion model.

[0311] 4) AFFINE_CLIP_REMOVE: Removes motion vector constraints for all affine motion vectors. The motion compensation process is designed to control the motion vector constraints themselves.

[0312]

[0313] Affine motion model

[0314] As described above, various affine motion models may be used or considered in affine inter-frame prediction. For example, the affine motion model may be four motions as in FIG. 14. An affine motion model capable of representing three motions (parallel translation, scaling and rotation), among the motions that can be represented by an affine motion model, can be called a similarity (or simplified) affine motion model. The number of CPMVs and/or the method of extracting the MV of a sample unit/subblock from the current block may be different depending on which one of the affine motion models is used.

[0315] In an embodiment of the invention, adaptive motion models with four or six parameters are used. In AF_INTER, a 6-parameter motion model is proposed in addition to the 4-parameter motion model present in JEM. The 6-parameter affine motion model is described similarly to Equation 15.

[0316] [Equation 15]

x’ = a * x+b * y+c

y’ = d * x+e * y+f

[0317] In this case, the coefficients a, b, c, d, e and f are affine motion parameters. (x, y) and (x',y') are the coordinates at the pixel locations before and after the affine motion model transformation. In video coding, in order to use the affine motion model, if CPMV0, CPMV1 and CPMV2 are the MVs for CP0 (top left), CP1 (top right) and CP2 (bottom left), then Equation 16 can be described as follows.

[0318] [Equation 16]

[0319] In this case, CPMV_0={v_0x, v_0y}, CPMV_1={v_1x, v_1y}, CPMV_2={v_2x, v_2y}, and w and h are the width and height of each encoding block. Equation 16 is the motion vector field (MVF) of the block.

[0320] The flag is parsed at the CU level to indicate whether a 4-parameter or 6-parameter affine motion model is used when adjacent blocks are encoded as an affine prediction. If a neighboring block coded as an affine prediction is not present, then the flag is omitted and the 4-parameter model is used for the affine prediction. In other words, a 6-parameter model is considered under the condition that one or more neighboring blocks are encoded into an affine motion model. In terms of the number of CPMVDs, each of two or three CPMVDs is signaled with respect to 4-parameter and 6-parameter affine motion models.

[0321] Additionally, in an embodiment of the invention, motion vector refinement with matching to a sample may be used. In sample-matched motion vector extraction (PMMVD, abbreviated as PMVD in the JEM encoder description) from JEM, the decoder needs to estimate some motion vectors (MVs) in order to determine an initial MV candidate for CU-level retrieval. When searching at the sub-CU level, some MV candidates are added in addition to the optimal CU-level MV. The decoder needs to evaluate such MV candidates in order to search for the optimal MV. This requires a lot of memory bandwidth. The proposed sample-matched motion vector refinement (PMVR) uses the concepts of JEM template matching and two-way matching in PMVD. When skip mode or merge mode is selected to indicate whether PMVR is available, a single PMVR_flag is signaled. In order to significantly reduce the memory bandwidth requirement compared to PMVD, a list of MV candidates is generated. If PMVR is used, the index of the initial MV candidate is explicitly signaled.

[0322] The candidate list is generated using the merge candidate list generation process, but sub-CU merge candidates, such as affine candidates and ATMVP candidates, are excluded. For two-way matching, only mono-prediction MV candidates are included. Bi-prediction MV candidates are divided into two mono-prediction MV candidates. In addition, similar MV candidates (with MV differences less than a predefined threshold) are also removed. When applied to CU-level search, MV refinement with diamond search is performed starting from the signaled candidate MV.

[0323] Sub-CU level search is only available in two-way matching merge mode. The search window for sub-CU level search for all sub-CUs is exactly the same as the search window for CU level search. Accordingly, no additional bandwidth is required when searching at the sub-CU level.

[0324] In order to refine the mode MVP, pattern matching is also used. In AMVP mode, two MVPs are generated using the HEVC MVP generation process, and one MVP index is signaled to select one of the two MVPs. The selected MVP is further refined using template matching in PMVR. If adaptive motion vector resolution (AMVR) is used, the MVP is rounded to an appropriate precision before being refined with template matching. This refinement process is called sample-matched motion vector predictor refinement (PMVPR). In the remainder of this document, PMVR includes template-matched PMVR, bidirectional-matched PMVR, and PMVPR unless specifically defined otherwise.

[0325] In order to reduce the memory bandwidth requirement, PMVR is not available for 4x4, 4x8 and 8x4 CUs. In order to reduce the additional memory bandwidth required, the search range {pattern matching, bidirectional matching} for a CU area of 64 can be reduced to {±2, ±4}. The search range {pattern matching, bidirectional matching} for CU zone greater than 64 can be reduced to {±6, ±8}. Compared to the worst case in HEVC, the required memory bandwidth was reduced from 45.9x in PMVD in JEM-7.0 to 3.1x in PMVR using all the methods described in the PMVR section of this document.

[0326]

[0327] General history-based motion vector prediction (HMVP)

[0328] In general, image compression technology uses the exploitation of spatial and temporal redundancy as two main schemes. For example, both High Efficiency Video Coding (HEVC) and VVC use two motion compensation schemes based on interframe coding. One is motion fusion and the other is advanced motion vector prediction (AMVP). In order to improve the two prediction modes, various modifications have been discussed. Various modifications include increasing the number of candidates to find more spatially expanded candidates and testing transient candidates in non-traditional locations. The two schemes include initially creating a list using the available candidates, minimizing the rate-distortion (RD) cost, and signaling the selected candidate in the bitstream.

[0329] In particular, in modern image compression technology, an HMVP is discussed in which motion information of a previously encoded block is stored, and the stored motion information is used to predict the motion of a subsequently encoded block. Such an HMVP may be added to a merge list (or merge candidate list) or an AMVP list (or AMVP candidate list).

[0330] The decoder maintains a lookup table (LUT) in a first-in-first-out (FIFO) system (or method) for HMVP. In the invention, a LUT is not limited by its name and may be referred to as a table, HMVP table, HMVP candidate table, buffer, HMVP buffer, HMVP candidate buffer, HMVP list, or HMVP candidate list. In particular, when a non-affine prediction unit (PU) (or coding unit (CU)) is decoded, the corresponding motion information is stored in the LUT. The decoder performs decoding on the next PU. Here, the stored motion information may include motion vectors, reference index information, and mode information in the x (horizontal) and y (vertical) directions.

[0331] The decoder maintains a LUT that stores motion information of the progressively decoded non-affine candidate. The size of the LUT can be limited to predefined S candidates. In one embodiment, the LUT may be reset at the start of a slice, the start of a CTU line, or the start of a CTU.

[0332] HMVP can be applied to both merge mode and AMVP mode. A merge list can have B candidates, and an AMVP list can have two candidates. In conventional image compression technology, the merge list is configured from the following candidates: i) spatial candidate, ii) temporal candidate, iii) bi-prediction candidate (Bi-Pred), iv) zero-motion candidate. Recently, a method to additionally consider advanced motion vector prediction (ATMVP) as a candidate has been discussed. For example, the ATMVP candidate may be inserted into the merge list before the temporary candidate. Merge list candidates are added to the merge list until they reach the maximum size of the merge list. The duplicate candidate may not be added to the merge list. Two candidates can be inserted into the AMVP list. For example, the first candidate of two candidates may be selected from the available spatial candidates, and the second candidate may be selected from the temporal candidates. If the list is not full, a null motion vector candidate can be added.

[0333] HMVP is applied based on FIFO queuing, in which candidates exit the LUT identical to their input queuing.

[0334] In one embodiment, when the HMVP is applied to a merge list configuration, the HMVP candidate may be inserted (or added) to a third list location as follows:

[0335] 1. Spatial Candidate

[0336] 2. Temporary Candidate

[0337] 3. Up to S HMVP Candidates for LUT

[0338] 4. Joint Bi-Pred Candidate

[0339] 5. Zero Motion Vector Candidate

[0340] In one embodiment, when the HMVP is applied to an AMVP list configuration, the HMVP may be inserted at a third location after the temporary candidate as follows:

[0341] 1. Spatial Candidate

[0342] 2. Temporary Candidate

[0343] 3. Up to K HMVP Candidates

[0344] 4. Zero Motion Vector Candidate

[0345] FIG. 27 is a flowchart for describing a method for storing HMVP in accordance with an embodiment of the invention.

[0346] Referring to FIG. 27, the decoder decodes the current PU (or CU) (S2701).

[0347] The decoder checks whether the current PU is a non-affine mode encoded block (S2702). To facilitate the use of HMVP candidates, if the current PU is an affine-mode encoded block, the decoder does not store the current PU's movement information in a table.

[0348] If the current PU is a non-affine mode encoded block, the decoder stores (or updates) the movement information of the current PU in a table (S2703).

[0349] In an embodiment of the invention, the HMVP table can be updated using two methods, i.e. methods i) FIFO without restrictions ii) FIFO with restrictions. In the first case, redundant motion information may be present, but the clipping process is not applied. This helps reduce the complexity of the entire process. On the other hand, in the latter case, the clipping process is applied and no redundant motion information is present. This is described by referring to the next figure.

[0350] FIG. 28 is a diagram for describing an HMVP table that operates according to a FIFO queue discipline without restrictions in accordance with an embodiment of the invention.

[0351] Referring to FIG. 28, the candidate that is added to the table is added to the end (right) of the table. In contrast, a candidate that is released from the table according to the FIFO method is located at the front (left side, oldest candidate) of the table.

[0352] If the table is not completely filled with the maximum number of predefined candidates at index L-1 (ie, the end), new candidates are added without removing the candidate. In contrast, if the table is already completely filled, i.e. if the maximum number of the table is satisfied, then the candidate who is located in the front, i.e. the oldest one in the table is removed and a new candidate is added.

[0353] FIG. 29 is a diagram for describing an HMVP table that operates according to a constrained FIFO queue discipline in accordance with an embodiment of the invention.

[0354] Referring to FIG. 29, in the case where a constrained FIFO is used, pruning is performed if adding a new candidate causes any redundancy (ie, the new candidate includes redundant motion information). In an embodiment, if a candidate with redundant motion information is present in the table, then the redundant candidate in the table is removed and the motion information of the current candidate can be added.

[0355]

[0356] With respect to HMVP candidates, in many cases, the most recent historical MVs may overlap the motion information of a spatial candidate (or a spatial neighbor candidate). Accordingly, the present embodiment proposes a method for setting the order of adding candidates differently from the order of HMVP LUT indexes when an HMVP candidate is added to a merge list or AMVP.

[0357] According to an embodiment of the invention, the candidate list can be efficiently configured by adaptively adjusting the HMVP candidate. Accordingly, the number of signaling bins that are used for conversion to binary representation can be reduced and coding efficiency can be improved. Those. The HMVP candidate that is added to the merge list or AMVP list may not be limited to an index in the HMVP list. As an embodiment, the following Table 1 illustrates a method for changing the order in which an HMVP candidate is added to a merge or AMVP list.

[0358] [Table 1]

HMVP LUT Index (0 denotes most recent historical MV) Priority to be added to the Merge or AMVP list 0 2 1 0 2 1 3 3 .. ..

[0359] Referring to Table 1, as described above, there is a good chance that the most recently inserted HMVP candidate may have the same motion information as the spatial candidate. Accordingly, the order of adding the HMVP candidate may be predetermined independently of the HMVP index based on probability.

[0360] Additionally, in one embodiment, the encoder or decoder may add an HMVP candidate to a merge list or AMVP list, starting with the HMVP candidate that begins with the nth candidate in the list. Table 2 below illustrates the modified order in which a candidate is added to the Merger List or AMVP.

[0361] [Table 2]

HMVP LUT Index (0 denotes most recent historical MV) Priority to be added to the Merge or AMVP list 0 1 0 2 1 3 2 .. ..

[0362] Referring to Table 2, the HMVP candidate may be added to the merge list or AMVP list from the second index.

[0363] In one embodiment, information regarding the order in which an HMVP candidate is added to a table (LUT) may be signaled from the encoder to the decoder. For example, such ordering information may be conveyed via High Level Syntax (HLS). The high-level syntax may be a sequence parameter set, a picture parameter set, a slice header, an encoding tree unit, an encoding unit, and/or other suitable syntax data header, for example.

[0364] Table 3 illustrates the high-level syntax structure to which the method of the invention can be applied.

[0365] [Table 3]

high_level_parameter_set(){ Description ... set_HMVP_order_flag u(1) ...

[0366] Referring to Table 3, set_HMVP_order_flag equal to 1 indicates that set_HMVP_order_flag is present in the slice header in non-IDR pictures in CVS. set_HMVP_order_flag equal to 0 indicates that set_HMVP_order_flag is not present in the slice headers and adaptive HMVP is not used in VCS.

[0367] Table 4 illustrates the structure of the slice segment header syntax to which the method of the invention can be applied.

[0368] [Table 4]

slice_segment_header(){ Description ... if(set_HMVP_order_flag) slice_HMVP_idx u(1) ... } ...

[0369] Referring to Table 4, slice_HMVP_idx means the index for the sequence of used candidates. For example, slice_HMVP_idx equal to 0 could represent a base HMVP sequence such as 0, 1, 2, 3. Likewise, an index value of 1 could be used to represent an HMVP sequence as 3, 2, 1, 0.

[0370]

[0371] In an embodiment of the invention, in addition to the HMVP LUT, a method of using a long-term list for motion prediction is provided. Accordingly, the number of supported HMVP candidates may be increased. In an embodiment, 2 HMVP tables may be considered. In this case, one can be used to store general HMVP candidates, and the other can be used as a long-term list that stores candidates for which further support is required.

[0372] The following illustrates methods for resetting and creating a long-term list (or long-term HMVP list).

[0373] - After the first CTU of a CTU row is decoded, one or more historical MVs of the subsequent CTU may be added to the long-term HMVP LUT. Such a long-term HMVP LUT may not be used or updated until the next CTU line.

[0374] - At the beginning of the next CTU line, the long-term HMVP LUT can be used to reset the general HMVP LUT. The reason for this is that the HMVP candidates of a CTU at the beginning of a CTU line may be more related compared to the historical MV at the end of the previous CTU line.

[0375] -- The above process can be repeated.

[0376] FIG. 30 is a diagram illustrating an HMVP LUT and a long-term HMVP LUT in accordance with an embodiment of the invention.

[0377] Referring to FIG. 30, the encoder or decoder may include two LUTs for storing HMVP candidates. One could be an HMVP LUT (or a general HMVP LUT or a short-term HMVP LUT) and the other could be a long-term HMVP LUT. When an HMVP candidate is added to both the merge list and the AMVP list, it may be added from an HMVP LUT or a long-term LUT, as illustrated in FIG. thirty.

[0378] The use of a long-term LUT can be signaled by using a new syntax element. In an embodiment, the syntax element may be signaled by high-level syntax. For example, a syntax element may be present in a sequence parameter set, a picture parameter set, a slice header, an encoding tree unit, an encoding unit, and/or other suitable syntax data header.

[0379] In an embodiment of the invention, a method is provided that takes into account flexibility for decoding when adding an HMVP candidate to an HMVP LUT. The encoder/decoder may consider a decision criterion for one or more PU (or CU) characteristics when adding an HMVP candidate to the HMVP LUT.

[0380] In an embodiment, the encoder/decoder may consider the following content when adding an HMVP candidate to the table. The encoder/decoder may add a candidate to the table based on characteristics such as the mode (eg, merge mode, affine mode, or AMVP mode) of the PU and/or block size, alone or in combination. In one embodiment, in addition to the characteristics, other additional characteristics may be taken into account. For example, the type of merge (eg, spatial candidate or temporal candidate) in which the HMVP LUT update is considered, whether it is a sub-PU, etc. may be considered as criteria for candidate selection. A selection criterion can be defined to reduce redundancy with previous history (or previous HMVP). For example, if the PU is encoded in merging mode and the merging type is spatial merging, then the decoder may not update the HMVP LUT with motion information of the corresponding PU.

[0381] FIG. 31 is a diagram illustrating an example of a method for updating an HMVP LUT in accordance with an embodiment of the invention.

[0382] Referring to FIG. 31, the encoder/decoder obtains motion information of the coded candidate (S3101).

[0383] The encoder/decoder judges whether to update the LUT with candidate motion information based on a predetermined decision criterion (S3102). As described above, the decision criterion may include characteristics associated with one or more of a mode (eg, merge mode, affine mode, or AMVP mode) of the candidate, candidate block size, and/or candidate merge type.

[0384] The encoder/decoder updates the LUT based on the decision criterion (S4303). Those. if the candidate satisfies the predetermined decision condition, then the encoder/decoder may add candidate motion information to the LUT.

[0385]

[0386] In an embodiment of the invention, there are proposed redundancy check restrictions for adding an HMVP candidate to a merge list (or AMVP list). Constraints regarding redundancy checking can be defined or implemented in a variety of ways.

[0387] In one embodiment, the encoder/decoder may limit the number of pruning checks for the first specified number of candidates in the merge list. As an embodiment, the encoder/decoder may limit the number of pruning checks for candidates from the first candidate of the merge list to a particular candidate. For example, the encoder/decoder may perform a pruning process on candidates from the first candidate of the merge list to a particular candidate. And the HMVP candidate, i.e. the purpose of the cutoff check may be limited to a predetermined number.

[0388] Additionally, in one embodiment, the encoder/decoder may limit the pruning check by performing a pruning check on a particular merge candidate type in the merge list. For example, the encoder/decoder may perform a pruning check only on a spatial merge list candidate when adding an HMVP candidate. Alternatively, for example, the encoder/decoder may perform a pruning check on only some of the spatial candidates of the merge list when adding an HMVP candidate. Some of the spatial candidates may be pre-defined. For example, the predetermined spatial candidates may be at least one of a left neighbor spatial candidate and/or a top neighbor spatial candidate. Or, for example, when adding an HMVP candidate, the encoder/decoder may perform a pruning check on only a portion of the spatial candidates of the merge list, and the portion of the spatial candidates may be pre-defined as the left and top sides. The embodiment of the invention is not limited to the above examples, and other types may be combined and limited as the purpose of the cutoff check.

[0389] FIG. 32 is a diagram illustrating a method for limiting the number of HMVP candidates, i.e. the purpose of checking the clipping, in accordance with an embodiment of the invention.

[0390] Referring to FIG. 32, in an embodiment of the invention, the number of HMVP candidates, i.e. the target of the pruning check may be limited to M. The encoder/decoder may check for redundancy of motion information among the first M HMVP LUT candidates and merge candidates in the merge list when creating a merge list using the HMVP candidates.

[0391] Alternatively, the encoder/decoder may check for redundancy of motion information among the first M HMVP LUT candidates and the currently decoded PU when adding motion information of a decoded processing unit (eg, PU) to the HMVP LUT.

[0392] FIG. 33 is a flowchart illustrating an example of a method for performing a clipping check in accordance with an embodiment of the invention.

[0393] Referring to FIG. 33, the encoder/decoder obtains motion information of the decoded candidate and determines (or decodes) the number of clipping checks (S3301, S3302). The number of clipping checks may be predetermined in the encoder/decoder in accordance with the method described above (eg, described in FIG. 32). The encoder/decoder performs a clipping check based on a certain number of clipping checks (S3303).

[0394] In one embodiment, as in Table 4 and Table 5, information that is associated with the clipping check may be signaled through high-level syntax. In this case, the syntax element that is passed from the encoder to the decoder may be signaled by high-level syntax to indicate the number of clipping checks. The high-level syntax may be included in a sequence parameter set, a picture parameter set, a slice header, an encoding tree unit, an encoding unit, and/or other appropriate syntax data header, for example.

[0395]

[0396] In an embodiment of the invention, an efficient method for selecting an HMVP candidate is provided. When a historical motion vector candidate (ie, an HMVP candidate) is inserted into a merge list (or AMVP list), then a pruning check can be performed such that the HMVP candidate does not overlap the existing merge list. At this time, in order to perform cumulative redundancy checks between a redundancy list of size M and a historical LUT of size N, checks (M-1)×N times are required.

[0397] Accordingly, in an embodiment of the invention, the number of HMVP candidates may depend on the merger candidate. For example, the number of HMVP candidates may depend on the number of spatial candidates that are present in the merge list. Alternatively, for example, the number of HMVP candidates may depend on the number of spatial candidates and temporal candidates that are present in the merge list.

[0398] If there is another merge candidate in the merge list, then the number of HMVP candidates on which the pruning check will be performed based on a particular criterion (or rule) based on the number of merge candidates and/or the number of HVVPs of the merge list may be reduced. Accordingly, the number of worst-case redundancy checks can be reduced.

[0399] For example, in the case of a merge list of size (or length) 6, if the merge list is completely full, then the merge list may include a maximum of 5 spatial or other merge candidates. In order to insert a HMVP candidate into 6 HMVP lists, in the worst case, 30 redundancy checks may be required.

[0400] In one embodiment, an example related to restrictions on the number of HMVPs that should be targets of a pruning check is illustrated in Equation 17 and Table 5.

[0401] [Equation 17]

if (existing_candidates >= 3)

number_hist_to_check=7 - existing_candidates

[0402] [Table 5]

# existing candidate # existing candidate to check # historical MV for verification # checks 1 1 6 6 2 2 6 12 3 3 4 12 4 4 3 12 5 5 2 10

[0403] Referring to Table 5, the number of redundancy checks for adding HMVPs in the worst case can be reduced by 12 times rather than 30 times by limiting the number of HMVPs, i.e. cutoff check targets, up to 2.

[0404]

[0405] In an embodiment of the invention, a method for creating a merge list using history-based spatio-temporal motion vector prediction (H-STMVP) is provided. H-STMVP indicates the candidate extracted as the average of two history-based spatial MVPs and TMVPs. Two spatial HMVPs can be obtained from the HMVP buffer. TMVP can be obtained from the current merge list. In this case, the spatial candidate may be a candidate obtained from the last 2 encoded MVs in the decoding sequence before the current block.

[0406] For example, the last encoded MV (referred to as MV_L in the invention), MV, i.e. the latter to the latter (referred to as MV_(L-1) in the invention), and MV_TMVP can be used to generate a candidate H-STMVP to be inserted into the merge list.

[0407] If all three candidates can be used, then the MVs that are added to the merge list can be calculated by Equation 18 below.

[0408] [Equation 18]

[0409] In one embodiment, if only two of three candidates are available, then only two candidates can be averaged to form the H-STMVP. Likewise, if only one candidate is available, then one candidate may be used. If an available candidate is not present, then H-STMVP may not be used for merge list configuration.

[0410] In an embodiment of the invention, a method is provided for obtaining the motion vector of an H-STMVP candidate using a different method than Equation 18.

[0411] For example, instead of averaging three or more candidates simultaneously, averaging the spatial candidates first and then averaging the two candidates again using the result of the averaging may be simpler from a computational standpoint. An example of this is illustrated in the following equations.

[0412] [Equation 19]

[0413] Alternatively, the average can be obtained as follows.

[0414] [Equation 20]

[0415] [Equation 21]

[0416] [Equation 22]

[0417] The encoder/decoder may first average the two candidates as in Equations 19 to 21, and then may average the resulting values using the third candidate. Alternatively, the encoder/decoder may assign a higher importance/weight to the candidate, i.e. MV_L, by applying the shift operation twice, as in Equation 22. The average can be extracted without the division operation, but only by the shift operation, using Equations 19 to 22.

[0418]

[0419] In an embodiment of the invention, a method is provided that uses a given number (n) of spatial candidates, instead of two history-based spatial candidates, when retrieving the H-STMVP. n candidates need not be essentially consecutive decoding sequences. n candidates can be selected randomly or according to some rule.

[0420] Accordingly, Equation 18 can be represented using a more general method, similar to Equation 23 below.

[0421] [Equation 23]

[0422] In another embodiment, assuming that 5 spatial candidates are used, the impact of increasing spatial candidates on the generation of the H-STMVP candidate can be minimized, and the spatial candidate and temporal candidate can be properly combined by improving the weighting factor that is applied to the temporal to the candidate.

[0423] Accordingly, for this purpose, after the spatial candidates are averaged together using Equation 24, the above objective can be achieved by averaging the MV-TMVP using the result of the averaging.

[0424] [Equation 24]

[0425]

[0426] In an embodiment of the invention, a method is provided for adding a weight factor (or weight multiplier) to a motion vector candidate that is used to extract an H-STMVP. In this case, the weighting factor may be determined empirically, or may be determined by taking into account the temporal distance to a fixed reference frame, or may be determined by taking into account a location in the history table. For example, a new candidate may have a heavier weighting than a previous candidate.

[0427] That is in the present embodiment, Equation 18 can be presented similar to Equation 25 below.

[0428] [Equation 25]

[0429] In this case, the weighting factor may have the same value or an unevenly distributed value.

[0430]

[0431] In an embodiment of the invention, a motion vector scaling method is proposed that is used to extract an H-STMVP candidate as one reference picture.

[0432] FIG. 34 is a diagram for describing a method for extracting an H-STMVP candidate using motion vectors that refer to different reference pictures in accordance with an embodiment of the invention.

[0433] Referring to FIG. 34, it is assumed that the candidates MV_L, MV_L-1 and MV_TMVP refer to (or indicate) respective different reference pictures. Those. Fig. 34 illustrates that the candidates that are used to generate H-STMVP candidates may have different reference indices and, as a result, have different reference frames.

[0434] The average of Equations 18 to 25 may not be equal to the result value because a frame with a closer reference frame may inherently have a greater influence on the motion vector of the H-STMVP. Accordingly, a method is proposed to scale all motion vectors in one reference frame for equal comparison and fusion.

[0435] In this case, the encoder may determine which one frame, performed as part of the RD optimization, is most suitable for use as a reference frame. In an embodiment, the selected reference frame may be signaled in the slice header similar to the TMVP array index that is present in the slice header. For example, a reference frame to be used may be generated using a fixed rule. Alternatively, for example, the list may be scaled as the first available reference frame from L0, or the list may be scaled based on the current picture index value.

[0436] In one embodiment, to achieve the above purpose, the encoder may transmit information to the decoder for one fixed picture using high-level syntax (HLS), which may be part of a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, and/or or other data header. For example, a high-level syntax structure such as Table 6 and/or Table 7 below may be defined.

[0437] [Table 6]

high_level_parameter_set(){ Description ... set_HSTMVP_ref_pic_flag u(1) ...

[0438] Referring to Table 6, set_HSTMVP_ref_pic_flag equal to 1 indicates that set_HSTMVP_idx is present in the non-IDR picture slice header in CVS. set_HSTMVP_ref_pic_flag equal to 0 indicates that set_HSTMVP_idx is not present in the slice header.

[0439] [Table 7]

slice_segment_header(){ Description ... if(set_HSTMVP_ref_pic_flag) slice_HSTMVP_idx u(1) ... } ...

[0440] Referring to Table 7, slice_HMVP_idx denotes the reference index. In one embodiment, the reference index may be selected relative to the list L0.

[0441]

[0442] In embodiments of the invention, more detailed embodiments will be described in relation to the above-described embodiments. In particular, a method is proposed to indirectly use an HMVP affine candidate by using position and dimensional information to calculate or retrieve the CPMV of the current block. In the invention, the extracted CPMV may be referred to as the legacy affinity candidate HMVP. The legacy HMVP Affine Candidate in accordance with an embodiment of the invention may be used in the Merge Affine List and/or AMVP Affine List generation process described above.

[0443] FIG. 35 is a diagram illustrating the location of a block for extracting a legacy HMVP affinity candidate in accordance with an embodiment of the invention.

[0444] Referring to FIG. 35, the CPMV of the current block 3501 can be extracted based on the location and size of the HMVP affine candidate in a manner similar to the method of extracting the common legacy CPMV from neighboring blocks. Those. the encoder/decoder may extract motion vectors of the control points of the current block 3501 based on information about the position and size (eg, width and height) of the reference block 3502, i.e. HMVP affine candidate.

[0445] As an embodiment, the CPMV of the current block's legacy affine HMVP can be extracted by using Equations 26 and 27 below.

[0446] [Equation 26]

V0x=VB0x + (VB2_x - VB0x) * (posCurCU_Y - posRefCU_Y) / RefCU_height + (VB1x - VB0x) * (posCurCU_X - posRefCU_X) / RefCU_width

[0447] [Equation 27]

V0y=VB0y + (VB2_y - VB0y ) * (posCurCU_Y - posRefCU_Y) / RefCU_height + (VB1y - VB0y ) * (posCurCU_X - posRefCU_X) / RefCU_width

[0448] In Equations 26 and 27, posCurCU_Y is the vertical coordinate value of the top left sample of the current block 3501, and posRefCU_Y is the vertical coordinate value of the top left sample of the reference block 3502. posCurCU_X is the horizontal coordinate value of the top left sample of the current block 3501, and posRefCU_X is the horizontal coordinate value of the top left sample of reference block 3502. RefCU_height is the height of reference block 3502, and RefCU_width is the width of reference block 3502.

[0449]

[0450] In one embodiment of the invention, when an HMVP affine candidate (direct or legacy HMVP) is added, constraints can be added to select the HMVP affine candidate that can be used to generate a merge affine list or an AMVP affine list.

[0451] As an example, an HMVP affine candidate may be added to a merge affine list or an AMVP affine list only when the HMVP affine candidate is adjacent to the current block.

[0452] As another example, an HMVP affine candidate may be added to a merge affine list or an AMVP affine list only when the HMVP affine candidate is located (or exists) within a specific distance from the current block. For example, the specific distance may be a predetermined pixel distance. The encoder/decoder may determine whether an HMVP affine candidate is located within a predetermined specific distance to determine whether an HMVP affine candidate is available.

[0453] As another example, an HMVP affine candidate may be added to a merge affine list or an AMVP affine list only when an HMVP affine candidate is located (or exists) at a specific location based on the current block. For example, when an HMVP affine candidate is present at a particular location, it may be the case that the HMVP affine candidate is the left or top neighbor of the current block.

[0454] With respect to an affine HMVP LUT that has N elements, the verification process described above for all elements or the first M elements can be performed until the merge list or AMVP is complete, or until it is previously reached a certain number of HMVP candidates.

[0455]

[0456] In one embodiment of the invention, a method is provided in which an HMVP affinity candidate is used to replace a legacy affinity candidate that is already present in a merge affine list and/or an AMVP affine list.

[0457] FIG. 36 is a diagram illustrating an affine merge list or an AMVP affine list in accordance with an embodiment of the invention.

[0458] Referring to FIG. 36, the encoder/decoder may replace a legacy candidate present in an existing merge affine list or AMVP affine list with a legacy HMVP affine candidate. Those. when a subblock-based merge mode is applied to the current block, the encoder/decoder may generate a subblock-based merge candidate list using the legacy affine candidate and the already created affine candidate, retrieve the legacy HMVP affine candidate, and replace at least one legacy affine candidate included into a subblock-based list of merge candidates inherited by the HMVP affine candidate.

[0459] In addition, in one embodiment of the invention, an affine HMVP lookup table (LUT) may be initialized at the beginning of a slice, CTU row, or CTU. By doing this, the efficiency of parallel processing can be improved.

[0460]

[0461] Further, in the following embodiments, a method is proposed for reducing the number of worst case pruning checks from an HMVP.

[0462] In an embodiment of the invention, when an HMVP candidate is added to the merge list, the number of pruning checks may be determined based on the number of available candidates in the merge list and the number of HMVP candidates that can be added to the merge list. In the following, when describing embodiments of the invention, for convenience of description, variables are defined as follows.

[0463] - NST: number of available (or existing) candidates in the merge list

[0464] - NHMVP: number of HMVP candidates in the table (i.e. HMVP table size)

[0465] - NmrgToBeAdded: number of HMVP candidates added to the merge list

[0466] - NHMVPChecked: The number of HMVP candidates that are checked with the cutoff.

[0467] - Nmax_hmvp_prunning: number of worst case pruning checks required to add an HMVP candidate to the merge list

[0468] In one embodiment of the invention, an HMVP candidate may be added to the merge list in accordance with the following conditions.

[0469] - First condition: When the LUT is previously pruned (i.e. there are no identical mvs among the HMVP LUT candidates)

[0470] - Second condition: When the HMVP LUT size is 6

[0471] - Third condition: When the maximum number of available (or existing) merge candidates for adding an HMVP candidate to the merge list is 4. That is, this occurs when the number of merge candidates in the merge list is less than the value obtained by subtracting 1 from the maximum size of the merge list (or the maximum number of merge candidates). For example, the maximum size of a merge list could be 6, and if the number of currently available merge candidates is less than 5, then an HMVP candidate can be added (or inserted). In other words, an HMVP candidate can only be added up to index 5 of the merge list.

[0472] When an HMVP candidate is added to the merge list (i.e., becomes a merge candidate), each HMVP candidate may require a pruning check to remove duplicates among the merge candidates. According to the existing image compression technique, the worst case number of pruning checks required to add HMVP to the merge list can be calculated as shown in Table 8 below.

[0473] [Table 8]

N _ST N _mrgToBeAdded N _HMVChecked N _{max_hmvp_prunning} 0 5 6 0 1 4 6 4 2 3 6 7 3 2 6 9 4 1 6 10

[0474] Referring to Table 8, according to the existing image compression technique, a pruning check can be performed on six HMVP candidates in the HMVP table (or HMVP list, HMVP candidate list).

[0475] In particular, 1) when there is one candidate in the merge list, 4 HMVP candidates can be added to the merge list. Then, cutoff tests can be performed on the six HMVP candidates. In this case, the number of worst pruning checks can be 4. 2) When there are two candidates in the merge list, 3 HMVP candidates can be added to the merge list. Then, cutoff tests can be performed on the six HMVP candidates. In this case, the number of worst pruning checks can be 7. 3) When there are three candidates in the merge list, 2 HMVP candidates can be added to the merge list. Then, cutoff tests can be performed on the six HMVP candidates. In this case, the number of worst pruning checks can be 9. 4) When there are four candidates in the merge list, 1 HMVP candidate can be added to the merge list. Then, cutoff tests can be performed on the six HMVP candidates. In this case, the number of worst cutoff checks could be 10.

[0476] In an embodiment of the invention, a method for reducing the number of worst-case pruning checks described above is provided. If there are many merge candidates in the merge list, since the HMVP encoding result will decrease as the merge candidates (ie, non-HMVP candidates) increase, it may be necessary to reduce the number of HMVP candidates that must be pruned. Accordingly, in an embodiment of the invention, the encoder/decoder may set the number of HMVP candidates that must be checked (NHMVPChecked) equal to the number of available HMVP candidates that must be added (NmrgToBeAdded) in order to reduce the number of worst-case pruning checks. In this case, the number of worst cutoff tests can be calculated as shown in Table 9 below.

[0477] [Table 9]

N _ST N _mrgToBeAdded N _HMVChecked N _{max_hmvp_prunning} 0 5 5 0 1 4 4 4 2 3 3 6 3 2 2 6 4 1 1 4

[0478] Referring to Table 9, the number of worst case clipping checks for HMVP can be reduced from 10 to 6 compared to conventional image compression techniques.

[0479] Referring to Table 9, in the embodiment, 1) when there is one candidate in the merge list, 4 HMVP candidates can be added to the merge list. Then, cutoff tests can be performed on the four HMVP candidates. In this case, the number of worst pruning checks can be 4. 2) When there are two candidates in the merge list, 3 HMVP candidates can be added to the merge list. Then, cutoff tests can be performed on the three HMVP candidates. In this case, the number of worst pruning checks can be 6. 3) When there are three candidates in the merge list, 2 HMVP candidates can be added to the merge list. Then, cutoff tests can be performed on the two HMVP candidates. In this case, the number of worst pruning checks can be 6. 4) When there are four candidates in the merge list, 1 HMVP candidate can be added to the merge list. Then, cutoff tests can be performed on one HMVP candidate. In this case, the number of worst cutoff checks can be 4.

[0480] In an embodiment of the invention, in order to reduce the number of worst case pruning checks, the encoder/decoder may set the number of HMVP candidates against which a pruning check should be performed to be exactly the same as the sum of the number of available HMVP candidates (NmrgToBeAdded ), which must be added, and K. Here K represents a predefined constant value. As an example, when K is 1, the worst case number of pruning checks can be calculated as shown in Table 10 below.

[0481] [Table 10]

_NST N _mrgToBeAdded N _HMVChecked N _{max_hmvp_prunning} 0 5 6 0 1 4 5 4 2 3 4 7 3 2 3 8 4 1 2 7

[0482] Referring to Table 10, in the embodiment, 1) when there is one candidate in the merge list, 4 HMVP candidates can be added to the merge list. Then, cutoff tests can be performed on the five HMVP candidates. In this case, the number of worst pruning checks can be 4. 2) When there are two candidates in the merge list, 3 HMVP candidates can be added to the merge list. Then, cutoff tests can be performed on the four HMVP candidates. In this case, the number of worst pruning checks can be 7. 3) When there are three candidates in the merge list, 2 HMVP candidates can be added to the merge list. Then, cutoff tests can be performed on the three HMVP candidates. In this case, the number of worst pruning checks can be 8. 4) When there are four candidates in the merge list, 1 HMVP candidate can be added to the merge list. Then, cutoff tests can be performed on the two HMVP candidates. In this case, the number of worst cutoff checks can be 7.

[0483] In an embodiment of the invention, in order to reduce worst-case pruning checks, the number of HMVP candidates that must be checked (NHMVPChecked) can be determined as in Equation 28 below.

[0484] [Equation 28]

[0485] In Equation 28, C is a predetermined constant value. If C is equal to 2, then the worst case number of pruning checks can be calculated as shown in Table 11 below.

[0486] [Table 11]

_NST N _mrgToBeAdded N _HMVChecked N _{max_hmvp_prunning} 0 5 6 0 1 4 6 4 2 3 6 7 3 2 4 9 4 1 2 7

[0487] Referring to Table 11, in the embodiment, 1) when there is one candidate in the merge list, 4 HMVP candidates can be added to the merge list. Then, cutoff tests can be performed on the six HMVP candidates. In this case, the number of worst pruning checks can be 4. 2) When there are two candidates in the merge list, 3 HMVP candidates can be added to the merge list. Then, cutoff tests can be performed on the six HMVP candidates. In this case, the number of worst pruning checks can be 7. 3) When there are three candidates in the merge list, 2 HMVP candidates can be added to the merge list. Then, cutoff tests can be performed on the four HMVP candidates. In this case, the number of worst pruning checks can be 9. 4) When there are four candidates in the merge list, 1 HMVP candidate can be added to the merge list. Then, cutoff tests can be performed on the two HMVP candidates. In this case, the number of worst cutoff checks can be 7.

[0488]

[0489] When an HMVP is added to the merge list and/or the AMVP list, pruning is again required to prevent duplication with candidates existing in the pruning list and/or the AMVP list. If the HMVP LUT has already been pruned through a limited FIFO operation as described in FIG. 29 above, then when inserting (or adding) an HMVP candidate to a merge list, a comparison (or cutoff check) between the HMVP candidates may not be required. For this reason, the number of pruning checks can be reduced by using a non-restrictive FIFO table, as described in FIG. 28 above. This is because cutoff checks between HMVP candidates are required when inserting HMVP candidates into the merge list. As described above, in the invention, the HMVP LUT is not limited by its name, and may be referred to as LUT, table, HMVP table, HMVP candidate table, buffer, HMVP buffer, HMVP candidate buffer, HMVP list, HMVP candidate list, HMVP candidate list, list based on the history of merger candidates, etc.

[0490] In one embodiment of the invention, the size of the HMVP lookup table (LUT) may be determined taking into account the HMVP candidate insertion process to create a merge list and/or an AMVP list. In particular, HMVP candidates can be added up to a predefined merge list size. For example, when the size of the maximum merge list is defined as 6, HMVP cannot be the 6th candidate. If 5 merge candidates are available (or present) in the merge list, then the HMVP candidate cannot be added. In this case, the sixth candidate may be selected from candidates other than HMVP (or according to another method). Therefore, in view of the above-described HMVP candidate insertion process, in one embodiment of the invention, the following method for selecting the size of an HMVP LUT is proposed.

[0491] As an embodiment, the size of the HMVP LUT may be determined or set to (MaxNumMergeCand-K). Here, MaxNumMergeCand indicates the maximum list of merge candidates (or the maximum number of candidates included in the list of merge candidates or the maximum number of merge candidates), and at this time MaxNumMergeCand can be defined as 6. K represents a predefined constant. For example, K could be 1 and the HMVP LUT size could be 5.

[0492] In accordance with an embodiment of the invention, by limiting the HMVP to MaxNumMergeCand-K (as described above, for example, K corresponds to 1), the worst case number of pruning checks (or worst case) can be reduced when adding the HMVP to the merge list and/or HMVP table. (Limited FIFO operation described previously in Fig. 29). In addition, according to an embodiment of the invention, the storage memory for the HMVP LUT may be reduced.

[0493] In an embodiment of the invention, an update process such as the embodiment described below may be applied to the HMVP motion candidate table given the HMVP table size described above. The following embodiments are examples of the HMVP table update process, and the embodiments of the invention are not limited to this.

[0494]

[0495] HMVP table update process

[0496] First, the input of the HMVP table update process can be defined as follows.

[0497] - Motion vectors mvL0 and mvL1

[0498] - Reference indexes refIdxL0 and refIdxL1

[0499] - Prediction list usage flags predFlagL0 and preFlagL1

[0500] The output of this update process may be a modified HMVP candidate list array. In this process, mvCand is a variable indicating a motion vector candidate with motion vectors, reference indices, and prediction list usage flags.

[0501] This update process can be performed in the following steps.

[0502] 1. The variable identicalCandExist is set to false and the variable tempIdx is set to 0. Here, identicalCandExist is a variable indicating whether the same traffic information is present in the HMVP candidate list, and tempIdx is a variable indicating the index of the HMVP candidate , in the HMVP candidate list, with the same motion information as the current motion vector.

[0503] 2. If HMVPCandNum is greater than 0, then for each index HMVPIdx with HMVPIdx=0..HMVPCandNum-1, the following steps can be applied as long as the variable identicalCandExist is true. Here, HMVPCandNum represents the number of HMVP candidates in the HMVP candidate list, and HMVPIdx represents the index assigned to the HMVP candidates in the HMVP candidate list.

[0504] - If mvCand has the same motion vector and the same reference index as HMVPCandList[HMVPIdx] (i.e. the HMVP candidate with HMVPIdx in the HMVP candidate list), then identicalCandExist may be set to true, and tempIdx can be set to HMVPIdx.

[0505] 3. And, the HMVP candidate list can be updated according to the following steps.

[0506] (1) If identicalCandExist is true or HMVPCandNum is equal to MaxNumMergeCand-K, then the following steps may be applied. Here, MaxNumMergeCand is a variable indicating the size of the merge list (or merge candidate list) (or the maximum number of merge list candidates, the maximum number of merge candidates), and K is an arbitrary constant. In an embodiment, K may be predetermined. Additionally, in an embodiment, K may be set to 1, and MaxNumMergeCand-K may be 5 when the value of MaxNumMergeCand is determined to be 6. Also, in an embodiment, MaxNumMergeCand-K may be set to 5. For example, if identicalCandExist matches the value true or HMVPCandNum matches 5, then the following steps can be applied.

[0507] - For each index idx with idx = (tempIdx+1) .. (HMVPCandNum-1), HMVPCandList [idx−1] can be set to HMVPCandList [idx]. Those. The HMVP candidate index with the index after tempIdx can be set to a value whose value is decremented by 1.

[0508] - HMVPCandList [HMVPCandNum-1] can be set to mvCand.

[0509] (2) Otherwise (ie, if identicalCandExist is false and HMVPCandNum is less than MaxNumMergeCand-K), the following steps may be applied. As described above, in an embodiment, MaxNumMergeCand-K may be set to 5. For example, otherwise (ie, identicalCandExist is false and HMVPCandNum is less than 5), the following steps may be applied.

[0510] - HMVPCandList[HMVPCandNum++] can be set by mvCand.

[0511] As an example, this update process may be invoked when the current slice matches a P or B slice. At this time, the HMVPCandNum variable is set to 0, and the HMVPCandList variable can be defined as an array of elements from MaxNumMergeCand-K. As described above, in an embodiment, MaxNumMergeCand-K can be set to 5. For example, at this time, the variable HMVPCandNum can be set to 0 and the variable HMVPCandList can be defined as an array of elements of 5.

[0512] Next, another example of the HMVP table update process will be described.

[0513] First, the update process input may be determined as follows.

[0514] - Motion vectors mvL0 and mvL1

[0515] - Reference indexes refIdxL0 and refIdxL1

[0516] - Prediction list usage flags predFlagL0 and preFlagL1

[0517] The output of this update process may be a modified HMVP candidate list array. In this process, mvCand is a variable indicating a motion vector candidate with motion vectors, reference indices, and prediction list usage flags.

[0518] This update process can be performed in the following steps.

[0519] 1. For each HMVPIdx index with HMVPIdx=0..HMVPCandNum-1, the following steps may be applied in order until the sameCand variable evaluates to true. Here, sameCand is a variable indicating whether the same motion information is present in the HMVP candidate list.

[0520] - If mvCand has the same motion vector and the same reference index as HMVPCandList[HMVPIdx], then sameCand is set to true.

[0521] - Otherwise, sameCand is set to false.

[0522] - HMVPIdx++ (i.e. HMVPIdx is increased by 1)

[0523] 2. The tempIdx variable is set to HMVPCandNum.

[0524] 3. If sameCand matches true or HMVPCandNum matches MaxNumMergeCand-K, then for each index tempIdx with tempIdx = (sameCand? HMVPIdx: 1)..HMVPCandNum-1, HMVPCandList[tempIdx] is copied or set to HMVPCandList[tempIdx- 1]. K is an arbitrary constant. In an embodiment, K may be predetermined. Further, in an embodiment, K may be set to 1, and MaxNumMergeCand-K may be 5 when the MaxNumMergeCand value is determined to be 6. Also, in an embodiment, MaxNumMergeCand-K may be set to 5.

[0525] 4. mvCand is copied to HMVPCandList[tempIdx].

[0526] 5. If HMVPCandNum is less than MaxNumMergeCand-K, then HMVPCandNum is increased by one. As described above, in an embodiment, MaxNumMergeCand-K can be set to 5.

[0527] The above embodiments of the invention may be separated and described for convenience of description, but the invention is not limited thereto. Those. the embodiments described above may be performed independently, or one or more embodiments may be combined and performed.

[0528] FIG. 37 is a flow chart illustrating a method of processing a video signal based on history-based motion vector prediction in accordance with an embodiment to which the invention is applied.

[0529] Referring to FIG. 37, the decoder is mainly described for convenience of description, but the invention is not limited to this. A method for processing a video signal based on history-based motion vector prediction according to an embodiment of the invention may be identically implemented in an encoder and a decoder.

[0530] In a video signal processing method based on history-based motion vector prediction, the decoder configures a merge candidate list based on neighboring blocks for the current block (S3701).

[0531] The decoder adds a history-based merge candidate for the current block to the list of merge candidates (S3702).

[0532] The decoder obtains an index indicating a merge candidate, which is used for inter-frame prediction of the current block in the merge candidate list (S3703).

[0533] The decoder generates prediction samples of the current block based on the motion information of the merge candidate indicated by the merge index (S3704).

[0534] The decoder updates the list of history-based merge candidates based on the motion information (S3705).

[0535] As described above, as an embodiment, a history-based merge candidate may be added to the list of merge candidates if the history-based merge candidate has traffic information that does not overlap with that of the predetermined candidate for merger, among the merger candidates included in the list of merger candidates.

[0536] As described above, as an embodiment, the history-based merge candidate list may be determined to have a size that is determined based on the maximum number of merge candidates of the merge candidate list.

[0537] As described above, as an embodiment, the history-based merge candidate list may be determined to have a size of a value obtained by subtracting 1 from the maximum number of merge candidates of the merge candidate list.

[0538] As described above, as an embodiment, the size of the history-based merge candidate list may be defined as 5.

[0539] As described above, as an embodiment, a history-based merge candidate may be added to the list of merge candidates if the history-based merge candidate has traffic information that does not overlap with that of a predetermined specific the number of merger candidates among the merger candidates included in the list of merger candidates.

[0540] As described above, as an embodiment, a history-based fusion candidate may be added to the list of fusion candidates if the history-based fusion candidate has motion information that does not overlap with that of a particular spatial candidate for a merger included in the list of merger candidates.

[0541] As described above, as an embodiment, a history-based merge candidate may be retrieved from a predetermined number of candidates in a list of history-based merge candidates.

[0542] As described above, as an embodiment, the decoder may add a null motion vector to the merge list when the merge list does not satisfy the maximum number of the merge list.

[0543] The embodiments described in the invention may be implemented and implemented in a processor, microprocessor, controller, or chip. For example, the functional units illustrated in the drawings may be implemented and implemented in a computer, processor, microprocessor, controller, or chip.

[0544] In addition, the decoder and encoder to which the invention applies may be included in a multimedia broadcast transmission/reception device, a mobile communication terminal, a home theater video device, a digital cinema video device, a security camera, a video dialogue device, a mode communication device real-time device such as video communication, mobile streaming device, storage medium, video camera, video on demand (VoD) device, over-the-top (OTT) video device, Internet streaming service device, three-dimensional (3D) device ) video, video telephony device and medical video device, and can be used to process video and data signals. For example, an OTT video device may include a game console, Blu-ray player, Internet access TV, home theater system, smartphone, tablet PC, and Digital Video Recorder (DVR).

[0545] Moreover, the processing method to which the present invention is applied may be created in the form of a computer executable program and may be stored on a computer-readable recording medium. Multimedia data with a data structure in accordance with the invention can also be stored on a machine-readable recording medium. Computer-readable recording media includes all types of storage devices in which computer-readable data is stored. The computer-readable recording medium may include a Blu-ray disc (BD), universal serial bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical storage device, for example. In addition, the computer-readable recording medium includes storage media implemented in the form of carriers (eg, transmission over the Internet). In addition, the bit stream generated using the encoding method may be stored in a computer-readable recording medium or transmitted through wired and wireless communication networks.

[0546] Additionally, an embodiment of the invention may be implemented as a computer program product using program code. The program code may be executed by a computer in accordance with an embodiment of the invention. The program code may be stored on a medium that can be read by a computer.

[0547] A decoding device and an encoding device to which the invention applies may be included in a digital device. The term "digital device" includes all digital devices configured to perform at least one of transmitting, receiving, processing and outputting, for example, data, content and services. Here, processing of data, content, services, etc. through a digital device, includes the operation of encoding and/or decoding data, content, services and the like. The digital device pairs or connects (hereinafter referred to as 'pairing') with other digital devices, external servers, etc. through a wired/wireless network to send and receive data and convert it as needed.

[0548] The digital device includes, for example, a fixed device (or stationary device) such as network TV, HBBTV (Hybrid Broadband TV), smart TV, IPTV (Internet Protocol Television), PC, etc. and a mobile device (or portable device) such as a PDA (Personal Digital Assistant), smartphone, tablet PC, laptop. In the invention described later, for convenience, FIG. 39 shows and describes digital TV as an embodiment for a digital device, and FIG. 40 shows and describes a mobile device as an embodiment of a digital device.

[0549] Meanwhile, the term “wired/wireless network” described herein refers to a communication network that supports various communication standards or protocols for interconnecting and/or transmitting and receiving data between digital devices or between digital devices and external servers. These wired/wireless networks may include both current and future supported communication networks, and communication protocols therefor, and may be formed by wired communication standard or protocols such as USB (Universal Serial Bus), CVBS (Composite Video signal), Component, S-Video (Analog), DVI (Digital Video Interface), HDMI (High Definition Multimedia Interface), RGB, D-SUB and the like, and are formed through communication standards for wireless communication such as Bluetooth, Radio Frequency Identification (RFID). RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Digital Home Network Alliance (DLNA), Wireless LAN (WLAN) (Wi-Fi), Wibro (Wireless Broadband), WiMAX (Worldwide Broadband Compatibility), HSDPA (High Speed Downlink Packet Access), LTE (Long Term Evolution), Wi-Fi Direct (Direct), etc.

[0550] Further, when simply referring to a digital device in the invention, it may mean a fixed device or a mobile device, or include both types depending on the context.

[0551] Meanwhile, the digital device is, for example, a smart device that supports a broadcast reception function, a computer function and at least one external input, and can support email, web browsing, banking, game and application via the wired/wireless network described above. In addition, the digital device may include an interface for supporting at least one input or control means (hereinafter referred to as an input means), such as a manual input device, a touch screen, and a remote control. The digital device may use a generic general purpose operating system (OS). For example, a digital device can add, remove, modify and update various applications in the general purpose OS kernel, and can configure and provide a more user-centric environment through them.

[0552] Meanwhile, the external input described in the invention includes external input devices, i.e. all input media or digital devices that are wired/wireless connected to the digital devices described above and transmit/receive associated data through it. Here, the external input may include all devices such as a high-definition multimedia interface (HDMI), a gaming device such as a Play Station or X-Box, a smartphone, a tablet PC, a printer, or a smart TV.

[0553] And the term “server” described in the invention includes a client, i.e. all digital devices or systems that supply data to the digital devices described above are called processor. Examples of such a server include a portal server that provides a web page or web content, an advertising server that provides advertising data, a content server that provides content, and a social networking service (SNS) server, service server, or manufacturer server provided by manufacturer, etc.

[0554] In addition, a “channel” described herein means a path, means, etc. for transmitting and receiving data, and can be represented, as an example, by a broadcast channel. Here, the broadcast channel is expressed in terms of a physical channel, a virtual channel and a logical channel in accordance with digital broadcast activation. A broadcast channel may be called a broadcast network. As described above, a broadcast channel refers to a channel for providing or accessing broadcast content that is provided by a broadcast station, and the broadcast content is mainly based on real-time broadcasting and is also called a live channel. However, recently, the broadcast environment has become more versatile and non-real-time broadcast is also enabled in addition to real-time broadcast. It can also be understood as a concept that refers to the entire channel. Thus, a live channel can be understood to mean not only a real-time broadcast, but also an entire broadcast channel, including a non-real-time broadcast in some cases.

[0555] In the invention, "random channel" is further defined with respect to a channel other than the broadcast channel described above. The arbitrary channel may provide a service guide, such as an electronic program guide (EPG), along with the broadcast channel. The Service Guide, GUI (Graphical User Interface) or OSD (On Screen Display) screen can be configured/provided using only a custom channel.

[0556] Meanwhile, unlike a broadcast channel with a predetermined channel number between transceivers, a random channel is a channel that is randomly allocated at the receiver, and a channel number that mainly does not overlap with the channel numbers for expressing broadcast channels is allocated to a random channel. For example, when a particular broadcast channel is configured, the receiver receives the broadcast signal and transmits the broadcast content and its signaling information via the configured channel. Here, the receiver analyzes the channel information from the signaling information and configures a channel browser, EPG and the like based on the analyzed channel information and provides it to the user. When the user makes a channel change request through the input facility, the receiver responds accordingly.

[0557] As described above, since the broadcast channel is pre-arranged between the transmitting and receiving terminals, when a random channel is allocated for the broadcast channel with overlap, it may cause confusion to the user or confusion may arise, so it is preferable not to allocate duplicates as described above . Meanwhile, even if the random channel number does not overlap with the broadcast channel number as described above, there is still confusion in the user's channel browsing process, and it is necessary to allocate the random channel number taking this into account. This is because a random channel according to the invention can also be implemented to be accessed in response to a user request to switch a channel through an input means in the same manner as a conventional broadcast channel. Accordingly, the arbitrary channel number can be determined and displayed in a form in which characters are written in parallel, such as arbitrary channel-1, arbitrary channel-2 and the like, instead of a numeric form similar to the broadcast channel, for the convenience of user access to the arbitrary channel and for ease of distinction or distinction from the broadcast channel number. Here, although the arbitrary channel number display may be implemented in a form in which the characters are written as in arbitrary channel-1, the receiver may recognize and implement the arbitrary channel in numerical form as a broadcast channel number internally. Also, the arbitrary channel number may be provided in the form of a number, such as a broadcast channel, and it is also possible to determine and display channel numbers in various ways, which may be different from the broadcast channels, such as video channel-1, title-1, and video-1.

[0558] The digital device executes a web browser for a web service and provides various types of web pages to the user. Here, a web page includes a web page including video content, and in the invention, the video is processed separately or independently from the web page. And the separated video can be allocated to an arbitrary channel as described above, and provided by a service guide or the like, and output in accordance with a channel switching request in the process of viewing the service guide or broadcast channel. Moreover, for services such as broadcast content, games and applications, in addition to web services, predefined content, images, audio, elements, etc. are processed separately from the broadcast content itself, games and applications, and an arbitrary channel number can be assigned for their playback, processing and the like, and can be implemented as described above.

[0559] FIG. 38 is a diagram schematically showing an example of a service system including a digital device.

[0560] Service systems including digital devices include a CP content provider 3810, an SP service provider 3820, an NP network operator 3830, and a home network end user (HNED) 1840. Here, HNED 3840 is, for example, a client 3800, i.e. digital device. Content provider 3810 creates and provides various content. As shown in FIG. 38, such content provider 3810 may be exemplified by a terrestrial broadcaster, a cable SO (System Operator) or MSO (Multiple SO), a satellite broadcaster, various Internet broadcasters, Private CPs and the like. Meanwhile, the content provider 3810 provides various applications in addition to the broadcast content.

[0561] The service provider 3820 provides the content provided by the content provider 3810 as a service package HNED 3840 by packaging the content. For example, service provider 3820 in FIG. 38 packages first terrestrial broadcast, second terrestrial broadcast, cable MSO, satellite broadcast, various Internet broadcasts, applications, etc. and provides them to HNED 3840.

[0562] Service provider 3820 provides services to client 3800 in a unicast or multicast manner. And the service provider 3820 can transmit data to multiple pre-registered clients 3800 at the same time and a protocol called Internet Group Management Protocol (IGMP) can be used to do this.

[0563] The content provider 3810 and service provider 3820 described above may be an identical or single entity. For example, content that is provided by content provider 3810 may be packaged into a service and provided by HNED 3840 such that the functions of service provider 3820 are performed together or vice versa.

[0564] Network operator 3830 provides a network for communication between content provider 3810 and/or service provider 3820 and client 3800.

[0565] The client 3800 may create a home network for transmitting and receiving data.

[0566] Meanwhile, the content provider 3810 and/or the service provider 3820 in the service system may use conditional access or content protection to protect the transmitted content. In this case, the client 3800 may use a processing facility such as a CableCARD (POD: Point of Deployment), DCAS (Downloadable CAS), etc. in response to restriction of reception or protection of content.

[0567] In addition, client 3800 may also use bidirectional service over a network (or communications network). In this case, the client 3800 may serve to some extent as a content provider, and the existing service provider 3820 may receive it and pass it back to another client.

[0568] FIG. 39 is a block diagram illustrating a digital device according to an embodiment. Here, Fig. 39, for example, may correspond to client 3800 in FIG. 38 and access the digital device described above.

[0569] The digital device 3900 includes a network interface 3901, a TCP/IP administration engine 3902, a service delivery administration engine 3903, an SI decoder 3904, a demultiplexer 3905 (demux), an audio decoder 3906, a video decoder 3907, a display module 3908 (display A /V and OSD), service control admin 3909, service discovery admin 3910, SI and metadata database 3911, metadata admin 3912, service admin 3913, UI admin 3914, and the like.

[0570] Network interface 3901 receives or transmits IP packets over the network. Those. network interface 3901 receives services, content, and the like from service provider 3820 via the network.

[0571] The TCP/IP manager 3902 is involved in packet transmission between the source and destination of IP packets received by the digital device 3900 and IP packets transmitted by the digital device 3900. And the TCP/IP manager 3902 classifies the received packets according to the appropriate protocol. and outputs the classified packets to a service delivery admin 3905, a service discovery admin 3910, a service control admin 3909, a metadata admin 3912, etc. The service delivery manager 3903 is responsible for managing the received service data. For example, the service delivery manager 3903 may use RTP/RTCP when managing real-time streaming data. When real-time streaming data is transmitted using RTP, the service delivery manager 3903 analyzes the received data packet in accordance with RTP and transmits it to the demultiplexer 3905 or stores it in the SI and metadata database 3911 under the control of the service manager 3913. And the service delivery administration means 3903 uses RTCP to feed back network reception information to the server that provides the service. A demultiplexer 3905 demultiplexes the received packets into audio, video, and SI system information data, and transmits them to the audio/video decoder 3906/3907 and the SI decoder 3904, respectively.

[0572] The SI decoder 3904 decodes system information such as program-specific PSI information, program and system information protocol PSIP, and DVB-SI digital video service information.

[0573] And the SI decoder 3904 stores the decoded service information in the SI and metadata database 3911, for example. Service information that is stored in this way can be read and used by the corresponding configuration, such as a user request.

[0574] The audio/video decoder 3906/3907 decodes each audio data and video data demultiplexed by the demultiplexer 3905. Thus, the decoded audio data and video data are provided to the user through the display unit 3908.

[0575] The application administration tool may include, for example, a UI administration tool 3914 and a service administration tool 3913. The application administration tool administers the overall state of the digital device 3900, provides a user interface, and performs administration of other administration tools.

[0576] The UI administration tool 3914 provides a GUI graphical user interface to the user using an OSD menu or the like, and receives key input from the user to perform a device operation according to the input. For example, when the UI administrator 3914 receives a channel selection key input from the user, the UI administrator 3914 transmits the key input signal to the service administrator 3913.

[0577] The service administrator 3913 manages the administrators associated with the service, such as the service delivery admin 3903, the service discovery admin 3910, the service control admin 3909, and the metadata admin 3912.

[0578] And the service administrator 3913 creates a channel map and selects a channel using the channel map in accordance with the key input received from the user interface administrator 3914. And the service management means 3913 receives the channel service information from the SI decoder 3904 and sets the audio/video packet ID PID of the selected channel in the demultiplexer 3905. The PID set in this manner is used in the demultiplexing process described above. Therefore, the demultiplexer 3905 filters the audio data, video data, and SI data using the PID.

[0579] Service discovery engine 3910 provides information necessary to select a service provider providing the service. When a channel selection signal is received from the service administrating means 3913, the service discovery administrating means 3910 finds the service using the information.

[0580] The service control manager 3909 is responsible for selecting and managing services. For example, the service control administrating means 3909 uses IGMP or RTSP or the like when the user selects a live broadcast service such as a conventional broadcast method, and uses RTSP to select and control the service when selecting a service such as VOD (Video on Demand). request). The RTSP protocol can provide a fast access mode to media objects for real-time streaming. And the service control administration engine 3909 may initiate and administer the session through the IMS gateway 3950 using the IMS IP multimedia subsystem and the SIP session initiation protocol. The protocol is one embodiment, and other protocols may be used in accordance with the example implementations.

[0581] The metadata administration engine 3912 administers the metadata associated with the service and stores the metadata in the SI and metadata database 3911 .

[0582] The SI and metadata database 3911 stores the service information decoded by the SI decoder 3904, metadata administered by the metadata administration engine 3912, and information required for service provider selection, which is provided by the service discovery administration engine 3910. And the SI and metadata database 3911 may store configuration data and the like for the system.

[0583] The SI and metadata database 3911 may be implemented using non-volatile RAM NVRAM, flash memory, or the like.

[0584] Meanwhile, the IMS gateway 3950 is a gateway that collects functions necessary to access the IMS-based IPTV service.

[0585] FIG. 40 is a configuration block diagram illustrating another embodiment of a digital device. In particular, FIG. 40 illustrates a block diagram of a mobile device as another embodiment of a digital device.

[0586] Referring to FIG. 40, the mobile device 4000 may include a wireless communication unit 4010, an A/V audio/video input unit 4020, a user input unit 4030, a touch unit 4040, an output unit 4050, a memory 4060, an interface unit 4070, a control unit 4080, and a feed unit 4090. nutrition. The components shown in FIG. 40 are not integral, so that a mobile device can be implemented with more or less components.

[0587] The wireless communication unit 4010 may include one or more modules that provide wireless communication between the mobile device 4000 and the wireless communication system or between the mobile device and the network on which the mobile device resides. For example, the wireless communication unit 4010 may include a broadcast receiving module 4011, a mobile communication module 4012, a wireless Internet network module 4013, a short range communication module 4014, and a location information module 4015.

[0588] The broadcast receiving module 4011 receives a broadcast signal and/or broadcast-related information from an external broadcast management server via a broadcast channel. Here, the broadcast channel may include a satellite channel and a terrestrial channel. The broadcast administration server may mean a server that generates and transmits broadcast signals and/or broadcast-related information, or a server that receives previously generated broadcast signals and/or broadcast-related information and transmits them to the terminal. The broadcast signal may include a TV broadcast signal, a radio broadcast signal, and a broadcast data signal, and may also include a combined signal in which the broadcast data signal is combined with a TV broadcast signal or a radio broadcast signal.

[0589] Broadcast-related information may refer to information associated with a broadcast channel, a broadcast program, or a broadcast service provider. Broadcast-related information may also be provided via a mobile communication network. In this case, it may be received by the mobile communication module 4012.

[0590] Broadcast-related information may be present in various forms, such as an electronic program schedule or an electronic directory of ESG services.

[0591] The broadcast receiving module 4011 can receive a digital broadcast signal using a digital broadcast system, such as, for example, ATSC, terrestrial DVB-T, satellite DVB-S, MediaFLO Multimedia Direct Line Only, portable DVB-H, terrestrial broadcasting with integrated digital services ISDB-T and similar. Of course, the broadcast receiving module 4011 may be configured to be suitable for other broadcast systems, as well as the digital broadcast system described above.

[0592] The broadcast signal and/or broadcast-related information received via the broadcast information receiving module 4011 may be stored in memory 4060.

[0593] Mobile communication module 4012 transmits and receives a radio signal to and from at least one of a base station, an external terminal, and a server in a mobile communication network. The wireless signal may include various types of data in accordance with the transmission and reception of a voice signal, a video call signal, or a text/multimedia message.

[0594] Wireless Internet module 4013 includes a module for wireless Internet access and may be built-in or external to mobile device 4000. Wireless Internet technology may include wireless LAN (WLAN), wireless broadband Wibro connectivity, worldwide WiMAX broadband wireless access interoperability and HSDPA high-speed packet downlink access.

[0595] Short range communication module 4014 refers to a module for short range communication. Short-range Bluetooth, RFID (Radio Frequency Identification), Infrared Communications (IrDA), UWB (Ultra Wideband), ZigBee, RS-232, RS-485, etc. can be used.

[0596] The location information module 4015 is a module for obtaining location information of the mobile device 4000, and may use the global positioning system module as an example.

[0597] An A/V input unit 4020 serves to input an audio signal and/or a video signal, and may include a camera 4021, a microphone 4022, and the like. The camera 4021 processes image frames, such as still images or moving pictures, obtained by an image sensor in a video call mode or shooting mode. The processed image frame may be displayed in display unit 4051.

[0598] Image frames processed by camera 4021 may be stored in memory 4060 or transmitted externally via wireless communication unit 4010. Two or more 4021 cameras can be provided depending on the use environment.

[0599] The microphone 4022 receives an external audio signal through a microphone in a call mode or recording mode, voice recognition mode, etc., and processes it as electrical voice data. The processed voice data may be converted and output in a form that can be transmitted to a mobile communication base station via the mobile communication module 4012 in a call mode. The 4022 microphone can be implemented with various noise reduction algorithms to remove noise that is generated when receiving an external audio signal.

[0600] The user input block 4030 generates input data for the user to control the operation of the terminal. The user input unit 4030 may be configured as a keypad, dome switch, touch pad (pressure/capacitive type), jog wheel, jog switch, or the like.

[0601] The touch unit 4040 displays the current state of the mobile device 4000, such as the open/closed state of the mobile device 4000, the location of the mobile device 4000, presence or absence of user contact, orientation of the mobile device, and acceleration/deceleration of the mobile device. It detects and generates a touch signal to control the operation of the mobile device 4000. For example, when the mobile device 4000 is moved or tilted, the position or tilt of the mobile device may be detected. And whether power is supplied to the power supply unit 4090 or whether external devices are connected to the interface unit 4070 can be recorded. Meanwhile, the sensor unit 4040 may include a proximity sensor 4041 including NFC near-field communication.

[0602] An output unit 4050 serves to generate output related to vision, hearing, or haptics, and may include a display unit 4051, an audio output unit 4052, an alert unit 4053, and a haptic unit 4054.

[0603] The display unit 4051 displays or outputs information processed by the mobile device 4000. For example, when the mobile device is in a call mode, a UI or graphical user interface GUI associated with the call is displayed. When the mobile device 4000 is in video call mode or shooting mode, the captured and/or received video, UI or GUI is displayed.

[0604] The display unit 4051 may include an LCD liquid crystal display, a TFT LCD liquid crystal display, an OLED organic light emitting diode and flexible display, and a 3D display.

[0605] Some of these displays may be of the transparent or light-transmitting type so that the outside can be seen through them. This may be called a transparent display and a typical example of a transparent display is a transparent OLED TOLED. The rear structure of the display unit 4051 can also be configured as a light transmitting structure. With this structure, the user can see objects located behind the terminal body through the area occupied by the terminal body display unit 4051.

[0606] Two or more display units 4051 may be present depending on the form of implementation of the mobile device 4000. For example, multiple display units may be located at a distance from a single surface or located as an integral part in the mobile device 4000, or may be respectively located on different surfaces.

[0607] When the display unit 4051 and the sensor detecting a touch operation (hereinafter referred to as a 'touch sensor') form a mutually layered structure (hereinafter referred to as a 'touch screen'), the display unit 4051 can be used as an input device in addition to the output device. The touch screen may take the form of, for example, a touch film, a touch sheet, or a touch pad.

[0608] The touch sensor may be configured to convert a change in pressure that is applied to a particular portion of the display unit 4051, or a capacitance that is formed in a particular portion of the display unit 4051, into an electrical input signal. The touch sensor may be configured to detect not only the position and area being touched, but also the pressure at the time of touch.

[0609] If touch input is made through the touch sensor, then corresponding signals are sent to the touch controller. The touch controller processes the signals and then transmits corresponding data to the controller 4080. Accordingly, the control unit 4080 can know which area of the display unit 4051 has been touched and the like.

[0610] The proximity sensor 4041 may be located in an internal area of the mobile device, surrounded by or adjacent to the touch screen. Proximity sensor refers to a sensor that detects the presence or absence of an object that is approaching a predetermined detection surface, or an object in the vicinity, using mechanical force or infrared light without mechanical contact. Proximity sensors have a longer lifespan and higher utilization than contact sensors.

[0611] Examples of the proximity sensor include a transmission type photoelectric sensor, a directional reflection type photoelectric sensor, a specular reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive type proximity sensor, a magnetic type proximity sensor, and an infrared proximity sensor. When the touch screen is capacitive, it is configured to detect the approach of the pointer due to a change in the electric field in accordance with the approach of the pointer. In this case, the touch screen (touch sensor) can be classified as a proximity sensor.

[0612] Further, for convenience of description, the action that causes the positioning of the pointer on the touch screen to be recognized without the pointer touching the touch screen is referred to as “touch by proximity”, and the action of actually touching the pointer of the touch screen is referred to as “touch by contact” . The location on the touch screen that is touched by proximity with the pointer means the location where the pointer corresponds vertically to the touch screen when the pointer is touched closely.

[0613] The proximity sensor detects a proximity touch and a proximity touch sample (e.g., a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, a proximity touch moving state, etc.). e.) Information corresponding to the registered proximity touch operation and the proximity touch sample can be displayed on the touch screen.

[0614] The audio output module 4052 may output audio data received from the wireless communication unit 4010 in a call mode, call signal reception or recording mode, voice recognition mode, broadcast reception mode or the like, or audio data that is stored in the memory 4060. Output module 4052 audio may also output audio signals associated with functions (eg, ringing sound, message receiving sound, etc.) that are performed in the mobile device 4000. The audio output module 4052 may include a receiver, a speaker, and a buzzer.

[0615] The notification module 4053 outputs a signal to notify about the occurrence of an event in the mobile device 4000. Examples of events that are generated in the mobile device include call signal reception, message reception, key input signal, and touch input signal. The notification unit 4053 may output a signal to notify of the occurrence of an event through other forms, such as vibration, in addition to a video signal or an audio signal.

[0616] The video signal or audio signal may also be output through the display unit 4051 or the audio output unit 4052, so that the display unit 4051 and the audio output unit 4052 may be classified as part of the notification unit 4053.

[0617] The haptic module 4054 generates various haptic effects that can be felt by the user. Vibration is a typical example of a haptic effect that is generated by the haptic module 4054. The intensity and sample of the vibration that is generated by the haptic module 4054 is controllable. For example, different vibrations can be synthesized and output or output sequentially.

[0618] In addition to vibration, the haptic module can generate various tactile effects, including stimulation effects, such as the arrangement of pins that move vertically relative to the skin contact surface, outflow forces or air suction forces through outflow or suction, contact with the skin surface, contact with electrodes, electrostatic force and the like, and the effect of reproducing a feeling of cold or heat using an element configured to absorb heat or generate heat.

[0619] The haptic module 4054 can not only deliver the haptic effect through direct contact, but also implements it such that the user can feel the haptic effect through muscle senses, such as fingers or hands. Two or more haptic modules 4054 may be provided in accordance with a configuration aspect of the mobile device 4000.

[0620] Memory 4060 may store a program for operating control unit 4080 and may temporarily store input/output data (eg, phone book, message, still image, video, etc.). The 4060's memory can also store data associated with various vibration and sound samples that are output when a touch is entered through the touch screen.

[0621] The memory 4060 may include a storage medium of at least one of a flash memory type, a hard disk type, a micro-media card type, a card type memory (e.g., SD or XD memory, etc.), memory random access RAM, static random access memory SRAM, read only memory ROM, electrically erasable programmable read only memory EEPROM, programmable read only memory PROM, magnetic memory, optical disk. The mobile device 4000 may operate in connection with a web storage device that performs the function of storing the memory 4060 on the Internet.

[0622] The interface block 4070 serves as a gateway for all external devices connected to the mobile device 4000. The interface block 4070 receives data from the external device, receives power, and transmits data to each component within the mobile device 4000, or provides data transmission, within the mobile device 4000, to an external device. For example, wired/wireless headphone port, external charger port, wired/wireless data port, memory card port, port for connecting to devices equipped with an identification module, audio input/output port, video I/O port, headphone port and similar may be included in interface block 4070.

[0623] The identity module is a chip that stores various information for authenticating rights to use the mobile device 4000, and may include a user identity module UIM, a subscriber identity module SIM, a universal subscriber identity module USIM, and the like. A device equipped with an identification module (hereinafter referred to as an ‘identification device’) can be manufactured in smart card format. Because of this, the identification device can be connected to the terminal 4000 via a port.

[0624] When the mobile terminal 4000 is coupled to an external support, the interface block 4070 may become a passage through which power from the support is supplied to the mobile terminal 4000, or a passage through which various command signals are input from the support by the user. Various command signals or power input from the support may be processed as a signal to recognize that the mobile terminal is correctly installed on the support.

[0625] The control unit 4080 generally controls all operation of the mobile device. For example, it carries out management and processing related to voice calls, data communications, video calls and the like. The control unit 4080 may include a media module 4081 for playing media. The multimedia module 4081 may be implemented in the control unit 4080 or may be implemented separately from the control unit 4080. The control unit 4080, in particular the multimedia module 4081, may include an encoding device 100 and/or a decoding device 200 described above.

[0626] The control unit 4080 may perform a sample recognition process capable of recognizing handwriting input or picture drawing input that is performed on the touch screen as characters and images, respectively.

[0627] The power supply unit 4090 receives external power and internal power under the control of the control unit 4080 and supplies the power that is required for the operation of each component.

[0628] Various embodiments described herein may be implemented in a computer or similar computer-readable recording medium using, for example, software, firmware, or a combination thereof.

[0629] In accordance with the hardware implementation, the embodiments described herein may be implemented using at least one of ASICs, DSPs, DSPDs, PLDs, programmable logic devices, programmable gate arrays FPGAs, processors, controllers, microcontrollers, microprocessors and electrical units to perform functions. In some cases, the embodiments described herein may be implemented by the control unit 4080 itself.

[0630] According to software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more of the functions and operations described in this document. The software code may be implemented in a software application written in a suitable programming language. Here, software code is stored in memory 4060 and can be executed by control unit 4080.

[0631] FIG. 41 is a block diagram illustrating a digital device according to another embodiment.

[0632] Another example of a digital device 4100 may include a broadcast receiving unit 4105, an external device interface unit 4156, a storage unit 4140, a user input interface unit 4150, a control unit 4170, a display unit 4180, an audio output unit 4185, a feed unit 4190 power supply and photographing unit (not shown). Here, the broadcast receiving unit 4105 may include at least one tuner 4110, a demodulation unit 4120, and a network interface unit 4130. However, depending on the case, the broadcast receiving unit 4105 may include a tuner 4110 and a demodulator 4120, but may not include a network interface unit 4130, and vice versa. Also, although not shown, the broadcast receiving unit 4105 may be equipped with a multiplexer to multiplex the signal tuned to by the tuner 4110 and then demodulated by the demodulation unit 4120 and the signal received through the network interface unit 4130. Also, although not shown, the broadcast receiving block 4105 may be equipped with a demultiplexer to demultiplex a multiplexed signal, a demodulated signal, or a signal that has passed through the network interface block 4130.

[0633] The tuner 4110 receives the RF broadcast signal by tuning to a channel selected by the user or all pre-stored channels among the RF broadcast signals received via the antenna. And the tuner 4110 converts the received RF broadcast signal into an intermediate frequency IF signal or a baseband signal.

[0634] For example, if the received RF broadcast signal is a digital broadcast signal, then it is converted to a DIF digital IF signal, and if it is an analog broadcast signal, then it is converted to an analog video or CVBS/SIF baseband audio signal. Those. The 4110 tuner can handle both digital broadcast signals and analog broadcast signals. The analog video or baseband CVBS/SIF audio signal that is output from the tuner 4110 can be directly input to the controller 4170.

[0635] The tuner 4110 may receive a single carrier RF wideband signal in accordance with the ATSC Advanced Television Systems Committee scheme or a multi-carrier RF broadcast signal in accordance with the DVB digital video broadcasting scheme.

[0636] Meanwhile, the tuner 4110 can sequentially tune to and receive the RF broadcast signals of all the broadcast signals stored by the channel storage function among the RF broadcast signals received via the antenna, and convert them into an intermediate frequency signal or a baseband signal.

[0637] The demodulation block 4120 receives and demodulates the digital IF signal DIF converted by the tuner 4110. For example, when the digital IF signal that is output from the tuner 4110 corresponds to the ATSC system, the demodulation block 4120 performs demodulation with 8-part-rejected sideband 8- VSB, for example. Also, demodulation unit 4120 may perform channel decoding. To this end, demodulation unit 4120 may include a trellis decoder, deinterleaver, Reed-Solomon decoder, and the like, and perform trellis decoding, deinterleaving, and Reed-Solomon decoding.

[0638] For example, when the digital IF signal that is output from the tuner 4110 corresponds to a DVB method, the demodulation unit 4120 performs COFDMA orthogonal frequency division coded modulation demodulations, for example. In addition, demodulation unit 4120 may perform channel decoding. To this end, the demodulation unit 4120 may include a convolutional decoder, a deinterleaver, and a Reed-Solomon decoder, and perform convolutional decoding, deinterleaving, and Reed-Solomon decoding.

[0639] The demodulation unit 4120 may output the TS stream signal after demodulation and channel decoding. Here, the streaming signal may be a signal in which a video signal, an audio signal and data are multiplexed. For example, the streaming signal may be an MPEG-2 TS transport stream in which an MPEG-2 video signal, a Dolby AC-3 audio signal, and the like are multiplexed. Specifically, MPEG-2 TS may include a 4-byte header and a 184-byte payload.

[0640] Meanwhile, the demodulation unit 4120 described above may be provided separately in accordance with the ATSC method and the DVB method. Those. the digital device may separately include an ATSC demodulator and a DVB demodulator.

[0641] The streaming signal that is output from the demodulator 4120 may be input to the controller 4170. The control unit 4170 may control demultiplexing, video/audio signal processing, and the like, and control the image that is output through the display unit 4180 and the audio that is output through audio output block 4185.

[0642] The external device interface block 4156 provides an environment in which various external devices interact with the digital device 4100. To this end, the external device interface block 4135 may include an A/V I/O block (not shown) or a wireless communication block. (not shown).

[0643] The external device interface unit 4156 can be wired/wirelessly connected to external devices such as digital versatile disc (DVD), Blu-ray, gaming devices, cameras, camcorders, computers (laptops, tablets), smartphones, Bluetooth devices and cloud. The external device interface unit 4156 transmits a video signal, an audio signal, or a data signal (including an image) that is input externally, via a connected external device, to a digital device controller 4170. The control unit 4170 may control a processed video signal, audio signal, or data signal to be output to a connected external device. To this end, the external device interface unit 4156 may further include an A/V input/output unit (not shown) or a wireless communication unit (not shown).

[0644] The A/V input/output unit may include a USB connector, a CVBS composite video connector, a component connector, an S-video (analog) connector and a DVI (Digital Video Interface) connector, an HDMI (High Definition Multimedia Interface) connector, RGB, D-SUB connector and the like to input video and audio signals from an external device into the 4100 digital device.

[0645] The wireless communication unit can perform short-range wireless communications with other electronic devices. The digital device 4100 can be networked with other electronic devices according to a communication protocol such as Bluetooth, RFID, IrDA, UWB, ZigBee, DLNA, etc. .d.

[0646] And the external device interface unit 4156 may be coupled to at least one of the various set-top boxes and various connectors described above, and perform I/O operations with the set-top box.

[0647] Meanwhile, the external device interface unit 4156 may receive an application or a list of applications in an adjacent external device and transmit it to the control unit 4170 or the storage unit 4140.

[0648] Network interface block 4130 provides an interface for connecting digital device 4100 to a wired/wireless network, including the Internet. The network interface unit 4130 may include, for example, an Ethernet connector or the like for connecting to a wired network, and use communication standards such as, for example, wireless LAN (WLAN) (Wi-Fi), wireless broadband (Wibro), worldwide Wireless Broadband (WiMAX) and High Speed Downlink Packet Access (HSDPA) compatibility to connect to a wireless network.

[0649] The network interface unit 4130 may transmit or receive data with other users or other digital devices through the connected network or another network associated with the connected network. In particular, some content data that is stored in the digital device 4100 may be transferred to another user registered in advance in the digital device 4100, or to a selected user or a selected digital device among other digital devices.

[0650] Meanwhile, the network interface unit 4130 may access a predefined web page through the connected network or another network associated with the connected network. Those. you can connect to a predefined web page via the network and transmit or receive data with the corresponding server. In addition, content or data provided by a content provider or network operator may be received. Those. You can receive content such as movie, advertisement, game, VOD, broadcast signal and related information that is provided by the content provider or network operator through the network. You can also receive update information and update firmware files provided by your network operator. The network interface unit 4130 may also send data to the Internet or to a content provider or network operator.

[0651] Also, the network interface unit 4130 can select and receive a desired application from applications that are open to the public via the network.

[0652] The storage unit 4140 may store a program for processing and controlling each signal in the controller 4170, or may store a post-signal image, an audio signal, or a data signal.

[0653] Also, the storage unit 4140 may perform a function for temporarily storing an image, audio, or data signal that is input from the external device interface unit 4135 or the network interface unit 4130. The storage unit 4140 may store information associated with a predetermined broadcast channel through a channel memory function.

[0654] A storage unit 4140 may store an application or a list of applications that is input from an external device interface unit 4135 or a network interface unit 4130 .

[0655] Also, the storage unit 4140 can store various platforms, which are described later.

[0656] The storage unit 4140 may include at least one storage medium among, for example, a flash memory type, a hard disk type, a micro-media card type, and a card type memory (for example, SD or XD memory, etc. .), RAM and ROM (EEPROM, etc.). The digital device 4100 can play back and provide content files (video files, still image files, music files, document files, application files, etc.) that are stored in the storage unit 4140 to the user.

[0657] FIG. 41 illustrates an embodiment in which the storage unit 4140 is provided separately from the control unit 4170, but the scope of the invention is not limited to this. Those. a storage unit 4140 may be included in a control unit 4170.

[0658] The user input interface block 4150 transmits a signal that is input by the user to the control block 4170, or a signal from the control block 4170 to the user.

[0659] For example, the user input interface unit 4150 may receive a control signal associated with power on/off controls, channel selection, and screen settings from the remote control device 4200 in accordance with various communication methods, such as an RF communication method and an infrared IR method. communications, and process it. Or, the user input interface unit 4150 may process a control signal from the control unit 4170 to be transmitted to the remote control device 4200.

[0660] And the user input interface unit 4150 may transmit a control signal that is input through a local key (not shown), such as a power key, a channel key, a volume key, and a set value key, to the controller 4170.

[0661] The user input interface unit 4150 may transmit a control signal that is input from a registration unit (not shown) that registers the user's gesture to the control unit 4170, or transmit a signal from the control unit 4170 to the registration unit (not shown). Here, the sensing unit (not shown) may include a touch sensor, a voice sensor, a position sensor, and a motion sensor.

[0662] The controller 4170 may demultiplex the stream that is input through the tuner 4110, demodulator 4120, or external device interface 4156, or process the demultiplexed signals to generate and output a signal for video or audio output. The control unit 4170 may include an encoding device and/or a decoding device as described above.

[0663] The image signal processed by the controller 4170 may be input to the display unit 4180 and displayed as an image corresponding to the image signal. Also, the image signal processed by the control unit 4170 can be input to an external output device through the external device interface unit 4156.

[0664] The audio signal processed by the control unit 4170 may be audio that is output to the audio output unit 4185 . Also, the audio signal processed by the controller 4170 can be input to an external output device through an external device interface block 4156.

[0665] Although not illustrated in FIG. 41, the control unit 4170 may include a demultiplexer, an image processor, and the like.

[0666] The control unit 4170 may control the entire operation of the digital device 4100. For example, the control unit 4170 may control the tuner 4110 to tune to an RF broadcast corresponding to a user-selected channel or a previously stored channel.

[0667] The control unit 4170 may control the digital device 4100 through a user command that is input through the user input interface unit 4150 or an internal program. Specifically, a network may be accessed and a desired application or list of applications may be downloaded to the digital device 4100.

[0668] For example, the control unit 4170 controls the tuner 4110 to output a signal of a channel selected in accordance with a predetermined channel selection command received through the user input interface unit 4150, and processes video signals, audio signals, or data signals of the selected channel. The control unit 4170 outputs channel information, etc., selected by the user through the display unit 4180 or the audio output unit 4185 along with the processed image signal or audio signal.

[0669] As another example, in accordance with an external device image playback command received through the user input interface block 4150, the control block 4170 may provide output of a video signal or an audio signal that is input through the external device interface block 4135 from an external device (eg, a camera or video cameras), through a display unit 4180 or an audio output unit 4185.

[0670] Meanwhile, the control unit 4170 may control the display unit 4180 to display an image. For example, the control unit 4170 may control the display unit 4180 to display a broadcast image that is input through the tuner 4110, an external input image that is input through the external device interface unit 4156, and an image that is input through the network interface unit or an image that is stored in storage block 4140. At this time, the image that is displayed in the display unit 4180 may be a still image or a video, and may be a 2D video or a 3D video.

[0671] Also, the control unit 4170 can control playback of content. At this time, the content may be content that is stored in the digital device 4100, or received broadcast content, or external input content that is input from outside. The content may be at least one of a broadcast image, an external input image, an audio file, a still image, a connected web screen, and a document file.

[0672] When navigating to the application view item, the control unit 4170 may control the display of a list of applications or applications that are located on the digital device 4100 or that can be downloaded from an external network.

[0673] The control unit 4170 may control the installation and operation of an application downloaded from an external network, along with various user interfaces. And the control unit 4170 can control an image that is associated with an application to be executed to be displayed in the display unit 4180 by user selection.

[0674] Although not shown in the drawing, it is also possible to optionally equip a channel browsing processing unit to generate a thumbnail image corresponding to a channel signal or an external input signal.

[0675] The channel view processing unit may receive a TS stream signal that is output from a demodulation block 4120 or a stream signal that is output from an external device interface block 4156, extract an image from the input stream signal, and generate a thumbnail image.

[0676] The generated thumbnail image may be input to control unit 4170 as is or as encoded. Also, the generated thumbnail image can be encoded in the form of a stream and input into the control unit 4170. The control unit 4170 may display a thumbnail list including a plurality of thumbnail images at the display unit 4180 using the input thumbnail images. The thumbnail images in a given thumbnail list can be updated sequentially or simultaneously. Accordingly, the user can easily understand the content of a plurality of broadcast channels.

[0677] The display unit 4180 converts the image signals, data signals, OSD signals and the like processed by the controller 4170, or the image signals and data signals received from the external device interface unit 4156 into R, G and B signals, respectively, and generates a signal excitement.

[0678] The display unit 4180 may be a PDP, LCD, OLED, flexible display, 3D display, or the like.

[0679] The display unit 4180 may be configured as a touch screen and used as an input device in addition to an output device.

[0680] The audio output unit 4185 receives a signal processed by the controller 4170, such as a stereo signal, a 3.1 channel signal, or a 5.1 channel signal, and outputs it as audio. The audio output block 4185 may be implemented as various types of speakers.

[0681] Meanwhile, in order to register a user's gesture as described above, a registration unit (not shown) with at least one of a touch sensor, a voice sensor, a position sensor and a motion sensor may be further provided in the digital device 4100. The signal detected by the recording unit (not shown) may be transmitted to the control unit 4170 via the user input interface unit 4150.

[0682] Additionally, a photographing unit (not shown) may be provided for photographing the user. Information of an image photographed by a photographing unit (not shown) may be input to the control unit 4170.

[0683] The control unit 4170 may detect the user's gesture alone or in combination with an image captured by a photographing unit (not shown) or a signal registered by a recording unit (not shown).

[0684] The power supply unit 4190 supplies power to the entire digital device 4100.

[0685] In particular, the power supply unit 4190 may supply power to a control unit 4170, which may be implemented in the form of an SOC, a display unit 4180 for displaying an image, and an audio output unit 4185 for outputting audio.

[0686] To this end, power supply unit 4190 may include a converter (not shown) that converts AC power to DC power. For example, when the display unit 4180 is implemented as a liquid crystal panel with a plurality of backlights, a PWM inverter (not shown) may be further provided for changing brightness or driving dimming.

[0687] Remote control device 4200 provides user input to user input interface unit 4150. For this purpose, the remote control device 4200 may use Bluetooth, RF (radio frequency) communication, infrared IR communication, UWB (Ultra Wide Band), ZigBee (ZigBee) method, etc.

[0688] In addition, the remote control device 4200 may receive an image, audio, or data signal that is output from the user input interface unit 4150, display it on the remote control device 4200, or output voice or vibration.

[0689] The digital device 4100 described above may be a digital broadcast receiver configured to process a fixed or mobile ATSC type or DVB type digital broadcast signal.

[0690] In addition, the digital device in accordance with the invention may omit certain components or additionally include components not illustrated, if necessary. As described above, the digital device may not have a tuner or demodulator, and may also receive and play content through a network interface unit or external device interface unit.

[0691] FIG. 42 is a block diagram illustrating a detailed configuration of the control unit in FIG. from 39 to 41.

[0692] An example control unit may include a demultiplexer 4210, an image processing unit 4220, an OSD generation unit 4240, a mixer 4250, an FRC frame rate converter 4255, and a formatter 4260. In addition, although not illustrated, the control unit may further include a voice processing unit and a data processing unit.

[0693] A demultiplexer 4210 demultiplexes the input stream. For example, the demultiplexer 4210 may demultiplex the input MPEG-2 TS into a video signal, an audio signal, and a data signal. Here, the streaming signal that is input to the demultiplexer 4210 may be a streaming signal that is output from a tuner or demodulator, or an external device interface.

[0694] The image processing unit 4220 performs image processing of the demultiplexed image signal. To this end, the image processing unit 4220 may include a video decoder 4225 and a scaler 4235.

[0695] Video decoder 4225 decodes the demultiplexed video signal, and scaler 4235 scales the resolution of the decoded video signal to be output to the display unit.

[0696] Video decoder 4225 can support various standards. For example, video decoder 4225 functions as an MPEG-2 decoder when the video signal is encoded in the MPEG-2 standard, and as an H.264 decoder when the video signal is encoded in DMB or H.264.

[0697] The video signal decoded by video processing unit 4220 is input to mixer 4250.

[0698] The OSD generating unit 4240 generates OSD data in accordance with user input or independently. For example, the OSD generating unit 4220 generates data for displaying various data on the screen of the display unit 4180 in graphical or text form based on a control signal of the user input interface unit. The generated OSD data includes various data such as the user interface screen of the digital device, various menu screens, widgets, icons, and view rate information.

[0699] The OSD generating unit 4240 may generate data for displaying subtitles of broadcast images or broadcast information based on the EPG.

[0700] The mixer 4250 mixes the OSD data generated by the OSD generating unit 4240 and the image signal processed by the video processing unit and provides it to the formatter 4260. Because the decoded video signal and OSD data are mixed, the OSD appears superimposed on the broadcast video or external input video.

[0701] The FRC frame rate converter 4255 converts the frame rate of the output video. For example, frame rate converter 4255 may convert an input 60Hz image frame rate to a frame rate of, for example, 120Hz or 240Hz depending on the output frequency of the display unit. As described above, various methods may exist in the method for converting the frame rate. For example, when the frame rate converter 4255 converts the frame rate from 60Hz to 120Hz, the same first frame is inserted between the first frame and the second frame, or a third frame predicted from the first frame and the second frame is inserted between the first frame and the second frame. As another example, when the frame rate converter 4255 converts the frame rate from 60Hz to 240Hz, then three identical frames or predicted frames can be inserted between the existing frames. If a separate frame conversion is not performed, the frame rate conversion block 4255 may be bypassed.

[0702] Formatter 4260 changes the output of frame rate converter 4255 to match the output format to the display unit. For example, formatter 4260 may output R, G, and B data signals, and the three R, G, and B data signals may be output as LVDS or mini-LVDS. Also, when the output of the frame rate converter 4255 is a 3D video signal, the formatter 4260 can support a 3D service through the display unit by configuring the output as a 3D format in accordance with the output format of the display unit.

[0703] An audio processing unit (not shown) in the control unit may perform audio processing of the demultiplexed audio signal. The audio processing unit (not shown) can support various audio formats. For example, even when an audio signal is encoded in formats such as MPEG-2, MPEG-4, AAC, HE-AAC, AC-3, BSAC, the audio processing unit may be provided with a decoder that matches the formats.

[0704] Also, an audio processing unit (not shown) in the control unit may process base control, treble control, volume control, and the like.

[0705] A data processing unit (not shown) in the control unit may perform data processing of the demultiplexed data signal. For example, the data processing unit may decode the demultiplexed data signal as it is encoded. Here, the encoded data signal may be EPG information including broadcast information such as a broadcast start time and a broadcast end time of a broadcast program on each channel.

[0706] Meanwhile, the digital device described above is an example in accordance with the invention, and each component may be integrated, added or omitted depending on the technical description of the actual digital device. Those. if necessary, two or more components can be combined into one component, or one component can be subdivided into two or more components. Also, the function that is performed in each block serves to describe an embodiment of the invention, and the specific operation or device does not limit the scope of the invention.

[0707] The digital device may be an image signal processing device that processes an image signal that is stored in the device or an input image. As another example of an image signal processing apparatus, a set-top box STB, a DVD player, a Blu-ray player, a gaming device, a computer and the like may be further cited, from which the display unit 4180 and the audio output unit 4185 shown in FIG. . 41.

[0708] FIG. 43 is a diagram illustrating an example in which a digital device screen displays a main image and a sub-image simultaneously, according to an embodiment.

[0709] A digital device in accordance with an embodiment may simultaneously display a main image 4310 and a sub-image or sub-image 4320 on a screen 4300. The main image 4310 may be referred to as a first image, and the sub-image 4320 may be referred to as a second image. The main picture 4310 and sub picture 4320 may include and are not limited to video, still picture, EPG, GUI, OSD and the like. The main image 4310 may mean an image that is relatively smaller in size than the electronic device screen 4300 while being simultaneously displayed on the electronic device screen 4300 along with the sub image 4320, and may be referred to as picture-in-picture PIP. In FIG. 43, the main image 4310 is displayed on the upper left side of the digital device screen 4300, but the location at which the main image 4310 is displayed is not limited to this, and the main image 4310 may be displayed at any location within the digital device screen 4300.

[0710] The main image 4310 and the auxiliary image 4320 may be associated with each other directly or indirectly. As an example, the main image 4310 may be a streaming video, and the sub image 4320 may be a GUI that sequentially displays thumbnails of video elements including information like the streaming video. As another example, the main image 4310 may be a broadcast image and the sub image 4320 may be an EPG. As another example, the main image 4310 may be a broadcast image and the sub image 4320 may be a GUI. Examples of the main image 4310 and the sub image are not limited to this.

[0711] In one embodiment, the main image 4310 is a broadcast image received via the broadcast channel, and the auxiliary image 4320 may be information that is associated with the broadcast image received via the broadcast channel. Information associated with a broadcast image received via a broadcast channel may include, for example, EPG information including an integrated channel schedule and detailed broadcast program information, and broadcast program overview information, but is not limited to it.

[0712] In another embodiment, the main image 4310 is a broadcast image received via a broadcast channel, and the sub image 4320 may be an image that is generated based on information previously stored in a digital device. The image generated based on information previously stored in the digital device may include, for example, a basic user interface UI of the EPG, basic channel information, an image resolution manipulation UI, and a bedtime reservation UI, and is not limited to this.

[0713] In another embodiment, the main image 4310 is a broadcast image received via a broadcast channel, and the sub image 4320 may be information associated with a broadcast image received via a network. Information associated with a broadcast image received via a network may, for example, be information obtained through a network-based search engine. In particular, for example, information related to a character currently displayed on the main image 4310 may be obtained through a web-based search engine.

[0714] However, the example is not limited to this, and information related to the broadcast image received via the network can be obtained by using, for example, an AI artificial intelligence system. Specifically, for example, the estimated location on a place map that is currently displayed on the main image 4310 may be obtained by using network-based deep learning, and the digital device may receive information about the estimated location on a place map that is currently displayed in the main image 4310, via the network.

[0715] The digital device according to an embodiment may receive at least one of the main image image information 4310 and the sub image image information 4320 from outside. The image information of the main image 4310 may include, for example, a broadcast signal received via a broadcast channel, source code information of the main image 4310, and IP packet (Internet Protocol Packet) information of the main image 4310 received via a network, but is not limited to it. Similarly, the image information of the sub-picture 4320 includes, for example, a broadcast signal received via a broadcast channel, source code information of the sub-picture 4320, IP packet information of the sub-picture 4320 received via a network, etc., but is not limited to this. . The digital device may decode and use image information of the main image 4310 received externally or image information of the sub-image 4320. However, in some cases, the digital device may store image information of the main image 4310 or image information of the sub-image 4320 internally.

[0716] The digital device may display a main image 4310 and a sub-image 4320 on a digital device screen 4300 based on image information of the main image 4310 and information associated with the sub-image 4320.

[0717] In one example, the digital device decoding apparatus 200 includes a main image decoding apparatus and a sub-image decoding apparatus, and the main image decoding apparatus and the sub-image decoding apparatus may respectively decode the main image image information 4310 and the sub-image image information 4320. The renderer includes a main video renderer (first renderer) and a auxiliary video renderer (second renderer). The main image renderer may cause the main image 4310 to be displayed in a first zone of the digital device screen 4300 based on information decoded by the main image decoding device, and the sub image renderer may cause the sub image 4320 to be displayed in a second zone of the digital device screen 4300 based on information decoded by the sub image decoding device. Images.

[0718] In another example, the digital device decoding apparatus 200 may decode the image information of the main image 4310 and the image information of the sub-image. Based on the information decoded by the decoding device 200, the renderer can process the main image 4310 and the sub-image 4320 together for simultaneous display on the digital device screen 4300.

[0719] That is according to this document, a method for processing an image service in a digital device may be provided. The image service processing method may comprise the steps of: receiving image information, decoding a (main) image based on the image information, rendering or displaying the decoded image in a first zone on a display, and rendering or displaying a sub-image in a second zone on a display. Here, the step of decoding the first image may follow a decoding procedure in the decoding apparatus 200 in accordance with FIG. 3 described above. For example, as described above, the step of decoding the first picture may include the steps of extracting prediction samples for the current block based on inter-frame or intra-frame prediction, extracting residual samples for the current block based on the received residual information, and generating reconstructed samples based on prediction samples and/or residual samples. Additionally, decoding the first image may include performing an in-loop filtering procedure on the reconstructed picture including the reconstructed samples.

[0720] For example, the auxiliary display may be an electronic program schedule, an OSD menu, or a GUI graphical user interface. For example, image information may be received via a broadcast network, and information regarding the sub-image may be received via the broadcast network. For example, image information may be received via a communication network, and information regarding the sub-image may be received via a communication network. For example, image information may be received via a broadcast network, and information regarding the sub-image may be received via a communication network. For example, image information may be received via a broadcast network or a communication network, and information regarding the sub-image may be stored in a storage medium in a digital device.

[0721] In the above embodiments, the elements and characteristics of the invention have been combined in a specific form. Each of the elements or characteristics may be considered optional unless otherwise expressly stated. Each of the elements or characteristics may be implemented in a form that does not need to be combined with other elements or characteristics. In addition, some of the elements and/or characteristics may be combined to form an embodiment of the invention. The sequence of operations described in the embodiments of the invention may be changed. Some of the elements or features of an embodiment may be included in another embodiment or may be replaced by corresponding elements or features of another embodiment. It is apparent that an embodiment may be created by combining claims that are not explicitly cited in the claims and may be incorporated as a new claim by post-filing amendments.

[0722] An embodiment in accordance with the invention may be implemented by various means, such as hardware, firmware, software, or a combination thereof. If implemented in hardware, an embodiment of the invention may be implemented using one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), programmable gate arrays (FPGA), processors, controllers, microcontrollers, microprocessors, etc.

[0723] If implemented through firmware or software, an embodiment of the invention may be implemented in the form of a module, procedure, or function to perform the above functions or operations. The software code may be stored in memory and executed by the processor. The memory may be located internally or externally to the processor and may communicate with the processor through a variety of known means.

[0724] Those skilled in the art will appreciate that the invention may be embodied in other specific forms without departing from the inherent characteristics of the invention. Accordingly, the detailed description should not be construed as limiting, but rather as illustrative in all respects. The scope of the invention shall be determined by a reasonable analysis of the appended claims, and all modifications within the equivalent scope of the invention shall be included within the scope of the invention.

Industrial applicability

[0725] The above preferred embodiments of the invention have been disclosed for purposes of illustration, and those skilled in the relevant art may improve, modify, substitute, and add various other embodiments without departing from the technical spirit and scope of the invention as disclosed in the appended claims.

1. A method of processing a video signal based on history-based motion vector prediction, comprising the steps of:

configuring a merge candidate list based on a spatial neighbor and a temporal neighbor of the current block;

adding a history-based merge candidate of the current block to the list of merge candidates;

obtaining a merge index indicating a merge candidate that is used for inter-frame prediction of the current block in the merge candidate list;

generating a prediction sample of the current block based on the motion information of the merge candidate indicated by the merge index; And

updating the list of history-based merger candidates based on the traffic information,

wherein, the history-based merger candidate is added to the merger candidate list when it has motion information different from that of the predetermined merger candidate from the merger candidates included in the merger candidate list.

2. The method of claim 1, wherein the size of the history-based merge candidate list is determined based on the maximum number of merge candidates in the merge candidate list.

3. The method of claim 1, wherein the size of the history-based merge candidate list is determined to be a value equal to 1, which is subtracted from the maximum number of merge candidates in the merge candidate list.

4. The method of claim 1, wherein the size of the list of history-based merger candidates is determined to be 5.

5. The method of claim 1, wherein the history-based merger candidate is added to the list of merger candidates when the history-based merger candidate includes movement information other than the movement information of a specific number of predetermined merger candidates among merger candidates included in the list of merger candidates.

6. The method of claim 1, wherein the history-based merge candidate is added to the list of merge candidates when the history-based merge candidate includes motion information other than the motion information of specific spatial merge candidates included in list of merger candidates.

7. The method of claim 6, wherein the history-based merge candidate is retrieved from a predetermined number of candidates in the list of history-based merge candidates.

8. A video signal processing device based on interframe prediction, comprising:

a memory configured to store video signals; And

processor combined with memory,

wherein the processor is configured to:

configuring a merge candidate list based on a spatial neighbor and a temporal neighbor of the current block;

adding the history-based merge candidate of the current block to the list of merge candidates;

obtaining a merge index indicating a merge candidate that is used for inter-frame prediction of the current block in the merge candidate list;

generating a prediction sample of the current block based on information about the movement of the merger candidate indicated by the merger index; And

updating the list of history-based merger candidates based on traffic information,

wherein, the history-based merger candidate is added to the merger candidate list when it has motion information different from that of the predetermined merger candidate from the merger candidates included in the merger candidate list.

9. The apparatus of claim 8, wherein the size of the history-based merge candidate list is determined based on the maximum number of merge candidates in the merge candidate list.

10. The apparatus of claim 8, wherein the size of the history-based merge candidate list is determined to be a value equal to 1, which is subtracted from the maximum number of merge candidates in the merge candidate list.

11. The apparatus of claim 8, wherein the size of the history-based merge candidate list is determined to be 5.

12. The apparatus of claim 8, wherein the history-based merger candidate is added to the list of merger candidates when the history-based merger candidate includes movement information other than the movement information of a specific number of predetermined merger candidates among merger candidates included in the list of merger candidates.

13. The apparatus of claim 8, wherein the history-based merge candidate is added to the list of merge candidates when the history-based merge candidate includes motion information other than the motion information of specific spatial merge candidates included in list of merger candidates.

14. The apparatus of claim 13, wherein the history-based merge candidate is retrieved from a predetermined number of candidates in the list of history-based merge candidates.

Claims (30)

1. A device for decoding images, comprising memory and at least one processor, wherein at least one processor is configured to: configuring a merge candidate list based on a spatial neighbor and a temporal neighbor for the current block; adding the history-based motion vector predictor (HMVP) candidate of the current block to the merge candidate list; And generating a prediction sample of the current block based on information about the movement of a merger candidate included in the list of merger candidates, wherein the list of HMVP candidates is updated based on information about the movement of the merger candidate, and wherein the HMVP candidate is added to the merge candidate list based on the fact that the HMVP candidate has motion information different from the motion information of predetermined spatial merge candidates included in the list of merge candidates, wherein the predetermined spatial merge candidates are only part of the spatial merge candidates in the merge candidate list. 2. The device of claim 1, wherein the maximum size of the HMVP candidate list is less than the maximum size of the merge candidate list. 3. The device according to claim 2, in which the maximum size of the HMVP candidate list is less than the maximum size of the merge candidate list by 1. 4. The device according to claim 2, in which the maximum size of the HMVP candidate list is defined as 5. 5. The device according to claim 1, wherein the number of predetermined spatial fusion candidates is defined as 2. 6. The apparatus of claim 1, wherein the predetermined spatial merge candidates include a left neighbor spatial merge candidate and a top neighbor spatial merge candidate for the current block. 7. An image encoding device comprising a memory and at least one processor, wherein the at least one processor is configured to: configuring a merge candidate list based on a spatial neighbor and a temporal neighbor for the current block; adding the history-based motion vector predictor (HMVP) candidate of the current block to the merge candidate list; generating a prediction sample of the current block based on information about the movement of a merger candidate included in the list of merger candidates; And encoding a merge index indicating a merge candidate used to generate a prediction sample, and wherein the list of HMVP candidates is updated based on information about the movement of the merger candidate, and wherein the HMVP candidate is added to the merge candidate list based on the fact that the HMVP candidate has motion information different from the motion information of predetermined spatial merge candidates included in the list of merge candidates, wherein the predetermined spatial merge candidates are only part of the spatial merge candidates in the merge candidate list. 8. The apparatus of claim 7, wherein the maximum size of the HMVP candidate list is less than the maximum size of the merge candidate list. 9. The device according to claim 8, wherein the maximum size of the HMVP candidate list is less than the maximum size of the merge candidate list by 1. 10. The device according to claim 8, wherein the maximum size of the HMVP candidate list is defined as 5. 11. The apparatus of claim 7, wherein the number of predetermined spatial fusion candidates is set to 2. 12. The apparatus of claim 7, wherein the predetermined spatial merge candidates include a left adjacent spatial merge candidate and an upper adjacent spatial merge candidate for the current block. 13. A method for transmitting a bit stream generated by an image encoding method, wherein the image encoding method comprises the steps of: configuring a merge candidate list based on a spatial neighbor and a temporal neighbor for the current block; adding the history-based motion vector predictor (HMVP) candidate of the current block to the merge candidate list; generating a prediction sample of the current block based on information about the movement of a merger candidate included in the list of merger candidates; And encoding a merge index indicating a merge candidate used to generate a prediction sample into a bit stream, wherein the list of HMVP candidates is updated based on information about the movement of the merger candidate, and wherein the HMVP candidate is added to the merge candidate list based on the fact that the HMVP candidate has motion information different from the motion information of predetermined spatial merge candidates included in the list of merge candidates, wherein the predetermined spatial merge candidates are only part of the spatial merge candidates in the merge candidate list.

Images