TW201924333A - Encoding device, decoding device, encoding method, and decoding method - Google Patents

Abstract

An encoding device is provided with a circuit and a memory. When a process for correcting the luminance of a prediction image must be applied to a block for determination that is at least one of an object block and an adjacent block adjoining the object block, the circuit uses the memory to apply a deblocking filtering process to the boundary between the object block and the adjacent block. When the luminance correction process does not need to be applied to the block for determination, the circuit does not apply the deblocking filtering process to the boundary between the object block and the adjacent block.

Description

Encoding device, decoding device, encoding method and decoding method

FIELD OF THE INVENTION The present invention relates to an encoding apparatus, a decoding apparatus, an encoding method, and a decoding method.

BACKGROUND OF THE INVENTION In the past, H.265 was used as a specification for encoding moving images. H.265 is also known as HEVC (High Efficiency Video Coding).
Advanced technical literature

Non-patent literature
[Non-Patent Document 1] H.265 (ISO/IEC 23008-2 HEVC (High Efficiency Video Coding))

SUMMARY OF THE INVENTION Problems to be Solved by the Invention This type of encoding method and decoding method are expected to improve the image quality of a decoded image.

An object of the present invention is to provide an encoding apparatus, a decoding apparatus, an encoding method, and a decoding method capable of improving the image quality of a decoded image.

Means for Solving the Problem A coding apparatus according to an aspect of the present invention includes a circuit and a memory, and the circuit uses the memory to block at least one of a target block and an adjacent block adjacent to the target block. When the brightness correction processing of the prediction image is applied to the determination block, the deblocking filtering process is applied to the boundary between the target block and the adjacent block, and when the brightness correction processing is not applied to the determination block, The aforementioned deblocking filtering process is applied not to the aforementioned boundary.

A decoding apparatus according to an aspect of the present invention includes a circuit and a memory, wherein the circuit uses the memory to determine at least one of a target block and an adjacent block adjacent to the target block, that is, a determination block. When the brightness correction processing of the predicted image is applied, the deblocking filtering process is applied to the boundary between the target block and the adjacent block, and when the brightness correction processing is not applied to the determination block, the foregoing is not applied to the boundary. Deblocking filtering.

Furthermore, such comprehensive or specific aspects may also be implemented by systems, devices, methods, integrated circuits, computer programs, or non-transitory recording media such as computer readable CD-ROMs, or by systems, devices, and methods. Any combination of integrated circuits, computer programs, and recording media.

Advantageous Effects of Invention The present invention can provide a decoding apparatus, an encoding apparatus, a decoding method, or an encoding method capable of improving the image quality of a decoded image.

EMBODIMENT OF THE INVENTION The coding apparatus according to one aspect of the present invention includes a circuit and a memory, and the circuit uses the memory to block at least one of a target block and an adjacent block adjacent to the target block. When the brightness correction processing of the prediction image is applied to the determination block, the deblocking filtering process is applied to the boundary between the target block and the adjacent block, and when the brightness correction processing is not applied to the determination block, The aforementioned deblocking filtering process is applied not to the aforementioned boundary.

According to this, the encoding apparatus performs deblocking filtering processing when the brightness correction processing is applied to the determination block. Thereby, since the subjectively conspicuous block noise can be reduced, the image quality of the decoded image can be improved.

For example, when the luminance correction processing is applied to one or both of the target block and the adjacent block, the deblocking filtering process may be applied to the boundary; and for any one of the target block and the adjacent block, When the above-described brightness correction processing is not applied, the above-described deblocking filtering processing may be applied without the above boundary.

For example, regardless of whether or not the luminance correction processing is applied to the target block, when the luminance correction processing is applied to the adjacent block, the deblocking filtering processing is applied to the boundary, and the luminance correction processing is not applied to the adjacent block. The foregoing deblocking filtering process may not be applied to the foregoing boundary.

According to this, the encoding apparatus can suppress excessive deblocking filtering processing.

For example, the brightness correction processing may be LIC (Local Illumination Compensation) processing.

For example, when the brightness correction processing is applied to the determination block, the Bs indicating the boundary strength of the boundary is set to a value other than 0, and the deblocking filtering process may be applied to the boundary; When the block is not subjected to the above-described brightness correction processing, by setting the Bs to 0, the above-described deblocking filtering process may be applied without the above boundary.

A decoding apparatus according to an aspect of the present invention includes a circuit and a memory, wherein the circuit uses the memory to determine at least one of a target block and an adjacent block adjacent to the target block, that is, a determination block. When the brightness correction processing of the predicted image is applied, the deblocking filtering process is applied to the boundary between the target block and the adjacent block, and when the brightness correction processing is not applied to the determination block, the foregoing is not applied to the boundary. Deblocking filtering.

According to this, the decoding apparatus performs deblocking filtering processing when the brightness correction processing is applied to the determination block. Thereby, since the subjectively conspicuous block noise can be reduced, the image quality of the decoded image can be improved.

For example, when the luminance correction processing is applied to one or both of the target block and the adjacent block, the deblocking filtering process may be applied to the boundary; and for any one of the target block and the adjacent block, When the above-described brightness correction processing is not applied, the above-described deblocking filtering processing may be applied without the above boundary.

For example, regardless of whether or not the luminance correction processing is applied to the target block, when the luminance correction processing is applied to the adjacent block, the deblocking filtering processing is applied to the boundary, and the luminance correction processing is not applied to the adjacent block. The foregoing deblocking filtering process may not be applied to the foregoing boundary.

According to this, the encoding apparatus can suppress excessive deblocking filtering processing.

For example, the brightness correction processing may be LIC (Local Illumination Compensation) processing.

For example, when the brightness correction processing is applied to the determination block, the Bs indicating the boundary strength of the boundary is set to a value other than 0, and the deblocking filtering process may be applied to the boundary; When the block is not subjected to the above-described brightness correction processing, by setting the Bs to 0, the above-described deblocking filtering process may be applied without the above boundary.

In the encoding method according to an aspect of the present invention, when the target block and the block adjacent to at least one of the adjacent blocks of the target block, that is, the brightness correction processing for applying the prediction image to the determination block, the object is The deblocking filtering process is applied to the boundary between the block and the adjacent block, and when the brightness correction processing is not applied to the determination block, the deblocking filtering process is not applied to the boundary.

According to this, the encoding method performs deblocking filtering processing when the brightness correction processing is applied to the determination block. Thereby, since the subjectively conspicuous block noise can be reduced, the image quality of the decoded image can be improved.

In the decoding method according to an aspect of the present invention, when at least one of the target block and the adjacent block adjacent to the target block, that is, the brightness correction processing of the prediction image is applied to the determination block, the object is The deblocking filtering process is applied to the boundary between the block and the adjacent block, and when the brightness correction process is not applied to the block for determination, the deblocking filtering process is not applied to the boundary.

According to this, the decoding method performs deblocking filtering processing when the brightness correction processing is applied to the determination block. Thereby, since the subjectively conspicuous block noise can be reduced, the image quality of the decoded image can be improved.

In addition, such comprehensive or specific aspects can also be realized by systems, devices, methods, integrated circuits, computer programs, or non-transitory recording media such as computer readable CD-ROMs, or by systems, devices, The method, the integrated circuit, the computer program, and any combination of recording media are implemented.

The embodiment will be specifically described below with reference to the drawings.

Furthermore, the embodiments described below represent comprehensive or specific examples. The numerical values, shapes, materials, constituent elements, arrangement positions and connection forms of the constituent elements, the order of the steps and the steps, and the like are exemplified in the following embodiments, and the gist of the invention is not limited to the scope of the patent application. Further, among the constituent elements of the following embodiments, the constituent elements of the independent request item indicating the highest-order concept are not described as arbitrary constituent elements.
(Embodiment 1)

First, an outline of the first embodiment will be described as an example of an encoding device and a decoding device which are applicable to the processing and/or configuration described in each aspect of the present invention to be described later. However, the first embodiment is only an example of an encoding device and a decoding device to which the processing and/or configuration described in each aspect of the present invention can be applied, and the processing and/or configuration described in each aspect of the present invention may be The encoding device and the decoding device are implemented in different embodiments.

In the case where the processing and/or configuration described in each aspect of the present invention is applied to the first embodiment, for example, any of the following may be employed.
(1) In the coding apparatus or the decoding apparatus according to the first embodiment, the constituent elements corresponding to the constituent elements described in the respective aspects of the present invention among the plurality of constituent elements constituting the encoding apparatus or the decoding apparatus are replaced by the invention. The constituent elements described in each aspect;
(2) The encoding device or the decoding device according to the first embodiment is configured as a component of a plurality of constituent elements constituting the encoding device or the decoding device, and is arbitrarily changed by adding, replacing, or deleting a function or a process. The constituent elements corresponding to the constituent elements described in the various aspects of the present invention are replaced by the constituent elements described in the various aspects of the invention;
(3) The method implemented by the encoding device or the decoding device according to the first embodiment, after the processing is added, and/or one of the plurality of processes included in the method is subjected to any change such as replacement or deletion, The processing described in connection with the aspects of the present invention replaces the processing described in the various aspects of the invention;
(4) A part of a plurality of constituent elements constituting the coding apparatus or the decoding apparatus according to the first embodiment, and constituent elements described in the aspects of the present invention, and the constituent elements described in the aspects of the present invention. The constituent elements of a part of the functions provided, or the constituent elements of a part of the processing performed by the constituent elements described in the various aspects of the present invention are combined;
(5) A component constituting a part of the functions of the plurality of components constituting the coding device or the decoding device according to the first embodiment, or a plurality of components constituting the coding device or the decoding device according to the first embodiment One of the requirements constitutes a part of the processing required by the requirements, and the constituent elements described in the aspects of the present invention, and the constituent elements of the functions of the constituent elements described in the aspects of the present invention, or Carrying out a combination of constituent elements of a part of the processing performed by the constituent elements described in the aspects of the present invention;
(6) The method performed by the encoding device or the decoding device according to the first embodiment, the processing corresponding to the processing described in each aspect of the present invention among the plurality of processings included in the method is replaced with the state of the invention. The treatment described;
(7) One of a plurality of processes included in the method performed by the encoding device or the decoding device according to the first embodiment is combined with the processing described in each aspect of the present invention.

Furthermore, the embodiments of the processes and/or configurations described in the various aspects of the present invention are not limited to the above examples. For example, in the moving image/image encoding device or the moving image/image decoding device disclosed in the first embodiment, the device is used for a different purpose, or the processing and/or configuration described in each aspect is performed separately. can. Further, it may be implemented by combining the processes and/or configurations described in different aspects.
[Summary of coding device]

First, an outline of an encoding apparatus according to the first embodiment will be described. Fig. 1 is a block diagram showing a functional configuration of an encoding apparatus 100 according to the first embodiment. The encoding device 100 is a moving image/image encoding device that encodes a moving image/image in units of blocks.

As shown in FIG. 1, the encoding apparatus 100 is an apparatus for encoding an image in units of blocks, and includes a dividing unit 102, a subtracting unit 104, a converting unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, and an inverse conversion. The unit 114, the adding unit 116, the block memory 118, the loop filter unit 120, the frame memory 122, the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128.

The encoding device 100 is realized by, for example, a general purpose processor and a memory. At this time, when the processor executes the software program stored in the memory, the processor functions as the division unit 102, the subtraction unit 104, the conversion unit 106, the quantization unit 108, the entropy coding unit 110, the inverse quantization unit 112, and the inverse conversion unit 114. The addition unit 116, the loop filter unit 120, the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128 function. Further, the encoding device 100 may correspond to the division unit 102, the subtraction unit 104, the conversion unit 106, the quantization unit 108, the entropy coding unit 110, the inverse quantization unit 112, the inverse conversion unit 114, the addition unit 116, and the loop filter unit 120. The intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128 are realized by a dedicated one or more electronic circuits.

Each constituent element included in the encoding device 100 will be described below.
[Division Department]

The division unit 102 divides each picture included in the input moving image into a plurality of blocks, and outputs each block to the subtraction unit 104. For example, the dividing unit 102 first divides the picture into blocks of a fixed size (for example, 128 ́128). This fixed size block is sometimes referred to as a coding tree unit (CTU). Then, the segmentation unit 102 divides the blocks of fixed size into blocks of variable size (for example, 64 ́64) according to the recursive quadtree and/or binary tree segmentation. . The variable size block is sometimes referred to as a coding unit (CU), a prediction unit (PU), or a conversion unit (TU). Furthermore, in this embodiment, it is not necessary to distinguish between CU, PU, and TU, and some or all of the blocks in the picture may be processing units of CU, PU, and TU.

Fig. 2 is a view showing an example of block division in the first embodiment; In FIG. 2, the solid line indicates the block boundary of the quaternary tree block division, and the broken line indicates the block boundary of the binary tree block division.

Here, block 10 is a 128 ́128 pixel square block (128 ́128 block). The 128 ́128 block 10 is first partitioned into 4 square 64 ́64 blocks (quaternary tree block partitioning).

The upper left 64 ́64 block is further vertically divided into two rectangular 32 ́64 blocks, and the left 32 ́64 block is further vertically divided into two rectangular 16 ́64 blocks (binary tree block segmentation). As a result, the upper left 64 ́64 block is divided into two 16 ́64 blocks 11, 12 and 32 ́64 blocks 13.

The upper right 64 ́64 block is horizontally divided into two rectangular 64 ́32 blocks 14, 15 (binary tree block segmentation).

The lower left 64 ́64 block is divided into 4 square 32 ́32 blocks (quaternary tree block partitioning). Among the four 32 ́32 blocks, the upper left block and the lower right block are further divided. The upper left 32 ́32 block is vertically divided into two rectangular 16 ́32 blocks, and the right 16 ́32 block is further horizontally divided into two 16 ́16 blocks (binary tree block segmentation). The lower right 32 ́32 block is horizontally divided into two 32 ́16 blocks (binary tree block partitioning). As a result, the lower left 64 ́64 block is divided into 16 ́32 blocks 16, two 16 ́16 blocks 17, 18, two 32 ́32 blocks 19, 20 and two 32 ́16 blocks 21, 22 .

The lower right 64 ́64 block 23 is undivided.

As above, in FIG. 2, the block 10 is divided into 13 variable-sized blocks 11 to 23 according to the recursive quadtree and binary tree block division. Such segmentation is sometimes referred to as QTBT (quad-tree plus binary tree) segmentation.

Furthermore, in FIG. 2, one block is divided into four or two blocks (quaternary tree or binary tree block division), but the division is not limited thereto. For example, one block can also be divided into three blocks (three-dimensional tree block division). This type of partitioning including ternary tree block partitioning is sometimes referred to as MBT (multi-type tree) partitioning.
[Reduction Department]

The subtraction unit 104 subtracts the prediction signal (predicted sample) from the original signal (original sample) in the block unit divided by the division unit 102. In summary, the subtraction unit 104 calculates a prediction error (also referred to as a residual) of a coding target block (hereinafter referred to as a current block). Then, the subtraction unit 104 outputs the calculated prediction error to the conversion unit 106.

The original signal is an input signal of the encoding device 100, which is a signal (for example, a luma signal and two chroma signals) indicating images of the respective pictures constituting the moving image. The signal representing the image is sometimes referred to as a sample below.
[conversion department]

The conversion unit 106 converts the prediction error in the spatial domain into a conversion coefficient in the frequency domain, and outputs the conversion coefficient to the quantization unit 108. Specifically, the conversion unit 106 performs a predetermined discrete cosine transform (DCT) or discrete sine transform (DST) on, for example, a prediction error in the spatial domain.

Furthermore, the conversion unit 106 may adaptively select a conversion type from a plurality of types of conversions, and convert the prediction error into a conversion coefficient by using a transform basis function corresponding to the selected conversion type. This type of conversion is sometimes called EMT (explicit multiple core transform) or AMT (adaptive multiple transform).

A plurality of types of conversion include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. FIG. 3 is a table showing basis functions corresponding to respective conversion types. In Fig. 3, N represents the number of input pixels. The choice of the type of conversion from among the plurality of types of conversion may depend, for example, on the type of prediction (intra prediction and inter prediction) or on the intra prediction mode.

This class indicates information applicable to EMT or AMT (for example, called AMT flag), and information indicating the type of conversion selected is signaled at the CU level. Furthermore, the signalization of such information need not be limited to the CU level, but may be other levels (such as sequence level, picture level, slice level, block level or CTU level).

Further, the conversion unit 106 may reconvert the conversion coefficient (conversion result). This type of retransformation is sometimes referred to as AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the conversion unit 106 performs retransformation one by one by sub-blocks (for example, 4 ́4 sub-blocks) included in the block corresponding to the conversion coefficient of the intra prediction error. Information indicating whether NSST is applicable or not, and information about the conversion matrix used for NSST is signaled at the CU level. Furthermore, the signalization of such information need not be limited to the CU level, but may be other levels (such as sequence level, picture level, slice level, block level or CTU level).

Here, Separable conversion refers to a method of performing multiple conversions in which only the dimensions of the input are separated in each direction, and Non-Separable conversion means that when the input is multi-dimensional, two or more are used. The dimension is treated as a one-dimensional and one-time conversion.

For example, as an example of the Non-Separable conversion, when a block of 4 ́4 is input, it is regarded as an array having 16 elements, and for the array, conversion processing is performed with a conversion matrix of 16 ́16. .

Similarly, after the 4 ́4 input block is regarded as one array having 16 elements, a plurality of Givens rotation (Hypercube Givens Transform) conversion is performed for the array, which is also Non. -Separable conversion example.
[Quantization Department]

The quantization unit 108 quantizes the conversion coefficients output from the conversion unit 106. Specifically, the quantization section 108 scans the conversion coefficient of the current block in a predetermined scanning order, and quantizes the conversion coefficient according to a quantization parameter (QP) corresponding to the scanned conversion coefficient. Then, the quantization unit 108 outputs the quantized conversion coefficients (hereinafter referred to as quantized coefficients) of the current block to the entropy coding unit 110 and the inverse quantization unit 112.

The predetermined order is the order of quantization/dequantization of the conversion coefficients. For example, the predetermined scanning order is defined in ascending order of frequency (in order from low frequency to high frequency) or descending order (in order from high frequency to low frequency).

The quantization parameter is a parameter that defines the quantization step (quantization width). For example, if the quantization parameter value is increased, the quantization step is also increased. In summary, if the quantization parameter value increases, the quantization error increases.
[Entropy coding unit]

The entropy coding unit 110 generates a coded signal (coded bit stream) by variable length coding of the quantized coefficients input from the quantization unit 108. Specifically, the entropy coding unit 110 binarizes the quantized coefficients, for example, and arithmetically encodes the binary signals.
[Anti-quantization department]

The inverse quantization unit 112 inversely quantizes the quantized coefficient, which is an input from the quantization unit 108. Specifically, the inverse quantization unit 112 inversely quantizes the quantized coefficients of the current block in a predetermined scanning order. Then, the inverse quantization unit 112 outputs the converted coefficients of the current block that have been inversely quantized to the inverse conversion unit 114.
[Anti-conversion department]

The inverse conversion unit 114 inversely converts the conversion coefficient, which is an input from the inverse quantization unit 112, to restore the prediction error. Specifically, the inverse conversion unit 114 restores the prediction error of the current block by performing inverse conversion corresponding to the conversion of the conversion unit 106 with respect to the conversion coefficient. Then, the inverse conversion unit 114 outputs the restored prediction error to the addition unit 116.

Furthermore, since the predicted prediction error is lost due to quantization, it does not match the prediction error calculated by the subtraction unit 104. That is, the prediction error in the restoration includes a quantization error.
[Additional Department]

The adding unit 116 reconstructs the current block by adding the prediction error, which is an input from the inverse conversion unit 114, and the prediction sample input from the prediction control unit 128. Then, the adding unit 116 outputs the reconstructed block to the block memory 118 and the loop filter unit 120. Reconstructed blocks are sometimes referred to as local decoded blocks.
[block memory]

The block memory 118 is a memory for storing a block to which intra prediction is referred, and is a block in a picture to be encoded (hereinafter referred to as a current picture). Specifically, the tile memory 118 stores the reconstructed block output from the addition unit 116.
[loop filter unit]

The loop filter unit 120 applies a loop filter to the block reconstructed by the addition unit 116, and outputs the filtered reconstructed block to the frame memory 122. The loop filter is a filter (intra-loop filter) used in the coding loop, and includes, for example, a deblocking filter (DF), a sample applicability offset (SAO), and an adaptive loop filter (ALF).

ALF is applicable to the least square error filter used to remove coding distortion. For example, for the 2 ́2 sub-blocks in the current block, the filter is applied one by one according to the direction and activity of the local gradient. 1 filter selected in .

Specifically, first, sub-blocks (for example, 2 ́2 sub-blocks) are classified into a plurality of groups (for example, 15 or 25 groups). The classification of sub-blocks is based on the direction and activity of the gradient. For example, the classification value C (for example, C=5D+A) is calculated by using the gradient direction value D (for example, 0 to 2 or 0 to 4) and the gradient activity value A (for example, 0 to 4). Then, based on the classification value C, the sub-blocks are classified into a plurality of groups (for example, 15 or 25 groups).

The direction value D of the gradient is derived by, for example, comparing gradients in a plurality of directions (eg, horizontal, vertical, and 2 diagonal directions). Further, the activity value A of the gradient is derived by, for example, adding a gradient in a plurality of directions and quantizing the addition result.

Based on the results of such classification, the filters for the sub-blocks are determined from a plurality of filters.

The shape of the filter used in the ALF can be, for example, a circularly symmetrical shape. 4A to 4C are diagrams showing a plurality of examples of shapes of filters used for ALF. Fig. 4A shows a 5 ́5 diamond shape filter, Fig. 4B shows a 7 ́7 diamond shape filter, and Fig. 4C shows a 9 ́9 diamond shape filter. The information representing the shape of the filter is signalized at the picture level. Furthermore, the signalization of the information indicating the shape of the filter need not be limited to the picture level, and may be other levels (for example, sequence level, slice level, block level, CTU level or CU level).

The ALF on/off is determined by, for example, a picture level or a CU level. For example, in terms of brightness, whether or not ALF is applied is determined by the CU level, and in terms of color difference, whether or not ALF is applied is determined at the picture level. Information indicating that ALF is turned on/off is signalized at the picture level or CU level. Furthermore, the signalization of the information indicating that the ALF is turned on/off is not limited to the picture level or the CU level, and may be other levels (for example, sequence level, slice level, block level, or CTU level).

The set of coefficients for a selectable plurality of filters (e.g., filters from 15 to 25) is signaled at the picture level. Furthermore, the signalization of the coefficient set need not be limited to the picture level, and may be other levels (such as sequence level, slice level, block level, CTU level, CU level or sub-block level).
[frame memory]

The frame memory 122 is a memory for storing reference pictures used for inter prediction, and is sometimes referred to as a frame buffer. Specifically, the frame memory 122 stores the reconstructed block filtered by the loop filter unit 120.
[Intra Prediction Unit]

The intra prediction unit 124 refers to the block stored in the current picture of the block memory 118 to perform intra prediction (also referred to as intra prediction) of the current block, thereby generating a prediction signal (intra prediction signal). . Specifically, the intra prediction unit 124 performs intra prediction by referring to samples (for example, luminance values and color difference values) of the blocks adjacent to the current block, thereby generating an intra prediction signal and outputting the intra prediction signal. To the prediction control unit 128.

For example, the intra prediction unit 124 performs intra prediction using one of a plurality of predetermined intra prediction modes. The plurality of intra prediction modes include one or more non-directional prediction modes and a plurality of directional prediction modes.

The non-directional prediction mode of 1 or more includes, for example, a Planar prediction mode and a DC prediction mode defined by the H.265/HEVC (High-Efficiency Video Coding) specification (Non-Patent Document 1).

The plurality of directional prediction modes include, for example, a prediction mode of 33 directions specified by the H.265/HEVC specification. Further, the plurality of directional prediction modes may further include a prediction mode of 32 directions (a total of 65 directional prediction modes) in addition to the 33 direction. 5A is a diagram showing 67 intra prediction modes (two non-directional prediction modes and 65 directional prediction modes) of intra prediction. The solid arrows indicate the 33 directions defined by the H.265/HEVC specifications, and the dotted arrows indicate the additional 32 directions.

Furthermore, in the intra prediction of the color difference block, the luminance block can also be referred to. In summary, the color difference component of the current block can also be predicted based on the brightness component of the current block. This type of intra prediction is sometimes referred to as CCLM (cross-component linear model) prediction. The intra prediction mode (for example, referred to as CCLM mode) of the color difference block of the reference luminance block may also be added as one of the intra prediction modes as the color difference block.

The intra prediction unit 124 can also correct the intra-predicted pixel value based on the gradient of the reference pixels in the horizontal/vertical direction. The intra prediction associated with this type of correction is sometimes referred to as PDPC (position dependent intra prediction combination). Information indicating whether or not the PDPC is applicable (referred to as a PDPC flag, for example) is signaled at the CU level. Furthermore, the signalization of the information need not be limited to the CU level, but may be other levels (such as sequence level, picture level, slice level, block level or CTU level).
[Inter Prediction Unit]

The inter prediction unit 126 refers to the reference picture stored in the frame memory 122 and the reference picture different from the current picture to perform inter prediction (also referred to as inter-picture prediction) of the current block, thereby generating a prediction signal (frame). Inter-predictive signal). Inter prediction is performed in units of current blocks or sub-blocks within the current block (eg, 4 ́4 blocks). For example, the inter prediction unit 126 performs reference intra-picture motion estimation for the current block or sub-block. Then, the inter prediction unit 126 performs motion compensation using the motion information (for example, motion vector) obtained by the motion estimation, thereby generating an inter prediction signal of the current block or sub-block. Then, the inter prediction unit 126 outputs the generated inter prediction signal to the prediction control unit 128.

Mobile information for motion compensation is signaled. The signal vectorization of the motion vector can also utilize a motion vector predictor. In summary, the difference between the motion vector and the predicted motion vector can also be signaled.

Furthermore, not only the movement information of the current block obtained by the motion estimation but also the movement information of the adjacent block can be used to generate the inter prediction signal. Specifically, the prediction signal according to the movement information obtained by the motion estimation and the prediction signal according to the movement information of the adjacent block may be added to the weight, thereby using the sub-block in the current block as a unit. To generate an inter prediction signal. This type of inter prediction (motion compensation) is sometimes referred to as OBMC (overlapped block motion compensation).

In this type of OBMC mode, information indicating the size of the OBMC sub-blocks (for example, called OBMC block size) is signaled at the sequence level. Also, information indicating whether or not the OBMC mode is applied (for example, referred to as an OBMC flag) is signaled at the CU level. Furthermore, the level of signalization of such information need not be limited to the sequence level and CU level, and may be other levels (such as picture level, slice level, block level, CTU level or sub-block level).

More specifically describes the OBMC mode. 5B and 5C are a flowchart and a conceptual diagram for explaining an outline of a predicted image correction process of the OBMC process.

First, a general motion compensated predicted image (Pred) is obtained by using a motion vector (MV) assigned to a coding target block.

Next, the motion vector (MV_L) of the encoded left adjacent block is applied to the coding target block, the predicted image (Pred_L) is obtained, and the predicted image and the Pred_L are weighted and overlapped, thereby performing the predicted image. The first revision.

Similarly, the motion vector (MV_U) of the encoded upper adjacent block is applied to the coding target block, the predicted image (Pred_U) is obtained, and the predicted image and the Pred_U that have been subjected to the first correction are given weights. By overlapping, the second correction of the predicted image is performed, and this is used as the final predicted image.

Furthermore, the method of two-stage correction using the left adjacent block and the upper adjacent block has been described here, but it is also possible to adopt a configuration in which the right adjacent block or the lower adjacent block is used to perform correction with more than two stages.

Furthermore, the pixel area in which the entire area of the overlapping area is not overlapped may be only a part of the area near the boundary of the block.

Furthermore, although the predicted image correction processing from one reference picture has been described here, the same applies to the case where the predicted image is corrected from the plurality of reference pictures, and the predicted images corrected from the respective reference pictures are acquired, and further overlapped. The obtained predicted image is taken as the final predicted image.

Furthermore, the processing target block may be in units of prediction blocks, or may be in units of sub-blocks further divided into prediction blocks.

As a method of determining whether or not the OBMC processing is applied, for example, a method of indicating whether or not the OBMC-processed signal obmc_flag is applied is used. The specific example is that the encoding device determines whether the encoding target block belongs to a mobile complex region, and if it belongs to a mobile complex region, the obmc_flag setting value is 1 and is encoded by the OBMC processing, and does not belong to the mobile complex region, the obmc_flag setting value. 0, not encoded for OBMC processing. Further, in the decoding apparatus, by decoding the obmc_flag described by the stream, it is switched according to the value of whether or not the OBMC processing is applied.

Furthermore, the mobile information is not signalized and may be derived from the decoding device side. For example, the merge mode specified in the H.265/HEVC specification can also be utilized. Further, for example, the mobile information may be derived by performing motion estimation on the decoding device side. At this time, the motion estimation is performed without using the pixel value of the current block.

Here, a mode for performing motion estimation on the decoding apparatus side will be described. The mode in which motion estimation is performed on the decoding device side is sometimes referred to as a PMMVD (pattern matched motion vector derivation) mode or a FRUC (frame rate up-conversion) mode.

An example of the FRUC process is shown in FIG. 5D. First, a plurality of candidate lists having predicted motion vectors (which may also be common to the merge list) are generated with reference to the motion vectors of the coded blocks that are spatially or temporally adjacent to the current block. Next, the optimal candidate MV is selected from among the plurality of candidate MVs registered in the candidate list. For example, an evaluation value of each candidate included in the candidate list is calculated, and one candidate is selected based on the evaluation value.

Then, based on the selected candidate motion vector, the motion vector for the current block is derived. Specifically, for example, the selected candidate motion vector (best candidate MV) is directly derived as the motion vector for the current block. Further, for example, pattern matching may be performed in the peripheral region of the position in the reference picture corresponding to the selected candidate motion vector, thereby deriving the motion vector for the current block. That is, for the peripheral region of the optimal candidate MV, the estimation is performed in the same manner. If there is an MV whose evaluation value is a better value, the optimal candidate MV is updated to the aforementioned MV as the final MV of the current block. Also. Further, a configuration in which the processing is not performed may be employed.

When processing in units of sub-blocks, the same processing can be used.

Furthermore, the evaluation value is obtained by obtaining a difference value of the reconstructed image by matching the region in the reference picture corresponding to the motion vector with the predetermined region. Furthermore, in addition to the difference value, the evaluation value can be calculated using information other than the difference value.

Pattern matching utilizes the first mode matching and the second mode matching. The first mode matching and the second mode matching are sometimes referred to as bidirectional matching and template matching, respectively.

In the first pattern matching, pattern matching is performed between two blocks in different two reference pictures and between two blocks of the motion trajectory of the current block. Therefore, in the first pattern matching, the region in the other reference picture along the moving track of the current block is used as the predetermined region for calculating the candidate evaluation value.

Fig. 6 is a view for explaining an example of pattern matching (bidirectional matching) between two blocks along a moving track. As shown in FIG. 6, in the first mode matching, two blocks in the moving track along the current block (Cur block) are estimated, and two regions in the two different reference pictures (Ref0, Ref1) The best matching pair in the pair of blocks is used to derive 2 motion vectors (MV0, MV1). Specifically, for the current block, the reconstructed image at the specified position in the first encoded reference picture (Ref0) specified by the candidate MV and the second encoded reference picture (Ref1) specified by the symmetric MV are derived. The difference of the reconstructed image at the specified position is calculated using the obtained difference value, and the symmetric MV is scaled by the candidate MV at the display time interval. The candidate MV whose evaluation value is the best value among the plurality of candidate MVs is selected as the final MV.

Assume that it is a continuous moving track, indicating that the motion vectors (MV0, MV1) of the two reference blocks are equal to the time distance (TD0, TD1) between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). proportion. For example, when the current picture is located between two reference pictures in time, when the time distance from the current picture to the two reference pictures is equal, the first mode matching is performed, and the bidirectional motion vector with reflection symmetry is derived.

In the second pattern matching, the template in the current picture (in the current picture, adjacent to the block of the current block (for example, the upper and/or left adjacent block)) and the block in the reference picture, the mode is performed. match. Therefore, in the second pattern matching, the block adjacent to the current block in the current picture is used as the predetermined area for calculating the candidate evaluation value.

FIG. 7 is a diagram for explaining an example of pattern matching (template matching) between a template in a current picture and a block in a reference picture. As shown in FIG. 7, in the second mode matching, by estimating the block in the reference picture (Ref0) that best matches the block in the current picture (Cur Pic) adjacent to the current block (Cur block), the block is derived. The current vector of the block. Specifically, for the current block, the reconstructed image of the coded region of both the left adjacent and the upper adjacent or the adjacent one is derived, and the reconstructed image in the encoded reference picture (Ref0) specified by the candidate MV is reconstructed. The difference between the images is calculated using the obtained difference value, and the candidate MV whose evaluation value is the optimum value among the plurality of MV candidates is selected as the optimal candidate MV.

Information indicating whether or not this type of FRUC mode is applied (for example, called the FRUC flag) is signaled at the CU level. Further, when the FRUC mode is applied (for example, when the FRUC flag is true), information indicating the pattern matching method (first pattern matching or second pattern matching) (for example, referred to as FRUC mode flag) is signaled at the CU level. Furthermore, the signalization of such information need not be limited to the CU level, but may be other levels (such as sequence level, picture level, slice level, block level, CTU level or sub-block level).

Here, a mode in which a motion vector is derived based on a model of assuming constant-speed linear motion will be described. This mode is sometimes referred to as BIO (bi-directional optical flow) mode.

Fig. 8 is a view for explaining a model assuming constant-speed linear motion. In Fig. 8, (v _x , v _y ) represents a velocity vector, and t ₀ and t ₁ respectively represent the temporal distance between the current picture (Cur Pic) and two reference pictures (Ref ₀ , Ref ₁ ). (MVx ₀ , MVy ₀ ) represents a motion vector corresponding to the reference picture Ref ₀ , and (MVx ₁ , MVy ₁ ) represents a motion vector corresponding to the reference picture Ref ₁ .

At this time, under the assumption of the iso-linear motion of the velocity vector (v _x , v _y ), (MVx ₀ , MVy ₀ ) and (MVx ₁ , MVy ₁ ) are expressed as (v _x τ ₀ , v _y τ _{0 , respectively).} And (-v _x τ ₁ , -v _y τ ₁ ), the following optical flow equation (1) holds.
[Number 1]

Here, I ^(k) represents the luminance value of the reference image k (k = 0, 1) after the motion compensation. The optical flow equation represents the sum of (i), (ii) and (iii) equal to zero, where (i) the time differential of the luminance value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image. (iii) the product of the velocity in the vertical direction and the vertical component of the spatial gradient of the reference image. According to the combination of the optical flow equation and the Hermitian interpolation, the motion vector of the block unit obtained from the merge list or the like is corrected in units of pixels.

Furthermore, it is also possible to derive a motion vector on the decoding device side by using a different method from the motion vector derivation of the model based on the assumed constant velocity linear motion. For example, according to the motion vector of a plurality of adjacent blocks, the motion vector may be derived in units of sub-blocks.

Here, a mode in which a motion vector is derived in units of sub-blocks according to a motion vector of a plurality of adjacent blocks will be described. This mode is sometimes referred to as an affine motion compensation prediction mode.

FIG. 9A is a diagram for explaining a motion vector of a sub-block unit derived from a motion vector of a plurality of adjacent blocks. FIG. In Figure 9A, the current block contains 16 4 ́4 sub-blocks. Here, the motion vector v _{0 of the} control point of the upper left corner of the current block is derived according to the motion vector of the adjacent block, and the motion vector v _{1 of the} control point of the upper right corner of the current block is derived according to the motion vector of the adjacent sub-block. Then, using two motion vectors v ₀ and v ₁ , the motion vectors (v _x , v _y ) of the respective sub-blocks in the current block are derived by the following equation (2).
[Number 2]

Here, x and y respectively indicate the horizontal position and the vertical position of the sub-block, and w represents a predetermined weighting coefficient.

The affine motion compensation prediction mode may also include several modes in which the motion vector derivation methods of the upper left and upper right control points are different. Information indicating this type of affine motion compensated prediction mode (for example, called an affine flag) is signaled at the CU level. Furthermore, the signalization of the information indicating the affine motion compensation prediction mode need not be limited to the CU level, and may be other levels (eg, sequence level, picture level, slice level, block level, CTU level, or sub-block level).
[Predictive Control Department]

The prediction control unit 128 selects any one of the intra prediction signal and the inter prediction signal, and outputs the selected signal as a prediction signal to the subtraction unit 104 and the addition unit 116.

Here, an example of deriving a motion vector of a coding target picture by a merge mode will be described. FIG. 9B is a diagram for explaining an outline of a motion vector derivation process using a merge mode.

First, a list of predicted MVs to which candidates for the predicted MV are registered is generated. The candidate for predicting the MV includes: a spatial adjacency prediction MV, which is an MV of a plurality of coded blocks spatially located around the block of the coding target; a temporally adjacent MV, which is a coded block in which the encoded reference picture is projected. The MV of the block near the location; combined with the prediction MV, which is the MV generated by combining the MV value of the spatial neighbor prediction MV with the temporal neighbor prediction MV; and the MV with zero value, that is, the zero prediction MV.

Next, by selecting one prediction MV from among a plurality of prediction MVs registered in the prediction MV list, the MV as the coding target block is determined.

Further, the variable length coding unit encodes the signal merge_idx indicating the selected prediction MV in the stream.

Further, the prediction MV registered in the prediction MV list described in FIG. 9B is an example, and the number may be different from the number in the figure, or may be a configuration that does not include a part of the prediction MV in the figure, or The configuration of the prediction MV other than the type of the prediction MV in the figure is added.

Furthermore, the final MV may be determined by performing DMVR processing, which will be described later, by using the MV of the encoding target block derived by the merge mode.

Here, an example in which MV is determined by DMVR processing will be described.

9C is a conceptual diagram for explaining an outline of DMVR processing.

First, the optimal MVP set in the processing target block is used as the candidate MV, and the first reference picture of the processed picture in the L0 direction and the second reference picture of the processed picture in the L1 direction are respectively selected according to the candidate MV. A reference pixel is obtained, and an average of each reference pixel is obtained to generate a template.

Next, using the template, the peripheral regions of the candidate MVs of the first reference picture and the second reference picture are respectively estimated, and the MV with the lowest cost is determined as the final MV. Further, the cost value is calculated by using a difference value and an MV value between each pixel value of the template and each pixel value of the estimated region.

Furthermore, the outlines of the processes described herein are basically common to both the encoding device and the decoding device.

Further, it is disadvantageous that the processing itself used in the above description may be performed by any other processing as long as it is a process in which the periphery of the candidate MV can be estimated and the final MV is derived.

Here, a mode in which a predicted image is generated by the LIC processing will be described.

9D is a diagram for explaining an outline of a predicted image generating method of brightness correction processing by LIC processing.

First, an MV for obtaining a reference image corresponding to the block of the encoding target is derived from the reference picture of the encoded picture.

Next, for the coding target block, the luminance pixel value of the coded peripheral reference region adjacent to the left adjacent and the upper adjacent and the luminance pixel value of the equivalent position in the reference picture specified by the MV are used to extract the luminance value in the reference picture and The information on how the image of the encoding object changes is calculated, and the brightness correction parameter is calculated.

For the reference image in the reference picture specified by the MV, the brightness correction processing is performed using the aforementioned brightness correction parameter, thereby generating a predicted image for the encoding target block.

Further, the shape of the peripheral reference region in FIG. 9D is an example, and a shape other than the shape may be used.

Further, although the process of generating a predicted image from one reference picture has been described here, the same applies to the case of generating a predicted image from a plurality of reference pictures, and the brightness is performed in the same manner for the reference picture obtained from each reference picture. After the correction process, a predicted image is generated.

As a method of determining whether or not the LIC processing is applied, for example, a method of indicating whether or not the LIC processing signal lic_flag is applied is used. In the specific example, the encoding device determines whether or not the encoding target block belongs to the region where the luminance change occurs, and belongs to the region where the luminance change occurs, and the lic_flag is set to 1 and is encoded by the LIC processing, and is not included in the region where the luminance change occurs, lic_flag The setting value is 0, and encoding is not applied to LIC processing. Further, in the decoding apparatus, by decoding the lic_flag described by the stream, it is switched according to the value of whether or not the LIC processing is applied.

The method of determining whether or not the LIC processing is applied also includes, for example, a method of determining whether or not the peripheral block is LIC-processed. As a specific example, when the encoding target block is in the merge mode, it is determined whether or not the surrounding coded block selected at the time of MV export of the merge process is encoded by the LIC process, and the result is switched whether or not the LIC process is applied. coding. Furthermore, in the case of this example, the decoding process is also identical.
[Summary of decoding device]

Next, an outline of a decoding device that can decode the encoded signal (encoded bit stream) output from the encoding device 100 will be described. FIG. 10 is a block diagram showing a functional configuration of a decoding device 200 according to the first embodiment. The decoding device 200 is a moving image/image decoding device that decodes a moving image/image in units of blocks.

As shown in FIG. 10, the decoding apparatus 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse conversion unit 206, an addition unit 208, a block memory 210, a loop filter unit 212, a frame memory 214, and an intra prediction unit 216. The inter prediction unit 218 and the prediction control unit 220.

The decoding device 200 is realized by, for example, a general purpose processor and a memory. At this time, when the processor executes the software program stored in the memory, the processor functions as the entropy decoding unit 202, the inverse quantization unit 204, the inverse conversion unit 206, the addition unit 208, the loop filter unit 212, and the intra prediction unit 216. The inter prediction unit 218 and the prediction control unit 220 function. Further, the decoding device 200 may correspond to the entropy decoding unit 202, the inverse quantization unit 204, the inverse conversion unit 206, the addition unit 208, the loop filter unit 212, the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220. A dedicated one or more electronic circuits are implemented.

Hereinafter, each constituent element included in the decoding device 200 will be described.
[Entropy decoding unit]

The entropy decoding unit 202 entropy decodes the encoded bit stream. Specifically, the entropy decoding unit 202 decodes, for example, a coded bit stream number into a binary signal. Then, the entropy decoding unit 202 debinarizes the binary signal. Thereby, the entropy decoding unit 202 outputs the quantized coefficients to the inverse quantization unit 204 in units of blocks.
[Anti-quantization department]

The inverse quantization unit 204 inversely quantizes the input coefficient from the entropy decoding unit 202, that is, the quantization coefficient of the decoding target block (hereinafter referred to as the current block). Specifically, the inverse quantization unit 204 inversely quantizes the quantized coefficients according to the quantization parameters corresponding to the quantized coefficients for the respective quantized coefficients of the current block. Then, the inverse quantization unit 204 outputs the quantized coefficients (i.e., conversion coefficients) of the current block that have been inversely quantized to the inverse conversion unit 206.
[Anti-conversion department]

The inverse conversion unit 206 inversely converts the conversion coefficient, which is an input from the inverse quantization unit 204, to restore the prediction error.

For example, when the information interpreted from the encoded bit stream is applied to EMT or AMT (for example, the AMT flag is true), the inverse conversion unit 206 inversely converts the conversion coefficient of the current block based on the information indicating the type of conversion of the interpretation.

Further, for example, when the information interpreted from the encoded bit stream is applied to the NSST, the inverse conversion unit 206 applies inverse retransformation to the conversion coefficient.
[Additional Department]

The addition unit 208 reconstructs the current block by adding the prediction error input from the inverse conversion unit 206 and the prediction sample input from the prediction control unit 220. Then, the adding unit 208 outputs the reconstructed block to the block memory 210 and the loop filter unit 212.
[block memory]

The tile memory 210 is a memory for storing a block to which intra prediction is referred, and is a block in a decoding target picture (hereinafter referred to as a current picture). Specifically, the tile memory 210 stores the reconstructed block output from the addition unit 208.
[loop filter unit]

The loop filter unit 212 applies a loop filter to the block reconstructed by the adding unit 208, and outputs the filtered reconstructed block to the frame memory 214, the display device, and the like.

The information indicating the on/off of the ALF read from the encoded bit stream indicates that one filter is selected from the plurality of filters according to the direction and the degree of activity of the local gradient when the ALF is turned on, and the selected filter is applied to Reconstruct the block.
[frame memory]

The frame memory 214 is a memory for storing reference pictures used for inter prediction, and is sometimes referred to as a frame buffer. Specifically, the frame memory 214 stores the reconstructed block filtered by the loop filter unit 212.
[Intra Prediction Unit]

The intra prediction unit 216 performs intra prediction by referring to a block stored in the current picture of the block memory 210 based on the intra prediction mode interpreted from the encoded bit stream, thereby generating a prediction signal (intra prediction) Signal)). Specifically, the intra prediction unit 216 performs intra prediction by referring to samples (for example, luminance values and color difference values) of the blocks adjacent to the current block, thereby generating an intra prediction signal and outputting the intra prediction signal. To the prediction control unit 220.

Furthermore, when the intra prediction mode of the reference luminance block is selected in the intra prediction of the color difference block, the intra prediction unit 216 predicts the color difference component of the current block based on the luminance component of the current block.

Further, when the information interpreted from the encoded bit stream indicates that the PDPC is applied, the intra prediction unit 216 corrects the intra-predicted pixel value based on the gradient of the reference pixels in the horizontal/vertical direction.
[Inter Prediction Unit]

The inter prediction section 218 refers to the reference picture stored in the frame memory 214 to predict the current block. The prediction is made in units of the current block or sub-blocks within the current block (for example, 4 ́4 blocks). For example, the inter prediction unit 218 performs motion compensation using motion information (for example, motion vector) interpreted from the encoded bit stream, thereby generating an inter prediction signal of the current block or sub-block, and inter-predicting the signal. The output is output to the prediction control unit 220.

Furthermore, when the information interpreted from the encoded bit stream indicates that the OBMC mode is applied, the inter prediction unit 218 can use not only the mobile information of the current block obtained by the motion estimation but also the mobile information of the adjacent block. To generate an inter prediction signal.

Further, when the information interpreted from the encoded bit stream indicates that the FRUC mode is applied, the inter predicting unit 218 performs motion estimation in accordance with the pattern matching method (bidirectional matching or template matching) interpreted from the encoded stream, thereby deriving the movement. News. Then, the inter predicting unit 218 performs motion compensation using the derived moving information.

Further, when the BIO mode is applied, the inter predicting unit 218 derives a motion vector based on a model that assumes a constant-speed linear motion. Further, when the information interpreted from the encoded bit stream indicates that the affine motion compensation prediction mode is applied, the inter prediction unit 218 derives the motion vector in units of sub-blocks based on the motion vectors of the plurality of adjacent blocks.
[Predictive Control Department]

The prediction control unit 220 selects any one of the intra prediction signal and the inter prediction signal, and outputs the selected signal as a prediction signal to the addition unit 208.
[The first aspect of deblocking filtering processing]

FIG. 11 is a flowchart showing a first aspect of the deblocking filtering process performed by the loop filter unit 120 included in the encoding device 100. The processing shown in Fig. 11 is performed for each block boundary of two adjacent blocks. In the following, the operation of the loop filter unit 120 included in the encoding device 100 will be mainly described, but the operation of the loop filter unit 212 included in the decoding device 200 is also the same.

First, in the deblocking filtering process, the loop filter unit 120 calculates a value called Bs by using the block boundary located at the processing target, that is, the information of the block on both sides of the object boundary (S201). Next, the loop filter unit 120 determines whether or not to perform deblocking filtering processing on the target boundary in accordance with the value of Bs (S202 and S203).

Specifically, when the loop filter unit 120 is Bs=2 (S202, YES (YES) and S203, YES), both the luminance and the chrominance are subjected to deblocking processing (S204 and S205). Further, when the loop filter unit 120 is Bs=1 (S202, YES, S203, NO), the luminance is subjected to deblocking processing (S206), and the chrominance is not subjected to deblocking processing. Further, when the loop filter unit 120 is Bs=0 (S202, NO), the deblocking filtering process is not performed on any of luminance and chrominance.

FIG. 12 is a diagram showing an example of a method of calculating Bs. In the Bs calculation of the deblocking filtering process of HEVC, the following four conditions are adopted: (1) at least one of the two blocks adjacent to the boundary is intra prediction; (2) adjacent to the boundary At least one of the two blocks contains dominant DCT coefficients; (3) the difference between the motion vectors of the two blocks adjacent to the boundary is above the threshold; (4) adjacent to the boundary crossing Two blocks, the number of moving vectors (MV) or the reference image is different. On the other hand, in the first aspect, the condition (5) is added to at least one of the two blocks adjacent to each other across the boundary, and the LIC processing is performed. Furthermore, (5) at least one of the two blocks adjacent to the boundary, when performing LIC processing, the Bs value may be 0 or 1 or 2.

FIG. 13 is a flowchart of Bs calculation processing (S201) when Bs=1 is set when the condition (5) is satisfied. FIG. 14 is a view showing an example of a method of calculating Bs at this time.

First, the loop filter unit 120 determines whether or not the condition (1) is satisfied (S211). When the condition (1) is satisfied (S211, YES), the loop filter unit 120 is set to Bs=2 (S213).

When the condition (1) is not satisfied (S211, NO), the loop filter unit 120 determines whether or not at least one of the conditions (2) to (5) is satisfied (S212). When at least one of the conditions (2) to (5) is satisfied (S212, YES), the loop filter unit 120 sets Bs=1 (S214). When any of the conditions (2) to (5) is not satisfied (S212, NO), the loop filter unit 120 sets Bs = 0 (S215).

FIG. 15 is a flowchart of Bs calculation processing (S201) when Bs=2 is set when the condition (5) is satisfied. FIG. 16 is a view showing an example of a method of calculating Bs at this time.

First, the loop filter unit 120 determines whether or not at least one of the conditions (1) and (5) is satisfied (S211A). When at least one of the conditions (1) and (5) is satisfied (S211A, YES), the loop filter unit 120 is set to Bs=2 (S213).

When any of the conditions (1) and (5) is not satisfied (S211A, NO), the loop filter unit 120 determines whether or not at least one of the conditions (2) to (4) is satisfied (S212A). When at least one of the conditions (2) to (4) is satisfied (S212A, YES), the loop filter unit 120 sets Bs=1 (S214). When any of the conditions (2) to (4) is not satisfied (S212A, NO), the loop filter unit 120 sets Bs=0 (S215).
[LIC processing]

The LIC processing (brightness correction processing) has been described using FIG. 9D, and the details thereof will be described below.

First, the inter predicting unit 126 derives a motion vector for acquiring a reference image corresponding to the encoding target block from the reference picture which is the encoded picture.

Next, the inter prediction unit 126 uses the luma pixel values of the left adjacent and upper adjacent coded peripheral reference regions and the luminance pixel values of the equivalent positions in the reference picture specified by the motion vector for the encoding target block. The brightness correction parameter is calculated based on the information of how the reference picture and the coding target picture change. For example, the luminance pixel value of a certain pixel in the peripheral reference region in the encoding target picture is p0, and the luminance pixel value of the pixel in the peripheral reference region in the reference picture at the same position as the pixel is set to p1. The inter predicting unit 126 calculates coefficients A and B which optimize A × p1 + B = p0 for a plurality of pixels in the peripheral reference region as luminance correction parameters.

Next, the inter predicting unit 126 performs luminance correction processing on the reference image in the reference picture specified by the motion vector using the luminance correction parameter, thereby generating a predicted image for the encoding target block. For example, the luminance pixel value in the reference image is set to p2, and the luminance pixel value of the predicted image after the luminance correction processing is set to p3. The inter prediction unit 126 calculates A×p2+B=p3 for each pixel in the reference image, thereby generating a predicted image after the luminance correction processing.

Furthermore, the shape of the peripheral reference region of FIG. 9D is an example, and other shapes may be employed. Also, a part of the peripheral reference area shown in Fig. 9D may be employed. Further, the peripheral reference region is not limited to the region adjacent to the encoding target block, and may not be adjacent to the region of the encoding target block. Further, in the example shown in FIG. 9D, the peripheral reference area in the reference picture is an area specified by the motion vector of the encoding target picture from the peripheral reference area in the encoding target picture, but may be specified by other motion vectors. For example, the other motion vector may also be a motion vector of a peripheral reference region within the encoding target picture.

Although the operation of the encoding device 100 has been described here, the operation of the encoding device 200 is also the same.
[Effect of the first aspect]

According to the composition of the first aspect, it is possible to reduce the subjectively blocky block noise. Here, since the luminance value of the predicted image changes when the LIC processing is performed, the subjective block noise may easily become used when the reconstructed image and the residual signal of the original image are not completely transmitted. Eye-catching. However, when the Bs calculation processing similar to HEVC is performed, even if the LIC processing is performed, Bs=0 may be obtained, and the deblocking filtering processing may not be performed. According to this aspect, this condition can be suppressed.

Further, the loop filter unit 212 included in the decoding device 200 performs the same processing as the above-described deblocking filter processing of the loop filter unit 120 included in the encoding device 100.

Further, not all the processes described in the first aspect are always required, and a part of the processes described in the first aspect may be performed.
[Second aspect of deblocking filtering processing]

The flowchart of the deblocking filtering process of the loop filter unit 120 included in the encoding apparatus 100 of the second aspect is the same as the flowchart of the deblocking filtering process of the first aspect shown in FIG.

In the second aspect, the content of the Bs determination process (S201) is different from that of the first aspect. In the first aspect, the condition (5) is that at least one of the two blocks adjacent to the boundary is subjected to LIC processing, but the positional relationship of the block is considered in the second aspect. Specifically, the loop filter unit 120 sets Bs ≠ 0 when performing LIC processing on adjacent blocks in two blocks (current block and adjacent block) across the boundary of the object. For example, the loop filter unit 120 is set to Bs ≠ 0 when the LIC process is performed on the left block (adjacent block) in the left and right blocks as shown in FIG. 17 . Further, as shown in FIG. 18, the loop filter unit 120 sets Bs≠0 when performing LIC processing on the upper block (adjacent block) in the upper and lower blocks. FIG. 19 is a view showing an example of a method of calculating Bs in this case. Furthermore, (5A) in the upper (left) block located in the upper and lower (left and right) blocks, when the LIC processing is performed, the Bs value may be 0 or 1 or 2.

FIG. 20 is a flowchart of Bs calculation processing (S201) when Bs=1 is set when the condition (5A) is satisfied. FIG. 21 is a view showing an example of a method of calculating Bs at this time.

First, the loop filter unit 120 determines whether or not the condition (1) is satisfied (S221). When the condition (1) is satisfied (S221, YES), the loop filter unit 120 sets Bs=2 (S223).

When the condition (1) is not satisfied (S221, NO), the loop filter unit 120 determines whether or not at least one of the conditions (2) to (5A) is satisfied (S222). When at least one of the conditions (2) to (5A) is satisfied (S222, YES), the loop filter unit 120 sets Bs=1 (S224). When any of the conditions (2) to (5A) is not satisfied (S222, NO), the loop filter unit 120 sets Bs = 0 (S225).

FIG. 22 is a flowchart of Bs calculation processing (S201) when Bs=2 is set in the condition (5A). FIG. 23 is a view showing an example of a method of calculating Bs at this time.

First, the loop filter unit 120 determines whether or not at least one of the conditions (1) and (5A) is satisfied (S221A). When at least one of the conditions (1) and (5A) is satisfied (S221A, YES), the loop filter unit 120 is set to Bs=2 (S223).

When any of the conditions (1) and (5A) is not satisfied (S221A, NO), the loop filter unit 120 determines whether or not at least one of the conditions (2) to (4) is satisfied (S222A). When at least one of the conditions (2) to (4) is satisfied (S222A, YES), the loop filter unit 120 sets Bs=1 (S224). When any of the conditions (2) to (4) is not satisfied (S222A, NO), the loop filter unit 120 sets Bs = 0 (S225).
[Effect of the second aspect]

According to the configuration of the second aspect, excessive smoothing may be reduced as compared with the first aspect. For example, when the LIC processing is performed using the area other than the two blocks adjacent to the boundary of the object as the LIC brightness correction peripheral reference area, the block boundary is subjectively conspicuous. This case is the case where the LIC processing is performed in the adjacent block. In other cases, for example, when the LIC processing is performed as the LIC brightness correction peripheral reference area by using the area in the two blocks adjacent to the target boundary, the deblocking filter may be used. Excessive smoothing. This case refers to the case where the LIC processing is performed on the target block. Therefore, this problem may be solved by the configuration of the second aspect.

Further, the loop filter unit 212 included in the decoding device 200 performs the same processing as the above-described deblocking filter processing of the loop filter unit 120 included in the encoding device 100.

Further, not all the processes described in the second aspect are always required, and a part of the processes described in the second aspect may be performed.
[Modification]

The loop filter unit 120 may or may not perform the above-described processing in units of slices. The loop filter unit 120 may or may not perform the above processing in units of blocks. The loop filter unit 120 may or may not perform the above processing in units of CTUs. The loop filter unit 120 may or may not perform the above processing in units of CU. The loop filter unit 120 may or may not perform the above-described processing depending on the frame type (P frame, B frame, etc.) of the frame to be processed.

Further, in the above-described embodiment, the LIC brightness correction peripheral reference region is an example in which the luminance pixel values of the left adjacent and adjacent coded peripheral reference regions of the block are adjacent to each other, but the present invention is not limited thereto. For example, as the LIC brightness correction peripheral reference region, the encoding device 100 may use the luminance pixel value of the adjacent coded peripheral reference region, and may not use the luminance pixel value of the left adjacent coded peripheral reference region. Further, the encoding device 100 can also change the LIC brightness correction peripheral reference region depending on the conditions. For example, the encoding device 100 may use a block having a small difference in motion vectors as a peripheral reference region for LIC luminance correction.

Further, in the above embodiment, an example in which the Bs value is set to a value of 0 to 2 is shown. However, the value at which the Bs value can be determined is not limited to this example, and other values may be employed. For example, the Bs value can take a value of 3 or more. At this time, when the loop filter unit 120 satisfies the above condition (5) or (5A), the Bs value may be set to a value of three or more.

Further, the loop filter unit 120 refers to the LIC flag indicating whether or not each block uses the LIC process, or does not use the LIC flag to determine whether or not the LIC process is used for each block of the two blocks. For example, when the LIC processing is used in the peripheral block, the LIC processing is also used in the target block, and when the LIC processing is not used in the peripheral block, the loop filter unit 120 can also be used when the target block does not use the LIC processing or the like. The LIC usage status of the neighboring block determines whether the target block uses LIC processing.

Further, in the above description, an example of determining whether or not to perform deblocking filtering processing is determined depending on whether or not the LIC processing is applied. However, the encoding apparatus 100 may change the filtering strength of the deblocking filtering processing depending on whether or not the LIC processing is applied. For example, the encoding device 100 can also make the filtering strength of the deblocking filtering process stronger than when the LIC processing is not applied when the LIC processing is applied.

Furthermore, the loop filter unit 212 included in the decoding device 200 is also the same.

As described above, the encoding device 100 and the decoding device 200 of the present embodiment perform the processing shown in FIG.

The encoding device 100 determines whether or not the brightness correction processing (for example, LIC processing) of the predicted image is applied to at least one of the target block and the adjacent block adjacent to the target block, that is, the determination block (S231).

When the encoding device 100 applies the brightness correction processing to the determination block (S231, YES), the deblocking filtering process is applied to the boundary between the target block and the adjacent block (S232). When the encoding apparatus 100 does not apply the brightness correction processing to the determination block (S231, NO), the deblocking processing is not applied to the boundary (S233).

Here, in the first aspect described above, the determination block refers to the target block and the adjacent block. In short, when the encoding device 100 applies the brightness correction processing to one or both of the target block and the adjacent block, the deblocking filtering process is applied to the boundary. When the encoding apparatus 100 does not apply the luminance correction processing to any of the target block and the adjacent block, the deblocking filtering process is not applied to the boundary.

Further, in the second aspect described above, the determination block refers to the adjacent block. In summary, the encoding apparatus 100 applies the deblocking processing to the boundary regardless of whether or not the luminance correction processing is applied to the target block, and when the luminance correction processing is applied to the adjacent block. The encoding apparatus 100 applies the luminance correction processing to the target block, and does not apply the deblocking processing to the boundary when the luminance correction processing is not applied to the adjacent block.

For example, when the encoding device 100 applies the brightness correction processing to the determination block, the de-block filtering process is applied to the boundary by setting Bs indicating the boundary strength of the boundary to a value other than 0. When the encoding apparatus 100 does not apply the luminance correction processing to the determination block, by setting Bs to 0, the deblocking filtering processing is applied without the boundary.

Further, the decoding apparatus 200 according to the present embodiment determines whether or not the brightness correction processing (for example, LIC processing) of the predicted image is applied to at least one of the target block and the adjacent block adjacent to the target block, that is, the determination block. ) (S231).

When the decoding device 200 applies the brightness correction processing to the determination block (S231, YES), the deblocking filtering process is applied to the boundary between the target block and the adjacent block (S232). When the decoding device 200 does not apply the brightness correction processing to the determination block (S231, NO), the deblocking processing is not applied to the boundary (S233).

Here, in the first aspect described above, the determination block refers to the target block and the adjacent block. In short, when the decoding device 200 applies the luminance correction processing to one or both of the target block and the adjacent block, the deblocking filtering process is applied to the boundary. When the decoding device 200 does not apply the luminance correction processing to any of the target block and the adjacent block, the deblocking filtering process is not applied to the boundary.

Further, in the second aspect described above, the determination block refers to the adjacent block. In summary, the decoding apparatus 200 applies the deblocking processing to the boundary regardless of whether or not the luminance correction processing is applied to the target block and the luminance correction processing is applied to the adjacent block. The decoding apparatus 200 applies the luminance correction processing to the target block, and does not apply the deblocking processing to the boundary when the luminance correction processing is not applied to the adjacent block.

For example, when the decoding device 200 applies the luminance correction processing to the determination block, the de-block filtering process is applied to the boundary by setting Bs indicating the boundary strength of the boundary to a value other than 0. When the decoding device 200 does not apply the luminance correction processing to the determination block, by setting Bs to 0, the deblocking filtering process is applied without the boundary.

Further, the encoding apparatus 100 of the present embodiment includes a dividing unit 102 that divides an image into a plurality of blocks, and the intra prediction unit 124 predicts a region included in the image by using a reference picture included in the image. a block; inter prediction unit 126 predicts a block included in the image by using a reference block included in another image different from the image; and a loop filter unit 120 for the block included in the image The filter unit 106 converts the prediction error of the prediction signal generated by the intra prediction unit 124 or the inter prediction unit 126 and the original signal to generate a conversion coefficient, and the quantization unit 108 quantizes the conversion coefficient to generate a quantization. The coefficient and the entropy coding unit 110 generate a coded bit stream by performing variable length coding on the quantized coefficients. The loop filter unit 120 determines whether or not the brightness correction processing (for example, LIC processing) of the predicted image is applied to at least one of the target block and the adjacent block adjacent to the target block, that is, the determination block (S231). When the loop filter unit 120 applies the brightness correction processing to the determination block (S231, YES), the deblocking filter processing is applied to the boundary between the target block and the adjacent block (S232). When the loop filter unit 120 does not apply the brightness correction processing to the determination block (S231, NO), the deblocking processing is not applied to the boundary (S233).

Further, the decoding apparatus 200 of the present embodiment includes a decoding unit (entropy decoding unit 202) that decodes a coded bit stream and outputs a quantized coefficient, and an inverse quantization unit 204 that inversely quantizes the quantized coefficient to output a conversion coefficient; The unit 206 inversely converts the conversion coefficient to output a prediction error, and the intra prediction unit 216 predicts a block included in the image by using a reference picture included in the image; the inter prediction unit 218 is included in A reference block of another image different from the image is used to predict a block included in the image; and a loop filter unit 212 applies a filter to the block included in the image. The loop filter unit 212 determines whether or not the brightness correction processing (for example, LIC processing) of the predicted image is applied to at least one of the target block and the adjacent block adjacent to the target block, that is, the determination block (S231). When the loop filter unit 212 applies the brightness correction processing to the determination block (S231, YES), the deblocking filter processing is applied to the boundary between the target block and the adjacent block (S232). When the loop filter unit 212 does not apply the brightness correction processing to the determination block (S231, NO), the deblocking processing is not applied to the boundary (S233).
[Example of coding device]

Fig. 25 is a block diagram showing a practical example of the coding apparatus 100 according to the first embodiment. The encoding device 100 includes a circuit 160 and a memory 162. For example, a plurality of components of the encoding device 100 shown in FIG. 1 are implemented by the circuit 160 and the memory 162 shown in FIG.

The circuit 160 is a circuit for performing information processing and is accessible to the memory 162. For example, circuit 160 is a dedicated or general purpose electronic circuit that encodes a dynamic image. Circuitry 160 can be a processor such as a CPU. Moreover, circuit 160 can also be a collection of a plurality of electronic circuits. Further, the circuit 160 can also function as a plurality of constituent elements excluding the constituent elements for storing information among the plurality of constituent elements of the encoding apparatus 100 shown in FIG. 1 and the like.

The memory 162 is a general purpose or special memory used by the memory circuit 160 to encode information of a moving image. The memory 162 may be an electronic circuit or may be connected to the circuit 160. Further, the memory 162 may be included in the circuit 160. Further, the memory 162 may be an aggregate of a plurality of electronic circuits. Moreover, the memory 162 may be a magnetic disk or a compact disk, or may be embodied as a storage or a recording medium. Further, the memory 162 may be either a non-volatile memory or a volatile memory.

Also, for example, in the memory 162, a coded moving image or a bit string corresponding to the encoded moving image may be used. Moreover, in the memory 162, the program used by the memory circuit 160 to encode a moving image can also be used.

Further, for example, the memory 162 can also function as a constituent element for storing information in a plurality of constituent elements of the encoding device 100 shown in FIG. 1 and the like. Specifically, the memory 162 can also function as the block memory 118 and the frame memory 122 shown in FIG. More specifically, in the memory 162, the reconstructed block, the reconstructed picture, and the like can also be memorized.

Further, in the encoding apparatus 100, all of the plurality of constituent elements shown in FIG. 1 and the like may not be realized, or all of the above-described plural kinds of processing may not be performed. A part of the plurality of constituent elements shown in FIG. 1 and the like may be included in other devices, or a part of the plurality of processes may be executed by other devices. Then, in the encoding apparatus 100, part of the plurality of constituent elements shown in FIG. 1 and the like can be realized, and by performing a part of the plurality of types of processing described above, the movement compensation can be efficiently performed.
[Example of decoding device]

Fig. 26 is a block diagram showing a practical example of the decoding device 200 of the first embodiment. The decoding device 200 includes a circuit 260 and a memory 262. For example, a plurality of components of the decoding device 200 shown in FIG. 10 are implemented by the circuit 260 and the memory 262 shown in FIG.

The circuit 260 is a circuit for performing information processing and is accessible to the memory 262. For example, circuit 260 is a dedicated or general purpose electronic circuit that decodes dynamic images. Circuitry 260 can be a processor such as a CPU. Also, circuit 260 can be an aggregate of a plurality of electronic circuits. Further, for example, the circuit 260 may function as a plurality of constituent elements excluding the constituent elements for storing information among the plurality of constituent elements of the decoding apparatus 200 shown in FIG. 10 and the like.

The memory 262 is a general-purpose or dedicated memory used by the memory circuit 260 to decode information of a moving image. The memory 262 may be an electronic circuit or may be connected to the circuit 260. Also, the memory 262 may be included in the circuit 260. Further, the memory 262 may be an aggregate of a plurality of electronic circuits. Moreover, the memory 262 may be a magnetic disk or a compact disk, or may be embodied as a memory or a recording medium. Further, the memory 262 may be either a non-volatile memory or a volatile memory.

For example, in the memory 262, a bit string corresponding to the encoded moving picture is memorized, or a moving picture corresponding to the decoded bit string can be memorized. Further, in the memory 262, a program for decoding a moving image by the memory circuit 260 is also possible.

Further, for example, the memory 262 can also function as a constituent element for storing information in a plurality of constituent elements of the decoding device 200 shown in FIG. Specifically, the memory 262 can also function as the block memory 210 and the frame memory 214 shown in FIG. More specifically, in the memory 262, the reconstructed block and the reconstructed picture can also be memorized.

Further, in the decoding device 200, all of the plurality of constituent elements shown in FIG. 10 and the like may not be implemented, or all of the above-described plural kinds of processing may not be performed. A part of the plurality of constituent elements shown in FIG. 10 and the like may be included in other devices, or a part of the above-described plurality of processes may be executed by other devices. Then, in the decoding device 200, a part of the plurality of constituent elements shown in FIG. 10 and the like can be realized, and the movement compensation can be efficiently performed by performing a part of the plurality of types of processing described above.
[supplement]

The encoding device 100 and the decoding device 200 according to the present embodiment are each used as an image encoding device and an image decoding device, or may be used as a moving image encoding device and a moving image decoding device, respectively. Alternatively, the encoding device 100 and the decoding device 200 can be utilized as loop filtering devices, respectively.

In other words, the encoding device 100 and the decoding device 200 may correspond to only the loop filter unit 120 and the loop filter unit 212, respectively. Then, other components such as the conversion unit 106 and the inverse conversion unit 206 may be included in other devices.

Further, in the present embodiment, each constituent element may be constituted by a dedicated hardware or by executing a software program suitable for each constituent element. Each component can be realized by a program execution unit such as a CPU or a processor, which reads and executes a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

Specifically, the encoding device 100 and the decoding device 200 may each include a processing circuit (Processing Circuitry) and a memory device (Storage) electrically connected to the processing circuit and accessible from the processing circuit. For example, the processing circuit corresponds to circuit 160 or 260, and the memory device corresponds to memory 162 or 262.

The processing circuit includes at least one of a dedicated hardware and a program execution unit, and performs processing using a memory device. Further, when the processing circuit includes the program execution unit, the memory device memorizes the software program executed by the program execution unit.

Here, the software that implements the encoding device 100 and the decoding device 200 of the present embodiment is the following program.

Further, as described above, each constituent element may be an electric circuit. These circuits may constitute one circuit as a whole or may be different circuits. Further, each component may be implemented by a general purpose processor or by a dedicated processor.

Further, other components may be used to execute the processing performed by the specific constituent elements. Further, the order of the execution processing may be changed, or a plurality of types of processing may be performed simultaneously. Further, the codec device may include the encoding device 100 and the decoding device 200.

Although the aspects of the encoding device 100 and the decoding device 200 have been described above based on the embodiment, the aspects of the encoding device 100 and the decoding device 200 are not limited to the embodiment. Various modifications, which are contemplated by those skilled in the art, may be made in the present embodiment or in combination with the constituent elements of the different embodiments, and may be included in the aspects of the encoding device 100 and the decoding device 200, without departing from the scope of the present invention. In the range.

This aspect can also be implemented in combination with at least a portion of other aspects of the invention. Further, a part of the processing described in the flowchart of the present aspect, a part of the configuration of the device, a part of the grammar, and the like may be combined with other aspects.
(Embodiment 2)

In each of the above embodiments, each of the functional blocks can be generally implemented by an MPU, a memory, or the like. Further, each processing of the function block is generally realized by a program execution unit such as a processor reading and executing a software (program) recorded on a recording medium such as a ROM. The software can be distributed by downloading or the like, or can be distributed on a recording medium such as a semiconductor memory. Furthermore, it is of course also possible to implement the functional blocks by hardware (dedicated circuit).

Further, the processing described in each embodiment may be realized by centralized processing by a single device (system) or by a plurality of device distributed processing. Further, the processor for executing the above program may be a single number or a plurality of processors. That is, centralized processing or distributed processing is possible.

The aspects of the present invention are not limited to the above embodiments, and various modifications can be made thereto, and such modifications are also included in the scope of the present invention.

Further, an application example of the moving image encoding method (image encoding method) or the moving image decoding method (image decoding method) described in each of the above embodiments and a system using the same will be described. This system is characterized in that it includes an image coding device using an image coding method, an image decoding device using an image decoding method, and an image coding and decoding device including both. The other components of the system can be appropriately changed depending on the situation.
[usage]

FIG. 27 is a view showing the overall configuration of a content supply system ex100 that realizes a content distribution service. The communication service providing area is divided into required sizes, and fixed base stations ex106, ex107, ex108, ex109, and ex110 are respectively set in each cell.

The content supply system ex100 connects the computers ex111, the game machine ex112, the camera ex113, the home appliance ex114, and the smart phone ex115 to each other via the Internet service provider ex102 and the communication network ex104, and the base stations ex106 to ex110. Internet ex101. The content supply system ex100 can also be connected in combination with any of the above requirements. The devices may be directly or indirectly connected to each other via a telephone network or short-range wireless without passing through the fixed wireless stations, that is, the base stations ex106 to ex110. Further, the streaming server ex103 is connected to each of the devices such as the computer ex111, the game machine ex112, the camera ex113, the home appliance ex114, and the smartphone ex115 via the Internet ex101 or the like. Further, the streaming server ex103 is connected to a terminal device or the like in the hot spot in the aircraft ex117 via the satellite ex116.

Furthermore, it is also possible to replace the base stations ex106 to ex110 by using wireless access points or hotspots. Further, the streaming server ex103 may be directly connected to the communication network ex104 via the Internet ex101 or the Internet service provider ex102, or may be directly connected to the aircraft ex117 via the satellite ex116.

The camera ex113 is a machine that can capture still images and animations, such as a digital camera. In addition, the smart phone ex115 is a smart phone, a mobile phone, or a PHS (Personal Handyphone System) that generally supports 2G, 3G, 3.9G, 4G, and a mobile communication system called 5G in the future.

The appliance ex118 is a machine included in a refrigerator or a household fuel cell cogeneration system.

In the content supply system ex100, the terminal device having the photographing function is connected to the streaming server ex103 via the base station ex106 or the like, thereby realizing live delivery and the like. In the live release, the terminal device (the computer ex111, the game machine ex112, the camera ex113, the home appliance ex114, the smart phone ex115, and the terminal device in the aircraft ex117) performs the still image or the animation content captured by the user using the terminal device. In the encoding process described in each of the above embodiments, the video data obtained by the encoding and the audio data encoded by the audio corresponding to the video are multiplexed, and the obtained data is transmitted to the streaming server ex103. That is, each terminal device functions as an image coding device according to an aspect of the present invention.

In addition, the streaming server ex103 distributes the content data sent by the client in need thereof. The client is a computer ex111, a game machine ex112, a camera ex113, a home appliance ex114, a smart phone ex115, and a terminal device in the aircraft ex117, which can decode the encoded data. Each machine that receives the published material decodes the received data and reproduces it. That is, each machine functions as an image decoding device of one aspect of the present invention.
[Distributed processing]

Moreover, the serialization server ex103 can also be a plurality of servers or a plurality of computers to process, record, or distribute data. For example, the streaming server ex103 can also be implemented by a CDN (Contents Delivery Network), which realizes content by connecting a network distributed between many edge servers and edge servers all over the world. release. At CDN, physically close edge servers are dynamically dispatched in response to the customer. Then, by caching and publishing content to the edge server, latency can be reduced. In addition, when a certain error occurs, or the communication status changes due to an increase in traffic, etc., the distributed edge server is distributed, or the distribution body is switched to another edge server, thereby bypassing the network that is in trouble. Partly to continue to release, so high-speed and stable release can be achieved.

Further, not only the distributed processing of the distribution itself but also the encoding processing of the captured data may be performed on each terminal device or on the server side, or may be performed separately. As an example, the encoding process generally performs a processing loop twice. In the first loop, the picture complexity or code size of the frame or scene unit is detected. Further, in the second loop, processing for maintaining image quality or improving encoding efficiency is performed. For example, the terminal device performs the first encoding device, and the server side that receives the content performs the second encoding process, thereby reducing the processing load of each terminal device and improving the quality and efficiency of the content. In this case, if it is required to receive and decode almost instantaneously, the data of the first encoding performed by the terminal device can be received and reproduced by the other terminal device, so that more flexible instant distribution can be realized.

As another example, the camera ex113 or the like extracts the feature amount from the image, compresses the data on the feature amount as metadata, and transmits it to the server. The server compresses according to the meaning of the image, for example, switching the quantum precision from the feature quantity, judging the importance of the object. The feature quantity data is especially effective for improving the accuracy and efficiency of motion vector prediction when the server is recompressed. Further, the terminal device performs simple coding such as VLC (variable length coding), and the server may perform coding with a large load such as CABAC (Context-Adapted Binary Arithmetic Coding).

Further, as another example, in a gymnasium, a shopping mall, a factory, or the like, a plurality of video files of substantially the same scene may be captured by a plurality of terminal devices. In this case, a plurality of terminal devices for photographing and other terminal devices and servers that are not photographed as needed are used, for example, a GOP (Group of Picture) unit, a picture unit, or a divided square unit of a picture. , respectively, assigning encoding processing to distribute processing. This can reduce latency and achieve immediacy.

Moreover, since the plurality of video data are substantially in the same scene, it is also possible to manage and/or instruct the video data captured by each terminal device with the server. Moreover, the server receives the encoded data from each terminal device, changes the reference relationship between the plurality of data, or corrects or replaces the image itself, and re-encodes it. Thereby, it is possible to generate a stream that improves the quality and efficiency of each piece of data.

In addition, the server can also perform image transcoding after changing the encoding method of the image data, and then release the image data. For example, the server converts the encoding method of the MPEG system into a VP system, or converts H.264 into H.265.

In this way, the encoding process can be performed by the terminal device or a server of one or more. Therefore, the following is a description of the "server" or "terminal" used as the main body for processing. However, part or all of the processing performed by the server may be performed by the terminal device or part of the processing performed by the terminal device. Or all, can also be carried out by the server. Also, regarding these, the same is true for the decoding process.
[3D, multi-angle]

In recent years, more and more images captured by a plurality of cameras ex113 and/or smart phones ex115, which are substantially synchronized with each other, or images or images of the same scene are captured from different angles. The images captured by the respective terminal devices are integrated based on the relative positional relationship between the acquired terminal devices, the regions in which the feature points included in the images match, and the like.

The server not only encodes the two-dimensional moving image, but also encodes the still image automatically or at the time specified by the user according to the scene analysis of the moving image, and transmits it to the receiving terminal device. Further, when the server can acquire the relative positional relationship between the photographing terminal devices, the three-dimensional shape of the scene can be generated based on not only the two-dimensional moving image but also the image of the same scene from different angles. Furthermore, the server additionally encodes the three-dimensional data generated by the point cloud or the like, or uses the three-dimensional data to identify the person or the object, or selects or reconstructs the image captured from the plurality of terminal devices according to the tracking result to be transmitted to the receiving terminal. The image of the device can be.

In this way, the user can arbitrarily select each image corresponding to each of the photographing terminal devices, and enjoy the scene, and can also view the content of the image of any viewpoint from the three-dimensional data reconstructed from the plurality of images or images. In other words, like the image, the sound can be collected from a plurality of different angles, and the server cooperates with the image to multiplex and transmit the sound from a specific angle or space.

In addition, in recent years, content such as Virtual Reality (VR) and Augmented Reality (AR) that make the real world correspond to the virtual world has become increasingly popular. In the VR image, the server creates the viewpoint images for the right eye and the left eye, and performs the encoding of the allowable reference between the respective viewpoint images by Multi-View Coding (MVC) or the like, or does not mutually For reference, it can be encoded as a different stream. In the case of different stream decoding, in response to the user's viewpoint, they can be synchronized with each other to reproduce the virtual three-dimensional space.

In the AR image, the server superimposes the virtual object information on the virtual space according to the three-dimensional position or the user's viewpoint movement. The decoding device can also acquire or maintain virtual object information and three-dimensional data, generate a two-dimensional image in response to the user's viewpoint movement, and smoothly join to create overlapping data. Moreover, in addition to the request for transmitting the virtual object information, the decoding device transmits the viewpoint movement of the user to the server, and the server cooperates with the viewpoint received from the three-dimensional data held by the server to generate overlapping data, and encodes the overlapping data. It can also be released to the decoding device. Furthermore, the superimposed data has a value indicating the degree of penetration in addition to RGB, and the server can also set the value of a other than the object created from the three-dimensional data to 0, etc., and encode the state in which the part is penetrated. . Alternatively, the server may also set the RGB value of the predetermined value in the background as the chroma key, and the part other than the object is set as the background color to generate the data.

Similarly, the decoding process of the distribution data may be performed on each terminal device as a client or on the server side, or may be shared with each other. As an example, when a certain terminal device transmits and receives a request to the server, the other terminal device receives the content requested, performs decoding processing, and transmits the decoded signal to the device having the display. It is not affected by the performance of the communication-enabled terminal device itself, and the processing is dispersed, and appropriate content is selected, thereby reproducing data with good image quality. Further, as another example, it is also possible to receive a large-sized image data on a TV or the like, and to display a partial region such as a divided square in a viewer's personal terminal device. Thereby, the entire image can be shared, and at the same time, the area of responsibility of the user or the area to be confirmed in more detail can be confirmed at hand.

In addition, in the future, it is expected that the use of multiple types of short-range, medium-range or long-distance wireless communication can be used to switch the appropriate data to the communication during the connection using the MPEG-DASH and other wireless communication systems. , receiving content seamlessly. Thereby, the user can switch to the decoding device or the display device such as a display installed indoors or outdoors without being limited to the terminal device of the home. Further, it is possible to decode the decoded terminal device and the displayed terminal device based on the position information of the user or the like. Thereby, it is also possible to move the map information while displaying the map information on the wall surface or a part of the floor of the adjacent building in which the display device is embedded while moving to the destination. Moreover, according to the ease of accessing the encoded data on the network, such as the encoded data being cached by a server that can be accessed from the receiving terminal device for a short time, or copied to an edge server of the content publishing service, etc. Switch the bit rate of the received data.
[Adaptability coding]

Regarding the content switching, the adaptive stream of the compression encoding by the moving image encoding method shown in each of the above embodiments will be described with reference to FIG. The server has a plurality of streams of the same content and different quality as individual streams, but it is also possible to use the characteristics of temporal/temporal adaptability realized by layered coding as shown in the figure. It is also possible to switch contents. In summary, the decoding side determines the layer to be decoded due to the external factors such as the internal factors of the performance and the state of the communication band, and the decoding side can freely switch the content of the low resolution and the content of the high resolution to decode. For example, in the case of moving the video viewed by the smart phone ex115, when the user wants to view the Internet TV or the like after returning home, the device can decode the same stream to different layers, thereby reducing the burden on the user side. .

Further, in addition to the above, the implementation of the code for each layer, the presence of the enhancement layer in the upper layer of the base layer, the enhancement layer includes meta-information based on image statistics and information, and the decoding side is based on the meta-information. It is also possible to super-resolution images of the base layer to generate high-quality images. Super-resolution can also refer to any of the SN ratio enhancement and resolution expansion of the same resolution. The meta information includes information for specifying linear or nonlinear filter coefficients used for super-resolution processing, or information for parameter values specific to filter processing, mechanical learning, or least squares operations for super-resolution processing.

Further, for example, the picture may be divided into squares or the like in accordance with the semantics of the object or the like in the image, and the decoding side selects the decoded block, thereby decoding only a part of the area. Moreover, the object attributes (person, car, ball, etc.) and the position within the image (coordinate position in the same image, etc.) are stored as meta information, whereby the decoding side can determine the position of the desired object based on the meta information, and decide to include The square of the object. For example, as shown in FIG. 29, the meta information is stored in a material storage structure different from the pixel material by using the SEI message of HEVC or the like. The meta information indicates, for example, the position, size, or color of the main object.

Further, the metadata may be stored in units of a plurality of pictures in a stream, a sequence, a random access unit, or the like. Thereby, the decoding side can obtain the time when the specific person appears in the image, and the information of the picture unit can be used to specify the picture existing in the object and the position of the object in the picture.
[Webpage optimization]

FIG. 30 is a view showing an example of a display screen of a web page such as the computer ex111. FIG. 31 is a view showing an example of a display screen of a web page such as the smartphone ex115. As shown in FIG. 30 and FIG. 31, a web page sometimes includes a plurality of connected images of links to image content, and the viewing mode may be different depending on the device being browsed. When a plurality of connected images are visible on the screen, the display device (decoding device) displays a still image or an I picture of each content as a connected image, or displays a plurality of images such as gif animation such as still images or I images. Or only receive the base layer, decode and display the image until the user explicitly selects the connected image, or the connected image is near the center of the image, or the connected image is all entered into the screen.

When the user selects the connected image, the display device decodes the base layer first. Furthermore, when the HTML constituting the web page has information indicating the content of the adaptability, the display device may decode the enhancement layer. Moreover, in order to ensure immediacy, before the selection or the communication band is very strict, the display device only decodes and displays the picture referenced in front (I picture, P picture, B picture only referenced in front), thereby reducing the initial picture. The delay between the decoding time and the display time (the delay from the start of decoding of the content to the start of display). Moreover, the display device can also deliberately ignore the reference relationship of the picture, perform forward reference, and roughly decode all B pictures and P pictures. As time passes, the received pictures increase and perform normal decoding.
[Automatic driving]

In addition, when receiving still images or video data such as two-dimensional or three-dimensional map information for automatic vehicle driving or support driving, the receiving terminal device may receive weather information or construction information in addition to image data belonging to one or more layers. As meta information, the corresponding decoding is performed. Furthermore, the meta information belongs to the layer, or it can be multiplexed with the image data alone.

At this time, since the vehicle including the receiving terminal device, the unmanned aerial vehicle, or the airplane moves, the receiving terminal device transmits the location information of the receiving terminal device when receiving the request, thereby switching the base stations ex106 to ex110. Seamless reception and decoding on one side. Moreover, the receiving terminal device can dynamically switch the degree of reception of the meta-information or the degree of update of the map information in response to the user's selection, the state of the user, or the state of the communication band.

As described above, in the content supply system ex100, the client can immediately receive the encoded information transmitted by the user, decode it, and reproduce it.
[Personal Content Release]

Further, in the content supply system ex100, not only high-quality, long-time content from the image distribution company but also unicast or multicast distribution of low-quality, short-time content from the individual can be performed. Also, such personal content should increase in the future. In order to make personal content into better content, the server may perform editing processing and then perform encoding processing. This can be achieved, for example, by the following constitution.

During photography, the server immediately or accumulates the image, and after the shooting, performs recognition processing such as shooting error, scene estimation, semantic analysis, and object detection from the original image or the encoded data. Then, according to the identification result, the server performs the following editing: manually or automatically corrects out-of-focus or hand-shock, etc., deletes a scene with low importance such as a scene with lower brightness than other pictures and a misaligned focus, and emphasizes the edge of the object, or Make the color tone change, etc. The server encodes the edited data based on the edited result. Moreover, it is known that if the shooting time is too long, the viewing rate will decrease, and the server can also not only automatically edit scenes with low importance, but also clip scenes with less movement, depending on the image processing result, in order to maintain the content. Within a specific time frame. In addition, the server can generate and encode a digest based on the semantic analysis of the scene.

In addition, in the case of personal content, cases in which the copyright infringement, the human rights of the author, or the right to portrait are directly photographed are also found, and the scope of the sharing exceeds the scope of the intention, which is inconvenient for the individual. Therefore, for example, the server can also change the image to specifically focus on the image, and the code is not aligned with the face or home of the person in the peripheral portion of the screen. Further, the server can recognize whether or not the face of the person different from the person registered in advance is captured in the image to be encoded, and when the photograph is taken, the face portion can be subjected to processing such as mosaic. Further, as the pre-processing or post-processing before encoding, a person or a background area in which the user wants to perform image processing may be specified based on the viewpoint of copyright, etc., and the server may replace the designated area with another image, or blur the focus, etc. deal with. If it is a character, you can also replace the image of the face while tracking the character on the moving image.

Further, since the viewing of personal content with a small amount of data strongly requires immediacy, the decoding device first receives the base layer first and performs decoding and reproduction, depending on the frequency band region. The decoding device receives the enhancement layer during this period, and may reproduce the high-quality image including the enhancement layer when the reproduction layer is reproduced two or more times during the reproduction cycle. In this way, if the stream is adaptively encoded, the following experience can be provided: the animation is rough at the stage of not selecting or starting to watch, but the image is improved as the stream is intelligent. In addition to the adaptive coding, the rough stream that is reproduced for the first time and the second stream that is encoded with reference to the first animation are configured as one stream, and the same experience can be provided.
[Other use cases]

Further, these encoding or decoding processes are generally processed by the LSI ex500 included in each terminal device. The LSI ex500 is a single wafer or a composite of a plurality of wafers. Furthermore, the software for encoding or decoding moving pictures is incorporated into a recording medium (CD-ROM, floppy disk, hard disk, etc.) readable by a computer such as ex111, and is also encoded or decoded by the software. can. In other words, when the smart phone ex115 is attached to a camera, the animation data obtained by the camera can also be sent. The animation data at this time is data that is encoded by the LSI ex500 included in the smart phone ex115.

Furthermore, the LSIex500 can also be configured to download application software and enable it. At this time, the terminal device first determines whether the terminal device supports the encoding method of the content or whether it has the execution capability of the specific service. When the terminal device does not support the content encoding method or does not have the execution capability of the specific service, the terminal device downloads the content or the application software, and then acquires and reproduces the content.

Further, the present invention is not limited to the content supply system ex100 via the Internet ex101, and may be incorporated in at least one of the above-described embodiments of the video encoding device (image encoding device) or the moving image decoding device. (Image decoding device). Since the broadcast radio-capable video and the multiplexed multiplexed data are transmitted and received by the satellite or the like, the unicast configuration is easily performed with respect to the content supply system ex100, and the difference is that it is suitable for multicast, but the encoding process and The same application can be performed by decoding processing.
[Hardware composition]

FIG. 32 is a diagram showing an example of the smartphone ex115. In addition, FIG. 33 is a view showing an example of the configuration of the smartphone ex115. The smart phone ex115 has an antenna ex450 for transmitting and receiving radio waves with the base station ex110, a camera unit ex465 for capturing images and a still image, and a display unit ex458 for displaying images captured by the camera unit ex465 and the antenna ex450. The decoded data such as the received image. The smart phone ex115 further includes an operation unit ex466, which is a touch panel or the like, a sound output unit ex457, a speaker for outputting sound or sound, a sound input unit ex456, a microphone for inputting sound, and the like; Ex467, can save the captured image or still picture, recorded sound, received image or still picture, mailed and other encoded data, or decoded data; and slot part ex464, which is the interface with SIMex468, the SIMex468 is used for specific users to perform authentication for various data accesses such as the Internet. Furthermore, an external memory can be used instead of the memory portion ex467.

Further, the main control unit ex460 such as the display control unit ex458 and the operation unit ex466 is integrated with the power supply circuit unit ex461, the operation input control unit ex462, the video signal processing unit ex455, the camera interface ex463, and the display control unit ex459 via the bus line ex470. The modulation/demodulation unit ex452, the multiplex/separation unit ex453, the audio signal processing unit ex454, the slot unit ex464, and the memory unit ex467 are connected.

When the power button is turned on by the user's operation, the power supply circuit unit ex461 supplies power to each unit from the battery pack, and activates the smartphone ex115 in an operable state.

The smartphone ex115 performs processing such as call and data communication under the control of the main control unit ex460 composed of a CPU, a ROM, and a RAM. At the time of the call, the audio signal processing unit ex454 converts the audio signal received by the voice input unit ex456 into a digital audio signal, and performs the spectrum diffusion processing on the modulation/demodulation unit ex452, and performs the transmission/reception unit ex451. The digital analog conversion processing and the frequency conversion processing are transmitted via the antenna ex450. Further, the received data is amplified, the frequency conversion process and the analog-to-digital conversion process are performed, and the spectrum de-diffusion process is performed by the modulation/demodulation unit ex452, and converted into analog sound data by the audio signal processing unit ex454, and then the audio output unit ex457 is used. Output. In the data communication mode, the text, the still image, or the video data is sent to the main control unit ex460 via the operation input unit ex462 by the operation unit ex466 of the main unit, and the transmission and reception processing is performed in the same manner. When transmitting a video, a still picture, or a video and audio in the data communication mode, the video signal processing unit ex455 updates the video signal stored in the memory unit ex467 or the video signal input from the camera unit ex465 by the above-described embodiments. The image encoding method compresses and encodes the image data that has been encoded to the multiplex/separation unit ex453. Further, the audio signal processing unit ex454 encodes the audio signal, and sends the encoded audio data to the multiplex/separation unit ex453, which is the audio input unit while capturing the video or the still picture by the camera unit ex465. Ex456 radio. The multiplexer/separation unit ex453 multiplexes the encoded video data and the encoded audio data in a predetermined manner, and uses a modulation/demodulation unit (modulation/demodulation circuit unit) ex452 and a transmission/reception unit ex451. The modulation processing and the conversion processing are transmitted via the antenna ex450.

When receiving an image attached to an e-mail or a chat or an image linked to a web page or the like, in order to decode the multiplexed data received via the antenna ex450, the multiplex/separation unit ex453 separates the multiplexed data by separating the multiplexed data. For the bit stream of the video data and the bit stream of the audio data, the encoded video data is supplied to the video signal processing unit ex455 via the synchronous bus ex470, and the encoded audio data is supplied to the audio signal. Processing unit ex454. The video signal processing unit ex455 decodes the video signal by the moving picture decoding method corresponding to the moving picture coding method described in each of the above embodiments, and displays the connected video file included in the display unit ex458 via the display control unit ex459. Image or still image. Further, the audio signal processing unit ex454 decodes the audio signal and outputs the sound from the audio output unit ex457. Furthermore, since real-time streaming has become widespread, depending on the user's situation, it is also possible that the reproduction of sound is not appropriate from a social point of view. Therefore, as an initial value, it is preferable to use a non-reproduced sound signal to reproduce only the image signal. It is also possible to synchronize the reproduced sound only when the user clicks on an image data or the like.

In addition, although the smart phone ex115 is described as an example here, in addition to the transceiving type terminal device having both the encoder and the decoder as the terminal device, it is also conceivable to have only the transmitting terminal device having the encoder and the receiving only with the decoder. Three implementation forms such as terminal devices. In other words, it describes the digital playback system that receives or transmits multiplexed data that has audio data for multiplexed video data. However, in addition to voice data, text data associated with images can also be received by multiplexed data. It is also possible to receive or transmit image data by multiplexing or not receiving or transmitting multiplexed data.

Furthermore, it has been explained that the main control unit ex460 including the CPU controls the encoding or decoding processing, but the terminal device often includes the GPU. Therefore, it is also possible to utilize the performance of the GPU to uniformly process a large area by using a memory that is common to the CPU and the GPU, or a memory that is managed by the address so as to be commonly used. Thereby, the encoding time can be shortened, the immediacy can be ensured, and low latency can be realized. In particular, the GPU is not used for the CPU, and the motion estimation, deblocking filtering, SAO (Sample Adaptive Offset), and conversion ‧ quantization processing are performed efficiently in units of pictures and the like.

This aspect can also be implemented in combination with at least a portion of other aspects of the invention. Further, a part of the processing described in the flowchart of the present aspect, a part of the configuration of the device, a part of the grammar, and the like may be combined with other aspects.

Industrial Applicability The present invention can be utilized, for example, in a television receiver, a digital video recorder, a car navigator, a mobile phone, a digital camera, a digital camera, a video conferencing system, or an electronic mirror.

10~23‧‧‧ Block

100‧‧‧ coding device

102‧‧‧ Division

104‧‧‧Decrease Department

106‧‧‧Transition Department

108‧‧‧Quantity Department

110‧‧‧ Entropy Coding Department

112, 204‧‧‧Anti-Quantization Department

114, 206‧‧‧Anti-conversion department

116, 208‧‧‧Additional Department

118, 210‧‧‧ Block memory

120, 212‧‧‧Circuit Filtering Department

122, 214‧‧‧ frame memory

124, 216‧‧ Intra Prediction Department

126, 218‧‧ interframe prediction department

128, 220‧‧‧Predictive Control Department

160, 260‧‧‧ circuits

162, 262‧‧‧ memory

200‧‧‧ decoding device

202‧‧‧ Entropy Decoding Department

ALF‧‧‧Adaptive loop filter

AMT‧‧‧Adaptive multiple conversion

AR‧‧‧Augmented Reality

AST‧‧ ‧ adaptive second conversion

BIO‧‧‧Two-way optical flow

CCLM‧‧‧cross-component linear mode

CABAC‧‧‧Context-adaptive binary arithmetic coding

CDN‧‧‧Content Publishing Network

CTU‧‧‧ coding tree unit

CU‧‧‧ coding unit

Cur block‧‧‧ current block

DCT‧‧ discrete cosine transform

DF‧‧‧deblock filter

DST‧‧‧discrete sine transform

EMT‧‧‧ Explicit multiple core conversion

Ex100‧‧‧Content Supply System

Ex101‧‧‧Internet

Ex102‧‧‧Internet Service Provider

Ex103‧‧‧Streaming server

Ex104‧‧‧Communication Network

Ex106～ex110‧‧‧base office

Ex111‧‧‧ computer

Ex112‧‧‧game machine

Ex113‧‧‧ camera

Ex114‧‧‧Home appliances

Ex115‧‧‧Smart mobile phone

Ex116‧‧‧ satellite

Ex117‧‧ aircraft

Ex450‧‧‧Antenna

Ex451‧‧‧Send/Receive Department

Ex452‧‧‧Modulation/Demodulation Department

Ex453‧‧‧Multiplex/Separation Department

Ex454‧‧‧Sound Signal Processing Department

Ex455‧‧‧Image Signal Processing Department

Ex456‧‧‧Sound Input Department

Ex457‧‧‧Sound Output Department

Ex458‧‧‧Display Department

Ex459‧‧‧Display Control Department

Ex460‧‧‧Main Control Department

Ex461‧‧‧Power Circuit Department

Ex462‧‧‧Operation Input Control Department

Ex463‧‧‧ camera face

Ex464‧‧‧Slots

Ex465‧‧‧Photo Camera Department

Ex466‧‧‧Operation Department

Ex467‧‧‧ memory department

Ex468‧‧‧SIM

Ex470‧‧‧ bus bar, synchronous bus

ex500‧‧‧LSI

FRUC‧‧‧ frame rate upgrade conversion

GOP‧‧‧Photo Group

HEVC‧‧‧High Efficiency Video Coding

LIC‧‧‧Local Lighting Compensation

MBT‧‧‧Multiple Tree

MV, MV0, MV1, MV_L, MV_U‧‧‧ mobile vector

MVC‧‧‧Multi-view coding

NSST‧‧‧ can not be divided into second conversion

OBMC‧‧‧Overlap block motion compensation

P0, p1, p2, p3‧‧‧ brightness pixel values

PDPC‧‧‧Independent Position Intra Prediction Combination

PMMVD‧‧‧ pattern matching mobile vector export

Pred, Pred_L, Pred_U‧‧‧ Forecast Image

PU‧‧‧ forecasting unit

QP‧‧‧Quantification parameters

QTBT‧‧‧ ternary tree plus binary tree

Ref0, Ref1‧‧‧ reference picture

Steps S201 to S206, S211 to S215, S211A, S212A, S221 to S225, S221A, S222A, S231 to S233‧‧

SAO‧‧‧ sample applicability offset

TU‧‧ conversion unit

v ₀ , v ₁ , v _x , v _y ‧‧‧moving vectors

VLC‧‧‧Variable length coding

VR‧‧‧Virtual Reality

Fig. 1 is a block diagram showing a functional configuration of an encoding apparatus according to a first embodiment.

Fig. 2 is a view showing an example of block division in the first embodiment;

FIG. 3 is a table showing a conversion basis function corresponding to each conversion type.

4A is a view showing an example of a shape of a filter used for ALF.

4B is a view showing another example of the shape of the filter used for the ALF.

4C is a view showing another example of the shape of a filter used for ALF.

FIG. 5A is a diagram showing 67 intra prediction modes of intra prediction.

FIG. 5B is a flowchart for explaining an outline of a predicted image correction process of the OBMC process.

FIG. 5C is a conceptual diagram for explaining an outline of a predicted image correction process of the OBMC process.

FIG. 5D is a diagram showing an example of FRUC.

Figure 6 is a diagram for explaining pattern matching (bidirectional matching) between two blocks along a moving track.

FIG. 7 is a diagram for explaining pattern matching (template matching) between a template in a current picture and a block in a reference picture.

Fig. 8 is a view for explaining a model assuming constant-speed linear motion.

9A is a diagram for explaining a motion vector of a sub-block unit derived from a motion vector of a plurality of adjacent blocks.

FIG. 9B is a diagram for explaining an outline of a motion vector derivation process using a merge mode.

9C is a conceptual diagram for explaining an outline of DMVR processing.

9D is a diagram for explaining an outline of a predicted image generating method of brightness correction processing by LIC processing.

Fig. 10 is a block diagram showing a functional configuration of a decoding device according to the first embodiment.

Fig. 11 is a flowchart showing a first aspect of the deblocking filtering process in the first embodiment.

FIG. 12 is a view showing an example of a Bs calculation method according to the first aspect of the first embodiment.

FIG. 13 is a flowchart showing a first example of Bs calculation processing in the first aspect of the first embodiment.

FIG. 14 is a view showing a first example of the Bs calculation method according to the first aspect of the first embodiment.

Fig. 15 is a flowchart showing a second example of Bs calculation processing in the first aspect of the first embodiment.

FIG. 16 is a view showing a second example of the Bs calculation method according to the first aspect of the first embodiment.

Fig. 17 is a view showing the state of the target block and the adjacent block in the first embodiment;

Fig. 18 is a view showing the state of the target block and the adjacent block in the first embodiment;

Fig. 19 is a view showing an example of a Bs calculation method according to a second aspect of the first embodiment.

FIG. 20 is a flowchart showing a first example of Bs calculation processing in the second aspect of the first embodiment.

FIG. 21 is a view showing a first example of the Bs calculation method according to the second aspect of the first embodiment.

Fig. 22 is a flowchart showing a second example of Bs calculation processing in the second aspect of the first embodiment.

FIG. 23 is a view showing a second example of the Bs calculation method according to the second aspect of the first embodiment.

Fig. 24 is a flowchart showing the deblocking filtering process of the first embodiment.

Figure 25 is a block diagram showing a practical example of an encoding device.

Fig. 26 is a block diagram showing a practical example of the decoding device.

Fig. 27 is a view showing the overall configuration of a content supply system for realizing a content distribution service.

FIG. 28 is a diagram showing an example of a coding structure at the time of adaptive coding.

FIG. 29 is a diagram showing an example of a coding structure at the time of adaptive coding.

FIG. 30 is a view showing an example of a display screen of a web page.

31 is a diagram showing an example of a display screen of a web page.

32 is a diagram showing an example of a smart phone.

Fig. 33 is a block diagram showing an example of the configuration of a smart phone.

Claims (12)

An encoding device having: Circuit; and Memory; The aforementioned circuit utilizes the aforementioned memory, When at least one of the target block and the adjacent block adjacent to the target block, that is, the brightness correction processing of the prediction block for the prediction block, is at the boundary between the target block and the adjacent block For deblocking filtering, When the aforementioned brightness correction processing is not applied to the above-described determination block, the above-described deblocking filtering processing is not applied to the above boundary. An encoding device as claimed in claim 1, wherein When the brightness correction processing is applied to one or both of the target block and the adjacent block, the deblocking filtering process is applied to the boundary; When the luminance correction processing is not applied to any of the target block and the adjacent block, the deblocking filtering process is not applied to the boundary. An encoding device as claimed in claim 1, wherein Regardless of whether the aforementioned brightness correction processing is applied to the aforementioned target block, When the foregoing brightness correction processing is applied to the adjacent blocks, the foregoing deblocking filtering processing is applied to the foregoing boundary. When the aforementioned luminance correction processing is not applied to the adjacent blocks, the above-described deblocking filtering processing is not applied to the foregoing boundary. The encoding device of any one of claims 1 to 3, wherein The aforementioned brightness correction processing is LIC (Local Illumination Compensation) processing. The encoding device of any one of claims 1 to 3, wherein When the brightness correction processing is applied to the determination block, the deblocking filtering process is applied to the boundary by setting Bs indicating the boundary strength of the boundary to a value other than 0; When the luminance correction processing is not applied to the determination block, the aforementioned de-block filtering processing is applied to the boundary without setting the Bs to 0. A decoding device having: Circuit; and Memory; The aforementioned circuit utilizes the aforementioned memory, When at least one of the target block and the adjacent block adjacent to the target block, that is, the brightness correction processing of the prediction block for the prediction block, is at the boundary between the target block and the adjacent block For deblocking filtering, When the aforementioned brightness correction processing is not applied to the above-described determination block, the above-described deblocking filtering processing is not applied to the above boundary. The decoding device of claim 6, wherein When the brightness correction processing is applied to one or both of the target block and the adjacent block, the deblocking filtering process is applied to the boundary; When the luminance correction processing is not applied to any of the target block and the adjacent block, the deblocking filtering process is not applied to the boundary. The decoding device of claim 6, wherein Regardless of whether the aforementioned brightness correction processing is applied to the aforementioned target block, When the foregoing brightness correction processing is applied to the adjacent blocks, the foregoing deblocking filtering processing is applied to the foregoing boundary. When the aforementioned luminance correction processing is not applied to the adjacent blocks, the above-described deblocking filtering processing is not applied to the foregoing boundary. The decoding device of any one of claims 6 to 8, wherein The aforementioned brightness correction processing is LIC (Local Illumination Compensation) processing. The decoding device of any one of claims 6 to 8, wherein When the brightness correction processing is applied to the determination block, the deblocking filtering process is applied to the boundary by setting Bs indicating the boundary strength of the boundary to a value other than 0; When the luminance correction processing is not applied to the determination block, the aforementioned de-block filtering processing is applied to the boundary without setting the Bs to 0. An encoding method in which the target block is adjacent to the aforementioned block when at least one of the target block and the adjacent block adjacent to the target block, that is, the brightness correction processing of the prediction image is applied to the determination block The boundary of the block is suitable for deblocking filtering. When the aforementioned brightness correction processing is not applied to the above-described determination block, the above-described deblocking filtering processing is not applied to the above boundary. A decoding method, in the case where at least one of a target block and an adjacent block adjacent to the target block, that is, a brightness correction process for applying a prediction image to the determination block, is adjacent to the target block The boundary of the block is suitable for deblocking filtering. When the aforementioned brightness correction processing is not applied to the above-described determination block, the above-described deblocking filtering processing is not applied to the above boundary.