TW201808004A - Encoding device, decoding device, encoding method, and decoding method - Google Patents

Abstract

An encoding device (100) for encoding blocks to be encoded in an image is provided with: an intra/inter determination unit (1061) that determines whether intra prediction or inter prediction is to be used for a block to be encoded; and a frequency transformation unit (1063) that selectively uses a plurality of frequency transformation bases including DCT-II and DCT-V to perform a first frequency transformation on the prediction error of the block to be encoded. In the first frequency transformation, the basis of DCT-V is used if intra prediction is used for the block to be encoded, and the basis of DCT-II is used if inter prediction is used for the block to be encoded.

Description

Encoding device, decoding device, encoding method and decoding method

FIELD OF THE INVENTION The present disclosure relates to an encoding device, a decoding device, an encoding method, and a decoding method.

BACKGROUND OF THE INVENTION An image coding standard called HEVC (High-Efficiency Video Coding) has been standardized by JCT-VC (Joint Collaborative Team on Video Coding). Prior art literature

Non-Patent Document Non-Patent Document 1: H.265 (ISO / IEC 23008-2 HEVC (High Efficiency Video Coding)

SUMMARY OF THE INVENTION Problems to be Solved by the Invention What is required in this encoding and decoding technology is a further improvement in compression efficiency.

Therefore, the present disclosure provides an encoding device, a decoding device, an encoding method, and a decoding method that can achieve further improvement in compression efficiency. Means to solve the problem

An encoding device according to one aspect of the present disclosure is an encoding device that encodes an encoding target block of an image, and includes a processor and a memory. The processor uses the memory and determines that it is the encoding target area. Which of intra-frame prediction and inter-frame prediction is used for the block, and a plurality of frequency conversion bases including DCT-II and DCT-V are selectively used to perform the first prediction error on the encoding target block. 1 frequency conversion. In the first frequency conversion, when in-frame prediction is used for the encoding target block, the base of DCT-V is used. When the frame is used for the encoding target block, a frame is used. In the case of inter-prediction, the base of DCT-II is used.

A decoding device according to one aspect of the present disclosure is a decoding device that decodes a decoding target block of an image, and includes a processor and a memory. The processor uses the memory and determines that it is the decoding target area. Which of intra-frame prediction and inter-frame prediction is used for the block, and a plurality of inverse frequency conversion bases including inverse conversion of DCT-II and DCT-V are selectively used to perform the foregoing decoding of the target block The first inverse frequency conversion of prediction error. In the first inverse frequency conversion, when in-frame prediction is used for the decoding target block, it is the basis of inverse conversion using DCT-V. When inter-frame prediction is used for the decoding target block, it is the basis for inverse conversion using DCT-II.

Furthermore, these comprehensive or specific aspects can be realized by using a system, method, integrated circuit, computer program, or a computer-readable recording medium such as a CD-ROM, or by using a system, method, or integrated circuit. It can be realized by any combination of circuits, computer programs, and recording media. Invention effect

The present disclosure can provide an encoding device, a decoding device, an encoding method, and a decoding method that can achieve further improvement in compression efficiency.

The form for implementing the invention (the knowledge insights that form the basis of this disclosure) In the conventional moving image coding such as H.265 / HEVC (High Efficiency Video Coding), it is used as a block for in-frame prediction The frequency conversion base is basically the base using DCT-II, but only the 4 × 4 size luminance block will use the base of DST-VII.

In addition, since blocks that are spatially adjacent are used in the intra-frame prediction, a residual signal that is a difference between an input signal and a prediction signal may be spatially shifted in most cases. In particular, there is a tendency that the residuals near the boundaries of the blocks adjacent to the left and upper sides become smaller. Therefore, if a flat DCT-II conversion characteristic is used for the DC (0th-order) conversion characteristics, higher-order components may become noticeable, which may cause the compression efficiency to deteriorate.

Therefore, an encoding device according to one aspect of the present disclosure is an encoding device that encodes an encoding target block of an image, and includes a processor and a memory. The processor uses the memory and determines that the encoding is for the encoding. Which of the intra-frame prediction and inter-frame prediction is used by the target block, and a plurality of frequency conversion bases including DCT-II and DCT-V are used selectively to perform prediction error on the aforementioned coding target block In the first frequency conversion, in the first frequency conversion, when in-frame prediction is used for the encoding target block, a DCT-V base is used. In the case of inter-frame prediction, it is the basis for using DCT-II.

Therefore, when intra-frame prediction is used for the coding target block, the base of DCT-V can be used to convert the coding target block. In DCT-V, the amplitude is reduced at a position close to the reference pixel in the DC component. Therefore, DCT-V is suitable for the conversion of prediction errors in the frame prediction. Therefore, the encoding device can achieve further improvement in compression efficiency.

In the encoding device according to one aspect of the present disclosure, for example, the processor may further determine whether the size of the encoding target block is equal to or smaller than a threshold size, and in the first frequency conversion, for the foregoing purpose, When the coding target block uses in-frame prediction: (i) if the size of the coding target block is below the threshold size, the base of DCT-V is used; (ii) if the coding target area is When the block size is larger than the aforementioned threshold size, the base of DCT-II is used.

Thereby, the bases of DCT-II and DCT-V can be switched in accordance with the size of the coding target block predicted using the frame to convert the coding target block. If the block size is large, the prediction error tends to become smaller in the block as a whole, and DCT-II is suitable for the conversion of the prediction error of the coding target block. On the other hand, if the block size is small, the pixel prediction error tends to be smaller as it gets closer to the reference pixel, making DCT-V suitable for conversion of the prediction error of the coding target block. Therefore, if the size of the coding target block is below the threshold size, the DCT-V base is used to convert the coding target block. If the size of the coding target block is greater than the threshold size, the DCT-II base is used to convert it. Encoding the target block, which can further improve the compression efficiency.

In addition, in the encoding device of one aspect of the present disclosure, for example, the processor may further write the information of the threshold size into the bit stream.

Thereby, the information of the threshold size can be written into the bit stream. Therefore, it becomes possible to use an adaptively determined threshold size in accordance with the input image, and to achieve further improvement in compression efficiency.

Furthermore, in the encoding device of one aspect of the present disclosure, for example, the processor may further determine which conversion mode among a plurality of conversion modes including the first conversion mode and the second conversion mode is applicable. Performing the first frequency conversion on the encoding target block when the first conversion mode is applied, and performing the second frequency conversion different from the first frequency conversion when the second conversion mode is applied .

Thereby, the frequency conversion can be switched using the conversion mode. As a result, it becomes possible to further improve the efficiency of frequency conversion, and further improve the compression efficiency.

Furthermore, in the encoding device of one aspect of the present disclosure, for example, the processor may further write information of a conversion mode applicable to the encoding target block into the bit stream.

Thereby, the information of the conversion mode applicable to the encoding target block can be written into the bit stream. Thus, it becomes possible to adaptively determine a conversion mode in response to an input image, and to achieve further improvement in compression efficiency.

One aspect of the present disclosure is an encoding method that encodes an encoding target block of an image. It determines which of intra-frame prediction and inter-frame prediction is used for the aforementioned encoding target block, and selects The first frequency conversion of the prediction error of the coding target block is performed using a base including a plurality of frequency conversions of DCT-II and DCT-V. In the first frequency conversion, in the area for the coding target, When the block uses intra-frame prediction, the base is DCT-V, and when the inter-frame prediction is used for the aforementioned coding target block, the base is DCT-II.

Thereby, the same effects as those of the encoding device can be exhibited.

A decoding device according to one aspect of the present disclosure is a decoding device that decodes a decoding target block of an image, and includes a processor and a memory. The processor uses the memory and determines that it is the decoding target area. Which of intra-frame prediction and inter-frame prediction is used for the block, and a plurality of inverse frequency conversion bases including inverse conversion of DCT-II and DCT-V are selectively used to perform the foregoing decoding of the target block The first inverse frequency conversion of prediction error. In the first inverse frequency conversion, when in-frame prediction is used for the decoding target block, it is the basis of inverse conversion using DCT-V. When inter-frame prediction is used for the decoding target block, it is the basis for inverse conversion using DCT-II.

With this, in the case that intra-frame prediction is used for the decoding target block, the base of the inverse conversion of DCT-V can be used to perform inverse conversion on the decoding target block. In DCT-V, the amplitude is reduced at a position close to the reference pixel in the DC component. Therefore, DCT-V is suitable for the conversion of prediction errors in the frame prediction. Therefore, the decoding device can achieve further improvement in compression efficiency.

In the decoding device according to one aspect of the present disclosure, for example, the processor may further determine whether the size of the decoding target block is equal to or smaller than a threshold size, and in the first inverse frequency conversion, In the case of using intra-frame prediction for the foregoing decoding target block: (i) if the size of the foregoing decoding target block is equal to or smaller than the aforementioned threshold size, the base of inverse conversion of DCT-V is used; (ii) If the size of the decoding target block is larger than the threshold size, the base of the inverse conversion of DCT-II is used.

Thereby, the basis of the inverse conversion of DCT-II and DCT-V can be switched according to the size of the decoding target block predicted in the frame to perform inverse conversion on the decoding target block. If the block size is large, the prediction error tends to become smaller in the block as a whole, and DCT-II is suitable for the conversion of the prediction error of the decoding target block. On the other hand, if the block size is small, the pixel prediction error tends to be smaller as it gets closer to the reference pixel, making DCT-V suitable for conversion of the prediction error of the decoding target block. Therefore, if the size of the decoding target block is below the threshold size, the base of the inverse conversion of DCT-V is used to inversely transform the decoding target block. If the size of the decoding target block is larger than the threshold size, DCT-II is used. Based on the inverse conversion of the image, inverse conversion is performed on the decoding target block, thereby further improving the compression efficiency.

In addition, in the decoding device of one aspect of the present disclosure, for example, the processor may further decode the information of the threshold size from the bit stream.

In this way, the information of the threshold size can be interpreted from the bit stream. Therefore, it becomes possible to use an adaptively determined threshold size in accordance with the input image, and to achieve further improvement in compression efficiency.

Further, in the decoding device of one aspect of the present disclosure, for example, the processor may further determine which conversion mode among a plurality of conversion modes including the first conversion mode and the second conversion mode is applicable. Perform the first inverse frequency conversion on the decoding target block when the first conversion mode is applied, and perform the second inverse frequency conversion that is different from the first inverse frequency conversion when the second conversion mode is applied. Inverse frequency conversion.

Thereby, the inverse frequency conversion can be switched using the conversion mode. As a result, it becomes possible to further improve the efficiency of frequency conversion, and further improve the compression efficiency.

In addition, in the decoding device of one aspect of the present disclosure, for example, the processor further interprets information of a conversion mode applicable to the decoding target block from a bit stream.

Thereby, the information of the conversion mode applicable to the decoding target block can be written into the bit stream. Thus, it becomes possible to adaptively determine a conversion mode in response to an input image, and to achieve further improvement in compression efficiency.

One aspect of the present disclosure is a decoding method that decodes a decoding target block of an image, which determines which of intra-frame prediction and inter-frame prediction is used for the aforementioned decoding target block, and selects The basis of the plurality of inverse frequency conversions including the inverse conversion of DCT-II and DCT-V is used to perform the first inverse frequency conversion on the prediction error of the decoding target block. In the first inverse frequency conversion, In the case where intra-frame prediction is used for the aforementioned decoding target block, it is the base of inverse conversion using DCT-V, and when the inter-frame prediction is used for the aforementioned decoding target block, DCT is used. -II The basis of the inverse transformation.

Thereby, the same effects as those of the decoding device can be exhibited.

Furthermore, these comprehensive or specific aspects can be realized using a recording medium such as a system, integrated circuit, computer program, or a computer-readable CD-ROM, or a system, integrated circuit, or computer program. And any combination of recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings.

The embodiments described below are general or specific examples. The numerical values, shapes, materials, constituent elements, arrangement positions and connection forms, steps, and order of the steps shown in the following embodiments are merely examples, and are not intended to limit the scope of the request. In addition, among the constituent elements of the following embodiments, constituent elements that are not described in the independent request item indicating the highest-level concept are described as arbitrary constituent elements. (Embodiment 1) [Outline of encoding device]

First, the outline of the encoding device according to the first embodiment will be described. FIG. 1 is a block diagram showing a functional configuration of an encoding device 100 according to the first embodiment. The encoding device 100 is a moving image / image encoding device that encodes moving images / images in units of blocks.

As shown in FIG. 1, the encoding device 100 is a device that encodes an image in units of blocks, and includes a division unit 102, a subtraction unit 104, a conversion unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, The inverse conversion unit 114, the addition unit 116, the block memory 118, the loop filtering unit 120, the frame memory 122, the in-frame prediction unit 124, the inter-frame prediction unit 126, and the prediction control unit 128.

The encoding device 100 can be implemented by, for example, a general-purpose processor and a memory. In this case, when the software program stored in the memory is executed by the processor, the processor functions as the division unit 102, the subtraction unit 104, the conversion unit 106, the quantization unit 108, the entropy encoding unit 110, the inverse quantization unit 112, and the inverse The conversion unit 114, the addition unit 116, the loop filtering unit 120, the intra-frame prediction unit 124, the inter-frame prediction unit 126, and the prediction control unit 128 function. The encoding device 100 may correspond to the division unit 102, the subtraction unit 104, the conversion unit 106, the quantization unit 108, the entropy encoding unit 110, the inverse quantization unit 112, the inverse conversion unit 114, the addition unit 116, the loop filter unit 120, One or more dedicated electronic circuits of the intra-frame prediction unit 124, the inter-frame prediction unit 126, and the prediction control unit 128 are implemented.

Hereinafter, each constituent element included in the encoding device 100 will be described. [Division]

The division unit 102 divides each picture included in the input moving image into a plurality of blocks, and outputs each block to the subtraction unit 104. For example, the dividing unit 102 may first divide a picture into blocks of a fixed size (for example, 128 × 128). This fixed-size block is sometimes called a coding tree unit (CTU). In addition, the division unit 102 divides each fixed-size block into a variable size (for example, 64 × 64 or less) based on recursive quadtree and / or binary tree block division. Block. This variable-sized block is sometimes called a coding unit (CU), a prediction unit (PU), or a conversion unit (TU). Furthermore, in this embodiment, it is not necessary to distinguish between CU, PU, and TU, and a part or all of the blocks in the picture may be processed by CU, PU, and TU.

FIG. 2 is a diagram showing an example of block division in the first embodiment. In FIG. 2, the solid line indicates a block boundary divided by a quaternary tree block, and the dotted line indicates a block boundary divided by a binary tree block.

Here, the block 10 is a square block of 128 × 128 pixels (128 × 128 blocks). This 128 × 128 block 10 is first divided into 64 square 64 × 64 blocks (quaternary tree block partition).

The upper left 64 × 64 block will be further vertically divided into 2 rectangular 32 × 64 blocks, and the left 32 × 64 block will be further vertically divided into 2 rectangular 16 × 64 blocks (two Meta-tree block partitioning). As a result, the upper left 64 × 64 block is divided into two 16 × 64 blocks 11, 12 and 32 × 64 block 13.

The upper right 64 × 64 block is horizontally divided into two rectangular 64 × 32 blocks 14, 15 (binary tree block division).

The 64 × 64 block in the lower left is divided into 4 square 32 × 32 blocks (quaternary tree block partition). Among the four 32 × 32 blocks, the upper left block and the lower right block are further divided. The upper left 32 × 32 block is vertically divided into two rectangular 16 × 32 blocks, and the right 16 × 32 block is further horizontally divided into two 16 × 16 blocks (binary tree block division ). The lower right 32 × 32 block is horizontally divided into two 32 × 16 blocks (binary tree block partition). As a result, the lower left 64 × 64 block is divided into: 16 × 32 block 16; two 16 × 16 blocks 17, 18; two 32 × 32 blocks 19 and 20; and two 32 × 16 blocks 21, 22.

The lower right 64 × 64 block 23 is not divided.

As above, in FIG. 2, the block 10 is divided according to the recursive quaternary and binary tree blocks, and is divided into 13 variable-size blocks 11 to 23. This segmentation is sometimes called QTBT (quad-tree plus binary tree) segmentation.

Moreover, in FIG. 2, although one block is divided into four or two blocks (quaternary tree or binary tree block division), the division is not limited to this. For example, one block may be divided into three blocks (ternary tree block division). This type of segmentation involving ternary tree block segmentation is sometimes referred to as MBT (multi type tree) segmentation. [Subtraction Division]

The subtraction unit 104 subtracts the prediction signal (prediction sample) from the original signal (original sample) in units of blocks divided by the division unit 102. That is, the subtraction unit 104 calculates a prediction error (also referred to as a residual) of a coding target block (hereinafter, referred to as a current block). Then, the subtraction unit 104 outputs the calculated prediction error to the conversion unit 106.

The original signal is an input signal of the encoding device 100 and is a signal (for example, a luma signal and two chroma signals) indicating an image of each picture constituting a moving image. Hereinafter, a signal representing an image may be referred to as a sample. [Conversion Department]

The conversion unit 106 converts the prediction error in the spatial region into a conversion coefficient in the frequency region, and outputs the conversion coefficient to the quantization unit 108. Specifically, the conversion unit 106 performs, for example, a predetermined discrete cosine transform (DCT) or discrete sine transform (DST) on the prediction error of the spatial region.

In addition, the conversion unit 106 may also adaptively select a conversion type from a plurality of conversion types, and use a transform basis function corresponding to the selected conversion type to convert the prediction error into a conversion coefficient. . This conversion is sometimes referred to as EMT (explicit multiple core transform) or AMT (adaptive multiple transform).

The plurality of conversion types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. FIG. 3 is a table showing conversion basis functions corresponding to each conversion type. In FIG. 3, N is the number of input pixels. The selection of the conversion type from among the plurality of conversion types may depend on, for example, the type of prediction (intra-prediction and inter-prediction), or it may depend on the in-frame prediction mode. set.

This type of information indicating whether EMT or AMT is applicable (may be called, for example, AMT flag) and information indicating the selected conversion type is signaled at the CU level. Furthermore, the signalization of this information does not need to be limited to the CU level, but may be other levels (e.g., sequence level, picture level, slice level, tile level ( tile level) or CTU level).

The conversion unit 106 may convert the conversion coefficient (conversion result) again. Such retransforms are sometimes referred to as AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the conversion unit 106 performs re-conversion for each sub-block (for example, a 4 × 4 sub-block) included in a block corresponding to a conversion coefficient of the in-frame prediction error. Information showing whether NSST is applicable and information related to the conversion matrix used in NSST is signaled at the CU level. Furthermore, the signalization of this information does not need to be limited to the CU level, but may be other levels (e.g., sequence level, picture level, slice level, tile level ( tile level) or CTU level). [Quantization Department]

The quantization unit 108 quantizes the conversion coefficients output from the conversion unit 106. Specifically, the quantization unit 108 scans the conversion coefficients of the current block in a predetermined scanning order, and quantizes the conversion coefficients according to a quantization parameter (QP) corresponding to the scanned conversion coefficients. In addition, the quantization unit 108 outputs the quantized conversion coefficients (hereinafter, referred to as quantization coefficients) of the current block to the entropy encoding unit 110 and the inverse quantization unit 112.

The predetermined order is an order for quantization / inverse quantization of the conversion coefficients. For example, the predetermined scanning order is defined in ascending order of frequency (order from low frequency to high frequency) or descending order (order from high frequency to low frequency).

The quantization parameter is a parameter that defines a quantization step (quantization width). For example, increasing the value of the quantization parameter will increase the quantization step. In other words, if the value of the quantization parameter increases, the quantization error increases. [Entropy coding section]

The entropy coding unit 110 generates a coding signal (encoded bit stream) by variable-length coding the quantization coefficient input from the quantization unit 108. Specifically, for example, the entropy coding unit 110 binarizes the quantization coefficient and performs arithmetic coding on the binary signal. [Inverse quantization section]

The inverse quantization unit 112 performs inverse quantization on the quantization coefficients input from the quantization unit 108. Specifically, the inverse quantization unit 112 performs inverse quantization on the quantization coefficients of the current block in a predetermined scanning order. Then, the inverse quantization section 112 outputs the inverse quantized conversion coefficients of the current block to the inverse conversion section 114. [Inverse Conversion Department]

The inverse conversion unit 114 restores the prediction error by inversely converting the conversion coefficient input by the inverse quantization unit 112. Specifically, the inverse conversion unit 114 restores the prediction error of the current block by inverse conversion of the conversion coefficient corresponding to the conversion performed by the conversion unit 106. Then, the inverse conversion unit 114 outputs the restored prediction error to the addition unit 116.

Furthermore, since the restored prediction error loses information due to quantization, it does not agree with the prediction error calculated by the subtraction unit 104. That is, the restored prediction error includes a quantization error. [Addition Department]

The addition unit 116 adds the prediction error input by the inverse conversion unit 114 and the prediction sample input by the self-prediction control unit 128 to thereby construct a current block. Then, the adding unit 116 outputs the reconstructed block to the block memory 118 and the loop filtering unit 120. The reconstructed block is sometimes called a local decoding block. [Block memory]

The block memory 118 is a storage unit for storing the blocks referred to in the in-frame prediction and also the blocks in the encoding target picture (hereinafter, referred to as the current picture). Specifically, the block memory 118 stores the reconstructed blocks output from the adding unit 116. [Loop Filtering Division]

The loop filtering unit 120 performs loop filtering on the blocks reconstructed by the adding unit 116, and outputs the filtered reconstructed blocks to the frame memory 122. The so-called loop filter is a filter (in-loop filter) used in the coding loop, and includes, for example, a deblocking filter (DF) and a sample adaptive offset (Sample). Adaptive Offset (SAO) and Adaptive Loop Filter (ALF).

In ALF, a least square error filter for removing coding distortion can be applied. For example, it can be applied to each of the 2 × 2 sub-blocks in the current block, according to the direction of the local gradient and activity. ) To select one filter from a plurality of filters.

Specifically, first, a sub-block (for example, a 2 × 2 sub-block) may be classified into a plurality of classes (for example, 15 or 25 classes). The classification of sub-blocks is based on the direction and activity of the gradient. For example, the classification value C (for example, C = 5D + A) can be calculated by using the direction value D (for example, 0 ~ 2 or 0 ~ 4) of the gradient and the activity value A (for example, 0 ~ 4) of the gradient. Furthermore, the sub-block is classified into a plurality of categories (for example, 15 or 25 categories) according to the classification value C.

The direction value D of the gradient can be derived by, for example, comparing gradients in a plurality of directions (for example, horizontal, vertical, and two diagonal directions). The activity value A of the gradient is derived by, for example, adding gradients in a plurality of directions and quantifying the addition result.

Based on the results of this classification, a filter for a sub-block can be determined from a plurality of filters.

As the shape of the filter used in ALF, for example, a circularly symmetric shape can be used. 4A to 4C are diagrams showing plural examples of the shape of a filter used in the ALF. FIG. 4A shows a 5 × 5 diamond shape filter, FIG. 4B shows a 7 × 7 diamond shape filter, and FIG. 4C shows a 9 × 9 diamond shape filter. Information on the shape of the display filter is signaled at the picture level. Furthermore, the signalization of the information showing the shape of the filter does not need to be limited to the picture level, but may be other levels (for example, sequence level, slice level, tile level, CTU level, or CU level).

The on / off of ALF is determined at the picture level or the CU level, for example. For example, whether to apply ALF is determined at the CU level for brightness, and whether to apply ALF is determined at the picture level for color difference. The display of ALF on / off information is signaled at the picture level or the CU level. Furthermore, the signalization of the information showing the on / off of ALF need not be limited to the picture level or the CU level, but may be other levels (for example, the sequence level, the fragment level, the tile level, or the CTU level).

The coefficient sets of a plurality of selectable filters (for example, filters up to 15 or 25) are signaled at the picture level. In addition, the signalization of the coefficient group does not need to be limited to the picture level, but may be other levels (such as sequence level, fragment level, tile level, CTU level, CU level, or sub-block level). [Frame memory]

The frame memory 122 is a storage unit for storing reference pictures used for inter-frame prediction, and is sometimes referred to as a frame buffer. Specifically, the frame memory 122 stores the reconstructed blocks that have been filtered by the loop filtering unit 120. [In-frame prediction section]

The in-frame prediction section 124 refers to the blocks already stored in the current picture of the block memory 118 to perform in-frame prediction (also called in-frame prediction) of the current block, thereby generating a prediction signal (in-frame prediction signal) ). Specifically, the in-frame prediction unit 124 performs in-frame prediction with reference to samples (e.g., luminance values and color difference values) of blocks adjacent to the current block, thereby generating an in-frame prediction signal, and performs in-frame prediction. The signal is output to the prediction control unit 128.

For example, the intra-frame prediction unit 124 performs intra-frame prediction using one of a plurality of predetermined intra-frame prediction modes. The plurality of in-frame prediction modes include one or more non-directional prediction modes and a plurality of directional prediction modes.

The one or more non-directional prediction modes include, for example, a planar prediction mode and a DC prediction mode specified by H.265 / HEVC (High-Efficiency Video Coding) specifications (Non-Patent Document 1).

The plurality of directional prediction modes include, for example, prediction modes in 33 directions prescribed by the H.265 / HEVC standard. Furthermore, the plurality of directional prediction modes may include, in addition to 33 directions, prediction modes in 32 directions (a total of 65 directional prediction modes). FIG. 5 is a diagram showing 67 intra-frame prediction modes (2 non-directional prediction modes and 65 directional prediction modes) in the intra-frame prediction. The solid arrows indicate the 33 directions defined by the H.265 / HEVC standard, and the dotted arrows indicate the 32 additional directions.

Furthermore, in the frame prediction of the color difference block, a luminance block may also be referred to. That is, the color difference component of the current block can also be predicted based on the brightness component of the current block. Such in-frame prediction may be referred to as CCLM (cross-component linear model) prediction. Such an in-frame prediction mode of a color-difference block that refers to a luminance block (for example, referred to as a CCLM mode) can also be added as one of the in-frame prediction modes of a color-difference block.

The intra-frame prediction unit 124 may also correct the pixel values after intra-frame prediction based on the gradient of the reference pixels in the horizontal / vertical direction. Such intra-frame prediction accompanied by correction is sometimes referred to as PDPC (position-dependent intra prediction combination). The applicable information (referred to as, for example, the PDPC flag) that indicates the presence or absence of PDPC is signaled at, for example, the CU level. Furthermore, the signalization of this information does not need to be limited to the CU level, but may be other levels (such as sequence level, picture level, fragment level, tile level, or CTU level). [Inter-frame prediction section]

The inter-frame prediction unit 126 refers to a reference picture stored in the frame memory 122 and is a reference picture different from the current picture to perform inter-frame prediction (also called inter-frame prediction) of the current block, thereby generating a prediction signal. (Inter-frame prediction signal). Inter-frame prediction is performed in units of a current block or a sub-block (for example, a 4 × 4 block) within the current block. For example, the inter-frame prediction unit 126 performs a dynamic search (motion estimation) in the reference picture for the current block or subblock. In addition, the inter-frame prediction unit 126 uses motion information (for example, a motion vector) obtained by dynamic search to perform motion compensation, thereby generating an inter-frame prediction signal of a current block or a sub-block. Then, the inter-frame prediction unit 126 outputs the generated inter-frame prediction signal to the prediction control unit 128.

The motion information used for motion compensation is signaled. To signal the motion vector, a motion vector predictor may be used. That is, the difference signal between the motion vector and the motion vector predictor may be converted into a signal.

Furthermore, not only the dynamic information of the current block obtained by dynamic search, but also the dynamic information of adjacent blocks can also be used to generate inter-frame prediction signals. Specifically, the prediction signals of the dynamic information obtained by the dynamic search and the prediction signals of the dynamic information of the neighboring blocks can also be weighted and added, so as to generate the frame between the sub-block units in the current block. Forecast signal. Such inter-frame prediction (dynamic compensation) is sometimes referred to as OBMC (overlapped block motion compensation).

In this OBMC mode, information (for example, called OBMC block size) showing the size of subblocks for OBMC is signaled at the sequence level. The information indicating whether the OBMC mode is applicable (for example, the OBMC flag) is signaled at the CU level. Furthermore, the level of signalization of these information does not need to be limited to the sequence level and the CU level, but may also be other levels (such as the picture level, the fragment level, the tile level, the CTU level, or the sub-block level).

Furthermore, instead of signalizing the dynamic information, it may be derived on the decoding device side. For example, a merge mode defined by the H.265 / HEVC standard may be used. In addition, for example, dynamic information may be derived by performing a dynamic search on the decoding device side. In this case, a dynamic search can be performed without using the pixel values of the current block.

Here, a mode for performing dynamic search on the decoding device side will be described. This mode of performing dynamic search on the decoding device side is sometimes referred to as a PMMVD (pattern matched motion vector derivation) mode or a FRUC (flame rate up-conversion) mode.

First, referring to the motion vector of a coded block that is adjacent to the current block in space or time, a plurality of candidate lists each having a motion vector predictor can be generated (or common to the merged list). Then, an evaluation value of each candidate included in the candidate list is calculated, and one candidate is selected based on the evaluation value.

Furthermore, the motion vector for the current block can be derived based on the selected candidate motion vector. Specifically, for example, the selected candidate motion vector is derived as it is as a motion vector for the current block. In addition, for example, in a peripheral area of a position in a reference picture corresponding to the selected candidate motion vector, a pattern-matching is performed to derive a motion vector for the current block.

The evaluation value is calculated by pattern matching between a region in the reference picture corresponding to the motion vector and a predetermined region.

As the pattern matching, the first pattern matching or the second pattern matching can be used. The first pattern matching and the second pattern matching are sometimes referred to as bilateral matching and template matching, respectively.

In the first pattern matching, pattern matching is performed between two blocks that are two blocks in two different reference pictures and that are motion trajectory along the current block. Therefore, in the first pattern matching, as a predetermined area for calculating the candidate evaluation value, an area in another reference picture along the movement track of the current block is used.

FIG. 6 is a diagram for explaining pattern matching (two-way matching) between two blocks along a moving track. As shown in FIG. 6, in the first type matching, it is 2 in the two reference pictures (Ref0, Ref1) that are two blocks along the current track (Cur block) and are different Among the pairs of blocks, the best matching pair is searched to derive 2 motion vectors (MV0, MV1).

Under the assumption of continuous moving trajectories, it means that the motion vectors (MV0, MV1) of 2 reference blocks will be relative to the time distance between the current picture (Cur Pic) and the 2 reference pictures (Ref0, Ref1). (TD0, TD1) are proportional. For example, if the current picture is located between two reference pictures in time, and the time distance from the current picture to the two reference pictures is equal, in the first type matching, a mirror-symmetric two-way Move vector.

In the second type matching, the template in the current picture (the block adjacent to the current block in the current picture (such as the upper and / or left adjacent blocks)) and the block in the reference picture Pattern matching between. Therefore, in the second type matching, a block adjacent to the current block in the current picture is used as a predetermined region for calculating the candidate evaluation value.

FIG. 7 is a diagram for explaining pattern matching (template matching) between a template in a current picture and a block in a reference picture. As shown in FIG. 7, in the second type of matching, the reference picture (Ref0) is searched for the block that matches the block closest to the current block in the current picture (Cur Pic). Block to derive the motion vector for the current block.

This type of information (for example, a FRUC flag) indicating whether the FRUC mode is applicable is signaled at the CU level. In addition, when the FRUC mode is applied (for example, when the FRUC flag is true), the information of the pattern matching method (the first pattern match or the second pattern match) is displayed (for example, it can be referred to as the FRUC mode flag) (Subscript) is signaled at the CU level. Furthermore, the signalization of such information does not need to be limited to the CU level, but may be other levels (for example, sequence level, picture level, fragment level, tile level, CTU level, or sub-block level).

Furthermore, the dynamic information can also be derived on the decoding device side by a method different from the dynamic search. For example, based on a model that assumes a constant-speed linear motion, the correction amount of the motion vector may be calculated using the surrounding pixel values in pixel units.

Here, a mode for deriving a motion vector from a model that assumes a constant-speed linear motion will be described. This mode is sometimes called a BIO (bi-directional optical flow) mode.

FIG. 8 is a diagram for explaining a model in which constant-speed linear motion is assumed. In FIG. 8, (v _x , v _y ) are velocity vectors, and τ ₀ and τ ₁ each represent a time distance between a current picture (Cur Pic) and two reference pictures (Ref ₀ , Ref ₁ ). (MVx ₀ , MVy ₀ ) is a motion vector corresponding to the reference picture Ref ₀ , and (MVx ₁ , MVy ₁ ) is a motion vector corresponding to the reference picture Ref ₁ .

At this time, under the assumption of constant velocity linear motion of the velocity vector (v _x , v _y ), (MVx ₀ , MVy ₀ ) and (MVx ₁ , MVy ₁ ) are each expressed as (v _x τ ₀ , v _y τ ₀ ) and (-v _x τ ₁ , -v _y τ ₁ ), and the following optical flow equation (1) is established. [Mathematical formula 1]

Here, I ^(k) represents the luminance value of the reference image k (k = 0, 1) after motion compensation. This optical flow equation represents the following (i), (ii), and (iii) equal to zero: (i) temporal differentiation of the luminance value, (ii) horizontal velocity and the level of the spatial gradient of the reference image The product of the components and (iii) the product of the vertical components of the velocity in the vertical direction and the spatial gradient of the reference image. According to the combination of the optical flow equation and Hermite interpolation, the motion vector of the block unit obtained from the combined list and the like can be corrected in pixel units.

In addition, the motion vector may be derived on the decoding device side by a method different from the derivation of the motion vector from a model in which a constant-speed linear motion is assumed. For example, the motion vector may be derived in units of sub-blocks based on the motion vectors of a plurality of adjacent blocks.

Here, a mode for deriving a motion vector in units of sub-blocks based on the motion vectors of a plurality of adjacent blocks will be described. This mode is sometimes called the affine motion compensation prediction mode.

FIG. 9 is a diagram for explaining derivation of a motion vector based on a sub-block unit of a motion vector of a plurality of adjacent blocks. In FIG. 9, the current block contains 16 4 × 4 sub-blocks. Here, the movement vector v ₀ of the upper left corner control point of the current block is derived based on the movement vector of the neighboring block, and the movement of the upper right corner control point of the current block is derived based on the movement vector of the neighboring sub-block Vector v ₁ . Then, using two motion vectors v ₀ and v ₁ , the motion vector (v _x , v _y ) of each sub-block in the current block is derived by the following formula (2). [Mathematical formula 2]

Here, x and y each indicate a horizontal position and a vertical position of a sub-block, and w is a predetermined weighting factor.

In this affine motion compensation prediction mode, the method of deriving the motion vectors of the upper left and upper right control points can also include several different modes. Information (such as affine flags) showing such an affine motion-compensated prediction mode is signaled at the CU level. Furthermore, the signalization of the information showing the affine dynamic compensation prediction mode does not need to be limited to the CU level, but may be other levels (such as sequence level, picture level, fragment level, tile level, CTU level, or sub-regions). Block level). [Predictive Control Department]

The prediction control unit 128 selects any one of the intra-frame prediction signal and the inter-frame prediction signal, and outputs the selected signal to the subtraction unit 104 and the addition unit 116 as a prediction signal. [Outline of Decoding Device]

Next, an outline of a decoding device capable of decoding an encoded signal (encoded bit stream) output from the encoding device 100 will be described. FIG. 10 is a block diagram showing a functional configuration of the decoding device 200 according to the first embodiment. The decoding device 200 is a moving image / image decoding device that decodes moving images / images in units of blocks.

As shown in FIG. 10, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse conversion unit 206, an addition unit 208, a block memory 210, a loop filtering unit 212, a frame memory 214, and an in-frame prediction unit 216. , Inter-frame prediction unit 218, and prediction control unit 220.

The decoding device 200 may be implemented by, for example, a general-purpose processor and a memory. In this case, when the processor executes the software program stored in the memory, the processor functions as the entropy decoding unit 202, the inverse quantization unit 204, the inverse conversion unit 206, the addition unit 208, the loop filtering unit 212, and the in-frame prediction. The unit 216, the inter-frame prediction unit 218, and the prediction control unit 220 function. The decoding device 200 may also correspond to the entropy decoding unit 202, the inverse quantization unit 204, the inverse conversion unit 206, the addition unit 208, the loop filtering unit 212, the intra-frame prediction unit 216, the inter-frame prediction unit 218, and the prediction control unit 220. More than 1 dedicated electronic circuit.

Hereinafter, each constituent element included in the decoding device 200 will be described. [Entropy decoding section]

The entropy decoding unit 202 performs entropy decoding on a coded bit stream. Specifically, the entropy decoding unit 202 arithmetically decodes a binary signal from the encoded bit stream, for example. The entropy decoding unit 202 debinarizes the binary signal. As a result, the entropy decoding unit 202 outputs the quantization coefficient to the inverse quantization unit 204 in units of blocks. [Inverse quantization section]

The inverse quantization unit 204 performs inverse quantization on the quantization coefficients of a decoding target block (hereinafter, referred to as a current block) input from the entropy decoding unit 202. Specifically, the inverse quantization unit 204 performs inverse quantization on each quantization coefficient of the current block based on a quantization parameter corresponding to the quantization coefficient. In addition, the inverse quantization unit 204 outputs the quantized coefficients (that is, conversion coefficients) of the current block that have been inversely quantized to the inverse conversion unit 206. [Inverse Conversion Department]

The inverse conversion unit 206 restores the prediction error by inversely converting the conversion coefficient input by the inverse quantization unit 204.

In the case where, for example, it is displayed that the information decoded from the encoded bit stream is applicable to EMT or AMT (for example, the AMT flag is true), the inverse conversion section 206 will perform the current block analysis based on the information indicating the type of the decoded conversion. The inverse conversion is performed.

When the NSST is applied to information that has been decoded from the encoded bit stream, for example, the inverse conversion unit 206 applies inverse reconversion to the conversion coefficient. [Addition Department]

The addition unit 208 adds the prediction error input by the inverse conversion unit 206 and the prediction sample input by the self-prediction control unit 220 to thereby reconstruct the current block. The addition unit 208 outputs the reconstructed blocks to the block memory 210 and the loop filter unit 212. [Block memory]

The block memory 210 is a storage unit for storing blocks referenced in the intra-frame prediction and blocks in a picture to be decoded (hereinafter referred to as a current picture). Specifically, the block memory 210 stores the reconstructed blocks output from the adding unit 208. [Loop Filtering Division]

The loop filtering unit 212 performs loop filtering on the blocks reconstructed by the adding unit 208, and outputs the filtered reconstructed blocks to the frame memory 214, the display device, and the like.

When the display of the ALF on / off information decoded from the encoded bit stream shows that ALF is on, one filter can be selected from a plurality of filters according to the direction and activity of the local gradient. And apply the selected filter to the reconstructed block. [Frame memory]

The frame memory 214 is a storage unit for storing reference pictures used for inter-frame prediction, and is sometimes referred to as a frame buffer. Specifically, the frame memory 214 stores the reconstructed blocks that have been filtered by the loop filtering unit 212. [In-frame prediction section]

The in-frame prediction unit 216 performs in-frame prediction based on the in-frame prediction mode that has been decoded from the encoded bit stream and refers to the blocks stored in the current picture of the block memory 210, thereby generating a prediction signal ( Frame prediction signal). Specifically, the in-frame prediction unit 216 performs in-frame prediction with reference to samples (e.g., luminance values and color difference values) of blocks adjacent to the current block, thereby generating an in-frame prediction signal, and performs intra-frame prediction. The signal is output to the prediction control unit 220.

When the intra-frame prediction mode that refers to the luminance block is selected in the intra-frame prediction of the color difference block, the intra-frame prediction unit 216 may predict the color difference component of the current block based on the luminance component of the current block.

When the information decoded from the coded bit stream indicates that PDPC is applicable, the intra-frame prediction unit 216 corrects the pixel values after intra-frame prediction based on the gradient of the reference pixels in the horizontal / vertical direction. [Inter-frame prediction section]

The inter-frame prediction unit 218 refers to a reference picture stored in the frame memory 214 to predict a current block. The prediction is performed in units of the current block or a sub-block (for example, a 4x4 block) within the current block. For example, the inter-frame prediction unit 218 uses motion information (such as a motion vector) decoded from the encoded bit stream to perform dynamic compensation, thereby generating an inter-frame prediction signal of the current block or sub-block, and The prediction signal is output to the prediction control unit 220.

Furthermore, in the case where the information decoded from the encoded bit stream is displayed as being applicable to the OBMC mode, the inter-frame prediction unit 218 will use not only the dynamic information of the current block obtained through dynamic search, but also relevant information. Dynamic information of neighboring blocks to generate inter-frame prediction signals.

When the FRUC mode is applied to the information decoded from the encoded bit stream, the inter-frame prediction unit 218 performs dynamic search according to the pattern matching method (two-way matching or template matching) decoded from the encoded stream. Use this to export dynamic information. Then, the inter-frame prediction unit 218 performs motion compensation using the derived motion information.

When the BIO mode is applied, the inter-frame prediction unit 218 derives a motion vector from a model that assumes a constant-speed linear motion. When the affine motion compensation prediction mode is applied to the information decoded from the encoded bit stream, the inter-frame prediction unit 218 derives the movement in units of sub-blocks based on the motion vectors of a plurality of adjacent blocks. vector. [Predictive Control Department]

The prediction control unit 220 selects one of the intra-frame prediction signal and the inter-frame prediction signal, and outputs the selected signal to the addition unit 208 as a prediction signal. [Internal Structure of Conversion Unit of Encoding Device]

Next, an example of the internal configuration of the conversion unit 106 of the encoding device 100 will be described with reference to FIG. 11.

FIG. 11 is a block diagram showing the internal configuration of the conversion unit 106 of the encoding device 100 according to the first embodiment. The conversion unit 106 includes an in-frame / inter-frame determination unit 1061, a base selection unit 1062, and a frequency conversion unit 1063.

The in-frame / inter-frame determination unit 1061 determines which of intra-frame prediction and inter-frame prediction is used for the coding target block. For example, the in-frame / inter-frame determination unit 1061 determines which of the intra-frame prediction and the inter-frame prediction is used based on a comparison result with an image signal obtained by locally decoding an input image signal and a compressed image.

The base selection unit 1062 selects one base from a plurality of frequency conversion bases including DCT-II and DCT-V based on the determination result of the in-frame / inter-frame determination unit 1061. Specifically, when intra-frame prediction is used for the coding target block, the base selection unit 1062 is a base for selecting DCT-V. On the other hand, when inter-frame prediction is used for the coding target block, the base selection unit 1062 is a base for selecting DCT-II.

The frequency conversion unit 1063 performs frequency conversion of the prediction error (residual error) on the encoding target block using the base selected by the base selection unit 1062. That is, the frequency conversion unit 1063 is a base that selectively uses a plurality of frequency conversions including DCT-II and DCT-V to perform frequency conversion on the prediction error of the coding target block. Specifically, when intra-frame prediction is used for the coding target block, the frequency conversion unit 1063 performs frequency conversion using the base of DCT-V. On the other hand, when inter-frame prediction is used for the coding target block, the frequency conversion unit 1063 performs frequency conversion using the base of DCT-II.

In addition, the coefficients of the coding target block output by the frequency conversion unit 1063 are quantized and inversely quantized by the quantization unit 108 and the inverse quantization unit 112. The inverse conversion unit 114 performs inverse frequency conversion on the coefficients of the quantized and inverse-quantized encoding target block. At this time, the inverse conversion unit 114 selects a base of the inverse frequency conversion based on the information of the selection result of the base selection unit 1062. In other words, in the case where intra-frame prediction is used for encoding the target block, the base of the inverse conversion of DCT-V is selected, and when inter-frame prediction is used for encoding the target block, DCT is selected. -II is the base of the inverse conversion, and the selected base is used to implement the inverse frequency conversion. [Operation of Conversion Unit of Encoding Device]

Next, the operation of the conversion unit 106 configured as described above will be specifically described with reference to FIG. 12. FIG. 12 is a flowchart showing processing performed by the conversion unit 106 of the encoding device 100 according to the first embodiment.

First, the in-frame / inter-frame determination unit 1061 determines which of the intra-frame prediction and the inter-frame prediction is used for the coding target block (S101). Here, when intra-frame prediction is used for the coding target block (in the frame of S101), the base selection unit 1062 selects the base of DCT-V (S102). On the other hand, when inter-frame prediction is used for the coding target block (between the frames of S101), the base selection unit 1062 selects the base of DCT-II (S103). Finally, the frequency conversion unit 1063 performs frequency conversion on the prediction error of the coding target block using the base selected in step S102 or step S103 (S104). [Internal configuration of the inverse conversion unit of the decoding device]

Next, the internal configuration of the inverse conversion unit 206 of the decoding device 200 will be described.

FIG. 13 is a block diagram showing the internal configuration of the inverse conversion unit 206 of the decoding device 200 according to the first embodiment. The inverse conversion unit 206 includes an in-frame / inter-frame determination unit 2061, a base selection unit 2062, and an inverse frequency conversion unit 2063.

The in-frame / inter-frame determination unit 2061 determines which of intra-frame prediction and inter-frame prediction is used to decode the target block. For example, the in-frame / inter-frame determination unit 2061 makes a determination based on the information obtained from the bit stream.

The base selection unit 2062 selects one base from a plurality of bases of inverse frequency conversion including the inverse conversion of DCT-II and DCT-V based on the determination result of the in-frame / inter-frame determination unit 1061. Specifically, when intra-frame prediction is used for decoding the target block, the base selection unit 2062 is a base for selecting the inverse conversion of DCT-V. On the other hand, when inter-frame prediction is used for the coding target block, the base selection unit 2062 is a base for selecting the inverse conversion of DCT-II.

The inverse frequency conversion unit 2063 performs inverse frequency conversion on the coefficients of the decoding target block using the base selected by the base selection unit 2062. In other words, the inverse frequency conversion unit 2063 selectively performs the inverse frequency conversion on the coefficients of the decoding target block by using a plurality of inverse frequency conversion bases including the inverse conversion of DCT-II and DCT-V.

More specifically, when intra-frame prediction is used for decoding the target block, the inverse frequency conversion unit 2063 performs inverse frequency conversion using the base of the inverse conversion of DCT-V. On the other hand, when inter-frame prediction is used for decoding a target block, the inverse frequency conversion unit 2063 performs inverse frequency conversion using the base of the inverse conversion of DCT-II. [Operation of the Inverse Conversion Unit of the Decoding Device]

Next, the operation of the inverse conversion unit 206 configured as described above will be specifically described with reference to FIG. 14. FIG. 14 is a flowchart showing processing performed by the inverse conversion unit 206 of the decoding device 200 according to the first embodiment.

First, the in-frame / inter-frame determination unit 2061 determines which of the intra-frame prediction and the inter-frame prediction is used to decode the target block (S201). Here, when intra-frame prediction is used for decoding the target block (in the frame of S201), the base selection unit 2062 is a base for selecting the inverse conversion of DCT-V (S202). On the other hand, when inter-frame prediction is used for decoding the target block (between the frames of S201), the base selection unit 2062 selects the base of the inverse conversion of DCT-II (S203). Finally, the inverse frequency conversion unit 2063 performs inverse frequency conversion on the coefficients of the decoding target block using the base selected in step S202 or step S203 (S204). [Effects, etc.]

As described above, according to the conversion unit 106 of the encoding device 100 and the inverse conversion unit 206 of the decoding device 200 according to this embodiment, when intra-frame prediction is used for the current block, the base of DCT-V or The base of DCT-V's inverse conversion is to convert or inverse convert the current block. In DCT-V, the amplitude is reduced at a position close to the reference pixel in the DC component. Therefore, DCT-V is suitable for the conversion / inverse conversion of the prediction error in the frame prediction. Therefore, the encoding device 100 and the decoding device 200 can achieve a further improvement in compression efficiency.

Here, the DCT-V will be specifically described with reference to FIGS. 15A to 16B.

FIG. 15A is a graph showing the conversion characteristics of DCT-II in a 32 × 32 size block. FIG. 15B is a graph showing the conversion characteristics of DCT-V in a 32 × 32 size block. FIG. 16A is a graph showing the conversion characteristics of DST-VII in a 4 × 4 size block. FIG. 16B is a graph showing the conversion characteristics of DCT-V in a 4 × 4 size block. In FIGS. 15A to 16B, the horizontal axis indicates the distance from the reference pixel, and the vertical axis indicates the amplitude.

DCT-II is a discrete cosine transform of type II. In DCT-II, the base (base function) shown in FIG. 3 is used. As shown in FIG. 15A, in the direct current (0th order), the amplitude value is fixed regardless of the distance. In addition, DCT-II is effective for blocks that have the same intra-frame prediction error as the intra-block prediction error.

DCT-V is a discrete cosine transform of type V. In DCT-V, the base (base function) shown in FIG. 3 is used. As shown in FIG. 15B, in a relatively large block, DCT-V has a conversion characteristic similar to DCT-II. Also, as shown in FIG. 16B, in a relatively small block, the amplitude becomes smaller at a position closer to the reference pixel in DC, so the conversion characteristics of DCT-V are similar to the conversion characteristics of DST-VII.

In a block predicted in a frame, pixels close to a reference pixel (pixels on the left and upper sides) tend to have a smaller prediction error. However, since a larger block tends to be used when the prediction error is small, in a larger block, it is difficult to show a tendency that the prediction error becomes smaller in pixels close to the reference pixel.

Therefore, DCT-V, which has similar conversion characteristics as DCT-II in larger blocks and similar conversion characteristics as DST-VII in smaller blocks, can be applied to areas predicted in the frame. The frequency conversion of the block, thereby effectively performing frequency conversion / inverse frequency conversion on the block predicted in the frame from the smaller size to the larger size. Therefore, the base of the DCT-V can be used to transform the blocks predicted in the frame, thereby achieving a further improvement in compression efficiency. (Modification 1 of Embodiment 1)

Next, a first modification of the first embodiment will be described. In this modification, the basis used for frequency conversion and inverse frequency conversion depends on the size of the current block, which is different from the first embodiment described above. Hereinafter, this modification will be specifically described with reference to Figs. 17 to 20, focusing on the differences from the first embodiment. [Internal Structure of Conversion Unit of Encoding Device]

FIG. 17 is a block diagram showing the internal configuration of the conversion unit 106A of the encoding device 100 according to the first modification of the first embodiment. The conversion unit 106A includes an in-frame / inter-frame determination unit 1061, a base selection unit 1062A, a frequency conversion unit 1063, and a size determination unit 1064A.

The size determination unit 1064A determines whether the size of the encoding target block is equal to or smaller than a threshold size.

When intra-frame prediction is used for the coding target block, as long as the size of the coding target block is equal to or smaller than the threshold size, the base selection unit 1062A selects the base of the DCT-V. On the other hand, even when intra-frame prediction is used for the coding target block, as long as the size of the coding target block is larger than the threshold size, the base selection unit 1062A selects the base of the DCT-II. When inter-frame prediction is used for the coding target block, the base selection unit 1062A selects the base of DCT-II in the same manner as in the first embodiment.

As the threshold size indicating the boundary of the block size for switching the base, for example, a fixed size (for example, 16 × 16 pixels) that has been previously defined in a standard specification is used. In addition, the threshold size may be determined according to a signal included in the bit stream, and may also be input by an external device or a user. For example, the threshold size may be determined according to an intra-frame prediction mode, a quantization parameter, or a prediction error. [Operation of Conversion Unit of Encoding Device]

Next, the operation of the conversion unit 106A of the present modification configured as described above will be specifically described with reference to FIG. 18. FIG. 18 is a flowchart showing processing performed by the conversion unit 106A of the encoding device 100 according to the first modification of the first embodiment.

First, the in-frame / inter-frame determination unit 1061 determines which of the intra-frame prediction and the inter-frame prediction is used for the coding target block (S101). Here, when intra-frame prediction is used for the encoding target block (in the frame of S101), the size determination unit 1064A determines whether the size of the encoding target block is equal to or smaller than a threshold size (S111). Here, when the size of the coding target block is equal to or smaller than the threshold size (YES in S111), the base selection unit 1062A selects the base of the DCT-V (S102).

When inter-frame prediction is used for the coding target block (between the frames in S101) or when the size of the coding target block is larger than the threshold size (No in S111), the base selection unit 1062A will The base of DCT-II is selected (S103). Finally, the frequency conversion unit 1063 performs frequency conversion on the prediction error of the coding target block using the base selected in step S102 or step S103 (S104). [Internal configuration of the inverse conversion unit of the decoding device]

Next, the internal configuration of the inverse conversion unit 206A of the decoding device 200 will be described. FIG. 19 is a block diagram showing the internal configuration of the inverse conversion unit 206A of the decoding device 200 according to the first modification of the first embodiment. The inverse conversion unit 206A includes an in-frame / inter-frame determination unit 2061, a base selection unit 2062A, an inverse frequency conversion unit 2063, and a size determination unit 2064A.

The size determination unit 2064A determines whether the size of the decoding target block is equal to or smaller than the threshold size.

When intra-frame prediction is used for the decoding target block, as long as the size of the decoding target block is equal to or smaller than the threshold size, the base selection unit 2062A selects the base of the DCT-V. On the other hand, even when intra-frame prediction is used for the decoding target block, as long as the size of the decoding target block is larger than the threshold size, the base selection unit 2062A selects the base of DCT-II. When inter-frame prediction is used for the decoding target block, the base selection unit 2062A selects the base of DCT-II in the same manner as in the first embodiment.

As the threshold size, the same size as the threshold size used by the conversion unit 106A of the encoding device 100 is used. [Operation of the Inverse Conversion Unit of the Decoding Device]

Next, the operation of the inverse conversion unit 206A of the present modification configured as described above will be specifically described with reference to FIG. 20. FIG. 20 is a flowchart showing processing of the inverse conversion unit 206A of the decoding device 200 according to the first modification of the first embodiment.

First, the in-frame / inter-frame determination unit 2061 determines which of the intra-frame prediction and the inter-frame prediction is used to decode the target block (S201). Here, when intra-frame prediction is used for the decoding target block (in the frame of S201), the size determination unit 2064A determines whether the size of the decoding target block is equal to or smaller than the threshold size (S211). Here, when the size of the decoding target block is equal to or smaller than the threshold size (YES in S211), the base selection unit 2062A selects the base of the inverse conversion of DCT-V (S202).

When inter-frame prediction is used for the decoding target block (between the frames of S201) or when the size of the decoding target block is larger than the threshold size (No in S211), the base selection unit 2062A will The base of the inverse conversion of DCT-II is selected (S203). Finally, the inverse frequency conversion unit 2063 performs inverse frequency conversion on the coefficients of the decoding target block using the base selected in step S202 or step S203 (S204). [Effects, etc.]

As described above, according to the conversion unit 106A of the encoding device 100 and the inverse conversion unit 206A of the decoding device 200 according to this embodiment, the bases of DCT-II and DCT-V can be switched in accordance with the size of the current block predicted using the frame. , And perform conversion / inverse conversion on the current block. If the block size is large, the prediction error tends to become smaller in the block as a whole, and DCT-II is suitable for the conversion of the prediction error of the block. On the other hand, if the block size is small, the pixels whose prediction errors are closer to the reference pixels tend to become smaller, and DCT-V is suitable for the conversion of the prediction errors of the blocks. Therefore, if the size of the current block is below the threshold size, the current block is converted / inversely converted by DCT-V. If the size of the block to be encoded is larger than the threshold size, it is converted by DCT-II. The current block is converted / inversely converted, so that further compression efficiency can be improved. (Modification 2 of Embodiment 1)

Next, a second modification of the first embodiment will be described. This modification is different from the first modification of the first embodiment in that the information of the threshold size is written in the bit stream. Hereinafter, this modification will be specifically described with reference to Figs. 21 to 25, focusing on differences from the first modification of the first embodiment. [Internal Structure of Conversion Unit of Encoding Device]

FIG. 21 is a block diagram showing the internal configuration of the conversion unit 106B of the encoding device 100 according to the second modification of the first embodiment. The conversion unit 106B includes an in-frame / inter-frame determination unit 1061, a base selection unit 1062A, a frequency conversion unit 1063, a size determination unit 1064A, and a threshold size determination unit 1065B.

The threshold size determination unit 1065B determines the threshold size adaptively in response to an input image signal or the like. The determined threshold size is used in the size determination section 1064A.

The information of the determined threshold size is output to the entropy coding unit 110 and written into the bit stream. The information of the threshold size refers to the information of the display threshold size, for example, the value of the display threshold size itself or the index of the display threshold size. The threshold size information is written in at least one of the plural headers shown in (i) to (v) of FIG. 22, for example.

22 is a diagram showing plural examples of positions in a bit stream of information of a threshold size or a conversion pattern in a modification 2 or 3 of the first embodiment. (I) of FIG. 22 is information showing that there is a threshold size or a conversion mode in the video parameter set. (Ii) of FIG. 22 is information showing that there is a threshold size or a conversion mode in the sequence parameter set of the video stream. (Iii) of FIG. 22 is information showing that there is a threshold size or a conversion mode in a picture parameter set of a picture. (Iv) of FIG. 22 is information showing that there is a threshold size or a conversion mode in a clip header of a clip. (V) of FIG. 22 shows a case where there is information of a threshold size or a conversion mode in a group of parameters for performing set-up or initialization of a motion picture system or a video decoder. In the case where the information of the threshold size or the conversion mode exists in a plurality of levels (for example, a picture parameter set and a fragment header), the information of the threshold size or the conversion mode in a lower layer (for example, a fragment header) is overwritten Information on threshold sizes or conversion patterns that exist in higher layers (such as picture parameter sets).

Note that the threshold size information may be written only when it is changed. That is to say, when using the same threshold size as the threshold size just used before, it is also possible to skip writing the information of the threshold size. [Operation of Conversion Unit of Encoding Device]

Next, the operation of the conversion unit 106B of the present modification configured as described above will be specifically described with reference to FIG. 23. FIG. 23 is a flowchart showing processing performed by the conversion unit 106B of the encoding device 100 according to the second modification of the first embodiment.

First, the in-frame / inter-frame determination unit 1061 determines which of the intra-frame prediction and the inter-frame prediction is used for the coding target block (S101). Here, when intra-frame prediction is used for the coding target block (in the frame of S101), the threshold size determination unit 1065B adaptively determines the threshold size and outputs the information of the determined threshold size to Entropy coding unit 110 (S121). After that, the processes from step S111 onward are executed. [Internal configuration of the inverse conversion unit of the decoding device]

Next, the internal configuration of the inverse conversion unit 206B of the decoding device 200 will be described. FIG. 24 is a block diagram showing the internal configuration of the inverse conversion unit 206B of the decoding device 200 according to the second modification of the first embodiment. The inverse conversion unit 206B includes an in-frame / inter-frame determination unit 2061, a base selection unit 2062A, an inverse frequency conversion unit 2063, a size determination unit 2064A, and a threshold size acquisition unit 2065B.

The threshold size acquisition unit 2065B obtains a threshold size from the bit stream. For example, the threshold size acquisition unit 2065B obtains the threshold size based on the information of the threshold size decoded from the bit stream by the entropy decoding unit 202. The threshold size obtained here is used in the size determination section 2064A. [Operation of the Inverse Conversion Unit of the Decoding Device]

Next, the operation of the inverse conversion unit 206B of the present modification configured as described above will be specifically described with reference to FIG. 25. FIG. 25 is a flowchart showing processing performed by the inverse conversion unit 206B of the decoding device 200 according to the second modification of the first embodiment.

First, the in-frame / inter-frame determination unit 2061 determines which of the intra-frame prediction and the inter-frame prediction is used to decode the target block (S201). Here, when intra-frame prediction is used for the decoding target block (in the frame of S201), the threshold size obtaining unit 2065B obtains the threshold size from the bit stream (S221). After that, the processing from step S211 is executed. [Effects, etc.]

As described above, according to the conversion unit 106A of the encoding device 100 and the inverse conversion unit 206A of the decoding device 200 according to this embodiment, the information of the threshold size can be included in the bit stream. Therefore, the threshold size can be adaptively determined according to the input image, and further compression efficiency can be improved. (Modification 3 of Embodiment 1)

Next, a third modification of the first embodiment will be described. In this modification, the first conversion mode using the conversion and inverse conversion of the second modification of the first embodiment and the second conversion mode using other conversions and inverse conversion can be switched. This is the same as the modification of the first embodiment. Example 2 is different. Hereinafter, this modification will be specifically described with reference to FIGS. 26 to 29, focusing on points that are different from the second modification of the first embodiment. [Internal Structure of Conversion Unit of Encoding Device]

FIG. 26 is a block diagram showing the internal configuration of the conversion unit 106C of the encoding device 100 according to the third modification of the first embodiment. The conversion unit 106C includes an in-frame / inter-frame determination unit 1061, a base selection unit 1062C, a frequency conversion unit 1063, a size determination unit 1064A, a threshold size determination unit 1065B, and a conversion mode determination unit 1066C.

The conversion mode determination unit 1066C determines which conversion mode among a plurality of conversion modes including the first conversion mode and the second conversion mode is to be applied to the encoding target block. In the plurality of conversion modes, for example, the selectable substrates may be different from each other, or the selectable substrates may be the same but the selection methods may be different from each other.

Information suitable for the conversion mode of the encoding target block can be output to the entropy encoding section 110 and written into the bit stream. The information of the conversion mode refers to information for identifying the conversion mode, and may be, for example, a flag or an index displaying the conversion mode. The information of the conversion mode is written in at least one of the plural headers shown in (i) to (v) of FIG. 22 in the same manner as the information of the threshold size. Moreover, the information of the conversion mode and the information of the threshold size need not be written in the same header, and can also be written in different headers.

When the first conversion mode is applied, the base selection unit 1062C selects one base from among a plurality of frequency conversion bases including DCT-II and DCT-V, as in the second modification of the first embodiment. On the other hand, when the second conversion mode is applied, the base selection unit 1062C selects one base from a plurality of frequency conversion bases different from the first conversion mode.

The frequency conversion unit 1063 performs frequency conversion on the prediction error of the coding target block using the base selected by the base selection unit 1062C. That is, the frequency conversion unit 1063 performs the first frequency conversion when the first conversion mode is applied, and performs the second frequency conversion when the second conversion mode is applied.

The first frequency conversion is the same as the frequency conversion of the second modification of the first embodiment. The second frequency conversion is different from the first frequency conversion. For example, in the second frequency conversion, a base different from the first frequency conversion is used. Furthermore, the base used in the first frequency conversion and the base used in the second frequency conversion need not be different in all conditions, but can also be used in specific conditions (for example, the block size is a threshold value). The same applies to the case below the size). [Operation of Conversion Unit of Encoding Device]

Next, referring to Fig. 27, the operation of the conversion unit 106C according to the present modification configured as described above will be specifically described. FIG. 27 is a flowchart showing processing of the conversion unit 106C of the encoding device 100 according to the third modification of the first embodiment.

First, the in-frame / inter-frame determination unit 1061 determines which of the intra-frame prediction and the inter-frame prediction is used for the coding target block (S101). Here, when intra-frame prediction is used for the encoding target block (in the frame of S101), the conversion mode determination unit 1066C determines the conversion mode applicable to the encoding target block and outputs it to the entropy encoding unit 110. (S131).

Here, if the determined conversion mode is the first conversion mode (the first conversion mode in S132), the processes after step S121 are executed. On the other hand, if the determined conversion mode is the second conversion mode (the second conversion mode of S132), the base selection unit 1062C selects the base for the second conversion mode (S133).

Thereafter, the frequency conversion unit 1063 performs frequency conversion of the encoding target block using the base selected in step S102, step S103, or step S133 (S104).

As the substrate for the second conversion mode, eight types of substrates of type I to type VIII defined by the boundary conditions and the symmetry in each of DCT and DST can also be used. In this case, from a total of 16 types of bases, the base for encoding the target block may be selected based on the evaluation value taking into account the prediction error or the encoding amount of the information about the prediction error and the encoding of the prediction error. . For example, in the selection based on the prediction error, a base for minimizing the residual error may also be selected. In addition, the base for encoding the target block may be selected based on the intra-frame prediction mode in which the direction of the intra-frame prediction is displayed. Moreover, a single base may be fixedly selected without selecting a base for encoding a target block from a plurality of bases for frequency conversion. For example, the substrate of DST-VII may be fixedly used in a 4 × 4 size, and the substrate may be adaptively selected in other sizes. In addition, when the substrate can be fixedly selected, the signal showing the selected substrate may not be written into the bit stream, and only when the substrate is adaptively selected, the signal of the selected substrate will be displayed. Write to bit stream.

In the first conversion mode and the second conversion mode, the selectable bases may be different, or the selectable bases may be the same but the selection methods are different. In addition, in the first conversion mode and the second conversion mode, exclusive and repetition can be switched depending on the block size, and the same base can be selected in different block sizes, and the same area can be selected in different block sizes. Different substrates can be selected in the block size.

As a more specific example, in the second conversion mode, the DST-VII base can be fixedly selected when the block size is 4 × 4, and when the block size exceeds 4 × 4, Either the DST-I or DST-VII substrate can be adaptively selected. At this time, in the first conversion mode, the base of DCT-V can be fixedly selected when the block size is 16 × 16 or less, and fixed when the block size exceeds 16 × 16. Select the base of DCT-II.

As another specific example, in the second conversion mode, when the block size is 4 × 4, the base of DST-VII can be fixedly selected. When the block size exceeds 4 × 4, from One substrate group is selected from a plurality of substrate groups each including one or more substrates, and one substrate is selected from the selected substrate groups. The plurality of substrate groups may include, for example, a first substrate group composed of a substrate of DST-I and a substrate of DST-VII, and a second substrate group composed of a substrate of DCT-VIII and a substrate of DST-VII. In addition, the selection of the base group may also be performed according to an intra-frame prediction mode such as displaying the direction of the intra-frame prediction. Also, in the first conversion mode, the base of DCT-V can be fixedly selected when the block size is 16 × 16 or less, and fixedly selectable when the block size exceeds 16 × 16. The base of DCT-II. [Internal configuration of the inverse conversion unit of the decoding device]

Next, the internal configuration of the inverse conversion unit 206C of the decoding device 200 will be described. FIG. 28 is a block diagram showing an internal configuration of the inverse conversion unit 206C of the decoding device 200 according to the third modification of the first embodiment. The inverse conversion unit 206C includes an in-frame / inter-frame determination unit 2061, a base selection unit 2062C, an inverse frequency conversion unit 2063, a size determination unit 2064A, a threshold size acquisition unit 2065B, and a conversion mode determination unit 2066C.

The conversion mode determination unit 2066C determines which conversion mode among a plurality of conversion modes including the first conversion mode and the second conversion mode is to be applied to the decoding target block. For example, the conversion mode determination unit 2066C determines the conversion mode based on the information of the conversion mode decoded from the bit stream by the entropy decoding unit 202.

When the first conversion mode is applied, the base selection unit 2062C selects, from the plurality of inverse frequency conversion bases including the inverse conversion of DCT-II and DCT-V, as in the second modification of the first embodiment. 1 substrate. On the other hand, when the second conversion mode is applied, the base selection unit 2062C selects one base from a plurality of inverse frequency conversion bases different from the first conversion mode. [Operation of the Inverse Conversion Unit of the Decoding Device]

Next, the operation of the inverse conversion unit 206C of the present modification configured as described above will be specifically described with reference to FIG. 29. FIG. 29 is a flowchart showing processing performed by the inverse conversion unit 206C of the decoding device 200 according to the third modification of the first embodiment.

First, the in-frame / inter-frame determination unit 2061 determines which of the intra-frame prediction and the inter-frame prediction is used to decode the target block (S201). Here, when intra-frame prediction is used for the decoding target block (in the frame of S201), the conversion mode determination unit 2066C determines which conversion mode among a plurality of conversion modes is to be applied to the decoding target area. Block (S232). Here, when the first conversion mode is applied (the first conversion mode of S232), the processes from step S221 onward are executed. On the other hand, when the second conversion mode is applied (the second conversion mode of S232), the base selection unit 2062C selects the base for the second conversion mode (S233). The base for the second conversion mode selected here is a base for inverse conversion corresponding to the base for the second conversion mode selected by the encoding device 100.

Thereafter, the inverse frequency conversion unit 2063 performs inverse frequency conversion of the decoding target block using the base selected in step S202, step S203, or step S233 (S204). [Effects, etc.]

As described above, according to the conversion unit 106C of the encoding device 100 and the inverse conversion unit 206C of the decoding device 200 in this modification, it is possible to switch the frequency conversion using the conversion mode. As a result, it becomes possible to further improve the efficiency of frequency conversion, and further improve the compression efficiency.

In addition, according to the conversion unit 106C of the encoding device 100 and the inverse conversion unit 206C of the decoding device 200 in this modification, the information of the conversion mode applicable to the current block can be included in the bit stream. Therefore, it becomes possible to adaptively determine the conversion mode in accordance with the input image, and to achieve further improvement in compression efficiency.

In addition, in this modification, a case where two conversion modes (a first conversion mode and a second conversion mode) are used will be mainly described, but the number of conversion modes is not limited to two. For example, in addition to the first conversion mode and the second conversion mode, a third conversion mode and / or a fourth conversion mode may be used. (Other Modifications of Embodiment 1)

Although the encoding device and the decoding device according to one or more aspects of the present disclosure have been described based on the embodiments and the modifications, the present disclosure is not limited to the embodiments and the modifications. Without departing from the gist of the present disclosure, various modifications that can be conceived by a person having ordinary skill in the technical field to which the present invention pertains can be implemented in this embodiment or the modification, or a combination of different modifications The forms constructed by the requirements may also be included in the scope of one or more aspects of the present disclosure.

For example, in the first embodiment and each modification described above, the base of DST-VII may be selected for a 4 × 4 size luminance block as in the prior art. In this case, the DCT-V base can be selected for frequency conversion of a block other than the 4 × 4 size luminance block that is used for the intra-frame prediction block. In addition, for the color difference block using the intra-frame prediction, the base of DCT-II can be selected in the same manner as in the prior art. In this case, for the luminance block using the intra-frame prediction, the base of DCT-V can be selected.

Furthermore, in the above-mentioned Embodiment 1 and each modification, the in-frame / inter-frame determination unit 1061 performs frame processing based on a comparison result of an input image signal and an image signal obtained by locally decoding a compressed image. Which of the intra prediction and the inter-frame prediction is to be determined, but the determination may be made based on other signals.

Furthermore, in the above-mentioned Embodiment 1 and each modified example, although the basis of orthogonal conversion of DCT-II and DCT-V is used, a basis of non-orthogonal conversion having similar conversion characteristics may also be used. It replaces the base of DCT-II and DCT-V.

In addition, in the first embodiment and each modification described above, the base is selected based on the type of prediction (inside / between frames) or block size, but it is not limited to this. For example, the in-frame prediction mode, quantization parameter, or residual of the coding target block may be evaluated, and the base may be selected according to the evaluation result.

In addition, in the first embodiment and each modification described above, although the substrate is selected regardless of the brightness and chromatic aberration, it is not limited to this. For example, for color-difference blocks, the base of DCT-II may be fixedly used regardless of intra-frame prediction / inter-frame prediction.

Furthermore, in the above-mentioned Embodiment 1 and each modification, although the intra-frame prediction is used for the current block, the bases of DCT-II and DCT-V are used, but it is not limited to this. For example, in addition to the substrates of DCT-II and DCT-V, the substrates of DST-VII can also be used. For example, in the case where the in-frame prediction is applied to the current block, (i) the base of DST-VII can be used when the size of the current block is below the threshold size; (ii) the size of the current block is larger than the threshold size Use larger DCT-V substrates.

In addition, in the first embodiment and each modification described above, although the conversion unit or the inverse conversion unit is provided with the base selection unit, it may be clearly indicated that the base selection unit is not provided. In this case, the function of the base selection section may be integrated into the frequency conversion section or the inverse frequency conversion section. (Embodiment 2)

In the above embodiments and modifications, each of the functional blocks can usually be realized by an MPU, a memory, or the like. In addition, the processing performed by each of the function blocks is usually realized by a program execution unit such as a processor reading out and executing software (program) stored in a recording medium such as a ROM. The software may be distributed by downloading or the like, or may be stored in a recording medium such as a semiconductor memory and distributed. Moreover, of course, each functional block can also be realized by hardware (dedicated circuit).

The processes described in the embodiments and the modifications can be implemented by using a single device (system) to perform the centralized processing, or by using a plurality of devices to perform the distributed processing. In addition, the processor executing the above program may be a single processor or a plurality of processors. That is, centralized processing may be performed, or distributed processing may be performed.

The present invention is not limited by the above embodiments, and various changes can be made, and these are also included in the scope of the present invention.

Furthermore, application examples of the moving image encoding method (image encoding method) or the moving image decoding method (image decoding method) shown in the above-mentioned embodiments and modifications will be described below, and a system using the same. This system is characterized by having an image encoding device using an image encoding method, an image decoding device using an image decoding method, and an image encoding and decoding device having both. Other components in the system can be changed as appropriate. [Example of use]

FIG. 30 is a diagram showing the overall configuration of a content supply system ex100 that implements a content delivery service. The communication service providing area is divided into desired sizes, and base stations ex106, ex107, ex108, ex109, and ex110, which are fixed radio stations, are provided in each cell.

In this content supply system ex100, computers ex111, game consoles ex112, cameras ex113, home appliances ex114, and smart phones ex115 can be connected to each other through the Internet service provider ex102 or the communication network ex104, and the base stations ex106 to ex110. Connected to the internet ex101. The content supply system ex100 may be configured to combine and connect any of the above-mentioned requirements. The devices can also be directly or indirectly connected to each other through a telephone network or near-field wireless without passing through base stations ex106 to ex110, which are fixed radio stations. The streaming server ex103 is connected to various devices such as a computer ex111, a game machine ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 through the Internet ex101 and the like. The streaming server ex103 is connected to a terminal or the like in a hot spot in the aircraft ex117 through the satellite ex116.

Furthermore, wireless access points or hotspots can be used instead of base stations ex106 ~ ex110. In addition, the streaming server ex103 may be directly connected to the communication network ex104 without using the Internet ex101 or the Internet service provider ex102, or may be directly connected to the aircraft ex117 without using the satellite ex116.

The camera ex113 is a digital camera or the like that can perform still image photography and motion image photography. The smart phone ex115 is a smart phone, a mobile phone, or a PHS (Personal Handy Phone System) that corresponds to a mobile communication system generally called 2G, 3G, 3.9G, 4G, and 5G. , Personal Handyphone System).

The home appliance ex118 is a refrigerator or a device included in a domestic fuel cell gas-electric co-generation system.

In the content supply system ex100, a terminal having a photographing function is connected to the streaming server ex103 through a base station ex106 or the like, so that live transmission or the like becomes feasible. In real-time live transmission, the terminal (computer ex111, game console ex112, camera ex113, home appliances ex114, smart phone ex115, and terminal in aircraft ex117, etc.) will display the content of static or dynamic pictures taken by the user using the terminal Perform the encoding processing described in the above embodiment and each modification, and multiplex the image data obtained by encoding and the audio data that has been encoded with the sound corresponding to the image to obtain the obtained data. The data is sent to the streaming server ex103. That is, each terminal functions as an image encoding device according to one aspect of the present invention.

On the other hand, the streaming server ex103 sends a stream of content data transmitted by a requesting client. The client refers to a computer ex111, a game machine ex112, a camera ex113, a home appliance ex114, a smart phone ex115, and a terminal in an aircraft ex117 that can decode the data that has been encoded as described above. Each machine that has received the transmitted data will decode the received data and play it. That is, each device functions as an image decoding device according to one aspect of the present invention. [Decentralized processing]

In addition, the streaming server ex103 may be a plurality of servers or a plurality of computers, or may be a data that is distributed and processed or recorded for the sender. For example, the streaming server ex103 can be implemented by a CDN (Contents Delivery Network), or it can be performed by a number of edge servers scattered around the world and between edge servers. Connected network for content delivery. On the CDN, the edge servers that are physically close to each other are dynamically allocated according to customers. And, latency can be reduced by caching and passing content to the edge server. In addition, when a certain error occurs or the communication status is changed due to an increase in traffic, etc., it is possible to use a plurality of edge servers to distribute the processing or switch the sending subject to another edge server to bypass the obstacle that has occurred The network part and continuous transmission, so you can achieve high-speed and stable transmission.

In addition, not only the distributed processing of the transmission itself, but also the encoding processing of the photographed data can be performed at each terminal, it can also be performed at the server side, or it can be performed by sharing each other. As an example, in the encoding process, a processing loop is generally performed twice. In the first loop, it is possible to detect the complexity or coding amount of the image in the frame or scene unit. In the second loop, processing to maintain image quality and improve coding efficiency can be performed. For example, the terminal can perform the first encoding process and the server receiving the content can perform the second encoding process, thereby reducing the processing load on each terminal and improving the quality and efficiency of the content. At this time, as long as there is a request for receiving and decoding in a near-instant manner, the first encoding completion data that has been performed by the terminal can also be received and played by other terminals, so more flexible instant transmission can also be achieved.

As another example, a camera ex113 or the like extracts feature quantities from an image, compresses data related to the feature quantities as meta data, and transmits the data to a server. The server performs compression corresponding to the meaning of the image, such as judging the importance of an object from a feature amount and switching the quantization accuracy. The feature quantity data is particularly effective for improving the accuracy and efficiency of motion vector prediction when the server is compressed again. In addition, simple coding such as VLC (Variable Length Coding) may be performed on the terminal, and coding with a large processing load such as CABAC (full text adaptive binary arithmetic coding method) may be performed on the server.

In addition, as another example, in a sports field, a shopping mall, a factory, or the like, there may be cases where a plurality of image data are captured by a plurality of terminals to capture almost the same scene. At this time, a plurality of terminals that have been photographed, and other terminals and servers that have not been photographed according to needs, may be used, for example, GOP (Group of Picture) units, picture units, or pictures have been divided The resulting tile units are assigned encoding processing and distributed processing. Thereby, the delay can be reduced, and real-time can be realized more.

In addition, since the plurality of image data are almost the same scene, it can also be managed and / or instructed on the server as: the image data photographed at each terminal can be referred to each other. Alternatively, the server may receive the coding completion data from each terminal, and change the reference relationship between the plurality of data, or correct or replace the picture itself and re-encode. In this way, a stream that has improved the quality and efficiency of each piece of data can be generated.

In addition, the server may transmit the image data after transcoding the encoding method of the image data. For example, the server can also convert MPEG encoding to VP encoding, and it can also convert H.264 to H.265.

In this way, the encoding process can be performed by a terminal or one or more servers. Therefore, although descriptions such as "server" or "terminal" are used as the subject of processing, some or all of the processing performed on the server may be performed on the terminal, and processing performed on the terminal may also be performed on the server. Part or all of it. The same applies to the decoding process. [3D, multi-angle]

In recent years, different scenes captured by terminals such as multiple cameras ex113 and / or smart phones ex115, which are almost synchronized with each other, or images or videos taken from different angles of the same scene are integrated and used. The practice is also gradually increasing. The images captured by each terminal are integrated based on the relative positional relationship between the terminals obtained separately, or the areas where the feature points of the images are consistent.

Not only the two-dimensional moving image coding, but also the server can automatically encode and transmit the static image to the receiving terminal according to the scene analysis of the moving image or at a time designated by the user. Furthermore, when the server can obtain the relative positional relationship between the photographing terminals, not only a two-dimensional moving image, but also an image photographed from different angles of the same scene to generate a three-dimensional shape of the scene. In addition, the server can additionally encode the three-dimensional data generated by the point cloud, etc., and can also use the three-dimensional data to identify or track people or targets, from the images taken by multiple terminals. Select, or reconstruct, and generate an image to be transmitted to the receiving terminal.

In this way, the user can arbitrarily select each image corresponding to each photographing terminal to enjoy the scene, and can also enjoy the content of cutting out an image of an arbitrary viewpoint from the three-dimensional data reconstructed using a plurality of images or images. In addition, like the video, the sound can also be received from a plurality of different angles, and the server can also multiplex and transmit the sound and video from a specific angle or space with the video.

In addition, in recent years, contents that establish a correspondence between the real world and the virtual world, such as Virtual Reality (VR) and Augmented Reality (AR), have also gradually spread. In the case of VR images, the server can also create right-view and left-view viewpoint images, and use Multi-View Coding (Multi-View Coding, MVC) to allow permissible reference between the viewpoint images. The encoding may be performed as a different stream without referring to each other. When decoding different streams, they can be synchronized with each other and played back to reproduce the virtual three-dimensional space in accordance with the user's point of view.

In the case of AR images, the server will superimpose the virtual object information in the virtual space with the camera information in the real space according to the three-dimensional position or the activity of the user's viewpoint. The decoding device can also obtain and maintain virtual object information and three-dimensional data, and generate a two-dimensional image and smoothly link it in response to the user's viewpoint activity to create overlapping data. Alternatively, in addition to the request of the virtual object information, the decoding device can also transmit the user's viewpoint activity to the server, and the server cooperates with the viewpoint activity received from the three-dimensional data held on the server to Create overlapping data, and encode and send the overlapping data to the decoding device. In addition, the superimposed data may have an alpha value that shows penetration beyond RGB, and the server may set the alpha value of parts other than the target made from the three-dimensional data to 0, etc. To encode. Alternatively, the server may set a RGB value of a predetermined value as the background in the form of a chroma key, and generate data for setting a part other than the target as the background color.

Similarly, the decoding process of the transmitted data can be performed at each terminal of the client, it can also be performed at the server side, and it can also be performed in a shared manner. As an example, a terminal may temporarily transmit a reception request to a server, and receive the content in response to the request at another terminal and perform a decoding process, and then transmit the decoded signal to a device with a display. It does not depend on the performance of the communicable terminal itself to decentralize the processing and select the appropriate content, which can play back good-quality data. Also, as another example, a TV or the like may be used to receive large-sized image data, and a part of a region such as a tile after the picture is divided may be decoded and displayed on the personal terminal of the viewer. In this way, you can share the overall picture, and you can confirm the area you are responsible for or the area you want to confirm in more detail.

In addition, in the future, the following situations can be expected: regardless of indoor and outdoor conditions, in the case of close-range, middle-range, or long-range wireless communication that can be used multiple times, using the transmission system specifications such as MPEG-DASH, Communication switches the appropriate data while receiving content seamlessly. Thereby, the user can freely select not only his own terminal, but also a decoding device or a display device such as a display installed inside or outside the house, and switch instantly. In addition, it is possible to perform decoding while switching between a terminal to be decoded and a terminal to be displayed based on its own location information and the like. Thereby, it is possible to move while moving to the destination while displaying map information on a part of the wall or floor of a nearby building in which displayable elements are embedded. In addition, the encoded data can be cached to a server that can be accessed from the receiving terminal in a short time, or copied to an edge server in a content delivery server, etc., based on the encoding on the network. Ease of data access to switch the bit-rate of the received data. [Adjustable coding]

The content switching will be described using the adjustable stream which is compression-encoded by applying the moving image encoding method shown in the above-mentioned embodiment and each modification shown in FIG. 31. Although the server has a plurality of streams with the same content but different qualities as individual streams, it is not a problem, but it can also be configured as a temporally / spatially adjustable stream realized by layering and encoding as shown in the figure. Stream characteristics are used to switch content. That is, the decoding side can decide which layer to decode according to the external factors such as performance and internal factors such as the state of the communication band. The decoding side can freely switch between low-resolution content and high-resolution content. To decode. For example, when you want to use a device such as Internet TV to watch the follow-up video on a mobile phone ex115 after you return home, the device only needs to decode the same stream to different layers, so it can be reduced. Burden on the server side.

In addition, as described above, in addition to realizing a structure in which a picture is coded for each layer and an enhancement layer exists above the base layer, the enhancement layer can also include an image based on the image. Metadata and other meta-information, and make the decoding side super-parse the pictures in the base layer based on the meta-information to generate high-quality content. The so-called super-resolution may be any of an increase in the SN ratio and an increase in the resolution in the same resolution. The meta-information includes: information for linear or non-linear filter coefficients used in specific super-analysis processing, or information about filter values used in specific super-analysis processing, mechanical learning, or parameter values in a least square operation.

Alternatively, it may be configured to divide a picture into tiles or the like in accordance with the meaning of an object or the like in the image, and cause the decoding side to select a tile to be decoded and decode only a part of the area. In addition, by storing the attributes of the target (person, car, ball, etc.) and the position in the image (coordinate position in the same image, etc.) as meta-information, the decoding side can specify the desired target's target based on the meta-information. Position, and decide which tile contains the target. For example, as shown in FIG. 32, the meta-information may be stored using a data storage structure different from the pixel data, such as the SEI message in HEVC. This meta information indicates, for example, the position, size, or color of the main target.

The meta information may be stored in units of a plurality of pictures, such as a stream, a sequence, or a random access unit. With this, the decoding side can obtain the moment when a specific person appears in the image, etc., and compare it with the information of the picture unit, so that the picture where the target exists and the position of the target in the picture can be specified. [Optimization of web pages]

FIG. 33 is a diagram showing an example of a display screen displaying a web page on a computer ex111 or the like. FIG. 34 is a diagram showing an example of a display screen displaying a web page in a smartphone ex115 or the like. As shown in FIG. 33 and FIG. 34, when a web page includes a plurality of linked images belonging to a link to an image content, its appearance will vary depending on the components viewed. When a plurality of linked images can be seen on the screen, the display device (decoding device) is all until the user explicitly selects the linked image, or the linked image is near the center of the screen or the entire linked image enters the screen. It displays static pictures or intra-frame coded pictures (Intra Picture, I-Picture) with various contents as linked images, or displays multiple still pictures or I-Pictures (I-Picture), such as animated gif images, Or only receive the base layer to decode and display the image.

When the linked image has been selected by the user, the display device sets the base layer as the highest priority for decoding. Furthermore, as long as there is information showing adjustable content in the HTML constituting the webpage, the display device can also be decoded to the enhancement layer. In addition, in order to guarantee the timeliness, the display device can decode and display only forward reference pictures (in-frame coded picture (I-Picture), predictive picture (Predictive)) before the selection or when the communication frequency band is very strict. Picture (P-Picture) and forward-referenced Bidirectionally Predictive Picture (B-Picture)) to reduce the delay between the decoding time and the display time of the previous picture (from the decoding of the content to the display) Delay between starts). In addition, the display device may deliberately ignore all the reference relationships of pictures and set all the bi-directionally estimated coded pictures (B-Picture) and predicted pictures (P-Picture) to forward reference for rough decoding, and over time, and The received picture is added for normal decoding. [Automatic driving]

In addition, when transmitting and receiving two-dimensional or three-dimensional map information and other static maps or image data for automatic driving or driving support of a car, the receiving terminal may not only include image data belonging to one or more layers, but also Weather and construction information are also received as meta-information and decoded in accordance with these. Furthermore, meta-information can belong to layers, or it can simply be multiplexed with image data.

At this time, since the vehicle including the receiving terminal, drone, or airplane will move, the receiving terminal will transmit the location information of the receiving terminal when receiving the request, thereby seamlessly switching base stations ex106 to ex110 while achieving Receiving and decoding. In addition, the receiving terminal can dynamically switch to what degree the meta-information is to be received or to which degree the map information is to be updated according to the user's selection, the user's condition, or the state of the communication band.

As described above, in the content supply system ex100, the client can immediately receive the encoded information transmitted by the user, and decode and play it. [Send personal content]

In addition, in the content supply system ex100, not only high-quality and long-term content from video distribution companies, but also unicast or multicast transmission of low-quality and short-term content from individuals. It can be done. It is thought that such personal content will continue to increase in the future. In order to make personal content better, the server may also perform encoding processing after editing processing. This can be achieved by, for example, the following configuration.

The server will do it in real time or cumulatively when shooting, and after shooting, it will identify the shooting errors, scene search, meaning analysis, and target detection from the original image or the coded data. Moreover, the server will perform the following manual or automatic editing based on the recognition results: correction of out-of-focus or camera shake, deletion of less important scenes such as scenes with lower brightness or unfocused scenes, emphasis on the edges of the target, Tone changes, etc. The server will encode the edited data according to the edit result. In addition, it is also well known that when the shooting time is too long, the ratings will decrease. The server will automatically perform not only the low-importance scenes, but also the less dynamic ones, according to the image processing results. Scenes, etc. are also clipped to make them content within a specific time range according to the shooting time. Alternatively, the server may generate and encode an excerpt based on the result of the meaning analysis of the scene.

In addition, there are cases where personal content infringes copyright, personality rights, or portrait rights, as it is, and it is inconvenient for individuals when the scope of sharing exceeds the scope of sharing. Case. Therefore, for example, the server may intentionally change and encode a human face in the periphery of the screen, or the inside of a house, into an unfocused image. In addition, the server can also recognize whether a face of a person different from the person registered in advance is photographed in the encoding target image, and if it is photographed, perform processing such as mosaicing the face portion. Alternatively, as a pre- or post-encoding process, from the viewpoint of copyright, the user can specify the person or background area to be processed in the image, and then make the server replace the specified area with another image, Alternatively, processing such as blurring the focus is also possible. If it is a person, it is possible to replace the face part of the face while tracking the person in the moving image.

In addition, since the viewing and listening of personal content with a small amount of data has strong requirements for immediacy, although it also depends on the frequency bandwidth, the decoding device first receives the base layer first and then decodes and plays it. The decoding device can also receive the enhancement layer during this period, and in the case of loop playback or more than two times, the enhancement layer is also included to play the high-quality image. As long as the stream can be adjusted and coded like this, it can provide a kind of rough dynamic picture when it is not selected or the first time it is seen, but the stream will gradually be smart and the image will be changed. Good experience. In addition to the adjustable encoding, the same experience can be provided even if the rough stream for the first playback and the second stream encoded with reference to the first motion picture are configured as one stream. [Other use cases]

In addition, these encoding or decoding processes are generally processed in the LSIex500 that each terminal has. The LSIex500 may be a single chip, or may be formed of a plurality of chips. In addition, you can also install software for encoding or decoding moving images on a recording medium (CD-ROM, flexible disk, or hard disk) that can be read on a computer such as ex111, and use it. The software performs encoding or decoding. In addition, when the smart phone ex115 is equipped with a camera, it can also transmit dynamic image data obtained by the camera. The dynamic map data at this time is data processed and encoded by the LSIex500 included in the smart phone ex115.

Furthermore, the LSIex500 may be configured to download and activate application software. At this time, the terminal first determines whether the terminal corresponds to the encoding method of the content, or whether it has the ability to execute specific services. When the terminal does not have a coding method corresponding to the content, or does not have the ability to execute specific services, the terminal downloads a codec or application software, and then obtains and plays the content.

Moreover, it is not limited to the content supply system ex100 via the Internet ex101, and at least a moving image encoding device (image encoding device) or a moving image decoding device of the above-mentioned embodiment and each modification may be installed in the digital playback system. (Image Decoding Device). Since satellites are used to transmit and receive images and sounds that have been multiplexed with multiplexed data on radio waves for transmission and reception, it will be more suitable for the content supply system ex100 which is easy to form a unicast. The multicast is different, but the encoding and decoding processes can still be used for the same application. [Hardware composition]

FIG. 35 is a diagram showing a smart phone ex115. FIG. 36 is a diagram showing a configuration example of a smartphone ex115. The smartphone ex115 includes an antenna ex450 for transmitting and receiving radio waves to and from the base station ex110, a camera section ex465 capable of capturing images and still images, a display showing images captured by the camera section ex465, and an antenna ex450. Display section ex458 of decoded data such as received images. The smart phone ex115 further includes an operation unit ex466 such as a touch panel, a sound output unit ex457 for outputting a sound or a speaker, a sound input unit ex456 for inputting a microphone, and the like, and can store a photographed image. Or still pictures, recorded sounds, received images or still pictures, mail and other encoded data, or decoded data memory part ex467, and the slot part ex464 as the interface between the SIMex468, the SIMex468 It is used for specific users and authenticates access to various data, including the Internet. Furthermore, an external memory may be used instead of the memory section ex467.

The main control unit ex460 that controls the display unit ex458 and the operation unit ex466 in an integrated manner is connected to the power circuit unit ex461, the operation input control unit ex462, the image signal processing unit ex455, the camera interface portion ex463, and the display control unit through a bus ex470. The ex459, the modulation / demodulation unit ex452, the multiplexing / demultiplexing unit ex453, the audio signal processing unit ex454, the slot unit ex464, and the memory unit ex467 are connected.

When the power supply circuit section ex461 sets the power key to the on state by a user's operation, the smart phone ex115 is activated to operate by supplying power to each section from a battery pack.

The smart phone ex115 performs processing such as calling and data communication according to the control of the main control unit ex460 including the CPU, ROM, and RAM. During a call, the sound signal received by the sound input unit ex456 is converted into a digital sound signal in the sound signal processing unit ex454, and spread-spectrum processing is performed by the modulation / demodulation unit ex452, and then the transmission / reception unit ex451 After digital analog conversion processing and frequency conversion processing are performed, transmission is performed through the antenna ex450. In addition, the received data is amplified and subjected to frequency conversion processing and analog digital conversion processing, and the demodulation processing is performed by the modulation / demodulation unit ex452, and then converted to the analog sound signal by the sound signal processing unit ex454, and then the sound is output. The mini ex457 outputs it. In the data communication mode, the main body part's operation part ex466 and other operations are used to send the text (text), still pictures, or image data to the main control part ex460 through the operation input control part ex462, and the same transmission and reception processing is performed. . When images, still images, or images and sounds are transmitted in the data communication mode, the image signal processing unit ex455 will be stored in the memory by the dynamic image encoding method shown in the above embodiment and each modification. The image signal of the unit ex467 or the image signal input from the camera unit ex465 is compression-encoded, and the encoded image data is sent to the multiplex / separation unit ex453. In addition, the sound signal processing unit ex454 encodes the sound signals received by the sound input unit ex456 when shooting images or still pictures with the camera unit ex465, and sends the encoded sound data to the multiplexing / separation unit ex453. The multiplexing / separating section ex453 multiplexes the encoded video data and the encoded audio data in a predetermined manner, and uses a modulation / demodulation section (modulation / demodulation circuit section) ex452 and transmission / The receiving unit ex451 performs modulation processing and conversion processing, and transmits it through the antenna ex450.

In the case where an image attached to an e-mail or a web chat or an image linked to a web page is received, in order to decode the received multiplexed data via the antenna ex450, the multiplex / separation section ex453 will separate the multiplex by The multiplexed data is divided into a bit stream of image data and a bit stream of sound data, and then the encoded image data is supplied to the image signal processing unit ex455 through the synchronous bus ex470, and the encoded sound data is supplied. To the sound signal processing unit ex454. The image signal processing unit ex455 decodes the image signal by a dynamic image decoding method corresponding to the moving image encoding method shown in the above embodiment and each modification, and displays the linked image from the display unit ex458 through the display control unit ex459. An image or still image contained in a moving image file. The audio signal processing unit ex454 decodes the audio signal and outputs audio from the audio output unit ex457. Furthermore, since real-time streaming has become popular, depending on the user's situation, sound playback may occur in places that are not suitable for sound generation in society. Therefore, as an initial value, it is desirable to have a structure that only plays video data without playing a sound signal. The sound can also be played synchronously only when the user performs operations such as clicking on image data.

In addition, although the smart phone ex115 has been described as an example here, as a terminal, the following three types of assembly can be considered: in addition to a transmitting and receiving terminal having both an encoder and a decoder, there is also an A transmitting terminal and a receiving terminal having only a decoder. In addition, although the digital broadcasting system is described by receiving or transmitting multiplexed data that has been multiplexed with audio data, such as video data, in the multiplexed data, in addition to audio data, Images are multiplexed with related text data, and can receive or send image data itself instead of multiplexed data.

In addition, although the case where the main control unit ex460 including the CPU is used to control the encoding or decoding process has been described, the terminal may include a GPU in many cases. Therefore, it can also be configured to utilize the performance of the GPU and process a wider area together by utilizing the memory common to the CPU and the GPU or the memory that manages addresses in a way that can be used in common. This can reduce encoding time, ensure immediateness, and achieve low latency. In particular, for dynamic search, deblock filter, SAO (Sample Adaptive Offset), and conversion and quantization processing, the GPU is used in units of pictures and other units without using the CPU. It is efficient when combined. Industrial availability

The present disclosure can be used in, for example, a television receiver, a digital video recorder, a car navigation system, a mobile phone, a digital camera, or a digital video camera.

10 ~ 23‧‧‧block
100‧‧‧ encoding device
102‧‧‧Division
104‧‧‧Subtraction Division
106, 106A, 106B, 106C‧‧‧ Conversion Department
108‧‧‧Quantitative Department
110‧‧‧Entropy coding department
112, 204‧‧‧ Inverse quantification department
114, 206, 206A, 206B, 206C‧‧‧ Inverse conversion department
116, 208‧‧‧Addition Department
118, 210‧‧‧ block memory
120, 212‧‧‧loop filtering department
122, 214‧‧‧ frame memory
124, 216‧‧‧ Frame prediction department
126, 218‧‧‧‧ Inter-frame prediction department
128, 220‧‧‧ Predictive Control Department
200‧‧‧ decoding device
202‧‧‧Entropy Decoding Department
1061, 1061A, 2061‧‧‧ In-frame / inter-frame determination unit
1062, 1062A, 1062C, 2062, 2062A, 2062C‧‧‧Base selection department
1063‧‧‧Frequency Conversion Department
1064A, 2064A‧‧‧Size determination unit
1065B‧‧‧Threshold size determination unit
1066C, 2066C ‧‧‧ Conversion mode determination unit
2063‧‧‧ Inverse frequency conversion section
2065B‧‧‧Threshold size acquisition unit
_{MV0, MV1, MVx 0, MVy} 0, MVx 1, MVy 1, v0, v1‧‧‧ motion vector
Ref0, Ref1‧‧‧ reference pictures
TD0, TD1, τ ₀ , τ ₁ ‧‧‧ distance
S101 ~ S104, S201 ~ S204, S111, S121, S131, S132, S133, S211, S221, S232, S233
ex100‧‧‧Content Supply System
ex101‧‧‧Internet
ex102‧‧‧Internet Service Provider
ex103‧‧‧streaming server
ex104‧‧‧Communication Network
ex106, ex107, ex108, ex109, ex110‧‧‧ base station
ex111‧‧‧Computer
ex112‧‧‧Game console
ex113‧‧‧Camera
ex114‧‧‧Household appliances
ex115‧‧‧Smartphone
ex116‧‧‧ satellite
ex117‧‧‧plane
ex450‧‧‧antenna
ex451‧‧‧Transmission / Reception Department
ex452‧‧‧Modulation / Demodulation Section (Modulation / Demodulation Circuit Section)
ex453‧‧‧Multiplexing / Separation Department
ex454‧‧‧Sound Signal Processing Department
ex455‧‧‧Image Signal Processing Department
ex456‧‧‧Voice input section
ex457‧‧‧Sound output
ex458‧‧‧Display
ex459‧‧‧Display Control
ex460‧‧‧Main Control Department
ex461‧‧‧Power circuit department
ex462‧‧‧Operation input control section
ex463‧‧‧camera face
ex464‧‧‧Slot
ex465‧‧‧Camera Department
ex466‧‧‧Operation Department
ex467‧‧‧Memory
ex468‧‧‧SIM
ex470‧‧‧Bus
ex500‧‧‧LSI

FIG. 1 is a block diagram showing a functional configuration of an encoding device according to the first embodiment. FIG. 2 is a diagram showing an example of block division in the first embodiment. FIG. 3 is a table showing conversion basis functions corresponding to each conversion type. FIG. 4A is a diagram showing an example of the shape of a filter used in ALF. FIG. 4B is a diagram showing another example of the shape of a filter used in ALF. FIG. 4C is a diagram showing another example of the shape of a filter used in ALF. FIG. 5 is a diagram showing 67 intra-frame prediction modes in the intra-frame prediction. FIG. 6 is a diagram for explaining pattern matching (two-way matching) between two blocks along a moving track. FIG. 7 is a diagram for explaining pattern matching (template matching) between a template in a current picture and a block in a reference picture. FIG. 8 is a diagram for explaining a model in which constant-speed linear motion is assumed. FIG. 9 is a diagram for explaining derivation of a motion vector based on a sub-block unit of a motion vector of a plurality of adjacent blocks. Fig. 10 is a block diagram showing a functional configuration of a decoding device according to the first embodiment. Fig. 11 is a block diagram showing an internal configuration of a conversion unit of the encoding device of the first embodiment. Fig. 12 is a flowchart showing processing performed by a conversion unit of the encoding device of the first embodiment. FIG. 13 is a block diagram showing an internal configuration of an inverse conversion section of the decoding device according to the first embodiment. Fig. 14 is a flowchart showing processing performed by an inverse conversion unit of the decoding device of the first embodiment. FIG. 15A is a graph showing the conversion characteristics of DCT-II in a 32 × 32 size block. FIG. 15B is a graph showing the conversion characteristics of DCT-V in a 32 × 32 size block. FIG. 16A is a graph showing the conversion characteristics of DCT-II in a 4 × 4 size block. FIG. 16B is a graph showing the conversion characteristics of DCT-V in a 4 × 4 size block. FIG. 17 is a block diagram showing an internal configuration of a conversion unit of an encoding device according to a first modification of the first embodiment. FIG. 18 is a flowchart showing processing performed by a conversion unit of an encoding device according to a first modification of the first embodiment. 19 is a block diagram showing an internal configuration of an inverse conversion section of a decoding device according to a first modification of the first embodiment. FIG. 20 is a flowchart showing processing by an inverse conversion unit of the decoding device according to the first modification of the first embodiment. FIG. 21 is a block diagram showing an internal configuration of a conversion unit of an encoding device according to a second modification of the first embodiment. 22 is a diagram showing plural examples of positions in a bit stream of information of a threshold size or a conversion pattern in a modification 2 or 3 of the first embodiment. FIG. 23 is a flowchart showing processing performed by a conversion unit of an encoding device according to a second modification of the first embodiment. FIG. 24 is a block diagram showing an internal configuration of an inverse conversion unit of a decoding device according to a second modification of the first embodiment. 25 is a flowchart showing processing performed by an inverse conversion unit of a decoding device according to a second modification of the first embodiment. FIG. 26 is a block diagram showing an internal configuration of a conversion unit of an encoding device according to a third modification of the first embodiment. FIG. 27 is a flowchart showing processing performed by a conversion unit of an encoding device according to a third modification of the first embodiment. FIG. 28 is a block diagram showing an internal configuration of an inverse conversion unit of a decoding device according to a third modification of the first embodiment. FIG. 29 is a flowchart showing a process of an inverse conversion unit of a decoding device according to a third modification of the first embodiment. FIG. 30 is an overall configuration diagram of a content supply system that implements a content delivery service. FIG. 31 is a diagram showing an example of a coding structure during adjustable coding. FIG. 32 is a diagram showing an example of a coding structure during adjustable coding. FIG. 33 is a diagram showing an example of a display screen displaying a web page. FIG. 34 is a diagram showing an example of a display screen for displaying a web page. FIG. 35 is a diagram showing an example of a smartphone. FIG. 36 is a block diagram showing a configuration example of a smartphone.

104‧‧‧Subtraction Division

106‧‧‧ Conversion Department

108‧‧‧Quantitative Department

114‧‧‧ Inverse Conversion Department

1061‧‧‧In-frame / between-frame judging section

1062‧‧‧Base Selection Division

1063‧‧‧Frequency Conversion Department

Claims (12)

An encoding device is an encoding device that encodes an encoding target block of an image, and includes a processor and a memory. The processor uses the memory and determines that a frame is used for the encoding target block. Which of intra-prediction and inter-frame prediction, and selectively using a plurality of frequency conversion bases including DCT-II and DCT-V to perform the first frequency conversion on the prediction error of the coding target block, as described above In the first frequency conversion, when intra-frame prediction is used for the aforementioned coding target block, the base of DCT-V is used, and when inter-frame prediction is used for the aforementioned coding target block, It is a substrate using DCT-II. The encoding device according to claim 1, wherein the processor further determines whether the size of the encoding target block is equal to or smaller than a threshold size, and in the first frequency conversion, what is used for the encoding target block is In the case of in-frame prediction: (i) if the size of the aforementioned coding target block is below the aforementioned threshold size, the base of DCT-V is used; (ii) if the size of the aforementioned coding target block is greater than the aforementioned threshold size Using DCT-II substrate. The encoding device as claimed in claim 2, wherein the processor further writes the information of the threshold size into the bit stream. The encoding device according to any one of claims 1 to 3, wherein the processor further determines which conversion mode among a plurality of conversion modes including the first conversion mode and the second conversion mode is applicable to the encoding. The target block performs the first frequency conversion when the first conversion mode is applied, and performs the second frequency conversion different from the first frequency conversion when the second conversion mode is applied. The encoding device as claimed in claim 4, wherein the processor further writes information of a conversion mode applicable to the encoding target block into the bit stream. An encoding method is an encoding method for encoding an encoding target block of an image. The encoding method determines which of intra-frame prediction and inter-frame prediction is used for the foregoing encoding target block, and selectively uses the same. The first frequency conversion of the prediction error of the coding target block is performed based on a plurality of frequency conversion bases of DCT-II and DCT-V. The first frequency conversion is used for the coding target block. In the case of intra-frame prediction, the base of DCT-V is used, and when the inter-frame prediction is used for the aforementioned coding target block, the base of DCT-II is used. A decoding device is a decoding device that decodes a decoding target block of an image, and includes a processor and a memory. The processor uses the memory and determines whether the decoding target block is used for the decoding target block. Which of intra-frame prediction and inter-frame prediction, and selectively uses a plurality of inverse frequency conversion bases including inverse conversion of DCT-II and DCT-V to perform the first prediction error of the foregoing decoding target block. Inverse frequency conversion. In the first inverse frequency conversion, when in-frame prediction is used for the decoding target block, it is a basis for inverse conversion using DCT-V. In the case of using inter-frame prediction, it is the basis of the inverse conversion using DCT-II. The decoding device according to claim 7, wherein the processor further determines whether the size of the decoding target block is equal to or smaller than a threshold size, and in the first inverse frequency conversion, the decoding unit uses the In the case of in-frame prediction, (i) if the size of the aforementioned decoding target block is below the aforementioned threshold size, the base of the inverse conversion of DCT-V is used; (ii) if the size of the aforementioned decoding target block is larger than For the aforementioned threshold size, the base of the inverse conversion of DCT-II is used. The decoding device according to claim 8, wherein the processor further interprets the information of the threshold size from the bit stream. The decoding device according to any one of claims 7 to 9, wherein the processor further determines which conversion mode among a plurality of conversion modes including the first conversion mode and the second conversion mode is suitable for the decoding. The target block performs the first inverse frequency conversion when the first conversion mode is applied, and performs the second inverse frequency conversion different from the first inverse frequency conversion when the second conversion mode is applied. . The decoding device according to claim 10, wherein the processor further interprets the information of the conversion mode applicable to the decoding target block from the bit stream. A decoding method is a decoding method for decoding a decoding target block of an image. The decoding method is to determine which of intra-frame prediction and inter-frame prediction is used for the foregoing decoding target block, and selectively use the same. The first inverse frequency conversion for the prediction error of the decoding target block is performed by a plurality of inverse frequency conversion bases including the inverse conversion of DCT-II and DCT-V. In the first inverse frequency conversion, When in-frame prediction is used for the decoding target block, it is the base of inverse conversion using DCT-V. When in-frame prediction is used for the foregoing decoding target block, DCT-II is used. Base of inverse transformation.