JP2022120221A - Image decoding device, image decoding method, and image decoding program - Google Patents

Abstract

To provide a technique for improving coding efficiency by performing prediction suitable for image coding and decoding.SOLUTION: An image decoding device includes an intra prediction unit that intra-predicts and refreshes a partial area in a picture, a filter unit that filters a decoded image, a refresh control unit that controls the filter unit such that the filter processing is performed by referring to the pixels in an area that has been refreshed, a decoded image memory that stores the output of the filter unit as a clean area, and an inter prediction unit that refers to the clean area in the inter prediction of the refreshed area.SELECTED DRAWING: Figure 2

Description

The present invention relates to image encoding and decoding techniques for dividing an image into blocks and performing prediction.

In image encoding and decoding, an image to be processed is divided into blocks each of which is a set of a predetermined number of pixels, and each block is processed. Encoding efficiency is improved by dividing into appropriate blocks and appropriately setting intra-prediction and inter-prediction.

Patent Literature 1 discloses an intra-prediction technique for obtaining a predicted image using decoded pixels adjacent to a block to be encoded/decoded.

JP 2009-246975 A

However, the technique of Patent Literature 1 uses only decoded pixels adjacent to a block to be encoded/decoded for prediction, resulting in poor prediction efficiency.

In one aspect of the present invention for solving the above problems, an intra prediction unit that performs intra prediction and refreshes a partial region within a picture, a filter unit that performs filtering processing on a decoded image, and in the filtering processing, the refreshing is a refresh control unit for controlling the filtering unit so as to refer to the pixels in the refreshed area and perform filtering; a decoded image memory for storing the output of the filtering unit as a clean area; and an inter prediction unit that refers to the clean region in prediction.

According to the present invention, highly efficient image encoding/decoding processing can be realized with a low load.

1 is a block diagram of an image coding device according to an embodiment of the present invention; FIG. 1 is a block diagram of an image decoding device according to an embodiment of the present invention; FIG. FIG. 11 is a flowchart for explaining an operation of dividing a treeblock; FIG. FIG. 4 is a diagram showing how an input image is divided into treeblocks; FIG. 4 is a diagram for explaining z-scan; It is a figure which shows the division|segmentation shape of a block. 4 is a flowchart for explaining an operation of dividing a block into four; 4 is a flowchart for explaining an operation of dividing a block into two or three; It is a figure for demonstrating intra prediction. FIG. 10 is a diagram for explaining uni-prediction motion-compensated prediction; FIG. 4 is a diagram for explaining motion compensation prediction of bi-prediction; FIG. 4 is a diagram for explaining motion compensation prediction of bi-prediction; It is a figure which shows the influence on the pixel value by each filter process. FIG. 10 is a diagram showing how intra refresh is performed in GDR encoding; FIG. 10 is a diagram showing how random access prevents correct decoding; FIG. 10 is a diagram showing how random access prevents correct decoding; 2 is a diagram showing details of a decoded image signal superimposing unit 110 in the image encoding device of FIG. 1; FIG. 3 is a diagram showing details of a decoded image signal superimposing unit 207 in the image decoding device of FIG. 2. FIG. FIG. 3 is a diagram illustrating processing of a refresh control unit 114 and a refresh control unit 214; FIG. 10 is a diagram showing pixels to be referred to in filtering and pixels to be changed; It is a figure which shows the mode of padding. 1 is a diagram for explaining an example of a hardware configuration of an encoding/decoding device according to a first embodiment; FIG.

Techniques and technical terms used in this embodiment are defined.

<POC>
POC (Picture Order Count) is a variable associated with a picture to be encoded, and is set to a value that increases by 1 in the order in which pictures are output. Based on the POC value, it is possible to determine whether the pictures are the same, determine the sequential relationship between the pictures in the output order, and derive the distance between the pictures. For example, if two pictures have the same POC value, it can be determined that they are the same picture. When two pictures have different POC values, it can be determined that the picture with the smaller POC value is the picture to be output first, and the difference between the POCs of the two pictures determines the inter-picture distance in the time axis direction. show.

<Tree Block>
In the embodiment, an image to be encoded/decoded is equally divided into a predetermined size. This unit is defined as a treeblock. Although the size of the treeblock is 128×128 pixels in FIG. 4, the size of the treeblock is not limited to this, and any size may be set. The treeblock to be processed (corresponding to the encoding target in the encoding process and the decoding target in the decoding process) is switched in raster scan order, that is, from left to right and from top to bottom. The interior of each treeblock can be further recursively split. A block to be coded/decoded after treeblock division is defined as a coded block. Also, tree blocks and coding blocks are collectively defined as blocks. Appropriate block division enables efficient encoding. The size of the treeblock can be a fixed value that is prearranged between the encoding device and the decoding device, or can be configured such that the treeblock size determined by the encoding device is transmitted to the decoding device.

<Prediction mode>
For each encoding block to be processed, the image to be processed has been processed (in the encoding process, the signal that has been encoded is used for the decoded image, image signal, tree block, block, encoding block, etc. In the decoding process, Intra prediction (MODE_INTRA), which performs prediction from the surrounding image signals of decoded images, image signals, tree blocks, blocks, coding blocks, etc.), and inter prediction, which performs prediction from the image signals of processed images Switch (MODE_INTER). A mode for distinguishing between intra prediction (MODE_INTRA) and inter prediction (MODE_INTER) is defined as a prediction mode (PredMode). The prediction mode (PredMode) has intra prediction (MODE_INTRA) or inter prediction (MODE_INTER) as a value.

<Inter Prediction>
In inter prediction, in which prediction is performed from image signals of processed images, a plurality of processed images can be used as reference pictures. In order to manage a plurality of reference pictures, two types of reference lists, L0 (reference list 0) and L1 (reference list 1), are defined, and reference pictures are specified using reference indices. L0 prediction (Pred_L0) is available for P slices. L0 prediction (Pred_L0), L1 prediction (Pred_L1), and bi-prediction (Pred_BI) are available for B slices. L0 prediction (Pred_L0) is inter prediction that refers to reference pictures managed by L0, and L1 prediction (Pred_L1) is inter prediction that refers to reference pictures that are managed by L1. Bi-prediction (Pred_BI) is inter-prediction in which both L0 prediction and L1 prediction are performed and one reference picture managed by each of L0 and L1 is referred to. Information specifying L0 prediction, L1 prediction, and bi-prediction is defined as an inter-prediction mode. In the subsequent processing, it is assumed that the constants and variables with the suffix LX attached to the output are processed for each of L0 and L1.

(First embodiment)
An image encoding device 100 and an image decoding device 200 according to the first embodiment of the present invention will be described.

FIG. 1 is a block diagram of an image coding device 100 according to the first embodiment. The moving image coding apparatus according to the embodiment includes an image coding apparatus 100, a block division unit 101, an inter prediction unit 102, an intra prediction unit 103, a decoded image memory 104, a prediction method determination unit 105, a residual signal generation unit 106, An orthogonal transform/quantization unit 107 , a bit stream encoding unit 108 , an inverse quantization/inverse orthogonal transform unit 109 , a decoded image signal superimposing unit 110 , and an encoded information storage memory 111 are provided.

The block division unit 101 recursively divides an input image to generate coding blocks. The block dividing unit 101 includes a 4-dividing unit that horizontally and vertically divides a block to be divided, and a 2-3 dividing unit that horizontally or vertically divides a block to be divided. . The generated image signal of the coding block to be processed is supplied to the inter prediction unit 102 , the intra prediction unit 103 and the residual signal generation unit 106 . Information indicating the determined recursive partitioning structure is also supplied to the bit stream encoding unit 108 . A detailed operation of the block division unit 101 will be described later.

The inter prediction unit 102 performs inter prediction of the coding block to be processed. A plurality of candidates for inter prediction information are derived from the prediction mode stored in the encoded information storage memory and the decoded image signal stored in the decoded image memory 104, and a suitable inter prediction mode is selected from among the plurality of candidates. is selected, and the selected inter-prediction mode and the predicted image signal corresponding to the selected inter-prediction mode are supplied to the prediction method determination unit 105 .

The intra prediction unit 103 performs intra prediction on the encoding block to be processed. Generate a predicted image signal by intra prediction from the decoded image signal stored in the decoded image memory 104, select a suitable intra prediction mode from among a plurality of intra prediction modes, select the selected intra prediction mode, and A predicted image signal corresponding to the selected intra prediction mode is supplied to the prediction method determination unit 105 . FIG. 9 shows an example of intra prediction. FIG. 9A shows the correspondence between prediction directions of intra prediction and prediction mode numbers. For example, prediction mode 50 produces intra-predicted images by copying pixels vertically. Prediction mode 1 is a DC mode, and is a mode in which all pixel values of a block to be processed are average values of reference pixels. Prediction mode 0 is a planar mode, and is a mode in which a two-dimensional intra prediction image is created from reference pixels in the vertical and horizontal directions. FIG. 9B is an example of generating an intra-prediction image in prediction mode 40. FIG. The value of the reference pixel in the direction indicated by the prediction mode is copied to each pixel of the block to be processed. When the reference pixel in the prediction mode is not at the integer position, the reference pixel value is determined by interpolation from the reference pixel values at the surrounding integer positions.

The intra prediction unit 103 can also perform intra block copy prediction. This is a process of encoding/decoding a block to be processed by referring to decoded pixels in the picture to be processed as predicted values. The distance from the block to be processed to the pixel to be referred to is represented by a block vector. The difference between a block vector and a motion vector is whether the picture referred to is a picture to be processed or a picture that has been processed.

The decoded image memory 104 stores the decoded image generated by the decoded image signal superimposing unit 110 . Also, the decoded image stored in the decoded image memory is supplied to the inter prediction unit 102 and the intra prediction unit 103 .

The prediction method determination unit 105 evaluates each prediction using the encoded information and the code amount of the residual signal, the amount of distortion between the predicted image signal and the image signal to be processed, and the like, thereby determining the optimum prediction. Determine the mode (inter prediction or intra prediction). The determined encoded information is supplied to the encoded information storage memory 111 .

The residual signal generation unit 106 generates a residual signal by subtracting the predicted image signal from the image signal to be processed, and supplies the residual signal to the orthogonal transformation/quantization unit 107 .

Orthogonal transformation/quantization section 107 performs orthogonal transformation and quantization on the residual signal according to a quantization parameter to generate an orthogonal transformation/quantized residual signal, and performs bit stream encoding section 108 and inverse quantization. supplied to the normalization/inverse orthogonal transformation unit 109 .

The bit stream encoding unit 108 encodes encoding information according to the prediction method determined by the prediction method determination unit 105 for each encoding block, in addition to the information for each sequence, picture, slice, and encoding block. Specifically, the prediction mode PredMode and the division mode PartMode are encoded for each encoding block. The coded information is coded according to a prescribed syntax (syntax rules for coded bit strings) to generate a first coded bit string. In addition, bit string encoding section 108 entropy-encodes the orthogonally transformed and quantized residual signal according to a prescribed syntax to generate a second encoded bit string. The first encoded bit stream and the second encoded bit stream are multiplexed according to a prescribed syntax, and a bit stream is output.

The inverse quantization/inverse orthogonal transformation unit 109 performs inverse quantization and inverse orthogonal transformation on the orthogonally transformed/quantized residual signal supplied from the orthogonal transformation/quantization unit 107 to calculate a residual signal, and decodes it. It is supplied to the image signal superimposing unit 110 .

The decoded image signal superimposition unit 110 superimposes the prediction image signal determined by the prediction method determination unit 105 and the residual signal inversely quantized and inverse orthogonally transformed by the inverse quantization/inverse orthogonal transformation unit 109 to generate a decoded image. is generated and stored in the decoded image memory 104 . Note that the decoded image may be stored in the decoded image memory 104 after being subjected to filtering processing for reducing distortion such as block distortion due to encoding. Details will be described later.

The coding information storage memory 111 stores coding information such as the prediction mode (inter prediction or intra prediction) determined by the prediction method determination unit 105 .

FIG. 2 is a block diagram showing the configuration of a video decoding device according to an embodiment of the present invention corresponding to the video encoding device of FIG. The video decoding device of the embodiment includes a bit string decoding unit 201, a block division unit 202, an inter prediction unit 203, an intra prediction unit 204, an encoded information storage memory 205, an inverse quantization/inverse orthogonal transformation unit 206, a decoded image signal A superimposing unit 207 and a decoded image memory 208 are provided.

The decoding process of the video decoding device in FIG. 2 corresponds to the decoding process provided inside the video encoding device in FIG. The components of the inverse orthogonal transform unit 206, the decoded image signal superimposition unit 207, and the decoded image memory 208 are the inverse quantization/inverse orthogonal transform unit 109, the decoded image signal superimposition unit 110, and the It has a function corresponding to each configuration of the encoded information storage memory 111 and the decoded image memory 104 .

The bitstream supplied to the bitstream decoding unit 201 is separated according to a prescribed syntax rule. The separated first encoded bit stream is decoded to obtain sequences, pictures, slices, information in units of encoded blocks, and encoded information in units of encoded blocks. Specifically, the prediction mode PredMode and the division mode PartMode for determining inter prediction (PRED_INTER) or intra prediction (PRED_INTRA) are decoded in coding block units. The coded information is decoded according to a prescribed syntax (syntax rules for coded bit strings), and the coded information is supplied to the inter prediction section 203 or intra prediction section 204 and the coded information storage memory 205 . The separated second encoded bit string is decoded to calculate an orthogonally transformed and quantized residual signal, and the orthogonally transformed and quantized residual signal is supplied to the inverse quantization and inverse orthogonal transformation section 206 .

The inter prediction unit 203 performs inter prediction when the prediction mode PredMode of the encoding block to be processed is inter prediction (PRED_INTER). The encoded information decoded by the bit stream decoding unit 201 contains inter prediction modes and motion vectors. A motion-compensated prediction signal is generated from the decoded image signal stored in the decoded image memory 208 according to the motion vector. The generated motion-compensated prediction signal is supplied to the decoded image signal superimposing unit 207 .

The intra prediction unit 204 performs intra prediction when the prediction mode PredMode of the encoding block to be processed is intra prediction (PRED_INTRA). The encoded information decoded by the bit string decoding unit 201 includes an intra prediction mode, and a predicted image signal is obtained by intra prediction from the decoded image signal stored in the decoded image memory 208 according to the intra prediction mode. is generated, and the predicted image signal is supplied to the decoded image signal superimposing unit 207 . Since the intra prediction unit 204 corresponds to the intra prediction unit 103 of the image encoding device 100, the same processing as the intra prediction unit 103 is performed.

The inverse quantization/inverse orthogonal transformation unit 206 performs inverse orthogonal transformation and inverse quantization on the orthogonally transformed/quantized residual signal decoded by the bit stream decoding unit 201, and performs inverse orthogonal transformation/inverse quantization. to obtain the residual signal.

The decoded image signal superimposing unit 207 performs inverse orthogonal transform/transform/ The decoded image signal is decoded by superimposing it with the inverse quantized residual signal and stored in the decoded image memory 208 . When storing the decoded image in the decoded image memory 208, the decoded image may be stored in the decoded image memory 208 after being subjected to a filtering process for reducing block distortion due to encoding. Details will be described later.

<Block division>
The operation of the block dividing section 101 in the image encoding device 100 will be described. FIG. 3 is a flowchart illustrating the operation of dividing an image into treeblocks and subdividing each treeblock. First, an input image is divided into tree blocks of a predetermined size (step S1001). Each treeblock is scanned in a predetermined order, that is, in raster scan order (step S1002), and the inside of the treeblock to be processed is divided (step S1003).

FIG. 7 is a flow chart showing the detailed operation of the division processing in step S1003. First, it is determined whether or not to divide the block to be processed into four (step S1101).

If it is determined that the block to be processed is to be divided into four, the block to be processed is divided into four (step S1102). Each block obtained by dividing the block to be processed is scanned in the order of Z scanning, that is, upper left, upper right, lower left, and lower right (step S1103). FIG. 5 shows an example of the Z scan order, and 601 in FIG. 6 shows an example in which the block to be processed is divided into four. Numbers 0 to 3 in 601 in FIG. 6 indicate the order of processing. Then, the flowchart of FIG. 7 is recursively called for each block divided in step S1101.

If it is determined that the block to be processed is not divided into 4, it is divided into 2-3 (step S1105).

FIG. 8 is a flow chart showing the detailed operation of the 2-3 division processing in step S1105. First, it is determined whether or not to divide the block to be processed into 2-3, that is, whether to divide into 2 or 3 (step S1201).

If it is determined not to divide the block to be processed into 2-3, that is, if it is determined not to be divided, the division is terminated (step S1211), and the block in the upper layer is returned to.

If it is determined to divide the block to be processed into 2 or 3, it is determined whether to further divide the block to be processed into two (step S1202).

If the block to be processed is determined to be divided into two, it is determined whether to divide the block to be processed vertically (step S1203), and based on the result, the block to be processed is vertically divided (step S1204). Alternatively, the block to be processed is horizontally divided (step S1205). As a result of step S1204, the block to be processed is divided into two vertically as indicated by 602 in FIG. 6, and as a result of step S1205, the block to be processed is divided into two horizontally as indicated by 604 in FIG. be done.

If it is determined in step S1202 that the block to be processed is not to be divided into two, that is, if it is determined to be divided into three, it is determined whether or not to divide the block to be processed in the vertical direction (step S1206). Based on this, the block to be processed is divided vertically (step S1207) or horizontally (step S1208). As a result of step S1207, the block to be processed is vertically divided into three parts as indicated by 603 in FIG. be done.

After executing any one of steps S1204 to S1205, each block obtained by dividing the block to be processed is scanned from left to right and from top to bottom (step S1209). Numbers 0 to 3 in 602 to 605 in FIG. 6 indicate the order of processing. The flowchart of FIG. 8 is recursively called for each divided block.

In the recursive block division described here, necessity of division may be restricted by the number of divisions, the size of the block to be processed, or the like. Information that limits whether or not division is necessary may be implemented by a configuration in which information is not transmitted by prior agreement between the encoding device and the decoding device, or the coding device may limit whether or not division is necessary. The information may be determined and recorded in the encoded bit string to be transmitted to the decoding device.

Here, when a certain block is divided, the block before division is called a parent block, and each block after division is called a child block.

Next, the operation of the block dividing section 202 in the image decoding device 200 will be described. The block dividing unit 202 divides the treeblocks in the same processing procedure as the block dividing unit 101 of the image encoding device 100 . However, the block division unit 101 of the image encoding device 100 applies optimization techniques such as estimation of the optimum shape by image recognition and distortion rate optimization to determine the optimum block division shape, whereas the image decoding device The block division unit 202 in 200 is different in that the block division shape is determined by decoding the block division information recorded in the encoded bit stream.

<Motion Compensation Prediction Processing>
The motion compensation prediction unit 306 acquires the position and size of the block currently being subjected to prediction processing in encoding. Also, the motion compensation prediction unit 306 acquires inter prediction information from the inter prediction mode determination unit 305 . A reference index and a motion vector are derived from the acquired inter prediction information, and the reference picture specified by the reference index in the decoded image memory is moved from the same position as the image signal of the prediction block by the amount of the motion vector. Generate a predicted signal after acquiring the signal.

When the inter prediction mode in inter prediction is prediction from a single reference picture such as L0 prediction or L1 prediction, the prediction signal obtained from one reference picture is the motion compensation prediction signal, and the inter prediction mode is BI When the prediction mode is prediction from two reference pictures, such as prediction, the weighted average of the prediction signals obtained from the two reference pictures is used as the motion-compensated prediction signal, and the motion-compensated prediction signal is used to determine the prediction method. Department. Although the ratio of weighted averaging of bi-prediction is set to 1:1 here, weighted averaging may be performed using other ratios. For example, the closer the picture interval between the picture to be predicted and the reference picture is, the higher the weighting ratio may be. Alternatively, the weighting ratio may be calculated using a correspondence table of combinations of picture intervals and weighting ratios.

The motion compensation prediction unit 406 has the same function as the motion compensation prediction unit 306 on the encoding side. The motion compensation prediction unit 406 passes the inter prediction information from the normal prediction motion vector mode derivation unit 401, the normal merge mode derivation unit 402, the sub-block prediction motion vector mode derivation unit 403, and the sub-block merge mode derivation unit 404 to switch 408. to get through.

The motion compensation prediction unit 406 supplies the obtained motion compensation prediction signal to the decoded image signal superimposing unit 207 .

<About inter-prediction mode>
The process of performing prediction from a single reference picture is defined as uniprediction, and in the case of uniprediction, either one of the two reference pictures registered in the reference lists L0 and L1, called L0 prediction or L1 prediction, is used. make predictions. L0 prediction and L1 prediction may be forward prediction (prediction referring to a forward reference image) or backward prediction (prediction referring to a backward reference image).

FIG. 10 shows a case where the inter prediction mode is L0 prediction and the L0 reference picture (RefL0Pic) is at a time before the processing target picture (CurPic). FIG. 10 shows a case in which L0 prediction is performed and the reference picture of L0 is at a time later than the picture to be processed. Similarly, uni-prediction can be performed by replacing the reference picture for L0 prediction in FIG. 10 with the reference picture for L1 prediction (RefL1Pic).

A process of performing prediction from two reference pictures is defined as bi-prediction, and bi-prediction is expressed as bi-prediction using both L0 prediction and L1 prediction. 11 and 12 are diagrams for explaining motion compensation prediction in bi-prediction. FIG. 11 shows a bi-predictive case where the reference picture for L0 prediction is before the picture to be processed and the reference picture for L1 prediction is after the picture to be processed. FIG. 12 shows a bi-prediction case in which the L0-prediction reference picture and the L1-prediction reference picture are located before the picture to be processed.

In this way, the relationship between the prediction type and time of L0/L1 is limited to forward prediction (prediction referring to a forward reference image) for L0 and backward prediction (prediction referring to a backward reference image) for L1. It is possible to use it without In the case of bi-prediction, L0 prediction and L1 prediction may be performed using the same reference picture. It should be noted that the determination of whether motion compensation prediction is performed by uni-prediction or bi-prediction is made based on information (for example, a flag) indicating whether to use L0 prediction and whether to use L1 prediction, for example. be.

<Filter processing>
Filter processing will be described. Filtering can be performed in the decoded image signal superimposing unit 110 and the decoded image signal superimposing unit 207 in order to reduce coding errors. In filtering, deblock filtering, sample adaptive offset, and adaptive loop filtering are performed in order.

FIG. 13 is a diagram showing the effect of each filtering process on pixel values. First, the case of the deblocking filter will be described with reference to FIG. 13(a). It is now assumed that the deblock filtering processes vertical block boundaries. The horizontal block boundary is the same as the vertical processing rotated 90 degrees, so the description is omitted. 4001 indicates a block boundary, p0 to p7 and q0 to q7 indicate pixels respectively. The deblocking filter has different strengths depending on the state of distortion at the block boundaries. In order to consider the effect on pixel values, the strongest filter shall be applied. In this case, 7 pixels to the left and right of the block boundary are changed. Let p_i and q_j be pixel values before filtering, and p_i' and q_j' be pixel values after filtering, respectively, for distances i and j (both from 0 to 6) from the block boundary.
p_i' = (f_i * M + (64 - f_i) * P + 32) >> 6
q_j' = (g_j * M + (64 - g_j) * Q + 32) >> 6
f_i = 59 - i * 9
g_j = 59 - j * 9
M = ( 2 * (p0 + q0) + (p1 + q1) + (p2 + q2) + (p3 + q3)
+ (p4 + q4) + (p5 + q5) + (p6 + q6) + 8 ) >> 4
P = (p6 + p7 + 1) >> 1
Q = (q6 + q7 + 1) >> 1
becomes. That is, the pixels p0 to p7 and q0 to q7 are referred to, and the pixel values p0 to p6 and q0 to q6 are changed.

Next, the case of sample adaptive offset will be described with reference to FIG. 13(b). p indicates a pixel to be processed. Also, a0 to a3 and b0 to b3 indicate pixels located in a 3×3 pixel range around p. When filtering edges, class c can be selected from 0 to 3 for filtering. Here, if the pixel value before filtering is p, the pixel value after filtering is p', and the surrounding pixels are a_c and b_c,
edgeIdx_c = 2 + Sign( p - a_c ) + Sign( p - b_c )
p' = Clip3( 0, ( 1 << bitDepth ) - 1, p + SaoOffsetVal[edgeIdx_c] )
becomes. Here, Clip3(x,y,z) is a function that limits the minimum value to x and the maximum value to y for the value z. Also, bitDepth is a bit depth, and SaoOffsetVal is a value encoded according to a prescribed syntax. That is, one pixel (a0 to a3 and b0 to b3) surrounding the pixel to be filtered is referred to, and the pixel p to be filtered is changed.

Next, the case of the adaptive loop filter will be described with reference to FIG. 13(c). p indicates a pixel to be processed. d0 to d11 and e0 to e11 indicate pixels located in a 7×7 pixel range around p. Here, let the position of the pixel value p before filtering be (hx, vy) and denote it as p[hx, vy]. Also, if the pixel value after filtering is p',
sum = f0 * ( Clip3( -c0, c0, p[ hx, vy + r3 ] - p ) +
Clip3( -c0, c0, p[ hx, vy - r3 ] - p ) )
+ f1 * ( Clip3( -c1, c1, p[ hx + 1, vy + r2 ] - p ) +
Clip3( -c1, c1, p[ hx - 1, vy - r2 ] - p ) )
+ f2 * ( Clip3( -c2, c2, p[ hx, vy + r2 ] - p ) +
Clip3( -c2, c2, p[ hx, vy - r2 ] - p ) )
+ f3 * ( Clip3( -c3, c3, p[ hx - 1, vy + r2 ] - p ) +
Clip3( -c3, c3, p[ hx + 1, vy - r2 ] - p ) )
+ f4 * ( Clip3( -c4, c4, p[ hx + 2, vy + r1 ] - p ) +
Clip3( -c4, c4, p[ hx - 2, vy - r1 ] - p ) )
+ f5 * ( Clip3( -c5, c5, p[ hx + 1, vy + r1 ] - p ) +
Clip3( -c5, c5, p[ hx - 1, vy - r1 ] - p ) )
+ f6 * ( Clip3( -c6, c6, p[ hx, vy + r1 ] - p ) +
Clip3( -c6, c6, p[ hx, vy - r1 ] - p ) ) +
+ f7 * ( Clip3( -c7, c7, p[ hx - 1, vy + r1 ] - p ) +
Clip3( -c7, c7, p[ hx + 1, vy - r1 ] - p ) )
+ f8 * ( Clip3( -c8, c8, p[ hx - 2, vy + r1 ] - p ) +
Clip3( -c8, c8, p[ hx + 2, vy - r1 ] - p ) )
+ f9 * ( Clip3( -c9, c9, p[ hx + 3, vy ] - p ) +
Clip3( -c9, c9, p[ hx - 3, vy ] - p ) )
+ f10 * ( Clip3( -c10, c10, p[ hx + 2, vy ] - p ) +
Clip3( -c10, c10, p[ hx - 2, vy ] - p ) )
+ f11 * ( Clip3( -c11, c11, p[ hx + 1, vy ] - p ) +
Clip3( -c11, c11, p[ hx - 1, vy ] - p ) )
sum = p + ( ( sum + 64 ) >> 7 )
p' = Clip3( 0, ( 1 << bitDepth ) - 1, sum )
becomes. Here, f_i, c_i (i is 0 to 11) are values encoded according to the prescribed syntax. Also, r1, r2, and r3 are values that vary from 0 to 3 depending on the position of the pixel to be processed. In other words, when performing filtering, pixels from p and d0 to d11 and e0 to e11 are referenced, and the pixel value of p is changed.

FIG. 13(d) is a diagram for explaining the overall effect of each filtering process on the pixel value. Consider the effect on the reference pixels on the right side of the boundary and the changed pixels on the left side of the boundary with respect to the block boundary to be deblock filtered. First, in the deblocking filtering process, 8 pixels (4002) on the right side of the block boundary 4001 are referred to, and 7 pixels (4003) on the left side of the block boundary 4001 are changed. Next, in the sample adaptive offset process, pixel p is changed by referring to pixel p6 changed by the deblocking filter process as b0. The adaptive loop filtering then modifies pixel p, referring to pixel p modified by the sample adaptive offset process as e9. As a result, when a block boundary is filtered, 8 pixels on the right side of the block boundary are referred to, and the pixel values of up to 11 pixels on the left side of the block boundary are changed.

<GDR>
In order to reduce the delay of encoding/decoding and to enable random access (decoding from the middle of the bitstream), some areas in the picture are refreshed by intra-encoding, and refreshed in stages while shifting the area. I have something to do. Such a mechanism is called GDR (GRADUAL DECODING REFRESH), and the encoding for it is called GDR encoding. In GDR coding, intra or inter coding is performed on areas other than some areas in the picture to be refreshed. Here, intra-encoding is encoding by the intra-prediction unit 103 by intra-prediction. Intra-coding of refreshed regions involves coding without reference to non-refreshed regions. Also, inter-encoding is encoding by inter-prediction performed by the inter-prediction unit 102 . In inter-coding, encoding is performed using forward prediction. In GDR encoding, the random access start point and recovery point information (the number of POCs until refresh is completed) are encoded according to a prescribed syntax. A recovery point is a point at which decoding is started from a random access start point, refresh is completed, and correct decoding is possible. When randomly accessing and decoding a GDR-encoded bitstream, the decoding process is performed from the random access start point to the recovery point, but the decoded image is not output. On the other hand, when decoding is performed without random access, decoding processing is performed and the decoded image is output. A GDR encoded bitstream can be decoded with or without random access.

FIG. 14 is a diagram showing how intra refresh is performed in GDR encoding. FIG. 14 shows three pictures in POC order from picture P1, which is the starting point of random access. Here, the picture P1 is a picture other than the first picture to be encoded.

In picture P1, 4102 indicates an area to be refreshed by intra-coding, and 4103 indicates an area that has not yet been refreshed. Picture P1 is refreshed by intra-encoding a region 4102 of one CTU column from the left end of the picture. Here, one CTU column indicates one column of treeblocks.

In picture P2, 4201 indicates the refreshed area, 4202 indicates the area to be refreshed by intra-coding, and 4203 indicates the area which has not yet been refreshed. Picture P2 is refreshed by intra-encoding a region 4202 to the right of region 4201 by one CTU column.

In picture P3, 4301 indicates the refreshed area, 4302 indicates the area to be refreshed by intra-coding, and 4303 indicates the area which has not yet been refreshed. Picture P3 is refreshed by intra-encoding a region 4302 to the right of region 4301 by one CTU column.

The picture P3 and subsequent pictures are encoded in the same manner, and when the right end of the picture is refreshed, the GDR encoding is completed. GDR encoding may be performed again after the picture for which GDR encoding has been completed. Here, the case where coding is performed while shifting the area to be refreshed to the right by one CTU column from the left end of the picture has been shown, but the refreshing method is not limited to this. For example, the right end of the picture may be shifted left by 1 CTU column, the top end of the picture may be shifted downward by 1 CTU row, or the bottom end of the picture may be shifted upward by 1 CTU row. good. Furthermore, the refresh unit may be any shape such as diagonal instead of vertical or horizontal such as 1 CTU column or 1 CTU row, or may be any unit such as one or a plurality of coding tree blocks or coding blocks. Additionally, there may be pictures that do not refresh. The method of refreshing can be determined in advance by the encoding device and the decoding device, or the encoding device can determine the refreshing method and transmit it to the decoding device.

In GDR coding, every region of the picture is intra-coded and refreshed at least once before the recovery point. Also, in inter-coding in a once refreshed area, reference is made to the refreshed area. If such a GDR-encoded bitstream is decoded, it can be correctly decoded after the recovery point. However, in inter-encoding in a once-refreshed area, if a non-refreshed area is referenced, decoding cannot be performed correctly. In addition, inter-coding in a once-refreshed region cannot be decoded correctly when referencing the refreshed region that has been affected by filtering.

FIG. 15 is a diagram showing how random access prevents correct decoding. It will be explained that, in inter-encoding in an area that has been refreshed once, decoding cannot be performed correctly if an area that has not been refreshed is referred to. The bitstream is now GDR-encoded from picture P1, and each picture is inter-encoded with reference to the immediately preceding picture. It is also assumed that this bitstream is not filtered.

Here, consider the case of decoding without performing random access to this bitstream. First, the picture Q1 is decoded and stored in the decoded picture memory 208 . After that, the picture Q2 is decoded and stored in the decoded picture memory 208. FIG. Next, picture P1 is decoded. Now, the coding block 4402 to be processed is inter-coded with reference to the reference block 4401 of picture Q2. The inter prediction unit 203 generates a predicted image signal by motion compensation from the image signal of the decoded picture Q2 stored in the decoded image memory 208 according to the motion vector. The decoded image signal is decoded by superimposing the prediction image signal and the residual signal inversely orthogonally transformed and inversely quantized by the inverse quantization and inverse orthogonal transformation unit 206 . Next, picture P2 is decoded. Now, the encoding block 4404 to be processed is inter-encoded with reference to the reference block 4403 of the picture P1. The inter prediction unit 203 generates a prediction image signal by motion compensation from the image signal of the decoded picture P1 stored in the decoded image memory 208 according to the motion vector. The decoded image signal is decoded by superimposing the prediction image signal and the residual signal inversely orthogonally transformed and inversely quantized by the inverse quantization and inverse orthogonal transformation unit 206 . As described above, this bitstream can be correctly decoded when it is decoded without random access.

On the other hand, consider the case of randomly accessing and decoding this bitstream. First, the picture Q1 is decoded and stored in the decoded picture memory 208 . Assume that picture P1 is randomly accessed. Region 4102 is intra-coded and decodes correctly. On the other hand, the coded block 4402 to be processed is inter-coded with reference to the reference block 4401 of picture Q2. However, since picture Q2 has not been decoded, it is not stored in the decoded picture memory 208 . Therefore, the inter prediction unit 203 cannot correctly generate the predicted image signal from the decoded image memory 208. However, the coded block 4402 to be processed is in an area 4103 that has not yet been refreshed. This area 4103 is intra-coded and refreshed in pictures after picture P1. Therefore, the encoding block 4402 to be processed does not have to be correctly decoded at the time of the picture P1. Since it is not always possible to correctly decode an area that has not been refreshed yet, even if the decoded image signal is stored in the decoded image memory 208, the decoded image signal is not correct. At this time, the decoded image signal of the area that has not yet been refreshed may be stored in the decoded image memory 208 so that it can be determined as being incorrect. Alternatively, the decoded image signal may not be stored in the decoded image memory 208 . Alternatively, regions that have not yet been refreshed may not be decoded.

Next, picture P2 is decoded. Now, the encoding block 4404 to be processed in the area 4201 is inter-encoded with reference to the reference block 4403 of the picture P1. In other words, in a once refreshed area, inter-coding is performed with reference to a non-refreshed area. Here, the picture P1 is decoded and stored in the decoded image memory 208 . However, since the encoded block 4402 is not correctly decoded in the picture P1, the reference image of the reference block 4403 is incorrect. In spite of the refreshed area 4201, some portions cannot be decrypted correctly, and decryption cannot be performed correctly even after the recovery point. To prevent this, inter-coding of once refreshed regions needs to refer to refreshed regions. This allows correct decryption after the recovery point.

Here, a reference area that can be correctly decoded after the recovery point is called a clean area. A reference area that cannot be correctly decoded after the recovery point is called a dirty area. Without filtering in GDR encoding, clean regions are the same as refreshed regions and dirty regions are the same as non-refreshed regions. However, this does not apply if the refreshed area is affected by the filtering process. The effect of filtering is to make the clean area narrower than the refreshed area.

FIG. 16 is a diagram showing how random access prevents correct decoding. It will be explained that, in inter-coding in a region that has been refreshed once, decoding cannot be performed correctly if the refreshed region that has been affected by filtering is referred to. The bitstream is now GDR-encoded from picture P1, and each picture is inter-encoded with reference to the immediately preceding picture. It is also assumed that this bitstream is filtered.

Now, the coding block 4211 to be processed in the region 4201 of the picture P2 is inter-coded with reference to the reference block 4111 of the picture P1. A region 4105 indicates a range affected by filtering the boundary 4104 . As described above, when the deblocking filter, the sample adaptive offset, and the adaptive loop filter are sequentially applied, the 8 pixels on the right side of the block boundary are referred to, and the pixel values of up to 11 pixels on the left side of the block boundary are changed. be. Region 4105 cannot be decoded correctly because boundary 4104 is filtered with reference to region 4103 which has not yet been refreshed. And since reference block 4111 contains region 4105, the reference image is incorrect. A clean area for correct decoding is a portion of area 4102 that does not include area 4105 .

Next, the decoded image signal superimposing unit 110 in the image encoding device 100 of this embodiment and the decoded image signal superimposing unit 207 in the image decoding device 200 will be described.

FIG. 17 is a diagram showing the details of the decoded image signal superimposition unit 110 in the image coding apparatus of FIG. In the present embodiment, the decoded image signal superimposing unit 110 generates a predicted image signal determined by the prediction method determining unit 105 and a residual signal inversely quantized and inverse orthogonally transformed by the inverse quantization/inverse orthogonal transformation unit 109. is supplied to the superimposition unit 112 . The superimposition unit 112 superimposes the predicted image signal and the residual signal to generate a decoded image, and supplies the decoded image to the filter unit 113 . The filter unit 113 performs filter processing on the decoded image under the control of the refresh control unit 114 in order to reduce coding errors. The control of refresh control unit 114 will be described later. The result of filtering is stored in the decoded image memory 104 .

FIG. 18 is a diagram showing details of the decoded image signal superimposing unit 207 in the image decoding device of FIG. In this embodiment, the decoded image signal superimposing unit 207 combines the predicted image signal inter-predicted by the inter prediction unit 203 or the predicted image signal intra-predicted by the intra prediction unit 204 with the inverse quantization/inverse orthogonal transform unit 206. A residual signal inversely orthogonally transformed and inversely quantized by is supplied to the superimposing unit 212 . The superimposition unit 212 superimposes the prediction image signal and the residual signal to generate a decoded image, and supplies the decoded image to the filter unit 213 . The filter unit 213 filters the decoded image under the control of the refresh control unit 214 in order to reduce coding errors. The control of refresh control unit 214 will be described later. The result of filtering is stored in the decoded image memory 208 .

Next, operations of the refresh control unit 114 and the refresh control unit 214 will be described. FIG. 19 is a diagram for explaining the processing of refresh control unit 114 and refresh control unit 214. As shown in FIG.

First, it is determined whether or not the filtering process is performed on the refresh boundary (S6001). A refresh boundary is the boundary between an area that has been refreshed and an area that has not yet been refreshed. Also, the refreshed area is a combined area of the area refreshed in the past picture and the area refreshed in the current picture. For example, in the case of picture P2 in FIG. 16, areas 4201 and 4202 have been refreshed, and area 4203 has not yet been refreshed. Also, the refresh boundary is in contact with the boundary 4204 and corresponds to a portion of the area 4202 including the area 4205 .

If the encoding device and the decoding device are refreshed by a predetermined method, whether or not it is a refresh boundary is determined according to the method. For example, it is determined in advance that the refresh is started from the left end of the picture and shifted to the right by one CTU column for each picture. Then, the encoding device and the decoding device can determine whether or not it is a refresh boundary based on a predetermined method. Alternatively, the encoding device may determine the refresh method and transmit it to the decoding device. Then, the encoding device determines whether or not it is a refresh boundary based on the self-determined refresh position, and transmits the refresh position. Also, the decoding device determines whether or not it is a refresh boundary based on the transmitted and decoded refresh position.

If it is not a refresh boundary (S6001: NO), it is determined whether it is a refreshed area (S6002). If the area has already been refreshed (S6002: YES), the decoded result or the filtered result is stored in the decoded image memory and set as a clean area (S6003). Here, if the decoded image output from the superimposition unit 112 or the superimposition unit 212 is not subjected to filtering, it is output from the filter unit 113 or the filter unit 213 as the decoding result. When filtering is performed, it is output from the filter unit 113 or the filter unit 213 as a filter result. The decoded result or filter result is stored in the decoded image memory 104 or decoded image memory 208 . If the area has not been refreshed (S6002: NO), the decoded result or the filtered result is stored in the decoded image memory as a dirty area (S6004). In this process, decoding and filtering are the same as in S6003.

If it is a refresh boundary (S6001: YES), it is determined whether or not to perform filtering (S6005). The encoding device and the decoding device may decide in advance whether or not to perform the filtering process, or the encoding device determines whether to perform the filtering process and transmits the result to the decoding device. It is good as If filtering is not to be performed (S6005: NO), the decoded image output from the superimposing unit 112 or superimposing unit 212 is stored in the decoded image memory 104 or decoded image memory 208 without being filtered, and is used as a clean area (S6008). .

If filter processing is to be performed (S6005: YES), from the decoded image output from the superimposing unit 112 or superimposing unit 212, pixels included in the area to be refreshed and in contact with the boundary are supplied to the filter unit (S6006). The decoded image output from the superimposing unit 112 or superimposing unit 212 and the pixels supplied from the refresh control unit 114 or 214 are referred to, and filtering is performed by the filter unit 113 or 213 . The filtered result is stored in the decoded image memory 104 or decoded image memory 208 as a clean area (S6007).

The above operation will be described using the filtering process for the picture P2 in FIG. 16 as an example. Assume that the coding device determines the refresh position and transmits it to the decoding device. Also, consider a case where filtering is not performed at refresh boundaries but is performed at areas other than refresh boundaries.

The refresh control unit 114 determines whether or not it is a refresh boundary based on the refresh position determined by the encoding device 100 (S6001). The refresh control unit 214 determines whether or not it is a refresh boundary based on the refresh position transmitted to and decoded by the decoding device 200 (S6001). An area 4201 is not a refresh boundary (S6001: NO), but a refreshed area (S6002: YES). Therefore, the filter result is stored in the decoded image memory 104 or the decoded image memory 208 as a clean area (S6003). A portion of the area 4202 that does not include the area 4205 is processed in the same manner as the area 4201 .

A portion of the area 4202 that includes the area 4205 is a refresh boundary (S6001: YES). Since filtering is not performed at this time (S6005: NO), the decoded result is stored in the decoded image memory 104 or decoded image memory 208 as a clean area (S6008).

Since the area 4203 is neither a refresh boundary (S6001: NO) nor a refreshed area (S6002: NO), the filter result is stored in the decoded image memory 104 or the decoded image memory 208 as a dirty area (S6004). The processing of the refresh control unit 114 and the refresh control unit 214 is thus completed.

As described above, the inter prediction unit 102 or the inter prediction unit 203 can perform inter prediction with reference to the clean region when the filter processing is not performed on the refresh boundary. Therefore, if the bitstream generated by this encoding device is randomly accessed and decoded, it can be correctly decoded after the recovery point. In addition, since filter processing can be performed on areas other than refresh boundaries, coding distortion can be reduced and coding efficiency can be improved.

On the other hand, consider the case of filtering at refresh boundaries. First, it is determined to be a refresh boundary (S6001: YES), and filtering is performed (S6005: YES). The pixels included in the region 4202 to be refreshed and in contact with the boundary 4204 are supplied to the filter unit 113 or the filter unit 213 (S6006). Then, the filtered result is stored in the decoded image memory 104 or the decoded image memory 208 as a clean area (S6007).

As described above, the inter prediction unit 102 or the inter prediction unit 203 can perform inter prediction with reference to the clean region when filtering is performed on the refresh boundary. Therefore, if the bitstream generated by this encoding device is randomly accessed and decoded, it can be correctly decoded after the recovery point. In addition, since filter processing can be performed including refresh boundaries, coding distortion can be reduced and coding efficiency can be improved.

If it is not known where the refresh boundary is located, it is not possible to change the presence or absence of filtering between the refresh boundary and the rest. Also, if you do not know where the refresh boundary is, filtering may refer to areas that have not yet been refreshed. Then, the reference image after the filtering process includes a dirty area, and if it is randomly accessed and decoded, it cannot be correctly decoded after the recovery point. Therefore, if it is not known at which position the refresh boundary is, filtering cannot be performed at any position regardless of whether it is a refresh boundary or not.

Now, the configuration is such that the encoding device determines the refresh position and transmits it to the decoding device. In addition, the encoding device and the decoding device preliminarily decide whether or not to perform filter processing on refresh boundaries and other areas. Then, the encoding device and the decoding device can determine whether or not to perform filter processing based on a predetermined method on refresh boundaries and other areas. Alternatively, the encoding device may determine whether or not to perform filtering and transmit the result to the decoding device. Then, the encoding device determines whether or not filtering is to be performed on the refresh boundary and other areas, performs filtering, and transmits whether or not filtering is to be performed. Also, the decoding device performs filtering on refresh boundaries and other areas according to the presence/absence of filtering that has been transmitted and decoded. Therefore, the encoder can choose whether or not to filter at refresh boundaries and otherwise.

FIG. 20 is a diagram showing pixels to be referenced and pixels to be changed in filtering. This shows the filtering of boundary 4104 in picture P1 of FIG. Now assume that the strongest filter is applied to consider the effect on pixel values. Filter unit 113 or filter unit 213 refers to pixels in ranges 4011 and 4021 and changes pixels in ranges 4023 and 4024 . Here, the refresh control unit 114 or the refresh control unit 214 determines whether or not to perform filter processing (S6005) because the filter processing is performed on the refresh boundary (S6001: YES). Since filter processing is now being performed (S6005: YES), the pixel p0 included in the refreshed area 4102 and in contact with the boundary 4104 is supplied to the filter unit 113 or the filter unit 213 (S6006). Filter unit 113 or filter unit 213 refers to pixels p0 included in refreshed area 4102 as pixels in range 4021 . The pixels referenced by the filter unit 113 or the filter unit 213 are all pixels included in the refreshed region 4102, and the region 4103 that has not yet been refreshed is not referenced. The filter unit 113 or the filter unit 213 performs filter processing with reference to the replaced pixels (S6007). This pixel replacement causes region 4102 to be filtered without reference to region 4103, which has not yet been refreshed. At this time, the clean area that can be decoded correctly is the entire area 4102 . Then, the filtered result is stored in the decoded image memory 104 or the decoded image memory 208 as a clean area (S6007). Then, in the picture P2 in FIG. 16, the reference block 4111 referred to by the target encoding block 4211 is included in the clean area. Therefore, even if random access is used for decoding, correct decoding is possible.

This pixel replacement is applied in the filtering across the refresh boundaries. On the other hand, pixel replacement is not applied to the filtering of regions that have not yet been refreshed. Also, instead of replacing the adjacent pixels as they are, they may be replaced with fixed values such as predicted values from adjacent pixels or gray images. Furthermore, since the range affected by the strength of the filter differs, the range of pixel replacement may be variable according to the filter strength. In addition, the size of the target block and the prediction mode may be changed in order to change the filter strength.

As described above, even if filter processing is performed at the refresh boundary, the reference image can be set as a clean area. As a result, the clean area that can be referred to by the inter prediction unit is widened when filtering is performed, so that encoding efficiency can be improved by selecting a more optimal reference image. Also, in a clean area close to the dirty area, a reference block spatially at the same position as the target encoding block, that is, the position of the motion vector (0, 0) can be referenced. This can improve coding efficiency, especially for images with little motion.

Further, pixel replacement in the refresh control section 114 and the refresh control section 214 can also be implemented as padding. FIG. 21 is a diagram showing how padding is performed. FIG. 21(a) shows padding in picture P1 in FIG. Pixels 4121 included in the area to be refreshed 4102 and touching the boundary 4104 are padded to the area 4103 that has not yet been refreshed. This is similar to what happens when a picture is referenced off-screen. The only difference is the position determined to be outside the screen, and no new processing occurs, so the amount of additional calculations can be reduced.

Padding for non-rectangular clean areas is shown in FIGS. 21(b) to (e). FIGS. 21B to 21E show the case where a plurality of areas 4102 to be refreshed are positioned around an area 4103 which has not yet been refreshed. As shown in FIG. 21(b), the pixels 4123 included in the region 4102 to be refreshed and in contact with the boundary 4122 are padded to the region 4103 that has not yet been refreshed. Alternatively, as shown in FIG. 21(c), pixels 4124 included in the area to be refreshed 4102 and in contact with the boundary 4122 may be padded in the area 4103 which has not yet been refreshed. Alternatively, as shown in FIG. 21(d), the pixels 4125 included in the area 4102 to be refreshed and in contact with the boundary 4122 may be padded in the area 4103 that has not yet been refreshed. In addition, as shown in FIG. 21(e), padding may be performed with an average value of pixels 4123, 4124, and 4125 included in the area 4102 to be refreshed and in contact with the boundary 4122. FIG. Further, instead of padding adjacent pixels, a predicted value from adjacent pixels or a fixed value such as a gray image may be used for padding.

This padding is applied in the filtering across the refresh boundary and is processed by the pixels referenced by the filtering. On the other hand, no padding is applied to filtering areas that have not yet been refreshed. Furthermore, since the range affected by the filter strength differs, the padding range may be variable according to the filter strength. In addition, the size of the target block and the prediction mode may be changed in order to change the filter strength. Since this padding is a different implementation of the pixel replacement described above, the effect is the same as in the case of pixel replacement.

Inter prediction section 102 in encoding apparatus 100 and inter prediction section 203 in decoding apparatus 200 of the present invention operate as follows. That is, the inter prediction section 102 and the inter prediction section 203 perform inter prediction with reference to the clean region in the refreshed region. In areas that have not yet been refreshed, inter-prediction is performed with reference to clean and dirty areas. In regions that have not yet been refreshed, no pixel replacement or padded pixels are used.

Consider the case where pixel replacement or padding is applied not only to the pixels to which the filtering process refers, but also to the entire region that has not yet been refreshed. Then, the following effects are obtained.

Due to a malfunction or the like, there is a possibility that a bitstream is generated by encoding with reference to a dirty area in a refreshed area. Alternatively, part of the bitstream may be missing due to packet loss or the like, and the decoded result may refer to such a dirty area. A bitstream that refers to such a dirty area cannot be correctly decoded after the recovery point. Also, in a decoding device that is not the present invention, pixel values are obtained from a decoded image memory that stores an incorrect decoding result as a reference block that refers to a dirty area. As a result, it is conceivable that the decoding result differs depending on the decoding device. On the other hand, if pixel replacement or padding is applied to an area that has not yet been refreshed, the pixel values of the dirty area will be the replaced or padded values of the pixel values of the clean area, or fixed values. Therefore, even if a bitstream that refers to such a dirty area is decoded, the pixel values of the reference blocks that refer to the dirty area are constant. As a result, the decoding result is the same depending on the decoding device, so that robustness can be improved.

Therefore, pixel replacement and padding may be performed selectively for the pixels referred to in filtering on the refresh boundary, or for the entire area that has not yet been refreshed. The pixel replacement and padding methods are determined in advance by the encoding device and the decoding device, or the encoding device determines the pixel replacement and padding methods and transmits them to the decoding device. Prioritizing coding efficiency, it is possible to perform pixel replacement and padding for pixels referred to in filter processing on the refresh boundary. Alternatively, in favor of robustness, pixels can be replaced or padded across regions that have not yet been refreshed.

In GDR encoding, intra refresh may be performed using intra block copy. However, intra block copy can refer only to the same CTU row as the target encoding tree block and some encoding blocks or encoding tree blocks that precede the target encoding tree block in encoding order. Therefore, in GDR encoding, in order to be able to refer to the clean area as a reference block, refresh may always be performed from left to right. Alternatively, it may be controlled not to refer to the dirty area as a reference block for intra block copy.

In all the embodiments described above, the encoded bitstream output by the image encoding device has a specific data format so that it can be decoded according to the encoding method used in the embodiments. is doing. The encoded bitstream may be provided by being recorded in a computer-readable recording medium such as an HDD, SSD, flash memory, or optical disk, or may be provided from a server through a wired or wireless network. Therefore, an image decoding device corresponding to this image encoding device can decode the encoded bitstream of this specific data format regardless of the providing means.

When a wired or wireless network is used to exchange an encoded bitstream between an image encoding device and an image decoding device, the encoded bitstream is converted into a data format suitable for the transmission mode of the communication channel. may be transmitted. In this case, a transmission device that converts an encoded bitstream output from an image encoding device into encoded data in a data format suitable for the transmission mode of a communication channel and transmits the encoded data to a network, and a transmission device that receives encoded data from the network. and a receiving device for restoring an encoded bitstream to supply to an image decoding device.

The transmission device includes a memory that buffers the encoded bitstream output from the image encoding device, a packet processing unit that packetizes the encoded bitstream, and a transmission unit that transmits the packetized encoded data via the network. including. The receiving device includes a receiving unit that receives packetized encoded data via a network, a memory that buffers the received encoded data, and packetizes the encoded data to generate an encoded bitstream, and a packet processing unit provided to the image decoding device.

When a wired or wireless network is used to exchange coded bitstreams between an image coding device and an image decoding device, in addition to the transmitting device and the receiving device, the coded data transmitted by the transmitting device is also used. A relay device may be provided for receiving and supplying to the receiving device. The relay device includes a receiver that receives packetized encoded data transmitted by the transmitter, a memory that buffers the received encoded data, and a transmitter that transmits the packetized encoded data and the network. include. Further, the relay device includes a reception packet processing unit that performs packet processing on packetized encoded data to generate an encoded bitstream, a recording medium that stores the encoded bitstream, and a packetization of the encoded bitstream. A transmission packet processing unit may be included.

Further, the display device may be configured by adding a display unit for displaying an image decoded by the image decoding device. In this case, the display unit reads the decoded image signal generated by the decoded image signal superimposition unit 207 and stored in the decoded image memory 208 and displays it on the screen.

Further, an imaging device may be configured by adding an imaging unit to the configuration and inputting a captured image to an image encoding device. In that case, the imaging unit inputs the image signal of the imaged image to the block dividing unit 101 .

FIG. 22 shows an example of the hardware configuration of the encoding/decoding device of the present application. The encoding/decoding device includes the configurations of the image encoding device and the image decoding device according to the embodiment of the present invention. The encoding/decoding device 9000 has a CPU 9001 , a codec IC 9002 , an I/O interface 9003 , a memory 9004 , an optical disk drive 9005 , a network interface 9006 and a video interface 9009 .

An image encoding unit 9007 and an image decoding unit 9008 are typically implemented as a codec IC 9002 . Image encoding processing of the image encoding device according to the embodiment of the present invention is executed by the image encoding unit 9007, and image decoding processing in the image decoding device according to the embodiment of the present invention is performed by the image encoding unit 9007. Executed by The I/O interface 9003 is realized by, for example, a USB interface, and connects with an external keyboard 9104, mouse 9105, and the like. The CPU 9001 controls the encoding/decoding device 9000 to execute the operation desired by the user based on the user's operation input via the I/O interface 9003 . User operations using the keyboard 9104, mouse 9105, etc. include selection of either encoding or decoding to be executed, encoding quality setting, encoded stream input/output destination, image input/output destination, and the like. .

When the user desires to reproduce the image recorded on the disc recording medium 9100, the optical disc drive 9005 reads the encoded bitstream from the inserted disc recording medium 9100 and transfers the read encoded stream to the bus 9010. to the image decoding unit 9008 of the codec IC 9002 via. The image decoding unit 9008 executes image decoding processing in the image decoding apparatus according to the embodiment of the present invention on the input coded bitstream, and sends the decoded image to the external monitor 9103 via the video interface 9009 . The encoding/decoding device 9000 also has a network interface 9006 and can be connected to an external distribution server 9106 and a mobile terminal 9107 via a network 9101 . If the user desires to reproduce the image recorded in the distribution server 9106 or portable terminal 9107 instead of the image recorded in the disk recording medium 9100, the network interface 9006 receives the image from the input disk recording medium 9100. Instead of reading the encoded bitstream, the encoded stream is obtained from the network 9101 . Further, when the user desires to reproduce the image recorded in the memory 9004, the image decoding process in the image decoding apparatus according to the embodiment of the present invention is performed on the encoded stream recorded in the memory 9004. Run.

When the user desires an operation to encode an image captured by the external camera 9102 and record it in the memory 9004, the video interface 9009 inputs the image from the camera 9102 and transmits the image to the image encoding unit 9007 of the codec IC 9002 via the bus 9010. send to The image encoding unit 9007 performs image encoding processing in the image encoding apparatus according to the embodiment of the present invention on the image input via the video interface 9009 to create an encoded bitstream. The encoded bitstream is then sent to memory 9004 via bus 9010 . When the user desires to record the encoded stream on the disk recording medium 9100 instead of the memory 9004, the optical disk drive 9005 writes the encoded stream to the inserted disk recording medium 9100. FIG.

It is also possible to realize a hardware configuration having an image encoding device but not having an image decoding device, or a hardware configuration having an image decoding device but not having an image encoding device. Such a hardware configuration is realized, for example, by replacing the codec IC 9002 with the image encoding unit 9007 or the image decoding unit 9008, respectively.

The above encoding and decoding processes may of course be implemented as transmission, storage, and reception devices using hardware, and are stored in ROM (read only memory), flash memory, or the like. It may be implemented by firmware or software such as a computer. The firmware program and software program may be recorded in a computer-readable recording medium and provided, may be provided from a server through a wired or wireless network, or may be provided from a terrestrial or satellite digital broadcasting data broadcast. can be provided as

The present invention has been described above based on the embodiments. It should be understood by those skilled in the art that the embodiments are examples, and that various modifications can be made to combinations of each component and each treatment process, and such modifications are also within the scope of the present invention. .

100 image coding device 101 block division unit 102 inter prediction unit 103 intra prediction unit 104 decoded image memory 105 prediction method determination unit 106 residual signal generation unit 107 orthogonal transformation/quantization unit 108 bit string code 110 decoded image signal superimposition unit 111 encoded information storage memory 200 image decoding device 201 bit string decoding unit 202 block division unit 203 inter prediction unit 204 intra prediction unit , 205 encoded information storage memory, 206 inverse quantization/inverse orthogonal transform unit, 207 decoded image signal superimposition unit, 208 decoded image memory.

Claims (2)

an intra prediction unit that intra-predicts and refreshes a partial region in a picture;
a filter unit for filtering the decoded image;
a refresh control unit that controls the filter unit so that in the filter processing, the filter processing is performed by referring to the pixels in the area that has been refreshed;
a decoded image memory that stores the output of the filter unit as a clean area;
an inter prediction unit that refers to the clean region in the inter prediction of the refreshed region;
An image decoding device comprising: an intra prediction unit that intra-predicts and refreshes a partial region in a picture;
a filter unit for filtering the decoded image;
In the filter processing, when there is a boundary between an area that has been refreshed and an area that has not been refreshed, a refresh control unit that refers to pixels in the area that has not been refreshed and performs control so that filtering is not performed. When,
a decoded image memory that stores the output of the filter unit in the refreshed area as a clean area;
an inter prediction unit that refers to the clean region in inter prediction of the refreshed region, or refers to pixels that are in contact with the boundary of the clean region when referring to the region that is not the clean region;
An image decoding device comprising:

Images