OA20185A - Virtual boundary processing for adaptive loop filtering. - Google Patents
Virtual boundary processing for adaptive loop filtering. Download PDFInfo
- Publication number
- OA20185A OA20185A OA1202100155 OA20185A OA 20185 A OA20185 A OA 20185A OA 1202100155 OA1202100155 OA 1202100155 OA 20185 A OA20185 A OA 20185A
- Authority
- OA
- OAPI
- Prior art keywords
- current sample
- boundary
- value
- virtual boundary
- sample
- Prior art date
Links
- 238000001914 filtration Methods 0.000 title claims abstract description 105
- 230000003044 adaptive Effects 0.000 title claims description 16
- 238000004590 computer program Methods 0.000 claims description 8
- 239000000969 carrier Substances 0.000 claims description 4
- 230000003287 optical Effects 0.000 claims description 3
- 101700002690 CLIP3 Proteins 0.000 description 71
- 241000023320 Luma <angiosperm> Species 0.000 description 44
- UIIMBOGNXHQVGW-UHFFFAOYSA-M buffer Substances [Na+].OC([O-])=O UIIMBOGNXHQVGW-UHFFFAOYSA-M 0.000 description 15
- 238000000034 method Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 9
- 239000010751 BS 2869 Class A2 Substances 0.000 description 6
- 239000010752 BS 2869 Class D Substances 0.000 description 6
- 239000010753 BS 2869 Class E Substances 0.000 description 6
- 239000010754 BS 2869 Class F Substances 0.000 description 6
- 230000004048 modification Effects 0.000 description 5
- 238000006011 modification reaction Methods 0.000 description 5
- 229940040608 SPS Drugs 0.000 description 4
- 230000000875 corresponding Effects 0.000 description 4
- 229910003460 diamond Inorganic materials 0.000 description 4
- 239000010432 diamond Substances 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 101700044444 TKT Proteins 0.000 description 1
- 102220187649 rs145044428 Human genes 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 1
- 229910052710 silicon Inorganic materials 0.000 description 1
- 239000010703 silicon Substances 0.000 description 1
- 230000001702 transmitter Effects 0.000 description 1
Abstract
A method for encoding or decoding an image of a video sequence is provided. The method comprises obtaining a set of sample values associated with the image. The method comprises determining a relative location of the current sample value with respect to a virtual boundary. The virtual boundary is defined with respect to a block boundary between the first block of sample values and a second block of sample values. The virtual boundary is parallel with the block boundary and separated from the block boundary by at least one row or column of sample values included in the first block of sample values. The method comprises a filter strength value based on the determined relative location of the current sample value with respect to the virtual boundary. The method comprises filtering the current sample value based on the selected filter strength value.
Description
VIRTUAL BOUNDARY PROCESSING FOR ADAPTIVE LOOP FILTERING
TECHNICAL FIELD
This disclosure relates to encoding and decoding of images of a video sequence.
INTRODUCTION
A video sequence consists of several images (also referred to herein as “pictures”). When viewed on a screen, the image consists of pixels, each pixel having a red, green and blue value (RGB). However, when encoding and decoding a video sequence, the image is often not represented using RGB but typically using another color space, including but not limited to YCbCr, ICtCp, non-constant-luminance YCbCr, and constant luminance YCbCr. If we take the example of YCbCr, it is made up of three components, Y, Cb and Cr. It is often the case that Y, which is called luma and which roughly représente luminance, is of full resolution, whereas the two other components, Cb and Cr, called chroma, are of a smaller resolution. A typical example is a high définition (HD) video sequence containing 1920x1080 RGB pixels, which is often represented with a 1920x1080-resolutîon Y component, a 960x540 Cb component and a 960x540 Cr component. The éléments in the components are called samples. In the example given above, there are therefore 1920x1080 samples in the Y component, and hence a direct relationship between samples and pixels. Therefore, in this document, we sometimes use the term pixels and samples interchangeably. For the Cb and Cr components, there is no direct relationship between samples and pixels; a single Cb sample typically influences several pixels.
In the latest video coding standard, Versatile Video Coding standard (WC), that is developed by the Joint Video Experts Team (JVET), the decoding of an image is carried out in two stages; prédiction coding and loop filtering. In the prédiction decoding stage, the samples of the components (Y, Cb and Cr) are partitioned into rectangular blocks. As an example, one block may be of size 4*8 samples, whereas another block may be of size 64x64 samples. The décoder obtains instructions for how to do a prédiction for each block, for instance to copy samples from a previously decoded image (an example of temporal prédiction), or copy samples from already decoded parts of the current image (an example of intra prédiction), or a combination thereof. To improve this prédiction, the décoder may obtain a residual, often encoded using transform coding such as discrète sine or cosine transform DST or DCT. This residual is added to the prédiction, and the décoder can proceed to décodé the following block.
The output from the prédiction decoding stage is the three components Y, Cb and Cr. However, it is possible to further improve the fidelity of these components, and this is diagonally top left to bottom right and filtDI measures gradients diagonally top right to bottom left:
| fiitH[i][jj = Abs((recPicture[hx4+i, ure[hX4+i + i, vy4+j]) | Vy4 + jJ « | 1) - recPicture[hX4+ j-ι, vy4+jj - recPict | (1429) |
| filtV[i][j] = Abs((recPicture[hx4+i, ure[hX4+i, vy4+i + i]) | Vy4 + j] « | 1) - recPicture[hx4+i, vy4+j-i] - recPict | (1430) |
| filtDO[i][j] = Abs((recPicture[hx4+ Picture[hx4+i + i, vy4+j + 1]) | G W4 + jl < | < 1) - recPicture[hx4 +, -1, vy4 + j -1] - rec | (1431) |
| filtDI [i][j] = Abs((recPicture[hx4 + Picture[hx4+i-i, vy4+j + 1]) | 1; Vy4 4- i] < | < 1 ) - recPicture[hx4 +, +1, vy4+j -1] - rec | (1432) |
Then these variables are summed up in a local neighborhood around the current sample to get a more reliable estimate of the directonallity of the neighborhood as follows where sumH indicates the sum of filtH, sumV the sum of filtV, sumDO the sum of filtDO, sumD1 the sum of filtDI and sumOfHV the sum of sumV and sumV from WG draft below:
| The variables sumH[x][y], sum\/[x][y], sumD0[x][y], sumD1[x][y] and sumOfHV[x][y] are | |
| derived as follows: sumH[x][y] = ZJj filtH[i][j], with i = - 2..5, j = minY..maxY | (1433) |
| sumV[x][y] = 1,¾ filtV[i][j], with i = - 2..5, j = minY..maxY | (1434) |
| sumD0[x][y] = ZiZj filtDO[i][j], with i = - 2..5, j = minY..maxY | (1435) |
| sumD1[x][y] - SZj filtDI [i][j], with i = - 2..5, j - minY..maxY | (1436) |
| sumOfHV[x][y] = sumH[x][y] + sumV[x][y] | (1437) |
Finally based on these metrics a classification is made to détermine which set of filters filtldx to use for the current sample and also a transposeldx such that several 10 directionalitiês can share the same filter coefficients, from WC draft below:
The classification filter index array fiïtldx and transpose index array transposeldx are derived by the following steps:
1. The variables dir1 [x][y], dir2[x][y] and dirS[x][y] with x, y = O..CtbSizeY - 1 are derived as follows:
The variables hv1, hvO and dirHV are derived as follows:
If sumV[x » 2][y » 2] is greater than sumH[x » 2][y » 2], the following applies:
hv1 = sumV[x » 2][y » 2] (1438) hvO = sumH[x » 2][y » 2] (1439) dirHV = 1 (1440)
- Otherwise, the following applies:
hv1 = sumH[x » 2][y » 2] (1441) hvO = sumV[x » 2][y » 2] (1442) dirHV = 3 (1443)
- The variables d1, dO and dirD are derived as follows:
- If sumD0[x » 2][y » 2] is greater than sumD1[x » 2][y » 2], the following applies:
d1 = sumD0[x » 2][y » 2] (1444) dO = sumD1[x » 2][y » 2] (1445) dirD = 0 (1446)
- Otherwise, the following applies:
d1 = sumD1[x » 2][y » 2] (1447) dO = sumDOfx » 2][y » 2] (1448) dirD = 2 (1449)
- The variables hvd1, hvdO, are derived as follows:
hvd1 = (d1 * hvO > hv1 * dO) ? d1 : hv1 (1450) hvdO = (d1 * hvO > hv1 * dO) ? dO : hvO (1451)
The variables dirS[x][y], dir1[x][y] and dir2[x][y] derived as follows:
dir1[x][y] = (d1 * hvO > hv1 * dO) ? dirD : dirHV (1452) dir2[x][y] = (d1 * hvO > hv1 * dO) ? dirHV : dirD (1453) dirS[x][y] = (hvd1 *2 > 9 * hvdO) ? 2 : ((hvd1 > 2 * hvdO) ? 1 :
0) (1454)
2. The variable avgVar[x][y] with x, y = O..CtbSizsY - 1 is derived as follows: varTab[] = { 0, 1,2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 4 } (1455) avgVar[x][y] = varTab[Clip3(0, 15, (1456) (sumOfHV[x » 2][y » 2] * ac[x » 2][y » 2]) » (4 + BitDepth))]
3. The classification filter index array filtldx[xj[y] and the transpose index array transposeldx[x][y] with x = y = O..CtbSizeY - 1 are derived as follows:
transposeTableQ = {0, 1, 0, 2, 2, 3, 1, 3} transposeldx[x][y] = transposeTable[dir1[x][y] * 2 + (dir2[x][y] » 1)] filtldx[x][y] = avgVar[x][y]
When dirS[x][y] is not equal 0, filtldx[x][y] is modified as follows: filtldx[x][y] += (((dir1 [x][y] & 0x1) « 1) + dirS[x][y]) * 5 (1457)
From above it can be seen that filtldx equal to 0 to 4 do not hâve any spécifie directional characteristics. A value of filierldx greater than 4 corresponds to directionality of the samples, since this means that dirS is greater than 0. Studying the addition to filtldx, filtldx[x][y] += (((dir1 [x][y] & 0x1) « 1) + dirS[x][y]) * 5, we see that if we hâve a diagonal directionality, i.e., if dir1 is either 0 or 2, the first term will be zéro and we can add either 1*5 (if dirS = 1) or 2*5 (if dirS = 2). (If dirS = 0 we will not do the addition.) Hence ail values of filterldx from 5 to 14 correspond to a diagonal directionality of the samples. Likewise, if we hâve a horizontal or vertical directionality, i.e., if dir1 is either 1 or 3, then the first term (dir1 & 1)«1 will become 2. Therefore, in this case, we will either add (2+1 )*5 (if dirS = 1 ) or we will add (2+2)*5 (if dirS = 2) resulting in values between 15 and 24. Hence we hâve concluded that filtldx indicates the directionality ofthe surrounding samples in the following way as described in Table 1:
Table 1: The value filtldx indicates directionality in the following way
| filtldx range | Directionality |
| 0...4 | No directionality |
| 5..14 | Diagnoa! directionality (dir1 = 0 or 2) |
| 15..24 | Horizontal or vertical directionality (dir1 = 1 or 3) |
Where transposeldx equal to 0 corresponds to no transpose of the filter coefficients, transposeldx equal to 1 corresponds to mirror the filter coefficients along the diagonal from top right to bottom left, transposeldx equal to 2 corresponds to mirror the filter coefficients along the vertical axis and transposeldx equal to 3 corresponds to rotating the filter coefficients 90 degrees.
This means that when the filteridx is between 15 and 24 and transposeldx is equal to 3 the local structure around the current sample has a vertical directionality and when transposeldx is equal to 0 the local structure around the current sample has a horizontal directionality.
It is possible for the encoder to signal one set of coefficients for each of the 25 classes. In WC the ALF coefficients etc., are signaled in adaptive parameter sets (APS) that then can be referred by an aps index that détermines which of the defined sets to use to when decoding pictures. The décoder will then first décidé which class a sample belongs to, and then select the appropriate set of coefficients to filter the sample. However, signaling 25 sets of coefficients can be costiy. Hence the WC standard also allows that only a few of the 25 classes are filtered using unique sets of coefficients. The remaining classes may reuse a set of coefficients used in another class, or it may be determined that it should not be filtered at ail. Another way to reduce cost is to use what is called the fixed coefficient set. This is a set of 64 hard-coded filters (i.e., 64 groups of coefficient values) that are known to the décoder. It is possible for the encoder to signal the use of one of these fixed (i.e., hard-coded) filters to the décoder very inexpensively, since they are already known to the décoder. For exampîe, the décoder stores a set of 16 different groups of N index values (e.g., N=25) and the encoder transmits an initial index value that points to one of the 16 groups of N index values, where each one of the index values included in the group of N index values is associated with a class and each one of the index values points to one of the 64 hard-coded filters. For example, the first of the N values in the group of index values points to the fixed filter that should be used for the first class, the second value points to the fixed filter that should be used for the second class, etc. Accordingly, the décoder obtains an index value for a particular filter based on the initial index value and the class. Although these filters are cheap, they may not match the desired filter perfectly and may thus resuit in slightly worse quality. For samples belonging to Cb or Cr, i.e., for chroma samples. no classification is used and the same set of coefficients is used for ail samples.
Transmitting the filter coefficients is costly, and therefore the same coefficient value is used for two filter positions. For luma (samples in the Y-component), the coefficients are re-used in the way shown in FIG. 1. As shown in FIG. 1, each coefficient is used twice in the filter, and FIG. 1 shows the spatial neighborhood the luma filter covers (i.e., FIG. 1 shows the other sample values that are used to filter the value of the current sample (i.e., the sample value in the center of neighborhood) and its configuration with regards to filter coefficients for luma. It can be seen that the filter coverage is symmetric and covers up to 3 samples away from the center both horizontally and vertically.
Assume R(x,y) is the sample to be filtered, situated in the middle of the FIG. 1. Then samples R(x,y-1) (the sample exactly above) and the sample R(x,y+1) (the sample exactly below) will be treated with the same coefficient 06.
The filtered version of the reconstructed sample in position (x,y), which we will dénoté RF(x,y\ is calculated in the following way from WC spécification équation 1411 to 1426 and Table 43, where (x,y)=(hx, v) and C0= f[idx[0]] , C1 = f[idx[1]], C2 = f[idx[2]], 03= f[idx[3]], C4=f[idx[4]], C5= f[idx[5]], 06= f[idx[6]], 07= f[idx[7]], 08= f[idx[8]], 09 = f[idx[9]], C10= f[idx[10]] and 011= f[idx[11]]:
- The array of luma filter coefficients fjj] and the array of luma clipping values c[j] comesponding to the filter specified by filtldx[x][y] is derived as follows with j = 0..11 : - If AlfCtbFiltSetldxY[xCtb » CtbLog2SizeY][yCtb >> CtbLog2SizeY] is less than 16, the following applies:
i = AlfCtbFiltSetldxYfxCtb » CtbLog2SizeY][yCtb » CtbLog2SizeY] (1411) fO] - AlfFixFiltCoeff[AlfClassToFiltMap[i][filtldx[x][y]]][j] (1412) c[j] = 2atDepth (1413)
- Otherwise (AlfCtbFiltSetldxY[xCtb » CtbLog2SizeY][yCtb » CtbLog2SizeY] is greater than or equal to 16, the following applies:
i = slice_aïf_apsJdJuma[AifCtbFiÎtSetidxY[xCtb » CtbLog2SizeY][yCtb » CtbL og2SizeY] - 16] (1414) fO] = AlfCoeffL[i][filtldx[x][y]][j] (1415) c[i] = AlfClipL[i][fi!tIdx[x][y]][j] (1416)
- The luma fiîter coefficients and clipping values index idx are derived depending on transposeldx[x][y] as follows:
- If transposelndex[x][y] is equal to 1, the following applies: idx[] = {9, 4, 10, 8, 1, 5, 11, 7, 3, 0, 2, 6 } (1417)
- Otherwise, if transposelndex[x][y] is equal to 2, the following applies:
idxD = { 0, 3, 2, 1, 8, 7,6, 5, 4, 9, 10, 11 } (1418)
- Otherwise, if transposelndex[x][y] is equal to 3, the following applies: idx[] = {9, 8, 10, 4, 3, 7, 11, 5, 1, 0, 2, 6 } (1419)
- Otherwise, the following applies:
idx[] = {0, 1, 2, 3, 4, 5,6, 7, 8, 9, 10, 11 } (1420)
- The locations ( hx+i, vy+j ) for each of the corresponding luma samples ( x, y ) inside the given array recPicture of luma samples with î, j — -3..3 are derived as follows:
hx + i = Clip3( 0, pic_width_in_luma_samples - 1, xCtb + x + i ) (1421)
Vy + j = Clip3( 0, pic_heightjn_luma_samples - 1, yCtb + y + j ) (1422)
- The variables clipLeftPos, clipRightPos, dipTopPos, clipBottomPos, clipTopLeftFlag and clipBotRightFlag are derived by invoking the ALF boundary position dérivation process as specified in clause 8.8.5.5 with ( xCtb, yCtb ) and ( x, y ) as inputs.
- The variables hx + i and vy + j are modified by invoking the ALF sample padding process as specified in clause 8.8.5.6 with ( xCtb, yCtb ), ( hx + i, vy + j ), 0, clipLeftPos, clipRightPos, clipTopPos, clipBottomPos, clipTopLeftFlag and clipBotRightFlag as input
- The variable appIyAIfLineBufBoundary is derived as follows:
- If the bottom boundary of the current coding tree block is the bottom boundary of current picture and pic_height_in_luma_samples - yCtb <= CtbSizeY - 4, appIyAIfLineBufBoundary is set equal to 0:
- Otherwise, appIyAIfLineBufBoundary is set equal to 1.
- The vertical sample position offsets y1, y2 and y3 are specified in Table 43 according to the vertical luma sample position y and appIyAIfLineBufBoundary.
curr = recPictureJhx, vy] (1423) sum = f[idx[O]] * (C!ip3(-c[idx[0]l, cridxfO]], recPictureJhx, vy+y3] - curr) + Clip3(-c[idx[Q]], c[idx[0]], recPictureJhx, vy-y3] - curr)) + f[idx[1]] * (Clip3(-c[idx[1]J, c[idx[1]], recPictureJhx + i, vy + yJ - curr) + Clip3(-c[idx[1]], c[idx[1]], recPictureL[hx-1, Vy-yJ - curr)) + f[idx[2]] * (Clip3(-c[idx[2]], c[idx[2]], recPicturei_[hx, vy+y2] - curr) + Clip3(-c[idx[2]], c[idx[2]], recPictureL[hx, vy-y2] - curr)) + f[idx[3]] * (Clip3(-c[idx[3]], c[idx[3]], recPictureJhx-1, vy+y2] - curr) + Clip3(-c[idx[3]], c[idx[3]], recPictureJhx + i, vy-y2] - curr)) + f[idx[4]] * (Clip3(-c[idx[4]], c[idx[4]], recPictureJhx+ 2, vy + y1j - curr) + Clip3(-c[idx[4]], c[idx[4]], recPictureJhx-2, vy-yi] - curr)) + f[idx[5]] * (Clip3(-c[idx[5]j, c[idx[5]], recPictureL[hx + i, vy+yi] - curr) + Ciip3(-c[idx[5]], c[idx[5]j, recPictureJhx-1, vy-yi] - curr)) + f[idx[6]] * (Clip3(-c[idx[6]], c[idx[61], recPictureL[hx, vy+y1] - curr) +
Clip3(“C[idx[Sj], c[idx[6j], recPicturêL[hx, vy - yJ — curr)) + f[idx[7]] * (Clip3(-c[idx[7]], c[idx[7]], recPictureJhx-1, vy+yi] - curr) +
Cl!p3(-c[idx[7]], c[idx{7]], recPictureJhx+ 1, vy-yi] - curr)) + f[idx[8]] * (Clip3(-c[idx[8]], c[idx[8]], recPictureJhx-2, vy + yi] - curr) + Clip3(-c[idx[8]], c[idx[8]], recPictureJhx+ 2, vy-yi] - curr)) + f[idx[9]] * (Clip3(-c[idx[9]j, c[idx[9]], recPictureJhx+ 3, vy] - curr) + Clip3(-c[idx[9]], c[idx[9]], recPictureJhx-3, vy] - curr)) + f[idx[10]] * (Clip3(-c[idx[10]], c[idx[10]], recPictureJhx+2, vy] - curr) + Clip3(-c[idx[10j], c[idx[10]], recPictureJhx- 2, vy] - curr)) + f[idx[11]] * (Clip3(-c[idx[11]], c[idx[11]], recPictureJhx+i, vy] - curr) + Clip3(-c[idx[11]], c[idx[11]]s recPictureJhx-1, vy] - curr)).
(1424) sum = curr + ((sum + 64) » 7) (1425)
- The modified filtered reconstructed luma picture sample alfPictureJxCtb + x][yCtb + y] is derived as follows:
alfPictureL[xCtb + x][yCtb + y] = Clip3(0, (1 « BitDepth) - 1, sum) (1426)
Table 43 - Spécification of y1, y2, and y3 according to the vertical luma sampie position y and appiyAifLineBufBoundary
| Condition | yl | y2 | y3 |
| (y = = CtbSizeY -5 {[ y = = CtbSizeY - 4) && (appiyAifLineBufBoundary = = 1) | 0 | 0 | 0 |
| (y = = CtbSizeY - 6 | | y = = CtbSizeY - 3) && (appiyAifLineBufBoundary = = 1) | 1 | 1 | 1 |
| (y = = CtbSizeY - 7 I | y = = CtbSizeY - 2) && (appiyAifLineBufBoundary = = 1) | 1 | 2 | 2 |
| Otherwise | 1 | 2 | 3 |
CtbSizeY is the vertical size of the CTU. CTU in VVC is typically 128x128. Here the Clip3(x,y,z) operation simply makes sure that the magnitude of the value z never exceeds y or goes below x:
rx ; z < x
Clip3(x, y, z) = jy ; z > y iz ; otherwise
The clipping parameters “c[xj” are also to be signaled from the encoder to the décoder. Decoding an image is done in sequential CTU order. As an example, if the video frame has 512x256 samples, it can consist of two rows of 128x128 CTUs where each row will hâve four CTUs. When decoding the second row of CTUs, it is sometimes required that the décoder accesses samples from the preceding (first) CTU row. In a practical hardware 10 décoder, this is most often solved by saving the bottom-most lines of samples in what is called a line buffer, from where it can be accessed easily. Continuing on with our example, if we need to access two samples above the current sample. we must store two lines of samples in our line buffer, in this case 512x2 samples. However, line buffers are expensive in terms of Silicon surface area. Therefore, in many parts ofthe VVC standard, 15 reading outside the CTU is prohibited or restricted, so that there is less of a need for expensive line buffers.
Before the concept of “virtual horizontal CTU boundary” was introduced the line buffer requirement for ALF was 6 additional lines (rows) as can be seen in FIG. 13 where ALF filtering of the above CTU only can be performed with the filter covering only samples above row 4 from the bottom of the above CTU as shown in position 106. When deblocking and SAO has been performed ALF in position 105 and below rows can be performed.
One attempt to reduce the need for horizontal CTU line buffer requirements for ALF is the concept of a “virtual horizontal CTU boundary” which is used in WC. The position of the Virtual horizontal boundary is four samples above the border between the current CTU and the CTU directly above the current CTU (i.e., the “above” CTU), as can be seen in FIG. 2, which shows the position of the Virtual boundary and two possible ALF filterings. The Virtual boundary is situated here because of how deblocking works: when deblocking samples that are at the CTU boundary, the deblocking process can read up to four samples above the CTU boundary. This means that samples below the Virtual boundary cannot be deblocked in the above CTU until the current CTU has been decoded - the last four lines are dépendent on the sample values in the current CTU.
After the samples below the Virtual boundary in the above CTU hâve been deblocked, they must be processed by SAO and ALF. Assume for simplicity that SAO is disabled and that only ALF processing needs to be carried out, If no considération to the Virtual boundary was taken when doing ALF filtering, then some samples above the Virtual boundary would be accessed. This can be seen in FIG. 2 when the filter is in position 101 - three rows of samples above the Virtual boundary are accessed. This would mean that 3 extra lines of linebuffer would be necessary, which is costly. To avoid this, it is specified that when ALF filtering occurs at or below the Virtual horizontal CTU boundary, no sample above the Virtual boundary samples are used (e.g., the spatial coverage of the filter is reduced). Similarly, when filtering above the Virtual horizontal CTU boundary no sample at the virtual horizontal CTU boundary or below is used. This is accomplished by using what is called symmetric padding. Symmetric padding can be described as follows. When the center of the filter is at the virtual horizontal CTU boundary or at the row above the virtual CTU boundary the neighboring rows are padded by the row centered at the filter. this corresponds to padding parameters y1=y2=y3=0 as defined in Table 43 in WC spécification. In FIG. 3 we hâve shown an exampie where the virtual boundary is right above the center sample. In this case, symmetric padding will mean that the samples as shown to the left will be filtered as if the sample values were as is shown to the right.
When the center row is two rows above the virtual boundary the rows at or below the virtual boundary are padded by the row adjacent and below the center row, the rows above the row adjacent and above the center row are padded by the row above the center row, this corresponds to padding parameters y1=y2=y3=1 as defined in Table 43 in VVC spécification, FIG. 4 shows an example where the virtual boundary is one step removed from the filter center, and the samples to the left will then be filtered as if they were the values shown to the right. This effectively means that the spatial coverage of the filter is reduced to only become a horizontal filter.
When the center row is three rows above the virtual boundary the rows at the virtual or below the virtual boundary are padded by the row two rows below the center row. The rows more than two rows above the center row are padded by row two rows above the center row, this corresponds to padding parameters y1 = 1 and y2=2 and y3=2 in Table 43 in the WC spécification. FIG. 5 shows an example of filtering when filter support just crosses the virtual boundary, where the samples to the left are filtered as if they were the values to the right.
l he symmetric padding approach is also used for filtering of rows below the virtual boundary. For the détermination of the classification the régions used for classification is restricted to avoid including samples on the other side of the virtual boundary and in addition to thatthe adjacent rowon the otherside ofthe virtual boundary is padded based on the row on the row the current side.
Symmetric padding also applies for ALF filtering of chroma components to avoid increasing the number of line buffers for chroma. For chroma the position of the virtual boundary is on row two from the bottom of above CTU since that is the last row that is used for chroma deblocking of a horizontal chroma CTU boundary.
Besides the concept of virtual horizontal CTU boundaries as shown above virtual boundaries in WC can also be defined at horizontal positions which are a multiple of 8 and vertical positions that are a multiple of 8. For such boundaries padding is only used on the side of the filter which has any filter coefficient on the another side of the virtual boundary than the position of the center sample of the filter, e.g. current sample, such that the sample adjacent to the boundary on same side as the current sample is used instead of a sample on the other side of the virtual boundary. The symmetry padding described for virtual horizontal CTU boundaries could be seen as a double sided padded where defined virtual boundaries use single sided padding.
sps_virtual_boundaries_present_flag equal to 1 spécifiés that information of virtual boundaries is signalled in the SPS. sps_virtual_boundaries_present_flag equal to 0 spécifiés that information of virtual boundaries is not signalled in the SPS. When there is one or more than one virtual boundary signalled in the SPS, the in-loop filtering operations are disabled across the virtual boundaries in pictures referring to the SPS. In-loop filtering operations include the deblocking filter, sample adaptive offset filter, and adaptive loop filter operations ph_virtual_boundaries_present_flag equal to 1 spécifiés that information of virtual boundaries is signalled in the PH. ph_virtual_boundaries_present_flag equal to 0 spécifiés that information of virtual boundaries is not signalled in the PH. When there is one or more than one virtual boundary signalled in the PH, the in-loop filtering operations are disabled across the virtual boundaries >n the picture. The in-loop filtering operations include the deblocking filter, sample adaptive offset filter, and adaptive loop filter operations. When not présent, the value of ph_virtual_boundaries_present_flag is inferred to be equal to 0.
In JVET-O0636 [1], a tool called the Cross-Component Adaptive Loop Filter (CC-ALF) was first proposed as part of the adaptive loop filter process. The CC-ALF was studied in a Core Experiment in JVET-P meeting and JVET-Q meeting. The CC-ALF makes use of luma sample values to refîne each chroma component. The luma sample values were reconstructed luma samples after SAO and before luma ALF operations. A linear, diamond shaped filter is applied to the luma samples for each chroma component i to dérivé a residual correction Af(x, y). The residual correction is applied to the reconstructed chroma sample after ALF chroma to dérivé the reconstructed chroma sample value. FIG. 6 illustrâtes the placement of CC-ALF with respect to other loop filters: In JVET-P2025 [2], description of Core experiment 5 (CE5) : Cross component Adaptive Loop filtering, the CC-ALF spécifiés an anchor CC-ALF (name as “CE anchor” in the following of the current invention) for core experiment test, which includes the following properties: 1) Filter shape is a 3 χ 4 diamond with 8 unique coefficients; 2) Filter coefficient dynamic range between [-32, 31], inclusive; 3) Filter sélection to performed at the CTU level with support for a maximum of 4 filters; 4) Symmetric line sélection is used at virtual boundary; 5) Temporal layer coefficient buffers are not used; and 6) Residual correction is clipped to to 2^^^^-1, inclusive.
The CE anchor applies an 84ap diamond CC-ALF filter to the co-located luma samples centered at the chroma sample to be refined. FIG. 7 shows the CE anchor CC-ALF filter shape.
Assume the Rc(xc, yc) is the ALF chroma reconstructed chroma sample to be refined by CC-ALF, where the (xc, yc) spécifiés the position of the chroma sample in the current picture. The co-located luma sample to the Rc(xc, yc) is Rl(xl, yi_), where (xl, yi_) spécifiés the positioni of the co-located luma sample in the current picture. The 8 luma samples to be filtered by CC-ALF are Rl(xl-1, yL), Rl(xl, yL-1), Rl(xl, yû, Rl(xl-1, yi+1), Rl(xl+1, yL1), Rl(xl+1, yû, Rl(xl+1, Yl+1), Rl(xl+2, yi_). FIG. 8 shows the 8-tap 3x4 diamond CCALF filter coefficient respect to the luma sample location.
The residual correction Alfx.y) is calculated in the following way:
Alite,y) = (Q0 * RLteL - LyJ + Q1 * RLtei.yi - D + Q2 * RLteL,yù + Cl3 * RLteL,yL + 1) + Q4 *RLteL + l,y£ - 1) + CL5 * RL(xL + fyj + CL6 * RL(XL + Ι,η +D + Cl7 » RL(xL + 4yL) + zsiâftFactor» shiftFactor (Eqn 4) where, Clî spécifiés the CC-ALF filter coefficients, i ranges from 0 to 7, each coefficient but CL2 is trained in the encoder side.
The CC-ALF filter coefficient Clî has a value ranges between [-32, 31], inclusive. This value range is also known as dynamic range.
Cl2 = (-1)* (Q0 + Q1 + Q3 + Q4+Q5 + Q6 + Q7) with clip to a value ranges between [-32, 31], inclusive.
shiftFactor = coefficient bit scale + (BitDepthY - BitDepthc), where in CE anchor, coefficient bit scale is equal to 7.
The residual correction Alite.y) is clipped to a value range between [-2BitDePthC 1, 2BltDepthC-1-i]! inclusive. The CC-ALF refined chroma sample Rf(xc, yc) is then derived as: Rf(xc<Yc) = ΔΜΑγ) + Rc(xoYc). with clip to a value range between [0, 2BitDepthC-1], inclusive.
The CC-ALF filter coefficient is coded with 6 bits fix length code which can represent value ranges from 0 to 63. The encoded / decoded CC-ALF coefficient names as Dl(î), which i ranges from 0,1,2 to 7. The CC-ALF coefficient CL(i) is equal to DL(i) - 32.
Similar to ALF, CC-ALF also use virtual boundary processing by means of symmetric padding for the filtering of luma samples to avoid increasing the line buffer requirements for luma.
SUMMARY
Certain challenges presently exist. It was observed that the ALF virtual boundary processing can generate Visual artifacts. The artifacts are most prominent at the virtual horizontal CTU boundaries and the row above the virtual horizontal CTU boundary. The reason for artifacts is due to the padding, which effectively reduces the spatial coverage of the filter - - i.e., the number of samples used for the filtering is reduced compared to the samples that were used during the determining of the filter coefficients, as is shown in FIG. 3 through FIG. 5. Because padding affects the samples most when filtering samples close to the virtual boundary (e.g., such as in FIG. 3) compared to further away from the boundary (e.g., such as in FIG. 5) the artifacts become largest there.
If we consider the horizontal virtual boundary, the ALF filter support due to the padding will only be horizontal. This can be seen in FIG. 9. In the top left diagram, we hâve subtracted the middle sample (120) from ail the values in FIG. 3. These values are then multiplied by the coefficients shown in the top right diagram of FIG. 9. However, this is the same thing as using zéros everywhere except in the middle row, as is shown in the right most diagram
FIG. 9. Accordingly, RG. 9 illustrâtes that when using padded sample differentials (top ieft), the filter (top right) becomes just a horizontal filter (bottom).
The contribution to the filtering from the samples horizontally adjacent to the center sample will be 5 times the intended différence between the center (at position of 012 in FIG. 1 ) and the closest adjacent sample (at position of C11 in FIG. 1), and 3 times the intended différence between the center sample (at position of 012 in FIG. 1) and the adjacent samples at distance two from the center sample (at positions of C10 in FIG. 1). Furthermore, the filter coefficient for respective adjacent sample will be equal to the sum of the coefficients vertically aligned with the adjacent sample, e.g. at position of C11 the coefficient will be C11+C1+C3+C5+C7, etc.
This disclosure présents alternatives to remove the Visual artifacts from filtering at virtual boundaries. One alternative is to omit filtering at the virtual boundary and the row above the virtual boundary, e.g., when the spatial coverage of the filter is reduced significantly compared to the intended spatial coverage to avoid using samples on the other side of the virtual boundary when filtering on one side of the Virtual boundary, effectively only having a filter with horizontal spatial coverage.
Another alternative is to reduce fte amount of change of the current sample from the filtering to reflect modification of the filter due to the padding near virtual boundaries, e.g., when the spatial coverage of the filter is reduced significantly compared to the intended spatial coverage to avoid using samples on the other side of the virtual boundary when filtering on one side of the virtual boundary, effectively only having a filter with horizontal spatial coverage.
Another alternative is to restrict the amount of réduction of the spatial coverage of the filter such that the spatial coverage of the filter at least includes neighboring samples at a distance of 1 from the center sample vertically and horizontally.
Each alternative provides the advantage of removing artifacts that the virtual boundary processing for ALF in WC can introduce. The second alternative also maintains some benefit of filtering also for virtual boundaries by reducing the filtering strength e.g., the amount of change of the current sample value from filtering at virtual boundaries.
Accordingly, according to a first aspect offrie présent disclosure there is provided a method for encoding or decoding an image of a video sequence. The method comprises obtaining a set of sample values associated with the image, the set of sample values comprising a current sample value and sample values neighboring the current sample value, wherein the obtained set of sample values is included within a first rectangular block of sample values set of sample values comprises a current sample value and sample values neighboring the current sample value. The method further comprises determining a relative location of the current sample value with respect to a virtual boundary. The virtual boundary is defined with respect to a block boundary between the first block of sample values and a second block of sample values. The virtual boundary is parallel with the block boundary and separated from the block boundary by at least one row or column of sample values included in the first block of sample values. The method further comprises selecting a filter strength value based on the determined relative location of the current sample value with respect to the virtual boundary. The method further comprises filtering the current sample value based on the selected filter strength value.
According to a second aspect of the présent disclosure there is provided a method for encoding or decoding an image of a video sequence. The method comprises obtaining a set of sample values associated with an image. The set of sample values comprises a current sample value and sample values neighboring the current sample value and the obtained set of sample values are included within a first rectangular block of sample values. The method further comprises determining a relative location of the current sample value with respect to a virtual boundary. The virtual boundary is defined with respect to a block boundary between the first block of sample values and a second block of sample values. The virtual boundary is parallel with the block boundary and separated from the block boundary by at least one row or column of sample values included in the first block of sample values. The method further comprises making a decision as to whether or not to filter the current sample value based on the determined relative location of the current sample value with respect to the virtual boundary?.
According to a third aspect of the présent disclosure there is provided a computer program comprising instructions which when executed by processing circuitry causes the processing circuitry to perform the method of any one of the first or the second aspect.
According to a fourth aspect of the présent disclosure there is provided a carrier comprising the computer program according to the third aspect, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.
According to a fifth aspect of the présent disclosure there is provided an apparatus, the apparatus being adapted to perform the method of the first or the second aspect.
DETAILED DESCRIPTION
FIG. 10 illustrâtes a system 1000 according to an example embodiment. System 1000 includes an encoder 1002 and a décoder 1004. In the example shown, décoder 1004 can receive via a network 110 (e.g., the Internet or other network) encoded images produced by encoder 1002.
FIG. 11 is a schematic block diagram of encoder 1002. As illustrated in FIG. 11, The encoder 1002 takes in an original image and subtracts a prédiction 41 that is selected 51 from either previously decoded samples (“Intra Prédiction” 49) or samples from previously decoded frames stored in the frame buffer 48 through a method called motion compensation 50. The task of finding the best motion compensation samples is typically called motion estimation 50 and involves comparing against the original samples. After subtracting the prédiction 41 the resulting différence is transformed 42 and subsequently quantized 43. The quantized results are entropy encoded 44 resulting in bits that can be stored, transmitted or further processed. The output from the quantization 43 is also inversely quantized 45 followed by an inverse transform 46. Then the prédiction from 51 is added 47 and the resuit is forwarded to both the intra prédiction unit 49 and to the Loopfilter Unit 100. The loopfilter unit 100 may do deblocking, SAO and/or ALF filtering (including CC-ALF filtering). The resuit is stored in the frame buffer 48, which is used for future prédiction. Not shown in FIG. 11 is that coding parameters for other blocks such as 42, 43, 49, 50, 51 and 100 also may also be entropy coded.
FIG. 12 is a corresponding schematic block diagram of décoder 1004 according to some embodiments. The décoder 1004 takes in entropy coded transform coeffcients which are then decoded by décoder 61. The output of décoder 61 then undergos inverse quantization 62 followed by inverse transform 63 to form a decoded residual. To this decoded residual, a prédiction is added 64. The prédiction is selected 68 from either a motion compensation unit 67 or from an intra prédiction unit 66. After having added the prédiction to the decoded residual 64, the samples can be forwarded for intra prédiction of subséquent blocks. The samples are also forwarded to the loopfilter unit 100, which may do deblocking, SAO processing, and/or ALF processing (including CC-ALF filtering). The output of the loopfilter unit 100 is forwarded to the frame buffer 65. which can be used for motion compensation prédiction of subsequently decoded images 67. The output ofthe loopfilter unit 100 can also be output the decoded images for viewing or subséquent processing outside the décoder. Not shown in FIG. 12 is that parameters for other blocks such as 63, 67, 66 and 100 may also be entropy decoded. As an examp le, the coefficients for the ALF filter in block 100 may be entropy decoded.
The embodiments disclosed herein can be used to avoid generating artifacts from filtering of Virtual boundaries in a still image or images of a video. This can be applied in encoder 1002 and/or décoder 1004. The approach is mainly described for Virtual horizontal CTU boundaries but can be applicable also for Virtual boundaries in general, such as for Virtual vertical boundaries referring to columns instead of rows. The approach is also mainly described for ALF and for luma but could also be used for ALF for chroma and for CCALF.
Embodiment 1
When a sample to be filtered (“current sample”) is “on” a virtua! horizontal CTU boundary (i.e., when the current sample is in the row immediately above the Virtual boundary) or current sample is in the row immediately above the row that is on the Virtual horizontal CTU boundary (e.g. when the filters spatial coverage is reduced significantly from the intended spatial coverage, which is done to avoid having the filter to use samples on the other side of the Virtual boundary), the filtering does not modify the current sample. Otherwise the filtering may modify the current sample.
This can be implemented by determining the vertical position before filtering and omit filtering if the determined position is at the Virtual horizontal CTU boundary or the row above the Virtual horizontal CTU boundary. If the Virtual horizontal CTU boundary is at y=CtbSizeY-4 given in samples of the CTU where the top row of the CTU is row 0, then no filtering is applied when y is equal to CtbSizeY-4 or y is equal to CtbSizeY-5 (e.g. row above Virtual boundary), otherwise filtering is applied.
Another approach to do exactly the same thing is to introduce horizontal padding such that ail neighboring samples of the filter support are pointing at the current sample when the determined position is at the Virtual horizontal CTU boundary or the row above the Virtual horizontal CTU boundary and otherwise obey padding parameters such that filtering may modify the current sample. Changes to VCC in équation 1424 and in Table 43 as shown below:
sum = f[idx[0]] * (Clip3(-c[idx[0]], c[idx[0]], recPictureL[hx, vy + y3] - curr) +
CHp3(-c[idx[0]], c[idx[0]], recPictureJhx, vy-y3] - curr)) + f[idx[1]] * (Clip3(-c[idx[1]], c[idx[1]], recPicturerfhx+i-xi, vy+y2] - curr) + Clip3(-c[idx[1]l. c[idx[1]]. recPictureJhv-i + vi, vy-y2] - curr)) + f[idx[2]j * (Clip3(-c[idx[2]], c[idx[2]J, recPictureL[hx, vy + y2] - curr) + Clip3(-c[idx[2]], c[idx[2]], recPictureJhx, Vy-^] - curr)) + f[idx[3]] * (Clip3(-c[idx[3]], c[idx[3]], recPictureL[hx_i + xi, vy+y2] - curr) + Clip3(-c[idx[3]], c[idx[3]], recPictureJhx + i-xi, vy-y2] - curr)) + f[idx[4]j * (Cïip3(-c[idx[4]], c[idx[4]], recPictureL[hx+2_x2, vy+yi] - curr) + Clip3(-c[idx[4]], c[idx[4]], recPictureL[hx_2 + x2, vy-yi] - curr)) + f[idx[5]] * (Clip3(-c[idx[5]], c[idx[5]], recPictureL[hx+1 - xi, vy+yi] - curr) + Clip3(-c[idx[5]], c[idx[5]], recPictureL[hx-i + xi, vy-yi] - curr)) + f[idx[6]] * (Clip3(—cndxr©]]. οΠάχΓΘ]], recPictureThx, vy+yi] - curr) +
Clip3(-c[idx[6]], c[idx[6]], recPicturei_[hx, vy-yi] - curr)) + f[idx[7]] * (Clip3(-c[idx[7]], c[idx[7]], recPictureL[hx_i + xi, vy+yi] - curr) +
Ciip3(-c[idx[7]], c[idx[7]], recPicturerfhx+i .x1, vy-y1] - curr)) + f[idx[8]] * (Clip3(-c[idx[8]], c[idx[8]], recPictureL[hx-2 + x2, vy + yi] - curr) +
Clip3(-c[idx[8J], c[idx[8]], recPictureL[hx+2.x2, vy-y1] - curr)) + f[idx[9]] * (Clip3(-c[idx[9]], c[idx[9]], recPictureL[hx + 3-x3, vy] - curr) +
Clip3(-c[idx[9]], c[idx[9]], recPictureJhx-s + xa, vy] - curr)) + f[idx[10]] * (Clip3(-c[idx[10]], c[idx[10]], recPictureJhx+a-xa, vy] - curr) +
Clip3(-c[idx[10]], c[idx[10]], recPictureL[hx_2 + x2, vy] - curr)) + f[idx[11]] * (Clip3(-c[idx[11]], c[idx[11]], recPictureL[hx+i _xi, vy] - curr) +
Clip3(-c[idx[11 ]], c[idx[11]], recPictureiJhx-! + x1, vy] - curr)). (1424)
| Condition | x1 | x2 | x3 | yl | y2 | y3 |
| (y = = CtbSizeY - 5 | | y = = CtbSizeY - 4) && IΛ Λ Λ L » Λ Ιί 1 îm A. D · a « > A m r Ί \ (ορ’ριγζΜΐΐ-ίΠσουιΟνυΠυαιγ--i} | 1 | 2 | 3 | 0 | 0 | 0 |
| (y = = CtbSizeY - 6 | | y = = CtbSizeY - 3) && (appIyAIfLineBufBoundary = = 1) | 0 | 0 | 0 | 1 | 1 | 1 |
| (y = = CtbSizeY - 7 | | y = = CtbSizeY - 2) && (appIyAIfLineBufBoundary = = 1) | 0 | 0 | 0 | 1 | 2 | 2 |
| Otherwise | 0 | 0 | 0 | 1 | 2 | 3 |
zmbodiment 2
In this embodiment a spécifie réduction of the amount of filtering is applied for filtering of the current sample immediately adjacent to the virtual horizontal boundary (i.e., immediately above or immediately below the virtual boundary) and the row immediately above/below the row that is immediately adjacent to the virtual horizontal boundary. In that case the spatial coverage of the filter is reduced to only cover samples in the horizontal direction.
This can be implemented by determining the vertical position before filtering and specifically reduce the amount of filtering if the determined position is at the virtual horizontal CTU boundary or the row above the virtual horizontal CTU boundary. If the virtual horizontal CTU boundary is at y=CtbSizeY-4 given in samples of the CTU where the top row of the CTU is row 0. Then specifically reduce the amount of filtering when y is equal to CtbSizeY-4 or y is equal to CtbSizeY-5 (e.g. row above virtual boundary) otherwise no specifically réduction in the amount of filtering is applied.
More specifically, the amount of filtering refers to the strength of filtering. As an examplê, assumewe hâvethree samples Left, Icenter, Iright. Afiltered version ofthe center sample can be expressed using différences as:
Ifiltered - Icenter + [5*(Ileft - Icenter) + 5*(Iright - Icenter)]/16.
The above is équivalent to:
i'filtered - [5*Ileft + 6*i'center) + 5*ÎrightJ/16, which is very strong filtering. However, by changing the division by 16 to division by 32 (i.e., changing the diviserfrom a value of 16 to a value of 32), we get
Ifiltered = Icenter + [5*(Ileft - Icenter) + 5*(Iright - Icenter)]/32.
This is équivalent to:
Ifiltered = [5*Ileft + 22*1center) + 5*Iright]/32, which results in much weaker filtering, since the filtered version of the Icenter value will on average deviate less from the original Icenter value. It is also clear that, when the filter is expressed using différences, it is easy to change the strength of the filter just by changing the divisorfrom 16 to 32. Typicaliy, the divisorcan be defined as χ*2Λ(-Ν) where N is greater than 0. This can preferably be implemented by right shift, χ*2Λ(-Ν) = x » N, e.g. a right shift of x.
One approach to implement this in WG is to détermine a right shift factor which has a higher value when the determined vertical position is at a Virtual horizontal CTU boundary or at the row above such a boundary than the value of the right shift factor otherwise. A reasonable increase of the right shift factor is by at least 2 e.g. basically reducing the contribution of the filtering to one fourth of the current value since the padding for luma corresponds to an increase of the contribution by 5 for the two horizontally adjacent samples to the current sample and for chroma corresponds to an increase of the contribution by 3 for the two horizontally adjacent samples to the current chroma sample. This can be achieved on top of the WC spécification by introducing an adaptive shift factor, alfShift yellow marked below. Thus, the amount of modification of the current sample curr can be reduced for samples on the vertical horizontal CTU boundary and the row above that row. In the example below for luma the right shift factor corresponds to alfShiftY and in this example the right shift is increased by 3 corresponding to reducing the contribution of the filtering to one eight of the current value.
sum = curr + ((sum + 64) » alfShiftY) (1425)
Table 43 - Spécification of y1, y2, y3, and alfShiftY according to the vertical luma sample position y and appIyAIfLineBufBoundary
| Condition | alfShiftY | y1 | y2 | yô |
| (y = = CtbSizeY - 5 | j y = = CtbSizeY - 4) && (appIyAIfLineBufBoundary = = 1) | 10 | 0 | 0 | 0 |
| (y = = CtbSizeY - 6 | | y = = CtbSizeY - 3) && (appiyAifLineBufBoundary = = 1) | 7 | 1 | 1 | 1 |
| (y = = CtbSizeY - 7 | | y = = CtbSizeY - 2) && (appiyAifLineBufBoundary = = 1) | 7 | 1 | 2 | 2 |
| Otherwise | 7 | 1 | 2 | 3 |
In the example below for chroma where the right shift factor corresponds to alfShiftC and in this exampie the right shift is increased by 3 corresponding to reducing the contribution of the filtering to one eight of the current value.
sum = curr + ( ( sum ·* 64 ) » alfShiftC- )(1466)
Table 44 - Spécification of y1, y2 and alfShiftC according to the vertical chroma sample position y and appiyAifLineBufBoundary
| Condition | alfShiftC | yi | y2 |
| ( y = = ctbHeightC - 2 I | y = = ctbHeightC - 3 ) && ( appiyAifLineBufBoundary = = 1 ) | 10 | 0 | 0 |
| ( y = = ctbHeightC - 1 | | y = = ctbHeightC - 4 ) && ( appiyAifLineBufBoundary = = 1 ) | 7 | 1 | 1 |
| Otherwise | 7 | 1 | 2 |
An alternative is to instead define an additional shift factor after the normal right shift factor as, example for luma below but same approach could be used for chroma as well, 10 follows:
sum = curr + (((sum + 64) » 7) » alfShiftY) (1425)
| Condition | alfShiftY | yi | y2 | y3 |
| (y = = CtbSizeY - 5 [ | y = = CtbSizeY - 4) && (appiyAifLineBufBoundary = = 1) | 3 | 0 | 0 | 0 |
| (y = = CtbSizeY -6 | j y = = CtbSizeY - 3) && (appiyAifLineBufBoundary = = 1) | 0 | 1 | i | 1 |
| (y = = CtbSizeY - 7 | | y = = CtbSizeY - 2) && (appiyAifLineBufBoundary = = 1) | 0 | 1 | 2 | 2 |
| Otherwise | 0 | 1 | 2 | 3 |
This would require one additional shift. Embodiment 3
As embodiment 2 wherein the réduction of the filtering is depending on how many 15 samples that are padded, such that most réduction of the contribution from the filtering is happening for filtering of samples immediately adjacent to the virtual horizontal boundary (i.e., sampies immediately above the Virtual boundary and samples immediately below the Virtual boundary) and the row immediately above/below a row that is immediately adjacent to the Virtual horizontal boundary.
One approach to do this on top of embodiment 2 is to vary the alfShiftY factor as follows: sum = cuit + ((sum + 64) » alfShiftY) (1425)
| Condition | alfShiftY | y1 | y2 | y3 |
| (y = = CtbSizeY - 5 H y = = CtbSizeY - 4) && (appIyAIfLineBufBoundary = = 1) | 10 | 0 | 0 | 0 |
| (y = = CtbSizeY - 6 | I y = = CtbSizeY - 3) && (appIyAIfLineBufBoundary = = 1) | 9 | 1 | 1 | 1 |
| (y = = CtbSizeY -7 | | y = = CtbSizeY - 2) && (appIyAIfLineBufBoundary = = 1) | 8 | 1 | 2 | 2 |
| Otherwise | 7 | 1 | 2 | 3 |
Thus the amount of change of the current sample curr is reduced more when more samples are padded.
Embodiment 4
In this embodiment an additional condition is checked to détermine if filtering shall be applied or if filtering shall not be applied.
One condition that relates to the characteristics of the neighborhood for the current sample is to check if the neighborhood has structure with a vertical (gradients dominate in horizontal direction) or diagonal directionality (gradients dominate in diagonal direction +45/-45 degrees from the vertical axis) and if that’s the case détermine that filtering of the sample is not performed otherwise filtering can be performed, The reason for avoiding filters when the neighborhood has structure with vertical or diagonal directionality it is that the padded filter only make use of samples in the horizontal direction when filtering on the Virtual horizontal CTU boundary and the row above the Virtual horizontal CTU boundary which do not fit with the intent of the filters for such neighborhooas and can thus create Visual artifacts. instead of improving the image.
With WC as an example this can be implemented as follows. First détermine if the current sample is on the Virtual horizontal CTU boundary or the row above that and in that case also set the filter coefficients to 0 for categories of filters that correspond to neighborhoods of a current sample that has structure with vertical (gradients dominate in horizontal direction) or diagonal directionality (gradients dominate in diagonal direction +45/-45 degree from vertical axis). Neighborhoods of a current sample that corresponds to structure of horizontal directionality (gradients dominate in vertical direction) or no directionality (no dominating gradient direction) can be allowed to be filtered. This can for exampie be determined by checking the filterldx and the transposeldx that is determined as part of the classification which hâve been determined from computations of gradients vertically, horizontally and diagonally. If filterldx is between 0 and 4 or between 15 and and transposeldx is equal to 0 filtering can be performed and otherwise it is not performed.
A variant which is more conservative is to only allow filtering for a current sample when it has a neighborhood that has structure with horizontal directionality. This can guarantee that the filtering does not dégradé structure that has both vertical and horizontal directionality. This can be determined by checking if filterldx is between 15 and 24 and transposeldx is equal to 0 otherwise no filtering is performed. On the other hand, it reduces the possibility to remove coding artifacts.
Another variant is to only allow filtering as in above variants when filter coefficients hâve been determined specifically for the content which is being coded, e.g., provided in the bitstream. In WC that can be discovered be checking if AlfCtbFiltSetldxY is greater than or equal to 16 which means that filter coefficients hâve been provided in the bitstream and are not from a predetermined set of fixed filter coefficients specified in the standard. Embodiment 5
For luma it first détermines if the current sample to be filtered by ALF is immediately adjacent to the Virtual horizontal CTU boundary or in a row immediately above/below the row that is immediately adjacent to the Virtual horizontal CTU boundary. If that is the case it then détermines if the filterldx is less than 5 or if filterldx is greater than 15 This embodiment is basically a combination of variants from embodiment 2 and 4.
For luma it first détermines a first condition that checks if the current rowto be filtered by ALF is at the Virtual horizontal CTU boundary or the row above the Virtual horizontal CTU boundary. If the condition is not fulfilled the ALF filtering is applied. If the condition is fulfilled it then checks a second condition that checks the filterldx for the 4x4 block that includes the current sample to be filtered by ALF. The second condition is true if the filterldx is less than 5 (isotropie neighborhood) or if filterldx is greater than or equal to 15 but less than 25 and the transposeldx is equal to 0 (neighborhood with horizontal directionality). If the second condition is true the amount of filtering is reduced by increasing the right shift factor in équation 1425 by two so that the contribution from filtering is reduced to a quarter ofthe normal filtering. Otherwise if the second condition not is true no filtering is applied for the current sample which can be described by setting x1=1, x2=2 and x3=3 (high-lighted in yellow) in the modified version of équation 1424. In software implémentation this is identical to just omit filtering in this case.
sum = f[idx[0]] * (Clip3(-c[idx[0]], c[idx[0]], recPicturei_[hx, vy + ys] - curr) + Clip3(-c[idx[0]], c[idx[0]], recPictureL[hx, vy-y3] - curr)) + f[idx[1]] * (Ciip3(-c[idx[1]], c[idx[1]], recPictureL[hx+i_xi, vy+y2j - curr) +
Clip3(-c[idx[1]], c[idx[1]], recPictureL[hx_-i + x1> vy-y2] - curr)) + f[idx[2]] * (Clip3(-c[idx[2]], c[idx[2]], recPictureL[hx, vy+y2] - curr) + Clip3(-c[idx[2]], c[idx[2]], recPictureJhx, vy-y2] - curr)) + f[idx[3]] * (Clip3(-cfidx[3]]. c(idx[3]l, recPictureL[hx_-i + x1. vy+y2] - curr) +
Clip3(-c[idx[3]], c[idx[3]], recPictureL[hx +1 - x-ι, vy-y2] - curr)) + f[idx[4]] * (Clip3(-c[idx[4]], c[idx[4]], recPictureL[hx + 2_x2, vy+yi] - curr) +
Clip3(-c[idx[4]], c[idx[4]], recPictureL[hx_2 + x2, vy-yi] - curr)) + f[idx[5]] * (Clip3(-c[idx[5]], c[idx[5]], recPictureL[hx+1 .xi, vy + y1] - curr) + Clip3(-c[idx[5]j, c[idx[5]j, recPictureJhx-i +xi, vy- yi] - curr)) + f[idx[6]] * (Clip3(-c[idx[6]], c[idx[6]], recPictureL[hx, vy + y1] - curr) + Clip3(-c[idx[6]], c[idx[6]], recPictureJhx, vy-yi] - curr)) + (1424) f[idx[7]] * (Clip3(— c[idx[7]], c[idx[7]], recPictureL[hx-i + xi, vy+yi] - curr) + Clip3(-c[idx[7]], cfidxP]], recPictureJhx+i -xi, vy-yi] - curr)) + f[idx[8]] * (Clip3(-c[idx[8]], c[idx[8]], recPictureL[hx_2 + x2i vy+yi] - curr) +
Clip3(-c[idx[8]], c[idx[8]], recPictureL[hx + 2 - x2, vy-yi] - curr)) + f[idx[9]] * (Clip3(-c[idx[9]], c[idx[9]], recPictureL[hx + 3-X3, vy] - curr) + Clip3(-c[idx[9]], c[idx[9]], recPictureL[hx_3 + x3, vy] - curr)) + f[idx[10]] * (Clip3(—c[idx[10]], c[idx[10]], recP!ctureL[hx + 2.x2, vy] - curr) +
Clip3(-c[idx[10]], c[idx[10]], recPictureL[hx-2 + x2, vy] - curr)) + f[idx[11]] * (Cîip3(-c[idx[11 j], c[idx[11]], rêcPictureJhx+i _x1, vy] - curr) + Clip3(-c[idx[11]], c[idx[11]], recPictureL[hx-i+ xi, vy] - curr)) sum - curr + ((sum + 64) » alfShiftY) (1425)
| Condition | alfShiftY | yi | y2 | y3 |
| (y = = CtbSizeY - 5 jj y = = CtbSizeY - 4) && (appIyAIfLineBufBoundary = = 1) | 9 | 0 | 0 | 0 |
| (y = = CtbSizeY - 6 I I y = = CtbSizeY - 3) && (appIyAIfLineBufBoundary = = 1) | 7 | 1 | 1 | 1 |
| (y = = CtbSizeY - 7 | | y = = CtbSizeY - 2) && (appIyAIfLineBufBoundary = = 1) | 7 | 1 | 2 | 2 |
| Otherwise | 7 | 1 | 2 | 3 |
For chroma it first détermines if the current row to be filtered by ALF is at the virtual horizontal CTU boundary or the row above the virtual horizontal CTU boundary. If that is the case it then reduces the amount of filtering by increasing the right shift factor in équation 1466 by two so that the contribution from filtering is reduced to a quarter of the normal filtering. Otherwise normal filtering is applied for the current sample.
sum = curr + ((sum + 64) » alfShiftC) (1466)
Table 44 - Spécification of y1 ,y2 and alfShiftC according to the vertical chroma sample position y and appIyAIfLineBufBoundary
| Condition | alfShiftC | y1 | y2 |
| (y = = ctbHeightC - 2 | | y = = ctbHeightC - 3) && (appIyAIfLineBufBoundary = = 1) | 9 | 0 | 0 |
| (y = = ctbHeightC - 1 | j y = = ctbHeightC - 4) && (appIyAIfLineBufBoundary = = 1) | 7 | 1 | 1 |
| Otherwise | 7 | 1 | 2 |
Embodiment 6
As in previous embodiments where a parameter is coded/decoded in encoded/decoded bitstream. When the parameter indicates if the approach is used or not used.
The parameter can be encoded/decoded at one of the following levels of the bitstream such as adaptive parameter set (APS), picture, sïice or block.
Embodiment 7
In this embodiment an offset (alfShiftOffset) to the right shift parameter (7) in équation 1425 is coded/decoded in the video bitstream. The additional parameter is highlighted in yellow:
sum = curr + ((sum + 64) » (7+alfShiftOffset) (1425)
The offset can be encoded/decoded at one of the following levels of the bitstream such as adaptive parameter set (APS), picture, slice or block. The range of the parameter can for example be 0 to 3 which corresponds to the capability to reduce the amount of filtering by 1/2, 1/4 or 1/8.
Embodiment 8
As other embodiments but where the Virtual boundary is a Virtual vertical boundary and the row instead is a column to the left of the Virtual vertical boundary. In this case, variants that only allow filtering when the structure has horizontal directionality is instead allowing filtering only when the structure has a vertical directionality. Embodiment 9
As other embodiments but where the approach applies to luma and chroma components.
Embodiment 10
As other embodiments but where the approach only applies to luma.
Embodiment 11
In this embodiment the amount of réduction of the spatiai coverage of the filter is restricted such that the spatiai coverage of the filter at least includes neighboring samples at a 5 distance of 1 from the center sample vertically and horizontally.
In the context of virtual horizontal CTU boundaries this can be achieved by modification of the position of the virtual horizontal CTU boundary by one row upwards (one row above the row that deblocking of current CTU can use) and at the same time allow for ALF filtering to read one row from the other side of the ALF virtual boundary. This enables the filter to 10 at least include neighboring samples values at a distant of 1 from the center sample vertically. This avoids the extreme case of padding ail neighboring rows, e.g. reduced the spatial coverage of a filter to only be horizontal, when filtering at the ALF virtual boundary or the row above the ALF virtual boundary.
The position of the virtual horizontal boundary in this case is five samples above the top 15 row of the current CTU inside the above CTU, as can be seen in FIG. 14.
The position of the filter when it is centered immediately below the virtual boundary. je., the position of the current sample is immediately below the virtual boundary, can be seen as in 103 and immediately above the virtual boundary as in 104, i.e., the current sample is immediately above the virtual boundary, in FIG. 14. Since the filter is allowed to read 20 samples from one row below the virtual boundary when the center of the filter is in the row above the virtual boundary as in 104 or read samples above the virtual boundary when the center of the filter is in the row immediately below the virtual boundary as in 103, the amount of padding in this case is reduced to 4 lines of padding which is the most amount of padding that happens when the filter is allowed to read samples from one row on the other side of 25 the virtual boundary. The padding in the case when the filter is at the row immediately above the virtual boundary, current sample on the row immediately above the virtual boundary, can be seen in FIG. 15. The padding in the case when the filter is at the row immediately below the virtual boundary. current sample on the row immediately below the virtual boundary, can be seen in FIG. 24. In FIG. 16 the filter is on the second row above 30 the virtual boundary, i.e., the current sample is in the row that is immediately above the row that is immediately above the virtual boundary, and in this case only two rows need to be padded. In FIG. 25 the filter is on the second row below the virtual boundary, i.e., the current sample is in the row that is immediately below the row that is immediately below the virtual boundary, and in this case only two rows need to be padded.
If we consider the virtual horizontal boundary, the ALF filter support due to the padding will be three rows, one row above and one row below the current row. This can be seen in FIG. 17. In the top left diagram, we hâve subtracted the middle sample (120) from ail the values in FIG. 3. These values are then multiplied by the coefficients shown in the top right diagram of FIG. 17. However, this is the same thing as using zéros everywhere except in the three middle rows, as is shown in the right most diagram of FIG. 17. The contribution to the filtering from the samples on the same row as the center sample will be as intended (at position of C9, C10 and C11 in Fig I). The contribution from the samples on the row above and the row below the center sample will be amplified when they are doser to the center and the filter coefficient for respective sample in the row above the center will be equal to the sum of the coefficients vertically aligned and above, and the filter coefficients for respective sample in the row below will be equal to the sum of the coefficients vertically and below (2*(C5+C1), 3*(C6+C2+C0), 2*(C7+C3) and in FIG. 17).
Compared to the extreme padding of ALF filtering of the virtual horizontal CTU boundary as in the current WC draft where amplified contributions from only the center row of samples are used as shown in FIG. 5 this embodiment reduce the amplification by about half and the amplification is spread for samples both above and below the current sample which makes the filtering more robust and without the artifacts of the extreme padding approach.
This filtering approach can be achieved for luma by replacing Table 43 of the current WC draft with the following table:
| Condition | y1 | y2 | y3 |
| ( y = = CtbSizeY - 6 | | y = = CtbSizeY - 5 ) && ( appIyAIfLineBufBoundary = = 1 ) | 1 | 1 | 1 |
| ( y = = CtbSizeY - 7 | | y = = CtbSizeY - 4 ) && ( appIyAIfLineBufBoundary = = 1 ) | 1 | 2 | 2 |
| Otherwise | 1 | 2 | 3 |
ror cnroma this can be achieved by replacing table 44 of the current WC draft with the following table:
| Condition | y1 | y2 |
| ( y = = ctbHeightC - 3 | | y = = ctbHeightC - 4 ) && ( appIyAIfLineBufBoundary = = 1 ) | 1 | 1 |
| Otherwise | 1 | 2 |
This embodiment increases the line buffer requirements by two lines (rows) as can be seen from position 103 in FIG. 14. This embodiment also affects luma with regards to the ALF classification which is performed for every 4x4 block based on a neighborhood of 8x8 (e.g. two sample outside the block in ail directions) since it will hâve a virtual horizontal boundary inside a 4x4 block. One solution to that is to base the classification on the bottom half of the rows (row 2 and 3 not row 0 and 1) and samples from the row below the 4x4 block (neighborhood 4x8) for fiitering below the Virtual boundary and base the classification on the upper half of the rows (row 0 and 1) and samples from the row above the 4x4 block (neighborhood 4x8) for filtering above the Virtual boundary. A simpler and perhaps also more robust solution is to use same classification both above and below the Virtual boundary. In such a solution the classification can be made for the whole 4x4 block (row 0 to 3) and samples from the row above the 4x4 block (neighborhood 6x8) and then this classification is used for filtering of ail samples inside the 4x4 block irrespectively if samples are below or above the Virtual boundary. That would require an additional line buffer for storing classification index (0-24) and transpose index (0-3) which corresponds to 7 bits times 3840/4 bits for 4K video (e.g. about 800 bytes). This would still be less than half of the number of line buffers for not using Virtual boundaries at ail since in that case you would need to store 6 additional hnes for luma (rows) where one line of luma samples for 4K video at a bit depth of 10 corresponds to 10*3840 bits (e.g. about 4800 bytes).
The simplest modification to the classification in WC is to use the same classification above and below the Virtual boundary considering that the Virtual boundary row also is available when filtering the row above the Virtual boundary row, which can be performed by just removing a adjustment for minY and maxY for the last 4x4 blocks in the CTU which starts when y4 is equal to CtbSizeY-4 shown in the table below which shows changes to the current VVC draft. We still need to keep the adjustment for the second last rows of 4x4 blocks in the CTU since we cannot read samples below the last row of that 4x4 block:
8.8.5.3 Dérivation process for ALF transpose and filter index for luma samples
- The variables appIyAIfLineBufBoundary, minY, maxY, and ac[ x ][ y ] are derived as follows:
- If y4 is equal to ( CtbSizeY - 8 ) and one of the following condition is true, minY is set equal to -2, maxY is set equal to 3, appIyAIfLineBufBoundary is set equal to 1, and ac[ x ][ y ] is set equal to 96.
- The bottom boundary of the current coding tree block is the bottom boundary of the picture and pic_height_in_luma_samples - yCtb > CtbSizeY - 4.
- The bottom boundary of the current coding tree block is not the bottom boudary of the picture.
—Otherwise;.....if y4 is equal to ( CtbSizeY-- 4 ) and one of the follewiFrg-Gondition is true, mmY is set equal to 0, maxY is set equal to 5,-appIyAIfLineBufBoundary is set equal to 1, and ac[ x ][ y ] is set equal to 96.
---The bottom boundary ofthe current coding tree block is the bottom-boundary of the pictureaRê-pie-height_in__iuma_sampi'os^ yCtb >-GtbSizeY - 4.
---The bottom boundary of the current coding tree block is not the bottom boudary of the picture
- Otherwise, minY is set equal to -2 and maxY is set equal to 5, appIyAIfLineBufBoundary is set equal to 0 and ac[ x ][ y ] is set equal to 64.
Another solution to the classification is to use the two rows in the upper half of the 4x4 blocks that includes the Virtual boundary for filtering above the boundary and the two rows in the lower half of the 4x4 block for filtering at or below the boundary. Thus, it is needed to perform spécifie classifications for respective half of the 4x4 blocks that includes the 5 Virtual boundary. The way the standard is written this requires storage of two versions of the classification for such 4x4 blocks. The modification of the classification can be specified in WC as shown in the table below:
8.8.5.3 Dérivation process for ALF transpose and filter index for luma samples
- The variables appIyAIfLineBufBoundary, minY, maxY, and ac[ x ][ y 1 are derived as follows:
- If y4 is equal to ( CtbSizeY - 8 ) and one of the following condition is true, minY is set equal to -2, maxY is set equal to 1, minYB is set equal to 2, maxYB is set equal to 5, appIyAIfLineBufBoundary is set equal to 1, and ac[ x ][ y ] is set equal to 128.
- The bottom boundary ofthe current coding tree block is the bottom boundary of the picture and pic_height_in_luma_samples - yCtb > CtbSizeY - 4.
- The bottom boundary of the current coding tree block is not the bottom boudary of the picture.
—Otherwise, if y4 is equal to ( CtbSizeY - 4 ) and-ene of the following condition is true, minY is set equal to 0, maxY is set equal to 5, appIyAIfLineBufBoundary is set equal to 1, and ac[ x][y ] is set equal to 96.
---The bottom boundary ofthe current coding tree block is the bottom boundary ofthe picture and pic_height_in_luma_samples - yCtb > CtbSizeY - 4.
boudary of the picture
- Otherwise, minY is set equal to -2 and maxY is set equal to 5, appIyAIfLineBufBoundary is set equal to 0 and ac[x ][ y ] is set equal to 64.
If y is equal to (CtbSizeY - 6) or if y is equal to (CtbSizeY -5) an additional classification is made:
The additional variables sumHB[x][y], sumVB[x][y], sumD0B[ x ][y ], sumD1 B[ x ][ y 1 and sumOfHVBf x 1[ y 1 are derived as follows:
sumHB[x][y ] = ΣΣ) fiîtH[ i ][ j ], with i = - 2..5, j = minYB..maxYB (1433) sumVB[ x ][ y ] = ΣΣ| filtV[ i ][ j ], with i = - 2..5, j = minYB..maxYB (1434) sumD0B[ x ][ y ] = ΣΣ) filtDO[ i ][ j ], with i = - 2..5, j = minYB..maxYB (1435) sumD1B[ x ][ y ] = ΣίΣ) filtDI[ i ][ j ], with i = - 2..5, j = minYB..maxYB (1436) sumOfHVB[ x ][ y ] = sumH[ x ][ y ] + sumV[ x ][ y ] (1437)
The variables dir1 B[ x ][ y ], dir2B[ x ][ y ] and dirSB[ x ][ y ] are derived as follows:
- The variables hv1B, hvOB and dirHVB are derived as follows:
- If sumVB[ x » 2 ][ y » 2 ] is greater than sumHB[ x » 2 ][ y » 2 ], the following applies:
hv1 B = sumVB[ x » 2 ][ y » 2 ] (1438) hvOB = sumHB[ x » 2 ][ y » 2 ] (1439) dirHVB - 1 (1440)
- Otherwise, the following applies:
hv1B = sumHB[x » 2 ][ y » 2 ] (1441) hvOB = sumVB[ x » 2 ][ y » 2 ] (1442) dirHVB = 3 (1443)
- The variables d1B, dOB and dirDB are derived as follows:
If sumD0B[ x » 2 ][ y » 2 ] is greater than sumD1 B[ x » 2 ][ y » 2 ], the following applies:
d1 B = sumD0B[ x » 2 ][y » 2 ] (1444) dOB = sumD1B[ x » 2 ][ y » 2 ] (1445) dirDB = 0 (1446)
- Otherwise, the following applies:
d1B = sumD1B[ x » 2 ][ y » 2 ] (1447) dOB = sumDOBf x » 2 ][ y » 2 ] (1448) dirDB = 2 (1449)
- The variables hvd1 B, hvdOB, are derived as follows:
hvd1B = ( d1B * hvOB > hv1B * dOB ) ? d1B : hv1B (1450) hvdOB = ( d1B * hvOB > hv1B * dOB ) ? dOB : hvOB (1451)
The variables dirSB[x][y], dir1B[x][y] and dir2B[x][y] derived as follows:
dir1 B[x ][y ] = ( d1B * hvOB > hv1B * dOB ) ? dirDB : dirHVB (1452) dir2Bf x ][y ] = ( d1B * hvOB > hv1B * dOB ) ? dirHVB ; dirDB (1453) dirSBf x ][ y ] = ( hvd1B *2 > 9 * hvdOB ) ? 2 : ( ( hvd1B > 2 * hvd OB) ? 1 : 0) (1454)
The variable avgVarBf x ][ y ] is derived as follows:
varTabf ] = { 0, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 4 } (1455) avgVarBf x ][ y ] = varTabf Clip3( 0, 15, (1456) ( sumOfHVBf x » 2 ][ y » 2 ] * acf x » 2 ][ y » 2 ] ) » ( 4 + BitDepth ) ) ]
The additional classification filter index array filt'dxBf x ][ y ] and the transpose index array transposeldxBf x ]f y ] are derived as follows: transposeTablef] = {0, 1, 0, 2, 2, 3, 1, 3 } transposeldxBf x ][ y ] = transposeTablef dir1 B[ x ][ y ] * 2 + ( dir2B[ x ]f y ] » 1)] filtldxBf x ][ y ] = avgVarBf x ][ y ]
- When dirSBf x ][ y ] is not equal 0, filtldxBf x ][ y ] is modifîed as follows: filtldxBf x ][ y ]+=((( dirlBf x ]f y ]& 0x1 ) << 1 ) + dirSBf x ][y ] ) * 5 (1457)
The classification filter index array filtldxf x ][ y ] and the transpose index array transposeldxf x ][ y ] with x = y = 0..CtbSizeY - 1 are derived as follows: transposeTablef ] = { 0, 1, 0, 2, 2, 3, 1,3} transposeldxf x ][ y ] = transposeTablef dîr1 [ x ]f y ] * 2 + ( dir2[ x ][ y ] » 1 ) ] filtldxf x ][ y ] = avgVarf x ][ y ]
When dirS[ x ][ y ] is not equal 0, filtldx[ x ][ y ] is modified as follows:
filtldx[ x ][ y ] += ( ( ( dir1 [ x ][ y ] & 0x1 ) « 1 ) + dirS[ x ][ y ] ) * 5 (1457)
If y is equal to (CtbSizeY-6) or if y is equal to (CtbSizeY-5) transposeldx [ x ][ y ] += transposeldxB [ x ][ y ] « 2 fiiteridx [ x ][ y j += fiiteridxB [ x ][ y j « 5
Then it is also needed to modify the filtering such that correct class is used for rows CtbSize-6 and CtbSize-5 which can be described as follows in VVC:
8.8.5.2 Coding tree block filtering process for luma samples
For the dérivation of the filtered reconstructed luma samples alfPictureL[ x ][ y ], each reconstructed luma sample inside the current luma coding tree block recPictureL[ x ][ y ] is filtered as follows with x, y = 0..CtbSizeY - 1 :
If y is equal to (CtbSizeY-6) or if y is equal to (CtbSizeY-5) then filterldx[ x ][ y ] = fiiterldx[ x ][ y ] >> 5 transposeldx[ x ][ y ] = transposeldxf x ][ y ] »2
Otherwise filterldx[ x ][ y ] = filterldx[ x ][ y ] && 31 transposeldxf x ][ y ] = transpose ldx[ x ][ y ] && 3 Embodiment 12
Combinations between one or several embodiments.
FIG. 18 is a flow chart illustrating a process 1800 for processing (e.g., encoding or decoding) an image. Process 1800 may begin in step s1802.
Step s1802 comprises obtaining a set of sample values associated with an image. The set of sample values comprises a current sample value and sample values neighboring 10 the current sample value and the obtained set of sample values are included within a first rectangular block of sample values.
Step s1804 comprises determining a relative location of the current sample value with respect to a virtual boundary. The virtual boundary is defined with respect to a block boundary between the first block of sample values and a second block of sample values. 15 The virtual boundary is parallel with the block boundary and separated from the block boundary by at least one row or column of sample values included in the first block of sample values.
Step s1806 comprises making a decision as to whether or not to filter the current sample value based on the determined relative location of the current sample value with respect to the virtual boundary.
In some embodiments, making a decision as to whether or not to filter the current sample value based on the determined relative location of the current sample comprises determining whether the current sample value is adjacent to (e.g., not more than one row or column away from) the virtual boundary and as a resuit of determining that the current sample value is adjacent to the virtual boundary, making a decision to refrain from filtering the current sample value.
In some embodiments, making a decision as to whether or not to filter the current sample value based on the determined relative location of the current sample with respect to the virtual boundary comprises making the decision based not only on the determined relative location of the current sample with respect to the virtual boundary but also on a characteristic of a block of sample values comprising the current sample value.
FIG. 19 is a flow chart illustrating a process 1900 for processing an image. Process 1900 may begin in step s1902.
Step s1902 comprises obtaining a set of sample values associated with an image. The set of sample values comprises a current sample value and sample values neighboring the current sample value. The obtained set of sample values is included within a first rectangular block of sample values.
Step s1904 comprises determining a relative location of the current sample value with respect to a virtual boundary. The virtual boundary is defined with respect to a block boundary between the first block of sample values and a second block of sample values. The virtual boundary is parallel with the block boundary and separated from the block boundary by at least one row or column of sample values included in the first block of sample values.
Step s1906 comprises selecting a filter strength value based on the determined relative location of the current sample value with respect to the virtual boundary.
Step s1908 comprises filtering the current sample value based on the selected filter strength value.
In some embodiments, selecting a filter strength value based on the determined relative location of the current sample with respect to the Virtual boundary comprises selecting a right shift value (e.g., N, where N is a whole number > 0) or a divisor (e.g., a value equal to 2-N) based on the determined relative location of the current sample with respect to the virtual boundary, and filtering the current sample based on the selected filter amount comprises filtering the current sample value using the selected right shift value or divisor.
In some embodiments, selecting a filter strength value based on the determined relative location of the current sample with respect to the virtual boundary comprises: selecting a first filter strength value if it is determined that the current sample value is adjacent to the virtual boundary; or selecting a second filter strength value if it is determined that the current sample value is more the one row or column away from the virtual boundary.
In some embodiments, the first filter strength value is greater than 7 and the second filter strength value is less than or equal to 7, or the first filter strength value is greater than or equal to 1 (e.g., 3) and the second filter strength value is 0 (in this embodiment the selected filter strength value function as a filter strength adjustment).
In some embodiments, selecting a filter strength value based on the determined relative location of the current sample with respect to the virtual boundary comprises: selecting a first filter strength value if it is determined that the current sample value is not more the one row or column away from the virtual boundary; or selecting a filter strength value from a set of filter strength values if it is determined that the current sample value is more the one row or column away from the virtual boundary, wherein the set of filter strength values comprises a second filter strength value (e.g., 9), a third filter strength value (e.g. 8), and a fourth filter strength value (e.g., 7).
In some embodiments, selecting a filter strength value from the set of filter strength values if it is determined that the current sample value is more the one row or column away from the virtual boundary comprises: selecting the second filter strength value if it is determined that the current sample value is two rows or columns away from the virtual boundary; selecting the third filter strength value if it is determined that the current sample value is three rows or columns away from the virtual boundary; or selecting the fourth filter strength value if it is determined that the current sample value is at least four rows or columns away from the virtual boundary.
In some embodiments, selecting a filter strength value based on the determined relative location of the current sample with respect to the virtual boundary comprises selecting the filter strength value based not only on the determined relative location of the current sample with respect to the virtual boundary but also on a characteristic of a block of sample values comprising the current sample value.
FIG. 20 is a flow chart illustrating a process 2000 for processing an image. Process 2000 may begin in step s2002.
Step s2002 comprises obtaining a set of sample values associated with an image, the set of sample values comprising a current sample value and a set of neighboring sample values neighboring the current sample value, wherein the obtained set of sample values is included within a first rectangular block of sample values, and wherein the obtained set of f
sample values is logically arranged in a plurality of rows of sample values, said plurality of rows consisting of a middle row, rows above the middle row, and rows below the middle row, the current sample value being in said middle row.
Step s2004 comprises determining a relative location of the middle row with respect to a virtual horizontal boundary, wherein the virtual horizontal boundary' is defined with respect to a horizontal block boundary between the first block of sample values and a second block of sample values, wherein the virtual horizontal boundary is parallel with the block boundary and separated from the block boundary by at least one row of sample values included in the first block of sample values.
Step s2006 comprises determining i) that the middle row is immediately above the virtual horizontal boundary (FIG. 15), ii) that that the middle row is immediately below the virtual horizontal boundary (FIG. 24), iii) that the middle row is immediately above the row that is immediately above the virtual horizontal boundary (FIG. 16), or iv) that the middle row is immediately below the row that is immediately below the virtual horizontal boundary (FIG. 25).
Step s2008 comprises applying ALF filtering of the current sample value based on samples only from: i) the middle row, ii) the row immediately below the middle row, and iii) the row immediately above the middle row if the middle row is determined to be immediately above or immediately below the virtual horizontal boundary (FIG. 15 and FIG. 24).
Step s2010 comprises applying ALF filtering of the current sample value based on samples only from: i) the middle row, ii) the two rows immediately above the middle row, and iii) the two rows immediately below the below row if the middle row is determined to be immediately above the row that is irnrnediateïy above the Virtual horizontal boundary or determined to be immediately below the row that is immediately below the virtual horizontal boundary (FIG. 16 and FIG. 25).
In some embodiments, the Virtual horizontal boundary for luma is five samples above the horizontal block boundary.
In some embodiments, the virtual horizontal boundary for chroma is three samples above the horizontal block boundary.
FIG, 21 is a block diagram of an apparatus 2101 for implementing encoder 502 or décoder 504, according to some embodiments. That is, apparatus 2101 can be adapted to perform the methods disclosed herein. in embodiments where apparatus 2101 impïements encoder 502, apparatus 2101 may be referred to as “encoding apparatus 2101,” and in embodiments where apparatus 2101 impïements décoder 504, apparatus 2101 may be referred to as a “decoding apparatus 2101.” As shown in FIG. 21, apparatus 2101 may comprise: processing circuitry (PC) 2102, which may include one or more processors (P)
2155 (e.g., one or more general purpose microprocessors and/or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gâte arrays (FPGAs), and the like), which processors may be co-iocated in a single housing or in a single data center or may be geographically distributed; one or more network interfaces 2148 (which may be co-iocated or geographically distributed) where each network interface includes a transmitter (Tx) 2145 and a receiver (Rx) 2147 for enabling apparatus 2101 to transmit data to and receive data from other nodes connected to network 110 (e.g., an Internet Protocol (IP) network) to which network interface 2148 is connected; and one or more storage units (a.k.a., “data storage Systems”) 2108 which may be colocated or geographically distributed and which may include one or more non-volatile storage devices and/or one or more volatile storage devices. In embodiments where PC 2102 includes a programmable processor, a computer program product (CPP) 2141 may be provided. CPP 2141 includes a computer readabie medium (CRM) 2142 storing a computer program (CP) 2143 comprising computer readabie instructions (CRI) 2144. CRM 2142 may be a non-transitory computer readabie medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 2144 of computer program 2143 is adapted such that when executed by PC 2102, the CRI causes apparatus 2101 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, apparatus 2101 may be adapted to perform steps described herein without the need for code. That is, for example, PC 2102 may consist merely of one ormore ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and/or software.
The embodiments disclosed herein avoid introducing artifacts from the filtering by reducing the amount of filtering for ALF at Virtual CTU boundaries and boundaries adjacent to the Virtual CTU boundaries, Below are objective results showing realized improvements.
RESULTS
Objective results (BDR where négative number is improvement) for embodiment 1 in comparison to VTM-7.0 (VVC reference software) as follows:
Ail Intra
| Y | U | V | |
| i Class A1 | 0,01% | 0,07% | 0,11% |
f
| Class A2 | 0,01% | 0,04% | 0,05% |
| Class B | 0,01% | 0,03% | 0,04% |
| Class C | -0,01% | 0,04% | 0,04% |
| Class E | 0,01% | 0,08% | 0,09% |
| Overail | 0,01% | 0,05% | 0,06% |
| Class D | 0,00% | 0,02% | 0,03% |
| Class F | 0,00% | 0,04% | 0,04% |
| Random | |||
| access | |||
| Y | U | V | |
| Class A1 | 0,02% | 0,06% | 0,03% |
| Class A2 | 0,02% | 0,04% | 0,14% |
| Class B | 0,03% | 0,09% | 0,12% |
| Class C | -0,04% | 0,00% | -0,03% |
| Class E | |||
| Overail | 0,01% | 0,05% | 0,06% |
| Class D | 0,00% | 0,11% | 0,16% |
| Class F | #VALUE! | #VALUE! | #VALUE! |
Low delay B
| Y | U | V | |
| Class A1 | |||
| Class A2 | |||
| Class B | 0,03% | -0,21% | 0,02% |
| Class C | -0,04% | -0,28% | -0,07% |
| Class E | -0,11% | 0,16% | -0,08% |
| Overail | -0,03% | -0,14% | -0,03% |
| Class D | -0,04% | 0,45% | 0,00% |
| Class F | 0,00% | -0,52% | 0,16% |
Objective results for embodiment 2 in comparison to VTM-7.0 (WC reference software at the time of the invention) as follows:
Ail intra
| Y | U | V | |
| Class Al | 0,01% | 0,04% | 0,07% |
| Class A2 | 0,01% | 0,02% | 0,03% |
| Ciass B | 0,01% | 0,02% | 0,03% |
| Class C | -0,01% | 0,03% | 0,02% |
| Class E | 0,01% | 0,05% | 0,06% |
| Overall | 0,00% | 0,03% | 0,04% |
| Class D | 0,00% | 0,01% | 0,02% |
| Class F | -0,01% | 0,03% | 0,02% |
| Random | |||
| access | Y | U | V |
| Class A1 | 0,02% | -0,01% | 0,03% |
| Class A2 | 0,00% | 0,10% | 0,09% |
| Class B | 0,01% | 0,13% | 0,11% |
| Class C | -0,01% | -0,03% | -0,06% |
| Class E | |||
| Overall | 0,00% | 0,05% | 0,04% |
| Class D | 0,00% | 0,07% | 0,06% |
| Class F | -0,04% | -0,01% | -0,01% |
| 1 ahi ID ^mvv MGsay i—f | Y | U | V |
| Class A1 | |||
| Class A2 | |||
| Class B | 0,01% | -0,24% | 0,32% |
| Class C | 0,04% | -0,15% | 0,02% |
| Class E | 0,01% | 0,72% | -0,03% |
| Overall | 0,02% | 0,03% | 0,13% |
| Class D | 0,00% | -0,18% | -0,15% |
| Class F | 0,05% | 0,07% | 0,24% |
FIG. 22 shows a first image processed the conventional way (left image) and a second image (bottom image) processed according to embodiment 1. The first image has an artifact (blurring of line) at a virtual boundary due to ALF. The second image does not include the artifact. Thus the second image is a higher quality version of the original image than the first image.
FIG. 23 shows a first image processed the conventional way (left image) and a second image (bottom image) processed according to embodiment 2. The first image has an artifact (blurring of line) at a virtual boundary due to ALF. The second image does not include the artifact. Thus the second image is a higher quality version of the original image than the first image.
While various embodiments are described herein, it shouki be understood that they hâve been presented by way of exampte only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described éléments in al! possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise cleariy contradicted by context
Additionally, white the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done sotely for the sake of illustration. Accordingly, it is contemplated that some steps may be added. some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.
REFERENCES
[1] Misra, et al., “Cross-Corr^onent Adaptive Loop Filter for chroma,” JVET-O0636_r1,
J u ly 2019 (available at htteffphenix.itsudparis.eu/ivet/doc end user/documents/15 Gothenburq/wg11/JVET-O0636-v2.zip)
[2] Chen, “Description of Core Experiment 5 (CE5): Cross-component adaptive loop filtering,” JVET-P2025-v4, October2019 (avafeble at http:ffphenix.it- sudparis.eu/jvet/doc end user/documents/16 Geneva/wq11/JVET-P2025-v4.zip)
Claims (12)
1. A method (1900) for encoding or decoding an image of a video sequence, the method comprising:
obtaining a set of sample values associated with the image, the set of sample values comprising a current sample value and sample values neighboring the current sample value, wherein the obtained set of sample values is included within a first rectangular block of sample values;
determining a relative location of the current sample value with respect to a Virtual boundary, wherein the Virtual boundary is defined with respect to a block boundary between the first block of sample values and a second block of sample values, wherein the Virtual boundary is parallel with the block boundary and separated from the block boundary by at least one row or column of sample values included in the first block of sample values;
selecting a filter strength value based on the determined relative location of the current sample with respect to the Virtual boundary; and filtering the current sample based on the selected filter strength value.
2. The method of claim 1, wherein selecting a filter strength value based on the determined relative location of the current sample with respect to the Virtual boundary comprises selecting a right shift value or a divisor based on the determined relative location of the current sample with respect to the Virtual boundary, and filtering the current sample based on the selected filter amount comprises filtering the current sample value using the selected right shift value or divisor.
3. The method of claims 1 or 2, wherein selecting a filter strength value based on the determined relative location of the current sample with respect to the Virtual boundary comprises:
selecting a first filter strength value if it is determined that the current sample value is adjacent to the Virtual boundary; or selecting a second filter strength value if it is determined that the current sample value is more the one row or column away from the Virtual boundary.
4. The method of claim 3, wherein the first filter strength value is greater than 7 and the second filter strength value is less than or equal to 7, or the first filter strength value is greater than or equal to 1 and the second filter strength value is 0.
5. The method of daims 1 or 2, wherein selecting a filter strength value based on the determined relative location of the current sample with respect to the virtual boundary comprises:
selecting a first filter strength value if it is determined that the current sample value is not more the one row or column away from the virtual boundary; or selecting a filter strength value from a set of filter strength values if it is determined that the current sample value is more the one row or column away from the virtual boundary, wherein the set of filter strength values comprises a second filter strength value, a third filter strength value, and a fourth filter strength value.
6. A method (1800) for encoding or decoding an image of a video sequence, the method comprising:
obtaining a set of sample values associated with an image, the set of sample values comprising a current sample value and a set of neighboring sample values neighboring the current sample value, wherein the obtained set of sample values is included within a first rectangular block of sample values;
determining a relative location ofthe current sample value with respect to a virtual boundary, wherein the virtual boundary is defined with respect to a block boundary between the first block of sample values and a second block of sample values, wherein the virtual boundary is parallel with the block boundary and separated from the block boundary by at least one row or column of sample values included in the first block of sample values; and making a decision as to whether or not to filter the current sample value based on the determined relative location of the current sample with respect to the virtual boundary.
7. The method of claim 6, wherein making a decision as to whether or not to filter the current sample value based on the determined relative location of the current sample comprises:
determining whether the current sample value is adjacent to the virtual boundary; and as a resuit of determining that the current sample value is adjacent to the virtual boundary, making a decision to refrain from filtering the current sample value.
8. The method of any one of claims 6-7, wherein making a decision as to whether or not to filter the current sample value based on the determined relative location of the current sample with respect to the Virtual boundary comprises making the decision based not only on the determined relative location of the current sample with respect to the
5 Virtual boundary but also on a characteristic of a block of sample values comprising the current sample value.
9. The method of any one of claims 6-8, wherein the decision to not filter the current sample value when it is adjacent to the Virtual boundary applies to cross-
10 component adaptive loop filter (CC-ALF).
10. A carrier containing a computer program (2143), wherein the computer program comprises instructions (2144) which when executed by processing circuitry (2102) causes the processing circuitry (2102) to perform the method of any one of the
15 above claims, and wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (2142).
11. An apparatus (2101), the apparatus being adapted to perform the method of any one of claims 1-9.
12. An apparatus (2101), the apparatus comprising:
processing circuitry (2102); and a memory (2142), said memory containing instructions (2144) exécutable by said processing circuitry, whereby said apparatus is operative to perform the method of any 25 one of the claims 1-9.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US62/953,310 | 2019-12-24 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| OA20185A true OA20185A (en) | 2021-12-30 |
Family
ID=
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12101484B2 (en) | Virtual boundary processing for adaptive loop filtering | |
| JP2019534629A (en) | Image processing method and apparatus therefor | |
| US11197032B2 (en) | Asymmetric deblocking in a video encoder and/or video decoder | |
| WO2018067722A1 (en) | Peak sample adaptive offset | |
| US20220167020A1 (en) | Deblocking of transform sub-block boundaries | |
| JP7450604B2 (en) | Deblocking filter for video coding and processing | |
| KR20220029742A (en) | Loop Filter Block Flexible Partitioning | |
| KR102632984B1 (en) | Image processing device and image processing method | |
| OA20185A (en) | Virtual boundary processing for adaptive loop filtering. | |
| US20210289202A1 (en) | Intra prediction method and apparatus for performing adaptive filtering on reference pixel | |
| US11381845B2 (en) | Deblocking between block boundaries and sub-block boundaries in a video encoder and/or video decoder | |
| KR102721536B1 (en) | Adaptive loop filtering between different video units | |
| WO2021198456A1 (en) | Adaptive loop filter (alf) virtual boundary processing | |
| KR102718068B1 (en) | Padding process in adaptive loop filtering | |
| US20230254483A1 (en) | Intra prediction method and apparatus for performing adaptive filtering on reference pixel | |
| US20220141457A1 (en) | Deblocking using sub-pel motion vector threshold | |
| GB2574425A (en) | Video coding and decoding |