WO2021045188A1 - 画像処理装置および方法 - Google Patents
画像処理装置および方法 Download PDFInfo
- Publication number
- WO2021045188A1 WO2021045188A1 PCT/JP2020/033573 JP2020033573W WO2021045188A1 WO 2021045188 A1 WO2021045188 A1 WO 2021045188A1 JP 2020033573 W JP2020033573 W JP 2020033573W WO 2021045188 A1 WO2021045188 A1 WO 2021045188A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- conversion
- quantization
- coefficient
- inverse
- unit
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/124—Quantisation
- H04N19/126—Details of normalisation or weighting functions, e.g. normalisation matrices or variable uniform quantisers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/157—Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
- H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/48—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/507—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction using conditional replenishment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/80—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
- H04N19/82—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation involving filtering within a prediction loop
Definitions
- the present disclosure relates to an image processing device and a method, and more particularly to an image processing device and a method capable of suppressing information loss due to coding and decoding.
- Non-Patent Document 1 a coding method has been proposed in which a predicted residual of a moving image is derived, coefficient-converted, quantized and encoded (for example, Non-Patent Document 1). Further, in the image coding, lossless coding has been proposed in which the coefficient conversion, quantization, etc. are skipped (omitted) by using the conversion quantization bypass, and the predicted residual is losslessly coded (for example, non-patent documents). 2).
- conversion skip + QP4 approach such an approach is called the "conversion skip + QP4" approach.
- VTM-5.0 in https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM Tsung-Chuan Ma, Yi-Wen Chen, Xiaoyu Xiu, Xianglin Wang, Tangi Poirier, Fabrice Le Leannec, Karam Naser, Edouard Francois, Hyeongmun Jang, Junghak Nam, Naeri Park, for VVC ", JVET-O1061, m49678, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019
- This disclosure has been made in view of such a situation, and makes it possible to suppress the loss of information due to coding and decoding.
- the image processing apparatus of one aspect of the present technology uses an inverse quantization unit that performs inverse quantization with respect to the quantization coefficient by using a quantization parameter according to whether conversion skip is applied, and when the conversion skip is not applied, When performing inverse coefficient conversion on the conversion coefficient generated by the inverse quantization by the inverse quantization unit to generate a predicted residual which is a residual between the image and the predicted image of the image, and applying the conversion skip.
- An image processing apparatus including an inverse conversion unit that skips the inverse coefficient conversion.
- inverse quantization is performed on the quantization coefficient using the quantization parameter according to whether the conversion skip is applied, and when the conversion skip is not applied, the inverse quantization is performed.
- the image processing apparatus of the other aspect of the present technology performs coefficient conversion for the predicted residual which is the residual of the image and the predicted image of the image, generates a conversion coefficient, and skips the conversion. Is applied, in an image processing apparatus including a conversion unit that skips the coefficient conversion and a quantization unit that performs quantization with respect to the conversion coefficient using a quantization parameter according to whether or not the conversion skip is applied. is there.
- the image processing method of another aspect of the present technology performs coefficient conversion for the predicted residual, which is the residual between the image and the predicted image of the image, generates a conversion coefficient, and skips the conversion. Is an image processing method that skips the coefficient conversion and performs quantization with respect to the conversion coefficient by using the quantization parameter according to whether or not the conversion skip is applied.
- the image processing apparatus of the other aspect of the present technology uses the inverse quantization unit that performs inverse quantization with respect to the quantization coefficient and the scaling parameter according to whether or not the conversion skip is applied, and the inverse quantization unit is used.
- An inverse quantization normalization unit that normalizes the conversion coefficient generated by the inverse quantization, and an inverse coefficient conversion for the conversion coefficient normalized by the inverse quantization normalization unit when the conversion skip is not applied. Is performed, a predicted residual which is a residual between the image and the predicted image of the image is generated, and when the conversion skip is applied, the image processing apparatus includes an inverse conversion unit that skips the inverse coefficient conversion.
- the conversion coefficient generated by the inverse quantization is normalized by performing the inverse quantization with respect to the quantization coefficient and using the scaling parameter according to whether the conversion skip is applied.
- the conversion skip is not applied, the inverse coefficient conversion is performed on the normalized conversion coefficient to generate a predicted residual which is a residual between the image and the predicted image of the image, and the conversion skip is performed.
- it is an image processing method that skips the inverse coefficient conversion.
- the image processing apparatus of the other aspect of the present technology performs coefficient conversion on the predicted residual which is the residual between the image and the predicted image of the image, and generates a conversion coefficient.
- the conversion unit which skips the coefficient conversion, and when the conversion skip is not applied the conversion coefficient is quantized, and when the conversion skip is applied, the quantum with respect to the predicted residual.
- the coefficient conversion when the conversion skip is not applied, the coefficient conversion is performed on the predicted residual which is the residual between the image and the predicted image of the image, and the conversion coefficient is generated.
- the coefficient conversion is skipped, when the conversion skip is not applied, quantization is performed on the conversion coefficient, and when the conversion skip is applied, the quantization is performed on the predicted residual.
- the inverse quantization with respect to the quantization coefficient is performed using the quantization parameter according to whether the conversion skip is applied, and when the conversion skip is not applied, the inverse quantization is performed.
- Inverse coefficient conversion is performed on the conversion coefficient generated by inverse quantization to generate a predicted residual that is the residual between the image and the predicted image of the image, and when the conversion skip is applied, the inverse coefficient conversion is skipped. Will be done.
- inverse quantization is performed on the quantization coefficient, and it is generated by the inverse quantization using the scaling parameter depending on whether the conversion skip is applied. If the conversion coefficients are normalized and no conversion skip is applied, an inverse coefficient conversion is performed on the normalized conversion coefficients to generate a predicted residual, which is the residual between the image and the predicted image of that image. When applying conversion skip, inverse coefficient conversion is skipped.
- coefficient conversion is performed on the predicted residual which is the residual between the image and the predicted image of the image, and the conversion coefficient is converted. Is generated and if the conversion skip is applied, the coefficient conversion is skipped, if the conversion skip is not applied, quantization is performed on the conversion coefficient, and if the conversion skip is applied, on the predicted residual. Quantization is performed and the quantization coefficients generated by the quantization are normalized using scaling parameters depending on whether the transformation skip is applied.
- Inverse Quantization Based on Transformation Skip It is a figure explaining an example of the control method of inverse quantization and transformation quantization. It is a block diagram which shows the main configuration example of the inverse quantization inverse conversion apparatus. It is a flowchart which shows the example of the flow of the reverse quantization reverse conversion process. It is a figure explaining the example of the quantization parameter control based on conversion skip. It is a figure explaining the example of the rounding offset control based on conversion skip. It is a figure explaining the example of the rounding offset control based on conversion skip. It is a block diagram which shows the main block diagram of the conversion quantization apparatus. It is a flowchart which shows the example of the flow of the conversion quantization processing.
- Non-Patent Document 1 (above)
- Non-Patent Document 2 (above)
- Non-Patent Document 3 Benjamin Bross, Jianle Chen, Shan Liu, "Versatile Video Coding (Draft 5)", N1001-v10, m48053, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 14th Meeting: Geneva, CH, 19-27 Mar.
- Non-Patent Document 4 Jianle Chen, Yan Ye, Seung Hwan Kim, "Algorithm description for Versatile Video Coding and Test Model 5 (VTM 5)", JVET-N1002-v2, m48054, Joint Video Experts Team (JVET) of ITU- T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 14th Meeting: Geneva, CH, 19-27 Mar.
- Non-Patent Document 5 Benjamin Bross, Jianle Chen, Shan Liu, "Versatile Video Coding (Draft 6)", JVET-O2001-vE, m49908, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019
- Non-Patent Document 6 Jianle Chen, Yan Ye, Seung Hwan Kim, "Algorithm description for Versatile Video Coding and Test Model 6 (VTM 6)", JVET-O2002-v2, m49914, Joint Video Experts Team (JVET) of ITU- T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29/WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019
- Non-Patent Document 7 Tsuung-Chuan Ma, Yi-Wen Chen, Xiaoyu Xiu, Xiangl
- the contents described in the above-mentioned non-patent documents are also the basis for determining the support requirements.
- Quad-Tree Block Structure and QTBT (Quad Tree Plus Binary Tree) Block Structure described in the above-mentioned non-patent documents are not directly described in the examples, they are within the disclosure range of the present technology. It shall meet the support requirements of the claims.
- technical terms such as Parsing, Syntax, and Semantics are within the scope of disclosure of the present technology even if they are not directly described in the examples. Shall meet the support requirements in the range of.
- a "block” (not a block indicating a processing unit) used in the description as a partial area of an image (picture) or a processing unit indicates an arbitrary partial area in the picture unless otherwise specified. Its size, shape, characteristics, etc. are not limited.
- “block” includes TB (Transform Block), TU (Transform Unit), PB (Prediction Block), PU (Prediction Unit), SCU (Smallest Coding Unit), and CU described in the above-mentioned non-patent documents.
- CodingUnit LCU (LargestCodingUnit), CTB (CodingTreeBlock), CTU (CodingTreeUnit), subblock, macroblock, tile, slice, etc., including any partial area (processing unit) And.
- the block size may be specified using the identification information that identifies the size.
- the block size may be specified by the ratio or difference with the size of the reference block (for example, LCU, SCU, etc.).
- the designation of the block size also includes the designation of the range of the block size (for example, the designation of the range of the allowable block size).
- coding includes not only the entire process of converting an image into a bitstream but also a part of the process. For example, it not only includes processing that includes prediction processing, orthogonal transformation, quantization, arithmetic coding, etc., but also includes processing that collectively refers to quantization and arithmetic coding, prediction processing, quantization, and arithmetic coding. Including processing, etc.
- decoding includes not only the entire process of converting a bitstream into an image, but also some processes.
- processing not only includes processing that includes inverse arithmetic decoding, inverse quantization, inverse orthogonal transformation, prediction processing, etc., but also processing that includes inverse arithmetic decoding and inverse quantization, inverse arithmetic decoding, inverse quantization, and prediction processing. Including processing that includes and.
- Non-Patent Document 2 is a coding method in which in the image coding of Non-Patent Document 1, the conversion quantization bypass is used to skip (omit) coefficient conversion, quantization, etc., and the predicted residual is reversibly coded. A lossless coding is disclosed.
- conversion skip + QP4 approach such an approach is called the "conversion skip + QP4" approach.
- the inverse quantization inverse conversion process is a process in which the quantization coefficient is inversely quantized to generate a conversion coefficient, and the conversion coefficient is inversely converted to generate a predicted residual.
- the predicted residual is the residual between the image and the predicted image of the image.
- the predicted image is generated based on the temporally peripheral information of the image, the spatially peripheral information of the image, or both.
- the information on the surroundings in terms of time is information in a frame different from that of the image.
- Spatial peripheral information is information in the same frame as the image.
- the inverse coefficient conversion is an inverse process of coefficient conversion, which is a process of generating a conversion coefficient by converting a predicted residual into a coefficient.
- Inverse quantization is the inverse process of quantization, which is the process of quantizing the conversion coefficient generated by the coefficient conversion to generate the quantization coefficient. That is, the inverse quantization inverse conversion process is the inverse process of the transformation quantization that performs the coefficient conversion and the quantization. Therefore, the inverse quantization inverse conversion process can generate the predicted residual by inverse quantization and inverse conversion of the quantization coefficient generated by transforming and quantizing the predicted residual.
- FIG. 2 is a block diagram showing an example of the configuration of an inverse quantization inverse conversion apparatus, which is an aspect of an image processing apparatus to which the present technology is applied.
- the inverse quantization inverse converter 100 shown in FIG. 2 is an apparatus that inversely quantizes and inversely transforms the quantization coefficient to generate a predicted residual.
- FIG. 2 shows the main things such as the processing unit and the data flow, and not all of them are shown in FIG. That is, in the inverse quantization inverse conversion device 100, there may be a processing unit that is not shown as a block in FIG. 2, or there may be a processing or data flow that is not shown as an arrow or the like in FIG. ..
- the inverse quantization inverse conversion device 100 has an inverse quantization unit 101 and an inverse conversion unit 102.
- the dequantization unit 101 performs processing related to dequantization.
- the inverse conversion unit 102 performs processing related to the inverse coefficient conversion.
- the inverse quantization unit 101 has an inverse quantization processing unit 111 and a normalization unit 112.
- the inverse quantization processing unit 111 acquires the quantization coefficient input to the inverse quantization inverse conversion device 100.
- the inverse quantization processing unit 11 performs inverse quantization on the acquired quantization coefficient and generates a conversion coefficient.
- the inverse quantization processing unit 111 supplies the generated conversion coefficient to the normalization unit 112.
- the normalization unit 112 acquires the conversion coefficient supplied from the inverse quantization processing unit 111.
- the normalization unit 112 normalizes the acquired conversion coefficient.
- the normalization unit 112 supplies the normalized conversion coefficient to the inverse conversion unit 102.
- the inverse conversion unit 102 includes a selection unit 121, an inverse conversion processing unit 122, a selection unit 123, a normalization unit 124, and an expansion processing unit 125.
- the selection unit 121 and the selection unit 123 select whether or not to apply conversion skip, which is a mode for skipping inverse coefficient conversion.
- the conversion skip is a mode in which the coefficient conversion process is skipped in the coefficient conversion process, and is a mode in which the inverse coefficient conversion process is skipped in the inverse coefficient conversion process.
- the selection unit 121 acquires the conversion coefficient supplied from the inverse quantization unit 101. When the conversion skip is not applied, that is, when the inverse coefficient conversion process is performed, the selection unit 121 supplies the acquired conversion coefficient to the inverse conversion processing unit 122. Further, the selection unit 123 acquires the prediction residual supplied from the inverse conversion processing unit 122 and supplies it to the normalization unit 124.
- the selection unit 121 When applying the conversion skip, that is, when skipping the inverse coefficient conversion process, the selection unit 121 supplies the acquired conversion coefficient to the decompression processing unit 125. In this case, since the coefficient conversion process is skipped, this conversion coefficient is the predicted residual. Further, the selection unit 123 acquires the conversion coefficient (that is, the predicted residual) supplied from the decompression processing unit 125 and supplies it to the normalization unit 124.
- the inverse conversion processing unit 122 acquires the conversion coefficient supplied from the inverse quantization unit 101 via the selection unit 121. This conversion coefficient is normalized by the normalization unit 112. The inverse conversion processing unit 122 performs inverse coefficient conversion on the acquired conversion coefficient to generate a predicted residual. The inverse conversion processing unit 122 supplies the generated predicted residual to the normalization unit 124 via the selection unit 123.
- the decompression processing unit 125 acquires the conversion coefficient supplied from the inverse quantization unit 101 via the selection unit 121. This conversion coefficient is normalized by the normalization unit 112. Moreover, this conversion coefficient is a predicted residual. The stretching processing unit 125 stretches the acquired conversion coefficient (that is, the predicted residual). The decompression processing unit 125 supplies the decompressed conversion coefficient (that is, the predicted residual) to the normalization unit 124 via the selection unit 123.
- the normalization unit 124 acquires the predicted residuals supplied from the inverse conversion processing unit 122 via the selection unit 123 when the conversion skip is not applied, that is, when the inverse coefficient conversion processing is performed. Further, when applying the conversion skip, that is, when skipping the inverse coefficient conversion process, the normalization unit 124 transmits the conversion coefficient (that is, the predicted residual) supplied from the decompression processing unit 125 via the selection unit 123. get. The normalization unit 124 normalizes the acquired predicted residuals. The normalization unit 124 outputs the normalized predicted residual to the outside of the inverse quantization inverse converter 100.
- the above-mentioned (method 1-1) is applied. That is, the inverse quantization inverse conversion device 100 sets the parameters related to the inverse quantization inverse conversion so as to suppress the loss of information.
- the dequantization processing unit 111 can perform dequantization using the parameters set in this way.
- the normalization unit 112 can perform normalization using the parameters set in this way.
- the normalization unit 124 can perform normalization using the parameters set in this way.
- the inverse quantization inverse conversion device 100 can suppress the loss of information due to decoding to which the inverse quantization inverse conversion process is applied. That is, the inverse quantization inverse converter 100 can bring the decoding by the "conversion skip + QP4" approach closer to the lossless decoding (lossless coding).
- the inverse quantization unit 101 sets the parameters related to the inverse quantization in step S101.
- step S102 the inverse quantization processing unit 111 performs inverse quantization on the quantization coefficient using the parameters set in step S101, and generates a conversion coefficient.
- step S103 the normalization unit 112 normalizes the conversion coefficient generated in step S102 using the parameters set in step S101.
- step S104 the selection unit 121 and the selection unit 123 determine whether or not to apply the conversion skip. If it is determined that the conversion skip is not applied, the process proceeds to step S105.
- step S105 the inverse conversion unit 102 sets a parameter related to the inverse coefficient conversion.
- step S106 the inverse conversion processing unit 122 performs inverse coefficient conversion on the conversion coefficient normalized in step S103 using the parameters set in step S105 to generate a predicted residual.
- step S108 the process proceeds to step S108.
- step S104 If it is determined in step S104 that the conversion skip is applied, the process proceeds to step S107.
- step S107 the decompression processing unit 125 decompresses the conversion coefficient (that is, the predicted residual) normalized in step S103.
- step S108 the process proceeds to step S108.
- step S108 the normalization unit 124 normalizes the predicted residual generated in step S106 or the conversion coefficient extended in step S107 (that is, the predicted residual) using the parameters set in step S105. To do.
- step S108 When the process of step S108 is completed, the inverse quantization inverse conversion process is completed.
- step S101 the inverse quantization unit 101 sets the parameters related to the inverse quantization so as to suppress the loss of information.
- the inverse quantization processing unit 111 can perform inverse quantization using the parameters set in this way.
- the normalization unit 112 can perform normalization using the parameters set in this way.
- step S105 the inverse conversion unit 102 sets a parameter related to the inverse coefficient conversion so as to suppress the loss of information.
- step S108 the normalization unit 124 can perform normalization using the parameters set in this way.
- the inverse quantization inverse conversion device 100 can suppress the loss of information due to decoding to which the inverse quantization inverse conversion process is applied. That is, the inverse quantization inverse converter 100 can bring the decoding by the "conversion skip + QP4" approach closer to the lossless decoding (lossless coding).
- ⁇ 1-1-1 Control of quantization parameters>
- the parameters related to inverse quantization may be controlled.
- inverse quantization with respect to the quantization coefficient may be performed using a quantization parameter depending on whether conversion skip is applied.
- the image processing apparatus may be provided with an inverse quantization unit that performs inverse quantization with respect to the quantization coefficient by using the quantization parameter according to whether or not the conversion skip is applied.
- the image processing apparatus can suppress the loss of information due to decoding to which the dequantization by the dequantization unit is applied. That is, the image processing apparatus can bring the decoding by the "conversion skip + QP4" approach closer to the reversible decoding (lossless coding).
- the quantization parameter is corrected and the inverse quantization is performed using the corrected quantization parameter. You may do it. That is, as shown in the second row from the top of the table in FIG. 1, the correction of the quantization parameter (QP) of the dependent quantization may be controlled based on the conversion skip (method 1-1-1). ..
- dependent quantization DQ (Dependent Quantization)
- QP quantization parameter
- the quantization step size for a certain quantization parameter is transformed (branched) according to the previous state.
- a of FIG. 4 is the semantics related to such dependent quantization described in Non-Patent Document 5.
- dependent quantization can be applied to inverse quantization.
- inverse quantization the inverse processing of dependent quantization (also called inverse dependent quantization) is applied. Therefore, in inverse quantization, if dependent quantization is applied, the quantization parameters are similarly corrected.
- the quantization step size should be set to 1 by correcting the quantization parameter by such dependent quantization (QP + 1). Can't be done. Therefore, there is a risk of information loss due to dequantization.
- the condition of the above equation (8-959) is not "and the value (IS_SKIP) indicating that the transformation skip flag (transform_skip_flag) applies the transformation skip". If "is added.
- the transform skip flag (transform_skip_flag) is flag information indicating whether or not to apply the transform skip. For example, if the conversion skip flag is true (for example, 1), it indicates that conversion skip is applied. If the conversion skip flag is false (for example, 0), it indicates that the conversion skip is not applied.
- “or when the transform skip flag (transform_skip_flag) is true (IS_SKIP) is added to the condition of the above equation (8-960).
- the inverse quantization processing unit 111 derives the scaling parameter ls according to such semantics. By doing so, when the conversion skip is applied, the equation (8-960) is applied, and the correction of the quantization parameter by the dependent quantization (inverse dependent quantization) is omitted. Therefore, the inverse quantization processing unit 111 can suppress the loss of information due to the inverse dependent quantization when the conversion skip is applied.
- the parameters related to the inverse quantization to be controlled may be other than the quantization parameters. For example, as shown in the third row from the top of the table in FIG. 1, for example, even if the sign of the rounding offset of the normalization process in the inverse quantization is set based on the sign of the conversion coefficient which is the input value. Good (Method 1-1-2).
- the image processing apparatus may further include an inverse quantization normalization unit that sets the sign of the rounding offset based on the sign of the conversion coefficient and normalizes the conversion coefficient using the rounding offset. .. Then, when the inverse conversion unit does not apply the conversion skip, the inverse coefficient conversion may be performed on the conversion coefficient normalized by the inverse quantization normalization unit.
- a of FIG. 5 is a semantics related to the normalization process (normalization process by the normalization unit 112) in the inverse quantization described in Non-Patent Document 5.
- normalization by logical bit shift is performed using the rounding offset bdOffset1 and the scaling parameter bdShift1 (Equation (8-963)).
- the rounding offset bdOffset1 is derived by the following equation (A) using the scaling parameter bdShift1.
- BdOffset1 (1 ⁇ (bdShift1-1)) ⁇ ⁇ ⁇ (A)
- BdOffset1 sign (dz [x] [y]) * (1 ⁇ (bdShift1-1)) ⁇ ⁇ ⁇ (B)
- the inverse quantization unit 101 derives the rounding offset bdOffset1 as in the equation (B), and the normalization unit 112 sets the rounding offset bdOffset1 and the scaling parameter bdShift1. Is used for normalization.
- the normalization unit 112 can perform the normalization process by the arithmetic bit shift, so that the bias of the sign (positive or negative) of the rounding error can be reduced and the loss of information in the inverse quantization can be suppressed. Can be done.
- ⁇ 1-1-3 Sign control of rounding offset> Parameters related to the inverse transformation may be controlled. For example, as shown in the fourth column from the top of the table in FIG. 1, for example, the sign of the rounding offset of the normalization process in the inverse coefficient conversion is set based on the sign of the predicted residual which is the input value. It may be good (method 1-1-3).
- the sign of the rounding offset is set based on the sign of the predicted residual, the predicted residual is normalized using the rounding offset, and the conversion skip is applied.
- an inverse conversion normalization unit that sets the sign of the rounding offset based on the sign of the conversion coefficient (that is, the predicted residual) and normalizes the conversion coefficient (that is, the predicted residual) using the rounding offset is further added. You may be prepared.
- a of FIG. 6 is a semantics related to the normalization process (normalization process by the normalization unit 124) in the inverse coefficient conversion described in Non-Patent Document 5.
- normalization by logical bit shift is performed using the rounding offset bdOffset2 and the scaling parameter bdShift2 (Equation (8-946)).
- the rounding offset bdOffset2 is derived by the following equation (C) using the scaling parameter bdShift2.
- BdOffset2 (1 ⁇ (bdShift2-1)) ⁇ ⁇ ⁇ (C)
- the normalization process is a logical bit shift in this way, the sign (positive or negative) of the rounding error (also called a dead zone) is biased. Therefore, the inverse coefficient conversion may be an irreversible process. That is, there is a possibility that information may be lost in the inverse coefficient conversion.
- the sign of the rounding offset of the normalization process in the inverse coefficient conversion is set based on the sign of the predicted residual which is the input value. That is, as shown in the following equation (D), the sign sign (r [x] [y]) of the predicted residual r [x] [y] is further multiplied.
- BdOffset2 sign (r [x] [y]) * (1 ⁇ (bdShift2-1)) ⁇ ⁇ ⁇ (D)
- the inverse conversion unit 102 derives the rounding offset bdOffset2 as in the equation (D), and the normalization unit 124 sets the rounding offset bdOffset2 and the scaling parameter bdShift2. Use to perform normalization.
- the rounding offset bdOffset2 derived by the equation (C) by the inverse transformant 102 multiplying the sign sign (r [x] [y]) of the predicted residual r [x] [y].
- the normalization unit 124 performs normalization by using the corrected rounding offset bdOffset2'and the scaling parameter bdShift2.
- the normalization unit 124 can perform the normalization process by the arithmetic bit shift, so that the bias of the sign (positive or negative) of the rounding error can be reduced and the loss of information in the inverse coefficient conversion can be suppressed. Can be done.
- ⁇ 1-1-4 Combination>
- ⁇ 1-1-1 Control of quantization parameter>
- ⁇ 1-1-2 Sign control of rounding offset>
- ⁇ 1-1-3 Sign control of rounding offset>.
- the methods can be applied in combination as appropriate.
- the inverse quantization inverse conversion device 100 can further suppress the loss of information in the inverse quantization inverse conversion process. That is, the inverse quantization inverse converter 100 can bring the decoding by the "conversion skip + QP4" approach closer to the reversible decoding (lossless coding) by combining more of the above methods.
- the conversion quantization process by controlling the parameters so as to suppress the loss of information, it is possible to suppress the loss of information due to the coding to which this conversion quantization process is applied.
- the coding by the "conversion skip + QP4" approach can be made closer to the lossless coding.
- FIG. 7 is a block diagram showing an example of the configuration of a conversion quantization device, which is an aspect of an image processing device to which the present technology is applied.
- the conversion quantization device 200 shown in FIG. 7 is a device that converts and quantizes the predicted residuals to generate a quantization coefficient.
- FIG. 7 shows the main things such as the processing unit and the data flow, and not all of them are shown in FIG. 7. That is, in the conversion quantization device 200, there may be a processing unit that is not shown as a block in FIG. 7, or there may be a processing or data flow that is not shown as an arrow or the like in FIG.
- the conversion quantization device 200 has a conversion unit 201 and a quantization unit 202.
- the conversion unit 201 performs processing related to coefficient conversion.
- the quantization unit 202 performs processing related to quantization.
- the conversion unit 201 includes a selection unit 211, a conversion processing unit 212, a selection unit 213, a normalization unit 214, and an expansion processing unit 215.
- the selection unit 211 and the selection unit 213 select whether or not to apply the conversion skip.
- the selection unit 211 acquires the predicted residual input to the conversion quantization device 200.
- the selection unit 211 supplies the acquired predicted residuals to the conversion processing unit 212.
- the selection unit 213 acquires the conversion coefficient supplied from the conversion processing unit 212 and supplies it to the normalization unit 214.
- the selection unit 211 When applying the conversion skip, that is, when skipping the coefficient conversion process, the selection unit 211 supplies the acquired predicted residual to the expansion processing unit 215.
- the selection unit 213 acquires the predicted residual (extended predicted residual) supplied from the decompression processing unit 215 and supplies it to the normalization unit 214.
- the conversion processing unit 212 acquires the predicted residual input to the conversion quantization device 200 via the selection unit 211.
- the conversion processing unit 212 performs coefficient conversion on the acquired predicted residual to generate a conversion coefficient.
- the conversion processing unit 212 supplies the generated conversion coefficient to the normalization unit 214 via the selection unit 213.
- the decompression processing unit 215 acquires the predicted residual input to the conversion quantization device 200 via the selection unit 211.
- the stretching processing unit 215 stretches the acquired predicted residuals.
- the decompression processing unit 215 supplies the decompressed predicted residual to the normalization unit 214 via the selection unit 213.
- the normalization unit 214 acquires the conversion coefficient supplied from the conversion processing unit 212 via the selection unit 213. Further, when the conversion skip is applied, that is, when the coefficient conversion process is skipped, the normalization unit 214 selects the predicted residual (extended predicted residual) supplied from the decompression processing unit 215 by selecting the selection unit 213. Get through. That is, when the conversion skip is applied, the coefficient conversion is skipped, so that the predicted residual is supplied to the normalization unit 214. The processing unit after the normalization unit 214 processes this predicted residual as a conversion coefficient. That is, this predicted residual is processed in the same manner as in the case of the conversion coefficient generated by the conversion processing unit 212. Therefore, in the following, the predicted residuals supplied from the decompression processing unit will also be described as conversion coefficients. The normalization unit 214 normalizes the acquired conversion coefficient. The normalization unit 214 supplies the normalized conversion coefficient to the quantization unit 202.
- the quantization unit 202 has a quantization processing unit 221 and a normalization unit 222.
- the quantization processing unit 221 acquires the conversion coefficient supplied from the conversion unit 201.
- the quantization processing unit 221 performs quantization on the acquired conversion coefficient to generate a quantization coefficient.
- the quantization processing unit 221 supplies the generated quantization coefficient to the normalization unit 222.
- the normalization unit 222 acquires the quantization coefficient supplied from the quantization processing unit 221.
- the normalization unit 222 normalizes the acquired quantization coefficient.
- the normalization unit 222 outputs the normalized quantization coefficient to the outside of the conversion quantization device 200.
- the above-mentioned (method 1-2) is applied. That is, the conversion quantization device 200 sets the parameters related to the conversion quantization so as to suppress the loss of information.
- the normalization unit 214 can perform normalization using the parameters set in this way.
- the quantization processing unit 221 can perform quantization using the parameters set in this way.
- the normalization unit 222 can perform normalization using the parameters set in this way.
- the conversion quantization device 200 can suppress the loss of information due to the coding to which this conversion quantization processing is applied. That is, the conversion quantization device 200 can bring the coding by the "conversion skip + QP4" approach closer to the lossless coding (lossless coding).
- step S201 determines whether or not to apply the conversion skip. If it is determined that the conversion skip is not applied, the process proceeds to step S202.
- step S202 the conversion unit 201 sets parameters related to coefficient conversion.
- step S203 the conversion processing unit 212 performs coefficient conversion on the predicted residual using the parameters set in step S202 to generate a conversion coefficient.
- step S203 the process proceeds to step S205.
- step S201 If it is determined in step S201 that the conversion skip is applied, the process proceeds to step S204.
- step S204 the stretch processing unit 215 stretches the predicted residuals.
- step S205 the process proceeds to step S205.
- step S205 the normalization unit 214 normalizes the conversion coefficient generated in step S203 or the predicted residual (that is, conversion coefficient) extended in step S204 using the parameters set in step S202. ..
- step S206 the quantization unit 202 sets parameters related to quantization.
- step S207 the quantization processing unit 221 performs quantization on the conversion coefficient normalized in step S205 using the parameters set in step S206, and generates a quantization coefficient.
- step S208 the normalization unit 222 normalizes the quantization coefficient generated in step S207 using the parameters set in step S206.
- step S208 When the process of step S208 is completed, the conversion quantization process is completed.
- step S202 the conversion unit 201 sets the parameters related to the coefficient conversion so as to suppress the loss of information.
- step S205 the normalization unit 214 can perform normalization using the parameters set in this way.
- step S206 the quantization unit 202 sets parameters related to quantization so as to suppress information loss.
- step S207 the quantization processing unit 221 can perform quantization using the parameters set in this way.
- step S208 the normalization unit 222 can perform normalization using the parameters set in this way.
- the conversion quantization device 200 can suppress the loss of information due to the coding to which this conversion quantization processing is applied. That is, the conversion quantization device 200 can bring the coding by the "conversion skip + QP4" approach closer to the lossless coding (lossless coding).
- ⁇ 1-2-1 Control of quantization parameters>
- the parameters related to quantization may be controlled.
- the conversion coefficient may be quantized by using the quantization parameter according to whether the conversion skip is applied.
- the image processing apparatus may be provided with a quantization unit that performs quantization with respect to the conversion coefficient by using the quantization parameter according to whether or not the conversion skip is applied.
- the image processing apparatus can suppress the loss of information due to the coding to which the quantization by the quantization unit is applied. That is, the image processing apparatus can bring the coding by the "conversion skip + QP4" approach closer to the lossless coding (lossless coding).
- the quantization unit applies the dependent quantization in the quantization and does not apply the conversion skip
- the quantization parameter is corrected and the quantization is performed using the corrected quantization parameter. May be good. That is, as shown in the sixth row from the top of the table in FIG. 1, the correction of the quantization parameter (QP) of the dependent quantization may be controlled based on the conversion skip (method 1-2-1). ..
- the quantization parameter is corrected according to the application of the dependent quantization as in the example of A in FIG.
- the quantization step size can be set to 1 by correcting the quantization parameter by dependent quantization (QP + 1). become unable. Therefore, there is a risk of information loss due to quantization.
- the quantization processing unit 221 derives the scaling parameter ls according to such semantics. By doing so, when the conversion skip is applied, the equation (8-960) is applied, and the correction of the quantization parameter by the dependent quantization is omitted. Therefore, the quantization processing unit 221 can suppress the loss of information due to the dependent quantization when the conversion skip is applied.
- the parameters related to the quantization to be controlled may be other than the quantization parameters. For example, as shown in the seventh column from the top of the table in FIG. 1, for example, even if the sign of the rounding offset of the normalization process in the quantization is set based on the sign of the quantization coefficient which is the input value. Good (Method 1-2-2).
- the image processing apparatus may be further provided with a quantization normalization unit that sets the sign of the rounding offset based on the sign of the quantization coefficient and normalizes the quantization coefficient using the rounding offset. Good.
- the round of the normalization process in the quantization is based on the sign of the quantization coefficient which is the input value. make sure that the sign of the ding offset is set.
- the quantization unit 202 derives the rounding offset fwdbdOffset2 using the scaling parameter fwdbdShift2 as in the equation (B), and the normalization unit 222 scales with the rounding offset fwdbdOffset2. Normalize using the parameter fwdbdShift2.
- the quantization unit 202 corrects the rounding offset fwdbdOffset2 derived using the scaling parameter fwdbdShift2 by multiplying the sign of the quantization coefficient level. Then, the normalization unit 222 performs normalization by using the corrected rounding offset fwdbdOffset2'and the scaling parameter fwdbdShift2.
- the normalization unit 222 can perform the normalization process by arithmetic bit shift, so that it is possible to reduce the bias of the sign (positive or negative) of the rounding error and suppress the loss of information in the quantization. it can.
- ⁇ 1-2-3 Sign control of rounding offset> Parameters related to coefficient conversion may be controlled. For example, as shown in the eighth column from the top of the table in FIG. 1, for example, the sign of the rounding offset of the normalization process in the coefficient conversion may be set based on the sign of the conversion coefficient which is the input value. (Method 1-2-3).
- the sign of the rounding offset is set based on the sign of the conversion coefficient, the conversion coefficient is normalized using the rounding offset, and the conversion skip is applied.
- the sign of the rounding offset is set based on the sign of the conversion coefficient (that is, the predicted residual), and the conversion normalization unit that normalizes the conversion coefficient (that is, the predicted residual) using the rounding offset is further provided. You may.
- the coefficient conversion may be an irreversible process. That is, there is a risk of information loss in coefficient conversion.
- the rounding of the normalization process in the coefficient conversion is based on the sign of the conversion coefficient which is the input value. make sure that the offset code is set.
- the conversion unit 201 derives the rounding offset fwdbdOffset1 using the scaling parameter fwdbdShift1 as in the equation (D), and the normalization unit 214 derives the rounding offset fwdbdOffset1 and the scaling parameter. Normalize using fwdbdShift1.
- the conversion unit 201 corrects the rounding offset fwdbdOffset1 derived using the scaling parameter fwdbdShift1 by multiplying the sign of the conversion coefficient. Then, the normalization unit 214 performs normalization by using the corrected rounding offset fwdbdOffset1'and the scaling parameter fwdbdShift1.
- the normalization unit 214 can perform the normalization process by arithmetic bit shift, so that it is possible to reduce the bias of the sign (positive or negative) of the rounding error and suppress the loss of information in the coefficient conversion. it can.
- ⁇ 1-2-4 Combination>
- ⁇ 1-2-1 Control of quantization parameter>
- ⁇ 1-2-2 Sign control of rounding offset>
- ⁇ 1-2-3 Sign control of rounding offset>.
- the methods can be applied in combination as appropriate.
- the conversion quantization device 200 can further suppress the loss of information in the conversion quantization process. That is, the conversion quantization device 200 can bring the coding by the "conversion skip + QP4" approach closer to the lossless coding by combining more of the above methods.
- ⁇ 1-2-1 Control of quantization parameter>
- ⁇ 1-2-2 Sign control of rounding offset>
- ⁇ 1-2-3 Sign control of rounding offset>
- Each method described in each can be applied in combination as appropriate. That is, the inverse quantization inverse converter 100 brings the decoding by the "conversion skip + QP4" approach closer to the reversible decoding (lossless coding), and the transformation quantization apparatus 200 encodes by the "conversion skip + QP4" approach. Can be closer to reversible coding (lossless coding).
- Second Embodiment> ⁇ Merge of decompression processing when conversion is skipped>
- redundant processing exists, which increases the load of the inverse quantization inverse conversion processing.
- the TS coefficient expansion process peculiar to the conversion skip is performed in the data path at the time of conversion skip.
- ⁇ tsShift the TS coefficient decompression process is a bit shift in the opposite direction to the normalization process (>> bdShift1) in the inverse quantization process. Therefore, it was redundant to perform these processes as different processes from each other.
- inverse quantization is performed on the quantization coefficient using the quantization parameter according to whether the conversion skip is applied, and when the conversion skip is not applied, the conversion coefficient generated by the inverse quantization is used.
- the inverse coefficient conversion is performed to generate the predicted residual which is the residual between the image and the predicted image of the image and the conversion skip is applied, the inverse coefficient conversion is skipped.
- a transformation generated by inverse quantization by the inverse quantization unit using an inverse quantization unit that performs inverse quantization with respect to the quantization coefficient and a scaling parameter depending on whether conversion skip is applied For example, in an image processing device, a transformation generated by inverse quantization by the inverse quantization unit using an inverse quantization unit that performs inverse quantization with respect to the quantization coefficient and a scaling parameter depending on whether conversion skip is applied.
- the inverse quantization normalization unit that normalizes the coefficients, and when the conversion skip is not applied, the inverse quantization normalization unit performs inverse quantization conversion on the conversion coefficient normalized by the inverse quantization normalization unit, and the image and the predicted image of that image are displayed.
- an inverse conversion unit that skips the inverse coefficient conversion is provided.
- FIG. 9 is a block diagram showing an example of the configuration of the inverse quantization inverse converter 100 in this case. Note that FIG. 9 shows the main things such as the processing unit and the data flow, and not all of them are shown in FIG. That is, in the inverse quantization inverse conversion device 100, there may be a processing unit that is not shown as a block in FIG. 9, or there may be a processing or data flow that is not shown as an arrow or the like in FIG. ..
- the inverse quantization inverse conversion device 100 in this case has an inverse quantization unit 101 and an inverse conversion unit 102 as in the case of FIG. 2.
- the inverse quantization unit 101 has an inverse quantization processing unit 111 and a normalization unit 112, as in the case of FIG.
- the inverse conversion unit 102 has a selection unit 121 to a normalization unit 124. That is, as compared with the case of FIG. 2, the decompression processing unit 125 is omitted.
- the above-mentioned (method 2-1) is applied. That is, the inverse quantization inverse conversion device 100 substitutes the decompression processing in the conversion skip with the normalization processing in the inverse quantization. That is, the normalization unit 112 normalizes the conversion coefficient generated by the inverse quantization by the inverse quantization processing unit 111 by using the scaling parameter according to whether the conversion skip is applied.
- the decompression processing unit 125 in FIG. 2 merges the decompression processing with the normalization processing in the inverse quantization by the normalization unit 112 in FIG.
- the inverse quantization inverse conversion device 100 can suppress an increase in the load of the inverse quantization inverse conversion processing. That is, the inverse quantization inverse converter 100 can suppress an increase in the decoding load due to the "conversion skip + QP4" approach.
- each process of steps S301 to S306 is basically executed in the same manner as each process of steps S101 to S106 of FIG.
- step S304 determines whether the conversion skip is applied. If it is determined in step S304 that the conversion skip is applied, the process proceeds to step S307. When the process of step S306 is completed, the process proceeds to step S307.
- step S307 is executed in the same manner as the process of step S108 of FIG.
- the process of step S307 is completed, the inverse quantization inverse conversion process is completed.
- step S303 the normalization unit 112 performs a normalization process on the conversion coefficient so as to replace the decompression process in the conversion skip. That is, the normalization unit 112 normalizes the conversion coefficient generated by the inverse quantization by the inverse quantization processing unit 111 by using the scaling parameter according to whether the conversion skip is applied.
- the decompression process performed in step S107 of FIG. 3 is merged with the normalization process in the inverse quantization in step S303.
- the inverse quantization inverse conversion device 100 can suppress an increase in the load of the inverse quantization inverse conversion processing. That is, the inverse quantization inverse converter 100 can suppress an increase in the decoding load due to the "conversion skip + QP4" approach.
- ⁇ 2-1-1 Setting scaling parameters according to conversion skip>
- the scaling parameter bdShift1 used in the normalization process and the scaling parameter tsShift used in the decompression process are, for example, a figure. It is derived as in the semantics shown in A of 11.
- the scaling parameter bdShift1 is a component based on the bit depth (bitDepth), a component based on the transformation block size and the transformation skip flag (transform_skip_flag) ((rectNonTsFlag? 1: 0)), and a component based on the transformation block size (((rectNonTsFlag? 1: 0)). It is derived by the sum of log2 (nTbW) + log2 (nTbH) / 2) -5) and the component (dep_quant_enabled_flag) based on dependent quantization.
- the scaling parameter tsShift is derived from the component (5+ (log2 (nTbW) + log2 (nTbH) / 2)) based on the conversion block size.
- the scaling parameter bdShift1 contains a component in which the positive and negative of the scaling parameter tsShift are reversed. Therefore, when the conversion skip is applied, the bit shift of the decompression processing in the conversion skip is reflected in the bit shift of the normalization processing in the inverse quantization, and the processing is merged.
- the conversion coefficient is normalized using a scaling parameter that does not reflect the expansion process in the conversion skip, and when the conversion skip is applied, the expansion process in the conversion skip is reflected.
- the conversion coefficient is normalized using the scaling parameters.
- the inverse quantization inverse conversion device 100 can reduce redundant processing, so that an increase in the load of the inverse quantization inverse conversion processing can be suppressed. That is, the inverse quantization inverse converter 100 can suppress an increase in the decoding load due to the "conversion skip + QP4" approach.
- the method of expressing semantics in this case is arbitrary and is not limited to the example of B in FIG.
- the variable tsShift is set according to whether or not the conversion skip is applied, and the variable tsShift is used as a component based on the conversion block size to derive the scaling parameter bdShift1. You may try to do it.
- the component based on the dependent quantization may be further set depending on whether or not the conversion skip is applied.
- the component based on the dependent quantization (dep_quant_enabled_flag) is added to the scaling parameter bdShift1 only when the conversion skip is not applied.
- the component based on the transform block size of the scaling parameter bdShift1 may be set to a value according to the quantization matrix.
- the component based on the conversion block size is set to ((log2 (nTbW) + log2 (nTbH) / 2) -5).
- the inverse quantization inverse converter 100 can suppress the processing from overflowing.
- the method of expressing semantics in this case is arbitrary and is not limited to the example of A in FIG.
- the variable tsShift is set according to whether or not the conversion skip is applied, and the variable tsShift is used as a component based on the conversion block size to derive the scaling parameter bdShift1. You may try to do it.
- the component based on the dependent quantization may be further set depending on whether or not the conversion skip is applied.
- the component based on the dependent quantization (dep_quant_enabled_flag) is added to the scaling parameter bdShift1 only when the conversion skip is not applied.
- the conversion quantization process is the same as in the case of the above-mentioned inverse quantization inverse conversion process. That is, when the conversion skip is applied, there is a possibility that redundant processing exists, which increases the load of the conversion quantization processing.
- the TS coefficient expansion process peculiar to conversion skip ( ⁇ fwdtsShift) ).
- this TS coefficient decompression process is a bit shift in the opposite direction to the normalization process (>> fwdbdShift2) in the quantization process. Therefore, it was redundant to perform these processes as different processes from each other.
- the coefficient conversion is performed on the predicted residual which is the residual between the image and the predicted image of the image, the conversion coefficient is generated, and the conversion skip is applied.
- the conversion coefficient is quantized, and if the conversion skip is applied, the prediction residual is quantized and the scaling parameter is set according to whether the conversion skip is applied. It is used to normalize the quantization coefficient generated by quantization.
- the coefficient conversion is performed on the predicted residual which is the residual between the image and the predicted image of the image, the conversion coefficient is generated, and the conversion skip is applied.
- FIG. 14 is a block diagram showing an example of the configuration of the conversion quantization device 200 in this case. It should be noted that FIG. 14 shows the main things such as the processing unit and the data flow, and not all of them are shown in FIG. That is, in the conversion quantization device 200, there may be a processing unit that is not shown as a block in FIG. 14, or there may be a processing or data flow that is not shown as an arrow or the like in FIG.
- the conversion quantization device 200 in this case has a conversion unit 201 and a quantization unit 202 as in the case of FIG. 7.
- the conversion unit 201 has a selection unit 211 to a normalization unit 214. That is, as compared with the case of FIG. 7, the decompression processing unit 215 is omitted.
- the quantization unit 202 has a quantization processing unit 221 and a normalization unit 222, as in the case of FIG. 7.
- the above-mentioned (method 2-2) is applied. That is, the conversion quantization device 200 replaces the decompression processing in the conversion skip with the normalization processing in the quantization. That is, the normalization unit 222 normalizes the quantization coefficient generated by the quantization by the quantization processing unit 221 by using the scaling parameter according to whether the conversion skip is applied.
- the decompression process by the decompression processing unit 215 of FIG. 7 is merged with the normalization process in the quantization by the normalization unit 222 of FIG.
- the conversion quantization device 200 can suppress an increase in the load of this conversion quantization processing. That is, the conversion quantization device 200 can suppress an increase in the coding load due to the "conversion skip + QP4" approach.
- each process of steps S321 to S327 is executed basically in the same manner as each process of steps S201 to S203 of FIG. 8 and steps S205 to S208. To.
- step S321 determines whether the conversion skip is applied. If it is determined in step S321 that the conversion skip is applied, the process proceeds to step S324. When the process of step S323 is completed, the process proceeds to step S324.
- step S327 When the process of step S327 is completed, the conversion quantization process is completed.
- step S327 the normalization unit 222 performs a normalization process on the quantization coefficient so as to replace the decompression process in the conversion skip. That is, the normalization unit 222 normalizes the quantization coefficient generated by the quantization by the quantization processing unit 221 by using the scaling parameter according to whether the conversion skip is applied. Then, when the conversion skip is applied, the decompression process performed in step S204 of FIG. 8 is merged with the normalization process in the quantization in step S327.
- the conversion quantization device 200 can suppress an increase in the load of this conversion quantization processing. That is, the conversion quantization device 200 can suppress an increase in the coding load due to the "conversion skip + QP4" approach.
- ⁇ 2-2-1 Setting scaling parameters according to conversion skip>
- the scaling parameter fwdbdShift2 used in the normalization process and the scaling parameter fwdtsShift used in the decompression process are A in FIG. Is derived in the same way as in the example of.
- the scaling parameter fwdbdShift2 contains a component in which the positive and negative of the scaling parameter fwdtsShift are reversed. Therefore, it is redundant to perform the normalization process in the childization and the decompression process in the conversion skip as separate processes. Therefore, when applying the conversion skip, the bit shift of the decompression processing in the conversion skip is reflected in the bit shift of the normalization processing in the quantization, and those processes are merged.
- the conversion processing unit 212 when the conversion skip is not applied, the conversion processing unit 212 performs the coefficient conversion for the predicted residual, and when the conversion skip is applied, the conversion processing unit 212 skips the coefficient conversion without performing the expansion processing, and the normalization unit 222 ,
- the quantization coefficient is normalized using a scaling parameter that does not reflect the decompression processing in the conversion skip, and when the conversion skip is applied, the quantization is performed using the scaling parameter that reflects the decompression processing. Try to normalize the coefficients.
- the conversion quantization device 200 can reduce redundant processing, so that an increase in the load of the conversion quantization processing can be suppressed. That is, the conversion quantization device 200 can suppress an increase in the coding load due to the "conversion skip + QP4" approach.
- the method of expressing semantics in this case is arbitrary.
- the variable ffwdtsShift is set according to whether or not the conversion skip is applied, and the variable ffwdtsShift is used as a component based on the conversion block size, and the scaling parameter ffwdbdShift1 is used. May be derived.
- the component based on the dependent quantization may be further set depending on whether or not the conversion skip is applied.
- the component based on the dependent quantization (dep_quant_enabled_flag) is added to the scaling parameter fwdbdShift2 only when the conversion skip is not applied.
- the conversion quantization device 200 can suppress the processing from overflowing.
- the method of expressing semantics in this case is arbitrary.
- the variable ffwdtsShift is set according to whether or not the conversion skip is applied, and the variable ffwdtsShift is used as a component based on the conversion block size to use the scaling parameter ffwdbdShift2. May be derived.
- the component based on the dependent quantization may be further set depending on whether or not the conversion skip is applied.
- the component based on the dependent quantization (dep_quant_enabled_flag) is added to the scaling parameter fwdbdShift2 only when the conversion skip is not applied.
- FIG. 16 is a block diagram showing an example of the configuration of the inverse quantization inverse converter 100 in this case. It should be noted that FIG. 16 shows the main things such as the processing unit and the data flow, and not all of them are shown in FIG. That is, in the inverse quantization inverse conversion device 100, there may be a processing unit that is not shown as a block in FIG. 16, or there may be a processing or data flow that is not shown as an arrow or the like in FIG. ..
- the inverse quantization reverse conversion device 100 in this case has an inverse quantization unit 101 and an inverse conversion unit 102 as in the case of FIG.
- the inverse quantization unit 101 has an inverse quantization processing unit 111 and a normalization unit 112, as in the case of FIG.
- the inverse conversion unit 102 has a selection unit 121 to a normalization unit 124. That is, as compared with the case of FIG. 2, the decompression processing unit 125 is omitted.
- the normalization unit 124 is provided between the inverse conversion processing unit 122 and the selection unit 123. Therefore, in this case, if the conversion skip is not applied, the normalization unit 124 acquires the predicted residuals supplied from the inverse conversion processing unit 122, performs normalization processing on the predicted residuals, and is normalized. The predicted residual is supplied to the selection unit 123. When the conversion skip is not applied, the selection unit 123 acquires the predicted residual (normalized predicted residual) supplied from the normalizing unit 124 and sends it to the outside of the inverse quantization inverse conversion device 100. Output.
- the above-mentioned (method 3-1) is applied. That is, in the case of conversion skip, the inverse quantization inverse conversion device 100 substitutes the decompression processing in conversion skip and the normalization processing in inverse coefficient conversion with the normalization processing in inverse quantization. That is, the normalization unit 112 normalizes the conversion coefficient generated by the inverse quantization by the inverse quantization processing unit 111 by using the scaling parameter according to whether the conversion skip is applied.
- the decompression processing by the decompression processing unit 125 in FIG. 2 and the normalization processing in the inverse coefficient conversion by the normalization unit 124 in FIG. 2 are performed in the inverse quantization by the normalization unit 112 in FIG. Merged with normalization.
- the inverse quantization inverse conversion device 100 can suppress an increase in the load of the inverse quantization inverse conversion processing. That is, the inverse quantization inverse converter 100 can suppress an increase in the decoding load due to the "conversion skip + QP4" approach.
- each process of steps S341 to S347 is executed basically in the same manner as each process of steps S301 to S307 of FIG.
- step S344 determines whether the conversion skip is applied. If it is determined in step S344 that the conversion skip is applied, each process of steps S345 to S347 is skipped, and the inverse quantization inverse conversion process ends.
- step S343 when the conversion skip is applied, the normalization unit 112 performs the normalization process on the conversion coefficient so as to replace the decompression process in the conversion skip and the normalization process in the inverse coefficient conversion. That is, the normalization unit 112 normalizes the conversion coefficient generated by the inverse quantization by the inverse quantization processing unit 111 by using the scaling parameter according to whether the conversion skip is applied.
- the decompression process performed in step S107 of FIG. 3 and the normalization process performed in step S108 of FIG. 3 are merged with the normalization process in the inverse quantization in step S343.
- the inverse quantization inverse conversion device 100 can suppress an increase in the load of the inverse quantization inverse conversion processing. That is, the inverse quantization inverse converter 100 can suppress an increase in the decoding load due to the "conversion skip + QP4" approach.
- ⁇ 3-1-1 Setting scaling parameters according to conversion skip>
- the scaling parameter bdShift1 used for the normalization process in inverse quantization and the decompression process The scaling parameter tsShift used and the scaling parameter bdShift2 used for the normalization process in the inverse coefficient conversion are derived, for example, as in the semantics shown in FIG. 18A.
- the scaling parameter bdShift1 and the scaling parameter tsShift are derived as described in the second embodiment, respectively.
- the scaling parameter bdShift2 is set to the larger value of the difference between "20" and the bit depth (bitDepth) and "0". Merge bit shifts with these scaling parameters.
- the normalization unit 124 when the normalization unit 124 does not apply the conversion skip, the predicted residual is normalized, and when the normalization unit 112 applies the conversion skip, the normalization process for the predicted residual by the normalization unit 124 is reflected. Try to normalize the conversion factors using scaling parameters.
- the scaling parameter bdShift1 used for the normalization process in inverse quantization is the variable trBdShift. Derived by sum with the component (dep_quant_enabled_flag) based on dependent quantization.
- This variable trBdShift is derived as the sum of the value according to the inverse quantization (INV_QUANT) and the value according to the scaling list (SCALING_LIST_BITS) as in the semantics shown in B of FIG.
- the value according to the inverse quantization is set to "6"
- the value according to the scaling list is set to "4
- the variable trBdShift is set to "10".
- the inverse quantization inverse conversion device 100 can reduce redundant processing, so that an increase in the load of the inverse quantization inverse conversion processing can be suppressed. That is, the inverse quantization inverse converter 100 can suppress an increase in the decoding load due to the "conversion skip + QP4" approach.
- the method of expressing semantics in this case is arbitrary.
- the variable trBdShift is set according to whether or not the conversion skip is applied, and the variable trBdShift is used as a component based on the conversion block size to derive the scaling parameter bdShift1. You may try to do it.
- a component based on dependent quantization may be further set depending on whether or not conversion skip is applied.
- the component based on the dependent quantization (dep_quant_enabled_flag) is added to the scaling parameter bdShift1 only when the conversion skip is not applied.
- the conversion quantization process is the same as in the case of the above-mentioned inverse quantization inverse conversion process. That is, when the conversion skip is applied, there is a possibility that redundant processing exists, which increases the load of the conversion quantization processing.
- the TS coefficient expansion process peculiar to conversion skip ( ⁇ fwdtsShift) ) And "fwdbdShift1" for normalization.
- the bit shift in this TS coefficient height processing and normalization processing can be merged with the bit shift in the normalization processing (>> fwdbdShift2) in the quantization process. That is, it was redundant to perform each of these bit shifts.
- FIG. 20 is a block diagram showing an example of the configuration of the conversion quantization device 200 in this case. It should be noted that FIG. 20 shows the main things such as the processing unit and the data flow, and not all of them are shown in FIG. 20. That is, in the conversion quantization device 200, there may be a processing unit that is not shown as a block in FIG. 20, or there may be a processing or data flow that is not shown as an arrow or the like in FIG. 20.
- the conversion quantization device 200 in this case has a conversion unit 201 and a quantization unit 202 as in the case of FIG. 7.
- the conversion unit 201 has a selection unit 211 to a normalization unit 214. That is, as compared with the case of FIG. 7, the decompression processing unit 215 is omitted.
- the normalization unit 214 is provided between the conversion processing unit 212 and the selection unit 213. Therefore, in this case, the normalization unit 214 acquires the conversion coefficient supplied from the conversion processing unit 212 when the conversion skip is not applied, performs normalization processing on the conversion coefficient, and normalizes the conversion coefficient. Is supplied to the selection unit 213. When the conversion skip is not applied, the selection unit 213 acquires the conversion coefficient (normalized conversion coefficient) supplied from the normalization unit 214 and outputs it to the outside of the conversion quantization device 200.
- the quantization unit 202 has a quantization processing unit 221 and a normalization unit 222, as in the case of FIG. 7.
- the above-mentioned (method 3-2) is applied. That is, the conversion quantization device 200 replaces the decompression processing in the conversion skip and the normalization processing in the coefficient conversion with the normalization processing in the quantization. That is, the normalization unit 222 normalizes the quantization coefficient generated by the quantization by the quantization processing unit 221 by using the scaling parameter according to whether the conversion skip is applied.
- the decompression processing by the decompression processing unit 215 of FIG. 7 and the normalization processing by the normalization unit 214 of FIG. 7 are merged with the normalization processing in the quantization by the normalization unit 222 of FIG. Will be done.
- the conversion quantization device 200 can suppress an increase in the load of this conversion quantization processing. That is, the conversion quantization device 200 can suppress an increase in the coding load due to the "conversion skip + QP4" approach.
- each process of steps S361 to S367 is executed basically in the same manner as each process of steps S321 to S327 of FIG.
- step S361 determines whether the conversion skip is applied. If it is determined in step S361 that the conversion skip is applied, each process of steps S362 to S364 is skipped, and the process proceeds to step S365.
- step S367 when the conversion skip is applied, the normalization unit 222 performs the normalization process on the conversion coefficient so as to replace the decompression process in the conversion skip and the normalization process in the coefficient conversion. That is, the normalization unit 222 normalizes the quantization coefficient generated by the quantization by the quantization processing unit 221 by using the scaling parameter according to whether the conversion skip is applied.
- the decompression process performed in step S204 of FIG. 8 and the normalization process performed in step S205 of FIG. 8 are merged with the normalization process in the quantization in step S367.
- the conversion quantization device 200 can suppress an increase in the load of this conversion quantization processing. That is, the conversion quantization device 200 can suppress an increase in the coding load due to the "conversion skip + QP4" approach.
- ⁇ 3-2-1 Setting of scaling parameters according to conversion skip>
- the conversion coefficient is normalized
- the normalization unit 222 applies the conversion skip
- the scaling parameter reflecting the normalization processing for the predicted residual by the normalization unit 214 is set. Use to normalize the quantization factor.
- the scaling parameter fwdbdShift2 used for the normalization process in quantization is a variable fwdtrBdShift and dependent quantization. Derived by sum with the component (dep_quant_enabled_flag) based on.
- This variable fwdtrBdShift is derived as the sum of the value according to the quantization (QUANT) and the value according to the scaling list (SCALING_LIST_BITS), as in the case of the semantics shown in B of FIG.
- the value according to the quantization (QUANT) is set to "6"
- the value according to the scaling list (SCALING_LIST_BITS) is set to "4". That is, the variable fwdtrBdShift is set to "10".
- the conversion quantization device 200 can reduce redundant processing, so that an increase in the load of the conversion quantization processing can be suppressed. That is, the conversion quantization device 200 can suppress an increase in the coding load due to the "conversion skip + QP4" approach.
- variable fwdtrBdShift may be set according to whether or not the conversion skip is applied, and the scaling parameter ffwdbdShift2 may be derived using the variable fwdtrBdShift. ..
- a component based on dependent quantization may be further set depending on whether or not conversion skip is applied.
- the component based on the dependent quantization (dep_quant_enabled_flag) is added to the scaling parameter fwdbdShift2 only when the conversion skip is not applied.
- the present technology described above in the first embodiment to the third embodiment and the like can be applied to any device, device, system and the like.
- the present technology can be applied to an image decoding device that decodes encoded data of image data. It can also be applied to an image coding device that encodes image data.
- FIG. 22 is a block diagram showing an example of the configuration of an image decoding device, which is an aspect of an image processing device to which the present technology is applied.
- the image decoding device 500 shown in FIG. 22 is a device that decodes the coded data of the moving image.
- the image decoding device 500 decodes the encoded data of the moving image encoded by the encoding method such as VVC, AVC, HEVC, etc. described in the above-mentioned non-patent document.
- the image decoding device 500 can decode the coded data (bit stream) generated by the image coding device 600 described later.
- FIG. 22 shows the main things such as the processing unit and the data flow, and not all of them are shown in FIG. 22. That is, in the image decoding device 500, there may be a processing unit that is not shown as a block in FIG. 22, or there may be a processing or data flow that is not shown as an arrow or the like in FIG. This also applies to other figures illustrating the processing unit and the like in the image decoding device 500.
- the image decoding device 500 includes a control unit 501, a storage buffer 511, a decoding unit 512, an inverse quantization inverse conversion unit 513, an arithmetic unit 514, an in-loop filter unit 515, a sorting buffer 516, a frame memory 517, and The prediction unit 518 is provided.
- the prediction unit 518 includes an intra prediction unit (not shown) and an inter prediction unit.
- the control unit 501 performs processing related to decoding control. For example, the control unit 501 acquires the coding parameters (header information Hinfo, prediction mode information Pinfo, conversion information Tinfo, residual information Rinfo, filter information Finfo, etc.) included in the bit stream via the decoding unit 512. Further, the control unit 501 can estimate the coding parameters not included in the bit stream. Further, the control unit 501 controls decoding by controlling each processing unit (accumulation buffer 511 to prediction unit 518) of the image decoding device 500 based on the acquired (or estimated) coding parameter.
- the control unit 501 controls decoding by controlling each processing unit (accumulation buffer 511 to prediction unit 518) of the image decoding device 500 based on the acquired (or estimated) coding parameter.
- control unit 501 supplies the header information Hinfo to the inverse quantization inverse conversion unit 513, the prediction unit 518, and the in-loop filter unit 515. Further, the control unit 501 supplies the prediction mode information Pinfo to the inverse quantization inverse conversion unit 513 and the prediction unit 518. Further, the control unit 501 supplies the conversion information Tinfo to the inverse quantization inverse conversion unit 513. Further, the control unit 501 supplies the residual information Rinfo to the decoding unit 512. Further, the control unit 501 supplies the filter information Finfo to the in-loop filter unit 515.
- each coding parameter may be supplied to an arbitrary processing unit.
- other information may be supplied to an arbitrary processing unit.
- Header information Hinfo includes header information such as VPS (Video Parameter Set) / SPS (Sequence Parameter Set) / PPS (Picture Parameter Set) / PH (picture header) / SH (slice header).
- the header information Hinfo includes, for example, image size (width PicWidth, height PicHeight), bit depth (brightness bitDepthY, color difference bitDepthC), color difference array type ChromaArrayType, maximum CU size MaxCUSize / minimum MinCUSize, quadtree division ( Maximum depth of Quad-tree division MaxQTDepth / Minimum depth MinQTDepth / Maximum depth of binary-tree division (Binary-tree division) MaxBTDepth / Minimum depth MinBTDepth, Maximum value of conversion skip block MaxTSSize (also called maximum conversion skip block size) ), Information that defines the on / off flag (also called the valid flag) of each coding tool is included.
- the on / off flags of the coding tool included in the header information Hinfo there are the on / off flags related to the conversion and quantization processing shown below.
- the on / off flag of the coding tool can also be interpreted as a flag indicating whether or not the syntax related to the coding tool exists in the coded data. Further, when the value of the on / off flag is 1 (true), it indicates that the coding tool can be used, and when the value of the on / off flag is 0 (false), it indicates that the coding tool cannot be used. The interpretation of the flag value may be reversed.
- the prediction mode information Pinfo includes, for example, information such as size information PBSize (prediction block size) of the processing target PB (prediction block), intra prediction mode information IPinfo, and motion prediction information MVinfo.
- Intra prediction mode information IPinfo includes, for example, prev_intra_luma_pred_flag, mpm_idx, rem_intra_pred_mode in JCTVC-W1005, 7.3.8.5 Coding Unit syntax, and the brightness intra prediction mode IntraPredModeY derived from the syntax.
- Intra-prediction mode information IPinfo includes, for example, inter-component prediction flag (ccp_flag (cclmp_flag)), multi-class linear prediction mode flag (mclm_flag), color difference sample position type identifier (chroma_sample_loc_type_idx), color difference MPM identifier (chroma_mpm_idx), , IntraPredModeC, etc., which are derived from these syntaxes.
- the multi-class linear prediction mode flag is information regarding the mode of linear prediction (linear prediction mode information). More specifically, the multi-class linear prediction mode flag (mclm_flag) is flag information indicating whether or not to set the multi-class linear prediction mode. For example, "0" indicates that it is a one-class mode (single class mode) (for example, CCLMP), and "1" indicates that it is a two-class mode (multi-class mode) (for example, MCLMP). ..
- the color difference sample position type identifier (chroma_sample_loc_type_idx) is an identifier that identifies the type of pixel position of the color difference component (also referred to as the color difference sample position type).
- this color difference sample position type identifier (chroma_sample_loc_type_idx) is transmitted (stored in) as information (chroma_sample_loc_info ()) regarding the pixel position of the color difference component.
- the color difference MPM identifier (chroma_mpm_idx) is an identifier indicating which prediction mode candidate in the color difference intra prediction mode candidate list (intraPredModeCandListC) is designated as the color difference intra prediction mode.
- the information included in the prediction mode information Pinfo is arbitrary, and information other than this information may be included.
- the conversion information Tinfo includes, for example, the following information.
- the information included in the conversion information Tinfo is arbitrary, and information other than this information may be included.
- Width size TBWSize and height TBHSize of the conversion block to be processed Each TBWSize having a base of 2 and the radix of TBHSize may be log2TBWSize and log2TBHSize.
- Conversion skip flag (ts_flag) A flag indicating whether to skip the (reverse) primary conversion and the (reverse) secondary conversion.
- the residual information Rinfo includes, for example, the following syntax.
- cbf (coded_block_flag): Residual data presence / absence flag last_sig_coeff_x_pos: Last non-zero coefficient X coordinate last_sig_coeff_y_pos: Last non-zero coefficient Y coordinate coded_sub_block_flag: Subblock non-zero coefficient presence / absence flag sig_coeff_flag: Non-zero coefficient presence / absence flag gr1_flag: Non-zero coefficient level Flag indicating whether it is greater than 1 (also called GR1 flag)
- gr2_flag Flag indicating whether the level of non-zero coefficient is greater than 2 (also called GR2 flag) sign_flag: A sign indicating the positive or negative of the non-zero coefficient (also called a sign code) coeff_abs_level_remaining: Non-zero coefficient residual level (also called non-zero coefficient residual level) Such.
- the information included in the residual information Rinfo is arbitrary, and information other than this information may be included.
- the filter information Finfo includes, for example, control information related to each of the following filter processes.
- Control information for deblocking filters DPF
- Control information for pixel adaptive offset SAO
- Control information for adaptive loop filters ALF
- the picture to which each filter is applied the information for specifying the area in the picture, the filter On / Off control information for each CU, the filter On / Off control information for the slice and tile boundaries, and the like. included.
- the information included in the filter information Finfo is arbitrary, and information other than this information may be included.
- the storage buffer 511 acquires and holds (stores) the bit stream input to the image decoding device 500.
- the storage buffer 511 extracts the coded data included in the stored bit stream at a predetermined timing or when a predetermined condition is satisfied, and supplies the coded data to the decoding unit 512.
- the decoding unit 512 performs processing related to image decoding. For example, the decoding unit 512 takes the coded data supplied from the storage buffer 511 as an input, and entropically decodes (reversibly decodes) the syntax value of each syntax element from the bit string according to the definition of the syntax table. , Derivation of parameters.
- the parameters derived from the syntax element and the syntax value of the syntax element include, for example, information such as header information Hinfo, prediction mode information Pinfo, conversion information Tinfo, residual information Rinfo, and filter information Finfo. That is, the decoding unit 512 parses (analyzes and acquires) this information from the bit stream.
- the decoding unit 512 performs such parsing according to the control of the control unit 501. Then, the decoding unit 512 supplies these information obtained by parsing to the control unit 501.
- the decoding unit 512 decodes the encoded data with reference to the residual information Rinfo. At that time, the decoding unit 512 applies entropy decoding (reversible decoding) such as CABAC or CAVLC. That is, the decoding unit 512 decodes the coded data by a decoding method corresponding to the coding method performed by the coding unit 614 of the image coding device 600.
- entropy decoding reversible decoding
- the decoding unit 512 performs arithmetic decoding using a context model on the coded data to derive the quantization coefficient level of each coefficient position in each conversion block.
- the decoding unit 512 supplies the derived quantization coefficient level to the inverse quantization inverse conversion unit 513.
- the decoding unit 512 performs arithmetic decoding on the coded data in the bypass mode to derive the residual data D.
- the decoding unit 512 supplies the derived residual data D to the inverse quantization inverse conversion unit 513.
- Inverse quantization Inverse conversion unit 513 performs processing related to inverse quantization and inverse coefficient conversion. For example, in the non-lossless coding mode, the inverse quantization inverse conversion unit 513 acquires the quantization coefficient level supplied from the decoding unit 512. The inverse quantization inverse conversion unit 513 scales (inverse quantization) the acquired quantization coefficient level to derive the conversion coefficient Coeff. The inverse quantization inverse conversion unit 513 performs inverse coefficient transformation such as inverse orthogonal transformation on the conversion coefficient Coeff, and derives residual data D'. The inverse quantization inverse conversion unit 513 supplies the residual data D'to the arithmetic unit 514.
- the inverse quantization inverse conversion unit 513 can skip (omit) these inverse quantizations and inverse coefficient conversions. For example, in the lossless coding mode, the inverse quantization inverse conversion unit 513 acquires the residual data D supplied from the decoding unit 512. The inverse quantization inverse conversion unit 513 skips (omitted) the inverse quantization and the inverse coefficient conversion, and supplies the residual data D as the residual data D'to the arithmetic unit 514.
- the inverse quantization inverse conversion unit 513 performs these processes according to the control of the control unit 501.
- the inverse quantization inverse conversion unit 513 can perform these processes based on the prediction mode information Pinfo and the conversion information Tinfo supplied from the control unit 501.
- the calculation unit 514 performs processing related to addition of information related to images. For example, the calculation unit 514 inputs the residual data D'supplied from the inverse quantization inverse conversion unit 513 and the prediction image supplied from the prediction unit 518. The calculation unit 514 adds the residual data and the predicted image (predicted signal) corresponding to the residual data to derive a locally decoded image. The calculation unit 514 supplies the derived locally decoded image to the in-loop filter unit 515 and the frame memory 517.
- the in-loop filter unit 515 performs processing related to the in-loop filter processing. For example, the in-loop filter unit 515 inputs the locally decoded image supplied from the calculation unit 514 and the filter information Finfo supplied from the control unit 501. The information input to the in-loop filter unit 515 is arbitrary, and information other than this information may be input.
- the in-loop filter unit 515 appropriately filters the locally decoded image based on the filter information Finfo.
- the in-loop filter unit 515 includes a bilateral filter, a deblocking filter (DBF (DeBlocking Filter)), an adaptive offset filter (SAO (Sample Adaptive Offset)), and an adaptive loop filter (ALF (Adaptive Loop Filter)). Apply two in-loop filters in this order. It should be noted that which filter is applied and which order is applied is arbitrary and can be appropriately selected.
- the in-loop filter unit 515 performs a filter process corresponding to the filter process performed by the coding side (for example, the in-loop filter unit 618 of the image coding apparatus 600).
- the filter processing performed by the in-loop filter unit 515 is arbitrary and is not limited to the above example.
- the in-loop filter unit 515 may apply a Wiener filter or the like.
- the in-loop filter unit 515 supplies the filtered locally decoded image to the sorting buffer 516 and the frame memory 517.
- the sorting buffer 516 receives the locally decoded image supplied from the in-loop filter unit 515 as an input, and holds (stores) it.
- the sorting buffer 516 reconstructs and holds (stores in the buffer) the decoded image for each picture unit using the locally decoded image.
- the sorting buffer 516 sorts the obtained decoded images from the decoding order to the reproduction order.
- the rearrangement buffer 516 outputs the rearranged decoded image group as moving image data to the outside of the image decoding apparatus 500.
- the frame memory 517 performs processing related to storage of data related to images. For example, the frame memory 517 takes a locally decoded image supplied from the arithmetic unit 514 as an input, reconstructs the decoded image for each picture, and stores it in the buffer in the frame memory 517.
- the frame memory 517 takes an in-loop filtered locally decoded image supplied from the in-loop filter unit 515 as an input, reconstructs the decoded image for each picture, and stores it in the buffer in the frame memory 517. To do.
- the frame memory 517 appropriately supplies the stored decoded image (or a part thereof) to the prediction unit 518 as a reference image.
- the frame memory 517 may store header information Hinfo, prediction mode information Pinfo, conversion information Tinfo, filter information Finfo, etc. related to the generation of the decoded image.
- the prediction unit 518 performs processing related to the generation of the prediction image. For example, the prediction unit 518 inputs the prediction mode information Pinfo supplied from the control unit 501 and the decoded image (or a part thereof) read from the frame memory 517. The prediction unit 518 performs prediction processing in the prediction mode adopted at the time of coding based on the prediction mode information Pinfo, and generates a prediction image by referring to the decoded image as a reference image. The prediction unit 518 supplies the generated prediction image to the calculation unit 514.
- the inverse quantization inverse conversion unit 513 of the image decoding apparatus 500 As the inverse quantization inverse conversion unit 513 of the image decoding apparatus 500 as described above, the inverse quantization inverse conversion apparatus 100 described above in the first to third embodiments may be applied. That is, the inverse quantization inverse conversion unit 513 may have a configuration as shown in FIG. 2, FIG. 9, or FIG. By doing so, the inverse quantization inverse conversion unit 513 can obtain the above-mentioned effect in the first embodiment to the third embodiment. That is, the image decoding device 500 can obtain the above-mentioned effects in the first embodiment to the third embodiment in the decoding process.
- the storage buffer 511 acquires (stores) a bit stream (encoded data) supplied from the outside of the image decoding device 500 in step S501.
- step S502 the decoding unit 512 parses (analyzes and acquires) various coding parameters from the bit stream.
- the control unit 501 sets the various coding parameters by supplying the acquired various coding parameters to the various processing units.
- step S503 the control unit 501 sets the processing unit based on the obtained coding parameter.
- step S504 the decoding unit 512 decodes the bit stream under the control of the control unit 501 to obtain coefficient data (quantization coefficient level or residual data r). For example, when CABAC is applied, in the non-lossless coding mode, the decoding unit 512 performs arithmetic decoding using the context model to derive the quantization coefficient level of each coefficient position in each conversion block. Further, in the case of the lossless coding mode, the decoding unit 512 performs arithmetic decoding on the coded data in the bypass mode to derive the residual data D.
- step S505 the inverse quantization inverse conversion unit 513 performs the inverse quantization inverse conversion process to generate the residual data r (D').
- the inverse quantization and inverse transformation processing will be described later.
- step S506 the prediction unit 518 executes the prediction process by the prediction method specified by the coding side based on the coding parameters and the like set in step S502, and obtains the reference image stored in the frame memory 517.
- a predicted image P is generated by reference or the like.
- step S507 the calculation unit 514 adds the residual data D'obtained in step S505 and the predicted image P obtained in step S506 to derive the locally decoded image Rlocal.
- step S508 the in-loop filter unit 515 performs an in-loop filter process on the locally decoded image Rlocal obtained by the process of step S507.
- step S509 the sorting buffer 516 derives the decoded image R using the locally decoded image Rlocal filtered by the process of step S508, and rearranges the order of the decoded image R group from the decoding order to the reproduction order.
- the decoded image R group sorted in the order of reproduction is output as a moving image to the outside of the image decoding device 500.
- the frame memory 517 stores at least one of the locally decoded image Rlocal obtained by the process of step S507 and the locally decoded image Rlocal filtered by the process of step S508.
- step S510 When the process of step S510 is completed, the image decoding process is completed.
- the inverse quantization inverse conversion process described above in the first to third embodiments may be applied. .. That is, the inverse quantization inverse conversion process may be executed in the flow as shown in the flowchart shown in FIG. 3, FIG. 10, or FIG.
- the inverse quantization inverse conversion unit 513 can obtain the above-mentioned effect in the first embodiment to the third embodiment. That is, the image decoding device 500 can obtain the above-mentioned effects in the first embodiment to the third embodiment in the image decoding process.
- FIG. 24 is a block diagram showing an example of the configuration of an image coding device, which is an aspect of an image processing device to which the present technology is applied.
- the image coding device 600 shown in FIG. 24 is a device that encodes image data of a moving image.
- the image coding apparatus 600 uses a coding method such as VVC (Versatile Video Coding), AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding) described in the above-mentioned non-patent document to obtain image data of a moving image. Encode.
- VVC Very Video Coding
- AVC Advanced Video Coding
- HEVC High Efficiency Video Coding
- FIG. 24 shows the main things such as the processing unit and the data flow, and not all of them are shown in FIG. 24. That is, in the image coding apparatus 600, there may be a processing unit that is not shown as a block in FIG. 24, or there may be a processing or data flow that is not shown as an arrow or the like in FIG. 24. This also applies to other figures for explaining the processing unit and the like in the image coding apparatus 600.
- the image coding device 600 has a control unit 601, a sorting buffer 611, a calculation unit 612, a conversion quantization unit 613, a coding unit 614, and a storage buffer 615. Further, the image coding device 600 includes an inverse quantization inverse conversion unit 616, an arithmetic unit 617, an in-loop filter unit 618, a frame memory 619, a prediction unit 620, and a rate control unit 621.
- the control unit 601 divides the moving image data held by the sorting buffer 611 into blocks (CU, PU, TU, etc.) of the processing unit based on the block size of the external or predetermined processing unit. Further, the control unit 601 determines the coding parameters (header information Hinfo, prediction mode information Pinfo, conversion information Tinfo, filter information Finfo, etc.) to be supplied to each block based on, for example, RDO (Rate-Distortion Optimization). To do. For example, the control unit 601 can set a conversion skip flag or the like.
- control unit 601 determines the coding parameters as described above, the control unit 601 supplies them to each block. Specifically, it is as follows.
- Header information Hinfo is supplied to each block.
- the prediction mode information Pinfo is supplied to the coding unit 614 and the prediction unit 620.
- the conversion information Tinfo is supplied to the coding unit 614, the conversion quantization unit 613, and the inverse quantization inverse conversion unit 616.
- the filter information Finfo is supplied to the in-loop filter unit 618.
- ⁇ Sort buffer> Each field (input image) of moving image data is input to the image coding device 600 in the reproduction order (display order).
- the sorting buffer 611 acquires and holds (stores) each input image in its reproduction order (display order).
- the sorting buffer 611 sorts the input images in the coding order (decoding order) or divides the input images into blocks of processing units based on the control of the control unit 601.
- the sorting buffer 611 supplies each processed input image to the calculation unit 612.
- the calculation unit 612 subtracts the prediction image P supplied from the prediction unit 620 from the image corresponding to the block of the processing unit supplied from the sorting buffer 611 to derive the residual data D, and converts it into a conversion quantum. It is supplied to the conversion unit 613.
- the conversion quantization unit 613 performs processing related to coefficient conversion and quantization. For example, the conversion quantization unit 613 acquires the residual data D supplied from the calculation unit 612. In the non-lossless coding mode, the conversion quantization unit 613 performs coefficient conversion such as orthogonal conversion on the residual data D, and derives the conversion coefficient Coeff. The conversion quantization unit 613 scales (quantizes) the conversion coefficient Coeff and derives the quantization coefficient level. The conversion quantization unit 613 supplies the quantization coefficient level to the coding unit 614 and the inverse quantization inverse conversion unit 616.
- the conversion quantization unit 613 can skip (omit) coefficient conversion and quantization. In the case of the lossless coding mode, the conversion quantization unit 613 skips the coefficient conversion and quantization, and supplies the acquired residual data D to the coding unit 614 and the inverse quantization inverse conversion unit 616.
- the conversion quantization unit 613 performs these processes under the control of the control unit 601. For example, the conversion quantization unit 613 can perform these processes based on the prediction mode information Pinfo and the conversion information Tinfo supplied from the control unit 601. Further, the rate of quantization performed by the conversion quantization unit 613 is controlled by the rate control unit 621.
- the coding unit 614 has a quantization coefficient level (or residual data D) supplied from the conversion quantization unit 613 and various coding parameters (header information Hinfo, prediction mode information Pinfo, conversion) supplied from the control unit 601. Information Tinfo, filter information Finfo, etc.), information about the filter such as the filter coefficient supplied from the in-loop filter unit 618, and information about the optimum prediction mode supplied from the prediction unit 620 are input.
- the coding unit 614 encodes the quantization coefficient level or the residual data D with entropy coding (lossless coding) such as CABAC (Context-based Adaptive Binary Arithmetic Code) or CAVLC (Context-based Adaptive Variable Length Code). ) To generate a bit string (encoded data). For example, when CABAC is applied, the coding unit 614 performs arithmetic coding using a context model on the quantization coefficient level in the non-lossless coding mode, and generates coded data. Further, in the lossless coding mode, the coding unit 614 performs arithmetic coding on the residual data D in the bypass mode to generate the coded data.
- entropy coding such as CABAC (Context-based Adaptive Binary Arithmetic Code) or CAVLC (Context-based Adaptive Variable Length Code).
- CABAC Context-based Adaptive Binary Arithmetic Code
- CAVLC Context-based Adaptive Vari
- the coding unit 614 derives the residual information Rinfo from the quantization coefficient level and the residual data, encodes the residual information Rinfo, and generates a bit string.
- the coding unit 614 includes the information about the filter supplied from the in-loop filter unit 618 in the filter information Finfo, and includes the information about the optimum prediction mode supplied from the prediction unit 620 in the prediction mode information Pinfo. Then, the coding unit 614 encodes the various coding parameters (header information Hinfo, prediction mode information Pinfo, conversion information Tinfo, filter information Finfo, etc.) described above to generate a bit string.
- the coding unit 614 multiplexes the bit strings of the various information generated as described above to generate the coded data.
- the coding unit 614 supplies the coded data to the storage buffer 615.
- the storage buffer 615 temporarily holds the coded data obtained in the coding unit 614.
- the storage buffer 615 outputs the held coded data as, for example, a bit stream or the like to the outside of the image coding device 600 at a predetermined timing.
- this coded data is transmitted to the decoding side via an arbitrary recording medium, an arbitrary transmission medium, an arbitrary information processing device, or the like. That is, the storage buffer 615 is also a transmission unit that transmits coded data (bit stream).
- Inverse quantization Inverse conversion unit 616 performs processing related to inverse quantization and inverse coefficient conversion. For example, in the non-lossless coding mode, the inverse quantization inverse conversion unit 616 inputs the quantization coefficient level supplied from the conversion quantization unit 613 and the conversion information Tinfo supplied from the control unit 601. Inverse quantization Inverse conversion unit 616 scales (inverse quantization) the value of the quantization coefficient level based on the conversion information Tinfo, and derives the conversion coefficient Coeff. This inverse quantization is an inverse process of quantization performed in the conversion quantization unit 613.
- the inverse quantization inverse conversion unit 616 performs inverse coefficient conversion (for example, inverse orthogonal transformation) with respect to the conversion coefficient Coeff based on the conversion information Tinfo, and derives the residual data D'. It should be noted that this inverse coefficient conversion is an inverse process of the coefficient conversion performed in the conversion quantization unit 613.
- the inverse quantization inverse conversion unit 616 supplies the derived residual data D'to the arithmetic unit 617.
- the inverse quantization inverse conversion unit 616 can skip (omit) this inverse quantization and inverse coefficient conversion. For example, when the lossless coding mode is applied, the inverse quantization inverse conversion unit 616 inputs the residual data D supplied from the conversion quantization unit 613 and the conversion information Tinfo supplied from the control unit 601. To do. The inverse quantization inverse conversion unit 616 skips the inverse quantization and the inverse coefficient conversion, and supplies the residual data D (as the residual data D') to the arithmetic unit 617.
- the inverse quantization inverse conversion unit 616 Since the inverse quantization inverse conversion unit 616 is the same as the inverse quantization inverse conversion unit (described later) on the decoding side, the inverse quantization inverse conversion unit 616 will be described for the decoding side (described later). Can be applied.
- the calculation unit 617 inputs the residual data D'supplied from the inverse quantization inverse conversion unit 616 and the prediction image P supplied from the prediction unit 620.
- the calculation unit 617 adds the residual data D'and the predicted image corresponding to the residual data D'to derive a locally decoded image.
- the calculation unit 617 supplies the derived locally decoded image to the in-loop filter unit 618 and the frame memory 619.
- the in-loop filter unit 618 performs processing related to the in-loop filter processing.
- the in-loop filter unit 618 inputs the locally decoded image supplied from the calculation unit 617, the filter information Finfo supplied from the control unit 601 and the input image (original image) supplied from the sorting buffer 611.
- the information input to the in-loop filter unit 618 is arbitrary, and information other than these information may be input. For example, if necessary, information such as prediction mode, motion information, code amount target value, quantization parameter QP, picture type, block (CU, CTU, etc.) may be input to the in-loop filter unit 618. Good.
- the in-loop filter unit 618 appropriately filters the locally decoded image based on the filter information Finfo.
- the in-loop filter unit 618 also uses an input image (original image) and other input information for the filter processing, if necessary.
- the in-loop filter unit 618 includes a bilateral filter, a deblocking filter (DBF (DeBlocking Filter)), an adaptive offset filter (SAO (Sample Adaptive Offset)), and an adaptive loop filter (ALF (Adaptive Loop Filter)).
- DPF DeBlocking Filter
- SAO Sample Adaptive Offset
- ALF adaptive Loop Filter
- the filter processing performed by the in-loop filter unit 618 is arbitrary and is not limited to the above example.
- the in-loop filter unit 618 may apply a Wiener filter or the like.
- the in-loop filter unit 618 supplies the filtered locally decoded image to the frame memory 619.
- the in-loop filter unit 618 supplies information about the filter to the coding unit 614.
- the frame memory 619 performs processing related to storage of data related to images. For example, the frame memory 619 receives the locally decoded image supplied from the arithmetic unit 617 and the filtered locally decoded image supplied from the in-loop filter unit 618 as inputs, and holds (stores) them. Further, the frame memory 619 reconstructs and holds the decoded image for each picture unit using the locally decoded image (stored in the buffer in the frame memory 619). The frame memory 619 supplies the decoded image (or a part thereof) to the prediction unit 620 in response to the request of the prediction unit 620.
- the prediction unit 620 performs processing related to the generation of the prediction image.
- the prediction unit 620 receives the prediction mode information Pinfo supplied from the control unit 601, the input image (original image) supplied from the sorting buffer 611, and the decoded image (or a part thereof) read from the frame memory 619. Input.
- the prediction unit 620 uses the prediction mode information Pinfo and the input image (original image) to perform prediction processing such as inter-prediction and intra-prediction, makes a prediction by referring to the decoded image as a reference image, and based on the prediction result. Motion compensation processing is performed to generate a predicted image.
- the prediction unit 620 supplies the generated prediction image to the calculation unit 612 and the calculation unit 617. Further, the prediction unit 620 supplies information regarding the prediction mode selected by the above processing, that is, the optimum prediction mode, to the coding unit 614 as needed.
- the rate control unit 621 performs processing related to rate control. For example, the rate control unit 621 controls the rate of the quantization operation of the conversion quantization unit 613 based on the code amount of the coded data stored in the storage buffer 615 so that overflow or underflow does not occur.
- the conversion quantization unit 613 of the image coding device 600 As the conversion quantization unit 613 of the image coding device 600 as described above, the conversion quantization device 200 described above in the first to third embodiments may be applied. That is, the conversion quantization unit 613 may have a configuration as shown in FIG. 7, FIG. 14, or FIG. 20. By doing so, the conversion quantization unit 613 can obtain the above-mentioned effects in the first to third embodiments. That is, the image coding apparatus 600 can obtain the above-mentioned effects in the first embodiment to the third embodiment in the decoding process.
- step S601 the sorting buffer 611 is controlled by the control unit 601 to sort the frame order of the input moving image data from the display order to the coding order.
- step S602 the control unit 601 determines (sets) the coding parameters for the input image held by the sorting buffer 611.
- step S603 the control unit 601 sets a processing unit (block division is performed) for the input image held by the sorting buffer 611.
- the prediction unit 620 performs prediction processing and generates a prediction image or the like of the optimum prediction mode. For example, in this prediction process, the prediction unit 620 performs intra-prediction to generate a prediction image or the like of the optimum intra-prediction mode, and performs inter-prediction to generate a prediction image or the like of the optimum inter-prediction mode.
- the optimum prediction mode is selected from among them based on the cost function value and the like.
- step S605 the calculation unit 612 calculates the difference between the input image and the prediction image of the optimum mode selected by the prediction processing in step S604. That is, the calculation unit 612 generates the residual data D between the input image and the predicted image.
- the amount of residual data D thus obtained is smaller than that of the original image data. Therefore, the amount of data can be compressed as compared with the case where the image is encoded as it is.
- step S606 the conversion quantization unit 613 performs a conversion quantization process on the residual data D generated by the process of step S605 according to the conversion mode information generated in step S601.
- step S607 the inverse quantization inverse conversion unit 616 performs the inverse quantization inverse conversion process.
- This inverse quantization reverse conversion process is the reverse process of the conversion quantization process in step S606, and the same process is executed in the image decoding apparatus 500 described above. Therefore, the dequantization reverse conversion process will be described when the decoding side (image decoding device 500) is described. Then, the description can be applied to this inverse quantization inverse conversion process (step S607).
- the inverse quantization inverse conversion unit 616 performs inverse quantization and inverse coefficient conversion on the input coefficient data (quantization coefficient level or residual data r (D)) as appropriate, and the residual. Generate data D'.
- step S608 the calculation unit 617 locally decodes the residual data D'obtained by the inverse quantization inverse conversion process of step S607 by adding the predicted image obtained by the prediction process of step S604. Generate the decoded image.
- step S609 the in-loop filter unit 618 performs in-loop filter processing on the locally decoded decoded image derived by the process of step S608.
- step S610 the frame memory 619 stores the locally decoded decoded image derived by the process of step S608 and the locally decoded decoded image filtered in step S609.
- step S611 the coding unit 614 encodes the quantization coefficient level or the residual data D obtained by the conversion quantization process of step S606 to generate the coded data.
- the coding unit 614 encodes various coding parameters (header information Hinfo, prediction mode information Pinfo, conversion information Tinfo). Further, the coding unit 614 derives the residual information RInfo from the quantization coefficient level and the residual data D, and encodes the residual information RInfo.
- step S612 the storage buffer 615 stores the coded data thus obtained and outputs it, for example, as a bit stream to the outside of the image coding device 600.
- This bit stream is transmitted to the decoding side via, for example, a transmission line or a recording medium.
- the rate control unit 621 performs rate control as needed.
- the conversion quantization process (step S606) of the image coding apparatus 600 as described above the inverse quantization inverse conversion process described above in the first to third embodiments may be applied. That is, the inverse quantization inverse conversion process may be executed in the flow as shown in the flowchart shown in FIG. 8, FIG. 15, or FIG.
- the conversion quantization unit 613 can obtain the above-mentioned effects in the first to third embodiments. That is, the image coding apparatus 600 can obtain the above-mentioned effects in the first embodiment to the third embodiment in the image coding process.
- FIG. 26 is a block diagram showing a configuration example of computer hardware that executes the above-mentioned series of processes programmatically.
- the CPU Central Processing Unit
- ROM Read Only Memory
- RAM Random Access Memory
- the input / output interface 810 is also connected to the bus 804.
- An input unit 811, an output unit 812, a storage unit 813, a communication unit 814, and a drive 815 are connected to the input / output interface 810.
- the input unit 811 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like.
- the output unit 812 includes, for example, a display, a speaker, an output terminal, and the like.
- the storage unit 813 includes, for example, a hard disk, a RAM disk, a non-volatile memory, or the like.
- the communication unit 814 is composed of, for example, a network interface.
- the drive 815 drives a removable medium 821 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
- the CPU 801 loads the program stored in the storage unit 813 into the RAM 803 via the input / output interface 810 and the bus 804 and executes the above-described series. Is processed.
- the RAM 803 also appropriately stores data and the like necessary for the CPU 801 to execute various processes.
- the program executed by the computer can be recorded and applied to the removable media 821 as a package media or the like, for example.
- the program can be installed in the storage unit 813 via the input / output interface 810 by attaching the removable media 821 to the drive 815.
- This program can also be provided via wired or wireless transmission media such as local area networks, the Internet, and digital satellite broadcasting.
- the program can be received by the communication unit 814 and installed in the storage unit 813.
- this program can be installed in advance in ROM 802 or storage unit 813.
- This technique can be applied to any image coding / decoding method. That is, as long as it does not contradict the above-mentioned technology, the specifications of various processes related to image coding / decoding such as conversion (inverse transformation), quantization (inverse quantization), coding (decoding), and prediction are arbitrary. It is not limited to the example. In addition, some of these processes may be omitted as long as they do not contradict the present technology described above.
- this technology can be applied to a multi-viewpoint image coding / decoding system that encodes / decodes a multi-viewpoint image including images of a plurality of viewpoints (views).
- the present technology may be applied to the coding / decoding of each viewpoint (view).
- this technology is applied to a hierarchical image coding (scalable coding) / decoding system that encodes / decodes a hierarchical image that is layered (layered) so as to have a scalability function for a predetermined parameter. can do.
- the present technology may be applied in the coding / decoding of each layer.
- the inverse quantization inverse conversion device 100, the conversion quantization apparatus 200, the image decoding apparatus 500, and the image coding apparatus 600 have been described, but the present technology has an arbitrary configuration. Can be applied to.
- this technology is a transmitter or receiver (for example, a television receiver or mobile phone) for satellite broadcasting, cable broadcasting such as cable TV, distribution on the Internet, and distribution to terminals by cellular communication, or It can be applied to various electronic devices such as devices (for example, hard disk recorders and cameras) that record images on media such as optical disks, magnetic disks, and flash memories, and reproduce images from these storage media.
- devices for example, hard disk recorders and cameras
- a processor as a system LSI (Large Scale Integration) or the like (for example, a video processor), a module using a plurality of processors (for example, a video module), a unit using a plurality of modules (for example, a video unit)
- a processor as a system LSI (Large Scale Integration) or the like
- a module using a plurality of processors for example, a video module
- a unit using a plurality of modules for example, a video unit
- it can be implemented as a configuration of a part of the device, such as a set (for example, a video set) in which other functions are added to the unit.
- this technology can be applied to a network system composed of a plurality of devices.
- the present technology may be implemented as cloud computing that is shared and jointly processed by a plurality of devices via a network.
- this technology is implemented in a cloud service that provides services related to images (moving images) to arbitrary terminals such as computers, AV (AudioVisual) devices, portable information processing terminals, and IoT (Internet of Things) devices. You may try to do it.
- the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..
- Systems, devices, processing units, etc. to which this technology is applied can be used in any field such as transportation, medical care, crime prevention, agriculture, livestock industry, mining, beauty, factories, home appliances, weather, nature monitoring, etc. .. Moreover, the use is arbitrary.
- this technology can be applied to systems and devices used for providing ornamental contents and the like.
- the present technology can be applied to systems and devices used for traffic such as traffic condition supervision and automatic driving control.
- the present technology can be applied to systems and devices used for security purposes.
- the present technology can be applied to a system or device used for automatic control of a machine or the like.
- the present technology can be applied to systems and devices used for agriculture and livestock industry.
- the present technology can also be applied to systems and devices for monitoring natural conditions such as volcanoes, forests and oceans, and wildlife. Further, for example, the present technology can be applied to systems and devices used for sports.
- the "flag” is information for identifying a plurality of states, and is not only information used for identifying two states of true (1) or false (0), but also three or more states. It also contains information that can identify the state. Therefore, the value that this "flag” can take may be, for example, 2 values of 1/0 or 3 or more values. That is, the number of bits constituting this "flag” is arbitrary, and may be 1 bit or a plurality of bits.
- the identification information (including the flag) is assumed to include not only the identification information in the bitstream but also the difference information of the identification information with respect to a certain reference information in the bitstream. In, the "flag” and “identification information” include not only the information but also the difference information with respect to the reference information.
- various information (metadata, etc.) regarding the coded data may be transmitted or recorded in any form as long as it is associated with the coded data.
- the term "associate" means, for example, to make the other data available (linkable) when processing one data. That is, the data associated with each other may be combined as one data or may be individual data.
- the information associated with the coded data (image) may be transmitted on a transmission path different from the coded data (image).
- the information associated with the coded data (image) may be recorded on a recording medium (or another recording area of the same recording medium) different from the coded data (image). Good.
- this "association" may be a part of the data, not the entire data. For example, an image and information corresponding to the image may be associated with each other in an arbitrary unit such as a plurality of frames, one frame, or a part within the frame.
- the embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.
- the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units).
- the configurations described above as a plurality of devices (or processing units) may be collectively configured as one device (or processing unit).
- a configuration other than the above may be added to the configuration of each device (or each processing unit).
- a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit). ..
- the above-mentioned program may be executed in any device.
- the device may have necessary functions (functional blocks, etc.) so that necessary information can be obtained.
- each step of one flowchart may be executed by one device, or may be shared and executed by a plurality of devices.
- the plurality of processes may be executed by one device, or may be shared and executed by a plurality of devices.
- a plurality of processes included in one step can be executed as processes of a plurality of steps.
- the processes described as a plurality of steps can be collectively executed as one step.
- the processing of the steps for writing the program may be executed in chronological order in the order described in the present specification, and may be executed in parallel or in calls. It may be executed individually at the required timing such as when it is broken. That is, as long as there is no contradiction, the processing of each step may be executed in an order different from the above-mentioned order. Further, the processing of the step for writing this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.
- a plurality of technologies related to this technology can be independently implemented independently as long as there is no contradiction.
- any plurality of the present technologies can be used in combination.
- some or all of the techniques described in any of the embodiments may be combined with some or all of the techniques described in other embodiments. It is also possible to carry out a part or all of any of the above-mentioned techniques in combination with other techniques not described above.
- the present technology can also have the following configurations.
- An inverse quantization unit that performs inverse quantization with respect to the quantization coefficient using quantization parameters according to whether conversion skip is applied, and When the conversion skip is not applied, the inverse quantization unit performs inverse coefficient conversion on the conversion coefficient generated by the inverse quantization to generate a predicted residual which is a residual between the image and the predicted image of the image.
- An image processing apparatus including an inverse conversion unit that skips the inverse coefficient conversion when the conversion skip is applied.
- the inverse quantization unit corrects the quantization parameter and uses the corrected quantization parameter.
- the image processing apparatus according to (1) which performs the inverse quantization.
- an inverse quantization normalization unit that sets the sign of the rounding offset based on the sign of the conversion coefficient and normalizes the conversion coefficient using the rounding offset.
- the image processing apparatus wherein the inverse conversion unit performs the inverse coefficient conversion on the conversion coefficient normalized by the inverse quantization normalization unit when the conversion skip is not applied.
- the sign of the rounding offset is set based on the sign of the predicted residual, the predicted residual is normalized by using the rounding offset, and the conversion skip is applied.
- the image processing according to (1) further comprising an inverse conversion normalization unit that sets the sign of the rounding offset based on the sign of the conversion coefficient and normalizes the conversion coefficient using the rounding offset. apparatus.
- Inverse quantization with respect to the quantization coefficient is performed using the quantization parameter according to whether conversion skip is applied.
- an inverse coefficient conversion is performed on the conversion coefficient generated by the inverse quantization to generate a predicted residual which is a residual between the image and the predicted image of the image, and the conversion skip is applied.
- An image processing method that skips the inverse coefficient conversion.
- An inverse quantization unit that performs inverse quantization with respect to the quantization coefficient
- An inverse quantization normalization unit that normalizes the conversion coefficient generated by the inverse quantization by the inverse quantization unit using a scaling parameter depending on whether or not the conversion skip is applied.
- the conversion coefficient normalized by the inverse quantization normalization unit is subjected to inverse coefficient conversion to generate a predicted residual which is a residual between the image and the predicted image of the image.
- An image processing device including an inverse conversion unit that skips the inverse coefficient conversion when the conversion skip is applied.
- the inverse quantization normalization unit normalizes the conversion coefficient using the scaling parameter that does not reflect the decompression processing in the conversion skip, and applies the conversion skip. In the case, the conversion coefficient is normalized by using the scaling parameter reflecting the expansion process.
- the inverse conversion unit converts the inverse coefficient with respect to the conversion coefficient normalized by the inverse quantization normalization unit using the scaling parameter that does not reflect the decompression processing.
- the inverse quantization normalization unit normalizes the conversion coefficient using the scaling parameter that reflects the normalization processing for the predicted residual by the inverse conversion normalization unit (8). ).
- the image processing apparatus (14)
- the inverse quantization normalization unit uses the scaling parameter including a value according to the inverse quantization and the scaling list and a value according to the application of the inverse dependence quantization.
- Inverse quantization with respect to the quantization coefficient is performed.
- the conversion coefficients generated by the inverse quantization are normalized using scaling parameters depending on whether conversion skip is applied.
- the inverse coefficient conversion is performed on the normalized conversion coefficient to generate a predicted residual which is a residual between the image and the predicted image of the image, and the conversion skip is applied. If the image processing method skips the inverse coefficient conversion.
- the quantization normalization unit When the conversion skip is not applied, the quantization normalization unit normalizes the quantization coefficient using the scaling parameter which does not reflect the decompression processing in the conversion skip, and when the conversion skip is applied, The image processing apparatus according to (17), wherein the quantization coefficient is normalized by using the scaling parameter reflecting the decompression processing. (19) When the conversion skip is not applied, a conversion normalization unit for normalizing the conversion coefficient is further provided. When the conversion skip is applied, the quantization normalization unit normalizes the quantization coefficient using the scaling parameter reflecting the normalization processing for the conversion coefficient by the conversion normalization unit (17). The image processing apparatus described.
- 100 Inverse Quantization Inverse Converter 101 Inverse Quantization Unit, 102 Inverse Quantization Unit, 111 Inverse Quantization Processing Unit, 112 Normalization Unit, 121 Selection Unit, 122 Inverse Conversion Processing Unit, 123 Selection Unit, 124 Normalization Unit, 125 decompression processing unit, 200 conversion quantization unit, 201 conversion unit, 202 quantization unit, 211 selection unit, 212 conversion processing unit, 213 selection unit, 214 normalization unit, 215 decompression processing unit, 221 quantization processing unit, 222.
- Normalization unit 500 image decoding device, 513 inverse quantization inverse conversion unit, 600 image coding device, 613 conversion quantization unit, 616 inverse quantization inverse conversion unit
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
Description
1.変換スキップによるニアロスレス符号化
2.第1の実施の形態(パラメータ制御)
3.第2の実施の形態(変換スキップ時の伸張処理のマージ)
4.第3の実施の形態(変換スキップ時の正規化処理のマージ)
5.第4の実施の形態(画像復号装置)
6.第5の実施の形態(画像符号化装置)
7.付記
<技術内容・技術用語をサポートする文献等>
本技術で開示される範囲は、実施の形態に記載されている内容だけではなく、出願当時において公知となっている以下の非特許文献等に記載されている内容や以下の非特許文献において参照されている他の文献の内容等も含まれる。
非特許文献2:(上述)
非特許文献3:Benjamin Bross, Jianle Chen, Shan Liu, "Versatile Video Coding (Draft 5)", N1001-v10, m48053, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH, 19-27 Mar. 2019
非特許文献4:Jianle Chen, Yan Ye, Seung Hwan Kim, "Algorithm description for Versatile Video Coding and Test Model 5 (VTM 5)", JVET-N1002-v2, m48054, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH, 19-27 Mar. 2019
非特許文献5:Benjamin Bross, Jianle Chen, Shan Liu, "Versatile Video Coding (Draft 6)", JVET-O2001-vE, m49908, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019
非特許文献6:Jianle Chen, Yan Ye, Seung Hwan Kim, "Algorithm description for Versatile Video Coding and Test Model 6 (VTM 6)", JVET-O2002-v2, m49914, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019
非特許文献7:Tsung-Chuan Ma, Yi-Wen Chen, Xiaoyu Xiu, Xianglin Wang, "Modifications to support the lossless coding", JVET-O0591, m48730, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019
非特許文献8:Hyeongmun Jang, Junghak Nam, Naeri Park, Jungah Choi Seunghwan Kim, Jaehyun Lim, "Comments on transform quantization bypassed mode", JVET-O0584, m48723, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019
非特許文献9:Tangi Poirier, Fabrice Le Leannec, Karam Naser, Edouard Francois, "On lossless coding for VVC" JVET-O0460, m48583, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019
非特許文献10:Recommendation ITU-T H.264 (04/2017) "Advanced video coding for generic audiovisual services", April 2017
非特許文献11:Recommendation ITU-T H.265 (02/18) "High efficiency video coding", february 2018
非特許文献2には、非特許文献1の画像符号化において、変換量子化バイパスを使用して係数変換や量子化等をスキップ(省略)し、予測残差を可逆符号化する符号化方法であるロスレス符号化が開示されている。
<逆量子化逆変換のパラメータ制御>
そこで、逆量子化逆変換処理において、図1の表の上から1番目の段(最上段)に示されるように、損失する情報量を低減させるように、逆量子化逆変換に関するパラメータを設定する(方法1-1)。
図2は、本技術を適用した画像処理装置の一態様である逆量子化逆変換装置の構成の一例を示すブロック図である。図2に示される逆量子化逆変換装置100は、量子化係数を逆量子化逆変換し、予測残差を生成する装置である。
次に、この逆量子化逆変換装置100により実行される逆量子化逆変換処理の流れの例を、図3のフローチャートを参照して説明する。
上述のように、逆量子化に関するパラメータを制御するようにしてもよい。例えば、画像処理において、変換スキップを適用するかに応じた量子化パラメータを用いて、量子化係数に対する逆量子化を行うようにしてもよい。
制御対象とする逆量子化に関するパラメータは、量子化パラメータ以外であってもよい。例えば、図1の表の上から3番目の段に示されるように、例えば、入力値である変換係数の符号に基づいて逆量子化における正規化処理のラウンディングオフセットの符号が設定されてもよい(方法1-1-2)。
逆変換に関するパラメータを制御対象としてもよい。例えば、図1の表の上から4番目の段に示されるように、例えば、入力値である予測残差の符号に基づいて逆係数変換における正規化処理のラウンディングオフセットの符号が設定されてもよい(方法1-1-3)。
<1-1-1:量子化パラメータの制御>、<1-1-2:ラウンディングオフセットの符号制御>、および<1-1-3:ラウンディングオフセットの符号制御>のそれぞれにおいて説明した各方法は、適宜、組み合わせて適用することができる。より多くの方法を組み合わせることにより、逆量子化逆変換装置100は、逆量子化逆変換処理における情報の損失をより抑制することができる。つまり、逆量子化逆変換装置100は、上述の各方法をより多く組み合わせることにより、"変換スキップ+QP4"アプローチによる復号を、より可逆復号(ロスレス符号化)に近づけることができる。
逆量子化逆変換処理の場合と同様に、変換量子化処理において、図1の表の上から5番目の段に示されるように、損失する情報量を低減させるように、変換量子化に関するパラメータを設定してもよい(方法1-2)。
図7は、本技術を適用した画像処理装置の一態様である変換量子化装置の構成の一例を示すブロック図である。図7に示される変換量子化装置200は、予測残差を変換量子化し、量子化係数を生成する装置である。
次に、この変換量子化装置200により実行される変換量子化処理の流れの例を、図8のフローチャートを参照して説明する。
上述のように、量子化に関するパラメータを制御するようにしてもよい。例えば、画像処理において、変換スキップを適用するかに応じた量子化パラメータを用いて、変換係数に対する量子化を行うようにしてもよい。
制御対象とする量子化に関するパラメータは、量子化パラメータ以外であってもよい。例えば、図1の表の上から7番目の段に示されるように、例えば、入力値である量子化係数の符号に基づいて量子化における正規化処理のラウンディングオフセットの符号が設定されてもよい(方法1-2-2)。
係数変換に関するパラメータを制御対象としてもよい。例えば、図1の表の上から8番目の段に示されるように、例えば、入力値である変換係数の符号に基づいて係数変換における正規化処理のラウンディングオフセットの符号が設定されてもよい(方法1-2-3)。
<1-2-1:量子化パラメータの制御>、<1-2-2:ラウンディングオフセットの符号制御>、および<1-2-3:ラウンディングオフセットの符号制御>のそれぞれにおいて説明した各方法は、適宜、組み合わせて適用することができる。より多くの方法を組み合わせることにより、変換量子化装置200は、変換量子化処理における情報の損失をより抑制することができる。つまり、変換量子化装置200は、上述の各方法をより多く組み合わせることにより、"変換スキップ+QP4"アプローチによる符号化を、より可逆符号化(ロスレス符号化)に近づけることができる。
<変換スキップ時の伸張処理のマージ>
変換スキップを適用する場合、冗長な処理が存在し、それにより逆量子化逆変換処理の負荷が増大するおそれがあった。例えば、変換スキップ時のデータパスにおいて、逆直交変換後の予測残差のダイナミックレンジと逆直交変換をスキップした場合の予測残差のダイナミックレンジを揃えるために、変換スキップ固有のTS係数伸張処理(<<tsShift)がある。しかしながら、このTS係数伸張処理は、逆量子化プロセスにおける正規化処理(>>bdShift1)と逆方向のビットシフトである。したがって、これらの処理を互いに異なる処理として行うことは冗長であった。
図9は、この場合の逆量子化逆変換装置100の構成の一例を示すブロック図である。なお、図9においては、処理部やデータの流れ等の主なものを示しており、図9に示されるものが全てとは限らない。つまり、逆量子化逆変換装置100において、図9においてブロックとして示されていない処理部が存在したり、図9において矢印等として示されていない処理やデータの流れが存在したりしてもよい。
次に、この場合の逆量子化逆変換処理の流れの例を、図10のフローチャートを参照して説明する。
逆量子化における正規化処理と、変換スキップにおける伸張処理とを別の処理として行う場合、その正規化処理に用いられるスケーリングパラメータbdShift1と、その伸張処理に用いられるスケーリングパラメータtsShiftとは、例えば、図11のAに示されるセマンティクスのように導出される。
変換スキップフラグが変換スキップの適用を示す場合(transform_skip_flag == IS_SKIP)、スケーリングパラメータbdShift1の変換ブロックサイズに基づく成分を量子化マトリックスに応じた値に設定してもよい。
変換量子化処理においても、上述の逆量子化逆変換処理の場合と同様である。つまり、変換スキップを適用する場合、冗長な処理が存在し、それにより変換量子化処理の負荷が増大するおそれがあった。例えば、変換スキップ時のデータパスにおいて、直交変換後の変換係数のダイナミックレンジと直交変換をスキップした場合の予測残差のダイナミックレンジを揃えるために、変換スキップ固有のTS係数伸張処理(<<fwdtsShift)がある。しかしながら、このTS係数伸張処理は、量子化プロセスにおける正規化処理(>>fwdbdShift2)と逆方向のビットシフトである。したがって、これらの処理を互いに異なる処理として行うことは冗長であった。
図14は、この場合の変換量子化装置200の構成の一例を示すブロック図である。なお、図14においては、処理部やデータの流れ等の主なものを示しており、図14に示されるものが全てとは限らない。つまり、変換量子化装置200において、図14においてブロックとして示されていない処理部が存在したり、図14において矢印等として示されていない処理やデータの流れが存在したりしてもよい。
次に、この場合の変換量子化処理の流れの例を、図15のフローチャートを参照して説明する。
量子化における正規化処理と、変換スキップにおける伸張処理とを別の処理として行う場合、その正規化処理に用いられるスケーリングパラメータfwdbdShift2と、その伸張処理に用いられるスケーリングパラメータfwdtsShiftとは、図11のAの例と同様に導出される。
また、この変換量子化処理の場合も、上述した逆量子化逆変換処理の場合と同様に、変換スキップフラグが変換スキップの適用を示す場合(transform_skip_flag == IS_SKIP)、スケーリングパラメータfwdbdShift2の変換ブロックサイズに基づく成分を量子化マトリックスに応じた値に設定してもよい。
<変換スキップ時の正規化処理のマージ>
例えば、変換スキップ時のデータパスにおいて、逆係数変換後の予測残差のダイナミックレンジと逆係数変換をスキップした場合の変換係数(つまり予測残差)のダイナミックレンジを揃えるために、変換スキップ固有のTS係数伸張処理(<<tsShift)、および”bdShift2”による正規化処理がある。しかしながら、このTS係数身長処理および正規化処理におけるビットシフトは、逆量子化プロセスにおける正規化処理(>>bdShift1)におけるビットシフトとマージすることができる。つまり、これらのビットシフトをそれぞれ行うことは冗長であった。
図16は、この場合の逆量子化逆変換装置100の構成の一例を示すブロック図である。なお、図16においては、処理部やデータの流れ等の主なものを示しており、図16に示されるものが全てとは限らない。つまり、逆量子化逆変換装置100において、図16においてブロックとして示されていない処理部が存在したり、図16において矢印等として示されていない処理やデータの流れが存在したりしてもよい。
次に、この場合の逆量子化逆変換処理の流れの例を、図17のフローチャートを参照して説明する。
逆量子化における正規化処理と、変換スキップにおける伸張処理と、逆係数変換における正規化処理とを別の処理として行う場合、逆量子化における正規化処理に用いられるスケーリングパラメータbdShift1と、伸張処理に用いられるスケーリングパラメータtsShiftと、逆係数変換における正規化処理に用いられるスケーリングパラメータbdShift2とは、例えば、図18のAに示されるセマンティクスのように導出される。
変換量子化処理においても、上述の逆量子化逆変換処理の場合と同様である。つまり、変換スキップを適用する場合、冗長な処理が存在し、それにより変換量子化処理の負荷が増大するおそれがあった。例えば、変換スキップ時のデータパスにおいて、直交変換後の変換係数のダイナミックレンジと直交変換をスキップした場合の予測残差のダイナミックレンジを揃えるために、変換スキップ固有のTS係数伸張処理(<<fwdtsShift)および"fwdbdShift1"による正規化処理がある。しかしながら、このTS係数身長処理および正規化処理におけるビットシフトは、量子化プロセスにおける正規化処理(>>fwdbdShift2)におけるビットシフトとマージすることができる。つまり、これらのビットシフトをそれぞれ行うことは冗長であった。
図20は、この場合の変換量子化装置200の構成の一例を示すブロック図である。なお、図20においては、処理部やデータの流れ等の主なものを示しており、図20に示されるものが全てとは限らない。つまり、変換量子化装置200において、図20においてブロックとして示されていない処理部が存在したり、図20において矢印等として示されていない処理やデータの流れが存在したりしてもよい。
次に、この場合の変換量子化処理の流れの例を、図21のフローチャートを参照して説明する。
正規化部214が、変換スキップを適用しない場合、変換係数を正規化し、正規化部222が、変換スキップを適用する場合、正規化部214による予測残差に対する正規化処理を反映したスケーリングパラメータを用いて量子化係数を正規化するようにする。
<画像復号装置>
第1の実施の形態乃至第3の実施の形態等において上述した本技術は、任意の装置、デバイス、システム等に適用することができる。例えば、画像データの符号化データを復号する画像復号装置に、本技術を適用することができる。画像データを符号化する画像符号化装置に適用することもできる。
制御部501は、復号の制御に関する処理を行う。例えば、制御部501は、ビットストリームに含まれる符号化パラメータ(ヘッダ情報Hinfo、予測モード情報Pinfo、変換情報Tinfo、残差情報Rinfo、フィルタ情報Finfoなど)を、復号部512を介して取得する。また、制御部501は、ビットストリームに含まれない符号化パラメータを推定しうる。さらに、制御部501は、取得した(または推定した)符号化パラメータに基づいて、画像復号装置500の各処理部(蓄積バッファ511乃至予測部518)を制御することにより、復号を制御する。
ヘッダ情報Hinfoは、例えば、VPS(Video Parameter Set)/SPS(Sequence ParameterSet)/PPS(Picture Parameter Set)/PH(ピクチャヘッダ)/SH(スライスヘッダ)などのヘッダ情報を含む。ヘッダ情報Hinfoには、例えば、画像サイズ(横幅PicWidth、縦幅PicHeight)、ビット深度(輝度bitDepthY, 色差bitDepthC)、色差アレイタイプChromaArrayType、CUサイズの最大値MaxCUSize/最小値MinCUSize、4分木分割(Quad-tree分割ともいう)の最大深度MaxQTDepth/最小深度MinQTDepth、2分木分割(Binary-tree分割)の最大深度MaxBTDepth/最小深度MinBTDepth、変換スキップブロックの最大値MaxTSSize(最大変換スキップブロックサイズともいう)、各符号化ツールのオンオフフラグ(有効フラグともいう)などを規定する情報が含まれる。
予測モード情報Pinfoには、例えば、処理対象PB(予測ブロック)のサイズ情報PBSize(予測ブロックサイズ)、イントラ予測モード情報IPinfo、動き予測情報MVinfo等の情報が含まれる。
変換情報Tinfoには、例えば、以下の情報が含まれる。もちろん、変換情報Tinfoに含まれる情報は任意であり、これらの情報以外の情報が含まれるようにしてもよい。
変換スキップフラグ(ts_flag):(逆)プライマリ変換および(逆)セカンダリ変換をスキップか否かを示すフラグである。
スキャン識別子(scanIdx)
量子化パラメータ(qp)
量子化マトリックス(scaling_matrix):例えば、JCTVC-W1005, 7.3.4 Scaling list data syntax
残差情報Rinfo(例えば、JCTVC-W1005の7.3.8.11 Residual Coding syntaxを参照)には、例えば以下のシンタックスが含まれる。
last_sig_coeff_x_pos:ラスト非ゼロ係数X座標
last_sig_coeff_y_pos:ラスト非ゼロ係数Y座標
coded_sub_block_flag:サブブロック非ゼロ係数有無フラグ
sig_coeff_flag:非ゼロ係数有無フラグ
gr1_flag:非ゼロ係数のレベルが1より大きいかを示すフラグ(GR1フラグとも呼ぶ)
gr2_flag:非ゼロ係数のレベルが2より大きいかを示すフラグ(GR2フラグとも呼ぶ)
sign_flag:非ゼロ係数の正負を示す符号(サイン符号とも呼ぶ)
coeff_abs_level_remaining:非ゼロ係数の残余レベル(非ゼロ係数残余レベルとも呼ぶ)
など。
フィルタ情報Finfoには、例えば、以下に示す各フィルタ処理に関する制御情報が含まれる。
画素適応オフセット(SAO)に関する制御情報
適応ループフィルタ(ALF)に関する制御情報
その他の線形・非線形フィルタに関する制御情報
蓄積バッファ511は、画像復号装置500に入力されたビットストリームを取得し、保持(記憶)する。蓄積バッファ511は、所定のタイミングにおいて、または、所定の条件が整う等した場合、蓄積しているビットストリームに含まれる符号化データを抽出し、復号部512に供給する。
復号部512は、画像の復号に関する処理を行う。例えば、復号部512は、蓄積バッファ511から供給される符号化データを入力とし、シンタックステーブルの定義に沿って、そのビット列から、各シンタックス要素のシンタックス値をエントロピ復号(可逆復号)し、パラメータを導出する。
逆量子化逆変換部513は、逆量子化および逆係数変換に関する処理を行う。例えば、非ロスレス符号化モードの場合、逆量子化逆変換部513は、復号部512から供給される量子化係数levelを取得する。逆量子化逆変換部513は、その取得した量子化係数levelをスケーリング(逆量子化)し、変換係数Coeffを導出する。逆量子化逆変換部513は、その変換係数Coeffに対して、例えば逆直交変換等の逆係数変換を行い、残差データD'を導出する。逆量子化逆変換部513は、その残差データD'を演算部514に供給する。
演算部514は、画像に関する情報の加算に関する処理を行う。例えば、演算部514は、逆量子化逆変換部513から供給される残差データD'と、予測部518から供給される予測画像とを入力とする。演算部514は、その残差データとその残差データに対応する予測画像(予測信号)とを加算し、局所復号画像を導出する。演算部514は、導出した局所復号画像を、インループフィルタ部515およびフレームメモリ517に供給する。
インループフィルタ部515は、インループフィルタ処理に関する処理を行う。例えば、インループフィルタ部515は、演算部514から供給される局所復号画像と、制御部501から供給されるフィルタ情報Finfoとを入力とする。なお、インループフィルタ部515に入力される情報は任意であり、これらの情報以外の情報が入力されてもよい。
並べ替えバッファ516は、インループフィルタ部515から供給された局所復号画像を入力とし、それを保持(記憶)する。並べ替えバッファ516は、その局所復号画像を用いてピクチャ単位毎の復号画像を再構築し、保持する(バッファ内に格納する)。並べ替えバッファ516は、得られた復号画像を、復号順から再生順に並べ替える。並べ替えバッファ516は、並べ替えた復号画像群を動画像データとして画像復号装置500の外部に出力する。
フレームメモリ517は、画像に関するデータの記憶に関する処理を行う。例えば、フレームメモリ517は、演算部514より供給される局所復号画像を入力とし、ピクチャ単位毎の復号画像を再構築して、フレームメモリ517内のバッファへ格納する。
予測部518は、予測画像の生成に関する処理を行う。例えば、予測部518は、制御部501から供給される予測モード情報Pinfoと、フレームメモリ517から読み出す復号画像(またはその一部)を入力とする。予測部518は、予測モード情報Pinfoに基づいて符号化の際に採用された予測モードで予測処理を行い、復号画像を参照画像として参照して予測画像を生成する。予測部518は、生成した予測画像を演算部514に供給する。
以上のような画像復号装置500の、逆量子化逆変換部513として、第1の実施の形態乃至第3の実施の形態において上述した逆量子化逆変換装置100を適用してもよい。つまり、この逆量子化逆変換部513が、図2、図9、または図16に示されるような構成を有するようにしてもよい。このようにすることにより、逆量子化逆変換部513は、第1の実施の形態乃至第3の実施の形態において上述した効果を得ることができる。つまり、画像復号装置500は、その復号処理において、第1の実施の形態乃至第3の実施の形態において上述した効果を得ることができる。
次に、以上のような画像復号装置500により実行される画像復号処理の流れの例を図23のフローチャートを参照して説明する。
以上のような画像復号装置500の、逆量子化逆変換処理(ステップS505)として、第1の実施の形態乃至第3の実施の形態において上述した逆量子化逆変換処理を適用してもよい。つまり、この逆量子化逆変換処理が、図3、図10、または図17に示されるフローチャートのような流れで実行されるようにしてもよい。このようにすることにより、逆量子化逆変換部513は、第1の実施の形態乃至第3の実施の形態において上述した効果を得ることができる。つまり、画像復号装置500は、画像復号処理において、第1の実施の形態乃至第3の実施の形態において上述した効果を得ることができる。
<画像符号化装置>
図24は、本技術を適用した画像処理装置の一態様である画像符号化装置の構成の一例を示すブロック図である。図24に示される画像符号化装置600は、動画像の画像データを符号化する装置である。例えば、画像符号化装置600は、上述した非特許文献に記載のVVC(Versatile Video Coding)、AVC(Advanced Video Coding)、HEVC(High Efficiency Video Coding)等の符号化方式で動画像の画像データを符号化する。
制御部601は、外部、または予め指定された処理単位のブロックサイズに基づいて、並べ替えバッファ611により保持されている動画像データを処理単位のブロック(CU,PU,TUなど)へ分割する。また、制御部601は、各ブロックへ供給する符号化パラメータ(ヘッダ情報Hinfo、予測モード情報Pinfo、変換情報Tinfo、フィルタ情報Finfoなど)を、例えば、RDO(Rate-Distortion Optimization)に基づいて、決定する。例えば、制御部601は、変換スキップフラグ等を設定することができる。
画像符号化装置600には、動画像データの各フィールド(入力画像)がその再生順(表示順)に入力される。並べ替えバッファ611は、各入力画像をその再生順(表示順)に取得し、保持(記憶)する。並べ替えバッファ611は、制御部601の制御に基づいて、その入力画像を符号化順(復号順)に並べ替えたり、処理単位のブロックに分割したりする。並べ替えバッファ611は、処理後の各入力画像を演算部612に供給する。
演算部612は、並べ替えバッファ611から供給される処理単位のブロックに対応する画像から、予測部620より供給される予測画像Pを減算して、残差データDを導出し、それを変換量子化部613に供給する。
変換量子化部613は、係数変換と量子化に関する処理を行う。例えば、変換量子化部613は、演算部612から供給される残差データDを取得する。非ロスレス符号化モードの場合、変換量子化部613は、その残差データDに対して例えば直交変換等の係数変換を行い、変換係数Coeffを導出する。変換量子化部613は、その変換係数Coeffをスケーリング(量子化)し、量子化係数levelを導出する。変換量子化部613は、その量子化係数levelを符号化部614および逆量子化逆変換部616に供給する。
符号化部614は、変換量子化部613から供給された量子化係数level(または残差データD)と、制御部601から供給される各種符号化パラメータ(ヘッダ情報Hinfo、予測モード情報Pinfo、変換情報Tinfo、フィルタ情報Finfoなど)と、インループフィルタ部618から供給されるフィルタ係数等のフィルタに関する情報と、予測部620から供給される最適な予測モードに関する情報とを入力とする。
蓄積バッファ615は、符号化部614において得られた符号化データを、一時的に保持する。蓄積バッファ615は、所定のタイミングにおいて、保持している符号化データを、例えばビットストリーム等として画像符号化装置600の外部に出力する。例えば、この符号化データは、任意の記録媒体、任意の伝送媒体、任意の情報処理装置等を介して復号側に伝送される。すなわち、蓄積バッファ615は、符号化データ(ビットストリーム)を伝送する伝送部でもある。
逆量子化逆変換部616は、逆量子化と逆係数変換に関する処理を行う。例えば、非ロスレス符号化モードの場合、逆量子化逆変換部616は、変換量子化部613から供給される量子化係数levelと、制御部601から供給される変換情報Tinfoとを入力とする。逆量子化逆変換部616は、変換情報Tinfoに基づいて、量子化係数levelの値をスケーリング(逆量子化)し、変換係数Coeffを導出する。なお、この逆量子化は、変換量子化部613において行われる量子化の逆処理である。また、逆量子化逆変換部616は、変換情報Tinfoに基づいて、変換係数Coeffに対して逆係数変換(例えば逆直交変換)を行い、残差データD'を導出する。なお、この逆係数変換は、変換量子化部613において行われる係数変換の逆処理である。逆量子化逆変換部616は、導出した残差データD'を演算部617に供給する。
演算部617は、逆量子化逆変換部616から供給される残差データD'と、予測部620から供給される予測画像Pとを入力とする。演算部617は、その残差データD'と、その残差データD'に対応する予測画像とを加算し、局所復号画像を導出する。演算部617は、導出した局所復号画像をインループフィルタ部618およびフレームメモリ619に供給する。
インループフィルタ部618は、インループフィルタ処理に関する処理を行う。例えば、インループフィルタ部618は、演算部617から供給される局所復号画像と、制御部601から供給されるフィルタ情報Finfoと、並べ替えバッファ611から供給される入力画像(元画像)とを入力とする。なお、インループフィルタ部618に入力される情報は任意であり、これらの情報以外の情報が入力されてもよい。例えば、必要に応じて、予測モード、動き情報、符号量目標値、量子化パラメータQP、ピクチャタイプ、ブロック(CU、CTU等)の情報等がインループフィルタ部618に入力されるようにしてもよい。
フレームメモリ619は、画像に関するデータの記憶に関する処理を行う。例えば、フレームメモリ619は、演算部617から供給される局所復号画像や、インループフィルタ部618から供給されるフィルタ処理された局所復号画像を入力とし、それを保持(記憶)する。また、フレームメモリ619は、その局所復号画像を用いてピクチャ単位毎の復号画像を再構築し、保持する(フレームメモリ619内のバッファへ格納する)。フレームメモリ619は、予測部620の要求に応じて、その復号画像(またはその一部)を予測部620に供給する。
予測部620は、予測画像の生成に関する処理を行う。例えば、予測部620は、制御部601から供給される予測モード情報Pinfoと、並べ替えバッファ611から供給される入力画像(元画像)と、フレームメモリ619から読み出す復号画像(またはその一部)を入力とする。予測部620は、予測モード情報Pinfoや入力画像(元画像)を用い、インター予測やイントラ予測等の予測処理を行い、復号画像を参照画像として参照して予測を行い、その予測結果に基づいて動き補償処理を行い、予測画像を生成する。予測部620は、生成した予測画像を演算部612および演算部617に供給する。また、予測部620は、以上の処理により選択した予測モード、すなわち最適な予測モードに関する情報を、必要に応じて符号化部614に供給する。
レート制御部621は、レート制御に関する処理を行う。例えば、レート制御部621は、蓄積バッファ615に蓄積された符号化データの符号量に基づいて、オーバフローあるいはアンダーフローが発生しないように、変換量子化部613の量子化動作のレートを制御する。
以上のような画像符号化装置600の、変換量子化部613として、第1の実施の形態乃至第3の実施の形態において上述した変換量子化装置200を適用してもよい。つまり、この変換量子化部613が、図7、図14、または図20に示されるような構成を有するようにしてもよい。このようにすることにより、変換量子化部613は、第1の実施の形態乃至第3の実施の形態において上述した効果を得ることができる。つまり、画像符号化装置600は、その復号処理において、第1の実施の形態乃至第3の実施の形態において上述した効果を得ることができる。
図25のフローチャートを参照して、画像符号化装置600により実行される画像符号化処理の流れの例を説明する。
以上のような画像符号化装置600の、変換量子化処理(ステップS606)として、第1の実施の形態乃至第3の実施の形態において上述した逆量子化逆変換処理を適用してもよい。つまり、この逆量子化逆変換処理が、図8、図15、または図21に示されるフローチャートのような流れで実行されるようにしてもよい。このようにすることにより、変換量子化部613は、第1の実施の形態乃至第3の実施の形態において上述した効果を得ることができる。つまり、画像符号化装置600は、画像符号化処理において、第1の実施の形態乃至第3の実施の形態において上述した効果を得ることができる。
<コンピュータ>
上述した一連の処理は、ハードウエアにより実行させることもできるし、ソフトウエアにより実行させることもできる。一連の処理をソフトウエアにより実行する場合には、そのソフトウエアを構成するプログラムが、コンピュータにインストールされる。ここでコンピュータには、専用のハードウエアに組み込まれているコンピュータや、各種のプログラムをインストールすることで、各種の機能を実行することが可能な、例えば汎用のパーソナルコンピュータ等が含まれる。
本技術は、任意の画像符号化・復号方式に適用することができる。つまり、上述した本技術と矛盾しない限り、変換(逆変換)、量子化(逆量子化)、符号化(復号)、予測等、画像符号化・復号に関する各種処理の仕様は任意であり、上述した例に限定されない。また、上述した本技術と矛盾しない限り、これらの処理の内の一部を省略してもよい。
本技術を適用したシステム、装置、処理部等は、例えば、交通、医療、防犯、農業、畜産業、鉱業、美容、工場、家電、気象、自然監視等、任意の分野に利用することができる。また、その用途も任意である。
なお、本明細書において「フラグ」とは、複数の状態を識別するための情報であり、真(1)または偽(0)の2状態を識別する際に用いる情報だけでなく、3以上の状態を識別することが可能な情報も含まれる。したがって、この「フラグ」が取り得る値は、例えば1/0の2値であってもよいし、3値以上であってもよい。すなわち、この「フラグ」を構成するbit数は任意であり、1bitでも複数bitでもよい。また、識別情報(フラグも含む)は、その識別情報をビットストリームに含める形だけでなく、ある基準となる情報に対する識別情報の差分情報をビットストリームに含める形も想定されるため、本明細書においては、「フラグ」や「識別情報」は、その情報だけではなく、基準となる情報に対する差分情報も包含する。
(1) 変換スキップを適用するかに応じた量子化パラメータを用いて、量子化係数に対する逆量子化を行う逆量子化部と、
前記変換スキップを適用しない場合、前記逆量子化部による前記逆量子化により生成された変換係数に対する逆係数変換を行い、画像と前記画像の予測画像との残差である予測残差を生成し、前記変換スキップを適用する場合、前記逆係数変換をスキップする逆変換部と
を備える画像処理装置。
(2) 前記逆量子化部は、前記逆量子化において逆依存量子化を適用し、かつ、前記変換スキップを適用しない場合、前記量子化パラメータを補正し、補正した前記量子化パラメータを用いて前記逆量子化を行う
(1)に記載の画像処理装置。
(3) 前記変換係数の符号に基づいてラウンディングオフセットの符号を設定し、前記ラウンディングオフセットを用いて前記変換係数を正規化する逆量子化正規化部をさらに備え、
前記逆変換部は、前記変換スキップを適用しない場合、前記逆量子化正規化部により正規化された前記変換係数に対して前記逆係数変換を行う
(1)に記載の画像処理装置。
(4) 前記変換スキップを適用しない場合、前記予測残差の符号に基づいてラウンディングオフセットの符号を設定し、前記ラウンディングオフセットを用いて前記予測残差を正規化し、前記変換スキップを適用する場合、前記変換係数の符号に基づいて前記ラウンディングオフセットの符号を設定し、前記ラウンディングオフセットを用いて前記変換係数を正規化する逆変換正規化部をさらに備える
(1)に記載の画像処理装置。
(5) 変換スキップを適用するかに応じた量子化パラメータを用いて、量子化係数に対する逆量子化を行い、
前記変換スキップを適用しない場合、前記逆量子化により生成された変換係数に対する逆係数変換を行い、画像と前記画像の予測画像との残差である予測残差を生成し、前記変換スキップを適用する場合、前記逆係数変換をスキップする
画像処理方法。
前記変換スキップを適用するかに応じた量子化パラメータを用いて、前記変換係数に対する量子化を行う量子化部と
を備える画像処理装置。
(7) 変換スキップを適用しない場合、画像と前記画像の予測画像との残差である予測残差に対する係数変換を行い、変換係数を生成し、前記変換スキップを適用する場合、前記係数変換をスキップし、
前記変換スキップを適用するかに応じた量子化パラメータを用いて、前記変換係数に対する量子化を行う
画像処理方法。
変換スキップを適用するかに応じたスケーリングパラメータを用いて、前記逆量子化部による前記逆量子化により生成された変換係数を正規化する逆量子化正規化部と、
前記変換スキップを適用しない場合、前記逆量子化正規化部により正規化された前記変換係数に対して逆係数変換を行い、画像と前記画像の予測画像との残差である予測残差を生成し、前記変換スキップを適用する場合、前記逆係数変換をスキップする逆変換部と
を備える画像処理装置。
(9) 前記逆量子化正規化部は、前記変換スキップを適用しない場合、前記変換スキップにおける伸張処理を反映していない前記スケーリングパラメータを用いて前記変換係数を正規化し、前記変換スキップを適用する場合、前記伸張処理を反映した前記スケーリングパラメータを用いて前記変換係数を正規化し、
前記逆変換部は、前記変換スキップを適用しない場合、前記逆量子化正規化部により前記伸張処理を反映していない前記スケーリングパラメータを用いて正規化された前記変換係数に対して前記逆係数変換を行い、前記変換スキップを適用する場合、前記伸張処理を行わずに前記逆係数変換をスキップする
(8)に記載の画像処理装置。
(10) 前記逆量子化正規化部は、前記変換スキップを適用する場合、前記スケーリングパラメータの変換ブロックサイズに基づいて定まる成分を「0」に設定する
(9)に記載の画像処理装置。
(11) 前記逆量子化正規化部は、前記変換スキップを適用する場合、前記スケーリングパラメータの変換ブロックサイズに基づいて定まる成分を、量子化マトリックスに応じた値に設定する
(9)に記載の画像処理装置。
(12) 前記量子化マトリックスに応じた値は、「-4」である
(11)に記載の画像処理装置。
(13) 前記変換スキップを適用しない場合、前記予測残差を正規化する逆変換正規化部をさらに備え、
前記逆量子化正規化部は、前記変換スキップを適用する場合、前記逆変換正規化部による前記予測残差に対する正規化処理を反映した前記スケーリングパラメータを用いて前記変換係数を正規化する
(8)に記載の画像処理装置。
(14) 前記逆量子化正規化部は、前記変換スキップを適用する場合、逆量子化およびスケーリングリストに応じた値と、逆依存量子化の適用に応じた値とを含む前記スケーリングパラメータを用いて前記変換係数を正規化する
(13)に記載の画像処理装置。
(15) 前記逆量子化および前記スケーリングリストに応じた値は、「10」である
(14)に記載の画像処理装置。
(16) 量子化係数に対する逆量子化を行い、
変換スキップを適用するかに応じたスケーリングパラメータを用いて、前記逆量子化により生成された変換係数を正規化し、
前記変換スキップを適用しない場合、正規化された前記変換係数に対して逆係数変換を行い、画像と前記画像の予測画像との残差である予測残差を生成し、前記変換スキップを適用する場合、前記逆係数変換をスキップする
画像処理方法。
前記変換スキップを適用しない場合、前記変換係数に対する量子化を行い、前記変換スキップを適用する場合、前記予測残差に対する前記量子化を行う量子化部と、
前記変換スキップを適用するかに応じたスケーリングパラメータを用いて、前記量子化部による前記量子化により生成された量子化係数を正規化する量子化正規化部と
を備える画像処理装置。
(18) 前記変換部は、前記変換スキップを適用しない場合、前記予測残差に対して前記係数変換を行い、前記変換スキップを適用する場合、伸張処理を行わずに前記係数変換をスキップし、
前記量子化正規化部は、前記変換スキップを適用しない場合、前記変換スキップにおける前記伸張処理を反映していない前記スケーリングパラメータを用いて前記量子化係数を正規化し、前記変換スキップを適用する場合、前記伸張処理を反映した前記スケーリングパラメータを用いて前記量子化係数を正規化する
(17)に記載の画像処理装置。
(19) 前記変換スキップを適用しない場合、前記変換係数を正規化する変換正規化部をさらに備え、
前記量子化正規化部は、前記変換スキップを適用する場合、前記変換正規化部による前記変換係数に対する正規化処理を反映した前記スケーリングパラメータを用いて前記量子化係数を正規化する
(17)に記載の画像処理装置。
(20) 変換スキップを適用しない場合、画像と前記画像の予測画像との残差である予測残差に対して係数変換を行い、変換係数を生成し、前記変換スキップを適用する場合、前記係数変換をスキップし、
前記変換スキップを適用しない場合、前記変換係数に対する量子化を行い、前記変換スキップを適用する場合、前記予測残差に対する前記量子化を行い、
前記変換スキップを適用するかに応じたスケーリングパラメータを用いて、前記量子化により生成された量子化係数を正規化する
画像処理方法。
Claims (20)
- 変換スキップを適用するかに応じた量子化パラメータを用いて、量子化係数に対する逆量子化を行う逆量子化部と、
前記変換スキップを適用しない場合、前記逆量子化部による前記逆量子化により生成された変換係数に対する逆係数変換を行い、画像と前記画像の予測画像との残差である予測残差を生成し、前記変換スキップを適用する場合、前記逆係数変換をスキップする逆変換部と
を備える画像処理装置。 - 前記逆量子化部は、前記逆量子化において逆依存量子化を適用し、かつ、前記変換スキップを適用しない場合、前記量子化パラメータを補正し、補正した前記量子化パラメータを用いて前記逆量子化を行う
請求項1に記載の画像処理装置。 - 前記変換係数の符号に基づいてラウンディングオフセットの符号を設定し、前記ラウンディングオフセットを用いて前記変換係数を正規化する逆量子化正規化部をさらに備え、
前記逆変換部は、前記変換スキップを適用しない場合、前記逆量子化正規化部により正規化された前記変換係数に対して前記逆係数変換を行う
請求項1に記載の画像処理装置。 - 前記変換スキップを適用しない場合、前記予測残差の符号に基づいてラウンディングオフセットの符号を設定し、前記ラウンディングオフセットを用いて前記予測残差を正規化し、前記変換スキップを適用する場合、前記変換係数の符号に基づいて前記ラウンディングオフセットの符号を設定し、前記ラウンディングオフセットを用いて前記変換係数を正規化する逆変換正規化部をさらに備える
請求項1に記載の画像処理装置。 - 変換スキップを適用するかに応じた量子化パラメータを用いて、量子化係数に対する逆量子化を行い、
前記変換スキップを適用しない場合、前記逆量子化により生成された変換係数に対する逆係数変換を行い、画像と前記画像の予測画像との残差である予測残差を生成し、前記変換スキップを適用する場合、前記逆係数変換をスキップする
画像処理方法。 - 変換スキップを適用しない場合、画像と前記画像の予測画像との残差である予測残差に対する係数変換を行い、変換係数を生成し、前記変換スキップを適用する場合、前記係数変換をスキップする変換部と、
前記変換スキップを適用するかに応じた量子化パラメータを用いて、前記変換係数に対する量子化を行う量子化部と
を備える画像処理装置。 - 変換スキップを適用しない場合、画像と前記画像の予測画像との残差である予測残差に対する係数変換を行い、変換係数を生成し、前記変換スキップを適用する場合、前記係数変換をスキップし、
前記変換スキップを適用するかに応じた量子化パラメータを用いて、前記変換係数に対する量子化を行う
画像処理方法。 - 量子化係数に対する逆量子化を行う逆量子化部と、
変換スキップを適用するかに応じたスケーリングパラメータを用いて、前記逆量子化部による前記逆量子化により生成された変換係数を正規化する逆量子化正規化部と、
前記変換スキップを適用しない場合、前記逆量子化正規化部により正規化された前記変換係数に対して逆係数変換を行い、画像と前記画像の予測画像との残差である予測残差を生成し、前記変換スキップを適用する場合、前記逆係数変換をスキップする逆変換部と
を備える画像処理装置。 - 前記逆量子化正規化部は、前記変換スキップを適用しない場合、前記変換スキップにおける伸張処理を反映していない前記スケーリングパラメータを用いて前記変換係数を正規化し、前記変換スキップを適用する場合、前記伸張処理を反映した前記スケーリングパラメータを用いて前記変換係数を正規化し、
前記逆変換部は、前記変換スキップを適用しない場合、前記逆量子化正規化部により前記伸張処理を反映していない前記スケーリングパラメータを用いて正規化された前記変換係数に対して前記逆係数変換を行い、前記変換スキップを適用する場合、前記伸張処理を行わずに前記逆係数変換をスキップする
請求項8に記載の画像処理装置。 - 前記逆量子化正規化部は、前記変換スキップを適用する場合、前記スケーリングパラメータの変換ブロックサイズに基づいて定まる成分を「0」に設定する
請求項9に記載の画像処理装置。 - 前記逆量子化正規化部は、前記変換スキップを適用する場合、前記スケーリングパラメータの変換ブロックサイズに基づいて定まる成分を、量子化マトリックスに応じた値に設定する
請求項9に記載の画像処理装置。 - 前記量子化マトリックスに応じた値は、「-4」である
請求項11に記載の画像処理装置。 - 前記変換スキップを適用しない場合、前記予測残差を正規化する逆変換正規化部をさらに備え、
前記逆量子化正規化部は、前記変換スキップを適用する場合、前記逆変換正規化部による前記予測残差に対する正規化処理を反映した前記スケーリングパラメータを用いて前記変換係数を正規化する
請求項8に記載の画像処理装置。 - 前記逆量子化正規化部は、前記変換スキップを適用する場合、逆量子化およびスケーリングリストに応じた値と、逆依存量子化の適用に応じた値とを含む前記スケーリングパラメータを用いて前記変換係数を正規化する
請求項13に記載の画像処理装置。 - 前記逆量子化および前記スケーリングリストに応じた値は、「10」である
請求項14に記載の画像処理装置。 - 量子化係数に対する逆量子化を行い、
変換スキップを適用するかに応じたスケーリングパラメータを用いて、前記逆量子化により生成された変換係数を正規化し、
前記変換スキップを適用しない場合、正規化された前記変換係数に対して逆係数変換を行い、画像と前記画像の予測画像との残差である予測残差を生成し、前記変換スキップを適用する場合、前記逆係数変換をスキップする
画像処理方法。 - 変換スキップを適用しない場合、画像と前記画像の予測画像との残差である予測残差に対して係数変換を行い、変換係数を生成し、前記変換スキップを適用する場合、前記係数変換をスキップする変換部と、
前記変換スキップを適用しない場合、前記変換係数に対する量子化を行い、前記変換スキップを適用する場合、前記予測残差に対する前記量子化を行う量子化部と、
前記変換スキップを適用するかに応じたスケーリングパラメータを用いて、前記量子化部による前記量子化により生成された量子化係数を正規化する量子化正規化部と
を備える画像処理装置。 - 前記変換部は、前記変換スキップを適用しない場合、前記予測残差に対して前記係数変換を行い、前記変換スキップを適用する場合、伸張処理を行わずに前記係数変換をスキップし、
前記量子化正規化部は、前記変換スキップを適用しない場合、前記変換スキップにおける前記伸張処理を反映していない前記スケーリングパラメータを用いて前記量子化係数を正規化し、前記変換スキップを適用する場合、前記伸張処理を反映した前記スケーリングパラメータを用いて前記量子化係数を正規化する
請求項17に記載の画像処理装置。 - 前記変換スキップを適用しない場合、前記変換係数を正規化する変換正規化部をさらに備え、
前記量子化正規化部は、前記変換スキップを適用する場合、前記変換正規化部による前記変換係数に対する正規化処理を反映した前記スケーリングパラメータを用いて前記量子化係数を正規化する
請求項17に記載の画像処理装置。 - 変換スキップを適用しない場合、画像と前記画像の予測画像との残差である予測残差に対して係数変換を行い、変換係数を生成し、前記変換スキップを適用する場合、前記係数変換をスキップし、
前記変換スキップを適用しない場合、前記変換係数に対する量子化を行い、前記変換スキップを適用する場合、前記予測残差に対する前記量子化を行い、
前記変換スキップを適用するかに応じたスケーリングパラメータを用いて、前記量子化により生成された量子化係数を正規化する
画像処理方法。
Priority Applications (9)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
BR112022003668A BR112022003668A2 (pt) | 2019-09-06 | 2020-09-04 | Dispositivo e método de processamento de imagem |
CA3149873A CA3149873A1 (en) | 2019-09-06 | 2020-09-04 | Image processing device and method |
MX2022002267A MX2022002267A (es) | 2019-09-06 | 2020-09-04 | Dispositivo y metodo de procesamiento de imagenes. |
CN202080060634.XA CN114342396A (zh) | 2019-09-06 | 2020-09-04 | 图像处理装置和方法 |
AU2020341156A AU2020341156A1 (en) | 2019-09-06 | 2020-09-04 | Image processing device and method |
US17/632,238 US20220224903A1 (en) | 2019-09-06 | 2020-09-04 | Image processing device and method |
KR1020227005511A KR20220053563A (ko) | 2019-09-06 | 2020-09-04 | 화상 처리 장치 및 방법 |
EP20861050.1A EP4013054A4 (en) | 2019-09-06 | 2020-09-04 | IMAGE PROCESSING DEVICE AND METHOD |
JP2021544048A JPWO2021045188A1 (ja) | 2019-09-06 | 2020-09-04 |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201962896989P | 2019-09-06 | 2019-09-06 | |
US62/896,989 | 2019-09-06 | ||
US201962901895P | 2019-09-18 | 2019-09-18 | |
US62/901,895 | 2019-09-18 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021045188A1 true WO2021045188A1 (ja) | 2021-03-11 |
Family
ID=74853204
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2020/033573 WO2021045188A1 (ja) | 2019-09-06 | 2020-09-04 | 画像処理装置および方法 |
Country Status (11)
Country | Link |
---|---|
US (1) | US20220224903A1 (ja) |
EP (1) | EP4013054A4 (ja) |
JP (1) | JPWO2021045188A1 (ja) |
KR (1) | KR20220053563A (ja) |
CN (1) | CN114342396A (ja) |
AU (1) | AU2020341156A1 (ja) |
BR (1) | BR112022003668A2 (ja) |
CA (1) | CA3149873A1 (ja) |
MX (1) | MX2022002267A (ja) |
TW (1) | TW202121897A (ja) |
WO (1) | WO2021045188A1 (ja) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2022549795A (ja) * | 2019-09-21 | 2022-11-29 | 北京字節跳動網絡技術有限公司 | 画像およびビデオコーディングのための高精度変換および量子化 |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114365490A (zh) | 2019-09-09 | 2022-04-15 | 北京字节跳动网络技术有限公司 | 高精度图像和视频编解码的系数缩放 |
WO2021158049A1 (ko) * | 2020-02-05 | 2021-08-12 | 엘지전자 주식회사 | 영상 정보 코딩에 대한 영상 디코딩 방법 및 그 장치 |
CN115336274B (zh) * | 2020-02-05 | 2023-05-26 | Lg电子株式会社 | 与残差编码关联的图像解码方法以及用于其的装置 |
WO2023220946A1 (zh) * | 2022-05-17 | 2023-11-23 | Oppo广东移动通信有限公司 | 视频编解码方法、装置、设备、系统及存储介质 |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2019003676A1 (ja) * | 2017-06-29 | 2019-01-03 | ソニー株式会社 | 画像処理装置と画像処理方法およびプログラム |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9270986B2 (en) * | 2012-04-13 | 2016-02-23 | Qualcomm Incorporated | Level decision in rate distortion optimized quantization |
JPWO2014050676A1 (ja) * | 2012-09-28 | 2016-08-22 | ソニー株式会社 | 画像処理装置および方法 |
WO2015104963A1 (ja) * | 2014-01-09 | 2015-07-16 | 株式会社日立国際電気 | 画像処理装置及び動画像伝送方法 |
JP2017073602A (ja) * | 2015-10-05 | 2017-04-13 | 富士通株式会社 | 動画像符号化装置、動画像符号化方法及び動画像符号化用コンピュータプログラム |
JP6822470B2 (ja) * | 2016-05-13 | 2021-01-27 | ソニー株式会社 | 画像処理装置および方法 |
CN109076226B (zh) * | 2016-05-13 | 2021-08-13 | 索尼公司 | 图像处理装置和方法 |
EP3349452A1 (en) * | 2017-01-11 | 2018-07-18 | Thomson Licensing | Scaling of transform coefficients for non-dyadic transform blocks |
EP3484151A1 (en) * | 2017-11-13 | 2019-05-15 | Thomson Licensing | Method and apparatus for generating quantization matrices in video encoding and decoding |
CN112771865A (zh) * | 2018-08-23 | 2021-05-07 | 交互数字Vc控股法国公司 | 使用参数化模型对量化矩阵进行编解码 |
US11546602B2 (en) * | 2018-08-24 | 2023-01-03 | Samsung Electronics Co., Ltd. | Method and apparatus for image encoding, and method and apparatus for image decoding |
EP3918797A4 (en) * | 2019-03-10 | 2023-01-25 | HFI Innovation Inc. | METHOD AND DEVICE FOR CALCULATION AND REPRESENTATION OF QUANTIZATION MATRICES FOR VIDEO CODING |
JP7402016B2 (ja) * | 2019-11-06 | 2023-12-20 | オッポ広東移動通信有限公司 | 画像復号装置及び画像符号化装置 |
WO2021107641A1 (ko) * | 2019-11-26 | 2021-06-03 | 주식회사 윌러스표준기술연구소 | 적응적 컬러 공간 변환을 이용하는 비디오 신호 처리 방법 및 장치 |
US11575896B2 (en) * | 2019-12-16 | 2023-02-07 | Panasonic Intellectual Property Corporation Of America | Encoder, decoder, encoding method, and decoding method |
US11490122B2 (en) * | 2020-09-24 | 2022-11-01 | Tencent America LLC | Method and apparatus for video coding |
-
2020
- 2020-09-04 TW TW109130353A patent/TW202121897A/zh unknown
- 2020-09-04 BR BR112022003668A patent/BR112022003668A2/pt unknown
- 2020-09-04 WO PCT/JP2020/033573 patent/WO2021045188A1/ja unknown
- 2020-09-04 JP JP2021544048A patent/JPWO2021045188A1/ja active Pending
- 2020-09-04 US US17/632,238 patent/US20220224903A1/en active Pending
- 2020-09-04 CA CA3149873A patent/CA3149873A1/en active Pending
- 2020-09-04 EP EP20861050.1A patent/EP4013054A4/en active Pending
- 2020-09-04 CN CN202080060634.XA patent/CN114342396A/zh active Pending
- 2020-09-04 KR KR1020227005511A patent/KR20220053563A/ko unknown
- 2020-09-04 MX MX2022002267A patent/MX2022002267A/es unknown
- 2020-09-04 AU AU2020341156A patent/AU2020341156A1/en active Pending
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2019003676A1 (ja) * | 2017-06-29 | 2019-01-03 | ソニー株式会社 | 画像処理装置と画像処理方法およびプログラム |
Non-Patent Citations (14)
Title |
---|
ADVANCED VIDEO CODING FOR GENERIC AUDIOVISUAL SERVICES, April 2017 (2017-04-01) |
ALBRECHT M; BARTNIK C; BOSSE S; BRANDENBURG J; BROSS B; ERFURT J; GEORGE V; HAASE P; HELLE P; HELMRICH C; HENKEL A; HINZ T; DE LUX: "Description of SDR, HDR, and 360°video coding technology proposal by Fraunhofer HHI", JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/WG 11, JVET-J0014- VL, 10TH MEETING, no. JVET-J0014, 2 April 2018 (2018-04-02), San Diego, US, pages 31 - 34, XP030151172 * |
BENJAMIN BROSSJIANLE CHENSHAN LIU: "Versatile Video Coding (Draft 5", NLOOL-VLO, M48053, JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/WG 11 14TH MEETING: GENEVA, CH, 19 March 2019 (2019-03-19) |
BENJAMIN BROSSJIANLE CHENSHAN LIU: "Versatile Video Coding (Draft 6", JVET-02001-VE, M49908, JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/WG 11 15TH MEETING: GOTHENBURG, SE, 3 July 2019 (2019-07-03) |
HIGH EFFICIENCY VIDEO CODING, February 2018 (2018-02-01) |
HYEONGMUN JANGJUNGHAK NAMNAERI PARKJUNGAH CHOI SEUNGHWAN KIMJAEHYUN LIM: "Comments on transform quantization bypassed mode", JVET-00584, M48723, JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/WG 11 15TH MEETING: GOTHENBURG, SE, 3 July 2019 (2019-07-03) |
JIANLE CHENYAN YESEUNG HWAN KIM: "Algorithm description for Versatile Video Coding and Test Model 5 (VTM 5", JVET-N1002-V2, M48054, JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/WG 11 14TH MEETING: GENEVA, CH, 19 March 2019 (2019-03-19) |
JIANLE CHENYAN YESEUNG HWAN KIM: "Algorithm description for Versatile Video Coding and Test Model 6 (VTM 6", JVET-02002-V2, M49914, JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/WG 11 15TH MEETING: GOTHENBURG, SE, 3 July 2019 (2019-07-03) |
KARCZEWICZ (QUALCOMM) M; Y-H CHAO (QUALCOMM); WANG (QUALCOMM) H; COBAN (QUALCOMM) M: "Non-CE8: Minimum QP for Transform Skip Mode", JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/WG 11, JVET-00919-V2, 15TH MEETING, no. JVET-00919, 5 July 2019 (2019-07-05), Gothenburg, SE, pages 1 - 3, XP030220538 * |
NGUYEN (FRAUNHOFER) T; BROSS B; SCHWARZ H; MARPE D; WIEGAND (HHI) T: "Non-CE8: Minimum Allowed QP for Transform Skip Mode", JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/ WG 11, JVET-00405-VL, 15TH MEETING, no. JVET-O0405 ; m48524, 25 June 2019 (2019-06-25), Gothenburg, SE, pages 1 - 3, XP030219469 * |
T.-C. MA (KWAI), Y.-W. CHEN (KWAI), X. XIU (KWAI), X. WANG (KWAI INC.), T. POIRIER, F. LE LÉANNEC, K. NASER, E. FRANCOIS (INTERDIG: "Lossless coding for VVC", JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/WG 11, JVET-01061, 15TH MEETING, no. JVET-01061, 8 July 2019 (2019-07-08), Gothenburg, SE, pages 1 - 17, XP030220701 * |
TANGI POIRIERFABRICE LE LEANNECKARAM NASEREDOUARD FRANCOIS: "On lossless coding for WC", JVET-00460, M48583, JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/WG 11 15TH MEETING: GOTHENBURG, SE, 3 July 2019 (2019-07-03) |
TSUNG-CHUAN MAYI-WEN CHENXIAOYU XIUXIANGLIN WANG: "Modifications to support the lossless coding", JVET-00591, M48730, JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/WG 11 15TH MEETING: GOTHENBURG, SE, 3 July 2019 (2019-07-03) |
TSUNG-CHUAN MAYI-WEN CHENXIAOYU XIUXIANGLIN WANGTANGI POIRIERFABRICE LE LEANNECKARAM NASEREDOUARD FRANCOISHYEONGMUN JANGJUNGHAK NA: "Lossless coding for WC", JVET-01061, M49678, JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/WG 11 15TH MEETING: GOTHENBURG, SE, 3 July 2019 (2019-07-03) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2022549795A (ja) * | 2019-09-21 | 2022-11-29 | 北京字節跳動網絡技術有限公司 | 画像およびビデオコーディングのための高精度変換および量子化 |
JP7323712B2 (ja) | 2019-09-21 | 2023-08-08 | 北京字節跳動網絡技術有限公司 | 画像およびビデオコーディングのための高精度変換および量子化 |
Also Published As
Publication number | Publication date |
---|---|
TW202121897A (zh) | 2021-06-01 |
KR20220053563A (ko) | 2022-04-29 |
BR112022003668A2 (pt) | 2022-05-24 |
EP4013054A1 (en) | 2022-06-15 |
CN114342396A (zh) | 2022-04-12 |
US20220224903A1 (en) | 2022-07-14 |
MX2022002267A (es) | 2022-03-22 |
CA3149873A1 (en) | 2021-03-11 |
EP4013054A4 (en) | 2022-10-26 |
AU2020341156A1 (en) | 2022-03-03 |
JPWO2021045188A1 (ja) | 2021-03-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021045188A1 (ja) | 画像処理装置および方法 | |
US10178391B2 (en) | Methods and devices for data compression using a non-uniform reconstruction space | |
US11343538B2 (en) | Image processing apparatus and method | |
WO2021125309A1 (ja) | 画像処理装置および方法 | |
US20220400285A1 (en) | Image processing apparatus and method | |
JP7235030B2 (ja) | 画像処理装置および方法 | |
WO2021039650A1 (ja) | 画像処理装置および方法 | |
WO2019188465A1 (ja) | 画像符号化装置、画像符号化方法、画像復号装置、および画像復号方法 | |
EP2575364A1 (en) | Methods and devices for data compression using a non-uniform reconstruction space | |
WO2021100588A1 (ja) | 画像処理装置および方法 | |
US20230007265A1 (en) | Image processing device, bit stream generation method, coefficient data generation method, and quantization coefficient generation method | |
US20230045106A1 (en) | Image processing apparatus and method | |
WO2023195330A1 (ja) | 画像処理装置および方法 | |
US20220394256A1 (en) | Image processing device and method | |
US20220286681A1 (en) | Image processing device and method | |
WO2020129636A1 (ja) | 画像符号化装置、画像符号化方法、画像復号装置、および画像復号方法 | |
KR20240024808A (ko) | 화상 처리 장치 및 방법 | |
US20220086489A1 (en) | Image processing apparatus and method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20861050 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2021544048 Country of ref document: JP Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 3149873 Country of ref document: CA |
|
ENP | Entry into the national phase |
Ref document number: 2020341156 Country of ref document: AU Date of ref document: 20200904 Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
REG | Reference to national code |
Ref country code: BR Ref legal event code: B01A Ref document number: 112022003668 Country of ref document: BR |
|
ENP | Entry into the national phase |
Ref document number: 2020861050 Country of ref document: EP Effective date: 20220307 |
|
ENP | Entry into the national phase |
Ref document number: 112022003668 Country of ref document: BR Kind code of ref document: A2 Effective date: 20220225 |