WO2020066642A1 - Dispositif de codage, procédé de codage, dispositif de décodage, et procédé de décodage - Google Patents

Dispositif de codage, procédé de codage, dispositif de décodage, et procédé de décodage Download PDF

Info

Publication number
WO2020066642A1
WO2020066642A1 PCT/JP2019/035819 JP2019035819W WO2020066642A1 WO 2020066642 A1 WO2020066642 A1 WO 2020066642A1 JP 2019035819 W JP2019035819 W JP 2019035819W WO 2020066642 A1 WO2020066642 A1 WO 2020066642A1
Authority
WO
WIPO (PCT)
Prior art keywords
merge
class
subclass
classes
activity
Prior art date
Application number
PCT/JP2019/035819
Other languages
English (en)
Japanese (ja)
Inventor
優 池田
Original Assignee
ソニー株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from JP2018246543A external-priority patent/JP2022002357A/ja
Application filed by ソニー株式会社 filed Critical ソニー株式会社
Priority to CN201980061396.1A priority Critical patent/CN112740678A/zh
Priority to US17/268,320 priority patent/US20210168407A1/en
Publication of WO2020066642A1 publication Critical patent/WO2020066642A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H04N19/14—Coding unit complexity, e.g. amount of activity or edge presence estimation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/117—Filters, e.g. for pre-processing or post-processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/182—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/80—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
    • H04N19/82—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation involving filtering within a prediction loop
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
    • H04N19/96—Tree coding, e.g. quad-tree coding

Definitions

  • the present technology relates to an encoding device, an encoding method, a decoding device, and a decoding method, and in particular, for example, an encoding device, an encoding method, a decoding device, and decoding that can reduce a processing amount. About the method.
  • VVC Very Video Coding
  • FVC Full Video Coding
  • HEVC High Efficiency Video Coding
  • ILF image encoding and decoding
  • a bilateral filter Bilateral @ Filter
  • ALF Adaptive @ Loop @ Filter
  • GALF Global Adaptive Loop Filter
  • JEM7 Joint Exploration Test Model 7
  • PCS Picture Coding Symposium
  • a class merge process for merging classes is performed in order to reduce the amount of tap coefficient data by sharing tap coefficients used for filtering among a plurality of classes.
  • an optimal merge pattern for merging classes is obtained for each merge class number, with each natural number value equal to or less than the original class number being the merge class number after class merge. Then, from among the optimal merge patterns for each number of merge classes, the merge pattern that minimizes the cost is determined as the adopted merge pattern to be adopted when performing the filter processing.
  • each value of a natural number equal to or less than the original number of classes is used as the number of merge classes after class merging, and an optimum merge pattern is obtained for each number of merge classes. Become. Note that the adopted merge pattern determined by the class merge process needs to be transmitted from the encoding device to the decoding device.
  • the present technology has been made in view of such a situation, and is intended to reduce a processing amount.
  • a decoding device decodes encoded data included in an encoded bit stream, generates a decoded image, and classifies a target pixel of the decoded image generated by the decoding unit for a pixel of interest into a plurality of classes.
  • a class classifying unit that performs subclass classification for each feature amount, and an initial class of the pixel of interest obtained by the class classification performed by the class classifying unit is configured by the feature amount according to a merge pattern set in advance for each number of merge classes.
  • a merge class that converts the initial class into a merge class by merging the subclasses of the target pixel; and a product-sum operation of the tap coefficient of the merge class of the pixel of interest converted by the merge convert unit and the pixel of the decoded image Performing a filtering process of applying a prediction formula for performing the above to the decoded image to generate a filtered image.
  • the decoding method decodes encoded data included in an encoded bit stream to generate a decoded image, and performs class classification on a target pixel of the decoded image by subclass classification of each of a plurality of feature amounts. And converting the initial class of the target pixel obtained by the class classification into a merge class in which the initial class is merged by merging the subclasses of the feature amount in accordance with a merge pattern preset for each number of merge classes. And performing a filter process of applying a prediction formula for performing a product-sum operation between a tap coefficient of a merge class of the pixel of interest and a pixel of the decoded image to the decoded image to generate a filtered image. Is the way.
  • encoded data included in an encoded bit stream is decoded, and a decoded image is generated. Further, a class classification for the target pixel of the decoded image is performed by subclass classification of each of a plurality of feature amounts, and an initial class of the target pixel obtained by the class classification is a merge pattern set in advance for each number of merge classes. Is converted into a merged class obtained by merging the initial class by merging the subclasses of the feature amount.
  • a filter process is performed to apply a prediction formula for performing a product-sum operation of the tap coefficient of the merge class of the pixel of interest and the pixel of the decoded image to the decoded image, thereby generating a filtered image.
  • An encoding device obtains a class classification for a pixel of interest of a locally decoded locally decoded image by a subclass classification of each of a plurality of feature amounts, and a class classification performed by the class classification unit.
  • the encoding method performs a class classification on a target pixel of a locally decoded image that is locally decoded by performing a subclass classification of each of a plurality of feature amounts, and an initial class of the target pixel obtained by the class classification. Converting the initial class into a merged class by merging the subclasses of the feature amount according to a merge pattern set in advance for each number of merged classes; tap coefficients of the merged class of the target pixel and the local decoded image Generating a filter image by performing a filter process of applying a prediction formula for performing a product-sum operation with pixels of the local decoded image to generate a filter image, and encoding an original image using the filter image.
  • the encoding device and the decoding device may be independent devices, or may be internal blocks forming one device.
  • the encoding device and the decoding device can be realized by causing a computer to execute a program.
  • the program can be provided by being transmitted via a transmission medium or by being recorded on a recording medium.
  • FIG. 4 is a diagram illustrating an outline of a process of a class classification unit 10 that performs a GALF class classification.
  • FIG. 9 is a diagram illustrating how to determine a direction of GALF as a specified direction (inclination direction) of a target pixel.
  • FIG. 3 is a diagram for explaining classes obtained by GALF class classification.
  • 6 is a flowchart illustrating a GALF process that the encoding device that encodes an image has as one of the ILFs. It is a flowchart explaining a merge pattern determination process of step S21.
  • FIG. 9 is a diagram for describing an example of an expression format of a merge pattern.
  • FIG. 9 is a diagram illustrating an example of a merge pattern for each number of merge classes.
  • FIG. 3 is a diagram illustrating an example of a merge pattern transmitted from an encoding device to a decoding device.
  • FIG. 7 is a diagram illustrating a first example of a preset merge pattern.
  • FIG. 7 is a diagram illustrating a first example of a preset merge pattern.
  • FIG. 11 is a diagram illustrating a method of setting a merge pattern corresponding to a merge class number of 25 in which 25 initial classes obtained by GALF class classification are merged into 25 merge classes.
  • FIG. 11 is a diagram illustrating a method of setting a merge pattern corresponding to a merge class number of 20 in which 25 initial classes obtained by GALF class classification are merged into 20 merge classes.
  • FIG. 3 is a diagram illustrating an example of a merge pattern transmitted from an encoding device to a decoding device.
  • FIG. 7 is a diagram illustrating a first example of a preset merge pattern.
  • FIG. 11 is a diagram illustrating a method of setting a merge pattern corresponding to a
  • FIG. 14 is a diagram illustrating a method of setting a merge pattern corresponding to the number of merge classes of two in which 25 initial classes obtained by GALF class classification are merged into two merge classes.
  • FIG. 9 is a diagram illustrating a method of setting a merge pattern corresponding to a merge class number of 1, in which 25 initial classes obtained by GALF class classification are merged into one merge class.
  • FIG. 9 is a diagram illustrating a second example of a preset merge pattern.
  • FIG. 9 is a diagram illustrating a second example of a preset merge pattern.
  • FIG. 9 is a diagram for describing class classification using ranking (Ranking) as a feature amount of a target pixel.
  • FIG. 9 is a diagram illustrating class classification using a pixel value as a feature amount of a target pixel.
  • FIG. 9 is a diagram illustrating class classification using reliability in a tilt direction as a feature amount of a target pixel. It is a figure explaining the last class calculated
  • FIG. 14 is a diagram illustrating a third example of a preset merge pattern.
  • FIG. 14 is a diagram illustrating a third example of a preset merge pattern.
  • FIG. 14 is a diagram illustrating a fourth example of a preset merge pattern. It is a figure explaining the class classification of GALF. It is a figure explaining the subclass merge of a gradient intensity ratio subclass. It is a figure explaining the subclass merge of a direction subclass.
  • FIG. 11 is a diagram illustrating an example of a merge pattern obtained by performing a subclass merge and a merge pattern selection.
  • FIG. 9 is a diagram illustrating an example of a merge pattern obtained by partially merging subclasses.
  • FIG. 7 is a diagram illustrating an example of a relationship between a merge pattern obtained by subclass merge and merge pattern selection and a merge pattern obtained by partial merge.
  • FIG. 11 is a diagram illustrating another example of a relationship between a merge pattern obtained by subclass merge and merge pattern selection and a merge pattern obtained by partial merge.
  • FIG. 11 is a diagram showing a merge pattern corresponding to a merge class number 25 obtained by subclass merge and a subclass merge from which the merge pattern is obtained.
  • FIG. 9 is a diagram showing a merge pattern corresponding to 9 merge classes obtained by subclass merge, and a subclass merge from which the merge pattern is obtained.
  • FIG. 9 is a diagram showing a merge pattern corresponding to a merge class number 8 obtained by subclass merge and a subclass merge from which the merge pattern is obtained.
  • FIG. 8 is a diagram showing a merge pattern corresponding to a merge class number 6 obtained by subclass merge and a subclass merge from which the merge pattern is obtained.
  • FIG. 9 is a diagram illustrating a merge pattern corresponding to a merge class number 5 obtained by subclass merge and a subclass merge from which the merge pattern is obtained.
  • FIG. 11 is a diagram showing a merge pattern corresponding to a merge class number 4 obtained by subclass merge and a subclass merge from which the merge pattern is obtained.
  • FIG. 11 is a diagram showing a merge pattern corresponding to the number of merge classes 3 obtained by subclass merge and a subclass merge from which the merge pattern is obtained.
  • FIG. 7 is a diagram showing a merge pattern corresponding to a merge class number 2 obtained by subclass merge and a subclass merge from which the merge pattern is obtained.
  • FIG. 9 is a diagram showing a merge pattern corresponding to a merge class number 1 obtained by subclass merge and a subclass merge from which the merge pattern is obtained.
  • FIG. 21 is a block diagram illustrating a configuration example of a classification prediction filter to which the present technology is applied.
  • FIG. 9 is a flowchart illustrating an example of a class classification prediction process performed by a class classification prediction filter 110.
  • 1 is a block diagram illustrating an outline of an embodiment of an image processing system to which the present technology is applied.
  • 6 is a flowchart illustrating an outline of an encoding process of the encoding device 160. It is a flow chart explaining an outline of decoding processing of decoding device 170.
  • FIG. 39 is a block diagram illustrating a detailed configuration example of an encoding device 160.
  • 15 is a flowchart illustrating an example of an encoding process of an encoding device 160. It is a flowchart explaining the example of a prediction encoding process.
  • FIG. 21 is a block diagram illustrating a detailed configuration example of a decoding device 170.
  • FIG. 9 is a diagram showing a merge pattern (2, # 1, # 4) and a subclass merge from which the merge pattern (2, # 1, # 4) is obtained.
  • FIG. 9 is a diagram showing a merge pattern (1, 2, 4) and a subclass merge from which the merge pattern (1, 2, 4) is obtained.
  • FIG. 9 is a diagram showing a merge pattern (3, # 2, # 3) and a subclass merge from which the merge pattern (3, # 2, # 3) is obtained.
  • FIG. 9 is a diagram showing a merge pattern (3, # 1, # 3) and a subclass merge from which the merge pattern (3, # 1, # 3) is obtained.
  • FIG. 11 is a diagram showing a merge pattern (1, 2, 3) and a subclass merge from which the merge pattern (1, 2, 3) is obtained.
  • FIG. 9 is a diagram showing a merge pattern (3, # 2, # 2) and a subclass merge from which the merge pattern (3, # 2, # 2) is obtained.
  • FIG. 9 is a diagram illustrating a merge pattern (3, # 1, # 2) and a subclass merge from which the merge pattern (3, # 1, # 2) is obtained.
  • FIG. 9 is a diagram showing a merge pattern (2, 1, 2) and a subclass merge from which the merge pattern (2, 1, 2) is obtained.
  • FIG. 6 is a diagram showing a merge pattern (1, 2, 2) and a subclass merge from which the merge pattern (1, 2, 2) is obtained.
  • FIG. 9 is a diagram showing a merge pattern (3, # 2, # 1) and a subclass merge from which the merge pattern (3, # 2, # 1) is obtained.
  • FIG. 9 is a diagram illustrating a merge pattern (3, # 1, # 1) and a subclass merge from which the merge pattern (3, # 1, # 1) is obtained.
  • FIG. 8 is a diagram showing a merge pattern (2, # 2, # 1) and a subclass merge from which the merge pattern (2, # 2, # 1) is obtained.
  • FIG. 9 is a diagram showing a merge pattern (2, 1, 1) and a subclass merge from which the merge pattern (2, 1, 1) is obtained.
  • FIG. 9 is a diagram showing a merge pattern (1, 2, 1) and a subclass merge from which the merge pattern (1, 2, 1) is obtained. It is a figure showing the example of the syntax which transmits the combination of the number of subclasses.
  • FIG. 8 is a diagram showing a merge pattern (2, # 2, # 1) and a subclass merge from which the merge pattern (2, # 2, # 1) is obtained.
  • FIG. 9 is a diagram showing a merge pattern (2, 1, 1) and a subclass merge from which the merge pattern (1, 2, 1) is obtained. It is
  • 21 is a block diagram illustrating a configuration example of a classification prediction filter to which the present technology is applied.
  • 11 is a flowchart illustrating an example of a class classification prediction process performed by a class classification prediction filter 410.
  • 1 is a block diagram illustrating an outline of an embodiment of an image processing system to which the present technology is applied.
  • 15 is a flowchart illustrating an outline of an encoding process of an encoding device 460. It is a flow chart explaining an outline of decoding processing of decoding device 470.
  • FIG. 39 is a block diagram illustrating a detailed configuration example of an encoding device 460.
  • 15 is a flowchart illustrating an example of an encoding process of an encoding device 460.
  • FIG. 39 is a block diagram illustrating a detailed configuration example of a decoding device 470. It is a flow chart explaining an example of decoding processing of decoding device 470. It is a flowchart explaining an example of a prediction decoding process. It is a figure explaining the class classification of GALF.
  • FIG. 4 is a diagram illustrating a relationship between a merge pattern and a subclass.
  • FIG. 4 is a diagram illustrating a first merge rule.
  • FIG. 4 is a diagram showing all types of merge patterns set according to a first merge rule.
  • FIG. 9 is a diagram illustrating a method of merging when setting all types of merge patterns according to a first merge rule.
  • FIG. 9 is a diagram illustrating a method of merging when setting all types of merge patterns according to a first merge rule.
  • FIG. 9 is a diagram illustrating a method of merging when setting all types of merge patterns according to a first merge rule.
  • FIG. 9 is a diagram illustrating a method of merging when setting all types of merge patterns according to a first merge rule.
  • FIG. 9 is a diagram illustrating a method of merging when setting all types of merge patterns according to a first merge rule.
  • FIG. 9 is a diagram illustrating a method of merging when setting all types of merge patterns according to a first merge rule.
  • FIG. 9 is a diagram illustrating a second merge rule.
  • FIG. 9 is a diagram showing all types of merge patterns set according to a second merge rule.
  • FIG. 9 is a diagram for explaining a merging method when setting all types of merge patterns according to a second merge rule.
  • FIG. 9 is a diagram for explaining a merging method when setting all types of merge patterns according to a second merge rule.
  • FIG. 9 is a diagram for explaining a merging method when setting all types of merge patterns according to a second merge rule.
  • FIG. 9 is a diagram for explaining a merging method when setting all types of merge patterns according to a second merge rule.
  • FIG. 9 is a diagram for explaining a merging method when setting all types of merge patterns according to a second merge rule.
  • FIG. 9 is a diagram for explaining a merging method when setting all types of merge patterns according to a second merge rule.
  • FIG. 9 is a diagram for explaining a merging method when setting all types of merge patterns according to a second merge rule.
  • FIG. 11 is a diagram illustrating a method of merging when setting all types of merge patterns according to a third merge rule.
  • FIG. 11 is a diagram illustrating a method of merging when setting all types of merge patterns according to a third merge rule.
  • FIG. 11 is a diagram illustrating a method of merging when setting all types of merge patterns according to a third merge rule.
  • FIG. 11 is a diagram illustrating a method of merging when setting all types of merge patterns according to a third merge rule.
  • FIG. 11 is a diagram illustrating a method of merging when setting all types of merge patterns according to a third merge rule.
  • FIG. 11 is a diagram illustrating a method of merging when setting all types of merge patterns according to a third merge rule.
  • FIG. 11 is a diagram illustrating a method of merging when setting all types of merge patterns according to a third merge rule.
  • Reference 1 AVC standard ("Advanced video coding for generic audiovisual services", ITU-T H.264 (04/2017))
  • Reference 2 HEVC standard ("High efficiency video coding”, ITU-T H.265 (12/2016))
  • Reference 3 Algorithm description of Joint Exploration Test Model 7 (JEM7), 2017-08-19
  • the contents described in the above-mentioned documents are also the basis for determining support requirements.
  • the Quad-Tree Block Structure described in Document 1 and the QTBT (Quad Tree Plus Binary Tree) and Block Structure described in Document 3 are not directly described in the embodiment, the present technology can be used. It is within the disclosure range and meets the support requirements of the claims. Also, for example, technical terms such as parsing, syntax, and semantics are also within the disclosure range of the present technology even when there is no direct description in the embodiment. Meet claims support requirements.
  • a “block” (not a block indicating a processing unit) used in the description as a partial region or a processing unit of an image (picture) indicates an arbitrary partial region in a picture unless otherwise specified.
  • “block” includes TB (Transform Block), TU (Transform Unit), PB (Prediction Block), PU (Prediction Unit), SCU (Smallest Coding Unit), CU ( An arbitrary partial area (processing unit) such as Coding Unit, LCU (Largest Coding Unit), CTB (Coding Tree Block), CTU (Coding Tree Unit), conversion block, subblock, macroblock, tile, or slice included.
  • the block size may be specified directly, but also the block size may be specified indirectly.
  • the block size may be specified using identification information for identifying the size.
  • the block size may be specified by a ratio or a difference from the size of a reference block (for example, an LCU or an SCU).
  • a reference block for example, an LCU or an SCU.
  • the designation of the block size also includes designation of a range of block sizes (for example, designation of a range of allowable block sizes, etc.).
  • Encoded data is data obtained by encoding an image, for example, data obtained by orthogonally transforming and quantizing an image (residual).
  • the coded bit stream is a bit stream including coded data, and includes coded information related to coding as necessary.
  • the coded information includes information necessary for decoding the coded data, that is, for example, a quantization parameter (QP) in the case where quantization is performed in encoding, and predictive coding (motion At least a motion vector when compensation is performed is included.
  • QP quantization parameter
  • predictive coding motion At least a motion vector when compensation is performed is included.
  • the prediction formula is a polynomial for predicting the second data from the first data.
  • the prediction formula is a polynomial for predicting the second image from the first image.
  • Each term of the prediction formula, which is such a polynomial is composed of a product of one tap coefficient and one or more prediction taps. Therefore, the prediction formula is a formula for performing a product-sum operation of the tap coefficient and the prediction tap. is there.
  • the pixel (its pixel value) as the i-th prediction tap used for prediction (calculation of a prediction formula) among the pixels of the first image is x i
  • the i-th tap coefficient is w i
  • the second image Predicted value of pixel value
  • y ′ a polynomial composed of only linear terms
  • y ′ ⁇ w i x i Is done.
  • y ′ ⁇ w i x i
  • ⁇ represents the summation for i .
  • Tap coefficient w i which constitutes the prediction equation
  • the value y 'obtained by the prediction equation is obtained by statistically learning to minimize the error y'-y of the true value y.
  • tap coefficient learning there is a least square method.
  • a student image as student data input x i to the prediction formula
  • a prediction formula is applied to the first image
  • a teacher image as a learning teacher
  • the normal equation is obtained by adding the coefficients of the terms (coefficient summation), and solving the normal equation minimizes the sum of the square errors (statistical error) of the predicted values y '.
  • a tap coefficient is determined.
  • the prediction process is a process of predicting a second image by applying a prediction formula to a first image.
  • a predicted value of the second image is obtained by performing a product-sum operation as an operation of a prediction expression using (pixel value of) a pixel of the first image.
  • Performing the sum-of-products operation using the first image can be referred to as a filtering process of applying a filter to the first image, and using the first image, the sum-of-products operation of the prediction formula (as the calculation of the prediction formula) Can be said to be a kind of filter processing.
  • the prediction tap is information such as (a pixel value of) a pixel used in the calculation of the prediction formula, and is multiplied by a tap coefficient in the prediction formula.
  • the prediction tap includes, in addition to the pixel (the pixel value) itself, a value obtained from the pixel, for example, a sum or an average value of the pixels (the pixel value) in a certain block.
  • selecting a pixel or the like as a prediction tap used in the calculation of the prediction equation is equivalent to providing (distributing) a connection line for supplying a signal to be input to a tap of the digital filter. Selecting a pixel as a prediction tap used in the calculation of the expression is also referred to as “setting a prediction tap”.
  • Class classification means that pixels are classified (clustered) into one of a plurality of classes.
  • the class classification can be performed, for example, using (pixel values of) pixels in a peripheral area of the target pixel and coding information related to the target pixel.
  • the coding information related to the target pixel includes, for example, a quantization parameter used for quantization of the target pixel, DF (Deblocking Filter) information on a deblocking filter applied to the target pixel, and the like.
  • the DF information is, for example, information indicating whether a strong filter or a weak filter has been applied or none of the filters have been applied in the deblocking filter.
  • Class classification prediction processing is filter processing as prediction processing performed for each class.
  • the basic principle of the classification prediction process is described in, for example, Japanese Patent No. 4449489.
  • the higher-order term is a term having a product of (two or more) prediction taps (of pixels) among the terms constituting a polynomial as a prediction equation.
  • the D-order term is a term having a product of D prediction taps among terms constituting a polynomial as a prediction equation.
  • the first-order term is a term having one prediction tap
  • the second-order term is a term having a product of two prediction taps.
  • the prediction taps taking the product may be the same prediction tap (pixel).
  • D-order coefficient means a tap coefficient constituting the D-order term.
  • the D-th tap means a prediction tap (pixel as) constituting the D-th term.
  • a certain pixel may be a D-th tap and a D 'next tap different from the D-th tap.
  • the tap structure of the D-th tap and the tap structure of the D 'next tap different from the D-th tap need not be the same.
  • the DC (Direct Current) prediction formula is a prediction formula including a DC term.
  • the DC term is a term of a product of a value representing a DC component of an image as a prediction tap and a tap coefficient among terms constituting a polynomial as a prediction equation.
  • DC tap means a prediction tap of a DC term, that is, a value representing a DC component.
  • DC coefficient means a tap coefficient of a DC term.
  • the primary prediction formula is a prediction formula consisting of only the first-order terms.
  • the higher-order prediction equation is a prediction equation including a higher-order term, that is, a prediction equation including a first-order term and a second-order or higher-order term, or a prediction equation including only a second-order or higher-order term.
  • the i-th prediction tap (pixel value or the like) used for prediction among the pixels of the first image is x i
  • the i-th tap coefficient is w i
  • DC prediction equation moistened with DC term to the primary prediction equation for example, the formula ⁇ w i x i + w DCB DCB .
  • w DCB represents a DC coefficient
  • DCB represents a DC tap.
  • the tap coefficients of the first-order prediction formula, the higher-order prediction formula, and the DC prediction formula can be obtained by performing the tap coefficient learning by the least square method as described above.
  • the tap structure means the arrangement of pixels as prediction taps (for example, based on the position of the pixel of interest).
  • the tap structure can be said to be a method of setting a prediction tap.
  • the tap structure considering a state where a tap coefficient to be multiplied by a pixel is arranged at a position of a pixel constituting the prediction tap, the tap structure can be said to be an arrangement of tap coefficients. . Therefore, the tap structure refers to the arrangement of the pixels forming the prediction tap of the target pixel, and the arrangement of the tap coefficients in a state where the tap coefficient multiplied by the pixel is arranged at the position of the pixel forming the prediction tap. Both are meant.
  • the activity (of the image) means the degree of change in the spatial pixel value of the image.
  • a decoded image is an image obtained by decoding encoded data obtained by encoding an original image.
  • the image obtained by local decoding of the prediction encoding is also included. included. That is, when the encoding apparatus predictively encodes the original image, the prediction image and the (decoded) residual are added in local decoding, and the addition result of the addition is a decoded image. is there.
  • the tilt direction (of the pixel) means the direction of the tilt of the pixel value, in particular, for example, the direction in which the tilt of the pixel value is the maximum.
  • the direction in which the inclination of the pixel value is the maximum is a direction orthogonal to the contour line of the pixel value, and is orthogonal to the tangent direction of the contour line of the pixel value, and thus has a one-to-one relationship with the tangent direction of the contour line of the pixel value. . Therefore, the direction in which the inclination of the pixel value is the maximum and the tangent direction of the contour line of the pixel value are equivalent information.
  • the direction in which the inclination of the pixel value is the largest is adopted as the inclination direction.
  • the specified direction means a predetermined discrete direction.
  • a method of expressing a direction for example, a method of expressing a continuous direction by a continuous angle, a method of expressing two kinds of discrete directions, that is, a horizontal direction and a vertical direction, and a method of expressing 360 degrees around an equal angle of 8 It is possible to adopt a method of classifying into eight directions and expressing them in eight discrete directions.
  • the specified direction means a direction represented by a predetermined discrete direction as described above.
  • the direction used in the GALF described in Non-Patent Document 2 and the direction represented by the direction class of the GALF Is an example of the prescribed direction.
  • inclination direction includes a direction continuously expressed by a continuous angle, and also includes a specified direction. That is, the inclination direction can be expressed by a continuous direction or a specified direction.
  • the tilt feature value is a feature value of an image indicating the tilt direction.
  • an activity in each direction and a gradient vector (gradient) obtained by applying a Sobel filter or the like to an image are examples of the gradient feature amount.
  • the tilt direction reliability means the reliability (probability) of the pixel tilt direction obtained by some method.
  • the initial class is a class in which tap coefficients are obtained in tap coefficient learning, and is a class before merging.
  • a merge class is a class obtained by merging one or more (one class) initial classes.
  • the number of merge classes is the number of merge classes obtained by merging the initial classes.
  • a merge pattern represents a correspondence between an initial class and a merge class obtained by merging the initial class. For example, a merge in which the initial class of the class number is merged in the order of the class number representing the initial class It is expressed in an expression format in which the class numbers of the classes are arranged.
  • FIG. 1 is a diagram for explaining the outline of the processing of the class classification unit 10 for classifying GALF.
  • FIG. 1 shows an outline of the classification of JVET (Joint @ Video. @ Exploration @ Team) -B0060.
  • the class classification unit 10 sequentially selects pixels to be subjected to the classification as a target pixel, and sets a plurality of pixels having the target pixel as a starting point. The activity in each direction is obtained as the inclination feature amount of the pixel of interest.
  • the class classification unit 10 employs, for example, four directions starting from the pixel of interest as a vertical direction, a vertical direction starting from the pixel of interest, a left direction as a horizontal direction, an upper left direction, and an upper right direction. Is done.
  • the upward direction, the left direction, the upper left direction, and the upper right direction are hereinafter also referred to as the V direction, the H direction, the D0 direction, and the D1 direction, respectively.
  • the point-symmetric directions (opposite directions) with respect to (the position of) the pixel of interest (the position) as the center of symmetry, respectively also referred to as D0 'direction and D1' direction.
  • the V direction, H direction, D0 direction, and D1 direction are directions in which an activity is required in the GALF class classification, and thus can be said to be the activity calculation directions.
  • the V direction, the H direction, the D0 direction, and the D1 direction, which are the activity calculation directions, are predetermined directions (one type) because they are predetermined discrete directions.
  • the classifying unit 10 applies, for example, a Laplacian filter to the decoded image including the target pixel, for the activity A (D) in the D direction (representing the V direction, the H direction, the D0 direction, or the D1 direction) of the target pixel. Ask by that.
  • the activities A (V), A (H), A (D0), and A (D1) in the V direction, H direction, D0 direction, and D1 direction of the pixel of interest are, for example, according to the following equation. You can ask.
  • a (V) abs ((L [y] [x] ⁇ 1)-L [y-1] [x]-L [y + 1] [x])
  • a (H) abs ((L [y] [x] ⁇ 1)-L [y] [x-1]-L [y] [x + 1])
  • a (D0) abs ((L [y] [x] ⁇ 1)-L [y-1] [x-1]-L [y + 1] [x + 1])
  • a (D1) abs ((L [y] [x] ⁇ 1)-L [y + 1] [x-1]-L [y-1] [x + 1]) ... (1)
  • L [y] [x] represents the pixel value (luminance value) of the pixel at the position of the y-th row and the x-th column of the decoded image, and in this case, the position of the y-th row and the x-th column of the decoded image Is the pixel of interest.
  • Abs (v) represents the absolute value of v
  • v ⁇ b represents shifting v left by b bits (multiplying by 2 b ).
  • the class classification unit 10 similarly calculates the activity of each of a plurality of pixels in the peripheral area of the target pixel. Then, the class classification unit 10 adds the activity of each of the plurality of pixels in the peripheral area of the target pixel for each of the V direction, the H direction, the D0 direction, and the D1 direction, and the V direction, the H direction, and the D0 direction. , And in each of the D1 directions, an added value of activities (hereinafter, also referred to as activity summation) is obtained.
  • an area of 3 ⁇ 3 pixels in the horizontal and vertical directions centering on the target pixel is a target to which the activity A (D) as the inclination feature amount is comprehensively used.
  • the activity sum (A) of the target pixel in the V direction is obtained by adding the activity A (V) of Equation (1) for each of the 3 ⁇ 3 pixels in the surrounding region.
  • Activity sums sumA (H), sumA (D0), and sumA (D1) of the target pixel in the H direction, D0 direction, and D1 direction are obtained in the same manner.
  • the peripheral area is not limited to the area of 3 ⁇ 3 pixels.
  • peripheral area to be used for the gradient feature amount comprehensively here, the peripheral area to be added with the activity A (D)
  • a 3 ⁇ 3 pixel area and a 6 ⁇ An area of six pixels or any other area including the pixel of interest can be employed.
  • the class classification unit 10 uses the activity sums sumA (V), sumA (H), sumA (D0), and sumA (D1) of the target pixel in the V direction, H direction, D0 direction, and D1 direction,
  • the direction of GALF is determined (set) as a specified direction indicating the inclination direction of the target pixel.
  • the GALF directions as defined directions are eight directions to which 000 to 111 are assigned in binary and 0 to 7 are assigned in decimal, as shown in FIG.
  • a direction between the H direction and the direction HD0 that bisects the H direction and the D0 direction, a direction between the directions HD0 and D0, a direction between the D0 direction, the D direction, and the V direction are defined as directions.
  • a binary number 110 in the direction between the directions HD0 and D0, a binary number 001, and in the direction between the D0 direction and the direction D0V, a binary number. 000, the binary number 010 in the direction between the direction D0V and the V direction, the binary number 011 in the direction between the V direction and the direction VD1, and the direction between the direction VD1 and the D1 direction.
  • GALF each of the above-described eight directions and a direction symmetrical with respect to each of the eight directions are treated as the same direction.
  • the ⁇ class classification unit 10 obtains (sets) a direction class representing the inclination direction of the target pixel from the direction as the specified direction of the target pixel.
  • the direction class of GALF indicates either of the V direction or the H direction, or two directions of the D0 direction or the D1 direction.
  • obtaining the direction class constitutes a part of the GALF class classification performed by the class classification unit 10, and can be called subclass classification.
  • the subclass classification for obtaining the direction class is hereinafter also referred to as a direction subclass classification.
  • the class classification unit 10 determines the direction class of the pixel of interest and the activity sums sumA (V), sumA (H), sumA (D0), and sumA (D1) in the V, H, D0, and D1 directions. ), The target pixel is classified into classes.
  • FIG. 2 is a diagram for explaining how to determine the direction of GALF as a specified direction (inclination direction) of the target pixel.
  • FIG. 2 shows the GALF class classification using the activity sums sumA (V), sumA (H), sumA (D0), and sumA (D1) in the V direction, H direction, D0 direction, and D1 direction.
  • An example of the required (set) MainDir and SecDir is shown.
  • FIG. 2 shows a direction class classification table showing the relationship between MainDir, SecDir, and direction, and the relationship between direction, transpose, and class used in the GALF class classification.
  • the class classification unit 10 After calculating the activity sums sumA (V), sumA (H), sumA (D0), and sumA (D1) in the V direction, H direction, D0 direction, and D1 direction, the class classification unit 10 The sumA (H) and the sumA (V) are compared, and the larger one is set as a first winner activity HVhigh, and the other is set as a first loser activity HVlow.
  • the class classification unit 10 compares the activity sums sumA (D0) and sumA (D1), and determines the larger one as the second winner activity Dhigh and the other as the second loser activity Dlow.
  • the classifying unit 10 calculates a product value HVhigh ⁇ Dlow of the first winner activity HVhigh and the second loser activity Dlow and a product value Dhigh ⁇ HVlow of the second winner activity Dhigh and the first loser activity HVlow. Compare with
  • the classifying unit 10 determines the direction (H direction or V direction) in which the first winner activity HVhigh was obtained as Main Dir (MainDirection). At the same time, the direction (D0 direction or D1 direction) in which the second winner activity Dhigh is obtained is determined as SecDir (SecondonDirection).
  • the classifying unit 10 determines the direction in which the second winner activity Dhigh is obtained as MainDir, and obtains the first winner activity HVhigh. The direction is determined to SecDir.
  • MainDir and SecDir of the target pixel are in the D0 direction and the V direction, respectively.
  • the class classification unit 10 determines the direction assigned to MainDir and SecDir of the target pixel in the direction class classification table to be the direction as the specified direction of the target pixel. Further, the class classification unit 10 determines the transpose and class assigned to the direction of the target pixel in the direction class classification table as the transpose and class of the target pixel.
  • the filter coefficient is transposed and used for the filtering process.
  • Transpose indicates the way of transposing the filter coefficient.
  • class represents a direction class.
  • GALF direction classes are two classes represented by decimal numbers 0 and 2.
  • the direction class can be obtained by taking the logical product of the direction of the target pixel and 010 in binary.
  • the direction class 0 indicates that the tilt direction is the D0 direction or the D1 direction
  • the direction class 2 indicates that the tilt direction is the V direction or the H direction.
  • FIG. 3 is a diagram for explaining classes obtained by GALF class classification.
  • the pixel of interest is classified into one of 25 classes (final) classes 0 to 24.
  • the class classification unit 10 determines the direction class of the pixel of interest, and the activity sums sumA (V), sumA (H), sumA (D0), and sumA in the V, H, D0, and D1 directions. Using (D1) as necessary, a gradient intensity ratio representing the gradient intensity of the pixel value of the pixel of interest is determined, and a class representing the gradient intensity ratio of the pixel of interest is determined (set) according to the gradient intensity ratio. ).
  • obtaining the class representing the gradient intensity ratio constitutes a part of the GALF class classification performed by the class classification unit 10, and thus can be called subclass classification.
  • the subclass classification for obtaining the class representing the gradient intensity ratio is hereinafter also referred to as the gradient intensity ratio subclass classification.
  • the class obtained by the subclass classification is hereinafter also referred to as a subclass.
  • the class classification unit 10 calculates the ratio r d1, d2 of the activity sums sumA (D0) and sumA (D1) in the D0 and D1 directions, and the activity sum sumA (V) in the V and H directions.
  • the ratio rh , v of sumA (H) is determined as the gradient intensity ratio according to equations (2) and (3), respectively.
  • the pixel of interest has a gradient intensity ratio
  • the gradient intensity ratio subclass is classified into a very small None class.
  • the class classification unit 10 invalidates (does not consider) the direction class (subclass) of the target pixel and sets V as the spatial feature amount of the target pixel.
  • V the activity sums sumA (V), sumA (H), sumA (D0), and sumA (D1) in the direction, the H direction, the D0 direction, and the D1 direction
  • the target pixel is classified into a final initial class ( Hereinafter, it is also referred to as a final class).
  • the class classification unit 10 obtains a class representing the size of the activity sum according to the activity sums sumA (V), sumA (H), sumA (D0), and sumA (D1).
  • obtaining the class representing the size of the activity sum is a subclass classification, like the gradient intensity ratio subclass classification, and is also referred to as an activity subclass classification hereinafter.
  • the activity sums sumA (V), sumA (H), sumA (D0), and sumA (D1) are used, and the activity sums sumA (V) and sumA (H) are used.
  • Clip (0, 15, X) indicates that X is clipped such that X takes a value in the range of 0 to 15.
  • an activity subclass is determined according to the index class_idx.
  • the activity subclass is set to 0 (Small class), and when the index class_idx is 1, the activity subclass is set to 1.
  • the index class_idx is 2 to 6, the activity subclass is 2.
  • the index class_idx is 7 to 14, the activity subclass is 3.
  • the index class_idx is 15, the activity subclass is 4 (Large class).
  • the target pixel classified as a non-class in the gradient intensity ratio subclass classification is classified into final classes 0 to 4, respectively.
  • the direction class of the pixel of interest is set to be valid (considered. Then, a gradient intensity ratio subclass classification is performed.
  • the gradient intensity ratio subclass classification (gradient intensity) according to the gradient intensity ratios r d1 and d2 of Expression (2)
  • the gradient intensity ratio subclass classification using the ratios r d1 and d2 and the gradient intensity ratio subclass classification of the gradient intensity ratios r d1 and d2 are performed.
  • the target pixel is classified into a gradient class having a small gradient intensity ratio in a weak class.
  • the class classification unit 10 determines the activity sum sumA (V, H, D0, and D1 directions as the spatial feature amount of the target pixel. According to V), sumA (H), sumA (D0), and sumA (D1), the target pixel is classified into the final class.
  • the gradient intensity ratio subclass classification is performed. Are classified into final classes 5 to 9, respectively.
  • Gradient intensity ratio r d1, d2 is, if the second threshold t 2 or more, the pixel of interest, gradient strength ratio is greater Strong (Strong) class gradient strength ratio sub-class classification.
  • the class classification unit 10 determines the activity sum sumA (V direction, H direction, D0 direction, and D1 direction as the spatial feature amount of the target pixel. According to V), sumA (H), sumA (D0), and sumA (D1), the target pixel is classified into the final class.
  • the gradient intensity ratio subclass classification is performed. Are classified into the final classes 10 to 14, respectively.
  • the gradient intensity ratio subclass classification is performed according to the gradient intensity ratio rh , v in Expression (3).
  • Gradient intensity ratio r h, v is the first case is a threshold value t 1 or more second less than the threshold t 2, the target pixel is inclined intensity ratio is smaller Week (Weak) class gradient strength ratio sub-class classification.
  • the class classification unit 10 determines the activity sum sumA (V, H, D0, and D1 directions as the spatial feature amount of the target pixel. In accordance with V), sumA (H), sumA (D0), and sumA (D1), the target pixel is classified into any of the final classes 15 to 19.
  • the gradient intensity ratio subclass classification is performed. Are classified into final classes 15 to 19, respectively.
  • Gradient intensity ratio r h, v is, if the second threshold t 2 or more, the pixel of interest, gradient strength ratio is greater Strong (Strong) class gradient strength ratio sub-class classification.
  • the class classification unit 10 determines the activity sum sumA (V, H, D0, and D1 directions as the spatial feature of the target pixel. According to V), sumA (H), sumA (D0), and sumA (D1), the target pixel is classified into one of the final classes 20 to 24.
  • the gradient intensity ratio subclass classification is performed. Are classified into the final classes 20 to 24, respectively.
  • the class c means a class having a class number c for specifying the class.
  • FIG. 4 is a flowchart illustrating a GALF process that the encoding device that encodes an image has as one of the ILFs.
  • step S11 GALF sequentially selects pixels of a decoded image (for example, one picture) obtained by local decoding in the encoding device as a pixel of interest, and the process proceeds to step S12.
  • a decoded image for example, one picture
  • step S12 the GALF classifies the target pixel as described with reference to FIGS. 1 to 3, sets the final classes 0 to 24 to 25 initial classes 0 to 24, and sets the target pixel to one of the initial classes. And the process proceeds to step S13.
  • the i-th prediction tap (pixel value or the like) used for prediction is x i
  • the i-th tap coefficient is w i
  • the pixel value of the pixel of the original image the pixel value of ) Is represented as y.
  • primary prediction equation y ⁇ w i x i
  • X represents a matrix of N rows and N columns having a sum of the products of the prediction taps x i and x j as elements, and W is a matrix of N rows and 1 column having tap coefficients w i as elements (column vector ).
  • step S14 GALF solves the normal equation for each initial class by, for example, Cholesky decomposition or the like to obtain tap coefficients for each initial class, and the process proceeds to step S15.
  • the process of obtaining the tap coefficients for each initial class as in steps S11 to S14 is tap coefficient learning.
  • step S15 GALF performs a class merge process of merging the initial class in order to reduce (the data amount of) the tap coefficient, and the process proceeds to step S16.
  • a merge pattern determination process is performed in step S21, and a adopted merge class number determination process is performed in step S22.
  • step S16 GALF performs a GALF filter process, and the process proceeds to step S17.
  • the tap coefficient for each merge class is obtained by the merge pattern determination processing in step S21.
  • step S17 GALF encodes the tap coefficient for each merge class obtained by converting the initial class according to the merge pattern corresponding to the number of adopted merge classes, the number of adopted merge classes, and the merge pattern corresponding to the number of adopted merge classes, The process proceeds to step S18.
  • step S18 the GALF makes an RD (Rate Distortion) determination to determine whether to perform filter processing on the decoded image, and the process ends.
  • RD Rate Distortion
  • FIG. 5 is a flowchart illustrating the merge pattern determination process in step S21 of FIG.
  • step S31 GALF sets the initial class number (initial class number) Cini as the initial value in the merge class number (variable representing) C, and the process proceeds to step S32.
  • the number of merged classes C is the number of initial classes Cini
  • none of the initial classes is merged, but for convenience, it is treated as a state in which 0 initial classes are merged.
  • step S32 GALF sets 0 to the merge class (variable representing), and the process proceeds to step S33.
  • the merge class number C is the initial class number Cini
  • the merge class c is the initial class c.
  • step S33 GALF obtains an X matrix and a Y vector constituting a normal equation (made when tap coefficients are obtained) of the merge class c, and the process proceeds to step S34.
  • the merge class number C is the initial class number Cini
  • the merge class c is the initial class c
  • the normal equation of the merge class c is the initial class c obtained in step S13 (FIG. 4). Is a normal equation. If the merge class number C is smaller than the initial class number Cini, the normal equation of the merge class c is the normal equation of the initial class c, and the normal equation of the merge class c formed (set) in step S36 described later. This is a normal equation, a normal equation of an initial class c ′, or a normal equation of a merge class c ′ configured in step S36, and the class number is sorted to c in step S44 described below.
  • step S34 GALF sets c + 1 in the merge class (variable representing m) m, and the process proceeds to step S35.
  • step S35 GALF obtains an X matrix and a Y vector constituting the normal equation of the merge class m as in step S33, and the process proceeds to step S36.
  • step S36 GALF adds the elements of the X matrix forming the normal equation of the merge class c and the elements of the X matrix forming the normal equation of the merge class m. Further, GALF adds the elements of the Y vector forming the normal equation of the merge class c and the Y vector forming the normal equation of the merge class m. Then, GALF sets a normal equation of a new merge class c obtained by merging the merge classes c and m, which is composed of the X matrix and the Y vector after addition, and the process proceeds from step S36 to step S37.
  • step S37 GALF obtains (calculates) a tap coefficient of the new merge class c by solving the normal equation of the new merge class c including the added X matrix and Y vector. Proceed to step S38.
  • step S38 GALF uses the tap coefficients of the new merge class c and the tap coefficients other than the merge classes c and m in the C class (C merge classes 1, 2,, C). Then, filter processing is performed on the decoded image. Then, GALF obtains an error of the filtered image obtained by the filtering process with respect to the original image, and the process proceeds to step S39.
  • step S39 If it is determined in step S39 that the merge class m is not equal to C-1, that is, if the merge class m is less than C-1, the process proceeds to step S40.
  • step S40 GALF increments the merge class m by 1, and the process returns to step S35, and the same process is repeated thereafter.
  • step S39 if it is determined in step S39 that the merge class m is equal to C-1, that is, the merge class c is merged with the merge classes c + 1, c + 2,,.
  • the process proceeds to step S41.
  • step S41 GALF determines whether (the class number of) the merge class c is equal to C-2.
  • step S41 If it is determined in step S41 that the merge class c is not equal to C-2, that is, if the merge class c is less than C-2, the process proceeds to step S42.
  • step S42 GALF increments the merge class c by one, and the process returns to step S33, and thereafter, the same process is repeated.
  • step S41 when it is determined in step S41 that the merge class c is equal to C-2, that is, for C merge classes 1, 2,,.
  • step S43 When (C-1) / 2 types of merging are performed and the error of the filter image is obtained for each of C (C-1) / 2 types of merging, the process proceeds to step S43.
  • step S43 in the C (C-1) / 2 types of merging of any two of the C merge classes 1, 2,,.
  • GALF merges the merge classes c and m that are the targets of the optimal merge into a new merge class c, assuming that the smallest merge is the optimal merge that merges the number of merge classes from C to C-1. Then, the process proceeds to step S44. That is, GALF sets the class number m of the merge class m to the class number c of the new merge class c.
  • step S44 the GALF converts the class numbers excluding m in the class numbers c + 1 to C-1 into class numbers c + 1 to C-2 in ascending order, and the process proceeds to step S45.
  • step S44 since the class number m is set to the class number c in step S43, when the process of step S44 is performed, the class number m does not exist in the class numbers c + 1 to C-1.
  • step S45 GALF decrements the number C of merge classes by one, and the process proceeds to step S46.
  • step S46 the merge pattern representing the correspondence between the Cini initial classes and the C merge classes after the merge classes c and m are merged into the new merge class c is the optimal merge of the merge class number C.
  • the pattern is a pattern
  • GALF stores the optimal merge pattern of the merge class number C as a merge pattern corresponding to the merge class number C, and the process proceeds to step S47.
  • step S47 GALF determines whether the number C of merge classes is equal to one.
  • step S47 If it is determined in step S47 that the number C of merge classes is not equal to 1, the process returns to step S32, and the same process is repeated.
  • step S47 when it is determined that the number C of merge classes is equal to 1, the merge pattern determination processing ends.
  • FIG. 6 is a diagram for explaining an example of the expression format of the merge pattern.
  • a merge pattern is expressed in the following expression format.
  • the merge pattern indicates the correspondence between the initial class and the merge class that merges the initial class. For example, in the order of the class numbers arranged in the initial class table, the merge class of the class having the class number is merged. Expressed by arranging class numbers.
  • the initial class table is a table in which the class numbers of the initial classes are arranged.
  • the class numbers 0 to 24 of the 25 initial classes obtained by the GALF class classification are arranged in ascending order.
  • BB in FIG. 6 shows an example of the merge pattern.
  • the class numbers of the merge class into which the class of the class number is merged are arranged in the order of the class numbers arranged in the initial class table.
  • the initial class table and the merge pattern are expressed in a 5 ⁇ 5 table format.
  • the expression format of the initial class table and the merge pattern is not limited to the table format. An expression format in which the class numbers are arranged separated by commas or spaces may be used.
  • the number of initial classes in which class numbers are arranged in the initial class table (the number of initial classes), and the number of merge classes obtained by performing merging according to the merge pattern (the number of merge classes) ) are shown at the top of the initial class table and the table as the merge pattern, as appropriate.
  • Numeral 25 at the upper left of the initial class table of FIG. 6A represents the number of initial classes
  • numeral 5 at the upper left of the merge pattern of FIG. 6B represents the number of merge classes.
  • FIG. 7 is a diagram illustrating an example of a merge pattern for each number of merge classes.
  • FIG. 7 shows an example of an optimal merge pattern of each merge class number, with each natural number value equal to or less than 25 as the number of initial classes (initial class number) obtained by the GALF class classification as the merge class number. I have.
  • a circle attached to the merge pattern of the merge class number C indicates that the merge class number is changed from C + 1 to C in the merge classes obtained according to the merge pattern corresponding to the merge class number C + 1. Represents a merge class to be merged into another merge class by the merge.
  • the circle number is given to the class number 6 arranged at the 16th position. This is because, in a merge where the number of merge classes is changed from 25 to 24, the merge class of the class number 15 arranged at the 16th position of the merge pattern corresponding to the merge class number 25 becomes the 16th of the merge pattern corresponding to the merge class number 24. This indicates that the merge pattern is to be merged with the merge class with the class number 6 placed at the seventh position (which is also the merge class with the class number 6 placed at the seventh position in the merge pattern corresponding to the merge class number 24).
  • the merge pattern corresponding to the number of merge classes 25 equal to the initial class number, none of the initial classes is used.
  • a merge pattern that is not merged but corresponds to a merge class number 25 equal to the initial class number is treated as a merge pattern in which 0 initial classes are merged for convenience of explanation.
  • the merge pattern corresponding to the merge class number 25 is equal to the initial class table.
  • the merge pattern determination process (FIG. 5) after merging the merge classes c and m into a new merge class c in step S43, series sorting of class numbers is performed in step S44. Therefore, in the merge pattern corresponding to each merge class number in FIG. 7, the maximum value of the class number is a value corresponding to the merge class number C, that is, the merge class number C-1.
  • merge two arbitrary merge classes C ( C-1) / 2 types of merge are performed. Then, of the C (C-1) / 2 types of merges, the merge that minimizes the error of the filter image is determined as the optimal merge to the merge class number C-1, and the merge pattern of the merge is It is obtained as a merge pattern corresponding to the merge class number C-1.
  • the merge class number C is 25.
  • the merge pattern corresponds to the number of merge classes 25 and the merge pattern corresponding to the number of merge classes 1 respectively.
  • the number C of merge classes is any one of 2 to 24
  • the number when merging any two of the merge classes of the number C of merge classes is C (C-1) /
  • C (C-1) / 2 types of merges are performed, and a filter process is performed using tap coefficients obtained by each merge, thereby obtaining a filter image error. Then, the merge pattern of the merge that minimizes the error of the filter image is determined as the merge class corresponding to the merge class number C-1.
  • FIG. 8 is a flowchart illustrating the adopted merge class number determination process in step S22 of FIG.
  • step S62 GALF obtains (loads) a merge pattern corresponding to the merge class number C obtained in the merge pattern determination process (FIG. 5), and the process proceeds to step S63.
  • step S63 GALF calculates the tap coefficient of the C class (minutes) when 25 initial classes are merged into the C class merge class (C merge classes) according to the merge pattern corresponding to the merge class number C. Is acquired (loaded), and the process proceeds to step S64.
  • the tap coefficients of the C class (merge class) when the 25 initial classes are merged into the C class merge class are determined in step S37 of the merge pattern determination process. Already sought.
  • step S64 GALF performs a GALF filter process using the tap coefficient of the C class, and the process proceeds to step S65.
  • GALF sequentially selects the pixels of the decoded image as the pixel of interest, and performs class classification of the pixel of interest (class classification for the pixel of interest). Further, GALF converts the initial class of the pixel of interest obtained by the class classification of the pixel of interest into a merge class according to a merge pattern corresponding to the number C of merge classes. Then, GALF performs a filter process using the tap coefficient of the merge class of the pixel of interest among the tap coefficients of the C class acquired in step S63, and obtains a filter image.
  • step S65 the GALF obtains an error dist of the filter image obtained by performing the filter process using the tap coefficient of the target pixel in the merge class with respect to the original image, and the process proceeds to step S66.
  • step S66 the GALF is obtained by merging the parameters necessary for the GALF filter processing in the decoding device, that is, the number C of merge classes, the merge pattern corresponding to the number C of merge classes, and the initial class according to the merge pattern.
  • the code amount coeffBit of the tap coefficient of the C class is obtained, and the process proceeds to step S67.
  • step S68 GALF determines whether or not the number C of merge classes is equal to one.
  • step S68 If it is determined in step S68 that the number C of merge classes is not equal to 1, the process proceeds to step S69.
  • step S69 GALF decrements the number C of merge classes by one, and the process returns to step S62, and thereafter, the same process is repeated.
  • step S68 If it is determined in step S68 that the number C of merge classes is equal to 1, the process proceeds to step S70.
  • step S70 the merge with the minimum cost among the merges into the 1 class or the Cini class is determined as the adoption merge adopted in the GALF filter processing.
  • the GALF determines the number of merge classes in the merge pattern when performing the adoption merge. , The number of adopted merge classes is determined, and the adopted merge class number determination process ends.
  • the merge pattern that minimizes the cost is selected.
  • the number of merge classes is determined as the number of adopted merge classes.
  • FIG. 9 is a diagram showing an example of a merge pattern transmitted from the encoding device to the decoding device.
  • the merge pattern is represented by an array variable mergeInfo [25] in which the class number of the merge class into which the 25 initial classes are merged is set.
  • the ith numeral j from the beginning (left) indicates that the initial class of the class number i is converted (merged) into the merge class of the class number j.
  • the merge pattern determination processing for determining the merge pattern corresponding to each of the merge class numbers C of 1 to 25, it is necessary to perform 2600 merges, which increases the processing amount. Furthermore, in GALF, it is necessary to transmit a merge pattern from an encoding device to a decoding device.
  • a merge pattern corresponding to the number of merge classes is set in advance for each number of merge classes, and the initial class is converted into a merge class according to the preset merge pattern.
  • FIGS. 10 and 11 are diagrams showing a first example of a preset merge pattern.
  • FIGS. 10 and 11 show examples of merge patterns corresponding to the number of merged classes 25, $ 20, $ 15, $ 10, $ 5, $ 3, $ 2, $ 1, which merge 25 initial classes obtained by GALF class classification.
  • FIG. 10 shows examples of merge patterns corresponding to the number of merged classes 25, $ 20, $ 15, $ 10, $ 5, $ 3, $ 2, $ 1, which merge 25 initial classes obtained by GALF class classification.
  • a merge pattern corresponding to the number of merge classes is set in advance, and the initial class is converted into a merge class according to the preset merge pattern.
  • the merge pattern is set in advance for each number of merge classes, if the number of merge classes is specified, the merge pattern is also uniquely specified. Therefore, by sharing the preset merge pattern between the encoding device and the decoding device, there is no need to transmit the merge pattern from the encoding device to the decoding device. Encoding efficiency can be improved.
  • the number of merge classes in which the merge pattern is set in advance does not need to be a continuous natural number, but may be a natural number of discrete values.
  • a merge pattern for each number of merge classes can be set by an arbitrary method. However, if a merge pattern is set by an arbitrary method, the performance of the filter processing may be degraded and the image quality of the filter image may be degraded.
  • performing a predetermined class classification for classifying the target pixel into the initial class, converting the initial class obtained by the predetermined class classification according to the merge pattern, and obtaining the merge class includes classifying the target pixel into the merge class. It can be understood that it is a class classification. In this case, it can be considered that the merge pattern for converting the initial class into the merge class determines the classification rule (class classification method) of class classification into the merge class. Therefore, the setting of the merge pattern can be performed by determining the classification rule of the class classification into the merge class.
  • Deterioration of the performance of the filtering process can be attributed to the information that affects the class classification into the merge class among the information such as the pixel feature amount used for the class classification for obtaining the initial class for each number of merge classes, and the merging of the information
  • a classification rule of class classification into a merge class such as a method of assigning a class (a subclass thereof) (for example, which merge class is to be allocated to which range of a feature amount), and set a merge pattern. Can be suppressed.
  • suppressing the deterioration of the performance of the filtering process is set as a setting policy for setting a merge pattern, and a merge pattern corresponding to each number of merge classes is set according to a setting rule that does not violate the setting policy.
  • a reduction setting for setting a merge pattern for each number of merge classes such that the number of classes decreases from an initial class obtained by predetermined class classification can be adopted.
  • a merge pattern for merging an initial class obtained by a predetermined class classification and a merge pattern for merging an initial class obtained by another class classification are mixed. Can be adopted.
  • a setting rule when an image for setting a merge pattern prepared in advance is encoded as an original image, the sign of parameters (tap coefficients for each merge class and the number of adopted merge classes) required for filter processing is used.
  • a statistical setting for setting a merge pattern for each number of merge classes can be employed so that one or both of the amount and the error of the filter image with respect to the original image are statistically optimized.
  • a merge pattern determination process performed by GALF is performed offline in advance, and the merge pattern determination performed offline is performed.
  • a merge pattern corresponding to each merge class number obtained by the processing can be set as a merge pattern for each merge class number.
  • FIGS. 10 and 11 show examples of reduction settings for merge patterns.
  • the merge pattern for each number of merge classes is set so that the number of classes decreases from the initial class obtained by the predetermined class classification.
  • GALF class classification is adopted as the predetermined class classification.
  • a merge pattern for each number of merge classes can be set so that a merge class in which any one of the information used for the predetermined class classification is preferentially influenced is obtained.
  • the information used for the GALF class classification includes, as described with reference to FIGS. 1 to 3, the gradient intensity ratio, the direction class, and the activity sum (the activity subclass). ).
  • a merge pattern for each number of merge classes can be set so that a merge class in which the gradient intensity ratio or the activity sum affects preferentially is obtained.
  • the merge patterns in FIGS. 10 and 11 are such that a merge class in which the gradient intensity ratio affects preferentially is obtained.
  • the merge pattern corresponding to the number of merge classes 25 is one of three subclasses of a non-class, a weak class, and a strong class according to the gradient intensity ratio.
  • Crab gradient intensity ratio subclass classification, activity subclass classification into one of 5 subclasses according to activity sum, and gradient intensity ratio subclass obtained by gradient intensity ratio subclass classification according to gradient intensity ratio is other than non-class
  • the class obtained by the classification rule of classifying into all 25 classes is obtained as a merge class It has a merge pattern. That is, the merge pattern corresponding to the merge class number 25 is a merge pattern that can obtain the same merge class as the initial class obtained by the GALF class classification.
  • the H / V class means the direction class 2 (a subclass indicating that the tilt direction is the V direction or the H direction) described in FIGS. 1 to 3.
  • the D0 / D1 class means the direction class 0 (a subclass indicating that the tilt direction is the D0 direction or the D1 direction) described with reference to FIGS.
  • the merge pattern corresponding to the merge class number 20 classifies the target pixel into one of three subclasses of a non-class, a weak class, and a strong class according to the gradient intensity ratio, According to the activity sum, the activity subclass is classified into one of the four subclasses. If the gradient intensity ratio subclass is other than the non-class, any one of the H / V class and the D0 / D1 class is selected according to the direction class. By performing the direction subclass classification, the class obtained by the classification rule for classification into all 20 classes is a merge pattern obtained as a merge class.
  • the merge pattern corresponding to the merge class number 15 classifies the pixel of interest into one of three subclasses of a non-class, a weak class, and a strong class according to the gradient intensity ratio, According to the activity sum, the activity subclass is classified into one of the three subclasses. If the gradient intensity ratio subclass is other than the non-class, any one of the H / V class and the D0 / D1 class is selected according to the direction class. By performing the direction subclass classification, the class obtained by the classification rule for classification into all 15 classes is a merge pattern obtained as a merge class.
  • the merge pattern corresponding to the merge class number 10 classifies the target pixel into one of three subclasses of a non-class, a weak class, and a strong class according to the gradient intensity ratio, According to the activity sum, the activity subclass is classified into one of the two subclasses. If the gradient intensity ratio subclass is other than the non-class, one of the two subclasses of the H / V class and the D0 / D1 class depending on the direction class.
  • the class obtained by the classification rule for classification into all 10 classes is a merge pattern obtained as a merge class.
  • the merge pattern corresponding to the merge class number 5 classifies the target pixel into one of three subclasses of a non-class, a weak class, and a strong class according to the gradient intensity ratio, If the gradient strength ratio subclass is other than the non-class, according to the direction class, by classifying the direction subclass into one of 2 subclasses of H / V class and D0 / D1 class, according to the classification rules to classify into all 5 classes
  • the obtained class is a merge pattern obtained as a merge class.
  • the merge pattern corresponding to the number of merge classes 3 is to classify the pixel of interest into one of three subclasses of a non-class, a weak class, and a strong class according to the gradient intensity ratio.
  • the class obtained by the classification rule for classifying into all three classes is a merge pattern obtained as a merge class.
  • the merge pattern corresponding to the number of merge classes 2 is obtained by classifying the pixel of interest into one of two subclasses of a non-class and a weak / strong class according to the gradient intensity ratio.
  • a class obtained by the classification rule for classifying into two classes is a merge pattern obtained as a merge class.
  • the weak / strong class is a combination of the weak class and the strong class when the gradient intensity ratio subclass is classified into one of three subclasses of a non-class, a weak class, and a strong class according to the gradient intensity ratio.
  • the (merged) class is a combination of the weak class and the strong class when the gradient intensity ratio subclass is classified into one of three subclasses of a non-class, a weak class, and a strong class according to the gradient intensity ratio.
  • the (merged) class is a combination of the weak class and the strong class when the gradient intensity ratio subclass is classified into one of three subclasses of a non-class, a weak class, and a strong class according to the gradient intensity ratio.
  • the class obtained by classifying the target pixel into one class is a merge pattern obtained as a merge class.
  • classifying the target pixel into one class does not perform class classification, that is, it can be considered that there is no class.
  • the one class is also referred to as a mono class.
  • the merge pattern corresponding to the merge class number 1 is a merge pattern in which a mono class is obtained as a merge class.
  • FIG. 12 is a diagram for explaining a method of setting a merge pattern corresponding to a merge class number of 25 in which 25 initial classes obtained by GALF class classification are merged into 25 merge classes.
  • FIG. 12 shows a classification rule for performing class classification into merge classes obtained (by converting an initial class) according to a merge pattern corresponding to the number of merge classes 25 in FIG.
  • the target pixel is a non-class, a weak class, and a weak class according to the gradient intensity ratio.
  • the gradient strength ratio subclass is classified into one of the three subclasses of the Strong class, and the activity subclass is classified into any one of the five subclasses according to the activity sum as the spatial feature.
  • the direction subclass is classified into one of two subclasses of the H / V class and the D0 / D1 class, so that the classification is performed into any of the merge classes 0 to 24.
  • the target pixel in the activity subclass classification into any of the five subclasses, the target pixel is 0 when the index class_idx is 0 according to the index class_idx obtained from the activity sum as described in FIG. 3. If it is classified into activity subclass 0 (Small class) and the index class_idx is 1, it is classified into activity subclass 1 and if the index class_idx is 2 to 6, it is classified into activity subclass 2 and the index class_idx is 7 to 14 If there is, it is classified into activity subclass 3, and if the index class_idx is 15, it is classified into activity subclass 4 (Large class).
  • the target pixel is classified into merge class 0 when the activity subclass is 0, and merged when the activity subclass is 1.
  • the target pixel is classified into merge class 0 when the activity subclass is 0, and merged when the activity subclass is 1.
  • merge class 2 when classified into merge class 2
  • the activity subclass when the activity subclass is 3, it is classified into merge class 3
  • the activity subclass when the activity subclass is 4, it is classified into merge class 4.
  • the target pixel is classified into the merge class 5 when the activity subclass is 0, and is merged into the merge class 6 when the activity subclass is 1.
  • the activity subclass is 2, it is classified into a merge class 7, when the activity subclass is 3, it is classified into a merge class 8, and when the activity subclass is 4, it is classified into a merge class 9.
  • the target pixel is classified into the merge class 10 when the activity subclass is 0, and is merged into the merge class 11 when the activity subclass is 1.
  • the activity subclass is 2, it is classified into the merge class 12, when the activity subclass is 3, it is classified into the merge class 13, and when the activity subclass is 4, it is classified into the merge class 14.
  • the target pixel is classified into the merge class 15 when the activity subclass is 0, and is merged into the merge class 16 when the activity subclass is 1.
  • the activity subclass is 2, it is classified into the merge class 17, when the activity subclass is 3, it is classified into the merge class 18, and when the activity subclass is 4, it is classified into the merge class 19.
  • the target pixel is classified into the merge class 20 when the activity subclass is 0, and is merged into the merge class 21 when the activity subclass is 1.
  • the activity subclass is 2, it is classified into the merge class 22, when the activity subclass is 3, it is classified into the merge class 23, and when the activity subclass is 4, it is classified into the merge class 24.
  • Merge classes 0 to 24 obtained by the classification according to the classification rules of FIG. 12 respectively correspond to initial classes 0 to 24 obtained by the GALF classification. Therefore, as a merge pattern corresponding to the number of merge classes 25, a merge pattern that converts (merges) the initial classes 0 to 24 into the merge classes 0 to 24 can be set.
  • FIG. 13 is a diagram for explaining a method of setting a merge pattern corresponding to a merge class number of 20 in which 25 initial classes obtained by GALF class classification are merged into 20 merge classes.
  • FIG. 13 shows a classification rule for performing class classification into merge classes obtained according to the merge pattern corresponding to the number of merge classes 20 in FIG.
  • the target pixel has an index class_idx of 0 or 1 according to the index class_idx obtained from the activity sum as described in FIG. If the activity subclass is classified into a small (small) class, if the index class_idx is 2 to 6, if the activity subclass is classified into the Middle 1 (Middle1) class, if the index class_idx is 7 to 14, the activity It is classified into a Middle2 class as a subclass, and when the index class_idx is 15, it is classified into a Large class as an activity subclass.
  • the number of activity subclasses is four, which is one less than the number of activity subclasses in the initial class.
  • the small class matches the activity subclasses 0 and 1 in the initial class, and the middle 1 class, the middle 2 class, and the large class match the activity subclasses 2, 3, and 4 in the initial class, respectively.
  • the assignment of the subclass to the activity sum is reduced by one subclass compared to the case of the GALF classification, so that the activity sum affects the classification of the merge class. Disappears.
  • a merge class is obtained in which the gradient intensity ratio and the direction class have a higher priority than the activity sum.
  • the target pixel is classified into the merge class 0 when the activity subclass is the small class, and when the activity subclass is the middle 1 class, When the activity subclass is the middle 2 class, it is classified into the merge class 2, and when the activity subclass is the large class, it is classified into the merge class 3.
  • the target pixel is classified into the merge class 4 when the activity subclass is the small class, and when the activity subclass is the middle 1 class, When the activity subclass is the middle 2 class, it is classified into the merge class 6, and when the activity subclass is the large class, it is classified into the merge class 7.
  • the target pixel is classified into the merge class 8 when the activity subclass is the small class, and when the activity subclass is the middle 1 class, If the activity subclass is a middle class, it is classified into a merge class 10, and if the activity subclass is a large class, it is classified into a merge class 11.
  • the pixel of interest is classified into the merge class 12 when the activity subclass is the small class, and when the activity subclass is the middle 1 class, If the activity subclass is a middle class, it is classified into a merge class 14, and if the activity subclass is a large class, it is classified into a merge class 15.
  • the pixel of interest is classified into the merge class 16 when the activity subclass is the small class, and when the activity subclass is the middle 1 class, If the activity subclass is a middle class, it is classified into a merge class 18. If the activity subclass is a large class, it is classified into a merge class 19.
  • merge class 0 matches initial classes 0 and 1 obtained by GALF class classification
  • merge classes 1 to 3 correspond to initial classes 2 to 4 obtained by GALF class classification
  • Match merge class 4 matches initial classes 5 and 6 obtained by GALF classification
  • merge classes 5 to 7 match initial classes 7 to 9 obtained by GALF classification
  • merge Class 8 matches the initial classes 10 and 11 obtained by the GALF classification
  • merge classes 9 to 11 respectively match the initial classes 12 to 14 obtained by the GALF classification
  • the merge class 12 Match the initial classes 15 and 16 obtained by the GALF classification
  • the merged classes 13 to 15 become the initial classes 17 to 19 obtained by the GALF classification.
  • Are matched, merged class 16 matches the initial class 20 and 21 obtained by the class classification GALF, merge classes 17 to 19, to an initial class 22 not obtained by the class classification GALF matches respectively 24.
  • the initial classes 0 and 1 become the merge class 0
  • the initial classes 2 to 4 become the merge classes 1 to 3, respectively
  • the initial classes 5 and 6 become the merge class 4.
  • the initial classes 7 to 9 respectively into the merge classes 5 to 7, the initial classes 10 and 11 into the merge class 8, the initial classes 12 to 14 into the merge classes 9 to 11, respectively, and the initial classes 15 and 16 into the merge class.
  • a merge pattern for converting each of the initial classes 17 to 19 into a merge class 13 to 15, each of the initial classes 20 and 21 into a merge class 16, and each of the initial classes 22 to 24 into a merge class 17 to 19, respectively. Can be set.
  • FIG. 14 is a diagram for explaining a method of setting a merge pattern corresponding to a merge class number of 15 in which 25 initial classes obtained by GALF class classification are merged into 15 merge classes.
  • FIG. 14 shows a classification rule for performing class classification into merge classes obtained according to the merge pattern corresponding to the number of merge classes 15 in FIG.
  • the pixel of interest is classified into one of three subclasses of a non-class, a weak class, and a strong class according to the gradient intensity ratio, and the activity as a spatial feature amount is performed.
  • the activity subclass is classified into one of the three subclasses. If the gradient intensity ratio subclass is other than the non-class, the activity subclass is classified into one of the H / V class and the D0 / D1 class according to the direction class. By being classified in the direction subclass, it is classified into one of the merge classes 0 to 14.
  • the classification rule in FIG. 14 is a rule in which the small class as the activity subclass and the middle 1 class in the classification rule in FIG. 13 are merged to reduce the number of activity subclasses from four to three.
  • the target pixel in the activity subclass classification into any of the three subclasses, has an index class_idx of 0 to 6 according to the index class_idx obtained from the activity sum as described in FIG. In the case, it is classified into a small (small) class as an activity subclass, and if the index class_idx is 7 to 14, it is classified into a middle (Middle) class as an activity subclass, and if the index class_idx is 15, it is classified as an activity subclass. It is classified into the Large class.
  • the number of activity subclasses is 3 which is reduced by 2 from the number 5 of activity subclasses in the initial class.
  • the small class matches the activity subclasses 0 to 2 in the initial class, and the middle class and the large class match the activity subclasses 3 and # 4 in the initial class, respectively.
  • the assignment of the subclass to the activity sum is reduced by two subclasses as compared with the case of the GALF classification, so that the activity sum affects the classification of the merge class. Disappears.
  • a merge class in which the gradient intensity ratio and the direction class have a higher priority than the activity sum is obtained.
  • the target pixel is classified into the merge class 0 when the activity subclass is the small class, and is merged when the activity subclass is the middle class. If the activity subclass is classified into class 1 and the activity subclass is a large class, it is classified into merge class 2.
  • the target pixel is classified into the merge class 3 when the activity subclass is the small class, and is merged when the activity subclass is the middle class. If the activity subclass is classified into class 4 and the activity subclass is a large class, it is classified into merge class 5.
  • the target pixel is classified into the merge class 6 when the activity subclass is the small class, and is merged when the activity subclass is the middle class. Classified into class 7, and when the activity subclass is a large class, it is classified into merge class 8.
  • the target pixel is classified into the merge class 9 when the activity subclass is the small class, and merged when the activity subclass is the middle class. If the activity subclass is a large class, it is classified into a merge class 11.
  • the target pixel is classified into the merge class 12 when the activity subclass is the small class, and is merged when the activity subclass is the middle class. If the activity subclass is a large class, it is classified into a merge class 14.
  • the merge class 0 matches the initial classes 0 and 2 obtained by the GALF class classification
  • the merge classes 1 and 2 correspond to the initial classes 3 and 4 obtained by the GALF class classification, respectively.
  • Merge class 3 matches the initial classes 5 to 7 obtained by the GALF classification
  • merge classes 4 and 5 respectively match the initial classes 8 and 9 obtained by the GALF classification.
  • Class 6 matches the initial classes 10 to 12 obtained by the GALF class classification
  • Merge Classes 7 and 8 respectively match the initial classes 13 and 14 obtained by the GALF class classification
  • Merge Class 9 The merged classes 10 and 11 correspond to the initial classes 18 and 19 obtained by the GALF classification, respectively.
  • the merge class 12 matches the initial classes 20 to 22 obtained by the GALF classification
  • the merge classes 13 and 14 match the initial classes 23 and 24 obtained by the GALF classification.
  • the initial classes 0 to 3 are set to the merge class 0
  • the initial classes 3 and 4 are set to the merge classes 1 and 2, respectively
  • the initial classes 5 to 7 are set to the merge class 3.
  • the initial classes 8 and 9 respectively into the merge classes 4 and 5, the initial classes 10 through 12 into the merge class 6, the initial classes 13 and 14 into the merge classes 7 and 8, respectively, and the initial classes 15 through 17 into the merge class.
  • FIG. 15 is a diagram for explaining a method of setting a merge pattern corresponding to 10 merge classes, in which 25 initial classes obtained by GALF class classification are merged into 10 merge classes.
  • FIG. 15 shows a classification rule for performing class classification into merge classes obtained according to a merge pattern corresponding to the number of merge classes 10 in FIG.
  • the target pixel is classified into one of three subclasses of a non-class, a weak class, and a strong class according to the gradient intensity ratio, and the activity as a spatial feature amount is performed. If the activity subclass is classified into one of the two subclasses according to the sum, and if the gradient intensity ratio subclass is other than the non-class, depending on the direction class, it is classified into one of the two subclasses of the H / V class and the D0 / D1 class. By being classified in the direction subclass, it is classified into one of the merge classes 0 to 14.
  • the classification rule of FIG. 15 is a rule in which the number of activity subclasses is reduced from three to two by merging a middle class and a large class as activity subclasses in the classification rule of FIG.
  • the target pixel in the activity subclass classification into any one of the two subclasses, has an index class_idx of 0 to 6, according to the index class_idx obtained from the activity sum as described in FIG. In this case, it is classified into a small class as an activity subclass, and if the index class_idx is 7 to 15, it is classified into a large class as an activity subclass.
  • the number of activity subclasses is 2 which is reduced by 3 from the number of activity subclasses in the initial class (FIG. 3). Then, the small class matches activity subclasses 0 to 2 in the initial class (FIG. 3), and the large class matches activity subclasses 3 and 4 in the initial class.
  • the assignment of the subclass to the activity sum is reduced by 3 subclasses as compared with the case of the GALF classification, and accordingly, the activity sum affects the classification of the merge class. Disappears.
  • a merge class in which the gradient intensity ratio and the direction class have a higher priority than the activity sum is obtained.
  • the target pixel is classified into the merge class 0 when the activity subclass is the small class, and is merged when the activity subclass is the large class.
  • the target pixel is classified into the merge class 2 when the activity subclass is the small class, and is merged when the activity subclass is the large class. Classified into Class 3.
  • the target pixel is classified into the merge class 4 when the activity subclass is the small class, and is merged when the activity subclass is the large class. Classified into Class 5.
  • the target pixel is classified into the merge class 6 when the activity subclass is the small class, and is merged when the activity subclass is the large class. Classified as Class 7.
  • the target pixel is classified into the merge class 8 when the activity subclass is the small class, and is merged when the activity subclass is the large class. Classified into class 9.
  • merge class 0 matches the initial classes 0 to 2 obtained by the GALF class classification
  • merge class 1 matches the initial classes 3 and 4 obtained by the GALF class classification
  • Class 2 corresponds to the initial class 5 to 7 obtained by the GALF classification
  • merge class 3 corresponds to the initial class 8 and 9 obtained by the GALF classification
  • merge class 4 is determined by the GALF classification.
  • the obtained initial classes 10 to 12 correspond
  • the merge class 5 corresponds to the initial classes 13 and 14 obtained by the GALF classification
  • the merge class 6 corresponds to the initial classes 15 to 17 obtained by the GALF classification.
  • the merge class 7 matches the initial classes 18 and 19 obtained by the GALF classification
  • the merge class 8 matches the initial class 20 obtained by the GALF classification.
  • merge class 9 corresponds to the initial class 23 and 24 obtained by the class classification GALF.
  • the initial classes 0 to 3 are set to the merge class 0
  • the initial classes 3 and 4 are set to the merge class 1
  • the initial classes 5 to 7 are set to the merge class 2
  • the initial class 8 is set to the merge class 8.
  • the initial classes 10 to 12 to the merge class 4 the initial classes 13 and 14 to the merge class 5, the initial classes 15 to 17 to the merge class 6, and the initial classes 18 and 19 to the merge class.
  • a merge pattern for converting the initial classes 20 to 22 into a merge class 8 and the initial classes 23 and 24 into a merge class 9 can be set.
  • FIG. 16 is a diagram for explaining a method of setting a merge pattern corresponding to a merge class number of 5 in which 25 initial classes obtained by GALF class classification are merged into 5 merge classes.
  • FIG. 16 shows a classification rule for performing class classification into merge classes obtained according to the merge pattern corresponding to the number of merge classes 10 in FIG.
  • the pixel of interest is classified into one of three subclasses of a non-class, a weak class, and a strong class according to the gradient intensity ratio, and the gradient intensity ratio subclass is non-class. If it is not a class, it is classified into one of the merge classes 0 to 4 by being classified into one of two subclasses of the H / V class and the D0 / D1 class according to the direction class.
  • the classification rule in FIG. 16 is a rule in which the small class and the large class as the activity subclass in the classification rule in FIG. 15 are merged to reduce the number of activity subclasses from 2 to 1.
  • the activity sum as the spatial feature does not affect the classification into the merge class. That is, in the classification rule of FIG. 16, only the gradient intensity ratio and the direction class of the gradient intensity ratio, the direction class, and the activity sum affect the classification into the merge class.
  • the target pixel is classified into the merge class 0.
  • the target pixel is classified into the merge class 1.
  • the target pixel is classified into the merge class 2.
  • the target pixel is classified into the merge class 3.
  • the target pixel is classified into the merge class 4.
  • merge class 0 matches the initial classes 0 to 4 obtained by the GALF class classification
  • merge class 1 matches the initial classes 5 to 9 obtained by the GALF class classification
  • Class 2 matches the initial class 10 to 14 obtained by the GALF classification
  • merge class 3 matches the initial class 15 to 19 obtained by the GALF classification
  • merge class 4 by the GALF classification. It corresponds to the initial class 20 to 24 obtained.
  • the initial classes 0 to 4 are set to the merge class 0
  • the initial classes 5 to 9 are set to the merge class 1
  • the initial classes 10 to 14 are set to the merge class 2
  • the initial class 15 is set.
  • To 19 can be set to the merge class 3 and the initial classes 20 to 24 can be set to the merge class 4, respectively.
  • FIG. 17 is a diagram for explaining a method of setting a merge pattern corresponding to three merge classes in which 25 initial classes obtained by GALF class classification are merged into three merge classes.
  • FIG. 17 shows a classification rule for performing class classification into merge classes obtained according to the merge pattern corresponding to the number of merge classes 3 in FIG.
  • the target pixel is classified into one of three subclasses of a non-class, a weak class, and a strong class according to the gradient intensity ratio. Or 2 classes.
  • the classification rule in FIG. 17 is a rule in which the number of direction classes is reduced from 2 to 1 by merging the D0 / D1 class and the H / V class as direction classes in the classification rule in FIG.
  • the activity sum as the direction class and the spatial feature does not affect the classification into the merge class. That is, according to the classification rule of FIG. 17, only the gradient intensity ratio among the gradient intensity ratio, the direction class, and the activity sum affects the classification into the merge class.
  • the target pixel is classified into the merge class 0.
  • the target pixel is classified into the merge class 1
  • the target pixel is classified into the merge class 2.
  • the merge class 0 matches the initial classes 0 to 4 obtained by the GALF classification
  • the merge class 1 corresponds to the initial classes 5 to 9 and 15 to 19 obtained by the GALF classification
  • the merge class 2 matches the initial classes 10 to 14 and 20 to 24 obtained by the GALF classification.
  • the initial classes 0 to 4 are set to the merge class 0
  • the initial classes 5 to 9 and 15 to 19 are set to the merge class 1
  • the initial classes 10 to 14 and 20 to 24 are set to the merge class.
  • a merge pattern to be converted can be set in merge class 2.
  • FIG. 18 is a view for explaining a method of setting a merge pattern corresponding to the number 2 of merge classes for merging 25 initial classes obtained by GALF class classification into two merge classes.
  • FIG. 18 shows a classification rule for performing class classification into merge classes obtained according to the merge pattern corresponding to the merge class number 2 in FIG.
  • the pixel of interest is classified into one of two subclasses of a non-class and a weak / strong class according to the gradient intensity ratio, so that the merge class 0 and the Classified into one of the following.
  • the classification rule in FIG. 18 is a rule in which the weak class and the strong class as the gradient intensity ratio subclass in the classification rule in FIG. 17 are merged to reduce the number of gradient intensity ratio subclasses from three to two.
  • the number of the gradient intensity ratio subclasses obtained by the gradient intensity ratio subclass classification is 2, which is reduced from the number 3 of the gradient intensity ratio subclasses in the initial class.
  • the non-class in the classification rule of FIG. 18 matches the non-class in the initial class, and the weak / strong class matches the weak and strong classes in the initial class.
  • the activity sum as the direction class and the spatial feature does not affect the classification into the merge class. That is, according to the classification rule of FIG. 18, only the gradient intensity ratio among the gradient intensity ratio, the direction class, and the activity sum affects the classification into the merge class.
  • the target pixel is classified into the merge class 0 and if the gradient intensity ratio subclass is the weak / strong class, the target pixel is the merge class. Classified into 1. In this case, the target pixel is classified into a texture pixel and a non-texture pixel.
  • merge class 0 matches initial classes 0 to 4 obtained by GALF class classification
  • merge class 1 matches initial classes 5 to 24 obtained by GALF class classification.
  • a merge pattern for converting the initial classes 0 to 4 to the merge class 0 and the initial classes 5 to 24 to the merge class 1 can be set.
  • FIG. 19 is a diagram for explaining a method of setting a merge pattern corresponding to the number of merge classes 1 in which 25 initial classes obtained by the GALF class classification are merged into one merge class.
  • FIG. 19 shows a classification rule for performing class classification into merge classes obtained according to the merge pattern corresponding to the number of merge classes 1 in FIG.
  • the target pixel is always classified into the merge class 0 as a monoclass.
  • the classification rule in FIG. 19 is a rule in which the non-class as a gradient intensity ratio subclass and the weak / strong class in the classification rule in FIG. 18 are merged to reduce the number of gradient intensity ratio subclasses from 2 to 1. I have.
  • the merge class 0 matches the initial classes 0 to 24 obtained by the GALF class classification.
  • a merge pattern that converts the initial classes 0 to 24 into the merge class 0 can be set as the merge pattern corresponding to the merge class number 1.
  • the gradient intensity ratio subclass in the GALF class classification should be merged as little as possible. Therefore, according to such a merge pattern, a merge class in which the gradient intensity ratio affects preferentially is obtained.
  • the feature amount other than the gradient intensity ratio for example, the merge class in which the activity sum influences preferentially, Can be set.
  • FIGS. 20 and 21 are diagrams showing a second example of a preset merge pattern.
  • FIG. 20 and 21 show examples of merge patterns corresponding to the number of merge classes 25, 15, 10, 5, 4, 3, 2, 1 for merging 25 initial classes obtained by the GALF class classification.
  • FIG. 20 and 21 show examples of merge patterns corresponding to the number of merge classes 25, 15, 10, 5, 4, 3, 2, 1 for merging 25 initial classes obtained by the GALF class classification.
  • the merge patterns in FIGS. 20 and 21 are set by the reduction setting, similarly to the merge patterns in FIGS. 10 and 11.
  • the merge patterns in FIGS. 10 and 11 are merge patterns in which a merge class in which the gradient intensity ratio affects preferentially is obtained, whereas in the merge patterns in FIGS.
  • the merge pattern is such that the affected merge class is obtained.
  • the merge pattern corresponding to the number of merge classes 25 indicates that the target pixel is one of three subclasses of a non (None) class, a weak (Weak) class, and a strong (Strong) class according to the gradient intensity ratio.
  • Crab gradient intensity ratio subclass classification activity subclass classification into one of 5 subclasses (activity subclass 0 to 4) according to activity sum, gradient intensity ratio obtained by gradient intensity ratio subclass classification according to gradient intensity ratio
  • the direction class is classified into one of the two subclasses of the H / V class and the D0 / D1 class according to the direction class.
  • the merge pattern corresponding to the merge class number 15 classifies the pixel of interest into a non-class, a weak class, and a strong class according to the gradient intensity ratio.
  • the class obtained by the classification rule of classifying into all 15 classes is a merge pattern obtained as a merge class.
  • the assignment of the subclass to the gradient intensity ratio is reduced by one subclass compared to the case of the GALF class classification. Accordingly, the gradient intensity ratio does not affect the classification into the merge class.
  • the merge pattern corresponding to the merge class number 10 a merge class in which the activity sum influences more preferentially than the gradient intensity ratio is obtained.
  • the merge pattern corresponding to the merge class number 4 is a class obtained by a classification rule that classifies the target pixel into one of four subclasses according to the activity sum and classifies the target pixel into all four classes. Is a merge pattern obtained as a merge class.
  • the merge pattern corresponding to the number of merge classes 3 is a class obtained by a classification rule that classifies the target pixel into one of three subclasses according to the activity sum and classifies the target pixel into all three classes. Is a merge pattern obtained as a merge class.
  • the merge pattern corresponding to the number of merge classes 2 is a class obtained by a classification rule that classifies the target pixel into one of two subclasses according to the activity sum and classifies the target pixel into all two classes. Is a merge pattern obtained as a merge class.
  • the merge pattern corresponding to the number of merge classes 1 is a merge pattern from which a merge class 0 as a monoclass is always obtained.
  • the GALF class classification is adopted as the class classification for obtaining the initial class (hereinafter, also referred to as the initial class classification), but a class classification other than the GALF class classification is adopted as the initial class classification. be able to.
  • FIG. 22 is a diagram for explaining a class classification using ranking (Ranking) as a feature amount of a target pixel, that is, a class classification of JVET-J0014.
  • (i, j) ⁇ (s ′ (i, j) ⁇ s ′ (k, l)? 1: 0)
  • (i, j) is the position of the target pixel (for example, I-th from the left and j-th position from the top).
  • s ′ (i, j) represents the pixel value (for example, luminance) of the pixel at the position (i, j).
  • the first summation ( ⁇ ) on the right side represents the summation where k is changed to an integer from i-1 to i + 1, and the second summation is that l is changed from j-1 to j + Represents the summation as an integer up to 1.
  • (X? 1: 0) means that it takes 1 when X is true and takes 0 when X is false.
  • r 8 (i, j) ⁇ (s ′ (i, j) ⁇ s ′ (k, l)? 1: 0), a pixel having a pixel value larger than the pixel of interest ,
  • the ranking r 8 (i, j) of the pixel of interest increases.
  • r 8 (i, j) takes an integer value in the range of 0 to 8.
  • the category of the pixel of interest can also be obtained as follows.
  • if the ⁇ T 3 is satisfied, the category of the pixel of interest as (category) 0, wherein T 3 ⁇
  • ⁇ T 4 is satisfied If the category of the attention pixel is 1, the formula
  • T 1 , T 2 , T 3 , and T 4 are preset thresholds.
  • T 1 and T 2 is related to the formula T 1 ⁇ T 2
  • T 3 and T 4 are related equation T 3 ⁇ T 4.
  • the class D 1 R (i, j) of the target pixel is obtained using the ranking r 8 (i, j) and the category of the target pixel.
  • the category of the pixel of interest 2
  • the target pixel is classified into any one of the 27 classes of classes 0 to 26.
  • FIG. 22 shows an example of an initial class table in which a class obtained by class classification using ranking is an initial class.
  • the initial class table in FIG. 22 is a table in which the horizontal axis is r 8 (i, j) and the vertical axis is a category.
  • FIG. 23 is a diagram for explaining class classification using a pixel value as a feature amount of a target pixel, that is, JVET-J0018.
  • the dynamic range of pixel values is divided into, for example, bands of the same size.
  • the target pixel is classified into classes according to which band the pixel value of the target pixel belongs to.
  • FIG. 24 is a diagram illustrating class classification using the reliability of the tilt direction as the feature amount of the pixel of interest.
  • the direction as the specified direction of the target pixel is obtained (set) as in GALF.
  • the Laplacian filter is applied to the decoded image, and as a peripheral region, for example, the horizontal direction and the vertical direction of the pixel of interest are 3 ⁇ 3 pixels in the V direction and the H direction. , D0 direction, and D1 direction, activities A (V), A (H), A (D0), and A (D1) are obtained.
  • the activity sum sumA (H) of each of the four directions is obtained by adding the activity A (D) of 3 ⁇ 3 pixels as a peripheral region for each of the four directions for the target pixel. , SumA (D0) and sumA (D1) are obtained.
  • a frequency distribution in a tilt direction (prescribed direction) is generated for the target pixel.
  • the horizontal ⁇ vertical 3 ⁇ 3 pixels around the target pixel are each The activities A (V), A (H), A (D0), and A (D1) in the four directions of the V direction, the H direction, the D0 direction, and the D1 direction are obtained.
  • the frequency distribution generation area is an area of pixels used for generating a frequency distribution in a specified direction.
  • the frequency distribution generation area is assumed to be an area that matches the peripheral area.
  • activities A (V), A (H), A (D0), and A (D1 ) are performed by the frequency distribution generation area matches the surrounding area.
  • the eight directions of the GALF described in FIG. 1 are defined as the defined directions indicating the classes of the frequency distribution, and the target pixel is a 4 ⁇ 3 ⁇ 3 pixel in the frequency distribution generation area.
  • Direction The activity A (V), A (H), A (D0), and A (D1) are determined for the specified direction represented by the GALF direction (set). By counting the frequency in the direction, a frequency distribution in a specified direction is generated.
  • the frequency in the specified direction determined (set) for each of the 3 ⁇ 3 pixels in the frequency distribution generation area is counted, so that the target pixel is obtained. Is generated in the specified direction.
  • the value corresponding to the frequency of the direction (class) as the specified direction of the target pixel is the reliability of the specified direction of the target pixel. (Set).
  • the specified direction of the target pixel is a specified direction 000 (a specified direction to which 0 (decimal number 000) is assigned) among the specified directions as eight directions of GALF.
  • a value corresponding to the frequency in the specified direction 000 is obtained (set) as the reliability of the pixel of interest in the specified direction.
  • the target pixel has the same final class 0 to 24 as the GALF class classification. It is classified into one of 25 classes.
  • the reliability of the specified direction as the tilt direction of the target pixel is determined using the frequency distribution of the tilt direction of the pixel in the frequency distribution generation region, but the reliability of the tilt direction of the target pixel is Other, for example, a value corresponding to the sum of the absolute value or the square of the inner product of the vector indicating the tilt direction of the target pixel and each of the vectors indicating the tilt directions of a plurality of pixels around the target pixel, and other target pixels Can be adopted as a value indicating the likelihood of the inclination direction of.
  • FIG. 25 is a diagram illustrating the final class obtained by the class classification using the reliability.
  • the direction subclass classification is performed in the same manner as the GALF class classification. However, in the class classification using the reliability, the direction subclass classification is performed according to the reliability of the specified direction in addition to the direction as the specified direction of the target pixel.
  • a non-class (None) indicating low reliability is prepared in addition to the direction classes 0 (D0 / D1 class) and 2 (H / V class).
  • the target pixel in the specified direction is less than the threshold value p, in the class classification using the reliability, the target pixel is classified into a non-class direction class in the direction subclass. Then, in the class classification using the reliability, the activity sums sumA (V), sumA (H), sumA (D0), in the V direction, the H direction, the D0 direction, and the D1 direction as the spatial feature amount of the pixel of interest. In addition, according to sumA (D1), the target pixel is classified into any of the final classes 0 to 4, similarly to the class classification of GALF.
  • the target image is determined according to the direction as the specified direction of the target pixel,
  • the direction subclass is classified into the direction class 0 or 2.
  • the gradient intensity ratio of the equation (2) or the equation (3) is similar to the GALF class classification. Desired. Then, according to the gradient intensity ratio, a gradient intensity ratio subclass classification for obtaining a class representing the gradient intensity ratio of the pixel of interest is performed.
  • the direction class 0 or 2 the non-class, weak class, or strong class obtained as a result of the gradient intensity ratio subclass classification, and the spatial feature amount of the pixel of interest According to the activity sums sumA (V), sumA (H), sumA (D0), and sumA (D1) in the V direction, H direction, D0 direction, and D1 direction, To 24.
  • the threshold value p of the reliability in the specified direction can be set according to the number of pixels in the frequency distribution generation area. For example, in the case where the frequency itself of the frequency distribution in the specified direction is adopted as the reliability in the specified direction, when the frequency distribution generation area is an area of 6 ⁇ 6 pixels, the threshold value p is, for example, a pixel of the frequency distribution generation area. It can be set to 1/4 or 1/8 of the number (for example, 36 pixels).
  • the class classification in FIG. 25 can be said to be a class classification in which the reliability in the inclination direction (prescribed direction) is introduced into the GALF class classification in FIG.
  • the pixels are classified according to the reliability of the tilt direction, so that when the reliability of the tilt direction is low, that is, when the reliability of the direction indicating the tilt direction is low, such a case is obtained.
  • the pixel is classified into the direction class 0 or 2 by the direction subclass classification according to the direction, and it is possible to prevent the pixel from being classified into the direction class not corresponding to the inclination direction.
  • the pixels can be classified into an appropriate class (final class), and filter processing can be performed as appropriate prediction processing using the tap coefficients of the class. Therefore, the performance of the filtering process can be improved.
  • FIGS. 26 and 27 are diagrams showing a third example of a preset merge pattern.
  • FIG. 26 and FIG. 27 correspond to the number of merge classes 27, # 24, # 21, # 18, # 12, # 9, # 6 obtained by merging 27 initial classes obtained by the class classification using the ranking described in FIG.
  • FIG. 9 is a diagram illustrating an example of a merge pattern.
  • the merge patterns in FIGS. 26 and 27 are set by the reduction setting, similarly to the merge patterns in FIGS. 10 and 11 and FIGS. 20 and 21.
  • the pixel of interest is subclass classified into any one of 9 subclasses indicating that the ranking r 8 (i, j) is 0 to 9 according to the ranking, and the category According to the above, it can be said that a classification rule for classifying into all 27 classes is adopted by subclass classification into any of three subclasses indicating that the category is 0 or 2.
  • the merge pattern corresponding to the number 27 of the merge classes is obtained by classifying the target pixel into nine subclasses according to the ranking and subclassing into three subclasses according to the category.
  • the class obtained by the classification rule for classifying the class is a merge pattern obtained as a merge class. That is, the merge pattern corresponding to the number of merge classes 27 is a merge pattern in which the same merge class as the initial class obtained by the class classification using the ranking is obtained.
  • the merge pattern corresponding to the number of merge classes 24, 21, 18, 12, 9, 6 converts the pixel of interest into 8, 7, 6, 4, 3, 2 subclass according to the ranking.
  • the class obtained by the classification rule of classifying into all 24, 21, 18, 12, 9, 6 classes is Each of the obtained merge patterns.
  • the target pixel is classified into 5 subclasses according to the ranking, and according to the category, the subclass is classified into one of 3 subclasses.
  • a merge pattern obtained by a class obtained by a classification rule classifying as a merge class can be adopted.
  • the target pixel is classified into one of the three subclasses according to the category, and the class obtained by the classification rule of classifying all three classes is merged.
  • a merge pattern obtained as a class can be adopted.
  • a merge pattern corresponding to the number of merge classes 1 is a merge pattern that always provides a merge class 0 as a monoclass.
  • FIG. 28 is a diagram showing a fourth example of a preset merge pattern.
  • FIG. 28 is a diagram illustrating an example of a merge pattern corresponding to the number of merge classes 32, # 16, # 8, and # 4 in which 32 initial classes obtained by the class classification using the pixel values described with reference to FIG. 23 are merged. is there.
  • the merge pattern in FIG. 28 is set by the reduction setting, similarly to the merge patterns in FIGS. 10 and 11 and FIGS. 20 and 21.
  • the 256 levels as the dynamic range of the pixel values are divided into 32 bands, and the target pixel is assigned to the band to which the pixel value belongs according to the pixel value of the target pixel. It can be said that a classification rule of classifying into all 32 classes is adopted by classifying into classes assigned to.
  • the merge pattern corresponding to the number of merge classes 32 divides 256 levels as a dynamic range of a pixel value into 32 bands, and divides a target pixel into a band to which the pixel value belongs according to the pixel value of the target pixel.
  • the class obtained by the classification rule for classifying into all 32 classes is a merge pattern obtained as a merge class. That is, the merge pattern corresponding to the number of merge classes 32 is a merge pattern from which the same merge class as the initial class obtained by the class classification using the pixel values is obtained.
  • the merge pattern corresponding to the number of merge classes 16, 8, 4 divides the 256 levels as the dynamic range of the pixel value into 16, 8, 4 bands, and sets the target pixel according to the pixel value of the target pixel. Then, by classifying the pixels into the classes assigned to the band to which the pixel value belongs, the classes obtained by the classification rules for classifying into all 16, 8, 4 classes are the merge patterns obtained as the merge classes.
  • the size of the bands is 8, # 16, # 32, and # 64 levels, respectively.
  • the number of merge classes of the merge pattern for merging the 32 initial classes obtained by the class classification using the pixel values for example, 2 or 1 can be adopted in addition to 32, 16, 8, 4 .
  • the 256 levels as the dynamic range of the pixel value are divided into two bands, and the target pixel is assigned to the band to which the pixel value belongs according to the pixel value of the target pixel.
  • a merge pattern corresponding to the number of merge classes 1 is a merge pattern that always provides a merge class 0 as a monoclass.
  • the merge pattern corresponding to each number of merge classes is set in a mixed setting, that is, a merge pattern for merging an initial class obtained by a predetermined class classification and an initial class obtained by another class classification are merged. It can be set so that the merge pattern and the merge pattern coexist.
  • the merge pattern corresponding to each merge class number is such that a merge pattern for merging the initial class obtained by the GALF class classification and a merge pattern for merging the initial class obtained by the class classification using the ranking are mixed.
  • the merge pattern for merging the initial classes obtained by the GALF class classification for example, the merge pattern corresponding to the number of merge classes 25, $ 20, $ 15, $ 10, $ 5, $ 3, $ 2, $ 1 shown in Figs. Can be adopted.
  • the merge pattern for merging the initial classes obtained by the class classification using the ranking the number of merge classes 27, 24, 21, 18, 15, 12, 9, 6, 3, 1 described in FIGS. A corresponding merge pattern can be employed.
  • merge patterns corresponding to the number of merge classes 25, # 20, # 15, # 10, # 5, # 3, # 2, and # 1 as merge patterns for merging the initial classes obtained by the GALF class classification (hereinafter, also referred to as GALF merge patterns)
  • the GALF merge pattern and the ranking merge pattern have the same number of merge classes of 15, # 3, and # 1.
  • the GALF merge pattern is adopted as the merge pattern corresponding to the number of merge classes 25, 20, 15, 10, 5, 3, 2, 1, and the number of merge classes 27, Ranking merge patterns are adopted as the merge patterns corresponding to 24, # 21, # 18, # 12, # 9, and # 6.
  • the merge pattern corresponding to each number of merge classes is a merge pattern that merges initial classes obtained by any two or more types of class classification in addition to the class classification using GALF class classification and ranking. Can be set to be mixed.
  • the merge pattern corresponding to each number of merge classes includes a merge pattern that merges an initial class obtained by GALF class classification and a merge pattern that merges an initial class obtained by class classification using pixel values. It can be set as follows.
  • the merge pattern for merging the initial classes obtained by the GALF class classification for example, the merge pattern corresponding to the number of merge classes 25, $ 20, $ 15, $ 10, $ 5, $ 3, $ 2, $ 1 shown in Figs. Can be adopted.
  • a merge pattern corresponding to the number of merge classes 32, # 16, # 8, # 4, # 2, # 1 described in FIG. 28 can be adopted.
  • a merge pattern corresponding to the number of merge classes 32, 16, 8, 4, 2, 1 (hereinafter also referred to as a pixel value merge pattern) as a merge pattern for merging the initial classes obtained by the classification is mixed
  • the GALF merge pattern and the pixel value merge pattern have the same number of merge classes, 2, 1.
  • the merge pattern mixing setting such that the merge pattern for merging the initial classes obtained by the predetermined class classification and the merge pattern for merging the initial classes obtained by the other class classifications are mixed. Is performed by interpolating the number of merge classes other than the number of merge classes of the merge pattern obtained by the predetermined class classification with the number of merge classes of the merge pattern obtained by merging the initial classes obtained by the other class classifications. It can be said that the merge pattern is set as follows.
  • the number of merge classes 32, 16, 8, 4 that does not exist as the number of merge classes of the GALF merge pattern is the number of merge classes 32, 16 of the pixel value merge pattern. , 8, 4.
  • the initial class classification (class classification method) differs depending on the (adopted) number of merge classes.
  • FIG. 29 is a diagram for explaining the class classification of GALF.
  • GALF class classification is performed using the gradient intensity ratio, direction (prescribed direction), and activity sum (spatial feature) as a plurality of features of the target pixel. Can be.
  • the GALF classification includes gradient intensity ratio subclass classification (using gradient intensity ratio), direction subclass classification (using direction), and activity sum (using activity sum). It can be said that it is performed by activity subclass classification.
  • the subclass obtained by the direction subclass classification is also referred to as a direction subclass (equivalent to the direction class described in FIG. 2).
  • the pixel of interest is one of three subclasses (gradient intensity ratio subclass) of a non-class, a weak class, and a strong class by threshold processing of the gradient intensity ratio. are categorized.
  • the target pixel is classified into one of two subclasses (direction subclasses) of the D0 / D1 class and the H / V class according to the direction, as shown in FIG.
  • the activity subclass classification the target pixel is classified into any one of the five subclasses of activity subclasses 0 to 4, based on an index class_idx in the range of 0 to 15 obtained from the activity sum.
  • the GALF class classification is performed by subclass classification (gradient intensity ratio subclass classification, direction subclass classification, and activity subclass classification) of the gradient intensity ratio, direction, and activity sum as a plurality of feature amounts as described above. It can be said.
  • subclass classification gradient intensity ratio subclass classification, direction subclass classification, and activity subclass classification
  • the classification using the reliability described in FIGS. 24 and 25 is performed by the subclass classification of the gradient intensity ratio, the direction, the activity sum, and the reliability. Therefore, it can be said that the class classification using the reliability is also performed by the subclass classification of each of the plurality of feature amounts, similarly to the class classification of GALF.
  • class classification performed by subclass classification of each of a plurality of feature amounts is adopted as a class classification for obtaining an initial class (initial class classification)
  • a merge pattern is set by reduction setting
  • subclasses of the feature amounts are merged.
  • a merge pattern can be set by merging the gradient intensity ratio subclass of the gradient intensity ratio, the direction subclass of the direction, and the activity subclass of the activity sum. it can.
  • merging of subclasses is also referred to as subclass merging.
  • FIG. 30 is a diagram illustrating subclass merging of the gradient intensity ratio subclass.
  • the gradient strength ratio subclass is obtained by subclass merging the weak class and the strong class among the original nonclass, weak class, and strong class into the high class, and as a whole, the nonclass and the high class Can be two subclasses. Further, the gradient strength ratio subclass can be obtained by subclass merging the nonclass and the high class into the N / A (Not Available) class so that the subclass merges into one subclass of only the N / A class. Merging the gradient intensity ratio subclass into one subclass of only the N / A class is equivalent to not performing the gradient intensity ratio subclass classification.
  • the N / A class as the gradient strength ratio subclass can be said to be a subclass obtained by merging two subclasses of a nonclass and a high class, as well as the original nonclass, weak class, and strong class. It can be said that it is a subclass obtained by merging three subclasses of a class.
  • FIG. 31 is a diagram for explaining the subclass merging of the direction subclass.
  • the direction subclass can be obtained by merging the original two subclasses of the D0 / D1 class and the H / V class into the N / A class to make one subclass of only the N / A class as a whole. Merging the direction subclass into one subclass of only the N / A class is equivalent to not performing the direction subclass classification.
  • FIG. 32 is a diagram for explaining the subclass merging of the activity subclass.
  • the activity subclasses are the same as the original activity subclasses 0, 1 corresponding to the index class_idx (value 0), the activity subclasses 1 corresponding to the index class_idx 1, and the activity subclasses 2, 7 to 14 corresponding to the index class_idx of 2 to 6.
  • Activity subclass 3 corresponding to index class_idx
  • activity subclass 0 corresponding to index class_idx of 0 and 1 among 5 subclasses of activity subclass 4 corresponding to 15 index class_idx, for example, activity subclass 0 To the activity subclass 0 corresponding to the index class_idx of 0 and 1, the activity subclass 1 corresponding to the index class_idx of 2 to 6, and the activity corresponding to the index class_idx of 7 to 14.
  • Subclass 2 and can be merged into 4 subclasses of activities subclass 3 corresponding to 15 index Class_idx.
  • the activity subclasses are: activity subclasses 0 and 1 corresponding to the index class_idx, activity subclasses 1 to 2 corresponding to the index class_idx, activity subclasses 2 and 7 corresponding to the index class_idx of the index 7 to 14, and an index of 15
  • activity subclasses 3 corresponding to class_idx for example, activity subclasses 0 and 1 are subclass-merged into an activity subclass 0 corresponding to an index class_idx of 0 to 6, and as a whole, correspond to an index class_idx of 0 to 6.
  • Activity subclass 0, activity subclass 1 corresponding to the index class_idx of 7 to 14, and activity subclass 2 corresponding to the index class_idx of 15 can be merged.
  • the activity subclass is an activity subclass 0 corresponding to the index class_idx of 0 to 6, an activity subclass 1 corresponding to the index class_idx of 7 to 14, and an activity subclass 2 corresponding to the index class_idx of 15, for example, Activity subclass 0 corresponding to index class_idx of 0 to 6, and activity subclass 1 corresponding to index class_idx of 7 to 14, subclass merge to activity subclass 0 corresponding to index class_idx of 0 to 14, as a whole, It can be merged into two subclasses, activity subclass 0 corresponding to index class_idx of 0 to 14 and activity subclass 1 corresponding to index class_idx of 15.
  • the activity subclass includes an activity subclass 0 corresponding to an index class_idx of 0 to 14 and an activity subclass 1 corresponding to an index class_idx of 15 by an N / A class (activity subclass 0) corresponding to an index class_idx of 0 to 15. ) Can be merged into a single subclass of only the N / A class corresponding to the index class_idx of 0 to 15. Merging the activity subclass into one subclass of only the N / A class is equivalent to not performing the activity subclass classification.
  • the activity subclass 0 corresponding to the index class_idx of 0 to 6 is, as described above, the activity subclass 0 corresponding to the index class_idx of 0 and 1, and the activity subclass 0 to 2
  • the activity subclass merged into two subclasses and the activity subclass merged into one subclass are, as described above, the activity subclass 0 corresponding to the index class_idx of 0 and 1, and the activity subclass 0 to 2
  • the activity subclasses are merged (subclass merge) from the activity subclass 0 representing a small activity to the activity subclass 4 representing a large activity, in which the number of assignments of the index class_idx is small.
  • the order of subclass merging is not limited to this. For example, a subclass merge of an activity subclass merges activity subclasses 0 and 1, further merges activity subclass 2, then merges activity subclasses 3 and 4, and finally merges into the N / A class. It can be performed in order or the like.
  • the initial class can be merged and the merge pattern can be set (generated) by the above-described subclass merge.
  • FIG. 33 is a diagram showing an example of merging an initial class by merging a subclass of an activity subclass.
  • the subclass merge of the activity subclass for example, as shown by a dotted line in FIG. 33, in the initial class table, a plurality of initial classes in the horizontal direction of each row are merged.
  • the subclass merge of the activity subclass 0 corresponding to the index class_idx of 0 and the activity subclass 1 corresponding to the index class_idx of 1 merges the initial class in the first column and the initial class in the second column of each row. Have been.
  • FIG. 34 is a diagram showing an example of merging of initial classes by subclass merging of the gradient intensity ratio subclass.
  • the initial classes in the second and third rows of each column are merged, and The initial class on line 5 is merged.
  • the initial classes in the second and third rows of each column are merged, and the initial classes in the fourth and fifth rows are merged by subclass merging of the weak class and the strong class.
  • FIG. 35 is a diagram showing an example of initial class merging by subclass merging of a direction subclass.
  • the initial classes in the second and fourth rows of each column are merged, and the third and fifth rows are merged.
  • the initial classes of the eyes are merged.
  • the subclass merge between the D0 / D1 class and the H / V class merges the initial classes in the second and fourth rows of each column and merges the initial classes in the third and fifth rows. Have been.
  • FIG. 36 is a diagram illustrating a relationship between the number of subclasses after the subclass merge of the gradient intensity ratio subclass, the direction subclass, and the activity subclass, and the number of merge classes.
  • FIG. 36 shows that, as described with reference to FIGS. 30 to 32, the gradient intensity ratio subclass is any of 1 to 3 subclasses, the direction subclass is any of 1 and 2 subclasses, and the activity subclass is 1 to 5 subclasses. Shows the relationship between the number of subclasses after the subclass merge and the number of merged classes when the subclasses are merged.
  • the fact that the number of subclasses of the gradient intensity ratio subclass after the subclass merge is 3 is equivalent to the fact that the subclass merge of the gradient intensity ratio subclass is not performed.
  • not performing subclass merging is regarded as subclass merging in which each subclass is merged with the subclass. The same applies to the merging of the initial class.
  • the direction subclass is invalid, and the class is determined without considering the direction subclass (regardless of the direction subclass). Classification is performed.
  • the number of merge classes is represented by the formula Nc ⁇ (Nb ⁇ (Na ⁇ 1) ) +1).
  • the gradient intensity ratio subclass is any of 1 to 3 subclasses
  • the direction subclass is any of 1 and 2 subclasses
  • the activity subclass is any of 1 to 5 subclasses.
  • the number of merge patterns that can be obtained in the case of subclass merging is 30, in terms of calculation, such that the number of subclasses after subclass merging becomes as shown in FIG.
  • the classification into the merged class is performed regardless of the gradient intensity ratio (subclass). Then, if it is not known whether the gradient intensity ratio is large or small, and if the direction contributes to the classification into the merge class, when the gradient intensity ratio is small, the direction as the gradient direction of the pixel value of the pixel of the flat image is small. In consideration of the above, the classification is performed. For a flat image, the pixel values are not (almost) inclined. When such a flat image is classified into a merge class in consideration of the inclination direction of the pixel value, the pixel of interest becomes appropriate. May not be classified into the same class, that is, for example, pixels having similar characteristics may be classified into another class instead of the same class (merged class) due to slight noise.
  • the directional subclass classification classified into the D0 / D1 class or the H / V class and furthermore, a merge pattern that is a class classification performed by such a directional subclass classification That is, the merge pattern corresponding to the number of subclasses in which the number of subclasses in the gradient intensity ratio subclass is 1 and the number of subclasses in the direction subclass is 2 (or more) is invalidated and not used (N / A).
  • the column where the number of merge classes is N / A indicates that the merge pattern corresponding to the number of merge classes is invalid. There are five invalid merge patterns.
  • the merge pattern obtained by subclass merging of the gradient intensity ratio subclass, the direction subclass, and the activity subclass described with reference to FIGS. 30 to 32 that is, the gradient intensity ratio subclass is set to one of 1 to 3 subclasses, and the direction subclass is set to
  • the effective merge pattern that can be obtained is 30 invalid patterns from 5 computational invalid patterns. There are 25 patterns excluding the pattern.
  • the merge patterns of 25 patterns obtained by subclass merge include merge patterns with the number of subclasses of 1, 2, 3, 4, 5, 6, 8, 9, 10, 12, 15, 20, and 25. Merge patterns with the same number exist.
  • the merge patterns (3, 1, 5) and (2, 2, 5) have the same number of merge classes (15).
  • a merge pattern is set for each number of merge classes, for a plurality of merge patterns having the same number of merge classes, costs are obtained using various images, and the merge pattern having the minimum cost is determined as the merge class. Merge pattern selection to select the merge pattern corresponding to the number is performed.
  • FIG. 37 is a diagram showing an example of a merge pattern obtained by performing a subclass merge and a merge pattern selection.
  • 13 patterns of merge patterns that is, the number of subclasses 1, 2, 3, 4, 5, 6, 8, 9, 10, 12, 15, 20 , $ 25 can be set for each of the merge patterns.
  • a merge pattern When a merge pattern is set in advance, it is desirable to set a certain number of merge patterns from the viewpoint of improving filter processing performance, that is, improving the image quality and coding efficiency of a filter image.
  • the number of classes in the initial class classification is 25, so if a merge pattern is set in the reduction setting for each subclass number, at most, 25 merge patterns with 1 to 25 merge classes can be set.
  • the merge patterns of the number of merge patterns missing in the subclass merge and the merge pattern selection can be interpolated by partially merging the subclasses.
  • the number of merge classes of the merge pattern set by subclass merge and merge pattern selection is interpolated between 25 and 20, between 20 and 15, between 15 and 12, etc.
  • a merge pattern corresponding to the number of merge classes can be set.
  • FIG. 38 is a diagram for explaining the partial merging of subclasses.
  • ⁇ Partial merge ⁇ means that when the subclass of one feature used in the initial class classification is a specific subclass, the subclass of another feature is merged.
  • FIG. 38 illustrates a merge pattern obtained by partial merging of the gradient intensity ratio subclasses when the activity subclass is the activity subclass 0 corresponding to the index class_idx of 0.
  • the initial classes in the second and third rows of each column are merged, and the initial classes in the fourth and fifth rows are merged.
  • the classes are merged.
  • FIG. 39 is a view for explaining partial merging of subclasses.
  • the partial merge that merges the gradient intensity ratio subclass when the activity subclass is the activity subclass 0 corresponding to the index class_idx of 0, as described with reference to FIG.
  • the initial classes 5 and 10 on the third and third lines are merged, and the initial classes 15 and 20 on the fourth and fifth lines are merged.
  • FIG. 40 is a diagram showing an example of a merge pattern obtained by partially merging subclasses.
  • a merge pattern corresponding to 23 merge classes can be obtained by the partial merge described in FIG.
  • ⁇ Also for example, a merge pattern corresponding to 21 merge classes can be obtained by the partial merge described in FIG.
  • the subclass merge the activity subclass, the activity subclass 0 corresponding to the index class_idx of 0 and 1, the activity subclass 1 corresponding to the index class_idx of 2 to 6, the activity subclass 2 corresponding to the index class_idx of 7 to 14, and After merging into 4 subclasses of activity subclass 3 corresponding to index class_idx of 15, partial merge is performed to merge gradient intensity ratio subclasses when the activity subclass is activity subclass 0 corresponding to index class_idx of 0 and 1.
  • a merge pattern corresponding to the number of merge classes 18 can be obtained.
  • the activity subclasses 0, 1 corresponding to the index class_idx of 0, and the index of the activity subclasses 1, 2 to 6 corresponding to the index class_idx of 1 From activity subclass 2 corresponding to class_idx, activity subclass 3 corresponding to index class_idx of 7 to 14, and activity subclass 4 corresponding to index index_idx of 15, from activity subclass 0 indicating that the activity is small,
  • a merge pattern is obtained by sequentially merging the activity subclasses toward the activity subclass 4 indicating that they are larger.
  • the activity subclass is from activity subclass 0 indicating that the activity is small to activity subclass 3 indicating that the activity is large.
  • the gradient intensity ratio subclasses are sequentially merged to obtain merge patterns corresponding to the number of merge classes 23, # 21, # 19, and # 17.
  • the merge pattern corresponding to each of the number of merge classes 23, # 21, # 19, and # 17 is an activity subclass 1 indicating that the activity is small since the activity subclass is an activity subclass 4 indicating that the activity is large.
  • the case can be obtained by performing a partial merge in which the gradient intensity ratio subclasses are sequentially merged.
  • a partial merge when a subclass other than the activity subclass is a specific subclass, a partial merge is performed to merge a subclass of another feature amount, and the merge pattern set by the subclass merge and the merge pattern selection And a merge pattern corresponding to another number of merge classes that interpolates between the numbers of merge classes.
  • FIG. 41 is a diagram illustrating an example of a relationship between a merge pattern obtained by subclass merging (and a merge pattern selection) and a merge pattern obtained by partial merging.
  • the subclass merge to make the number of subclasses of the gradient strength ratio subclass, the direction subclass, and the activity subclass 3, 3, 2, 5, respectively, that is, the number of subclasses of the gradient strength ratio subclass, the direction subclass, and the activity subclass, the GALF According to the subclass merging in which the original class classification is maintained, a merge pattern corresponding to 25 merge classes can be obtained.
  • the merge pattern corresponding to the merge class number 15 Can be obtained.
  • the number of subclasses of the activity subclass is changed from the original 5 to 4
  • the number of subclasses of the gradient intensity ratio subclass is changed from the original 3 to 2. According to the subclass merge, a merge pattern corresponding to the number of merge classes 12 can be obtained.
  • a merge pattern corresponding to 21 merge classes can be obtained.
  • a merge pattern corresponding to the merge class number 15 can be obtained.
  • the merge pattern corresponding to the number of merge classes 15 coincides with the merge pattern corresponding to the number of merge classes 15 obtained by subclass merge in which the number of subclasses of the gradient intensity ratio subclass is changed from 3 to 2.
  • FIG. 42 is a diagram showing another example of the relationship between the merge pattern obtained by subclass merging and the merge pattern obtained by partial merging.
  • a subclass merge that obtains a merge pattern corresponding to the merge class number 20, i.e., an activity subclass, an activity subclass 0 corresponding to an index class_idx of 0 and 1, and an activity subclass 1, 7 to 7 corresponding to an index class_idx of 2 to 6
  • Activity subclass 2 corresponding to the index class_idx of 14 and activity subclass 2 corresponding to the index class_idx of 0 and 1 after performing a subclass merge for merging into 4 subclasses of the activity subclass 3 corresponding to the index class_idx of 15
  • a merge pattern corresponding to the number of merge classes 18 can be obtained.
  • a merge pattern corresponding to the number of merge classes 12 can be obtained by partially merging the gradient intensity ratio subclasses when the activity subclass is the activity subclass 3 corresponding to the index class_idx of 15. .
  • the merge pattern corresponding to the number of merged classes 12 is the number of merged classes obtained by subclass merging that reduces the number of subclasses of the activity subclass from the original 5 to 4 and the number of subclasses of the gradient intensity ratio subclass from the original 3 to 2 Matches the merge pattern corresponding to 12.
  • merge patterns for each number of merge classes set by sub-class merge that is, 1, 2, 3, 4, 5, 6, 8, 9, 10, 12, 15, 20, 25 respectively Will be described again.
  • FIG. 43 is a diagram showing a merge pattern corresponding to the number of merge classes 25 obtained by subclass merge and a subclass merge from which the merge pattern is obtained.
  • the merge pattern corresponding to the number of merge classes 25 is such that the gradient intensity ratio subclass is subclass merged into 3 subclasses of non-class, weak class, and strong class, and the direction subclass is 2 of D0 / D1 class and H / V class.
  • a merge pattern of 25 merge classes can be obtained by leaving the gradient intensity ratio subclass of 3 subclasses, the direction subclass of 2 subclasses, and the activity subclass of 5 subclasses as they are.
  • FIG. 44 is a diagram showing a merge pattern corresponding to the number of merge classes 20 obtained by the subclass merge and a subclass merge from which the merge pattern is obtained.
  • the merge pattern corresponding to the number of merge classes 20 is such that the gradient intensity ratio subclass is subclass merged into three subclasses of non-class, weak class, and strong class, and the direction subclass is D0 / D1 class and 2 of H / V class.
  • FIG. 46 is a diagram showing a merge pattern corresponding to the number of merge classes 12 obtained by the subclass merge, and a subclass merge from which the merge pattern is obtained.
  • the merge pattern corresponding to the number of merge classes is 12, the gradient intensity ratio subclass is subclass merged into two subclasses of non-class and high class, and the direction subclass is subclassed into two subclasses of D0 / D1 class and H / V class.
  • FIG. 47 is a diagram showing a merge pattern corresponding to the number of merge classes 10 obtained by subclass merge and a subclass merge from which the merge pattern is obtained.
  • the merge pattern corresponding to the number of merge classes is 10, the gradient intensity ratio subclass is subclass merged into two subclasses of non-class and high class, the direction subclass is subclass merged into one subclass of N / A class, the activity subclass The activity subclass 0 corresponding to the index class_idx of 0, the activity subclass 1 corresponding to the index class_idx of 1, the activity subclass 2 corresponding to the index class_idx of 2 to 6, the activity subclass 3 corresponding to the index class_idx of 7 to 14, And, it can be obtained by subclass merging into 5 subclasses of activity subclass 4 corresponding to 15 index class_idx.
  • FIG. 48 is a diagram showing a merge pattern corresponding to the merge class number 9 obtained by the subclass merge and a subclass merge from which the merge pattern is obtained.
  • the merge pattern corresponding to 9 merge classes is to merge the gradient intensity ratio subclass into two subclasses, a nonclass and a high class, and subclass the direction subclass into two subclasses of D0 / D1 class and H / V class.
  • FIG. 49 is a diagram showing a merge pattern corresponding to the merge class number 8 obtained by the subclass merge and a subclass merge from which the merge pattern is obtained.
  • the merge pattern corresponding to the merge class number 8 is to merge the gradient intensity ratio subclass into two subclasses, a nonclass and a high class, to merge the direction subclass into one subclass of the N / A class, and to merge the activity subclass.
  • the activity subclass 0 corresponding to the index class_idx of 0 and 1
  • the activity subclass 1 corresponding to the index class_idx of 2 to 6
  • the activity subclass 2 corresponding to the index class_idx of 7 to 14, and the index class_idx of 15 It can be obtained by subclass merging into 4 subclasses of activity subclass 3.
  • FIG. 50 is a diagram showing a merge pattern corresponding to the number of merge classes 6 obtained by subclass merge and a subclass merge from which the merge pattern is obtained.
  • the merge pattern corresponding to the merge class number 6 is to merge the gradient intensity ratio subclass into two subclasses, a nonclass and a high class, to merge the direction subclass into one subclass of the N / A class, and to merge the activity subclass.
  • the merge pattern corresponding to the number of merge classes 5 is such that the gradient strength ratio subclass is subclass merged into one subclass of N / A class, the direction subclass is subclass merged into one subclass of N / A class, and the activity subclass is 0.
  • Activity subclasses 0 and 1 corresponding to the index class_idx, activity subclasses 1 and 2 corresponding to the index class_idx, an activity subclass 2 corresponding to the index class_idx of 2 to 6, and activity subclasses 3 and 15 corresponding to the index class_idx of 7 to 14 Can be obtained by subclass merging into 5 subclasses of the activity subclass 4 corresponding to the index class_idx.
  • FIG. 52 is a diagram showing a merge pattern corresponding to the merge class number 4 obtained by the subclass merge and a subclass merge from which the merge pattern is obtained.
  • the merge pattern corresponding to the merge class number 4 is that the gradient strength ratio subclass is subclass merged into one N / A class subclass, the direction subclass is subclass merged into one N / A class subclass, and the activity subclass is 0.
  • Activity subclass 0 corresponding to the index class_idx of 1 and 1; activity subclass 1 corresponding to the index class_idx of 2 to 6; activity subclass 2 corresponding to the index class_idx of 7 to 14; and activity subclass 3 corresponding to the index class_idx of 15 It can be obtained by subclass merging into 4 subclasses.
  • FIG. 53 is a diagram showing a merge pattern corresponding to three merge classes obtained by subclass merging and a subclass merge from which the merge pattern is obtained.
  • the merge pattern corresponding to the number of merge classes 3 is that the gradient strength ratio subclass is subclass merged into one subclass of N / A class, the direction subclass is subclass merged into one subclass of N / A class, and the activity subclass is 0.
  • Activity subclass 0 corresponding to an index class_idx of 6 to 6
  • activity subclass 1 corresponding to an index class_idx of 7 to 14
  • activity subclass 2 corresponding to an index class_idx of 15 can be obtained by subclass merging into three subclasses. .
  • FIG. 54 is a diagram showing a merge pattern corresponding to the number of merge classes 2 obtained by subclass merge and a subclass merge from which the merge pattern is obtained.
  • the merge pattern corresponding to the number of merge classes 2 is such that the gradient intensity ratio subclass is subclass merged into one subclass of N / A class, the direction subclass is subclass merged into one subclass of N / A class, and the activity subclass is 0. Alternatively, it can be obtained by subclass merging into two subclasses of the activity subclass 0 corresponding to the 14th index class_idx and the activity subclass 1 corresponding to the 15 index class_idx.
  • FIG. 55 is a diagram showing a merge pattern corresponding to the number of merge classes 1 obtained by subclass merge and a subclass merge from which the merge pattern is obtained.
  • the merge pattern corresponding to the number of merge classes 1 is such that the gradient intensity ratio subclass is subclass merged into one subclass of N / A class, the direction subclass is subclass merged into one subclass of N / A class, and the activity subclass is N It can be obtained by subclass merging into one subclass of the / A class (activity subclass corresponding to the index class_idx of 0 to 15).
  • FIG. 56 is a block diagram illustrating a configuration example of a classification prediction filter to which the present technology is applied.
  • the class classification prediction filter 110 performs class classification prediction processing.
  • a predetermined class classification is performed, and an initial class obtained by the predetermined class classification is converted into a merge class.
  • a filter process is performed as a prediction process in which a prediction equation using a tap coefficient of the merge class is applied.
  • the class classification prediction filter 110 includes a class classification unit 111, a merge conversion unit 112, a tap coefficient acquisition unit 113, and a prediction unit 114.
  • the target image (for example, the decoded image) to be subjected to the filter processing is supplied to the classifying unit 111 and the prediction unit 114.
  • the class classification unit 111 sequentially selects pixels of the target image as pixels of interest.
  • the class classification unit 111 obtains an initial class of the target pixel by performing, for example, a GALF class classification or the like as an initial class classification performed by subclass classification of each of the plurality of feature amounts with respect to the target pixel. It is supplied to the conversion unit 112.
  • the merge conversion unit 112 merges the initial class of the target pixel from the class classification unit 111 by merging the subclasses of the subclass classification (subclass merge) according to a merge pattern set in advance for each number of merge classes. Convert to merge class. That is, the merge conversion unit 112 stores, for example, a merge pattern preset for each number of merge classes by subclass merge of the gradient intensity ratio subclass, direction subclass, and activity subclass, and necessary partial merge. . Then, the merge conversion unit 112 converts the initial class of the pixel of interest into a merge class according to the merge pattern corresponding to the number of adopted merge classes among the merge patterns preset for each number of merge classes. The merge conversion unit 112 supplies the merge class of the target pixel to the tap coefficient acquisition unit 113.
  • the tap coefficient acquisition unit 113 stores tap coefficients for each merge class, and acquires tap coefficients used for filter processing as prediction processing of the target pixel according to the merge class of the target pixel from the merge conversion unit 112. .
  • the tap coefficient acquisition unit 113 selects the tap coefficient of the merge class of the pixel of interest from the merge conversion unit 112 from among the tap coefficients for each merge class (the tap coefficients for the number of adopted merge classes), and To supply.
  • the prediction unit 114 performs a filter process as a prediction process of applying a prediction formula using a tap coefficient of a merge class of the pixel of interest from the tap coefficient acquisition unit 113 to the target image, and a filter image generated by the filter process Is output.
  • the prediction unit 114 selects, for example, a plurality of pixels in the vicinity of the target pixel among the pixels of the target image as the prediction tap of the target pixel. Further, the prediction unit 114 performs a prediction process of applying a prediction expression composed of tap coefficients of the class of the pixel of interest to the target image, that is, the pixel (the pixel value thereof) x as the prediction tap of the pixel of interest.
  • the classifying / prediction filter 110 can include a learning unit 121 for performing tap coefficient learning. If the function of performing tap coefficient learning is referred to as a learning function, it can be said that the classification prediction filter 110 having the learning unit 121 is a classification prediction filter 110 with a learning function.
  • the learning unit 121 can use the teacher image and the student image to determine tap coefficients for each merge class, and store the tap coefficients in the tap coefficient acquisition unit 113. Further, the learning unit 121 can determine the number of adopted merge classes and supply the number to the merge conversion unit 112.
  • the learning unit 121 performs the same class classification as that of the class classification unit 111 using the decoded image as the student image, and performs a prediction formula including a tap coefficient and a prediction tap for each initial class obtained by the class classification. Tap coefficient learning is performed to find a tap coefficient that statistically minimizes the prediction error of the prediction value of the teacher image obtained by using the least square method.
  • the learning unit 121 stores the merge pattern corresponding to each of the plurality of merge classes as the same merge pattern as the preset merge pattern for each merge class stored in the merge conversion unit 112. I have.
  • the learning unit 121 sets the merge pattern in advance by performing the same process as the adopted merge class number determination process (FIG. 8) using each of the merge patterns corresponding to the plurality of preset merge class numbers.
  • the number of merge classes that minimizes the cost (for example, the cost dist + lambda ⁇ coeffBit obtained in step S67 of FIG. 8) among the plurality of merge classes is determined as the number of adopted merge classes.
  • step S63 performs the merge pattern determination process in step S63 before performing the process of step S64 which is a filter process for obtaining a cost for determining the number of adopted merge classes in the adopted merge class number determination process (FIG. 8).
  • step S64 is a filter process for obtaining a cost for determining the number of adopted merge classes in the adopted merge class number determination process (FIG. 8).
  • the learning unit 121 supplies the number of adopted merge classes to the merge conversion unit 112, and supplies the tap coefficients for each merge class of the number of adopted merge classes to the tap coefficient acquisition unit 113.
  • the merge conversion unit 112 outputs the target pixel from the class classification unit 111 in accordance with the merge pattern corresponding to the number of adopted merge classes supplied to the merge pattern corresponding to each of the plurality of preset merge classes. Convert initial class to merge class.
  • the merge pattern corresponding to each of the plurality of merge classes stored in the merge conversion unit 112 and the learning unit 121 is a merge pattern set for each merge class number, the merge pattern is uniquely identified by the number of merge classes. can do.
  • class classification prediction filter 110 associates the number of merge classes with a merge pattern set in advance as a merge pattern corresponding to the number of merge classes.
  • merge information information in which the number of merge classes is associated with a merge pattern set in advance as a merge pattern corresponding to the number of merge classes.
  • the encoding device and the decoding device to which the present technology is applied share the merge information. Then, the encoding device determines the number of adopted merge classes from the plurality of merge classes, and transmits the determined number of merge classes to the decoding device. The decoding device specifies a merge pattern from the number of adopted merge classes from the encoding device. Then, the decoding device performs the initial class classification, and converts the initial class obtained by the initial class classification into a merge class according to a merge pattern specified from the number of adopted merge classes (a merge pattern corresponding to the number of adopted merge classes). I do.
  • FIG. 57 is a flowchart for explaining an example of the classification prediction process performed by the classification prediction filter 110 of FIG.
  • step S111 the class classification unit 111 sequentially selects pixels to be selected as target pixels of the decoded image as the target image as target pixels, and the process proceeds to step S112.
  • step S112 the class classification unit 111 performs an initial class classification of the target pixel, and obtains an initial class of the target pixel.
  • the class classification unit 111 supplies the initial class of the pixel of interest to the merge conversion unit 112, and the process proceeds from step S112 to step S113.
  • step S113 the merge conversion unit 112 converts the initial class of the pixel of interest from the class classification unit 111 into a merge class according to the merge pattern corresponding to the number of adopted merge classes.
  • the merge conversion unit 112 supplies the merge class of the target pixel to the tap coefficient acquisition unit 113, and the process proceeds from step S113 to step S114.
  • step S114 the tap coefficient obtaining unit 113 obtains the tap coefficient of the merge class of the target pixel from the merge conversion unit 112 from among the tap coefficients for each merge class, and the process proceeds to step S115.
  • step S115 the prediction unit 114 performs a filtering process as a prediction process of applying a prediction expression composed of tap coefficients of a merge class of the pixel of interest from the tap coefficient acquisition unit 113 to the decoded image.
  • the prediction unit 114 selects, from the decoded image, a pixel to be a prediction tap of the target pixel, and calculates a primary prediction expression configured using the prediction tap and a tap coefficient of a merge class of the target pixel. , The predicted value of (the pixel value of) the pixel of the original image with respect to the pixel of interest. Then, the prediction unit 114 generates an image having the prediction value as a pixel value, outputs the image as a filter image, and ends the class classification prediction process.
  • FIG. 58 is a block diagram illustrating an outline of an embodiment of an image processing system to which the present technology is applied.
  • the image processing system includes an encoding device 160 and a decoding device 170.
  • the encoding device 160 includes an encoding unit 161, a local decoding unit 162, and a filter unit 163.
  • the encoding unit 161 uses the filter image from the filter unit 163 to convert the original image into, for example, a predetermined block unit such as CU of Quad-Tree ⁇ Block ⁇ Structure or QTBT (Quad ⁇ Tree ⁇ Plus ⁇ Binary ⁇ Tree) ⁇ Block ⁇ Structure.
  • a predetermined block unit such as CU of Quad-Tree ⁇ Block ⁇ Structure or QTBT (Quad ⁇ Tree ⁇ Plus ⁇ Binary ⁇ Tree) ⁇ Block ⁇ Structure.
  • the encoded data obtained by the (prediction) encoding and the encoding is supplied to the local decoding unit 162.
  • the encoding unit 161 subtracts the predicted image of the original image obtained by performing motion compensation on the filtered image from the filter unit 163 from the original image, and encodes the resulting residual.
  • the filter information is supplied from the filter unit 163 to the encoding unit 161.
  • the encoding unit 161 generates and transmits (transmits) an encoded bit stream including encoded data and filter information from the filter unit 163.
  • the local decoding unit 162 is supplied with the encoded data from the encoding unit 161 and the filtered image from the filter unit 163.
  • the local decoding unit 162 performs local decoding of the encoded data from the encoding unit 161 using the filter image from the filter unit 163, and supplies the (local) decoded image obtained as a result to the filter unit 163.
  • the local decoding unit 162 decodes the encoded data from the encoding unit 161 into a residual, and applies a prediction image of an original image obtained by performing motion compensation of the filter image from the filter unit 163 to the residual. By adding, a decoded image (local decoded image) obtained by decoding the original image is generated.
  • the filter unit 163 is configured, for example, in the same manner as the class classification prediction filter 110 with a learning function (FIG. 56), and includes a class classification unit 164 and a merge conversion unit 165.
  • the filter unit 163 performs tap coefficient learning using the decoded image from the local decoding unit 162 and the original image for the decoded image as a student image and a teacher image, and obtains a tap coefficient for each class.
  • the filter unit 163 performs a process similar to the adopted merge class number determination process (FIG. 8) using a merge pattern preset for each of the plurality of merge classes, whereby a merge pattern is set in advance.
  • the number of merge classes that minimizes the cost is determined as the number of adopted merge classes.
  • the filter unit 163 uses the normal equations (the X matrix and the Y vector thereof) obtained by the tap coefficient learning to determine the merge pattern in steps S36 and S36 of the merge pattern determination process (FIG. 5). By performing the same processing as in S37, the tap coefficient for each merge class of the number of adopted merge classes is obtained.
  • the filter unit 163 performs, for example, a GALF class classification or the like as an initial class classification performed by subclass classification of a plurality of feature amounts using the decoded image from the local decoding unit 162 in the class classification unit 164. , The initial class of the pixel of interest is determined. Further, the filter unit 163 causes the merge conversion unit 165 to convert the initial class of the target pixel into a merge class in which the initial class is merged by merging the subclasses of the subclass classification according to the merge pattern corresponding to the number of adopted merge classes. .
  • the filter unit 163 performs a filter process as a prediction process for applying a prediction formula for performing a product-sum operation of a tap coefficient of a merge class of the pixel of interest obtained by the conversion of the merge conversion unit 165 and a pixel of the decoded image to the decoded image. I do.
  • the filter unit 163 supplies the filter image obtained by the filter processing to the encoding unit 161 and the local decoding unit 162. Further, the filter unit 163 supplies the encoding unit 161 with the number of adopted merge classes and the tap coefficient for each merge class of the number of adopted merge classes as filter information.
  • the number of merge classes that minimizes the cost among a plurality of merge classes in which merge patterns are set in advance is determined as the number of adopted merge classes.
  • the number of merge classes of a specific merge pattern among a plurality of merge classes in which merge patterns are set in advance can be determined in advance as the number of adopted merge classes. In this case, it is not necessary to obtain a cost in order to determine the number of employed merge classes, so that the processing amount of the encoding device 160 can be reduced.
  • determining the number of adopted merge classes in advance is effective, for example, particularly when the performance of the encoding device 160 is not high.
  • the decoding device 170 includes a parsing unit 171, a decoding unit 172, and a filter unit 173.
  • the parsing unit 171 receives the encoded bit stream transmitted by the encoding device 160, performs parsing, and supplies filter information obtained by the parsing to the filter unit 173. Further, the parsing unit 171 supplies the encoded data included in the encoded bit stream to the decoding unit 172.
  • the decoding unit 172 is supplied with the encoded data from the parsing unit 171 and the filtered image from the filtering unit 173.
  • the decoding unit 172 decodes the encoded data from the parsing unit 171 using the filter image from the filter unit 173, for example, in the same manner as the encoding unit 161, in units of predetermined blocks such as CUs.
  • the obtained decoded image is supplied to the filter unit 173.
  • the decoding unit 172 decodes the encoded data from the parsing unit 171 into a residual, and obtains the residual by performing motion compensation on the filter image from the filter unit 173.
  • a decoded image obtained by decoding the original image is generated by adding the predicted image of the original image.
  • the filter unit 173 performs the same filter processing as the filter unit 163 on the decoded image from the decoding unit 172, generates a filtered image, and supplies the generated filtered image to the decoding unit 172.
  • the filter unit 173 performs the same initial class classification as the class classification unit 164 using the decoded image from the decoding unit 172 in the class classification unit 174, and obtains the initial class of the pixel of interest. Further, the filter unit 173 merges the initial class of the target pixel and the subclass of the subclass classification in the merge conversion unit 175 according to the merge pattern corresponding to the number of adopted merge classes included in the filter information from the parse unit 171 (subclass merge). ) To convert the initial class into a merged class.
  • the filter unit 173 supplies the filtered image obtained by the filtering process to the decoding unit 172 and outputs the filtered image as a final decoded image obtained by decoding the original image.
  • FIG. 59 is a flowchart illustrating an outline of the encoding process of the encoding device 160 in FIG. 58.
  • the processing according to the flowchart of FIG. 59 is performed, for example, on a frame (picture) basis.
  • step S161 the encoding unit 161 (FIG. 58) encodes (predicts) the original image using the filter image from the filter unit 163, and sends the encoded data obtained by the encoding to the local decoding unit 162. Then, the process proceeds to step S162.
  • step S162 the local decoding unit 162 performs local decoding of the encoded data from the encoding unit 161 using the filter image from the filter unit 163, and outputs the (local) decoded image obtained as a result to the filter unit 163. , And the process proceeds to Step S163.
  • step S163 the filter unit 163 performs tap coefficient learning using the decoded image from the local decoding unit 162 and the original image for the decoded image as a student image and a teacher image, and obtains a tap coefficient for each initial class. , The process proceeds to step S164.
  • step S164 the filter unit 163 performs merging of the initial classes according to the merge pattern corresponding to the number of merge classes for each of the plurality of merge classes for which the merge pattern is set in advance, and obtains a tap coefficient for each initial class.
  • the initial class is merged according to the merge pattern corresponding to the number of merge classes.
  • the filter unit 163 determines the number of merge classes with the minimum cost as the number of adopted merge classes using the tap coefficient for each merge class, and the process proceeds from step S164 to step S165.
  • step S165 the class classification unit 164 of the filter unit 163 performs the initial class classification of the target pixel of the decoded image from the local decoding unit 162, and the process proceeds to step S166.
  • step S166 the merge conversion unit 165 of the filter unit 163 converts the initial class of the pixel of interest obtained by the class classification of the class classification unit 164 into a merge class according to the merge pattern corresponding to the number of adopted merge classes. Then, the process proceeds to step S167.
  • step S167 the filter unit 163 converts the prediction expression for performing the product-sum operation between the tap coefficient of the merge class of the target pixel and the pixel of the decoded image among the tap coefficients for each merge class obtained in step S164, into the decoded image.
  • a filter process is performed as a prediction process to be applied to the image data, and a filter image is generated.
  • the filter image is supplied from the filter unit 163 to the encoding unit 161 and the local decoding unit 162.
  • the filter image supplied from the filter unit 163 to the encoding unit 161 and the local decoding unit 162 is used in the processing of steps S161 and S162 performed for the next frame.
  • the filter unit 163 supplies the number of adopted merge classes and the tap coefficients for each merge class to the encoding unit 161 as filter information.
  • step S167 the encoding unit 161 determines the encoded data obtained in step S161, the number of merge classes adopted as filter information obtained by the filter unit 163, and the merge class. And generates and transmits an encoded bit stream that includes a tap coefficient for each.
  • FIG. 60 is a flowchart for explaining the outline of the decoding process of the decoding device 170 in FIG.
  • step S181 the parsing unit 171 (FIG. 58) receives the encoded bit stream transmitted from the encoding device 160, and determines the number of merge classes adopted as filter information included in the encoded bit stream, and the number of merge classes. The tap coefficient for each is parsed and supplied to the filter unit 173. Further, the parsing unit 171 supplies the encoded data included in the encoded bit stream to the decoding unit 172, and the process proceeds from step S181 to step S182.
  • step S182 the decoding unit 172 decodes the encoded data from the parsing unit 171 using the filter image from the filter unit 173, and supplies the resulting decoded image to the filter unit 173 to perform processing. Proceeds to step S183.
  • step S183 the class classification unit 174 of the filter unit 173 performs initial class classification on the target pixel of the decoded image from the decoding unit 172, and the process proceeds to step S184.
  • step S184 the merge conversion unit 175 of the filter unit 173 converts the initial class of the pixel of interest obtained by the class classification of the class classification unit 174 into a merge class according to the merge pattern corresponding to the number of adopted merge classes from the parsing unit 171. After the conversion, the process proceeds to step S185.
  • step S185 the filter unit 173 performs a filtering process as a class classification prediction process on the decoded image from the decoding unit 172 using the tap coefficient for each merge class from the parsing unit 171 to generate a filtered image.
  • the filter unit 173 applies, to the decoded image, a prediction expression that performs a product-sum operation between the tap coefficient of the merge class of the target pixel and the pixel of the decoded image among the tap coefficients for each merge class from the parsing unit 171. Filter processing is performed as prediction processing to generate a filtered image.
  • the filter image is supplied from the filter unit 173 to the decoding unit 172, and is output as a final decoded image obtained by decoding the original image.
  • the filter image supplied from the filter unit 173 to the decoding unit 172 is used in the process of step S182 performed on the next frame of the decoded image.
  • a method of signaling a merge pattern (adopted merge pattern) for converting the initial class to the merge class a method of transmitting the number of adopted merge classes in the coded bit stream is adopted.
  • a signaling method as in the case of GALF, it is possible to adopt a method in which the adopted merge pattern is included in the coded bit stream and transmitted together with or instead of the adopted merge class number. .
  • transmitting the number of adopted merge classes can reduce the overhead compared to transmitting the adopted merge pattern.
  • the same syntax as the GALF class classification can be adopted.
  • FIG. 61 is a block diagram showing a detailed configuration example of the encoding device 160 in FIG.
  • the encoding device 160 includes an A / D conversion unit 201, a rearrangement buffer 202, an operation unit 203, an orthogonal transformation unit 204, a quantization unit 205, a lossless encoding unit 206, and a storage buffer 207. Further, the encoding device 160 includes an inverse quantization unit 208, an inverse orthogonal transform unit 209, an operation unit 210, an ILF 211, a frame memory 212, a selection unit 213, an intra prediction unit 214, a motion prediction compensation unit 215, and a predicted image selection unit 216. , And a rate control unit 217.
  • the A / D converter 201 A / D converts an original image of an analog signal into an original image of a digital signal, and supplies the original image to the rearrangement buffer 202 for storage.
  • the rearrangement buffer 202 rearranges the frames of the original image in the order of display from the display order to the encoding (decoding) according to the GOP (Group Of Picture), and calculates the arithmetic unit 203, the intra prediction unit 214, the motion prediction compensation unit 215, and , ILF211.
  • the operation unit 203 subtracts the predicted image supplied from the intra prediction unit 214 or the motion prediction compensation unit 215 via the predicted image selection unit 216 from the original image from the rearrangement buffer 202, and obtains a residual obtained by the subtraction. (Prediction residual) is supplied to the orthogonal transform unit 204.
  • the calculation unit 203 subtracts the predicted image supplied from the motion prediction compensation unit 215 from the original image read from the rearrangement buffer 202.
  • the orthogonal transform unit 204 performs an orthogonal transform such as a discrete cosine transform or a Karhunen-Loeve transform on the residual supplied from the arithmetic unit 203.
  • an orthogonal transform such as a discrete cosine transform or a Karhunen-Loeve transform
  • the method of this orthogonal transformation is arbitrary.
  • the orthogonal transform unit 204 supplies the orthogonal transform coefficients obtained by the orthogonal exchange to the quantization unit 205.
  • the quantization unit 205 quantizes the orthogonal transform coefficients supplied from the orthogonal transform unit 204.
  • the quantization unit 205 sets the quantization parameter QP based on the target code amount (code amount target value) supplied from the rate control unit 217, and quantizes the orthogonal transform coefficients. Note that this quantization method is optional.
  • the quantization unit 205 supplies the encoded data, which is the quantized orthogonal transform coefficient, to the lossless encoding unit 206.
  • the lossless encoding unit 206 encodes the quantized orthogonal transform coefficients as encoded data from the quantization unit 205 using a predetermined lossless encoding method. Since the orthogonal transform coefficients are quantized under the control of the rate control unit 217, the code amount of the coded bit stream obtained by the lossless coding of the lossless coding unit 206 depends on the code set by the rate control unit 217. It becomes the amount target value (or approximates the code amount target value).
  • the lossless encoding unit 206 acquires, from each block, encoding information necessary for decoding by the decoding device 170, out of encoding information related to predictive encoding by the encoding device 160.
  • the coding information for example, prediction modes of intra prediction and inter prediction, motion information such as a motion vector, a code amount target value, a quantization parameter QP, a picture type (I, P, B), CU (Coding Unit) and CTU (Coding Tree Unit) information.
  • motion information such as a motion vector, a code amount target value, a quantization parameter QP, a picture type (I, P, B), CU (Coding Unit) and CTU (Coding Tree Unit) information.
  • the prediction mode can be obtained from the intra prediction unit 214 or the motion prediction compensation unit 215.
  • the motion information can be acquired from the motion prediction compensation unit 215.
  • the lossless encoding unit 206 acquires the coding information, and also acquires, from the ILF 211, a tap coefficient for each class as filter information related to the filtering process in the ILF 211.
  • the lossless encoding unit 206 converts the encoding information and the filter information into, for example, variable length encoding such as CAVLC (Context-Adaptive Variable Length Coding) or CABAC (Context-Adaptive Binary Arithmetic Coding), arithmetic coding, or other reversible coding.
  • variable length encoding such as CAVLC (Context-Adaptive Variable Length Coding) or CABAC (Context-Adaptive Binary Arithmetic Coding), arithmetic coding, or other reversible coding.
  • a coded bit stream including the coded information and the filter information after the coding and the coded data from the quantization unit 205 is generated and supplied to the storage buffer 207.
  • the storage buffer 207 temporarily stores the coded bit stream supplied from the lossless coding unit 206.
  • the coded bit stream stored in the storage buffer 207 is read and transmitted at a predetermined timing.
  • the encoded data which is the orthogonal transform coefficient quantized by the quantization unit 205, is supplied to the lossless encoding unit 206 and also to the inverse quantization unit 208.
  • the inverse quantization unit 208 inversely quantizes the quantized orthogonal transform coefficient by a method corresponding to the quantization by the quantization unit 205, and outputs the orthogonal transform coefficient obtained by the inverse quantization to the inverse orthogonal transform unit 209. Supply.
  • the inverse orthogonal transform unit 209 performs an inverse orthogonal transform on the orthogonal transform coefficient supplied from the inverse quantization unit 208 by a method corresponding to the orthogonal transform process by the orthogonal transform unit 204, and calculates a residual obtained as a result of the inverse orthogonal transform. , To the arithmetic unit 210.
  • the arithmetic unit 210 adds the prediction image supplied from the intra prediction unit 214 or the motion prediction compensation unit 215 via the prediction image selection unit 216 to the residual supplied from the inverse orthogonal transform unit 209, and thereby calculates the original A decoded image (part of) obtained by decoding the image is obtained and output.
  • the decoded image output from the operation unit 210 is supplied to the ILF 211.
  • the ILF 211 is configured, for example, in the same manner as the class classification prediction filter 110 with a learning function (FIG. 56), and performs a filtering process as a class classification prediction process to perform a deblocking filter, an adaptive offset filter, a bilateral filter, and an ALF. , Or as two or more filters.
  • the ILF 211 functions as two or more filters among the deblocking filter, the adaptive offset filter, the bilateral filter, and the ALF, the arrangement order of the two or more filters is arbitrary.
  • the ILF 211 is supplied with the decoded image from the arithmetic unit 210 and the original image for the decoded image from the rearrangement buffer 202.
  • the $ ILF 211 stores merge information in which a plurality of merge classes are associated with a merge pattern set in advance for each merge class.
  • the ILF 211 performs tap coefficient learning by using, for example, the decoded image from the arithmetic unit 210 and the original image from the rearrangement buffer 202 as a student image and a teacher image, respectively, and obtains a tap coefficient for each initial class.
  • initial class classification is performed using a decoded image as a student image, and for each initial class obtained by the initial class classification, a teacher obtained by a prediction formula composed of a tap coefficient and a prediction tap is used.
  • a tap coefficient that statistically minimizes a prediction error of a prediction value of an original image as an image is obtained by a least square method.
  • the ILF 211 performs a process similar to the adopted merge class number determination process (FIG. 8) using the merge patterns corresponding to the plurality of merge classes included in the merge information, thereby obtaining the plurality of merge classes included in the merge information.
  • the merge class number that minimizes the cost (for example, the cost dist + lambda ⁇ coeffBit obtained in step S67 in FIG. 8) is determined as the number of adopted merge classes.
  • step S63 which is a filter process for obtaining a cost for determining the number of adopted merge classes in the adopted merge class number determination process (FIG. 8)
  • a merge pattern determination process (FIG. 8) is performed.
  • steps S36 and S37 of 5) a plurality of plural variables included in the merge information are obtained by using a normal equation (an X matrix and a Y vector of the normal equations) established when obtaining the tap coefficients for each initial class by tap coefficient learning. For each number of merge classes, a tap coefficient for each merge class is determined.
  • the ILF 211 supplies the number of adopted merge classes and the tap coefficient of each adopted merge class to the lossless encoding unit 206 as filter information.
  • the ILF 211 sequentially selects, for example, pixels of the decoded image from the arithmetic unit 210 as target pixels.
  • the ILF 211 performs an initial class classification on the pixel of interest and obtains an initial class of the pixel of interest.
  • the ILF 211 converts the initial class of the pixel of interest into a merge class according to a merge pattern corresponding to the number of adopted merge classes.
  • the ILF 211 acquires (reads) the tap coefficient of the merge class of the target pixel among the tap coefficients for each merge class obtained by conversion according to the merge pattern corresponding to the number of adopted merge classes. Then, the ILF 211 selects, from the decoded image, a pixel in the vicinity of the pixel of interest as a prediction tap, and decodes a prediction equation for performing a product-sum operation between a tap coefficient of a merge class of the pixel of interest and a pixel of the decoded image as a prediction tap.
  • Filter processing is performed as prediction processing applied to the image to generate a filtered image.
  • the class classification by the ILF 211 for example, the class obtained by the class classification of the upper left pixel of 2 ⁇ 2 pixels of the decoded image can be adopted as the class of each 2 ⁇ 2 pixel.
  • the filter image generated by the ILF 211 is supplied to the frame memory 212.
  • the frame memory 212 temporarily stores the filter image supplied from the ILF 211.
  • the filter image stored in the frame memory 212 is supplied to the selection unit 213 at a necessary timing as a reference image used for generating a predicted image.
  • the selection unit 213 selects a supply destination of the reference image supplied from the frame memory 212. For example, when the intra prediction is performed by the intra prediction unit 214, the selection unit 213 supplies the reference image supplied from the frame memory 212 to the intra prediction unit 214. Further, for example, when inter prediction is performed in the motion prediction compensation unit 215, the selection unit 213 supplies the reference image supplied from the frame memory 212 to the motion prediction compensation unit 215.
  • the intra prediction unit 214 uses the original image supplied from the rearrangement buffer 202 and the reference image supplied from the frame memory 212 via the selection unit 213, and uses, for example, a PU (Prediction @ Unit) as a processing unit to perform intra prediction. Perform prediction (in-screen prediction).
  • the intra-prediction unit 214 selects an optimal intra-prediction mode based on a predetermined cost function (for example, RD cost or the like), and sends the predicted image generated in the optimal intra-prediction mode to the predicted image selection unit 216. Supply. Further, as described above, the intra prediction unit 214 appropriately supplies the prediction mode indicating the intra prediction mode selected based on the cost function to the lossless encoding unit 206 and the like.
  • the motion prediction compensation unit 215 uses the original image supplied from the rearrangement buffer 202 and the reference image supplied from the frame memory 212 via the selection unit 213, and performs motion prediction (inter Prediction). Further, the motion prediction compensation unit 215 performs motion compensation according to the motion vector detected by the motion prediction, and generates a predicted image. The motion prediction compensation unit 215 performs inter prediction in a plurality of inter prediction modes prepared in advance to generate a predicted image.
  • the motion prediction compensation unit 215 selects an optimal inter prediction mode based on a predetermined cost function of a predicted image obtained for each of the plurality of inter prediction modes. Further, the motion prediction compensation unit 215 supplies the predicted image generated in the optimal inter prediction mode to the predicted image selection unit 216.
  • the motion prediction compensation unit 215 performs a prediction mode indicating the inter prediction mode selected based on the cost function, and a motion vector such as a motion vector required when decoding the encoded data encoded in the inter prediction mode. Information and the like are supplied to the lossless encoding unit 206.
  • the predicted image selection unit 216 selects a supply source (the intra prediction unit 214 or the motion prediction compensation unit 215) of the predicted image to be supplied to the calculation units 203 and 210, and the prediction supplied from the selected supply source.
  • the image is supplied to the calculation units 203 and 210.
  • the rate control unit 217 controls the rate of the quantization operation of the quantization unit 205 based on the code amount of the coded bit stream stored in the storage buffer 207 so that overflow or underflow does not occur. That is, the rate control unit 217 sets the target code amount of the coded bit stream so as to prevent overflow and underflow of the accumulation buffer 207, and supplies the target code amount to the quantization unit 205.
  • the arithmetic unit 203 to the lossless encoding unit 206 are in the encoding unit 161 in FIG. 58
  • the inverse quantization unit 208 to the arithmetic unit 210 are in the local decoding unit 162 in FIG. 58
  • the ILF 211 is the filter in FIG.
  • the section 163 corresponds to each.
  • FIG. 62 is a flowchart for explaining an example of the encoding process of the encoding device 160 in FIG.
  • the ILF 211 temporarily stores the decoded image supplied from the arithmetic unit 210 and also temporarily stores the original image for the decoded image supplied from the arithmetic unit 210 and supplied from the rearrangement buffer 202.
  • step S201 the encoding device 160 (the control unit (not shown)) determines whether the current timing is an update timing for updating the filter information.
  • the update timing of the filter information is, for example, every one or more frames (pictures), every one or more sequences, every one or more slices, every one or more lines of a predetermined block such as a CTU, etc. , Can be determined in advance.
  • the update timing of the filter information in addition to the periodic (fixed) timing such as the timing of one or more frames (pictures), the timing at which the S / N of the filter image becomes equal to or less than the threshold (filter image Dynamic timing, such as the timing at which the error with respect to the original image exceeds the threshold, or the timing at which the residual (the sum of absolute values thereof) exceeds the threshold.
  • filter image Dynamic timing such as the timing at which the error with respect to the original image exceeds the threshold, or the timing at which the residual (the sum of absolute values thereof) exceeds the threshold.
  • the ILF 211 performs tap coefficient learning using one frame of the decoded image and the original image, and the timing of each frame is the update timing of the filter information.
  • step S201 If it is determined in step S201 that the current timing is not the update timing of the filter information, the process skips steps S202 to S205 and proceeds to step S206.
  • step S201 If it is determined in step S201 that the current timing is the update timing of the filter information, the process proceeds to step S202, and the ILF 211 performs tap coefficient learning for obtaining a tap coefficient for each initial class.
  • the ILF 211 uses, for example, the decoded image and the original image stored between the previous update timing and the current update timing (here, the latest one-frame decoded image and original image supplied to the ILF 211). Then, tap coefficient learning is performed to determine tap coefficients for each initial class.
  • step S203 the ILF 211 converts each of the plurality of merge classes included in the merge information into a merge class by merging the initial class according to a merge pattern corresponding to the number of merge classes, and converts the merge classes into steps S36 and S37 in FIG. Similarly, a tap coefficient for each merge class is obtained by using a normal equation created by tap coefficient learning.
  • the ILF 211 obtains a cost (for example, the cost dist + lambda ⁇ coeffBit obtained in step S67 in FIG. 8) by performing a filter process using a tap coefficient for each of the plurality of merge classes. . Then, the ILF 211 determines the number of merge classes with the lowest cost among the plurality of merge classes as the number of adopted merge classes, and the process proceeds from step S203 to step S204.
  • a cost for example, the cost dist + lambda ⁇ coeffBit obtained in step S67 in FIG. 8
  • step S204 the ILF 211 supplies the number of adopted merge classes and the tap coefficient of each adopted merge class to the lossless encoding unit 206 as filter information.
  • the lossless encoding unit 206 sets the filter information from the ILF 211 as a transmission target, and the process proceeds from step S204 to step S205.
  • the filter information set as the transmission target is included in the coded bit stream and transmitted in the predictive coding process performed in step S206 described below.
  • step S205 the ILF 211 updates the number of adoption merges and the tap coefficient used in the class classification prediction process by using the number of adoption merge classes determined in step S203 and the tap coefficient for each merge class of the number of adoption merge classes. Proceeds to step S206.
  • step S206 the predictive encoding of the original image is performed, and the encoding ends.
  • FIG. 63 is a flowchart illustrating an example of the predictive encoding process in step S206 in FIG.
  • step S211 the A / D conversion unit 201 performs A / D conversion on the original image and supplies it to the rearrangement buffer 202, and the process proceeds to step S212.
  • step S212 the rearrangement buffer 202 stores the original images from the A / D converter 201, rearranges and outputs the images in the order of encoding, and the process proceeds to step S213.
  • step S213 the intra prediction unit 214 performs an intra prediction process in an intra prediction mode, and the process proceeds to step S214.
  • the motion prediction / compensation unit 215 performs an inter motion prediction process for performing motion prediction or motion compensation in the inter prediction mode, and the process proceeds to step S215.
  • step S215 the prediction image selection unit 216 determines an optimal prediction mode based on each cost function obtained by the intra prediction unit 214 and the motion prediction compensation unit 215. Then, the predicted image selection unit 216 selects and outputs a predicted image in an optimal prediction mode from the predicted image generated by the intra prediction unit 214 and the predicted image generated by the motion prediction compensation unit 215, and outputs the selected image. Proceeds from step S215 to step S216.
  • step S216 the arithmetic unit 203 calculates the residual between the encoding target image, which is the original image output from the rearrangement buffer 202, and the predicted image output from the predicted image selection unit 216. , And the process proceeds to step S217.
  • step S217 the orthogonal transformation unit 204 performs orthogonal transformation on the residual from the calculation unit 203, and supplies the resulting orthogonal transformation coefficient to the quantization unit 205, and the process proceeds to step S218.
  • step S218 the quantization unit 205 quantizes the orthogonal transform coefficient from the orthogonal transform unit 204, and supplies the quantized coefficient obtained by the quantization to the lossless encoding unit 206 and the inverse quantization unit 208, The process proceeds to step S219.
  • step S219 the inverse quantization unit 208 inversely quantizes the quantized coefficient from the quantization unit 205, and supplies the resulting orthogonal transform coefficient to the inverse orthogonal transform unit 209, and the process proceeds to step S220. move on.
  • step S220 the inverse orthogonal transform unit 209 performs an inverse orthogonal transform on the orthogonal transform coefficient from the inverse quantization unit 208, and supplies the resulting residual to the arithmetic unit 210, and the process proceeds to step S221. .
  • step S221 the arithmetic unit 210 adds the residual from the inverse orthogonal transform unit 209 and the predicted image output from the predicted image selecting unit 216, and calculates the element for which the arithmetic unit 203 has calculated the residual. Generate a decoded image corresponding to the image.
  • the arithmetic unit 210 supplies the decoded image to the ILF 211, and the process proceeds from step S221 to step S222.
  • step S222 the ILF 211 performs a filtering process as a class classification prediction process on the decoded image from the calculation unit 210, and supplies a filtered image obtained by the filtering process to the frame memory 212. Then, the process proceeds to step S223.
  • step S222 the same process as that of the classification prediction filter 110 (FIG. 56) is performed.
  • the ILF 211 performs initial class classification on the target pixel of the decoded image from the arithmetic unit 210, and obtains the initial class of the target pixel. Further, the ILF 211 converts the initial class of the target pixel into a merge class according to the merge pattern corresponding to the number of adopted merge classes updated in step S205 in FIG. The ILF 211 acquires the tap coefficient of the merge class of the target pixel from the tap coefficients for each merge class updated in step S205 in FIG. After that, the ILF 211 performs a filtering process as a prediction process of applying a prediction formula configured using a tap coefficient of a merge class of the pixel of interest to the decoded image, and generates a filtered image. The filter image is supplied from the ILF 211 to the frame memory 212.
  • step S223 the frame memory 212 stores the filter image supplied from the ILF 211, and the process proceeds to step S224.
  • the filter image stored in the frame memory 212 is used as a reference image from which a predicted image is generated in steps S213 and S114.
  • the lossless encoding unit 206 encodes the encoded data, which is the quantized coefficient from the quantization unit 205, and generates an encoded bit stream including the encoded data. Further, the lossless encoding unit 206 includes a quantization parameter QP used for quantization in the quantization unit 205, a prediction mode obtained in intra prediction processing in the intra prediction unit 214, and a prediction mode obtained in the motion prediction compensation unit 215. Encoding information such as a prediction mode and motion information obtained by the inter motion prediction processing is encoded as necessary, and included in an encoded bit stream.
  • the lossless encoding unit 206 encodes the filter information set as a transmission target in step S203 of FIG. 62 as necessary, and includes the encoded filter information in the encoded bit stream. Then, the lossless encoding unit 206 supplies the encoded bit stream to the accumulation buffer 207, and the process proceeds from step S224 to step S225.
  • step S225 the accumulation buffer 207 accumulates the encoded bit stream from the lossless encoding unit 206, and the process proceeds to step S226.
  • the coded bit stream stored in the storage buffer 207 is appropriately read and transmitted.
  • step S226 the rate control unit 217 determines the quantum of the quantization unit 205 based on the code amount (generated code amount) of the coded bit stream stored in the storage buffer 207 so that overflow or underflow does not occur.
  • the rate of the encoding operation is controlled, and the encoding process ends.
  • FIG. 64 is a block diagram showing a detailed configuration example of the decoding device 170 in FIG.
  • the decoding device 170 includes an accumulation buffer 301, a lossless decoding unit 302, an inverse quantization unit 303, an inverse orthogonal transformation unit 304, an operation unit 305, an ILF 306, a reordering buffer 307, and a D / A conversion unit 308.
  • the decoding device 170 includes a frame memory 310, a selection unit 311, an intra prediction unit 312, a motion prediction compensation unit 313, and a selection unit 314.
  • the accumulation buffer 301 temporarily stores the encoded bit stream transmitted from the encoding device 160 and supplies the encoded bit stream to the lossless decoding unit 302 at a predetermined timing.
  • the lossless decoding unit 302 receives the encoded bit stream from the storage buffer 301, and decodes the encoded bit stream using a method corresponding to the encoding method of the lossless encoding unit 206 in FIG.
  • the lossless decoding unit 302 supplies the quantization coefficient as encoded data included in the decoding result of the encoded bit stream to the inverse quantization unit 303.
  • the lossless decoding unit 302 has a function of performing parsing.
  • the lossless decoding unit 302 parses necessary encoding information and filter information included in the decoding result of the encoded bit stream, and supplies the encoded information to the intra prediction unit 312, the motion prediction compensation unit 313, and other necessary blocks. I do. Further, the lossless decoding unit 302 supplies the filter information to the ILF 306.
  • the inverse quantization unit 303 inversely quantizes the quantized coefficient as the encoded data from the lossless decoding unit 302 by a method corresponding to the quantization method of the quantization unit 205 in FIG. 61, and is obtained by the inverse quantization.
  • the orthogonal transform coefficient is supplied to the inverse orthogonal transform unit 304.
  • the inverse orthogonal transform unit 304 performs an inverse orthogonal transform on the orthogonal transform coefficient supplied from the inverse quantization unit 303 using a method corresponding to the orthogonal transform method of the orthogonal transform unit 204 in FIG. It is supplied to the arithmetic unit 305.
  • the operation unit 305 is supplied with the residual from the inverse orthogonal transform unit 304, and is also supplied with a predicted image from the intra prediction unit 312 or the motion prediction compensation unit 313 via the selection unit 314.
  • the operation unit 305 adds the residual from the inverse orthogonal transform unit 304 and the predicted image from the selection unit 314, generates a decoded image, and supplies the decoded image to the ILF 306.
  • $ IFL 306 stores the same merge information as ILF 211 (FIG. 61).
  • the ILF 306 is configured, for example, in the same manner as the class classification prediction filter 110 without a learning function (FIG. 56), and performs a filtering process as a class classification prediction process to perform a deblocking filter, an adaptive It functions as one of an offset filter, a bilateral filter, and an ALF, or two or more filters.
  • the ILF 306 sequentially selects pixels of the decoded image from the calculation unit 305 as pixels of interest.
  • the ILF 306 performs an initial class classification on the target pixel, and obtains an initial class of the target pixel. Further, the ILF 211 assigns the initial class of the target pixel to the merge class according to the merge pattern corresponding to the number of adopted merge classes included in the filter information supplied from the lossless decoding unit 302 among the merge patterns included in the merge information. Convert.
  • the ILF 306 acquires the tap coefficient of the merge class of the target pixel among the tap coefficients for each merge class included in the filter information supplied from the lossless decoding unit 302.
  • the ILF 306 selects a pixel in the vicinity of the pixel of interest from the decoded image as a prediction tap, and calculates a prediction equation for performing a product-sum operation between a tap coefficient of the class of the pixel of interest and a pixel of the decoded image as a prediction tap. Performs a filtering process as a prediction process to be applied to the image data, and generates and outputs a filtered image.
  • the filter image output from the ILF 306 is similar to the filter image output from the ILF 211 in FIG. 61, and is supplied to the rearrangement buffer 307 and the frame memory 310.
  • the rearrangement buffer 307 temporarily stores the filter image supplied from the ILF 306, rearranges the frame (picture) arrangement of the filter image from the encoding (decoding) order to the display order, and supplies the same to the D / A conversion unit 308. .
  • the D / A conversion unit 308 performs D / A conversion of the filter image supplied from the rearrangement buffer 307, and outputs the image to a display (not shown) for display.
  • the frame memory 310 temporarily stores the filter image supplied from the ILF 306. Further, the frame memory 310 uses the filter image as a reference image to be used for generating a prediction image at a predetermined timing or based on an external request from the intra prediction unit 312, the motion prediction compensation unit 313, or the like. To supply.
  • the selection unit 311 selects the supply destination of the reference image supplied from the frame memory 310.
  • the selection unit 311 supplies the reference image supplied from the frame memory 310 to the intra prediction unit 312.
  • the selection unit 311 supplies the reference image supplied from the frame memory 310 to the motion prediction compensation unit 313.
  • the intra prediction unit 312 According to the prediction mode included in the encoding information supplied from the lossless decoding unit 302, the intra prediction unit 312 outputs the data from the frame memory 310 via the selection unit 311 in the intra prediction mode used in the intra prediction unit 214 in FIG. Intra prediction is performed using the supplied reference image. Then, the intra prediction unit 312 supplies the prediction image obtained by the intra prediction to the selection unit 314.
  • the motion prediction / compensation unit 313 sends the selection unit 311 from the frame memory 310 to the inter prediction mode used in the motion prediction / compensation unit 215 in FIG. 61 according to the prediction mode included in the encoded information supplied from the lossless decoding unit 302. Inter prediction is performed using the reference image supplied through the inter-prediction. The inter prediction is performed using motion information and the like included in the encoded information supplied from the lossless decoding unit 302 as necessary.
  • the motion prediction compensation unit 313 supplies the prediction image obtained by the inter prediction to the selection unit 314.
  • the selection unit 314 selects a prediction image supplied from the intra prediction unit 312 or a prediction image supplied from the motion prediction compensation unit 313, and supplies the prediction image to the calculation unit 305.
  • the lossless decoding unit 302 corresponds to the parsing unit 171 of FIG. 58
  • the inverse quantization unit 303 to the arithmetic unit 305 corresponds to the decoding unit 172 of FIG. 58
  • the ILF 306 corresponds to the filter unit 173 of FIG. .
  • FIG. 65 is a flowchart for explaining an example of the decoding process of the decoding device 170 in FIG.
  • step S301 the accumulation buffer 301 temporarily stores the encoded bit stream transmitted from the encoding device 160 and appropriately supplies the encoded bit stream to the lossless decoding unit 302, and the process proceeds to step S302.
  • step S302 the lossless decoding unit 302 receives and decodes the coded bit stream supplied from the accumulation buffer 301, and dequantizes the quantized coefficient as coded data included in the decoding result of the coded bit stream.
  • the signal is supplied to the unit 303.
  • the lossless decoding unit 302 parses the filter information and the encoded information. Then, the lossless decoding unit 302 supplies necessary encoding information to the intra prediction unit 312, the motion prediction compensation unit 313, and other necessary blocks. Further, the lossless decoding unit 302 supplies the filter information to the ILF 306.
  • step S302 proceeds from step S302 to step S303, and the ILF 306 determines whether or not the lossless decoding unit 302 has supplied filter information including the number of adopted merge classes and the tap coefficient for each merge class of the number of adopted merge classes.
  • step S303 If it is determined in step S303 that the filter information has not been supplied, the process skips step S304 and proceeds to step S305.
  • step S303 If it is determined in step S303 that the filter information has been supplied, the process proceeds to step S304, where the ILF 306 merges the number of adopted merge classes and the number of adopted merge classes included in the filter information from the lossless decoding unit 302. Get the tap coefficient for each class. Further, the ILF 306 updates the number of adopted merge classes and the tap coefficient used in the class classification prediction process based on the number of adopted merge classes acquired from the filter information from the lossless decoding unit 302 and the tap coefficient for each merge class. .
  • step S304 proceeds from step S304 to step S305, where a predictive decoding process is performed, and the decoding process ends.
  • FIG. 66 is a flowchart illustrating an example of the predictive decoding process in step S305 in FIG.
  • step S311 the inverse quantization unit 303 inversely quantizes the quantized coefficient from the lossless decoding unit 302, and supplies the resulting orthogonal transform coefficient to the inverse orthogonal transform unit 304, and the process proceeds to step S312. move on.
  • step S312 the inverse orthogonal transform unit 304 performs an inverse orthogonal transform on the orthogonal transform coefficient from the inverse quantization unit 303, and supplies the resulting residual to the calculation unit 305, and the process proceeds to step S313. .
  • step S313 the intra prediction unit 312 or the motion prediction compensation unit 313 performs prediction using the reference image supplied from the frame memory 310 via the selection unit 311 and the encoded information supplied from the lossless decoding unit 302. An intra prediction process or an inter motion prediction process for generating an image is performed. Then, the intra prediction unit 312 or the motion prediction compensation unit 313 supplies the prediction image obtained by the intra prediction process or the inter motion prediction process to the selection unit 314, and the process proceeds from step S313 to step S314.
  • step S314 the selection unit 314 selects a prediction image supplied from the intra prediction unit 312 or the motion prediction compensation unit 313, supplies the prediction image to the calculation unit 305, and the process proceeds to step S315.
  • step S315 the arithmetic unit 305 generates a decoded image by adding the residual from the inverse orthogonal transform unit 304 and the predicted image from the selection unit 314. Then, the arithmetic unit 305 supplies the decoded image to the ILF 306, and the process proceeds from step S315 to step S316.
  • step S316 the same process as that of the class classification prediction filter 110 (FIG. 56) is performed.
  • the ILF 306 performs the same initial class classification as that of the ILF 211 on the target pixel of the decoded image from the arithmetic unit 305, and obtains the initial class of the target pixel. Further, the ILF 306 converts the initial class of the pixel of interest into a merge class according to the merge pattern corresponding to the number of adopted merge classes updated in step S304 of FIG. 65 among the merge patterns included in the merge information. The ILF 306 acquires the tap coefficient of the merge class of the pixel of interest from among the tap coefficients for each merge class updated in step S304 of FIG.
  • the ILF 306 performs a filtering process as a prediction process of applying a prediction formula including a tap coefficient of a merge class of the pixel of interest to the decoded image, and generates a filtered image.
  • the filter image is supplied from the ILF 306 to the rearrangement buffer 307 and the frame memory 310.
  • step S317 the reordering buffer 307 temporarily stores the filter image supplied from the ILF 306. Further, the rearrangement buffer 307 rearranges the stored filter images in the display order and supplies the rearranged filter images to the D / A converter 308, and the process proceeds from step S317 to step S318.
  • step S318 the D / A conversion unit 308 performs D / A conversion on the filter image from the reordering buffer 307, and the process proceeds to step S319.
  • the filter image after the D / A conversion is output and displayed on a display (not shown).
  • step S319 the frame memory 310 stores the filter image supplied from the ILF 306, and the decoding process ends.
  • the filter image stored in the frame memory 310 is used as a reference image from which a predicted image is generated in the intra prediction process or the inter motion prediction process in step S313.
  • the merge pattern is selected by selecting the merge pattern.
  • One merge pattern is selected (set and set) as a merge pattern corresponding to a predetermined number of merge classes.
  • the merge pattern is referred to as a candidate pattern, and a merge pattern corresponding to the predetermined number of merge classes among the plurality of candidate patterns. Is referred to as a selected pattern.
  • a merge pattern used for merging the classes with less overhead than when transmitting the merge pattern. Is adopted.
  • the merge pattern (Na, Nb, Nc) is obtained by the combination (Na, Nb, Nc) of the number of subclasses Na, Nb, Nc of the gradient strength ratio subclass, the direction subclass, and the activity subclass after the subclass merge.
  • the specifying method is also called a specifying method based on the number of subclasses.
  • each of a plurality of merge patterns having the same number of merge classes can be specified. Therefore, compared to the case where a merge pattern is set for each number of merge classes, an adopted merge pattern can be determined from more merge patterns. As a result, the initial class can be merged by the merge pattern in which the class classification more suitable for the original image is performed, and the encoding efficiency and the image quality of the decoded image can be improved.
  • the gradient strength ratio subclass after the subclass merge, the direction subclass, and the number of subclasses Na, Nb, Nc of the activity subclass are 1 to 3 subclasses, 1 or 2 subclasses, and 1 to 5 subclasses, respectively.
  • the data amount is smaller than the merge pattern of GALF (FIG. 9), which is a series of 25 numbers. Therefore, the adopted merge pattern (Na, Nb, Nc) is specified by the combination (Na, Nb, Nc) of the subclass numbers Na, Nb, Nc of the gradient strength ratio subclass, direction subclass, and activity subclass after the subclass merge. According to the specifying method based on the number of subclasses, the overhead can be reduced and the coding efficiency can be improved as compared with the case of GALF.
  • the subclass merge from which the merge pattern (3, # 2, # 5) is obtained is as shown in FIG.
  • FIG. 67 shows a merge pattern (3, 1, 5) corresponding to a combination (3, 1, 5) in which the number of subclasses of the gradient strength ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 3, 1, 5. ) And a subclass merge from which the merge pattern (3, 1, 5) is obtained.
  • the merge pattern (2, 2, 5) corresponding to the combination (2, 2, 5) where the number of subclasses of the gradient strength ratio subclass, direction subclass, and activity subclass after the subclass merge is 2, 22, 5, and
  • the subclass merge from which the merge pattern (2, # 2, # 5) is obtained is as shown in FIG.
  • the merge pattern (2, 1, 5) corresponding to the combination (2, 1, 5) where the number of subclasses of the gradient intensity ratio subclass, direction subclass, and activity subclass after the subclass merge is 2, 1, 5, and
  • the subclass merge from which the merge pattern (2, # 1, # 5) is obtained is as shown in FIG.
  • FIG. 68 shows a merge pattern (1, 2, 5) corresponding to a combination (1, 2, 5) in which the number of subclasses of the gradient strength ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 1, 2, 5. ) And a subclass merge from which the merge pattern (1, 2, 5) is obtained.
  • the gradient intensity ratio subclass is merged into one subclass (N / A class), and D0 / This corresponds to the case where the direction subclass classification classified into the D1 class or the H / V class is performed.
  • the gradient intensity ratio subclass is merged into one subclass and the directional subclass classification is performed which is classified into the D0 / D1 class or the H / V class, the directional subclass classification is invalid as described in FIG. A merge pattern that is a class classification performed by a proper direction subclass classification is also invalid. In the present technology, an invalid merge pattern is not used.
  • the subclass merge from which the merge pattern (1, 1, 5) is obtained is as shown in FIG.
  • the subclass merge from which the merge pattern (3, # 2, # 4) is obtained is as shown in FIG.
  • FIG. 69 shows a merge pattern (3, 1, 4) corresponding to a combination (3, 1, 4) in which the number of subclasses of the gradient intensity ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 3, 1, 4. ) And a subclass merge from which a merge pattern (3, 1, 4) is obtained.
  • the merge pattern (3, 1, 4) merges the gradient strength ratio subclass into three subclasses of non-class, weak class, and strong class, and merges the direction subclass into one subclass of N / A class.
  • the activity subclass, the activity subclass 0 corresponding to the index class_idx of 0 and 1, the activity subclass 1 corresponding to the index class_idx of 2 to 6, the activity subclass 2 corresponding to the index class_idx of 7 to 14, and the index class_idx of 15 Can be obtained by subclass merging into 4 subclasses of activity subclass 3 corresponding to.
  • FIG. 70 shows a merge pattern (2, 1, 4) corresponding to a combination (2, 1, 4) in which the number of subclasses of the gradient strength ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 2, 1, 4. ) And a subclass merge from which the merge pattern (2, 1, 4) is obtained.
  • the merge pattern (2, 1, 4) merges the gradient intensity ratio subclass into two subclasses, a nonclass and a high class, merges the direction subclass into one subclass of the N / A class, and merges the activity subclass.
  • the activity subclass 0 corresponding to the index class_idx of 0 and 1
  • the activity subclass 1 corresponding to the index class_idx of 2 to 6
  • the activity subclass 2 corresponding to the index class_idx of 7 to 14, and the index class_idx of 15 It can be obtained by subclass merging into 4 subclasses of activity subclass 3.
  • FIG. 71 shows a merge pattern (1, 2, 4) corresponding to a combination (1, 2, 4) in which the number of subclasses of the gradient intensity ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 1, 2, 4. ) And a subclass merge from which the merge pattern (1, 2, 4) is obtained.
  • the gradient intensity ratio subclass is merged into one subclass (N / A class), and D0 / This corresponds to the case where the direction subclass classification classified into the D1 class or the H / V class is performed.
  • the gradient intensity ratio subclass is merged into one subclass and the directional subclass classification is performed which is classified into the D0 / D1 class or the H / V class, the directional subclass classification is invalid as described in FIG. A merge pattern that is a class classification performed by a proper direction subclass classification is also invalid. In the present technology, an invalid merge pattern is not used.
  • the subclass merge from which the merge pattern (1, 1, 4) is obtained is as shown in FIG.
  • FIG. 72 shows a merge pattern (3, 2, 3) corresponding to a combination (3, 2, 3) in which the number of subclasses of the gradient intensity ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 3, 2, 3. ) And a subclass merge from which a merge pattern (3, # 2, # 3) is obtained.
  • the merge pattern (3, 2, 3) subclass merges the gradient strength ratio subclass into three subclasses, non-class, weak class, and strong class, and the direction subclass, D0 / D1 class and H / V class 2
  • the subclass is merged into the subclass, and the activity subclass is defined as the activity subclass 0 corresponding to the index class_idx of 0 to 6, the activity subclass 1 corresponding to the index class_idx of 7 to 14, and the activity subclass 2 corresponding to the index class_idx of 15. It can be obtained by subclass merging into a subclass.
  • FIG. 73 shows a merge pattern (3, 1, 3) corresponding to a combination (3, 1, 3) in which the number of subclasses of the gradient intensity ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 3, 1, 3. ) And a subclass merge from which the merge pattern (3, 1, 3) is obtained.
  • the merge pattern (3, 1, 3) merges the gradient strength ratio subclass into three subclasses of non-class, weak class, and strong class, and merges the direction subclass into one subclass of N / A class.
  • the activity subclasses are merged into three subclasses: activity subclass 0 corresponding to index class_idx of 0 to 6, activity subclass 1 corresponding to index class_idx of 7 to 14, and activity subclass 2 corresponding to index class_idx of 15. Can be obtained.
  • the subclass merge from which the merge pattern (2, # 2, # 3) is obtained is as shown in FIG.
  • the subclass merge from which the merge pattern (2, # 1, # 3) is obtained is as shown in FIG.
  • FIG. 74 shows a merge pattern (1, 2, 3) corresponding to a combination (1, 2, 3) in which the number of subclasses of the gradient intensity ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 1, 2, 3. ) And a subclass merge from which the merge pattern (1, 2, 3) is obtained.
  • the gradient intensity ratio subclass is merged into one subclass (N / A class), and D0 / This corresponds to the case where the direction subclass classification classified into the D1 class or the H / V class is performed.
  • the gradient intensity ratio subclass is merged into one subclass and the directional subclass classification is performed which is classified into the D0 / D1 class or the H / V class, the directional subclass classification is invalid as described in FIG. A merge pattern that is a class classification performed by a proper direction subclass classification is also invalid. In the present technology, an invalid merge pattern is not used.
  • the merge pattern (1, 1, 3) corresponding to the combination (1, 1, 3) where the number of subclasses of the gradient strength ratio subclass, direction subclass, and activity subclass after the subclass merge is 1, 1, 3, and
  • the subclass merge from which the merge pattern (1, 1, 3) is obtained is as shown in FIG.
  • FIG. 75 shows the merge pattern (3, 2, 2) corresponding to the combination (3, 2, 2) in which the number of subclasses of the gradient intensity ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 3, 2, 2. ) And a subclass merge from which a merge pattern (3, 2, 2) is obtained.
  • the merge pattern (3, 2, 2) merges the gradient strength ratio subclass into three subclasses of non-class, weak class, and strong class, and combines the direction subclass with two classes of D0 / D1 class and H / V class.
  • a subclass can be merged into a subclass, and the activity subclass can be obtained by subclass merging into two subclasses, activity subclass 0 corresponding to index class_idx of 0 to 14 and activity subclass 1 corresponding to index class_idx of 15.
  • FIG. 76 shows a merge pattern (3, 1, 2) corresponding to a combination (3, 1, 2) in which the number of subclasses of the gradient intensity ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 3, 1, 2. ) And a subclass merge from which the merge pattern (3, 1, 2) is obtained.
  • the merge pattern (3, 1, 2) merges the gradient strength ratio subclass into three subclasses of non-class, weak class, and strong class, and merges the direction subclass into one subclass of N / A class.
  • the activity subclass can be obtained by subclass merging into two subclasses, activity subclass 0 corresponding to index class_idx of 0 to 14 and activity subclass 1 corresponding to index index_idx of 15.
  • the merge pattern (2, 2, 2) corresponding to the combination (2, 2, 2) where the number of subclasses of the gradient strength ratio subclass, direction subclass, and activity subclass after the subclass merge is 2, 22, 2, and
  • the subclass merge from which the merge pattern (2, # 2, # 2) is obtained is as shown in FIG.
  • FIG. 77 shows a merge pattern (2, 1, 2) corresponding to a combination (2, 1, 2) in which the number of subclasses of the gradient strength ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 2, 1, 2. ) And a subclass merge from which the merge pattern (2, 1, 2) is obtained.
  • the merge pattern (2, 1, 2) merges the gradient intensity ratio subclass into two subclasses, non-class and high class, merges the direction subclass into one subclass of N / A class, and merges the activity subclass. Can be obtained by subclass merging into two subclasses, activity subclass 0 corresponding to index class_idx of 0 to 14 and activity subclass 1 corresponding to index class_idx of 15.
  • FIG. 78 shows merge patterns (1, 2, 2) corresponding to combinations (1, 2, 2) in which the number of subclasses of the gradient strength ratio subclass, the direction subclass, and the activity subclass after the subclass merge are 1, 2, 2. ) And a subclass merge from which the merge pattern (1, 2, 2) is obtained.
  • the gradient intensity ratio subclass is merged into one subclass (N / A class), and D0 / This corresponds to the case where the direction subclass classification classified into the D1 class or the H / V class is performed.
  • the gradient intensity ratio subclass is merged into one subclass and the directional subclass classification is performed which is classified into the D0 / D1 class or the H / V class, the directional subclass classification is invalid as described in FIG. A merge pattern that is a class classification performed by a proper direction subclass classification is also invalid. In the present technology, an invalid merge pattern is not used.
  • the merge pattern (1, 1, 2) corresponding to the combination (1, 1, 2) where the number of subclasses of the gradient strength ratio subclass, direction subclass, and activity subclass after the subclass merge is 1, 1, 2, and
  • the subclass merge from which the merge pattern (1, 1, 2) is obtained is as shown in FIG.
  • FIG. 79 shows a merge pattern (3, 2, 1) corresponding to a combination (3, 2, 1) in which the number of subclasses of the gradient intensity ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 3, 2, 1. ) And a subclass merge from which a merge pattern (3, # 2, # 1) is obtained.
  • the gradient strength ratio subclass is subclass-merged into three subclasses of a non-class, a weak class, and a strong class, and the direction subclass is divided into two classes, a D0 / D1 class and an H / V class. It can be obtained by merging a subclass into a subclass and merging an activity subclass into one subclass of the N / A class.
  • FIG. 80 shows a merge pattern (3, 1, 1) corresponding to a combination (3, 1, 1) in which the number of subclasses of the gradient strength ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 3, 1, 1. ) And a subclass merge from which the merge pattern (3, 1, 1) is obtained.
  • the merge pattern (3, 1, 1) merges the gradient strength ratio subclass into three subclasses of non-class, weak class, and strong class, and merges the direction subclass into one subclass of N / A class.
  • the activity subclass can be obtained by subclass merging into one subclass of the N / A class.
  • FIG. 81 shows merge patterns (2, 2, 1) corresponding to a combination (2, 2, 1) in which the number of subclasses of each of the gradient strength ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 2, 2,) 1. ) And a subclass merge from which the merge pattern (2, 2, 1) is obtained.
  • the merge pattern (2, 2, 1) merges the gradient intensity ratio subclass into two subclasses, a nonclass and a high class, and subclasses the direction subclass into two subclasses, a D0 / D1 class and an H / V class. It can be obtained by merging and subclassing the activity subclass into one subclass of the N / A class.
  • FIG. 82 shows a merge pattern (2, 1, 1) corresponding to a combination (2, 1, 1) in which the number of subclasses of the gradient strength ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 2, 1, 1. ) And a subclass merge from which the merge pattern (2, 1, 1) is obtained.
  • the merge pattern (2, 1, 1) merges the gradient strength ratio subclass into two subclasses, a nonclass and a high class, merges the direction subclass into one subclass of the N / A class, and merges the activity subclass. Can be obtained by subclass merging into one subclass of the N / A class.
  • FIG. 83 shows a merge pattern (1, 2, 1) corresponding to a combination (1, 2, 1) in which the number of subclasses of the gradient strength ratio subclass, the direction subclass, and the activity subclass after the subclass merge is 1, 2, 1. ) And a subclass merge from which the merge pattern (1, 2, 1) is obtained.
  • the gradient intensity ratio subclass is merged into one subclass (N / A class) and D0 / This corresponds to the case where the direction subclass classification classified into the D1 class or the H / V class is performed.
  • the gradient intensity ratio subclass is merged into one subclass and the directional subclass classification is performed which is classified into the D0 / D1 class or the H / V class, the directional subclass classification is invalid as described in FIG. A merge pattern that is a class classification performed by a proper direction subclass classification is also invalid. In the present technology, an invalid merge pattern is not used.
  • the merge pattern (1, 1, 1) corresponding to the combination (1, 1, 1) where the number of subclasses of the gradient strength ratio subclass, direction subclass, and activity subclass after the subclass merge is 1, 1, 1 and
  • the subclass merge from which the merge pattern (1, 1, 1) is obtained is as shown in FIG.
  • FIG. 84 is a diagram illustrating an example of syntax for transmitting a combination of the number of subclasses.
  • the adopted merge pattern (Na, Nb, Nc) is specified by the identification method based on the number of subclasses
  • a combination of the number of subclasses that specifies the adopted merge pattern (Na, Nb, Nc) (hereinafter also referred to as an adopted combination) (Na, Nb, Nc) must be transmitted from the encoding device to the decoding device.
  • alf_dirRatio_minus1, alf_dir_minus1, and alf_act_var_minus1 represent the gradient strength ratio subclass, direction subclass, and activity subclass after the subclass merge in which the adopted merge pattern is obtained, and the number of subclasses Na, Nb, Nc.
  • alf_dirRatio_minus1 the number of subclasses Na-1 of the gradient intensity ratio subclass after the subclass merge that obtains the adopted merge pattern is set.
  • alf_dir_minus1 the number of subclasses Nb-1 of the direction subclass after the subclass merge that can obtain the adopted merge pattern is set.
  • alf_act_var_minus1 the number of subclasses Nc-1 of the activity subclass after the subclass merge that can obtain the adoption merge pattern is set.
  • the number of subclasses of the gradient intensity ratio subclass is one of one to three
  • the number of subclasses of the direction subclass is one or two
  • the number of subclasses of the activity subclass is one of one to five. Therefore, 2-bit, 1-bit, and 3-bit (or more) variables are used as alf_dirRatio_minus1, alf_dir_minus1, and alf_act_var_minus1, which represent the number of subclasses of the gradient intensity ratio subclass, the direction subclass, and the activity subclass, respectively.
  • alf_dir_minus1 representing the number Nb of subclasses of the direction subclass is transmitted only when alf_dirRatio_minus1 representing the number Na of subclasses of the gradient intensity ratio subclass is greater than zero.
  • the direction subclass classification classified into the D0 / D1 class or the H / V class and A merge pattern that is a class classification performed by such a direction subclass classification, that is, a merge pattern corresponding to a combination of the number of subclasses in which the number of subclasses Nb of the gradient strength ratio subclass is 1 and the number of subclasses Nb of the direction subclass is 2 is invalid. And is not used.
  • the combination of the number of subclasses to be adopted is that the number of subclasses of the gradient intensity ratio subclass (the number of subclasses of the subclass classification of the gradient intensity ratio) is 1 and the number of subclasses of the direction subclass (the subclass of the subclass of the direction subclass classification) Number) does not include a combination of two or more.
  • the number of subclasses Na of the gradient intensity ratio subclass is 1, the number of subclasses Nb of the direction subclass is determined to be 1, so it is not necessary to transmit the number of subclasses Nb of the direction subclass. . Then, when it is necessary to transmit the number of subclasses Nb of the direction subclass, it means that the number of subclasses Na of the gradient intensity ratio subclass is 2 or more.
  • the adopted combination transmitted by the syntax in FIG. 84 includes the number of subclasses Nb in the direction subclass when the number of subclasses Na in the gradient intensity ratio subclass is 2 or more.
  • FIG. 85 is a block diagram illustrating a configuration example of a classification prediction filter to which the present technology is applied.
  • FIG. 85 illustrates a configuration example of the class classification prediction filter 410 that specifies an adopted merge pattern by a specifying method based on the number of subclasses.
  • portions corresponding to the class classification prediction filter 110 in FIG. 56 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate.
  • the classification prediction filter 410 includes a classification unit 111, a tap coefficient acquisition unit 113, a prediction unit 114, and a merge conversion unit 412.
  • the classification prediction filter 410 is common to the classification prediction filter 110 in that the classification prediction filter 410 includes the classification unit 111, the tap coefficient acquisition unit 113, and the prediction unit 114. However, the class classification prediction filter 410 differs from the class classification prediction filter 110 in having a merge conversion unit 412 instead of the merge conversion unit 112.
  • the merge conversion unit 412 sets the initial class of the pixel of interest from the classifying unit 111 to the number of subclasses of the gradient intensity ratio subclass, the direction subclass, and the activity subclass after the subclass merge (the gradient intensity ratio, direction, and activity sum, respectively). Is converted into a merge class according to a merge pattern (hereinafter, also simply referred to as a merge pattern determined for each combination of the number of subclasses) determined for each combination of the subclasses of the subclass classification. That is, for example, the merge conversion unit 412 determines the initial class of the target pixel according to the merge pattern corresponding to the adopted combination among the 25 (effective) merge patterns determined for each combination of the number of subclasses described with reference to FIG. , Convert to merge class. The merge conversion unit 412 supplies the merge class of the target pixel to the tap coefficient acquisition unit 113.
  • a merge pattern hereinafter, also simply referred to as a merge pattern determined for each combination of the number of subclasses
  • the tap coefficient acquisition unit 113 selects the tap coefficient of the merge class of the target pixel from the merge conversion unit 412 from the tap coefficients for each merge class, and supplies the selected tap coefficient to the prediction unit 114. Then, the prediction unit 114 performs a filtering process as a prediction process of applying a prediction equation using a tap coefficient of a merge class of the pixel of interest from the tap coefficient acquisition unit 113 to the target image, and generates the filter image by the filtering process. A filtered image is output.
  • the adopted combinations and the tap coefficients for each merge class can be supplied to the class classification prediction filter 410 from outside.
  • the classifying / prediction filter 410 can incorporate a learning unit 421 that performs tap coefficient learning.
  • the classification prediction filter 410 having the learning unit 421 can be said to be the classification prediction filter 410 with a learning function.
  • the learning unit 421 can use the teacher image and the student image to determine the tap coefficient for each merge class, and store the tap coefficient in the tap coefficient acquisition unit 113. Further, the learning unit 421 can determine an adopted combination and supply it to the merge conversion unit 412.
  • a decoded image obtained by employing an original image to be encoded as a teacher image and encoding and locally decoding the original image as a student image can be adopted.
  • the learning unit 421 performs the same class classification as that of the class classification unit 111 using the decoded image as the student image, and for each initial class obtained by the class classification, a prediction expression composed of a tap coefficient and a prediction tap. Tap coefficient learning is performed to find a tap coefficient that statistically minimizes the prediction error of the prediction value of the teacher image obtained by using the least square method.
  • the learning unit 421 performs the same process as the adopted merge pattern number determination process (FIG. 8) by using each merge pattern corresponding to each of a plurality of combinations of the number of subclasses as a merge pattern determined for each combination of the number of subclasses.
  • the number of subclasses that specify the merge pattern that minimizes the cost for example, the cost dist + lambda ⁇ coeffBit obtained in step S67 in FIG. 8)
  • the combination is determined as the adoption combination.
  • the learning unit 421 performs the merge pattern determination process (see FIG. 8) in step S63 before performing the process of step S64, which is a filter process for obtaining a cost for determining an adopted combination in the adopted merge pattern number determination process (FIG. 8).
  • step S64 is a filter process for obtaining a cost for determining an adopted combination in the adopted merge pattern number determination process (FIG. 8).
  • the learning unit 421 supplies the adopted combination to the merge conversion unit 412, and supplies the tap coefficient for each merge class obtained according to the merge pattern corresponding to the adopted combination to the tap coefficient acquisition unit 113.
  • the encoding device and the decoding device to which the present technology is applied share that the merging of the initial class is performed by the subclass merging of FIGS. Then, the encoding device determines an adopted combination from a plurality of combinations of the number of subclasses that specify the merge pattern obtained by the subclass merge, and transmits the combination to the decoding device. The decoding device specifies the merge pattern from the combination adopted from the encoding device. Then, the decoding device performs the initial class classification, and converts the initial class obtained by the initial class classification into a merge class according to the merge pattern specified from the adopted combination (the merge pattern corresponding to the adopted combination).
  • FIG. 86 is a flowchart illustrating an example of the classification prediction process performed by the classification prediction filter 410 of FIG.
  • step S411 the class classification unit 111 sequentially selects pixels to be selected as target pixels of the decoded image as the target image as target pixels, and the process proceeds to step S412.
  • step S412 the class classification unit 111 performs an initial class classification of the target pixel, and obtains an initial class of the target pixel.
  • the class classification unit 111 supplies the initial class of the pixel of interest to the merge conversion unit 412, and the process proceeds from step S412 to step S413.
  • step S413 the merge conversion unit 412 converts the initial class of the target pixel from the class classification unit 111 into a merge class according to the merge pattern corresponding to the adopted combination.
  • the merge conversion unit 412 supplies the merge class of the pixel of interest to the tap coefficient acquisition unit 113, and the process proceeds from step S413 to step S414.
  • step S414 the tap coefficient acquisition unit 113 acquires the tap coefficient of the merge class of the target pixel from the merge conversion unit 412 from among the tap coefficients for each merge class, and the process proceeds to step S415.
  • step S415 the prediction unit 114 performs a filtering process as a prediction process of applying a prediction expression composed of tap coefficients of a merge class of the pixel of interest from the tap coefficient acquisition unit 113 to the decoded image.
  • the prediction unit 114 selects, from the decoded image, a pixel to be a prediction tap of the target pixel, and calculates a primary prediction expression configured using the prediction tap and a tap coefficient of a merge class of the target pixel. , The predicted value of (the pixel value of) the pixel of the original image with respect to the pixel of interest. Then, the prediction unit 114 generates an image having the prediction value as a pixel value, outputs the image as a filter image, and ends the class classification prediction process.
  • FIG. 87 is a block diagram illustrating an outline of an embodiment of an image processing system to which the present technology is applied.
  • the image processing system has an encoding device 460 and a decoding device 470.
  • the encoding device 460 includes an encoding unit 161, a local decoding unit 162, and a filter unit 463.
  • the encoding device 460 is common to the encoding device 160 in FIG. 58 in having the encoding unit 161 and the local decoding unit 162, and is different in that the encoding device 460 has a filter unit 463 instead of the filter unit 163. It differs from the device 160.
  • the filter unit 463 is configured, for example, in the same manner as the class classification prediction filter 410 with a learning function (FIG. 85), and includes a class classification unit 164 and a merge conversion unit 465. Therefore, filter section 463 is common to filter section 163 in FIG. 58 in having class classification section 164, and differs from filter section 163 in having merge conversion section 465 instead of merge conversion section 165.
  • the filter unit 463 performs tap coefficient learning using the decoded image from the local decoding unit 162 and the original image for the decoded image as a student image and a teacher image, and obtains a tap coefficient for each class.
  • the filter unit 463 performs the same process as the adopted merge pattern number determination process (FIG. 8) by using (a plurality of) merge patterns determined for each combination of the number of subclasses obtained by the subclass merge. Of the combinations of the number of subclasses obtained by the above, the combination of the number of subclasses that specifies the merge pattern that minimizes the cost is determined as the adopted combination.
  • the filter unit 463 uses the normal equations (the X matrix and the Y vector) obtained by the tap coefficient learning to perform steps S36 and S37 of the merge pattern determination process (FIG. 5). By performing similar processing, tap coefficients for each merge class obtained by the merge pattern corresponding to the adopted combination are obtained.
  • the filter unit 463 performs, for example, GALF class classification or the like as an initial class classification performed by subclass classification of a plurality of feature amounts using the decoded image from the local decoding unit 162 in the class classification unit 164. And the initial class of the pixel of interest. Further, the filter unit 463 causes the merge conversion unit 465 to convert the initial class of the target pixel into a merge class according to the merge pattern corresponding to the adopted combination. Then, the filter unit 463 performs a filter process as a prediction process of applying, to the decoded image, a prediction expression for performing a product-sum operation of the tap coefficient of the merge class of the target pixel obtained by the conversion of the merge conversion unit 465 and the pixel of the decoded image. I do.
  • the filter unit 463 supplies the filter image obtained by the filter processing to the encoding unit 161 and the local decoding unit 162. Further, the filter unit 463 supplies the encoding unit 161 as the filter information, with the adopted combination and the tap coefficient for each merge class obtained by the conversion of the initial class according to the merge pattern corresponding to the adopted combination.
  • the cost of the merge pattern obtained by subclass merging (25 effective merge patterns among the merge patterns corresponding to the 30 combinations of the number of subclasses in FIG. 36) is described.
  • the combination of the number of subclasses that specifies the merge pattern that minimizes is determined to be the adopted combination, but the adopted combination is the number of subclasses that specify the specific merge pattern among the merge patterns obtained in advance by subclass merging. Can be determined in advance as the adopted combination. In this case, it is not necessary to obtain a cost to determine an adopted combination, so that the processing amount of the encoding device 460 can be reduced.
  • determining the adopted combination in advance is effective, for example, particularly when the performance of the encoding device 460 is not high.
  • the decoding device 470 includes a parsing unit 171, a decoding unit 172, and a filter unit 473. Therefore, decoding apparatus 470 is common to decoding apparatus 170 in FIG. 58 in having parsing section 171 and decoding section 172, and is different from decoding apparatus 170 in having filtering section 473 instead of filtering section 173. .
  • the filter unit 473 is configured in the same manner as, for example, the class classification prediction filter 410 without the learning function (FIG. 85), and includes the class classification unit 174 and the merge conversion unit 475. Therefore, filter section 473 is common to filter section 173 of FIG. 58 in having class classification section 174, and differs from filter section 173 in having merge conversion section 475 instead of merge conversion section 175.
  • the filter unit 473 performs the same filter processing as the filter unit 463 on the decoded image from the decoding unit 172, generates a filtered image, and supplies the filtered image to the decoding unit 172.
  • the filter unit 473 performs the same initial class classification as the class classification unit 164 in the class classification unit 174 using the decoded image from the decoding unit 172, and obtains the initial class of the pixel of interest. Further, the filter unit 473 merges the initial class of the target pixel with the subclass of the subclass classification in the merge conversion unit 475 according to the merge pattern corresponding to the adopted combination included in the filter information from the parse unit 171. Is converted to a merged class.
  • the filter unit 473 performs a filter process as a prediction process of applying a product-sum operation of a tap coefficient of a merge class of the pixel of interest obtained by the conversion of the merge conversion unit 475 and a pixel of the decoded image to the decoded image.
  • the tap coefficient of the merge class of the pixel of interest used for the filter processing is obtained from the tap coefficient for each merge class included in the filter information from the parse unit 171.
  • the filter unit 473 supplies the filtered image obtained by the filtering process to the decoding unit 172, and outputs the filtered image as a final decoded image obtained by decoding the original image.
  • FIG. 88 is a flowchart for explaining the outline of the encoding process of the encoding device 460 in FIG.
  • the processing according to the flowchart in FIG. 88 is performed, for example, on a frame (picture) basis.
  • steps S461 to S463 the same processes as those in steps S161 to S163 in FIG. 59 are performed. Then, after the tap coefficient for each initial class is obtained in step S463, the process proceeds to step S464.
  • step S464 the filter unit 463 sets a plurality of combinations of the number of subclasses specifying the merge pattern obtained by the subclass merge (for example, a combination of the number of subclasses specifying each of the 25 valid merge patterns described in FIG. 36).
  • the initial class is merged according to the merge pattern corresponding to the combination of the number of subclasses, and the normal equation (X matrix and Y vector of) obtained by tap coefficient learning for obtaining the tap coefficient for each initial class is used.
  • the tap coefficients for each merge class obtained by merging the initial classes according to the merge pattern corresponding to the combination are obtained as described in steps S36 and S37 in FIG.
  • filter section 463 determines the combination of the number of subclasses that specifies the merge pattern with the minimum cost as the adoption combination using the tap coefficient for each merge class, and the process proceeds from step S464 to step S465.
  • step S465 the class classification unit 164 of the filter unit 463 performs initial class classification of the target pixel of the decoded image from the local decoding unit 162, and the process proceeds to step S466.
  • step S466 the merge conversion unit 465 of the filter unit 463 converts the initial class of the pixel of interest obtained by the class classification of the class classification unit 164 into a merge class according to the merge pattern corresponding to the adopted combination. Proceed to S467.
  • step S467 the filter unit 463 calculates a prediction expression for performing a product-sum operation between the tap coefficient of the merge class of the target pixel and the pixel of the decoded image among the tap coefficients for each merge class obtained in step S464.
  • a filter process is performed as a prediction process to be applied to the image data to generate a filtered image.
  • the filter image is supplied from the filter unit 463 to the encoding unit 161 and the local decoding unit 162.
  • the filter image supplied from the filter unit 463 to the encoding unit 161 and the local decoding unit 162 is used in the processing of steps S461 and S462 performed for the next frame.
  • the filter unit 463 supplies the adopted combination and the tap coefficient for each merge class to the encoding unit 161 as filter information.
  • step S467 the encoding unit 161 determines the combination of the encoded data obtained in step S461, the adoption combination as the filter information obtained in the filter unit 463, and each merge class.
  • a coded bit stream including the tap coefficients is generated and transmitted.
  • FIG. 89 is a flowchart for explaining the outline of the decoding process of the decoding device 470 in FIG.
  • the processing according to the flowchart in FIG. 89 is performed in frame units, for example, as in the encoding processing in FIG. 88.
  • step S481 the parsing unit 171 (FIG. 87) receives the encoded bit stream transmitted from the encoding device 460, adopts the combination adopted as the filter information included in the encoded bit stream, and The tap coefficients are parsed and supplied to the filter unit 473. Further, the parsing unit 171 supplies the encoded data included in the encoded bit stream to the decoding unit 172, and the process proceeds from step S481 to step S482.
  • step S482 the decoding unit 172 decodes the encoded data from the parsing unit 171 using the filter image from the filter unit 473, and supplies the decoded image obtained as a result to the filter unit 473, and Proceeds to step S483.
  • step S483 the class classification unit 174 of the filter unit 473 performs initial class classification on the target pixel of the decoded image from the decoding unit 172, and the process proceeds to step S484.
  • step S484 the merge conversion unit 475 of the filter unit 473 converts the initial class of the pixel of interest obtained by the class classification of the class classification unit 174 into a merge class according to the merge pattern corresponding to the combination adopted from the parse unit 171. , The process proceeds to step S485.
  • step S485 the filter unit 473 performs a filtering process as a class classification prediction process on the decoded image from the decoding unit 172 using the tap coefficient for each merge class from the parsing unit 171 to generate a filtered image.
  • the filter image is supplied from the filter unit 473 to the decoding unit 172, and is output as a final decoded image obtained by decoding the original image.
  • the filter image supplied from the filter unit 473 to the decoding unit 172 is used in the process of step S482 performed on the next frame of the decoded image.
  • a method of signaling a merge pattern (adopted merge pattern) for converting an initial class into a merge class a method of transmitting an embedded combination included in an encoded bit stream is adopted, but an adopted merge pattern is signaled.
  • a method as in the case of GALF, a method of transmitting an adopted merge pattern by including an adopted merge pattern in an encoded bit stream together with or instead of the adopted combination can be adopted.
  • transmitting the adopted combination can reduce the overhead compared to transmitting the adopted merge pattern.
  • the same syntax as the GALF class classification can be adopted.
  • FIG. 90 is a block diagram illustrating a detailed configuration example of the encoding device 460 of FIG.
  • the encoding device 460 includes an A / D conversion unit 201, a rearrangement buffer 202, an operation unit 203, an orthogonal transformation unit 204, a quantization unit 205, a lossless encoding unit 206, and a storage buffer 207. Further, the encoding device 460 includes an inverse quantization unit 208, an inverse orthogonal transform unit 209, an operation unit 210, a frame memory 212, a selection unit 213, an intra prediction unit 214, a motion prediction compensation unit 215, a predicted image selection unit 216, a rate It has a control unit 217 and an ILF 511.
  • encoding apparatus 460 is common to encoding apparatus 160 in FIG. 61 in having A / D conversion section 201 through arithmetic section 210 and frame memory 212 through rate control section 217. However, the encoding device 460 is different from the encoding device 160 in having an ILF 511 instead of the ILF 211.
  • the ILF 511 is configured, for example, in the same manner as the class classification prediction filter 410 with a learning function (FIG. 85), and performs a filtering process as a class classification prediction process to perform a deblocking filter, an adaptive offset filter, a bilateral filter, and an ALF. , Or as two or more filters.
  • the ILF 511 functions as two or more filters among the deblocking filter, the adaptive offset filter, the bilateral filter, and the ALF, the arrangement order of the two or more filters is arbitrary.
  • the ILF 511 is supplied with the decoded image from the arithmetic unit 210 and the original image for the decoded image from the rearrangement buffer 202.
  • the ILF 511 performs tap coefficient learning using, for example, a decoded image from the arithmetic unit 210 and an original image from the rearrangement buffer 202 as a student image and a teacher image, respectively, and obtains a tap coefficient for each initial class.
  • initial class classification is performed using a decoded image as a student image, and for each initial class obtained by the initial class classification, a teacher obtained by a prediction formula composed of a tap coefficient and a prediction tap is used.
  • a tap coefficient that statistically minimizes a prediction error of a prediction value of an original image as an image is obtained by a least square method.
  • the ILF 511 uses each of a plurality of merge patterns (for example, a combination of the number of subclasses that specify each of the 25 valid merge patterns described with reference to FIG. 36) obtained by subclass merge to determine the number of adopted merge patterns (FIG. 8). ), The merge pattern that minimizes the cost (for example, the cost dist + lambda ⁇ coeffBit obtained in step S67 in FIG. 8) among the plurality of merge patterns obtained by the subclass merge is specified. The combination of the number of subclasses to be performed is determined as the adopted combination.
  • step S63 which is a filter process for obtaining a cost for determining an adopted combination in the adopted merge pattern number determination process (FIG. 8)
  • a merge pattern determination process (FIG. 5) is performed.
  • steps S36 and S37 a plurality of merge patterns obtained by subclass merging using a normal equation (X matrix and Y vector of) obtained when tap coefficients for each initial class are obtained by tap coefficient learning. For each, a tap coefficient for each merge class is determined.
  • the ILF 511 supplies the adoption combination and the tap coefficient for each merge class obtained by the conversion of the initial class according to the merge pattern corresponding to the adoption combination to the lossless encoding unit 206 as filter information.
  • the ILF 511 sequentially selects, for example, pixels of the decoded image from the arithmetic unit 210 as pixels of interest.
  • the ILF 511 performs an initial class classification on the target pixel, and obtains an initial class of the target pixel.
  • the ILF 511 converts the initial class of the target pixel into a merge class according to the merge pattern corresponding to the adopted combination.
  • the ILF 511 acquires (reads) the tap coefficient of the merge class of the target pixel from among the tap coefficients for each merge class obtained by conversion according to the merge pattern corresponding to the adopted combination. Then, the ILF 511 selects, from the decoded image, a pixel in the vicinity of the pixel of interest as a prediction tap, and decodes a prediction equation for performing a product-sum operation between a tap coefficient of a merge class of the pixel of interest and a pixel of the decoded image as a prediction tap.
  • Filter processing is performed as prediction processing applied to the image to generate a filtered image.
  • the class classification by the ILF 511 for example, the class obtained by the class classification of the upper left pixel of the 2 ⁇ 2 pixel of the decoded image can be adopted as the class of each 2 ⁇ 2 pixel.
  • the operation unit 203 or the lossless encoding unit 206 is used as the encoding unit 161 in FIG. 87
  • the inverse quantization unit 208 or the operation unit 210 is used as the local decoding unit 162 in FIG. 87
  • the ILF 511 is used as the filter shown in FIG.
  • the section 463 corresponds to each.
  • FIG. 91 is a flowchart illustrating an example of an encoding process of encoding apparatus 460 in FIG.
  • the ILF 511 temporarily stores the decoded image supplied from the arithmetic unit 210, and temporarily stores the original image for the decoded image from the arithmetic unit 210 supplied from the rearrangement buffer 202.
  • steps S501 and S502 the same processes as those in steps S201 and S202 in FIG. 62 are performed.
  • step S503 the ILF 511 merges the initial class according to the merge pattern corresponding to the combination of the number of subclasses for each of the plurality of combinations of the number of subclasses that specify each of the plurality of merge patterns obtained by the subclass merge. Then, as in steps S36 and S37 in FIG. 5, tap coefficients for each merge class are obtained by using the normal equation obtained by tap coefficient learning.
  • the ILF 511 obtains a cost by performing a filter process using a tap coefficient for each merge class for each of a plurality of combinations of the number of subclasses. Then, the ILF 511 determines the combination of the number of subclasses that minimizes the cost among the plurality of combinations of the number of subclasses as the adoption combination, and the process proceeds from step S503 to step S504.
  • step S504 the ILF 511 supplies, as filter information, the adopted combination and the tap coefficients for each merge class obtained by converting the initial class according to the merge pattern corresponding to the adopted combination to the lossless encoding unit 206.
  • the lossless encoding unit 206 sets the filter information from the ILF 511 as a transmission target, and the process proceeds from step S504 to step S505.
  • the filter information set as the transmission target is included in the coded bit stream and transmitted in the predictive coding process performed in step S506 described below.
  • step S505 the ILF 511 adopts the adoption used in the class classification prediction process by using the adoption combination determined in the latest step S503 and the tap coefficient for each merge class obtained by converting the initial class according to the merge pattern corresponding to the adoption combination.
  • the combination and the tap coefficient are updated, and the process proceeds to step S506.
  • step S506 the predictive encoding of the original image is performed, and the encoding ends.
  • FIG. 92 is a flowchart illustrating an example of the predictive encoding process in step S506 of FIG.
  • steps S511 to S521 the same processes as those in steps S211 to S221 in FIG. 63 are performed.
  • step S522 the ILF 511 performs a filtering process as a class classification prediction process on the decoded image from the calculation unit 210, and supplies a filtered image obtained by the filtering process to the frame memory 212.
  • the process proceeds from step S522 to step S523.
  • step S522 the same process as that of the class classification prediction filter 410 (FIG. 85) is performed.
  • the ILF 511 performs initial class classification on the target pixel of the decoded image from the arithmetic unit 210, and obtains the initial class of the target pixel. Further, the ILF 511 converts the initial class of the target pixel into a merge class according to the merge pattern corresponding to the adopted combination updated in step S505 in FIG. The ILF 511 obtains the tap coefficient of the merge class of the target pixel from the tap coefficients for each merge class updated in step S505 in FIG. After that, the ILF 511 performs a filtering process as a prediction process of applying a prediction formula configured using a tap coefficient of a merge class of the pixel of interest to the decoded image, and generates a filtered image. The filter image is supplied from the ILF 511 to the frame memory 212.
  • steps S523 to S526 the same processes as those in steps S223 to S226 in FIG. 63 are performed.
  • FIG. 93 is a block diagram showing a detailed configuration example of the decoding device 470 of FIG.
  • the decoding device 470 includes an accumulation buffer 301, a lossless decoding unit 302, an inverse quantization unit 303, an inverse orthogonal transformation unit 304, an operation unit 305, a reordering buffer 307, a D / A conversion unit 308, and an ILF 606.
  • the decoding device 470 includes a frame memory 310, a selection unit 311, an intra prediction unit 312, a motion prediction compensation unit 313, and a selection unit 314.
  • the decoding device 470 is common to the decoding device 170 in FIG. 64 in that the decoding device 470 includes the accumulation buffer 301 to the calculation unit 305, the rearrangement buffer 307, the D / A conversion unit 308, the frame memory 310, and the selection unit 314. However, the decoding device 470 differs from the decoding device 170 in having an ILF 606 instead of the ILF 306.
  • the ILF 606 is configured, for example, in the same manner as the class classification prediction filter 410 without a learning function (FIG. 85), and performs a filtering process as a class classification prediction process, thereby performing a deblocking filter, an adaptive It functions as one of an offset filter, a bilateral filter, and an ALF, or two or more filters.
  • the ILF 606 sequentially selects pixels of the decoded image from the calculation unit 305 as pixels of interest.
  • the ILF 606 performs an initial class classification on the target pixel, and obtains an initial class of the target pixel. Further, the ILF 511 sets the initial class of the target pixel to the merge class according to the merge pattern corresponding to the adopted combination included in the filter information supplied from the lossless decoding unit 302 among the merge patterns determined for each combination of the number of subclasses. Convert.
  • the ILF 606 acquires the tap coefficient of the merge class of the target pixel among the tap coefficients for each merge class included in the filter information supplied from the lossless decoding unit 302.
  • the ILF 606 selects a pixel in the vicinity of the pixel of interest from the decoded image as a prediction tap, and calculates a prediction equation for performing a product-sum operation between a tap coefficient of the class of the pixel of interest and a pixel of the decoded image as the prediction tap. Performs a filtering process as a prediction process to be applied to the image data, and generates and outputs a filtered image.
  • the class classification by the ILF 606 for example, similarly to the ILF 511, the class obtained by the class classification of the upper left pixel of 2 ⁇ 2 pixels can be adopted as the class of each 2 ⁇ 2 pixel.
  • the filter image output from the ILF 606 is the same as the filter image output from the ILF 511 in FIG. 90, and is supplied to the rearrangement buffer 307 and the frame memory 310.
  • the lossless decoding unit 302 corresponds to the parsing unit 171 in FIG. 87
  • the inverse quantization unit 303 or the arithmetic unit 305 corresponds to the decoding unit 172 in FIG. 87
  • the ILF 606 corresponds to the filter unit 473 in FIG. .
  • FIG. 94 is a flowchart for explaining an example of the decoding process of the decoding device 470 in FIG.
  • step S601 the accumulation buffer 301 temporarily stores the encoded bit stream transmitted from the encoding device 460 and supplies the encoded bit stream to the lossless decoding unit 302 as appropriate, and the process proceeds to step S602.
  • step S602 the lossless decoding unit 302 receives and decodes the coded bit stream supplied from the accumulation buffer 301, and dequantizes the quantized coefficient as coded data included in the decoding result of the coded bit stream.
  • the signal is supplied to the unit 303.
  • the lossless decoding unit 302 parses the filter information and the encoded information. Then, the lossless decoding unit 302 supplies necessary encoding information to the intra prediction unit 312, the motion prediction compensation unit 313, and other necessary blocks. Further, the lossless decoding unit 302 supplies the filter information to the ILF 606.
  • step S602 the process proceeds from step S602 to step S603, in which the ILF 606 filters from the lossless decoding unit 302 a tap coefficient for each merge class obtained by transforming the initial class according to the adopted combination and the merge pattern corresponding to the adopted combination. Determine whether information has been supplied.
  • step S603 If it is determined in step S603 that the filter information has not been supplied, the process skips step S604 and proceeds to step S605.
  • step S603 If it is determined in step S603 that the filter information has been supplied, the process proceeds to step S604, in which the ILF 606 determines the adopted combination included in the filter information from the lossless decoding unit 302 and the merge pattern corresponding to the adopted combination.
  • the tap coefficient for each merge class obtained by the conversion of the initial class is obtained.
  • the ILF 606 is used in the class classification prediction process by using the tap combination for each merge class obtained by conversion of the initial class according to the adopted combination acquired from the filter information from the lossless decoding unit 302 and the merge pattern corresponding to the adopted combination.
  • the adopted combination and the tap coefficient are updated.
  • step S604 the predictive decoding process is performed, and the decoding process ends.
  • FIG. 95 is a flowchart illustrating an example of the predictive decoding process in step S605 in FIG.
  • steps S611 to S615 the same processes as those in steps S311 to S315 in FIG. 66 are performed.
  • step S616 the ILF 606 performs a filtering process as a class classification prediction process on the decoded image from the calculation unit 305, and supplies the filtered image obtained by the filtering process to the rearrangement buffer 307 and the frame memory 310. Then, the process proceeds from step S616 to step S617.
  • step S616 the same processing as the classification prediction filter 410 (FIG. 85) is performed.
  • the ILF 606 performs the same initial class classification as that of the ILF 511 on the target pixel of the decoded image from the arithmetic unit 305, and obtains the initial class of the target pixel. Further, the ILF 606 converts the initial class of the target pixel into a merge class according to the merge pattern corresponding to the adopted combination updated in step S604 in FIG. 94 among the merge patterns determined for each combination of the number of subclasses. The ILF 606 acquires the tap coefficient of the merge class of the target pixel from among the tap coefficients for each merge class updated in step S604 of FIG.
  • the ILF 606 performs a filtering process as a prediction process of applying a prediction formula including a tap coefficient of a merge class of the pixel of interest to the decoded image, and generates a filtered image.
  • the filter image is supplied from the ILF 606 to the rearrangement buffer 307 and the frame memory 310.
  • steps S617 to S619 the same processes as those in steps S317 to S319 in FIG. 66 are performed.
  • the present technology employs the GALF class classification as the initial class classification.However, this technology uses the initial class classification to perform the class classification based on the subclass classification of a plurality of features other than the GALF class classification. It can be applied when adopted as.
  • the class classification using the reliability in the inclination direction described with reference to FIGS. 24 and 25 is performed by subclass classification of the gradient strength ratio, direction, activity sum, and reliability in the inclination direction as a plurality of feature amounts. It can be said that it is a classification. Therefore, the present technology can be applied to a case where the class classification using the reliability in the inclination direction described with reference to FIGS. 24 and 25 is adopted as the initial class classification.
  • the class prediction filter 110 includes, for example, the motion prediction compensation unit 215 and the motion prediction compensation unit 313 in the encoding device 160 (FIG. 61) and the decoding device 170 (FIG. 64) in addition to the ILF 211 and the ILF 306. Can be applied to an interpolation filter or the like used for generating a predicted image. The same applies to the classification prediction filter 410 (FIG. 86).
  • FIG. 96 is a diagram for explaining the class classification of GALF.
  • FIG. 96 shows an initial class (final class) obtained by GALF class classification.
  • the target pixel is classified into one of three subclasses of a non-class, a weak (Weak) class, and a strong (Strong) class according to the gradient intensity ratio.
  • the activity subclass is classified into one of five subclasses of activity subclasses 0 to 5.
  • the gradient intensity ratio subclass is other than the non-class, the H / V class and the D0 / D1 class (depending on the direction)
  • the target pixel is classified into one of the 25 classes of the initial classes 0 to 24 by performing the direction subclass classification into one of the two subclasses of the direction subclasses 0 and 2).
  • the activity subclasses 0 to 4 are subclasses with lower (smaller) activities as the index #i of the activity subclass #i is smaller.
  • a purge pattern is set for each number of merge classes.
  • a merge pattern corresponding to each of 1 to 25 merge classes (the number of 25 merge classes equal to the initial class number) of each value of a natural number equal to or less than the initial class number can be set.
  • the number of merge classes of each natural number value equal to or smaller than the initial class number (hereinafter, also referred to as all types of merge classes) It is desirable to set a merge pattern corresponding to each.
  • the setting of the merge pattern corresponding to each of the 1 to 25 merge classes is performed by setting the merge class (merge pattern that constitutes the merge pattern corresponding to the maximum 25 merge classes).
  • merge class merge pattern that constitutes the merge pattern corresponding to the maximum 25 merge classes.
  • two merge classes of the merge classes constituting the merge pattern corresponding to the merge class number C are Merge into one merge class.
  • a predetermined rule to be followed when two merge classes of the merge classes constituting the merge pattern corresponding to the merge class number C are merged into one merge class is hereinafter also referred to as a merge rule.
  • FIG. 97 is a view for explaining the relationship between the merge pattern and the subclass.
  • FIG. 97 shows a merge pattern with 25 merge classes.
  • the horizontal direction corresponds to the activity subclass.
  • the first column (the first column from the left) corresponds to activity subclass 0 (without subclass merging). That is, the merge class in the first column is a merge class whose activity subclass is 0.
  • the second to fifth columns correspond to activity subclasses 1 to 4 (without subclass merging), respectively.
  • the vertical direction corresponds to the gradient intensity ratio subclass and the direction subclass.
  • the first row first row from the top
  • the second and fourth rows correspond to the weak class of the gradient intensity ratio subclass
  • Rows and the fifth row correspond to the strong class of the gradient intensity ratio subclass.
  • the second and third rows correspond to the D0 / D1 class of the direction subclass
  • the fourth and fifth rows correspond to the H / V class of the direction subclass.
  • the merge class 15 is expressed as a subclass
  • the activity subclass is 0, the direction subclass is H / V class
  • the gradient intensity ratio subclass is a weak class merge class. be able to.
  • the merge class 20 is expressed as a subclass
  • the activity subclass is 0, the direction subclass is the H / V class
  • the gradient intensity ratio subclass is a strong class merge class. Therefore, in the merge pattern of 25 merge classes, the merge between the merge class 15 and the merge class 20 is performed, for example, with the weak class of the gradient intensity ratio subclass when the activity subclass is 0 and the direction subclass is the H / V class. It can be said that it is a merge with the strong class.
  • FIG. 98 is a view for explaining the first merge rule.
  • the weak class and the strong class of the gradient intensity ratio subclass are respectively reduced from the activity subclass of the low activity. Will be merged.
  • the H / V class and the D0 / D1 class of the direction subclass are merged from the activity subclass of the low activity.
  • the subclass after the merge between the weak class and the strong class of the gradient intensity ratio subclass (hereinafter, also referred to as a merge subclass) is referred to as a high class.
  • the non-class and high class of the ratio subclass are merged from the activity subclass of the low activity.
  • activity subclasses are merged from low activity activity subclasses.
  • the merge class 15 and the merge class 20 forming the merge pattern corresponding to the number of merge classes 25 are merged into the merge class 15, so that the merge class A merge pattern corresponding to Equation 24 is set. Further, the merge class 5 and the merge class 10 constituting the merge pattern corresponding to the merge class number 24 are merged into the merge class 5, so that the merge pattern corresponding to the merge class number 23 is set, and the merge class number is set.
  • a merge pattern corresponding to the number of merge classes 22 is set.
  • merge patterns corresponding to the respective merge class numbers 21 to 15 are set.
  • the merge class 5 and the merge class 10 forming the merge pattern corresponding to the merge class number 15 are merged into the merge class 5 to correspond to the merge class number 14.
  • a merge pattern is set, and a merge pattern corresponding to the merge class number 13 is set by merging the merge class 6 and the merge class 10 constituting the merge pattern corresponding to the merge class number 14 into the merge class 6.
  • merge patterns corresponding to each of the 12 or 1 merge classes are set.
  • FIG. 99 is a diagram showing all the merge patterns set according to the first merge rule.
  • FIGS. 100, 101, 102, 103, 104, and 105 are views for explaining a merging method when setting all types of merge patterns according to the first merge rule.
  • the merge pattern corresponding to the number of merge classes 25 is a merge pattern in which the same merge class as the initial class obtained by the GALF class classification as the initial class classification is obtained.
  • the merge pattern corresponding to the merge class number 24 is a weak pattern (merge class 15) and a strong class (merge class 15) when the activity subclass is 0 and the direction subclass is the H / V class in the merge pattern corresponding to the merge class number 25.
  • the merge class 20) can be obtained by merging into one merge class 15 (first step).
  • the merge pattern corresponding to the merge class number 23 is a weak pattern (merge class 5) and a strong class (merge class 5) when the activity subclass is 0 and the direction subclass is the D0 / D1 class in the merge pattern corresponding to the merge class number 24.
  • Merge class 10) can be obtained by merging into one merge class 5 (first step).
  • the merge pattern corresponding to the merge class number 22 is the weak pattern (merge class 15) and the strong class (the merge class 15) when the activity subclass is 1 and the direction subclass is the H / V class in the merge pattern corresponding to the merge class number 23.
  • the merge class 19) can be obtained by merging into one merge class 15 (first step).
  • the merge pattern corresponding to the merge class number 21 is a weak class (merge class 6) in which the activity subclass is 1 and the direction subclass is the D0 / D1 class in the merge pattern corresponding to the merge class number 22.
  • the strong class (merge class 10) can be obtained by merging into one merge class 6 (first step).
  • the merge pattern corresponding to the merge class number 20 is the weak class (merge class 15) and the strong class (the merge class corresponding to the merge class number 21) when the activity subclass is 2 and the direction subclass is the H / V class.
  • the merge class 18) can be obtained by merging into one merge class 15 (first step).
  • the merge pattern corresponding to the merge class number 19 is a weak pattern (merge class 7) and a strong class (merge class 7) when the activity subclass is 2 and the direction subclass is the D0 / D1 class in the merge pattern corresponding to the merge class number 20.
  • Merge class 10) can be obtained by merging into one merge class 7 (first step).
  • the merge pattern corresponding to the number of merge classes 18 is the same as the merge pattern corresponding to the number of merge classes 19, except that the activity subclass is 3, and the direction subclass is the H / V class.
  • the merge class 17) can be obtained by merging into one merge class 15 (first step).
  • the merge pattern corresponding to the merge class number 17 is a weak class (merge class 8) in the case where the activity subclass is 3 and the direction subclass is the D0 / D1 class in the merge pattern corresponding to the merge class number 18.
  • a strong class (merge class 10) can be obtained by merging into one merge class 8 (first step).
  • the merge pattern corresponding to the merge class number 16 is a weak pattern (merge class 15) and a strong class (merge class 15) when the activity subclass is 4 and the direction subclass is the H / V class in the merge pattern corresponding to the merge class number 17.
  • the merge class 16) can be obtained by merging into one merge class 15 (first step).

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

La présente invention concerne un dispositif de codage, un procédé de codage, un dispositif de décodage et un procédé de décodage qui permettent la réduction du montant de traitement. Lesdits dispositif de codage et de décodage réalisent une classification de classe d'un pixel cible dans une image décodée (une image décodée locale) au moyen d'une classification de sous-classe de chacune de multiples valeurs de caractéristique, et, conformément à un ensemble de motifs de fusion à l'avance pour chaque numéro de classe de fusion, convertissent la classe initiale du pixel cible, obtenue par la classification de classe, en une classe de fusion obtenue par fusion de classes initiales par fusion de sous-classes de valeurs de caractéristiques. En outre, lesdits dispositif de codage et de décodage génèrent une image filtrée par réalisation d'un traitement de filtre, qui consiste à appliquer à l'image décodée une formule de prédiction qui consiste à effectuer une opération de produit-somme de pixels de l'image décodée et un coefficient de prise de la classe de fusion du pixel cible. Cette technique peut être appliquée, par exemple, à des cas dans lesquels une image est codée ou décodée.
PCT/JP2019/035819 2018-09-25 2019-09-12 Dispositif de codage, procédé de codage, dispositif de décodage, et procédé de décodage WO2020066642A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201980061396.1A CN112740678A (zh) 2018-09-25 2019-09-12 编码装置、编码方法、解码装置和解码方法
US17/268,320 US20210168407A1 (en) 2018-09-25 2019-09-12 Encoding device, encoding method, decoding device, and decoding method

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
JP2018-178412 2018-09-25
JP2018178412 2018-09-25
JP2018246543A JP2022002357A (ja) 2018-09-25 2018-12-28 符号化装置、符号化方法、復号装置、及び、復号方法
JP2018-246543 2018-12-28

Publications (1)

Publication Number Publication Date
WO2020066642A1 true WO2020066642A1 (fr) 2020-04-02

Family

ID=69952119

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/035819 WO2020066642A1 (fr) 2018-09-25 2019-09-12 Dispositif de codage, procédé de codage, dispositif de décodage, et procédé de décodage

Country Status (4)

Country Link
US (1) US20210168407A1 (fr)
CN (1) CN112740678A (fr)
TW (1) TW202029750A (fr)
WO (1) WO2020066642A1 (fr)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019065261A1 (fr) * 2017-09-27 2019-04-04 ソニー株式会社 Dispositif de codage, procédé de codage, dispositif de décodage et procédé de décodage
TWI800180B (zh) * 2021-07-13 2023-04-21 財團法人工業技術研究院 特徵資料編碼方法、編碼器、特徵資料解碼方法及解碼器

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2013150084A (ja) * 2012-01-18 2013-08-01 Nippon Telegr & Teleph Corp <Ntt> 画像符号化方法，画像符号化装置，画像復号方法，画像復号装置およびそれらのプログラム
JP2017523668A (ja) * 2014-06-13 2017-08-17 インテル コーポレイション ビデオ符号化用の高コンテンツ適応型品質回復フィルタ処理のためのシステムおよび方法
WO2017142946A1 (fr) * 2016-02-15 2017-08-24 Qualcomm Incorporated Fusion de filtres pour de multiples classes de blocs pour un codage vidéo
WO2017196852A1 (fr) * 2016-05-09 2017-11-16 Qualcomm Incorporated Signalisation d'informations de filtrage
JP2018509074A (ja) * 2015-02-11 2018-03-29 クアルコム，インコーポレイテッド コーディングツリーユニット（ｃｔｕ）レベル適応ループフィルタ（ａｌｆ）

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4003128B2 (ja) * 2002-12-24 2007-11-07 ソニー株式会社 画像データ処理装置および方法、記録媒体、並びにプログラム
KR101444675B1 (ko) * 2011-07-01 2014-10-01 에스케이 텔레콤주식회사 영상 부호화 및 복호화 방법과 장치

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2013150084A (ja) * 2012-01-18 2013-08-01 Nippon Telegr & Teleph Corp <Ntt> 画像符号化方法，画像符号化装置，画像復号方法，画像復号装置およびそれらのプログラム
JP2017523668A (ja) * 2014-06-13 2017-08-17 インテル コーポレイション ビデオ符号化用の高コンテンツ適応型品質回復フィルタ処理のためのシステムおよび方法
JP2018509074A (ja) * 2015-02-11 2018-03-29 クアルコム，インコーポレイテッド コーディングツリーユニット（ｃｔｕ）レベル適応ループフィルタ（ａｌｆ）
WO2017142946A1 (fr) * 2016-02-15 2017-08-24 Qualcomm Incorporated Fusion de filtres pour de multiples classes de blocs pour un codage vidéo
WO2017196852A1 (fr) * 2016-05-09 2017-11-16 Qualcomm Incorporated Signalisation d'informations de filtrage

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
KARCZEWICZ, MARTA ET AL.: "CE2-related: Two- dimensional ALF classification", JOINT VIDEO EXPERTS TEAM (JVET) OF ITU-T SG 16 WP 3 AND ISO/IEC JTC 1/SC 29/WG 11 11TH MEETING, no. JVET-K0373, 3 July 2018 (2018-07-03), Ljubljana, SI, XP030198939 *

Also Published As

Publication number Publication date
US20210168407A1 (en) 2021-06-03
TW202029750A (zh) 2020-08-01
CN112740678A (zh) 2021-04-30

Similar Documents

Publication Publication Date Title
US9661344B2 (en) Method and apparatus for encoding and decoding image through intra prediction
US11032557B2 (en) Decoding device, coding device, and method
JP2023133553A (ja) 画像処理装置及び画像処理方法
JP2021518076A (ja) 映像符号化／復号化方法及び装置
WO2012134246A2 (fr) Procédé de décodage entropique et appareil de décodage l&#39;utilisant
KR20200092314A (ko) 화상 처리 장치 및 방법
WO2020066642A1 (fr) Dispositif de codage, procédé de codage, dispositif de décodage, et procédé de décodage
KR20200092315A (ko) 화상 처리 장치 및 방법
US20120281749A1 (en) Encoder, decoder, and data configuration
WO2020175145A1 (fr) Dispositif de traitement d&#39;image et procédé de traitement d&#39;image
WO2019220947A1 (fr) Dispositif de codage, procédé de codage, dispositif de décodage et procédé de décodage
US10116944B2 (en) Video encoding device, video encoding method, and program
KR20220021471A (ko) 화상 처리 장치 및 화상 처리 방법
WO2020066643A1 (fr) Dispositif de codage, procédé de codage, dispositif de décodage, et procédé de décodage
CN111386703B (zh) 编码装置、编码方法、解码装置和解码方法
WO2020008910A1 (fr) Dispositif de codage, procédé de codage, dispositif de décodage, et procédé de décodage
JP2022002357A (ja) 符号化装置、符号化方法、復号装置、及び、復号方法
US20230108722A1 (en) Allocating bit rate between video streams using machine learning
WO2019208258A1 (fr) Dispositif de codage, procédé de codage, dispositif de décodage et procédé de décodage
WO2019131161A1 (fr) Dispositif de codage, procédé de codage, dispositif de décodage et procédé de décodage
WO2012090286A1 (fr) Procédé de codage d&#39;image vidéo, et procédé de décodage d&#39;image vidéo
WO2020262370A1 (fr) Dispositif et procédé de traitement d&#39;image
US20240040124A1 (en) Flexible coefficient coding in video compression
US20230316582A1 (en) Information processing apparatus and method
WO2020175146A1 (fr) Dispositif de traitement d&#39;image et procédé de traitement d&#39;image

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19865954

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19865954

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP