WO2021060484A1

WO2021060484A1 - Image processing device and image processing method

Info

Publication number: WO2021060484A1
Application number: PCT/JP2020/036316
Authority: WO
Inventors: 健治近藤
Original assignee: ソニー株式会社
Priority date: 2019-09-25
Filing date: 2020-09-25
Publication date: 2021-04-01

Abstract

The present disclosure relates to an image processing device and an image processing method which make it possible to suppress degradation of image quality and a decrease in encoding efficiency. The image processing device is provided with: a setting unit which sets identifying information identifying whether the pixels of a current prediction block of an image to be subjected to an encoding process is to be subjected to a motion compensating process in which an optical flow process is applied; and an encoding unit which encodes the image and generates a bit stream including the identifying information set by the setting unit. The present techniques are applicable to, for example, an image processing system in which encoding and decoding are performed according to a versatile video coding (VVC) scheme.

Description

Image processing device and image processing method

The present disclosure relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and an image processing method capable of suppressing deterioration of image quality and deterioration of coding efficiency.

In recent years, in order to further improve the coding efficiency for AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding), etc., standardization of a coding method called VVC (Versatile Video Coding) has been promoted (the embodiment described later). See also support for).

For example, Non-Patent Document 1 discloses a technique for applying motion compensation to a luminance component using an optical flow.

By the way, conventionally, motion compensation using optical flow is applied only to the luminance component, and motion compensation using optical flow is not applied to the color difference component. Therefore, for example, in an image in a specific case where the affine transformation is effective, such as when an object is rotating, the deviation between the luminance component and the color difference component becomes large, and the subjective image quality is deteriorated and coded. It is thought that a decrease in efficiency will occur.

The present disclosure has been made in view of such a situation, and makes it possible to suppress deterioration of image quality and deterioration of coding efficiency.

The image processing apparatus of the first aspect of the present disclosure sets identification information for identifying whether to perform motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be encoded. It includes a setting unit and a coding unit that encodes the image and generates a bit stream including identification information set by the setting unit.

The image processing method of the first aspect of the present disclosure identifies whether the image processing apparatus performs motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be encoded. It includes setting the identification information and encoding the image to generate a bitstream containing the set identification information.

In the first aspect of the present disclosure, identification information is set to identify whether to perform motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be encoded, and the image is displayed. It is encoded to generate a bitstream containing the set identification information.

The image processing apparatus of the second aspect of the present disclosure is a bit stream including identification information that identifies whether to perform motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be decoded. From the perspective unit that parses the identification information, a control unit that controls whether to perform motion compensation processing to which the optical flow processing is applied by referring to the identification information parsed by the perspective unit, and a control unit that controls the control unit. It is provided with a decoding unit that decodes the current prediction block by using the prediction image generated by the motion compensation processing.

The image processing method of the second aspect of the present disclosure identifies whether the image processing apparatus performs motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be decoded. Parsing the identification information from a bit stream containing information, controlling whether to perform motion compensation processing to which optical flow processing is applied by referring to the parsed identification information, and controlled motion compensation processing. Includes decoding the current prediction block using the prediction image generated by.

The second aspect of the present disclosure is from a bitstream containing identification information that identifies whether to perform motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be decoded. The identification information is parsed, and with reference to the parsed identification information, it is controlled whether or not motion compensation processing to which optical flow processing is applied is performed, and current prediction is performed using a prediction image generated by the controlled motion compensation processing. The block is decrypted.

It is a figure which shows an example of a high level syntax. It is a figure explaining a block and a subblock. It is a figure explaining a motion vector. It is a block diagram which shows the structural example of one Embodiment of the image processing system to which this technique is applied. It is a figure explaining the 1st method to obtain the motion vector for a color difference component. It is a figure explaining the 2nd method to obtain the motion vector for a color difference component. It is a figure explaining an effective situation to apply this technology. It is a block diagram which shows the configuration example of one Embodiment of the computer-based system to which this technique is applied. It is a block diagram which shows the structural example of one Embodiment of an image coding apparatus. It is a flowchart explaining the coding process. It is a block diagram which shows the structural example of one Embodiment of an image decoding apparatus. It is a flowchart explaining the decoding process. It is a block diagram which shows the structural example of one Embodiment of the computer to which this technique is applied.

<Documents that support technical contents and technical terms>
The scope disclosed herein is not limited to the contents of the examples, and the contents of the following references REF1 to REF5, which are known at the time of filing, are also incorporated herein by reference. In other words, the contents described in the following references REF1 to REF5 are also the basis for judging the support requirements. In addition, the references referenced in references REF1 to REF5 are also the basis for determining support requirements.

For example, even if Quad-Tre Block Structure, QTBT (Quad Tree Plus Binary Tree), Block Structure, MTT (Multi-type Tree) Block Structure, etc. are not directly defined in the detailed description of the invention, the present disclosure And shall meet the support requirements of the claims. Similarly, technical terms such as Parsing, Syntax, and Semantics are also within the scope of the present disclosure, even if they are not directly defined in the detailed description of the invention. Yes, and shall meet the support requirements of the claims.

REF1: Recommendation ITU-T H.264 (04/2017) “Advanced video coding for generic audiovisual services”, April 2017
REF2: Recommendation ITU-T H.265 (02/2018) “High efficiency video coding”, February 2018
REF3: Benjamin Bross, Jianle Chen, Shan Liu, Versatile Video Coding (Draft 6), JVET-O2001-v14 (version 14 --date 2019-07-31)
REF4: Jianle Chen, Yan Ye, Seung Hwan Kim, Algorithm description for Versatile Video Coding and Test Model 6 (VTM 6), JVET-O2002-v1 (version 1 --date 2019-08-15)
REF5: Jiancong (Daniel) Luo, Yuwen He, CE4: Prediction refinement with optical flow for affine mode (Test 2.1), JVET-O0070, (version 5 --date 2019-07-10)

For example, REF3 describes PROF (Prediction refinement with optical flow) and high-level syntax related to this technology, which will be described later, as shown in FIG.

<Terminology>
In this application, the following terms are defined as follows.

<Block>
Unless otherwise specified, a "block" (not a block indicating a processing unit) used as a partial area or a processing unit of an image (picture) indicates an arbitrary partial area in the picture, and its size, shape, and processing. The characteristics are not limited. For example, "block" includes TB (Transform Block), TU (Transform Unit), PB (Prediction Block), PU (Prediction Unit), SCU (Smallest Coding Unit), CU (Coding Unit), LCU (Largest Coding Unit). ), CTB (Coding TreeBlock), CTU (Coding Tree Unit), conversion block, subblock, macroblock, tile, slice, etc., any partial area (processing unit) shall be included.

<Specify block size>
Further, when specifying the size of such a block, not only the block size may be directly specified, but also the block size may be indirectly specified. For example, the block size may be specified using the identification information that identifies the size. Further, for example, the block size may be specified by the ratio or difference with the size of the reference block (for example, LCU, SCU, etc.). For example, when transmitting information for specifying a block size as a syntax element or the like, the information for indirectly specifying the size as described above may be used as the information. By doing so, the amount of information of the information can be reduced, and the coding efficiency may be improved. Further, the designation of the block size also includes the designation of the range of the block size (for example, the designation of the range of the allowable block size).

<Unit of information / processing>
The data unit in which various information is set and the data unit targeted by various processes are arbitrary and are not limited to the above-mentioned examples. For example, these information and processes are TU (Transform Unit), TB (Transform Block), PU (Prediction Unit), PB (Prediction Block), CU (Coding Unit), LCU (Largest Coding Unit), and subblock, respectively. , Blocks, tiles, slices, pictures, sequences, or components, or data in those data units may be targeted. Of course, this data unit can be set for each information or process, and it is not necessary that the data unit of all the information or process is unified. The storage location of these information is arbitrary, and may be stored in the header, parameter set, or the like of the above-mentioned data unit. Further, it may be stored in a plurality of places.

<Control information>
The control information related to the present technology may be transmitted from the coding side to the decoding side. For example, control information (for example, enabled_flag) that controls whether or not the application of the present technology described above is permitted (or prohibited) may be transmitted. Further, for example, control information indicating an object to which the present technology is applied (or an object to which the present technology is not applied) may be transmitted. For example, control information may be transmitted that specifies the block size (upper limit, / lower limit, or both) to which the present technology is applied (or permission or prohibition of application), a frame, a component, a layer, or the like.

<Flag>
In the present specification, the "flag" is information for identifying a plurality of states, and is not only information used for identifying two states of true (1) or false (0), but also three or more states. It also contains information that can identify the state. Therefore, the value that this "flag" can take may be, for example, 2 values of 1/0 or 3 or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be 1 bit or a plurality of bits. Further, the identification information (including the flag) is assumed to include not only the identification information in the bit stream but also the difference information of the identification information with respect to a certain reference information in the bit stream. In, the "flag" and "identification information" include not only the information but also the difference information with respect to the reference information.

<Associate metadata>
Further, various information (metadata, etc.) regarding the coded data (bit stream) may be transmitted or recorded in any form as long as it is associated with the coded data. Here, the term "associate" means, for example, to make the other data available (linkable) when processing one data. That is, the data associated with each other may be combined as one data or may be individual data. For example, the information associated with the coded data (image) may be transmitted on a transmission path different from the coded data (image). Further, for example, the information associated with the coded data (image) may be recorded on a recording medium (or another recording area of the same recording medium) different from the coded data (image). Good. Note that this "association" may be a part of the data, not the entire data. For example, an image and information corresponding to the image may be associated with each other in an arbitrary unit such as a plurality of frames, one frame, or a part within the frame.

In addition, in this specification, "synthesize", "multiplex", "add", "integrate", "include", "store", "insert", "insert", "insert". A term such as "" means combining a plurality of objects into one, for example, combining encoded data and metadata into one data, and means one method of "associating" described above. Further, in the present specification, the coding includes not only the whole process of converting an image into a bit stream but also a part of the process. For example, it not only includes processing that includes prediction processing, orthogonal transformation, quantization, arithmetic coding, etc., but also includes processing that collectively refers to quantization and arithmetic coding, prediction processing, quantization, and arithmetic coding. Including processing, etc. Similarly, decoding includes not only the entire process of converting a bitstream into an image, but also some processes. For example, it not only includes processing that includes inverse arithmetic decoding, inverse quantization, inverse orthogonal transformation, prediction processing, etc., but also processing that includes inverse arithmetic decoding and inverse quantization, inverse arithmetic decoding, inverse quantization, and prediction processing. Including processing that includes and.

The prediction block means a block that is a processing unit when performing inter-prediction, and includes sub-blocks in the prediction block. If the processing unit is unified with the orthogonal conversion block, which is the processing unit when performing orthogonal conversion, or the coding block, which is the processing unit when performing coding processing, and the orthogonal conversion block, It means the same flock as the coded block.

Inter-prediction is a general term for processing that involves prediction between frames (prediction blocks), such as derivation of motion vectors by motion detection (MotionPrediction / MotionEstimation) and motion compensation using motion vectors (MotionCompensation). , Some processing (for example, motion compensation processing only) or all processing (for example, motion detection processing + motion compensation processing) used when generating a predicted image is included. The inter-prediction mode is referred to when deriving the inter-prediction mode, such as the mode number when performing inter-prediction, the index of the mode number, the block size of the prediction block, and the size of the sub-block that is the processing unit in the prediction block. It means a comprehensive set of variables (parameters).

In the present disclosure, identification data that identifies a plurality of patterns can be set as a bitstream syntax. In this case, the decoder can perform processing more efficiently by parsing + referencing the identification data. The method (data) for identifying the block size is not only to quantify (bite) the block size itself, but also to identify the difference value with respect to the reference block size (maximum block size, minimum block size, etc.) ( Data) is also included.

Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

<Conventional motion compensation>
With reference to FIGS. 2 and 3, the processing of motion compensation (MC: Motion Compensation) using the affine transformation when the format of the input image is Chroma Format: 4: 2: 0 will be described.

In VVC, motion compensation processing using affine transformation is performed by further dividing the motion compensation block into 4 × 4 samples called subblocks.

As shown in FIG. 2, the luminance component Y is subjected to motion compensation processing in an 8 × 8 block, and the color difference components Cb and Cr are subjected to motion compensation processing in a 4 × 4 block. That is, the sizes of the 8 × 8 block of the luminance component Y and the 4 × 4 block of the color difference components Cb and Cr match. At this time, in the motion compensation using the affine transformation, the size of the subblock is 4 × 4, so that the 8 × 8 block of the luminance component Y is divided into four subblocks.

And, by the technique of JVET-O0070, the processing of optical flow is added to the motion compensation of the luminance component Y.

As shown in FIG. 3, _{after performing motion compensation for the luminance component Y using the motion vector VSB} of the 4 × 4 subblock, the motion vector ΔV at the pixel level indicated by the white arrow. Optical flow processing is applied using _{(i, j).} On the other hand, since the optical flow processing is not applied to the color difference components Cb and Cr, it is considered that subjective deterioration of image quality and deterioration of coding efficiency occur as described above.

Therefore, in this technology, it is proposed to apply the optical flow processing to the color difference components Cb and Cr (chroma signal). Furthermore, the present technology proposes to skip the optical flow processing so that it is not applied to the image of a specific case.

<Configuration example of image processing system>
FIG. 4 is a block diagram showing a configuration example of an embodiment of an image processing system to which the present technology is applied.

As shown in FIG. 4, the image processing system 11 includes an image coding device 12 and an image decoding device 13. For example, in the image processing system 11, the image input to the image coding device 12 is encoded, the bit stream obtained by the coding is transmitted to the image decoding device 13, and the image decoding device 13 decodes the bit stream from the bit stream. The decoded image is output.

The image coding device 12 includes an inter-prediction unit 21, a coding unit 22, and a setting unit 23, and the image decoding device 13 includes an inter-prediction unit 31, a decoding unit 32, a perspective unit 33, and a control unit 34. have.

The inter-prediction unit 21 performs motion compensation processing by applying an interpolation filter to the current prediction block to be coded, and performs inter-prediction to generate prediction pixels in the current prediction block. At this time, the inter-prediction unit 21 is configured to perform motion compensation processing (hereinafter, referred to as color difference optical flow processing) to which optical flow processing is applied to the color difference component of the current prediction block to be coded. Will be done. That is, the inter-prediction unit 21 performs the color difference optical flow processing on the color difference component as well as the luminance component. Of course, the inter-prediction unit 21 can perform normal optical flow processing.

The coding unit 22 encodes the current pixel in the current prediction block using the prediction pixel generated by the inter prediction unit 21 to generate a bit stream.

The setting unit 23 sets identification data for identifying whether to apply the color difference optical flow processing, block size identification data for identifying the block size of the predicted block to which the color difference optical flow processing is applied, and the like. Then, the coding unit 22 generates a bit stream including the identification data set by the setting unit 23.

Further, the setting unit 23 identifies identification information for identifying whether to perform motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be encoded, and motion to which optical flow processing is applied. Set identification information that identifies whether compensation processing is performed on the slice. For example, the setting unit 23 sets the identification information for identifying whether to perform the motion compensation processing to which the optical flow processing is applied to the slice as the header of the bitstream, and the coding unit 22 sets the identification information as the header syntax. Generate a bitstream containing as. Further, the setting unit 23 sets the existence identification information for identifying that the identification information is included in the bit stream as a sequence parameter set, and the coding unit 22 sets the bit stream including the existence identification information as the sequence parameter set. Generate.

Similar to the inter-prediction unit 21, the inter-prediction unit 31 also performs color-difference optical flow processing on the color-difference component of the current prediction block to be decoded, and generates prediction pixels in the current prediction block. The inter-prediction unit 31 refers to the identification data contained in the bitstream to identify whether or not to apply the cross-component inter-prediction, and to identify the block size of the prediction block to which the cross-component inter-prediction is applied. can do. Of course, the inter-prediction unit 31 can perform normal optical flow processing.

The decoding unit 32 decodes the current pixel in the current prediction block by using the prediction pixel generated by the inter prediction unit 31 by the motion compensation process controlled by the control unit 34.

The parsing unit 33 parses the identification information from the bit stream including the identification information that identifies whether to perform the motion compensation processing to which the optical flow processing is applied to the pixels of the current prediction block of the image to be the decoding processing. .. For example, the identification information is included as a bitstream header, and the parsing unit 33 parses the bitstream header. Further, the parse unit 33 parses the identification information only when it indicates that the existence identification information included in the bit stream includes the identification information. For example, the bitstream includes the existence identification information as a sequence parameter set, and the parse unit 33 parses the existence identification information included in the bitstream by setting the sequence parameters.

The control unit 34 controls whether to perform motion compensation processing to which optical flow processing is applied with reference to the identification information parsed by the perspective unit 33, and the decoding unit 32 controls the motion compensation processing controlled by the control unit 34. The current prediction block is decoded using the prediction image generated by. For example, the control unit 34 controls to perform the motion compensation process to which the optical flow process is applied only when the identification information indicates that the motion compensation process to which the optical flow process is applied is performed. On the other hand, when the identification information indicates that the motion compensation process to which the optical flow process is applied is not performed, the control unit 34 controls to skip the motion compensation process to which the optical flow process is applied.

The image processing system 11 is configured as described above, and inter-prediction is performed using motion prediction called PROF. For example, motion compensation is performed at the subblock level using an affine motion model, and then PROF is applied to perform correction using optical flow processing at the pixel level.

By the way, PROF is usually applied only to luminance signals and not to chroma signals in order to reduce the amount of processing. Therefore, for example, in an image in a specific case where the affine transformation is effective, such as when an object is rotating, the image quality of the chroma signal deteriorates and the coding efficiency deteriorates due to the difference in the handling of the luminance signal and the chroma signal. May occur.

Therefore, in the image processing system 11, PROF can be controlled so as not to be applied to the image of a specific case as described below. For example, the image processing system 11 controls the PROF by using the high-level syntax of the bitstream as shown in FIG. 1 described above.

For example, slices that are likely to correspond to a specific case where you want to control RPOF are slices with large movements of the affine movement model as well as translational movements, slices whose reference direction is only the past, and slices whose reference direction is only the future. There are slices, slices with a large time distance of the referenced frame, and so on. Here, the slice whose reference direction is only the past cannot be used for bidirectional prediction between the past and the future, and the slice with a large time distance of the referenced frame is concerned that the change in movement is large. is there.

In this way, the image processing system 11 can signal each coding unit (slice) to be controlled. Specifically, in the image encoding device 12, the setting unit 23 sets the existence identification information sps_prof_enable_slice_present_flag to 1, sets the identification information slice_disable_prof_flag to 1 in the slice header of the current slice to be encoded, and sets the identification information slice_disable_prof_flag to 1. Send to 13. Then, in the image decoding device 13, the parse unit 33 parses the existence identification information sps_prof_enable_slice_present_flag, and if 1 is set, the identification information slice_disable_prof_flag is also read by the slice header. Then, in the image decoding device 13, the parsing unit 33 parses the identification information slice_disable_prof_flag, and if 1 is set, the parsing unit 33 processes the current slice so that the PROF processing is not performed.

In this way, in the image processing system 11, it is possible to skip the optical flow processing so that neither the luminance signal nor the chroma signal is applied to the image in a specific case. As a result, it is possible to suppress deterioration of the image quality of the chroma signal due to the difference in handling of the luminance signal and the chroma signal, and it is expected that the coding efficiency of the chroma signal is improved. Further, in the image processing system 11, the PROF can be flexibly controlled from the viewpoint of the processing amount.

For example, set the syntax as follows. If you want to control by sequence or slice, set the semantics as follows.

In the Sequence parameter set RBSP semantics shown in FIG. 1 above, when sps_prof_enable_slice_present_flag is 1, it indicates that slice_disable_prof_flag is sent by slice header. On the other hand, when sps_prof_enable_slice_present_flag is 0, it means that slice_disable_prof_flag is not sent by slice header. If sps_prof_enable_slice_resent_flag is not sent because motion compensation using the affine motion model is not used in SPS or PROF is not used in SPS, sps_prof_enable_slice_resent_flag is inferred to 0.

In the Slice header semantics shown in FIG. 1 above, when slice_disable_prof_flag is 1, it indicates that the current slice is not corrected by PROF. On the other hand, when slice_disable_prof_flag is 0, it indicates that PROF can be used in the current slice. If slice_disable_prof_flag is not sent, it is inferred to 0.

Further, in the image processing system 11, the inter-prediction unit 21 and the inter-prediction unit 31 have already calculated _{the pixel-level motion vectors ΔV Cb (i, j)} and ΔV _{Cr (i, j) of the color difference components Cb and Cr.} It is derived from _{the motion vector ΔV (i, j)} used for the luminance component Y. Then, in the image processing system 11, by applying the color difference optical flow processing to the color difference components Cb and Cr, it is possible to suppress subjective deterioration of image quality and deterioration of coding efficiency.

FIG. 5 is a diagram illustrating a first method of obtaining a motion vector for the color difference components Cb and Cr from the motion vector for the luminance component Y.

For example, in the first method, the pixel-level motion vector ΔV _{Cb of} the color difference component Cb and the pixel-level motion vector ΔV Cr of the color difference component _Cr are calculated from the average of the motion vector ΔV used for the brightness component Y. How to do it.

That is, as shown in FIG. 5, one pixel of the color difference components Cb and Cr corresponds to four pixels of the brightness component Y, and the four motion vectors ΔV _{(i, j)} used in the optical flow processing for the four pixels. The average is calculated and used as the motion vectors ΔV _{Cb (i, j)} and ΔV _{Cr (i, j)} of the color difference components Cb and Cr.

X component ΔV _{Cbx (i, j)} of the motion vector of the color difference component Cb is, the x component [Delta] V _lx upper left of the motion vector of the luminance component Y _{(i, j),} x component [Delta] V _lx at the top right of the motion vector of the luminance component Y _{(I + 1, j)} _{, using the x component ΔV lix (i, j + 1)} of the lower left motion vector of the luminance component Y and the x component ΔV _{lix (i + 1, j + 1)} of the lower right motion vector of the luminance component Y, It is calculated according to the following equation (1). _{Similarly, the y component ΔV Cby (i, j)} of the motion vector of the color difference component Cb _{is the y component ΔV ly (i, j)} of the motion vector on the upper left of the brightness component Y, and the y component y of the motion vector on the upper right of the brightness component Y. The component ΔV _{ly (i + 1, j)} _{, the y component ΔV ly (i, j + 1)} of the lower left motion vector of the brightness component Y, and the y component ΔV _{ly (i + 1, j + 1)} of the lower right motion vector of the brightness component Y. It is calculated according to the following equation (1).

Similarly, using this equation (1), the x component ΔV _{Crx (i, j)} and the y component ΔV _{Cry (i, j)} of the motion vector of the color difference component Cr can be obtained.

Chrominance components _{Cb (i, j)} of the amount of change .DELTA.Cb _{(i, j)} is, x direction of the gradient _{g Cbx (i, j)} of the color difference components Cb and y direction of the gradient _{g Cby (i, j),} and the formula It is obtained according to the following equation (2) using the x component ΔV _{Cbx (i, j)} and the y component ΔV _{Cby (i, j)} _{of the motion vector ΔV Cb (i, j)} obtained from (1).

The position (i, j) the color difference components _{Cb (i, j)} color corrected by applying the color difference optical flow processing to difference components Cb _{'(i, j)} of was determined by the equation (2) By adding the amount of change ΔCb _{(i, j)} to the color difference component Cb _{(i, j)} as a correction value, it can be obtained according to the following equation (3).

_{Similarly, the color difference component Cr (i, j)} at the position (i, j) is also corrected by applying the color difference optical flow processing using the equations (2) and (3). Cr' _{(i, j)} can be obtained.

FIG. 6 is a diagram illustrating a second method of obtaining a motion vector for the color difference components Cb and Cr from the motion vector for the luminance component Y.

For example, in the second method, one of the motion vectors ΔV _{(i, j)} _{used for the luminance component Y is used as the pixel-level motion vectors ΔV Cb (i, j)} and ΔV _{Cr (} for the color difference components Cb and Cr). This is the method used as _{i, j).}

That is, as shown in FIG. 6, one pixel of the color difference components Cb and Cr corresponds to four pixels of the brightness component Y, and the four motion vectors ΔV _{(i, j)} used in the optical flow processing for the four pixels. Of these, one with similar motion (in the example shown in FIG. 6, the motion vector of the upper left pixel) is defined as the motion vectors ΔV _{Cb (i, j)} and ΔV _{Cr (i, j)} of the color difference components Cb and Cr. Use.

_{The x component ΔV Cbx (i, j)} and the y component ΔV _{Cby (i, j)} of the motion vector of the color difference component Cb are the x component of the motion vector on the upper left of the luminance component Y, as shown in the following equation (4). It becomes ΔV _{lp (i, j)} and y component ΔV _{ly (i, j).}

_{Then, similarly to the first method, the color difference component Cb'(i, j)} and the color difference component Cr'corrected by applying the color difference optical flow processing using the above-mentioned equations (2) and (3). _{(I, j)} can be obtained. By adopting the second method, the amount of calculation can be reduced as compared with the first method.

As described above, the image processing system 11 can improve the accuracy of motion compensation by performing motion compensation for the color difference components Cb and Cr at the sub-block level and then performing color difference optical flow processing. Then, the luminance component Y is subjected to the optical flow treatment, and the color difference components Cb and Cr are subjected to the color difference optical flow treatment to reduce the deviation between the corrected luminance component Y and the color difference components Cb and Cr. , Deterioration of image quality and deterioration of coding efficiency can be suppressed.

With reference to FIG. 7, an effective situation in which this technology is applied will be described.

For example, when applying this technology, there is a concern that the processing amount will increase, and it is preferable to apply this technology in an effective situation where the processing amount can be suppressed. That is, it is effective to apply this technology with motion compensation in which the motion of the affine conversion is large. Therefore, the condition that the reference time of the motion compensation is large can be expected to be effective by using this technique.

As shown in A of FIG. 7, the reference POC distance or Temporal ID is used as a threshold value, and whether or not the present technology is applied is determined based on whether or not the correction of the affine transformation is expected to be large. Can be done. For example, it is conceivable to use this technique in the refine conversion of a reference with a large POC distance. Further, when hierarchical coding is used, the same effect can be obtained with Temporal ID. That is, under the condition that the Temporal ID is smaller or larger than the predetermined threshold value, a large movement is compensated, and this technique can be an effective condition.

Further, as shown in B of FIG. 7, it is considered that the judgment using the reference direction is also effective.

For example, in POC8, even if it is Bi-prediction, L0 and L1 are references in the same time direction as shown by the solid arrow. For other POC4 etc., the direction of L0 and L1 prediction is the past and the future as shown by the broken line arrow. Therefore, when the past and future can be used as a reference like POC4, motion compensation with a certain degree of accuracy can be performed without correction by optical flow.

On the other hand, in the case of a reference only in the past such as POC8, it can be expected that the correction of optical flow will be effective. Therefore, when the reference directions are the same, the present technology can be a more promising condition.

In the present embodiment, it has been described that the optical flow processing is applied to the luminance signal Y and the color difference optical flow processing is applied to the chroma signal Cb and the chroma signal Cr, but the present invention is limited to this. It will not be done. For example, the optical flow processing can be applied to the luminance signal Y, and the color difference optical flow processing can be applied to the difference signal U and the difference signal V.

<Computer-based system configuration example>
FIG. 8 is a block diagram showing a configuration example of an embodiment of a computer-based system to which the present technology is applied.

FIG. 8 is a block diagram showing a configuration example of a network system in which one or more computers, servers, and the like are connected via a network. The hardware and software environment shown in the embodiment of FIG. 8 is shown as an example of being able to provide a platform for implementing the software and / or method according to the present disclosure.

As shown in FIG. 8, the network system 101 includes a computer 102, a network 103, a remote computer 104, a web server 105, a cloud storage server 106, and a computer server 107. Here, in the present embodiment, a plurality of instances are executed by one or more of the functional blocks shown in FIG.

Further, in FIG. 8, a detailed configuration of the computer 102 is illustrated. The functional block shown in the computer 102 is shown for establishing an exemplary function, and is not limited to such a configuration. Further, although the detailed configurations of the remote computer 104, the web server 105, the cloud storage server 106, and the computer server 107 are not shown, they include the same configurations as the functional blocks shown in the computer 102. ing.

The computer 102 may be a personal computer, desktop computer, laptop computer, tablet computer, netbook computer, personal digital assistant, smartphone, or other programmable electronic device capable of communicating with other devices on the network. Can be done.

The computer 102 includes a bus 111, a processor 112, a memory 113, a non-volatile storage 114, a network interface 115, a peripheral device interface 116, and a display interface 117. Each of these features is implemented in an individual electronic subsystem (integrated circuit chip or a combination of chips and associated devices) in some embodiments, or in some embodiments, some of the features are combined. It may be mounted on a single chip (system on chip or SoC (System on Chip)).

Bus 111 can adopt various proprietary or industry standard high-speed parallel or serial peripheral interconnection buses.

Processor 112 may employ one designed and / or manufactured as one or more single or multi-chip microprocessors.

The memory 113 and the non-volatile storage 114 are storage media that can be read by the computer 102. For example, the memory 113 can employ any suitable volatile storage device such as DRAM (Dynamic Random Access Memory) or SRAM (Static RAM). The non-volatile storage 114 includes a flexible disk, a hard disk, an SSD (Solid State Drive), a ROM (Read Only Memory), an EPROM (Erasable and Programmable Read Only Memory), a flash memory, a compact disk (CD or CD-ROM), and a DVD (CD or CD-ROM). At least one or more of Digital Versatile Disc), card type memory, or stick type memory can be adopted.

The program 121 is stored in the non-volatile storage 114. Program 121 is, for example, a collection of machine-readable instructions and / or data used to create, manage, and control specific software features. In a configuration in which the memory 113 is much faster than the non-volatile storage 114, the program 121 can be transferred from the non-volatile storage 114 to the memory 113 before being executed by the processor 112.

The computer 102 can communicate and interact with other computers via the network 103 via the network interface 115. The network 103 can adopt, for example, a LAN (Local Area Network), a WAN (Wide Area Network) such as the Internet, or a combination of LAN and WAN, including a wired, wireless, or optical fiber connection. .. In general, network 103 consists of any combination of connections and protocols that support communication between two or more computers and related devices.

The peripheral device interface 116 can input and output data to and from other devices that can be locally connected to the computer 102. For example, the peripheral interface 116 provides a connection to the external device 131. The external device 131 includes a keyboard, mouse, keypad, touch screen, and / or other suitable input device. The external device 131 may also include, for example, a thumb drive, a portable optical or magnetic disk, and a portable computer readable storage medium such as a memory card.

In embodiments of the present disclosure, for example, software and data used to implement program 121 may be stored on such a portable computer readable storage medium. In such embodiments, the software may be loaded directly into the non-volatile storage 114 or into the memory 113 via the peripheral interface 116. Peripheral device interface 116 may use an industry standard such as RS-232 or USB (Universal Serial Bus) for connection with the external device 131.

The display interface 117 can connect the computer 102 to the display 132, and the display 132 can be used to present a command line or graphical user interface to the user of the computer 102. For example, industry standards such as VGA (Video Graphics Array), DVI (Digital Visual Interface), DisplayPort, and HDMI (High-Definition Multimedia Interface) (registered trademark) can be adopted for the display interface 117.

<Configuration example of image coding device>
FIG. 9 shows the configuration of an embodiment of an image coding device as an image processing device to which the present disclosure is applied.

The image coding device 201 shown in FIG. 9 encodes the image data by using the prediction process. Here, as the coding method, for example, a VVC (Versatile Video Coding) method, a HEVC (High Efficiency Video Coding) method, or the like is used.

The image coding device 201 of FIG. 9 has an A / D conversion unit 202, a screen rearrangement buffer 203, a calculation unit 204, an orthogonal conversion unit 205, a quantization unit 206, a lossless coding unit 207, and a storage buffer 208. Further, the image coding device 201 includes an inverse quantization unit 209, an inverse orthogonal conversion unit 210, an arithmetic unit 211, a deblocking filter 212, an adaptive offset filter 213, an adaptive loop filter 214, a frame memory 215, a selection unit 216, and an intra prediction. It has a unit 217, a motion prediction / compensation unit 218, a prediction image selection unit 219, and a rate control unit 220.

The A / D conversion unit 202 A / D-converts the input image data (Picture (s)) and supplies it to the screen sorting buffer 203. The digital data image may be input without providing the A / D conversion unit 202.

The screen rearrangement buffer 203 stores the image data supplied from the A / D conversion unit 202, and encodes the images of the frames in the stored display order according to the GOP (Group of Picture) structure. Sort by frame order. The screen rearrangement buffer 203 outputs the images in which the frame order is rearranged to the calculation unit 204, the intra prediction unit 217, and the motion prediction / compensation unit 218.

The calculation unit 204 subtracts the prediction image supplied from the intra prediction unit 217 or the motion prediction / compensation unit 218 via the prediction image selection unit 219 from the image output from the screen rearrangement buffer 203, and obtains the difference information. Output to the orthogonal conversion unit 205.

For example, in the case of an image to be intra-encoded, the calculation unit 204 subtracts the prediction image supplied from the intra prediction unit 217 from the image output from the screen rearrangement buffer 203. Further, for example, in the case of an image to be intercoded, the calculation unit 204 subtracts the prediction image supplied from the motion prediction / compensation unit 218 from the image output from the screen rearrangement buffer 203.

The orthogonal transform unit 205 performs orthogonal transforms such as discrete cosine transform and Karhunen-Loève transform on the difference information supplied from the arithmetic unit 204, and supplies the conversion coefficients to the quantization unit 206.

The quantization unit 206 quantizes the conversion coefficient output by the orthogonal conversion unit 205. The quantized unit 206 supplies the quantized conversion coefficient to the lossless coding unit 207.

The lossless coding unit 207 applies lossless coding such as variable length coding and arithmetic coding to the quantized conversion coefficient.

The lossless coding unit 207 acquires parameters such as information indicating the intra prediction mode from the intra prediction unit 217, and acquires parameters such as information indicating the inter prediction mode and motion vector information from the motion prediction / compensation unit 218.

The lossless coding unit 207 encodes the quantized conversion coefficient and encodes each acquired parameter (syntax element) to be a part (multiplex) of the header information of the coded data. The lossless coding unit 207 supplies the coded data obtained by coding to the storage buffer 208 and stores it.

For example, in the lossless coding unit 207, lossless coding processing such as variable length coding or arithmetic coding is performed. Examples of variable-length coding include CAVLC (Context-Adaptive Variable Length Coding). Examples of arithmetic coding include CABAC (Context-Adaptive Binary Arithmetic Coding).

The storage buffer 208 temporarily holds the coded stream (Encoded Data) supplied from the reversible coding unit 207, and at a predetermined timing, as a coded image, for example, not shown in the subsequent stage. Output to a recording device or transmission line. That is, the storage buffer 208 is also a transmission unit that transmits a coded stream.

Further, the conversion coefficient quantized in the quantization unit 206 is also supplied to the inverse quantization unit 209. The dequantization unit 209 dequantizes the quantized conversion coefficient by a method corresponding to the quantization by the quantization unit 206. The inverse quantization unit 209 supplies the obtained conversion coefficient to the inverse orthogonal conversion unit 210.

The inverse orthogonal conversion unit 210 performs inverse orthogonal conversion of the supplied conversion coefficient by a method corresponding to the orthogonal conversion processing by the orthogonal conversion unit 205. The inverse orthogonally converted output (restored difference information) is supplied to the calculation unit 211.

The calculation unit 211 supplies the inverse orthogonal conversion result supplied from the inverse orthogonal conversion unit 210, that is, the restored difference information from the intra prediction unit 217 or the motion prediction / compensation unit 218 via the prediction image selection unit 219. The predicted images are added to obtain a locally decoded image (decoded image).

For example, when the difference information corresponds to an image to be intra-encoded, the calculation unit 211 adds the prediction image supplied from the intra prediction unit 217 to the difference information. Further, for example, when the difference information corresponds to an image to be inter-encoded, the calculation unit 211 adds the predicted image supplied from the motion prediction / compensation unit 218 to the difference information.

The decoded image that is the addition result is supplied to the deblocking filter 212 and the frame memory 215.

The deblocking filter 212 suppresses the block distortion of the decoded image by appropriately performing the deblocking filter processing on the image from the calculation unit 211, and supplies the filter processing result to the adaptive offset filter 213. The deblocking filter 212 has parameters β and Tc obtained based on the quantization parameter QP. The parameters β and Tc are threshold values (parameters) used for determining the deblocking filter.

Note that the parameters β and Tc of the deblocking filter 212 are extended from β and Tc specified by the HEVC method. Each offset of the parameters β and Tc is encoded by the lossless coding unit 207 as a parameter of the deblocking filter and transmitted to the image decoding device 301 of FIG. 11 to be described later.

The adaptive offset filter 213 mainly performs an offset filter (SAO: Sample adaptive offset) process that suppresses ringing on the image after filtering by the deblocking filter 212.

There are 9 types of offset filters: 2 types of band offset, 6 types of edge offset, and no offset. The adaptive offset filter 213 uses a quad-tree structure in which the type of offset filter is determined for each divided area and an offset value for each divided area to filter the image after filtering by the deblocking filter 212. Apply processing. The adaptive offset filter 213 supplies the filtered image to the adaptive loop filter 214.

In the image coding apparatus 201, the quad-tree structure and the offset value for each divided region are calculated and used by the adaptive offset filter 213. The calculated quad-tree structure and the offset value for each divided region are encoded by the lossless coding unit 207 as adaptive offset parameters and transmitted to the image decoding device 301 of FIG. 11 to be described later.

The adaptive loop filter 214 performs adaptive loop filter (ALF: Adaptive Loop Filter) processing for each processing unit on the image after filtering by the adaptive offset filter 213, using the filter coefficient. In the adaptive loop filter 214, for example, a two-dimensional Wiener filter is used as the filter. Of course, a filter other than the Wiener filter may be used. The adaptive loop filter 214 supplies the filter processing result to the frame memory 215.

Although not shown in the example of FIG. 9, in the image coding apparatus 201, the filter coefficient is an adaptive loop filter 214 for each processing unit so as to minimize the residual with the original image from the screen rearrangement buffer 203. It is calculated and used by. The calculated filter coefficient is encoded by the lossless coding unit 207 as an adaptive loop filter parameter and transmitted to the image decoding device 301 of FIG. 11, which will be described later.

The frame memory 215 outputs the stored reference image to the intra prediction unit 217 or the motion prediction / compensation unit 218 via the selection unit 216 at a predetermined timing.

For example, in the case of an image to be intra-encoded, the frame memory 215 supplies the reference image to the intra-prediction unit 217 via the selection unit 216. Further, for example, when intercoding is performed, the frame memory 215 supplies the reference image to the motion prediction / compensation unit 218 via the selection unit 216.

When the reference image supplied from the frame memory 215 is an image to be intra-encoded, the selection unit 216 supplies the reference image to the intra prediction unit 217. Further, when the reference image supplied from the frame memory 215 is an image to be intercoded, the selection unit 216 supplies the reference image to the motion prediction / compensation unit 218.

The intra prediction unit 217 performs intra prediction (in-screen prediction) that generates a prediction image using the pixel values in the screen. The intra prediction unit 217 performs intra prediction in a plurality of modes (intra prediction mode).

The intra prediction unit 217 generates prediction images in all intra prediction modes, evaluates each prediction image, and selects the optimum mode. When the optimum intra prediction mode is selected, the intra prediction unit 217 supplies the prediction image generated in the optimum mode to the calculation unit 204 and the calculation unit 211 via the prediction image selection unit 219.

Further, as described above, the intra prediction unit 217 appropriately supplies parameters such as intra prediction mode information indicating the adopted intra prediction mode to the lossless coding unit 207.

The motion prediction / compensation unit 218 uses the input image supplied from the screen rearrangement buffer 203 and the reference image supplied from the frame memory 215 via the selection unit 216 for the image to be intercoded. Predict movement. Further, the motion prediction / compensation unit 218 performs motion compensation processing according to the motion vector detected by the motion prediction, and generates a prediction image (inter-prediction image information).

The motion prediction / compensation unit 218 performs inter-prediction processing in all candidate inter-prediction modes and generates a prediction image. The motion prediction / compensation unit 218 supplies the generated predicted image to the calculation unit 204 and the calculation unit 211 via the prediction image selection unit 219. Further, the motion prediction / compensation unit 218 supplies parameters such as inter-prediction mode information indicating the adopted inter-prediction mode and motion vector information indicating the calculated motion vector to the reversible coding unit 207.

The prediction image selection unit 219 supplies the output of the intra prediction unit 217 to the calculation unit 204 and the calculation unit 211 in the case of an image to be intra-encoded, and in the case of an image to be inter-encoded, the motion prediction / compensation unit 218 of the prediction image selection unit 219. The output is supplied to the calculation unit 204 and the calculation unit 211.

The rate control unit 220 controls the rate of the quantization operation of the quantization unit 206 based on the compressed image stored in the storage buffer 208 so that overflow or underflow does not occur.

The image coding device 201 is configured in this way, the lossless coding unit 207 corresponds to the coding unit 22 of FIG. 4, and the motion prediction / compensation unit 218 corresponds to the inter prediction unit 21 of FIG. .. Further, the motion prediction / compensation unit 218 also corresponds to the setting unit 23 in FIG. 4, and can supply the above-mentioned identification information, existence identification information, and the like to the lossless coding unit 207. Therefore, as described above, the image coding device 201 can suppress more subjective deterioration of image quality and deterioration of coding efficiency.

<Operation of image coding device>
With reference to FIG. 10, the flow of the coding process executed by the image coding apparatus 201 as described above will be described.

In step S101, the A / D conversion unit 202 A / D-converts the input image.

In step S102, the screen rearrangement buffer 203 stores the A / D-converted images by the A / D conversion unit 202, and rearranges the images from the display order of each picture to the coding order.

When the image to be processed supplied from the screen rearrangement buffer 203 is an image of a block to be intra-processed, the referenced decoded image is read from the frame memory 215 and is read from the frame memory 215 and is passed through the selection unit 216 to the intra prediction unit. It is supplied to 217.

Based on these images, in step S103, the intra-prediction unit 217 intra-predicts the pixels of the block to be processed in all the candidate intra-prediction modes. As the decoded pixel to be referred to, a pixel not filtered by the deblocking filter 212 is used.

By this process, intra-prediction is performed in all candidate intra-prediction modes, and cost function values are calculated for all candidate intra-prediction modes. Then, the optimum intra prediction mode is selected based on the calculated cost function value, and the prediction image generated by the intra prediction of the optimum intra prediction mode and the cost function value thereof are supplied to the prediction image selection unit 219.

When the image to be processed supplied from the screen rearrangement buffer 203 is an interprocessed image, the referenced image is read from the frame memory 215 and supplied to the motion prediction / compensation unit 218 via the selection unit 216. Will be done. Based on these images, in step S104, the motion prediction / compensation unit 218 performs motion prediction / compensation processing.

By this processing, motion prediction processing is performed in all candidate inter-prediction modes, cost function values are calculated for all candidate inter-prediction modes, and optimal inter-prediction is calculated based on the calculated cost function values. The mode is determined. Then, the predicted image generated by the optimum inter prediction mode and the cost function value thereof are supplied to the predicted image selection unit 219.

In step S105, the prediction image selection unit 219 optimizes one of the optimum intra prediction mode and the optimum inter prediction mode based on each cost function value output from the intra prediction unit 217 and the motion prediction / compensation unit 218. Determine to predict mode. Then, the prediction image selection unit 219 selects the determined prediction image of the optimum prediction mode and supplies it to the

calculation units

204 and 211. This predicted image is used for the calculation of steps S106 and S111 described later.

The selection information of this prediction image is supplied to the intra prediction unit 217 or the motion prediction / compensation unit 218. When the prediction image of the optimum intra prediction mode is selected, the intra prediction unit 217 supplies information indicating the optimum intra prediction mode (that is, parameters related to the intra prediction) to the lossless coding unit 207.

When the prediction image of the optimum inter prediction mode is selected, the motion prediction / compensation unit 218 reversibly encodes the information indicating the optimum inter prediction mode and the information corresponding to the optimum inter prediction mode (that is, the parameters related to the motion prediction). Output to unit 207. Examples of the information according to the optimum inter-prediction mode include motion vector information and reference frame information.

In step S106, the calculation unit 204 calculates the difference between the images sorted in step S102 and the predicted image selected in step S105. The predicted image is supplied to the calculation unit 204 from the motion prediction / compensation unit 218 in the case of inter-prediction and from the intra-prediction unit 217 in the case of intra-prediction via the prediction image selection unit 219.

The amount of difference data is smaller than that of the original image data. Therefore, the amount of data can be compressed as compared with the case where the image is encoded as it is.

In step S107, the orthogonal conversion unit 205 orthogonally converts the difference information supplied from the calculation unit 204. Specifically, orthogonal transforms such as the discrete cosine transform and the Karhunen-Loève transform are performed, and the transform coefficients are output.

In step S108, the quantization unit 206 quantizes the conversion coefficient. In this quantization, the rate is controlled as described in the process of step S118 described later.

The difference information quantized as described above is locally decoded as follows. That is, in step S109, the inverse quantization unit 209 dequantizes the conversion coefficient quantized by the quantization unit 206 with a characteristic corresponding to the characteristic of the quantization unit 206. In step S110, the inverse orthogonal conversion unit 210 performs inverse orthogonal conversion of the conversion coefficient inversely quantized by the inverse quantization unit 209 with a characteristic corresponding to the characteristic of the orthogonal conversion unit 205.

In step S111, the calculation unit 211 adds the predicted image input via the predicted image selection unit 219 to the locally decoded difference information, and locally decodes (that is, locally decoded) the image. (Image corresponding to the input to the calculation unit 204) is generated.

In step S112, the deblocking filter 212 performs a deblocking filter process on the image output from the calculation unit 211. At this time, as the threshold value for the determination regarding the deblocking filter, the parameters β and Tc extended from β and Tc defined by the HEVC method are used. The filtered image from the deblocking filter 212 is output to the adaptive offset filter 213.

Note that each offset of the parameters β and Tc used in the deblocking filter 212, which is input by the user by operating the operation unit or the like, is supplied to the reversible coding unit 207 as a parameter of the deblocking filter.

In step S113, the adaptive offset filter 213 performs adaptive offset filter processing. By this processing, the filter processing is performed on the image after filtering by the deblocking filter 212 by using the quad-tree structure in which the type of the offset filter is determined for each divided area and the offset value for each divided area. Be given. The filtered image is fed to the adaptive loop filter 214.

The determined quad-tree structure and the offset value for each divided region are supplied to the lossless coding unit 207 as an adaptive offset parameter.

In step S114, the adaptive loop filter 214 performs adaptive loop filter processing on the image after filtering by the adaptive offset filter 213. For example, the image after filtering by the adaptive offset filter 213 is filtered for each processing unit by using the filter coefficient, and the filter processing result is supplied to the frame memory 215.

In step S115, the frame memory 215 stores the filtered image. Images that have not been filtered by the deblocking filter 212, the adaptive offset filter 213, and the adaptive loop filter 214 are also supplied and stored in the frame memory 215 from the calculation unit 211.

On the other hand, the conversion coefficient quantized in step S108 described above is also supplied to the lossless coding unit 207. In step S116, the lossless coding unit 207 encodes the quantized conversion coefficient output from the quantizing unit 206 and each supplied parameter. That is, the difference image is losslessly coded and compressed by variable length coding, arithmetic coding, and the like. Here, each of the encoded parameters includes a deblocking filter parameter, an adaptive offset filter parameter, an adaptive loop filter parameter, a quantization parameter, motion vector information and reference frame information, prediction mode information, and the like.

In step S117, the storage buffer 208 stores the coded difference image (that is, the coded stream) as a compressed image. The compressed image stored in the storage buffer 208 is appropriately read out and transmitted to the decoding side via the transmission line.

In step S118, the rate control unit 220 controls the rate of the quantization operation of the quantization unit 206 based on the compressed image stored in the storage buffer 208 so that overflow or underflow does not occur.

When the process of step S118 is completed, the coding process is completed.

In the above coding process, when the motion prediction / compensation unit 218 performs the motion prediction / compensation process in step S104 to generate a prediction image, the color difference optical flow with respect to the color difference components Cb and Cr of the current prediction block. The process is applied. Further, in step S104, the motion prediction / compensation unit 218 sets identification information for identifying whether to perform motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be encoded. ..

<Configuration example of image decoding device>
FIG. 11 shows the configuration of an embodiment of an image decoding device as an image processing device to which the present disclosure is applied. The image decoding device 301 shown in FIG. 11 is a decoding device corresponding to the image coding device 201 of FIG.

It is assumed that the encoded stream (Encoded Data) encoded by the image coding device 201 is transmitted to the image decoding device 301 corresponding to the image coding device 201 via a predetermined transmission line and decoded. ..

As shown in FIG. 11, the image decoding device 301 includes a storage buffer 302, a reversible decoding unit 303, an inverse quantization unit 304, an inverse orthogonal conversion unit 305, an arithmetic unit 306, a deblocking filter 307, an adaptive offset filter 308, and an adaptive device. It has a loop filter 309, a screen sorting buffer 310, a D / A conversion unit 311, a frame memory 312, a selection unit 313, an intra prediction unit 314, a motion prediction / compensation unit 315, and a selection unit 316.

The storage buffer 302 is also a receiving unit that receives the transmitted encoded data. The storage buffer 302 receives the transmitted coded data and stores it. This coded data is encoded by the image coding device 201. The lossless decoding unit 303 decodes the coded data read from the storage buffer 302 at a predetermined timing by a method corresponding to the coding method of the lossless coding unit 207 of FIG.

The reversible decoding unit 303 supplies parameters such as information indicating the decoded intra prediction mode to the intra prediction unit 314, and supplies parameters such as information indicating the inter prediction mode and motion vector information to the motion prediction / compensation unit 315. .. Further, the reversible decoding unit 303 supplies the decoded deblocking filter parameters to the deblocking filter 307, and supplies the decoded adaptive offset parameters to the adaptive offset filter 308.

The inverse quantization unit 304 dequantizes the coefficient data (quantization coefficient) obtained by decoding by the reversible decoding unit 303 by a method corresponding to the quantization method of the quantization unit 206 of FIG. That is, the inverse quantization unit 304 performs the inverse quantization of the quantization coefficient by the same method as the inverse quantization unit 209 of FIG. 9 using the quantization parameters supplied from the image coding device 201.

The inverse quantized unit 304 supplies the inverse quantized coefficient data, that is, the orthogonal conversion coefficient to the inverse orthogonal conversion unit 305. The inverse orthogonal conversion unit 305 is a method corresponding to the orthogonal conversion method of the orthogonal conversion unit 205 of FIG. 9, and the orthogonal conversion coefficient is inversely orthogonally converted to the residual data before the orthogonal conversion by the image coding apparatus 201. Obtain the corresponding decoding residual data.

The decoding residual data obtained by the inverse orthogonal conversion is supplied to the calculation unit 306. Further, the calculation unit 306 is supplied with a prediction image from the intra prediction unit 314 or the motion prediction / compensation unit 315 via the selection unit 316.

The calculation unit 306 adds the decoded residual data and the predicted image, and obtains the decoded image data corresponding to the image data before the predicted image is subtracted by the calculation unit 204 of the image coding device 201. The calculation unit 306 supplies the decoded image data to the deblocking filter 307.

The deblocking filter 307 suppresses the block distortion of the decoded image by appropriately performing the deblocking filter processing on the image from the calculation unit 306, and supplies the filter processing result to the adaptive offset filter 308. The deblocking filter 307 is basically configured in the same manner as the deblocking filter 212 of FIG. That is, the deblocking filter 307 has parameters β and Tc obtained based on the quantization parameters. The parameters β and Tc are threshold values used for determining the deblocking filter.

Note that the parameters β and Tc of the deblocking filter 307 are extended from β and Tc defined by the HEVC method. Each offset of the parameters β and Tc of the deblocking filter encoded by the image coding device 201 is received by the image decoding device 301 as a parameter of the deblocking filter, decoded by the reversible decoding unit 303, and deblocking. Used by filter 307.

The adaptive offset filter 308 mainly performs an offset filter (SAO) process for suppressing ringing on the image after filtering by the deblocking filter 307.

The adaptive offset filter 308 uses a quad-tree structure in which the type of offset filter is determined for each divided region and an offset value for each divided region to filter the image after filtering by the deblocking filter 307. Apply processing. The adaptive offset filter 308 supplies the filtered image to the adaptive loop filter 309.

The quad-tree structure and the offset value for each divided region are calculated by the adaptive offset filter 213 of the image coding device 201, encoded as an adaptive offset parameter, and sent. Then, the quad-tree structure encoded by the image coding device 201 and the offset value for each divided region are received by the image decoding device 301 as adaptive offset parameters, decoded by the reversible decoding unit 303, and the adaptive offset. Used by filter 308.

The adaptive loop filter 309 filters the image filtered by the adaptive offset filter 308 for each processing unit using the filter coefficient, and supplies the filter processing result to the frame memory 312 and the screen sorting buffer 310. To do.

Although not shown in the example of FIG. 11, in the image decoding device 301, the filter coefficient is calculated for each LUC by the adaptive loop filter 214 of the image coding device 201, and is encoded and sent as an adaptive loop filter parameter. What has been obtained is decoded by the reversible decoding unit 303 and used.

The screen sorting buffer 310 sorts the images and supplies them to the D / A conversion unit 311. That is, the order of the frames rearranged for the coding order by the screen rearrangement buffer 203 of FIG. 9 is rearranged in the original display order.

The D / A conversion unit 311 D / A-converts the image (Decoded Picture (s)) supplied from the screen rearrangement buffer 310, outputs it to a display (not shown), and displays it. The image may be output as digital data without providing the D / A conversion unit 311.

The output of the adaptive loop filter 309 is further supplied to the frame memory 312.

The frame memory 312, the selection unit 313, the intra prediction unit 314, the motion prediction / compensation unit 315, and the selection unit 316 are the frame memory 215, the selection unit 216, the intra prediction unit 217, and the motion prediction / compensation unit of the image encoding device 201. It corresponds to 218 and the prediction image selection unit 219, respectively.

The selection unit 313 reads the interprocessed image and the referenced image from the frame memory 312, and supplies the motion prediction / compensation unit 315. Further, the selection unit 313 reads the image used for the intra prediction from the frame memory 312 and supplies it to the intra prediction unit 314.

Information and the like indicating the intra prediction mode obtained by decoding the header information are appropriately supplied to the intra prediction unit 314 from the reversible decoding unit 303. Based on this information, the intra prediction unit 314 generates a prediction image from the reference image acquired from the frame memory 312, and supplies the generated prediction image to the selection unit 316.

Information (prediction mode information, motion vector information, reference frame information, flags, various parameters, etc.) obtained by decoding the header information is supplied to the motion prediction / compensation unit 315 from the reversible decoding unit 303.

The motion prediction / compensation unit 315 generates a prediction image from the reference image acquired from the frame memory 312 based on the information supplied from the reversible decoding unit 303, and supplies the generated prediction image to the selection unit 316.

The selection unit 316 selects the prediction image generated by the motion prediction / compensation unit 315 or the intra prediction unit 314 and supplies it to the calculation unit 306.

The image decoding device 301 is configured in this way, the reversible decoding unit 303 corresponds to the decoding unit 32 of FIG. 4, and the motion prediction / compensation unit 315 corresponds to the inter prediction unit 31 of FIG. Further, the reversible decoding unit 303 also corresponds to the parsing unit 33 and the control unit 34 in FIG. 4, and controls whether to perform motion compensation processing to which the optical flow processing is applied by parsing the above-mentioned identification information and existence identification information. can do. Therefore, as described above, the image decoding device 301 can suppress more subjective deterioration of image quality and deterioration of coding efficiency.

<Operation of image decoding device>
An example of the flow of the decoding process executed by the image decoding apparatus 301 as described above will be described with reference to FIG.

When the decoding process is started, in step S201, the storage buffer 302 receives the transmitted coded stream (data) and stores it. In step S202, the reversible decoding unit 303 decodes the coded data supplied from the storage buffer 302. The I picture, P picture, and B picture encoded by the lossless coding unit 207 of FIG. 9 are decoded.

Prior to decoding the picture, parameter information such as motion vector information, reference frame information, and prediction mode information (intra prediction mode or inter prediction mode) is also decoded.

When the prediction mode information is the intra prediction mode information, the prediction mode information is supplied to the intra prediction unit 314. When the prediction mode information is inter-prediction mode information, the motion vector information corresponding to the prediction mode information is supplied to the motion prediction / compensation unit 315. In addition, the parameters of the deblocking filter and the adaptive offset parameter are also decoded and supplied to the deblocking filter 307 and the adaptive offset filter 308, respectively.

In step S203, the intra prediction unit 314 or the motion prediction / compensation unit 315 each performs a prediction image generation process corresponding to the prediction mode information supplied from the reversible decoding unit 303.

That is, when the intra prediction mode information is supplied from the reversible decoding unit 303, the intra prediction unit 314 generates an intra prediction image of the intra prediction mode. When the inter-prediction mode information is supplied from the reversible decoding unit 303, the motion prediction / compensation unit 315 performs the motion prediction / compensation processing in the inter-prediction mode and generates the inter-prediction image.

By this process, the prediction image (intra prediction image) generated by the intra prediction unit 314 or the prediction image (inter prediction image) generated by the motion prediction / compensation unit 315 is supplied to the selection unit 316.

In step S204, the selection unit 316 selects the predicted image. That is, the prediction image generated by the intra prediction unit 314 or the prediction image generated by the motion prediction / compensation unit 315 is supplied. Therefore, the supplied predicted image is selected and supplied to the calculation unit 306, and is added to the output of the inverse orthogonal conversion unit 305 in step S207 described later.

The conversion coefficient decoded by the reversible decoding unit 303 in step S202 described above is also supplied to the inverse quantization unit 304. In step S205, the inverse quantization unit 304 dequantizes the conversion coefficient decoded by the reversible decoding unit 303 with a characteristic corresponding to the characteristic of the quantization unit 206 of FIG.

In step S206, the inverse orthogonal conversion unit 305 performs inverse orthogonal conversion of the conversion coefficient inversely quantized by the inverse quantization unit 304 with a characteristic corresponding to the characteristic of the orthogonal conversion unit 205 of FIG. As a result, the difference information corresponding to the input of the orthogonal conversion unit 205 (output of the calculation unit 204) of FIG. 9 is decoded.

In step S207, the calculation unit 306 adds the predicted image selected in the process of step S204 described above and input via the selection unit 316 to the difference information. This decodes the original image.

In step S208, the deblocking filter 307 performs a deblocking filter process on the image output from the calculation unit 306. At this time, as the threshold value for the determination regarding the deblocking filter, the parameters β and Tc extended from β and Tc defined by the HEVC method are used. The filtered image from the deblocking filter 307 is output to the adaptive offset filter 308. In the deblocking filter processing, the offsets of the parameters β and Tc of the deblocking filter supplied from the reversible decoding unit 303 are also used.

In step S209, the adaptive offset filter 308 performs adaptive offset filter processing. By this processing, the filter processing is performed on the image after filtering by the deblocking filter 307 using the quad-tree structure in which the type of the offset filter is determined for each divided area and the offset value for each divided area. Be given. The filtered image is fed to the adaptive loop filter 309.

In step S210, the adaptive loop filter 309 performs adaptive loop filter processing on the image after filtering by the adaptive offset filter 308. The adaptive loop filter 309 performs filter processing for each processing unit on the input image using the filter coefficient calculated for each processing unit, and supplies the filter processing result to the screen sorting buffer 310 and the frame memory 312. To do.

In step S211 the frame memory 312 stores the filtered image.

In step S212, the screen rearrangement buffer 310 rearranges the images after the adaptive loop filter 309 and then supplies the screen rearrangement buffer 310 to the D / A conversion unit 311. That is, the order of the frames sorted for coding by the screen sort buffer 203 of the image coding device 201 is rearranged to the original display order.

In step S213, the D / A conversion unit 311 D / A-converts the images sorted by the screen sorting buffer 310 and outputs them to a display (not shown), and the images are displayed.

When the process of step S213 is completed, the decoding process is completed.

In the decoding process as described above, when the motion prediction / compensation unit 315 performs the motion prediction / compensation process in step S203 to generate a predicted image, the color difference optical flow process is performed on the color difference components Cb and Cr of the current prediction block. Is given. Further, in step S202, the reversible decoding unit 303 parses the identification information from the bit stream and controls whether to perform the motion compensation processing to which the optical flow processing is applied by referring to the identification information.

<Computer configuration example>
Next, the series of processes (image processing method) described above can be performed by hardware or software. When a series of processes is performed by software, the programs constituting the software are installed on a general-purpose computer or the like.

FIG. 13 is a block diagram showing a configuration example of an embodiment of a computer in which a program for executing the above-mentioned series of processes is installed.

The program can be recorded in advance on the hard disk 1005 or ROM 1003 as a recording medium built in the computer.

Alternatively, the program can be stored (recorded) in the removable recording medium 1011 driven by the drive 1009. Such a removable recording medium 1011 can be provided as so-called package software. Here, examples of the removable recording medium 1011 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, a semiconductor memory, and the like.

The program can be installed on the computer from the removable recording medium 1011 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 1005. That is, for example, the program transfers wirelessly from a download site to a computer via an artificial satellite for digital satellite broadcasting, or transfers to a computer by wire via a network such as LAN (Local Area Network) or the Internet. be able to.

The computer has a built-in CPU (Central Processing Unit) 1002, and the input / output interface 1010 is connected to the CPU 1002 via the bus 1001.

When a command is input by the user by operating the input unit 1007 via the input / output interface 1010, the CPU 1002 executes a program stored in the ROM (Read Only Memory) 1003 accordingly. .. Alternatively, the CPU 1002 loads the program stored in the hard disk 1005 into the RAM (Random Access Memory) 1004 and executes it.

As a result, the CPU 1002 performs the processing according to the above-mentioned flowchart or the processing performed according to the above-mentioned block diagram configuration. Then, the CPU 1002 outputs the processing result from the output unit 1006 or transmits it from the communication unit 1008, and further records it on the hard disk 1005, if necessary, via the input / output interface 1010.

The input unit 1007 is composed of a keyboard, a mouse, a microphone, and the like. Further, the output unit 1006 is composed of an LCD (Liquid Crystal Display), a speaker, or the like.

Here, in the present specification, the processing performed by the computer according to the program does not necessarily have to be performed in chronological order in the order described as the flowchart. That is, the processing performed by the computer according to the program also includes processing executed in parallel or individually (for example, parallel processing or processing by an object).

Further, the program may be processed by one computer (processor) or may be distributed processed by a plurality of computers. Further, the program may be transferred to a distant computer and executed.

Further, in the present specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..

Further, for example, the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). On the contrary, the configurations described above as a plurality of devices (or processing units) may be collectively configured as one device (or processing unit). Further, of course, a configuration other than the above may be added to the configuration of each device (or each processing unit). Further, if the configuration and operation of the entire system are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit). ..

Further, for example, this technology can have a cloud computing configuration in which one function is shared and jointly processed by a plurality of devices via a network.

Also, for example, the above-mentioned program can be executed in any device. In that case, the device may have necessary functions (functional blocks, etc.) so that necessary information can be obtained.

Further, for example, each step described in the above flowchart can be executed by one device or can be shared and executed by a plurality of devices. Further, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by one device or shared by a plurality of devices. In other words, a plurality of processes included in one step can be executed as processes of a plurality of steps. On the contrary, the processes described as a plurality of steps can be collectively executed as one step.

The program executed by the computer may be such that the processing of the steps for describing the program is executed in chronological order in the order described in this specification, or is called in parallel or in parallel. It may be executed individually at a necessary timing such as time. That is, as long as there is no contradiction, the processing of each step may be executed in an order different from the above-mentioned order. Further, the processing of the step for writing this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

It should be noted that the present techniques described in the present specification can be independently implemented independently as long as there is no contradiction. Of course, any plurality of the present technologies can be used in combination. For example, some or all of the techniques described in any of the embodiments may be combined with some or all of the techniques described in other embodiments. It is also possible to carry out a part or all of any of the above-mentioned techniques in combination with other techniques not described above.

<Example of configuration combination>
The present technology can also have the following configurations.
(1)
A setting unit that sets identification information that identifies whether to perform motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be coded.
An image processing device including a coding unit that encodes the image and generates a bit stream including identification information set by the setting unit.
(2)
The image processing apparatus according to (1) above, wherein the setting unit sets identification information for identifying whether to perform motion compensation processing to which the optical flow processing is applied for a unit to be encoded.
(3)
The setting unit sets identification information for identifying whether to perform motion compensation processing to which the optical flow processing is applied for a unit to be encoded as a header of the bit stream.
The image processing apparatus according to (2) above, wherein the coding unit generates a bit stream including the identification information as a header syntax.
(4)
The unit to be encoded is a slice. The image processing apparatus according to (3) above.
(5)
The setting unit sets existence identification information for identifying that the identification information is included in the bit stream.
The image processing apparatus according to (3) above, wherein the coding unit generates a bit stream including existence identification information set by the setting unit.
(6)
The setting unit sets the existence identification information as a sequence parameter set, and sets the existence identification information.
The image processing apparatus according to (5) above, wherein the coding unit generates a bit stream including the existence identification information set by the setting unit as a sequence parameter set.
(7)
The image processing device
Setting identification information that identifies whether to perform motion compensation processing to which optical flow processing is applied for the pixels of the current prediction block of the image to be coded, and
An image processing method including encoding the image to generate a bitstream containing the set identification information.
(8)
A parse unit that parses the identification information from a bit stream that includes identification information that identifies whether to perform motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be decoded.
With reference to the identification information parsed by the parse unit, a control unit that controls whether to perform motion compensation processing to which optical flow processing is applied, and a control unit.
An image processing device including a decoding unit that decodes the current prediction block using a prediction image generated by motion compensation processing controlled by the control unit.
(9)
The control unit controls to perform the motion compensation process to which the optical flow process is applied only when the identification information indicates that the motion compensation process to which the optical flow process is applied is performed. Image processing equipment.
(10)
The control unit controls to skip the motion compensation process to which the optical flow process is applied when the identification information indicates that the motion compensation process to which the optical flow process is applied is not performed. The image processing apparatus described.
(11)
The image processing apparatus according to (8) above, wherein the identification information is information for identifying whether or not motion compensation processing to which the optical flow processing is applied is performed on a unit to be encoded.
(12)
The identification information is included as a header of the bitstream.
The image processing apparatus according to (11) above, wherein the parsing unit parses the header of the bit stream.
(13)
The image processing apparatus according to (12) above, wherein the unit to be encoded is a slice.
(14)
The bitstream includes existence identification information that identifies that the identification information is included in the bitstream.
The image processing apparatus according to (12) above, wherein the parsing unit parses the identification information only when the existence identification information included in the bit stream indicates that the identification information is included.
(15)
The bitstream contains the presence identification information as a sequence parameter set.
The image processing apparatus according to (14) above, wherein the parsing unit parses the existence identification information included in the bit stream by setting sequence parameters.
(16)
The image processing device
Parsing the identification information from a bit stream containing identification information that identifies whether to perform motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be decoded.
By referring to the parsed identification information, it is possible to control whether or not motion compensation processing to which optical flow processing is applied is performed.
An image processing method including decoding the current prediction block using a prediction image generated by controlled motion compensation processing.

Note that the present embodiment is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present disclosure. Further, the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

11 image processing system, 12 image coding device, 13 image decoding device, 21 inter-prediction unit, 22 coding unit, 23 setting unit, 31 inter-prediction unit, 32 decoding unit

Claims

A setting unit that sets identification information that identifies whether to perform motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be coded.
An image processing device including a coding unit that encodes the image and generates a bit stream including identification information set by the setting unit.
The image processing apparatus according to claim 1, wherein the setting unit sets identification information for identifying whether to perform motion compensation processing to which the optical flow processing is applied for a unit to be encoded.
The setting unit sets identification information for identifying whether to perform motion compensation processing to which the optical flow processing is applied for a unit to be encoded as a header of the bit stream.
The image processing apparatus according to claim 2, wherein the coding unit generates a bit stream including the identification information as a header syntax.
The image processing apparatus according to claim 3, wherein the unit to be encoded is a slice.
The setting unit sets existence identification information for identifying that the identification information is included in the bit stream.
The image processing apparatus according to claim 3, wherein the coding unit generates a bit stream including existence identification information set by the setting unit.
The setting unit sets the existence identification information as a sequence parameter set, and sets the existence identification information.
The image processing apparatus according to claim 5, wherein the coding unit generates a bit stream including the existence identification information set by the setting unit as a sequence parameter set.
The image processing device
Setting identification information that identifies whether to perform motion compensation processing to which optical flow processing is applied for the pixels of the current prediction block of the image to be coded, and
An image processing method including encoding the image to generate a bitstream containing the set identification information.
A parsing unit that parses the identification information from a bit stream that includes identification information that identifies whether to perform motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be decoded.
A control unit that controls whether or not motion compensation processing to which optical flow processing is applied is performed with reference to the identification information parsed by the parsing unit.
An image processing device including a decoding unit that decodes the current prediction block using a prediction image generated by motion compensation processing controlled by the control unit.
The eighth aspect of the present invention, wherein the control unit controls to perform the motion compensation process to which the optical flow process is applied only when the identification information indicates that the motion compensation process to which the optical flow process is applied is performed. Image processing device.
The ninth aspect of claim 9 is that the control unit controls to skip the motion compensation process to which the optical flow process is applied when the identification information indicates that the motion compensation process to which the optical flow process is applied is not performed. Image processing equipment.
The image processing apparatus according to claim 8, wherein the identification information is information for identifying whether or not motion compensation processing to which the optical flow processing is applied is performed on a unit to be encoded.
The identification information is included as a header of the bitstream.
The image processing apparatus according to claim 11, wherein the parsing unit parses the header of the bit stream.
The image processing apparatus according to claim 12, wherein the unit to be encoded is a slice.
The bitstream includes existence identification information that identifies that the identification information is included in the bitstream.
The image processing apparatus according to claim 12, wherein the parsing unit parses the identification information only when the existence identification information included in the bit stream indicates that the identification information is included.
The bitstream contains the presence identification information as a sequence parameter set.
The image processing apparatus according to claim 14, wherein the parsing unit parses the existence identification information included in the bit stream by setting sequence parameters.
The image processing device
Parsing the identification information from a bit stream containing identification information that identifies whether to perform motion compensation processing to which optical flow processing is applied to the pixels of the current prediction block of the image to be decoded.
By referring to the parsed identification information, it is possible to control whether or not motion compensation processing to which optical flow processing is applied is performed.
An image processing method including decoding the current prediction block using a prediction image generated by controlled motion compensation processing.