WO2013154026A1

WO2013154026A1 - Image processing apparatus and method

Info

Publication number: WO2013154026A1
Application number: PCT/JP2013/060362
Authority: WO
Inventors: 佐藤　数史
Original assignee: ソニー株式会社
Priority date: 2012-04-13
Filing date: 2013-04-04
Publication date: 2013-10-17

Abstract

The present disclosure relates to an image processing apparatus and method capable of reducing the information sent to a decoding side and improving coding efficiency. Whether to apply an edge offset (EO) or a band offset (BO) is set by a classification unit depending on the TU size from an orthogonal transformation unit. Therefore, an adaptation offset unit determines whether to perform an offset or not (on/off status of an adaptation offset filter), and if it is determined that offset is to be performed, the category and offset value in the type of the offset set by the classification unit are obtained, and offset processing is performed with respect to a decoded image from a calculation unit. The present disclosure can be applied, for example, to an image processing apparatus.

Description

Image processing apparatus and method

The present disclosure relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method capable of reducing information to be sent to a decoding side and improving encoding efficiency.

In recent years, image information has been handled as digital data, and at that time, for the purpose of efficient transmission and storage of information, encoding is performed by orthogonal transform such as discrete cosine transform and motion compensation using redundancy unique to image information. An apparatus that employs a method to compress and code an image is becoming widespread. Examples of this encoding method include MPEG (Moving Picture Experts Group).

In particular, MPEG2 (ISO / IEC 13818-2) is defined as a general-purpose image encoding system, and is a standard that covers both interlaced scanning images and progressive scanning images, as well as standard resolution images and high-definition images. For example, MPEG2 is currently widely used in a wide range of applications for professional and consumer applications. By using the MPEG2 compression method, for example, a code amount (bit rate) of 4 to 8 Mbps is assigned to an interlaced scanned image having a standard resolution of 720 × 480 pixels. Further, by using the MPEG2 compression method, for example, a high resolution interlaced scanned image having 1920 × 1088 pixels is assigned a code amount (bit rate) of 18 to 22 Mbps. As a result, a high compression rate and good image quality can be realized.

MPEG2 was mainly intended for high-quality encoding suitable for broadcasting, but it did not support encoding methods with a lower code amount (bit rate) than MPEG1, that is, a higher compression rate. With the widespread use of mobile terminals, the need for such an encoding system is expected to increase in the future, and the MPEG4 encoding system has been standardized accordingly. Regarding the image coding system, the standard was approved as an international standard in December 1998 as ISO / IEC 14496-2.

The standardization schedule is H.03 in March 2003. H.264 and MPEG-4 Part 10 (Advanced Video Coding, hereinafter referred to as AVC).

Furthermore, this H. As an extension of H.264 / AVC, FRExt including RGB, 4: 2: 2, 4: 4: 4 coding tools necessary for business use, 8x8DCT and quantization matrix defined by MPEG-2 (FidelityFiRange Extension) standardization was completed in February 2005. This makes it possible to use AVC to encode film noise contained in movies well, and is used in a wide range of applications such as Blu-Ray Disc (trademark).

However, these days, we want to compress images with a resolution of 4000 x 2000 pixels, which is four times higher than high-definition images, or deliver high-definition images in a limited transmission capacity environment such as the Internet. There is a growing need for encoding. For this reason, in the above-described VCEG (= Video Coding Expert Group) under the ITU-T, studies on improving the coding efficiency are being continued.

As one such improvement in coding efficiency, a method having an FIR filter in a motion compensation loop has been proposed (see Non-Patent Document 1, for example). In the encoding device, this FIR filter coefficient is obtained by Wiener Filter so as to minimize the error from the input image, thereby minimizing deterioration in the reference image and encoding efficiency of the image compression information to be output. It is possible to improve.

And now H. It is called HEVC (High Efficiency) Video (Coding) by JCTVC (Joint Collaboration (Team-Video Coding)), which is a joint standardization organization of ITU-T and ISO / IEC, for the purpose of further improving encoding efficiency than H.264 / AVC. Standardization of the encoding method is underway. Regarding the HEVC standard, CommitteeCommitdraft, which is the first draft version specification, was issued in February 2012 (see Non-Patent Document 2, for example).

In this HEVC, a coding unit (CU (Coding Unit)) is defined as an encoding unit similar to a macroblock in AVC. Further, one coding unit can be divided into one or more prediction units (Prediction Unit: PU) that mean a unit of prediction processing. Then, intra prediction or inter prediction is performed for each prediction unit.

Also, one coding unit can be divided into one or more transform units (Transform Unit: TU) which means a unit of orthogonal transform. Then, for each transform unit, orthogonal transform from image data to transform coefficient data and quantization of transform coefficient data are performed.

For example, in Non-Patent Document 3, a short-distance intra prediction method that enables selection of a relatively small non-square prediction unit (for example, a linear or rectangular prediction unit) in the intra prediction mode. A method called "suggestion" is proposed. In this case, the shape of the conversion unit may also be a non-square according to the shape of the prediction unit.

By the way, in HEVC, a method called an adaptive offset filter proposed in Non-Patent Document 4 is adopted. In HEVC, the adaptive offset filter is provided between the deblocking filter and the adaptive loop filter.

¡There are two types of adaptive offsets called band offsets and six types called edge offsets, and it is also possible not to apply offsets. Then, the image can be divided into quad-trees, and the type of adaptive offset described above can be selected for each divided region. By using this method, encoding efficiency can be improved.

Incidentally, whether or not the block includes an edge has a correlation with the TU size and the PU size. That is, for example, for a flatter region, a larger TU size tends to be selected. For example, for a region including an edge, a smaller TU size tends to be selected.

However, in the method proposed in Non-Patent Document 3, it is determined by calculating a cost function value or the like which of the edge offset and the band offset is better without using such a correlation with the TU size. It was. Therefore, the amount of calculation is large, and information indicating whether an edge offset or a band offset is selected must be sent to the decoding side.

The present disclosure has been made in view of such a situation, and reduces information to be transmitted to the decoding side and improves encoding efficiency.

An image processing apparatus according to an aspect of the present disclosure includes a decoding unit that generates an image by decoding an encoded stream that is encoded in units having a hierarchical structure, and a block size of an image generated by the decoding unit, or An offset setting unit that sets an offset type for adaptive offset processing according to the area, and the adaptive offset processing for the image generated by the decoding unit with the offset type set by the offset setting unit. And an adaptive offset processing unit to perform.

The offset setting unit sets a band offset for the block when the size or area of the block is large, and sets an edge offset for the block when the size or area of the block is small. Can do.

The block is a TU (Transform Unit).

A receiver that receives the encoded stream and on / off information indicating whether the adaptive offset processing is on or off; and the decoder generates the image by decoding the encoded stream received by the receiver. The adaptive offset processing unit, when the on / off information received by the receiving unit is on of the adaptive offset processing, is an image generated by the decoding unit with the type of offset set by the offset setting unit. The adaptive offset processing can be performed on the target.

The offset setting unit sets a band offset when the size or area of the block is equal to or larger than the first size or the first area, and the block size or area is the first size or the first area. If it is smaller, an edge offset can be set.

The offset setting unit may set the adaptive offset processing to be off when the size or area of the block is equal to or larger than the first size or the second size or the second area larger than the first area. it can.

The offset setting unit can set an edge offset for the block when NSQT (Non-Square-Quadtree Transform) is applied to the block.

When NSQT (Non-Square Quadtree Transform) is applied to the block, the offset setting unit sets the type of offset of the adaptive offset processing according to the size or area of the short side of the block. Can do.

The block is an LCU (Largest Coding Unit), and the offset setting unit can set the type of offset of the adaptive offset processing according to the integration of the area of the sub-blocks included in the LCU.

According to an image processing method of one aspect of the present disclosure, an image processing device generates an image by decoding an encoded stream encoded in a unit having a hierarchical structure, and sets the size or area of a block of the generated image. Accordingly, the type of offset of the adaptive offset process is set, and the adaptive offset process is performed on the generated image with the set type of offset.

An image processing apparatus according to another aspect of the present disclosure includes an offset setting unit that sets an offset type of adaptive offset processing according to a size or area of a block of an image subjected to local decoding processing when an image is encoded, With the type of offset set by the offset setting unit, using the image as the target, the adaptive offset processing unit that performs the adaptive offset processing, and the image that has been subjected to the adaptive offset processing by the adaptive offset processing unit, An encoding unit that encodes the image in a unit having a hierarchical structure.

The block is a TU (Transform Unit).

A transmission unit that transmits an image encoded by the encoding unit; the adaptive offset processing unit determines whether the adaptive offset processing is on or off; and when the adaptive offset processing is on, the offset setting The adaptive offset processing is performed on the image with the type of offset set by the unit, and the transmission unit can transmit on / off information indicating whether the adaptive offset processing is on or off.

An image processing method according to another aspect of the present disclosure sets an offset type of adaptive offset processing according to a size or area of an image block subjected to local decoding processing when an image processing device encodes an image. Then, the adaptive offset processing is performed on the image with the set offset type, and the image is encoded in units having a hierarchical structure using the image on which the adaptive offset processing has been performed.

In one aspect of the present disclosure, an image is generated by decoding an encoded stream that is encoded in units having a hierarchical structure, and an offset of adaptive offset processing is performed according to the size or area of the block of the generated image Is set. Then, the adaptive offset processing is performed on the generated image with the set offset type.

In another aspect of the present disclosure, the type of offset of the adaptive offset process is set according to the size or area of the block of the image that has been locally decoded when the image is encoded. Then, the adaptive offset processing is performed on the image with the set offset type, and the image is encoded in a unit having a hierarchical structure using the image subjected to the adaptive offset processing. The

Note that the above-described image processing apparatus may be an independent apparatus, or may be an internal block constituting one image encoding apparatus or image decoding apparatus.

According to one aspect of the present disclosure, an image can be decoded. In particular, the information sent to the decoding side can be reduced and the encoding efficiency can be improved.

According to another aspect of the present disclosure, an image can be encoded. In particular, the information sent to the decoding side can be reduced and the encoding efficiency can be improved.

1 is a block diagram illustrating a main configuration example of an AVC image encoding device. FIG. It is a block diagram which shows the main structural examples of the image decoding apparatus of an AVC system. It is a block diagram which shows the main structural examples of the image coding apparatus to which an adaptive loop filter is applied. It is a block diagram which shows the main structural examples of the image decoding apparatus to which an adaptive loop filter is applied. It is a figure explaining the structural example of a coding unit. It is a figure explaining Non-Square | Quadtree | Transform. It is a figure explaining the adaptive offset process in a HEVC system. It is a figure explaining a quad-tree structure. It is a figure explaining a band offset. It is a figure explaining edge offset. It is a figure which shows the rule list of edge offset. It is a block diagram which shows the main structural examples of the image coding apparatus of this indication. It is a figure which shows the example of the relationship between TU size and an adaptive offset filter. It is a figure which shows the other example of the relationship between TU size and an adaptive offset filter. It is a figure explaining the case of LCU. It is a figure which shows the structural example of an orthogonal transformation part and an adaptive offset part. It is a flowchart explaining the example of the flow of an encoding process. It is a flowchart explaining the example of the flow of an adaptive offset process. It is a block diagram which shows the main structural examples of an image decoding apparatus. It is a block diagram which shows the structural example of an inverse orthogonal transformation part and an adaptive offset part. It is a flowchart explaining the example of the flow of a decoding process. It is a flowchart explaining the example of the flow of an adaptive offset process. It is a figure which shows the example of a multiview image encoding system. It is a figure which shows the main structural examples of the multiview image coding apparatus to which this technique is applied. It is a figure which shows the main structural examples of the multiview image decoding apparatus to which this technique is applied. It is a figure which shows the example of a hierarchy image coding system. It is a figure which shows the main structural examples of the hierarchy image coding apparatus to which this technique is applied. It is a figure which shows the main structural examples of the hierarchy image decoding apparatus to which this technique is applied. And FIG. 20 is a block diagram illustrating a main configuration example of a computer. It is a block diagram which shows an example of a schematic structure of a television apparatus. It is a block diagram which shows an example of a schematic structure of a mobile telephone. It is a block diagram which shows an example of a schematic structure of a recording / reproducing apparatus. It is a block diagram which shows an example of a schematic structure of an imaging device.

Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. 1. Description of conventional method First Embodiment (Image Encoding Device)
3. Second embodiment (image decoding apparatus)
4). Third embodiment (multi-view image encoding / multi-view image decoding apparatus)
5. Fourth embodiment (hierarchical image encoding / hierarchical image decoding apparatus)
6). Fifth embodiment (personal computer)
7). Application examples

<1. Description of conventional methods>
[AVC image encoding device]
FIG. 1 illustrates a configuration of an embodiment of an image encoding apparatus that encodes an image using an H.264 and MPEG (Moving Picture Experts Group) 4 Part 10 (AVC (Advanced Video Coding)) encoding method. Hereinafter, H.M. The H.264 and MPEG encoding methods are referred to as AVC methods.

In the example of FIG. 1, the image encoding device 1 includes an A / D conversion unit 11, a screen rearrangement buffer 12, a calculation unit 13, an orthogonal transformation unit 14, a quantization unit 15, a lossless encoding unit 16, an accumulation buffer 17, An inverse quantization unit 18, an inverse orthogonal transform unit 19, and a calculation unit 20 are included. The image encoding device 1 is also configured to include a deblock filter 21, a frame memory 22, a selection unit 23, an intra prediction unit 24, a motion prediction / compensation unit 25, a predicted image selection unit 26, and a rate control unit 27. Has been.

The A / D converter 11 A / D converts the input image data, outputs it to the screen rearrangement buffer 12, and stores it. The screen rearrangement buffer 12 rearranges the stored frame images in the display order in the order of frames for encoding in accordance with the GOP (Group of Picture) structure. The screen rearrangement buffer 12 supplies the image with the rearranged frame order to the arithmetic unit 13. The screen rearrangement buffer 12 also supplies the image in which the frame order has been rearranged to the intra prediction unit 24 and the motion prediction / compensation unit 25.

The calculation unit 13 subtracts the prediction image supplied from the intra prediction unit 24 or the motion prediction / compensation unit 25 via the prediction image selection unit 26 from the image read from the screen rearrangement buffer 12, and the difference information Is output to the orthogonal transform unit 14.

For example, in the case of an image on which intra coding is performed, the calculation unit 13 subtracts the prediction image supplied from the intra prediction unit 24 from the image read from the screen rearrangement buffer 12. For example, in the case of an image on which inter coding is performed, the calculation unit 13 subtracts the prediction image supplied from the motion prediction / compensation unit 25 from the image read from the screen rearrangement buffer 12.

The orthogonal transform unit 14 performs orthogonal transform such as discrete cosine transform and Karhunen-Loeve transform on the difference information supplied from the computation unit 13 and supplies the transform coefficient to the quantization unit 15.

The quantization unit 15 quantizes the transform coefficient output from the orthogonal transform unit 14. The quantization unit 15 sets a quantization parameter based on the information regarding the target value of the code amount supplied from the rate control unit 27, and performs quantization. The quantization unit 15 supplies the quantized transform coefficient to the lossless encoding unit 16.

The lossless encoding unit 16 performs lossless encoding such as variable length encoding and arithmetic encoding on the quantized transform coefficient. Since the coefficient data is quantized under the control of the rate control unit 27, the code amount becomes a target value set by the rate control unit 27 (or approximates the target value).

The lossless encoding unit 16 acquires information indicating intra prediction from the intra prediction unit 24, and acquires information indicating inter prediction mode, motion vector information, and the like from the motion prediction / compensation unit 25. Note that information indicating intra prediction (intra-screen prediction) is hereinafter also referred to as intra prediction mode information. In addition, information indicating an information mode indicating inter prediction (inter-screen prediction) is hereinafter also referred to as inter prediction mode information.

The lossless encoding unit 16 encodes the quantized transform coefficient, and converts various information such as a filter coefficient, intra prediction mode information, inter prediction mode information, and a quantization parameter into one piece of header information of encoded data. Part (multiplex). The lossless encoding unit 16 supplies the encoded data obtained by encoding to the accumulation buffer 17 for accumulation.

For example, the lossless encoding unit 16 performs lossless encoding processing such as variable length encoding or arithmetic encoding. Examples of variable length coding include H.264. CAVLC (Context-Adaptive Variable Length Coding) defined by H.264 / AVC format. Examples of arithmetic coding include CABAC (Context-Adaptive Binary Arithmetic Coding).

The accumulation buffer 17 temporarily holds the encoded data supplied from the lossless encoding unit 16. The accumulation buffer 17 stores the accumulated encoded data at a predetermined timing in an H.264 format. As an encoded image encoded by the H.264 / AVC format, for example, it is output to a recording device or a transmission path (not shown) in the subsequent stage. That is, the accumulation buffer 17 is also a transmission unit that transmits encoded data.

Also, the transform coefficient quantized by the quantization unit 15 is also supplied to the inverse quantization unit 18. The inverse quantization unit 18 inversely quantizes the quantized transform coefficient by a method corresponding to the quantization by the quantization unit 15. The inverse quantization unit 18 supplies the obtained transform coefficient to the inverse orthogonal transform unit 19.

The inverse orthogonal transform unit 19 performs inverse orthogonal transform on the supplied transform coefficient by a method corresponding to the orthogonal transform process by the orthogonal transform unit 14. The inversely orthogonal transformed output (restored difference information) is supplied to the calculation unit 20.

The calculation unit 20 is supplied from the intra prediction unit 24 or the motion prediction / compensation unit 25 to the inverse orthogonal transform result supplied from the inverse orthogonal transform unit 19, that is, the restored difference information, via the predicted image selection unit 26. Predicted images are added to obtain a locally decoded image (decoded image).

For example, when the difference information corresponds to an image on which intra coding is performed, the calculation unit 20 adds the prediction image supplied from the intra prediction unit 24 to the difference information. For example, when the difference information corresponds to an image on which inter coding is performed, the calculation unit 20 adds the predicted image supplied from the motion prediction / compensation unit 25 to the difference information.

The addition result is supplied to the deblock filter 21 or the frame memory 22.

The deblocking filter 21 removes block distortion of the decoded image by appropriately performing deblocking filter processing. The deblocking filter 21 supplies the filter processing result to the frame memory 22. Note that the decoded image output from the arithmetic unit 20 can be supplied to the frame memory 22 without going through the deblocking filter 21. That is, the deblocking filter process of the deblocking filter 21 can be omitted.

The frame memory 22 stores the supplied decoded image, and outputs the stored decoded image as a reference image to the intra prediction unit 24 or the motion prediction / compensation unit 25 via the selection unit 23 at a predetermined timing. .

For example, in the case of an image on which intra coding is performed, the frame memory 22 supplies the reference image to the intra prediction unit 24 via the selection unit 23. For example, when inter coding is performed, the frame memory 22 supplies the reference image to the motion prediction / compensation unit 25 via the selection unit 23.

The selection unit 23 supplies the reference image to the intra prediction unit 24 when the reference image supplied from the frame memory 22 is an image to be subjected to intra coding. The selection unit 23 supplies the reference image to the motion prediction / compensation unit 25 when the reference image supplied from the frame memory 22 is an image to be inter-encoded.

The intra prediction unit 24 performs intra prediction (intra-screen prediction) that generates a prediction image using the pixel value in the processing target picture supplied from the frame memory 22 via the selection unit 23. The intra prediction unit 24 performs this intra prediction in a plurality of modes (intra prediction modes) prepared in advance.

In the AVC method, an intra 4 × 4 prediction mode, an intra 8 × 8 prediction mode, and an intra 16 × 16 prediction mode are defined for the luminance signal. Regarding the color difference signal, a prediction mode independent of the luminance signal can be defined for each macroblock. For intra 4 × 4 prediction mode, one intra prediction mode is defined for each 4 × 4 luminance block, and for intra 8 × 8 prediction mode, for each 8 × 8 luminance block. become. For the intra 16 × 16 prediction mode and the color difference signal, one prediction mode is defined for each macroblock.

The intra prediction unit 24 generates prediction images in all candidate intra prediction modes, evaluates the cost function value of each prediction image using the input image supplied from the screen rearrangement buffer 12, and selects the optimum mode. select. When the optimal intra prediction mode is selected, the intra prediction unit 24 supplies the prediction image generated in the optimal mode to the calculation unit 13 and the calculation unit 20 via the predicted image selection unit 26.

Moreover, as described above, the intra prediction unit 24 supplies information such as intra prediction mode information indicating the adopted intra prediction mode to the lossless encoding unit 16 as appropriate.

The motion prediction / compensation unit 25 uses the input image supplied from the screen rearrangement buffer 12 and the reference image supplied from the frame memory 22 via the selection unit 23 for the image to be inter-coded, Perform motion prediction (inter prediction). The motion prediction / compensation unit 25 performs a motion compensation process according to the detected motion vector, and generates a prediction image (inter prediction image information). The motion prediction / compensation unit 25 performs such inter prediction in a plurality of modes (inter prediction modes) prepared in advance.

The motion prediction / compensation unit 25 generates prediction images in all candidate inter prediction modes, evaluates the cost function value of each prediction image, and selects an optimal mode. The motion prediction / compensation unit 25 supplies the generated predicted image to the calculation unit 13 and the calculation unit 20 via the predicted image selection unit 26.

Also, the motion prediction / compensation unit 25 supplies the inter prediction mode information indicating the adopted inter prediction mode and the motion vector information indicating the calculated motion vector to the lossless encoding unit 16.

The predicted image selection unit 26 supplies the output of the intra prediction unit 24 to the calculation unit 13 and the calculation unit 20 in the case of an image to be subjected to intra coding, and in the case of an image to be subjected to inter coding, the motion prediction / compensation unit 25. The output is supplied to the calculation unit 13 and the calculation unit 20.

The rate control unit 27 controls the quantization operation rate of the quantization unit 15 based on the compressed image stored in the storage buffer 17 so that overflow or underflow does not occur.

[AVC image decoding device]
FIG. 2 is a block diagram illustrating a main configuration example of an image decoding apparatus that realizes image compression by orthogonal transformation such as discrete cosine transformation or Karhunen-Labe transformation and motion compensation. An image decoding device 31 shown in FIG. 2 is a decoding device corresponding to the image encoding device 1 of FIG.

The encoded data encoded by the image encoding device 1 is supplied to an image decoding device 31 corresponding to the image encoding device 1 via an arbitrary path such as a transmission path or a recording medium, and is decoded. .

As shown in FIG. 2, the image decoding device 31 includes a storage buffer 41, a lossless decoding unit 42, an inverse quantization unit 43, an inverse orthogonal transform unit 44, a calculation unit 45, a deblock filter 46, a screen rearrangement buffer 47, And a D / A converter 48. Further, the image decoding device 31 includes a frame memory 49, a selection unit 50, an intra prediction unit 51, a motion compensation unit 52, and an image selection unit 53.

The accumulation buffer 41 receives and accumulates the transmitted encoded data. That is, the accumulation buffer 41 is also a receiving unit for the transmitted encoded data. This encoded data is encoded by the image encoding device 1. The lossless decoding unit 42 decodes the encoded data read from the accumulation buffer 41 at a predetermined timing by a method corresponding to the encoding method of the lossless encoding unit 16 in FIG.

Also, when the frame is intra-coded, intra prediction mode information is stored in the header portion of the encoded data. The lossless decoding unit 42 also decodes the intra prediction mode information and supplies the information to the intra prediction unit 51. On the other hand, when the frame is inter-encoded, motion vector information is stored in the header portion of the encoded data. The lossless decoding unit 42 also decodes the motion vector information and supplies the information to the motion compensation unit 52.

The inverse quantization unit 43 inversely quantizes the coefficient data (quantization coefficient) obtained by decoding by the lossless decoding unit 42 by a method corresponding to the quantization method of the quantization unit 15 in FIG. That is, the inverse quantization unit 43 performs inverse quantization of the quantization coefficient by the same method as the inverse quantization unit 18 of FIG.

The inverse quantization unit 43 supplies the inversely quantized coefficient data, that is, the orthogonal transform coefficient, to the inverse orthogonal transform unit 44. The inverse orthogonal transform unit 44 is a method corresponding to the orthogonal transform method of the orthogonal transform unit 14 in FIG. 1 (the same method as the inverse orthogonal transform unit 19 in FIG. 1), and inverse orthogonal transforms the orthogonal transform coefficient to obtain an image code. The decoding residual data corresponding to the residual data before being orthogonally transformed in the encoding apparatus 1 is obtained. For example, fourth-order inverse orthogonal transform is performed.

The decoded residual data obtained by the inverse orthogonal transform is supplied to the calculation unit 45. In addition, a prediction image is supplied to the calculation unit 45 from the intra prediction unit 51 or the motion compensation unit 52 via the image selection unit 53.

The computing unit 45 adds the decoded residual data and the predicted image, and obtains decoded image data corresponding to the image data before the predicted image is subtracted by the computing unit 13 of the image encoding device 1. The arithmetic unit 45 supplies the decoded image data to the deblock filter 46.

The deblock filter 46 removes block distortion of the supplied decoded image, and then supplies it to the screen rearrangement buffer 47.

The screen rearrangement buffer 47 rearranges images. That is, the order of frames rearranged for the encoding order by the screen rearrangement buffer 12 in FIG. 1 is rearranged in the original display order. The D / A converter 48 D / A converts the image supplied from the screen rearrangement buffer 47, and outputs and displays it on a display (not shown).

The output of the deblock filter 46 is further supplied to the frame memory 49.

The frame memory 49, the selection unit 50, the intra prediction unit 51, the motion compensation unit 52, and the image selection unit 53 are the frame memory 22, the selection unit 23, the intra prediction unit 24, and the motion prediction / compensation unit 25 of the image encoding device 1. , And the predicted image selection unit 26, respectively.

The selection unit 50 reads the image to be interprocessed and the image to be referenced from the frame memory 49 and supplies them to the motion compensation unit 52. In addition, the selection unit 50 reads an image used for intra prediction from the frame memory 49 and supplies the image to the intra prediction unit 51.

The information indicating the intra prediction mode obtained by decoding the header information is appropriately supplied from the lossless decoding unit 42 to the intra prediction unit 51. The intra prediction unit 51 generates a prediction image from the reference image acquired from the frame memory 49 based on this information, and supplies the generated prediction image to the image selection unit 53.

The motion compensation unit 52 acquires information (prediction mode information, motion vector information, reference frame information, flags, various parameters, and the like) obtained by decoding the header information from the lossless decoding unit 42.

The motion compensation unit 52 generates a prediction image from the reference image acquired from the frame memory 49 based on the information supplied from the lossless decoding unit 42 and supplies the generated prediction image to the image selection unit 53.

The image selection unit 53 selects the prediction image generated by the motion compensation unit 52 or the intra prediction unit 51 and supplies the selected prediction image to the calculation unit 45.

[Details of adaptive loop filter]
Next, an adaptive loop filter (ALF (Adaptive Loop Filter)) proposed in Non-Patent Document 1 will be described.

FIG. 3 is a block diagram illustrating a configuration example of an image encoding device to which an adaptive loop filter is applied. In the example of FIG. 3, for convenience of explanation, the A / D conversion unit 11, the screen rearrangement buffer 12, the accumulation buffer 17, the selection unit 23, the intra prediction unit 24, the predicted image selection unit 26, and the rate control of FIG. The part 27 is omitted. Also, arrows and the like are omitted as appropriate. Therefore, in the example of FIG. 3, the reference image from the frame memory 22 is directly input to the motion prediction / compensation unit 25, and the prediction image from the motion prediction / compensation unit 25 is directly output to the

calculation units

13 and 20. ing.

That is, the image encoding device 61 in FIG. 3 differs from the image encoding device 1 in FIG. 1 only in that an adaptive loop filter 71 is added between the deblock filter 21 and the frame memory 22.

The adaptive loop filter 71 calculates an adaptive loop filter coefficient so as to minimize a residual with the original image from the screen rearrangement buffer 12 (not shown), and uses this adaptive loop filter coefficient to perform deblocking. Filter processing is performed on the decoded image from the filter 21. As this filter, for example, a Wiener filter is used.

In addition, the adaptive loop filter 71 sends the calculated adaptive loop filter coefficient to the lossless encoding unit 16. In the lossless encoding unit 16, this adaptive loop filter coefficient is subjected to lossless encoding processing such as variable length encoding and arithmetic encoding, and inserted into the header portion of the compressed image.

FIG. 4 is a block diagram showing a configuration example of an image decoding apparatus corresponding to the image encoding apparatus of FIG. In the example of FIG. 4, the storage buffer 41, the screen rearrangement buffer 47, the D / A conversion unit 48, the selection unit 50, the intra prediction unit 51, and the image selection unit 53 of FIG. Yes. Also, arrows and the like are omitted as appropriate. Therefore, in the example of FIG. 4, the reference image from the frame memory 49 is directly input to the motion compensation unit 52, and the predicted image from the motion compensation unit 52 is directly output to the calculation unit 45.

That is, the image decoding device 81 in FIG. 4 differs from the image decoding device 31 in FIG. 2 only in that an adaptive loop filter 91 is added between the deblock filter 46 and the frame memory 49.

The adaptive loop filter 91 is supplied with the adaptive loop filter coefficient decoded from the lossless decoding unit 42 and extracted from the header. The adaptive loop filter 91 performs a filter process on the decoded image from the deblocking filter 46 using the supplied filter coefficient. As this filter, for example, a Wiener filter is used.

Thereby, the image quality of the decoded image can be improved, and further the image quality of the reference image can be improved.

[Cost function]
In AVC, selection of an appropriate prediction mode is important to achieve higher encoding efficiency.

As an example of such a selection method, a method implemented in AVC reference software called JM 呼ば (Joint Model) published at http://iphome.hhi.de/suehring/tml/index.htm. I can list them.

In JM, it is possible to select the following two mode determination methods: High Complexity Mode and Low Complexity Mode. In both cases, a cost function value for each prediction mode Mode is calculated, and a prediction mode that minimizes the cost function value is selected as the optimum mode for the block or macroblock.

The cost function in High Complexity Mode is as shown in the following formula (1).

Cost (Mode∈Ω) = D + λ * R (1)

Here, Ω is the entire set of candidate modes for encoding the block or macroblock, and D is the difference energy between the decoded image and the input image when encoded in the prediction mode Mode. λ is a Lagrange undetermined multiplier given as a function of the quantization parameter. R is a total code amount when encoding is performed in the mode Mode, including orthogonal transform coefficients.

That is, in order to perform encoding in High Complexity Mode, in order to calculate the parameters D and R, it is necessary to perform provisional encoding processing once in all candidate modes (Mode), which requires a higher calculation amount. .

The cost function in Low Complexity Mode is as shown in the following formula (2).

Cost (Mode∈Ω) = D + QP2Quant (QP) * HeaderBit (2)

Here, D is the difference energy between the predicted image and the input image, unlike the case of High Complexity Mode. QP2Quant (QP) is given as a function of the quantization parameter QP, and HeaderBit is a code amount related to information belonging to Header, such as a motion vector and a mode, which does not include an orthogonal transform coefficient.

That is, in Low Complexity Mode, it is necessary to perform prediction processing for each candidate mode (Mode), but it is not necessary to perform decoding processing because there is no need for a decoded image. For this reason, it is possible to realize with a calculation amount lower than that of High Complexity Mode.

[Coding unit]
Next, a coding unit defined in the High Efficiency Video Coding (HEVC) encoding method (hereinafter referred to as HEVC method) described in Non-Patent Document 2 will be described.

In the AVC method, it was possible to divide one macro block into a plurality of motion compensation blocks and to have different motion information for each. That is, in the AVC system, a hierarchical structure is defined by macroblocks and sub-macroblocks. For example, in the HEVC system, a coding unit (CU (Coding Unit)) is defined as shown in FIG. ing.

CU is also called Coding Tree Block (CTB) and is a coding unit similar to a macroblock in the AVC method. The latter is fixed to a size of 16 × 16 pixels, whereas the size of the former is not fixed, and is specified in the image compression information in each sequence.

For example, in a sequence parameter set (SPS (Sequence Coding Unit)) included in encoded data to be output, the maximum size (LCU (Largest Coding Unit)) and minimum size ((SCU (Smallest Coding Unit)) of the CU are defined. Is done.

Within each LCU, it is possible to divide into smaller CUs by setting split-flag = 1 within a range not smaller than the SCU size. In the example of FIG. 5, the LCU size is 128 and the maximum hierarchical depth is 5. When the value of split_flag is “1”, the 2N × 2N size CU is divided into N × N size CUs that are one level below.

Furthermore, the CU may be divided into one or more prediction units (Prediction Units: PUs) that mean units of intra or inter prediction processing. Moreover, PU can be divided | segmented into one or more transformation units (Transform | Unit: TU) which means the unit of orthogonal transformation. Then, for each transform unit, orthogonal transform from image data to transform coefficient data and quantization of transform coefficient data are performed. At present, in the HEVC system, it is possible to use 16 × 16 and 32 × 32 orthogonal transforms in addition to 4 × 4 and 8 × 8.

In the case of an encoding method in which a CU is defined and various processes are performed in units of the CU as in the HEVC method described above, a macroblock in the AVC method can be considered to correspond to an LCU. However, since the CU has a hierarchical structure as shown in FIG. 5, the size of the LCU in the highest hierarchy is H.264, for example, 128 × 128 pixels. Generally, it is set larger than the macroblock of the H.264 / AVC format.

In this specification, each processing unit of LCU, CU, PU, and TU is also referred to as a block as appropriate.

[Non-Square Quadtree Transform]
In the HEVC method, a rectangular TU called NSQT (Non-Square Quadtree Transform) as shown in FIG. 6 can be used.

In the HEVC method, when Depth = 1, a single TU is applied to the CU. On the other hand, when Depth = 2, a plurality of squares or a rectangular TU called NSQT can be applied to the CU. For example, in FIG. 6, as an example of Depth = 2, when composed of four square TUs, composed of four horizontally long TUs, composed of four vertically long TUs CUs are shown. Among these TUs, a horizontally long TU and a vertically long TU are NSQT.

This is described, for example, as a short distance intra prediction method in Non-Patent Document 3 in the case of intra prediction. In the short-range intra prediction method, for example, prediction units of various sizes such as 1 × 4 pixels, 2 × 8 pixels, 4 × 16 pixels, 4 × 1 pixels, 8 × 2 pixels, and 16 × 4 pixels are included in the image. Can be set to In this case, which of the vertical size and the horizontal size of the prediction unit is larger depends on the setting of the prediction unit.

[Adaptive offset processing in HEVC]
Next, an adaptive offset filter in the HEVC scheme will be described. In the HEVC method, the Sample Adaptive Offset method described in Non-Patent Document 3 is adopted.

The adaptive offset filter (Picture Quality Adaptive Offset: PQAO) is provided between the deblock filter (DB) and the adaptive loop filter (ALF) as shown in FIG.

¡There are two types of adaptive offsets called band offsets and six types called edge offsets, and it is also possible not to apply offsets. Then, the image is divided into quad-trees, and it is possible to select which of the above-described adaptive offset types is used for encoding each region.

This selection information is encoded as PQAO Info. By the encoding unit (Entropy Coding), a bit stream is generated, and the generated bit stream is transmitted to the decoding side. By using this method, encoding efficiency can be improved.

Here, the quad-tree structure will be described with reference to FIG.

For example, on the encoding side, as shown in A1 of FIG. 8, a cost function value J0 of Level-0 (division depth 0) indicating a state where the region 0 is not divided is calculated. Further, cost function values J1, J2, J3, and J4 of Level-1 (division depth 0) indicating a state where the area 0 is divided into four areas 1 to 4 are calculated.

Then, as shown in A2, the cost function values are compared, and a partition region (Partitions) of Level-1 is selected by J0> (J1 + J2 + J3 + J4).

Similarly, as shown in A3, cost function values J5 to J20 of Level-2 (division depth 2) indicating a state where the area 0 is divided into 16 areas 5 to 20 are calculated.

Then, as shown in A4, the cost function values are respectively compared, and a partition region (Partitions) of Level-1 is selected in region 1 by J1 <(J5 + J6 + J9 + J10). In region 2, a Level-2 partition region (Partitions) is selected by J2> (J7 + J8 + J11 + J12). By region J3> (J13 + J14 + J17 + J18), in region 3, the Level-2 partition region (Partitions) is selected. By J4> (J15 + J16 + J19 + J20), the division region (Partitions) of Level-1 is selected in the region 4.

As a result, the final quad-tree region (Partitions) indicated by A4 in the quad-tree structure is determined. Then, for each region of the quad-tree structure determined, cost function values are calculated for all of the two types of band offsets, six types of edge offsets, and no offset, and it is determined which offset is used for encoding. The

For example, in the example of FIG. 8, EO (4), that is, the fourth type of edge offset is determined for the region 1 as indicated by the white arrow. For region 7, OFF, that is, no offset is determined, and for region 8, EO (2), that is, the second type of edge offset is determined. For

regions

11 and 12, OFF, that is, no offset is determined.

For region 13, BO (1), that is, the first type of band offset is determined, and for region 14, EO (2), that is, 2 of edge offset, is determined. The type has been determined. For region 17, BO (2), that is, the second type of band offset is determined, and for region 18, BO (1), that is, the first type of band offset. Has been determined. For region 4, EO (1), that is, the first type of edge offset is determined.

Next, details of the band offset will be described with reference to FIG.

Regarding the band offset, in the example of FIG. 9, one scale represents one band = 8 pixels, the luminance pixel value is divided into 32 bands, and each band has an offset value independently.

That is, in the example of FIG. 9, among the 0 to 255 pixels (32 bands), the central 16 bands are divided into the first group, and the 8 bands on both sides are divided into the second group.

Then, the offset of only one of the first group and the second group is encoded and sent to the decoding side. In general, in one region, there are often either black and white clearly or subtle hues, and it is rare that both the first group and the second group have pixels. For this reason, by sending only one offset, it is possible to suppress an increase in the amount of coding due to transmission of pixel values that are not included in each quad-tree region.

When the input signal is broadcast, the luminance signal is limited to 16,235, and the color difference signal is limited to 16,240. At this time, the broadcast legal shown in the lower part of FIG. 9 is applied, and the offset value for each of the two bands on both sides indicated by the crosses is not transmitted.

Next, details of the edge offset will be described with reference to FIG.

In the edge offset, the pixel value is compared with the adjacent pixel value adjacent to the pixel value, and the offset value is transmitted to the category corresponding thereto.

In the edge offset, there are four one-dimensional patterns shown in FIGS. 10A to 10D and two two-dimensional patterns shown in FIG. 10E and FIG. 10F. The offset is transmitted in the indicated category.

10A shows that 1-D, 0-degree in which adjacent pixels are arranged one-dimensionally on the left and right sides with respect to the pixel C, that is, 0 degrees with respect to the pattern of A in FIG. Represents a pattern. In FIG. 10B, adjacent pixels are arranged one-dimensionally above and below the pixel C, that is, 1-D, 90-degree, which forms 90 degrees with respect to the pattern of A in FIG. Represents a pattern.

C in FIG. 10 is such that adjacent pixels are arranged one-dimensionally in the upper left and lower right with respect to the pixel C, that is, 1-D, which forms 135 degrees with respect to the pattern of A in FIG. Represents a 135-degree pattern. In FIG. 10D, adjacent pixels are arranged one-dimensionally on the upper right and lower left with respect to the pixel C, that is, 45 degrees with respect to the pattern of A in FIG. Represents the -degree pattern.

E in FIG. 10 represents a 2-D, cross pattern in which adjacent pixels are arranged two-dimensionally in the vertical and horizontal directions with respect to the pixel C, that is, intersect with the pixel C. FIG. 10F shows that 2-D, diagonal in which adjacent pixels are two-dimensionally arranged with respect to the pixel C in the upper right lower left and upper left lower right, that is, obliquely intersect the pixel C. Represents a pattern.

11A shows a rule list of one-dimensional patterns (Classification rule for 1-D patterns). The patterns A in FIG. 11 to D in FIG. 11 are classified into five types of categories as shown in FIG. 11A, offsets are calculated based on the categories, and sent to the decoding unit.

When the pixel value of the pixel C is smaller than the pixel values of two adjacent pixels, it is classified into category 1. When the pixel value of the pixel C is smaller than the pixel value of one adjacent pixel and matches the pixel value of the other adjacent pixel, it is classified into category 2. When the pixel value of the pixel C is larger than the pixel value of one adjacent pixel and matches the pixel value of the other adjacent pixel, it is classified into category 3. When the pixel value of the pixel C is larger than the pixel values of two adjacent pixels, it is classified into category 4. If none of the above, it is classified into category 0.

B in FIG. 11 shows a two-dimensional pattern rule list (Classification rule for 2-D 、２patterns). The patterns of E of FIG. 10 and F of FIG. 10 are classified into seven types of categories as shown in B of FIG. 11, and offsets are sent to the decoding unit according to the categories.

When the pixel value of the pixel C is smaller than the pixel values of the four adjacent pixels, it is classified into category 1. When the pixel value of the pixel C is smaller than the pixel values of the three adjacent pixels and matches the pixel value of the fourth adjacent pixel, the pixel C is classified into category 2. When the pixel value of the pixel C is smaller than the pixel values of the three adjacent pixels and larger than the pixel value of the fourth adjacent pixel, the pixel C is classified into category 3.

When the pixel value of the pixel C is larger than the pixel values of the three adjacent pixels and smaller than the pixel value of the fourth adjacent pixel, it is classified into category 4. When the pixel value of the pixel C is larger than the pixel values of the three adjacent pixels and matches the pixel value of the fourth adjacent pixel, the pixel C is classified into category 5. When the pixel value of the pixel C is larger than the pixel values of the four adjacent pixels, it is classified into category 6. If none of the above, it is classified into category 0.

As described above, in the edge offset, since the one-dimensional pattern only needs to compare two adjacent pixels, the amount of calculation is low. Note that, in the high-efficiency encoding condition, the 1-bit offset value is sent to the decoding side with higher accuracy than the low-delay encoding condition.

In the above-described adaptive offset processing, the quad-tree structure (including information about the type of offset and no offset) described above with reference to FIG. 8 and the offset value are sent to the decoding side. Note that the category may be sent to the decoding side, or may be obtained in each device.

<2. First Embodiment>
[Configuration Example of Image Encoding Device]
FIG. 12 illustrates a configuration of an embodiment of an image encoding device as an image processing device to which the present disclosure is applied.

The image encoding apparatus 101 shown in FIG. 12 encodes image data using a prediction process. Here, as the encoding method, for example, a method according to HEVC (High Efficiency Video Coding) is used.

12 includes an A / D conversion unit 11, a screen rearrangement buffer 12, a calculation unit 13, a quantization unit 15, a lossless encoding unit 16, a storage buffer 17, an inverse quantization unit 18, and an inverse unit. It is common with the image coding apparatus 1 of FIG. 1 by the point provided with the orthogonal transformation part 19. FIG. The image encoding device 101 in FIG. 12 includes a calculation unit 20, a deblock filter 21, a frame memory 22, a selection unit 23, an intra prediction unit 24, a motion prediction / compensation unit 25, a predicted image selection unit 26, and a rate control unit 27. 1 in common with the image encoding device 1 of FIG.

12 is different from the image encoding device 1 of FIG. 1 in that the adaptive loop filter 71 of FIG. 3 described above is added.

Furthermore, the image coding apparatus 101 in FIG. 12 is different from the image coding in FIG. 1 in that the orthogonal transform unit 14 is replaced with the orthogonal transform unit 111 and that a class classification unit 112 and an adaptive offset unit 113 are added. Different from the device 1.

That is, the orthogonal transform unit 111 performs orthogonal transform such as discrete cosine transform and Karhunen-Loeve transform on the difference information supplied from the computation unit 13 in the same manner as the orthogonal transform unit 14 of FIG. Is supplied to the quantization unit 15. At that time, the orthogonal transform unit 111 determines the orthogonal transform size for the block (processing unit, TU in this case) by mode determination based on the cost function value. Then, the orthogonal transform unit 111 supplies the determined TU size, which is a unit of the determined orthogonal transform, to the lossless encoding unit 16 and the class classification unit 112.

The class classification unit 112 performs class classification according to the TU size from the orthogonal transformation unit 111. The class classification in the class classification unit 112 is setting (classification) of the type of offset. That is, the class classification unit 112 sets whether to apply edge offset (EO) or band offset (BO) according to the TU size from the orthogonal transform unit 111. The class classification unit 112 supplies information indicating the set offset type to the adaptive offset unit 113.

The deblock filter 21, the adaptive offset unit 113, and the adaptive loop filter 71 are provided in the motion compensation loop in that order. The motion compensation loop is an arithmetic unit 13, an orthogonal transformation unit 111, a quantization unit 15, an inverse quantization unit 18, an inverse orthogonal transformation unit 19, an arithmetic unit 20, a frame memory 22, a selection unit 23, an intra prediction unit 24, or a motion. This block includes a prediction / compensation unit 25 and a predicted image selection unit 26. Hereinafter, the filter processing performed by the deblock filter 21, the adaptive offset unit 113, and the adaptive loop filter 71 in the motion compensation loop is also collectively referred to as in-loop filter processing.

The adaptive offset unit 113 performs an offset process on the decoded image (baseband information after local decoding) from the deblocking filter 21 based on the information indicating the type of offset from the class classification unit 112.

That is, whether to apply edge offset (EO) or band offset (BO) is set by the class classification unit 112 according to the TU size from the orthogonal transform unit 111. Therefore, the adaptive offset unit 113 determines whether or not to perform the offset (on / off of the adaptive offset filter), and when it is determined to perform the offset, obtains the category and the offset value, and calculates the arithmetic unit 20. The offset process is performed on the decoded image from. The adaptive offset unit 113 supplies the image after the offset process to the adaptive loop filter 71.

Also, the adaptive offset unit 113 supplies the determined on / off information and information indicating the offset value to the lossless encoding unit 16.

The adaptive loop filter 71 calculates an adaptive loop filter coefficient so as to minimize the residual from the original image (not shown) from the screen rearrangement buffer 12, and uses this adaptive loop filter coefficient to perform an adaptive offset. Filter processing is performed on the decoded image from the unit 113. As this filter, for example, a Wiener filter is used. This improves the image quality. Although not shown, the adaptive loop filter 71 sends the calculated adaptive loop filter coefficient to the lossless encoding unit 16.

The lossless encoding unit 16 in FIG. 12 encodes the quantized transform coefficient as well as various types of filter coefficients, prediction mode information, quantization parameters, and the like, similar to the lossless encoding unit 16 in FIG. The information is part of the header information of the encoded data. At this time, the lossless encoding unit 16 uses the on / off information from the adaptive offset unit 113 and the information indicating the offset value as part of the header information of the encoded data as the adaptive offset parameter. Further, the lossless encoding unit 16 also uses information about the TU size from the orthogonal transform unit 111, information indicating an adaptive loop filter coefficient, and the like as part of the header information of the encoded data.

[Adaptive offset processing of this technology]
In the HEVC system, the orthogonal transform size for the luminance signal can be any of 4 × 4, 8 × 8, 16 × 16, and 32 × 32. In such an orthogonal transform size, a smaller TU size tends to be selected for a block (processing unit) including an edge. On the other hand, for flat blocks, larger TU sizes tend to be selected. That is, whether or not the block includes an edge has a correlation with the size of the block.

Therefore, in the present technology, it is determined whether to use the edge offset or the band offset in the adaptive offset filter by using the correlation with the block size described above. Thereby, the information regarding the adaptive offset filter sent to the decoding side is reduced, and the coding efficiency is improved.

That is, an edge offset is applied to a block for which a smaller TU size is selected, and a band offset is applied to a block for which a larger TU size is selected.

Specifically, as an example, the class classification unit 112 sets an edge offset for a block having a TU size of 4 × 4 or 8 × 8, and blocks other TU sizes (or more). Set the band offset.

Note that in the method proposed in Non-Patent Document 4, there may be an option of not applying an offset.

Therefore, in the present technology, as illustrated in FIG. 13, for a TU having a TU size of 4 × 4 or 8 × 8, an option of applying an edge offset or not applying an adaptive offset is given. . Also, for TUs having a TU size equal to or larger than those sizes (for example, 16 × 16 or 32 × 32), an option of applying a band offset or not applying an adaptive offset is given.

In the present technology, two options as described above are given for 16 × 16 or more TUs, but an adaptive offset may not be applied to 32 × 32 TUs.

This is because a 32 × 32 TU is assumed to be a flatter area than a block of 16 × 16 TU size. That is, it is considered that the contribution to the improvement of the coding efficiency by performing the adaptive offset process on the 32 × 32 TU is considered to be smaller than the applied offset process to other sizes.

Of course, what kind of adaptive offset processing is performed for a TU of 16 × 16 or less may be set in advance. For example, as shown in the example of FIG. 14, an edge offset can be set to be applied to a TU having a TU size of 4 × 4 or 8 × 8. In addition, it is possible to set a band offset to be applied to a TU having a TU size of 16 × 16 and not to apply an adaptive offset to a 32 × 32 TU.

Further, for example, an edge offset is applied to a TU to which NSQT described above with reference to FIG. 6 is applied. This is because a block to which NSQT is applied is considered to include an edge in the texture.

Note that, for a TU to which NSQT is applied, whether to apply an edge offset or a band offset may be set for the size of the shorter side.

As described above, using the correlation with the size of the block (for example, TU), the adaptive offset filter determines whether to use the edge offset or the band offset, and based on this, the adaptive offset filter is applied to the decoded image. I did it.

As a result, it is not necessary to send information indicating whether the region division and edge offset or band offset (that is, the quad-tree structure map information described above with reference to FIG. 8) to the decoding side, and the compressed image The amount of information in the information is reduced, and the coding efficiency is improved.

In the above description, an example is described in which class classification is performed according to the TU size as a block (processing unit), but is not limited to a TU. That is, class classification (setting of filter type) may be performed according to PU size or CU size, and an adaptive offset filter may be performed for each PU or CU.

In particular, in an intra frame, a block having a larger PU size or CU size is likely to be a flat region, and is therefore set to use a band offset. On the other hand, a block having a smaller large PU size or CU size is more likely to include an edge, and thus is set to use an edge offset.

Here, as described above with reference to FIG. 5, in the HEVC hierarchical structure, the CU includes a PU and the PU includes a TU. In other words, the amount of calculation for determining the edge offset and the band offset by the existing method is the smallest based on the CU size, followed by the case based on the PU size and the TU size.

On the other hand, there is a possibility that a single CU includes both a flat area and an edge area, and this can be known only by performing decoding processing up to the PU or TU hierarchy. For this reason, classification based on PU or TU can be performed with higher accuracy.

Furthermore, an edge offset or band offset may be set for each LCU. In the case of the LCU unit, the processing unit (for example, TU, PU, or CU) included in the LCU is determined depending on whether there are more units larger than a predetermined size or smaller than a predetermined size. That is, setting of the edge offset or band offset is performed according to how much area the processing unit of each size occupies in the LCU, not the number of processing units.

For example, as shown in FIG. 15, the LCU has a size of 32 × 32 and is configured to include three 16 × 16 TUs, two 8 × 8 TUs, and eight 4 × 4 TUs.

In this case, 4 × 4 TU is the largest number, but the area occupied by 16 × 16 TU is the largest. Therefore, in the case of the LCU shown in FIG. 15, the band offset is set.

Thereby, even in the case of LUC unit, since it is not necessary to send to the information decoding side of the edge offset or band offset, the amount of information in the compressed image information is reduced and the encoding efficiency is improved. In this case, for example, the class classification unit 112 includes a counter for each size, and the area is obtained from the number and size indicated by the counter.

The method according to the present technology described above can be applied to both the luminance signal and the color difference signal.

[Configuration example of orthogonal transform unit and adaptive offset unit]
Next, each unit of the image encoding device 101 will be described. FIG. 16 is a block diagram illustrating a configuration example of the orthogonal transform unit 111 and the adaptive offset unit 113.

In the example of FIG. 16, the orthogonal transform unit 111 includes a 4 × 4 orthogonal transform unit 131, an 8 × 8 orthogonal transform unit 132, a 16 × 16 orthogonal transform unit 133, a 32 × 32 orthogonal transform unit 134, a cost function calculation unit 135, And a TU size determination unit 136.

The adaptive offset unit 113 is configured to include an on / off determination unit 141, a category classification unit 142, and an offset processing unit 143.

Difference information (PU) indicating a difference value from the calculation unit 13 is supplied to the 4 × 4 orthogonal transform unit 131, the 8 × 8 orthogonal transform unit 132, the 16 × 16 orthogonal transform unit 133, and the 32 × 32 orthogonal transform unit 134. Is done.

The 4 × 4 orthogonal transform unit 131 performs orthogonal transform on the difference information from the calculation unit 13 with a 4 × 4 TU size, and supplies the transform coefficient to the cost function calculation unit 135. The 8 × 8 orthogonal transform unit 132 performs orthogonal transform on the difference information from the calculation unit 13 with an 8 × 8 TU size, and supplies the transform coefficient to the cost function calculation unit 135. The 16 × 16 orthogonal transform unit 133 performs orthogonal transform on the difference information from the calculation unit 13 with a 16 × 16 TU size, and supplies the transform coefficient to the cost function calculation unit 135. The 32 × 32 orthogonal transform unit 134 performs orthogonal transform on the difference information from the calculation unit 13 with a 32 × 32 TU size, and supplies the transform coefficient to the cost function calculation unit 135.

The cost function calculation unit 135 calculates the cost function value using the conversion coefficient of each TU size, and supplies the conversion coefficient of each TU size and the corresponding cost function value to the TU size determination unit 136.

The TU size determination unit 136 determines the optimal TU size for the block based on the cost function value calculated by the cost function calculation unit 135 and supplies the orthogonal transform coefficient of the determined TU size to the quantization unit 15. Further, the TU size determination unit 136 supplies information on the determined TU size to the class classification unit 112 and the lossless encoding unit 16.

Based on the TU size from the TU size determination unit 136, the class classification unit 112 sets the type of offset (edge offset or band offset) to be applied to the block by the method of the present technology described above. To do. The class classification unit 112 supplies the set offset type information to the on / off determination unit 141.

The on / off determination unit 141 determines the on / off of the adaptive offset processing for the block (for example, TU) using the pixel value after the deblocking filter processing from the deblocking filter 21. For example, the on / off determination unit 141 calculates the cost function value in the adaptive offset process according to the type of offset set by the class classification unit 112 and the cost function value without the adaptive offset process. Then, the on / off determination unit 141 determines on / off of the adaptive offset process based on the calculated cost function value.

The on / off determination unit 141 supplies the pixel value after the deblocking filter processing from the deblocking filter 21 and the determined on / off information to the category classification unit 142. Further, the on / off determination unit 141 supplies the determined on / off information to the lossless encoding unit 16.

When the on / off information from the on / off determination unit 141 indicates “on”, the category classification unit 142 uses the pixel value after the deblocking filter processing to determine the category in the offset type set by the class classification unit 112. Classify. The category classification unit 142 supplies information indicating the classified category and the pixel value after deblocking filter processing to the offset processing unit 143.

When the on / off information from the on / off determination unit 141 indicates off, the category classification unit 142 stores the pixel value after the deblocking filter processing from the on / off determination unit 141 as it is in the offset processing unit 143. Supply.

When the on / off information from the on / off determination unit 141 indicates on, the offset processing unit 143 performs adaptive offset processing on the pixel value after the deblocking filter processing from the category classification unit 142. The offset processing unit 143 supplies the pixel value after adaptive offset processing to the adaptive loop filter 71.

That is, the offset processing unit 143 uses the pixel value from the screen rearrangement buffer 12 and the pixel value after the deblocking filter processing from the category classification unit 142 for the set offset type and the classified category. Find the offset value.

The offset processing unit 143 performs offset processing on the pixel value after the deblocking filter processing from the category classification unit 142 with the set offset type, the classified category, and the obtained offset value. The offset processing unit 143 also supplies the obtained offset value to the lossless encoding unit 16.

When the on / off information from the on / off determination unit 141 indicates off, the offset processing unit 143 uses the pixel value after the deblocking filter processing from the category classification unit 142 as it is (without performing the offset processing). To the adaptive loop filter 71.

The lossless encoding unit 16 adds information on the TU size from the TU size determination unit 136 to the encoded stream. The lossless encoding unit 16 uses the on / off information from the on / off determination unit 141 and the offset value information from the offset processing unit 143 (when the on / off information indicates on) as an adaptive offset parameter. Is added to the encoded stream.

[Flow of encoding process]
Next, the flow of each process executed by the image encoding device 101 as described above will be described. First, an example of the flow of encoding processing will be described with reference to the flowchart of FIG.

In step S101, the A / D conversion unit 11 performs A / D conversion on the input image. In step S102, the screen rearrangement buffer 12 stores the A / D converted image, and rearranges the picture from the display order to the encoding order.

When the image to be processed supplied from the screen rearrangement buffer 12 is an image of a block to be intra-processed, the decoded image to be referred to is read from the frame memory 22 and the intra-prediction unit via the selection unit 23 24.

Based on these images, in step S103, the intra prediction unit 24 performs intra prediction on the pixels of the block to be processed in all candidate intra prediction modes. As decoded pixels to be referred to, pixels that are not filtered or offset by the deblock filter 21, the adaptive offset unit 113, and the adaptive loop filter 71 are used.

With this process, intra prediction is performed in all candidate intra prediction modes, and the cost function shown in the equation (1) or equation (2) is used for all candidate intra prediction modes, A cost function value is calculated. Then, based on the calculated cost function value, the optimal intra prediction mode is selected, and the predicted image generated by the intra prediction in the optimal intra prediction mode and its cost function value are supplied to the predicted image selection unit 26.

When the processing target image supplied from the screen rearrangement buffer 12 is an inter-processed image, the referenced image is read from the frame memory 22 and supplied to the motion prediction / compensation unit 25 via the selection unit 23. Is done. Based on these images, in step S104, the motion prediction / compensation unit 25 performs motion prediction / compensation processing.

By this processing, motion prediction processing is performed in all candidate inter prediction modes, and the cost function shown in Equation (1) or Equation (2) is used for all candidate inter prediction modes. A cost function value is calculated. Based on the calculated cost function value, the optimal inter prediction mode is determined, and the predicted image generated in the optimal inter prediction mode and its cost function value are supplied to the predicted image selection unit 26.

In step S <b> 105, the predicted image selection unit 26 optimizes one of the optimal intra prediction mode and the optimal inter prediction mode based on the cost function values output from the intra prediction unit 24 and the motion prediction / compensation unit 25. Determine the prediction mode. Then, the predicted image selection unit 26 selects the predicted image in the determined optimal prediction mode and supplies it to the

calculation units

13 and 20. This predicted image is used for calculations in steps S106 and S112 described later.

Note that the prediction image selection information is supplied to the intra prediction unit 24 or the motion prediction / compensation unit 25. When the prediction image in the optimal intra prediction mode is selected, the intra prediction unit 24 supplies information indicating the optimal intra prediction mode (that is, intra prediction mode information) to the lossless encoding unit 16.

When the prediction image of the optimal inter prediction mode is selected, the motion prediction / compensation unit 25 further includes information indicating the optimal inter prediction mode and, if necessary, information corresponding to the optimal inter prediction mode as a lossless encoding unit. 16 is output. Information according to the optimal inter prediction mode includes motion vector information and reference frame information.

In step S106, the calculation unit 13 calculates a difference between the image rearranged in step S102 and the predicted image selected in step S105. The predicted image is supplied from the motion prediction / compensation unit 25 in the case of inter prediction, and from the intra prediction unit 24 in the case of intra prediction, to the calculation unit 13 via the predicted image selection unit 26, respectively.

差分 Difference data has a smaller data volume than the original image data. Therefore, the data amount can be compressed as compared with the case where the image is encoded as it is.

In step S107, the orthogonal transform unit 111 determines an orthogonal transform size. That is, the 4 × 4 orthogonal transform unit 131 performs orthogonal transform on the difference information from the calculation unit 13 with a 4 × 4 TU size, and supplies the transform coefficient to the cost function calculation unit 135. The 8 × 8 orthogonal transform unit 132 performs orthogonal transform on the difference information from the calculation unit 13 with an 8 × 8 TU size, and supplies the transform coefficient to the cost function calculation unit 135. The 16 × 16 orthogonal transform unit 133 performs orthogonal transform on the difference information from the calculation unit 13 with a 16 × 16 TU size, and supplies the transform coefficient to the cost function calculation unit 135. The 32 × 32 orthogonal transform unit 134 performs orthogonal transform on the difference information from the calculation unit 13 with a 32 × 32 TU size, and supplies the transform coefficient to the cost function calculation unit 135.

The cost function calculation unit 135 calculates the cost function value using the conversion coefficient of each TU size, and supplies the conversion coefficient of each TU size and the corresponding cost function value to the TU size determination unit 136. The TU size determination unit 136 determines an optimal TU size for the block based on the cost function value calculated by the cost function calculation unit 135. The TU size determination unit 136 supplies information on the determined TU size to the class classification unit 112 and the lossless encoding unit 16.

In step S108, the TU size determination unit 136 performs orthogonal transformation. For example, the TU size determination unit 136 performs orthogonal transform with the TU size determined in step S <b> 107 and supplies the orthogonal transform coefficient to the quantization unit 15.

In step S109, the quantization unit 15 quantizes the transform coefficient. The quantization unit 15 sets the quantization parameter based on the information regarding the target value of the code amount supplied from the rate control unit 27 and performs quantization, as will be described in the process of step S119 described later.

The difference information quantized as described above is locally decoded as follows. That is, in step S110, the inverse quantization unit 18 inversely quantizes the transform coefficient quantized by the quantization unit 15 with characteristics corresponding to the characteristics of the quantization unit 15. In step S <b> 111, the inverse orthogonal transform unit 19 performs inverse orthogonal transform on the transform coefficient inversely quantized by the inverse quantization unit 18 with characteristics corresponding to the characteristics of the orthogonal transform unit 14.

In step S112, the calculation unit 20 adds the predicted image input via the predicted image selection unit 26 to the locally decoded difference information, and the locally decoded (that is, locally decoded) image. (An image corresponding to the input to the calculation unit 13) is generated.

In step S113, the deblocking filter 21 performs a deblocking process on the image from the calculation unit 20, and supplies the pixel value after the deblocking process to the adaptive offset unit 113. By this processing, block distortion is suppressed.

In step S114, the class classification unit 112 and the adaptive offset unit 113 perform an adaptive offset process according to the orthogonal transform size determined in step S107. Details of the adaptive offset processing will be described later with reference to FIG.

The pixel value after adaptive offset processing is supplied to the adaptive loop filter 71 by the processing in step S114. Information indicating on / off of the adaptive offset processing and information indicating the offset value are supplied to the lossless encoding unit 16. Ringing and the like are suppressed by this adaptive offset processing.

In step S115, the adaptive loop filter 71 performs an adaptive loop filter on the pixel value after the adaptive offset process, and supplies the pixel value after the adaptive loop filter to the frame memory 22.

For example, the adaptive loop filter 71 calculates the adaptive loop filter coefficient so as to minimize the residual with the original image (not shown) from the screen rearrangement buffer 12, and uses the adaptive loop filter coefficient to Filter processing is performed on the decoded image from the adaptive offset unit 113. Although not shown, the adaptive loop filter 71 sends the calculated adaptive loop filter coefficient to the lossless encoding unit 16.

In step S116, the frame memory 22 stores the filtered image. In the frame memory 22, images that are not filtered or offset by the deblocking filter 21, the adaptive offset unit 113, and the adaptive loop filter 71 are also supplied from the arithmetic unit 20 and stored.

On the other hand, the transform coefficient quantized in step S109 described above is also supplied to the lossless encoding unit 16. In step S117, the lossless encoding unit 16 encodes the quantized transform coefficient output from the quantization unit 15. That is, the difference image is subjected to lossless encoding such as variable length encoding and arithmetic encoding, and is compressed.

At this time, the intra prediction mode information from the intra prediction unit 24 or the information corresponding to the optimal inter prediction mode from the motion prediction / compensation unit 25 input to the lossless encoding unit 16 in step S105 described above, etc. Is also encoded and added to the header information. Furthermore, the information indicating the orthogonal transform size input to the lossless encoding unit 16 in step S107 described above, the information indicating on / off input to the lossless encoding unit 16 in step S114 described above, and the offset information are also encoded. And added to the header information.

For example, information indicating the inter prediction mode is encoded for each LCU. Motion vector information and reference frame information are encoded for each target PU.

In step S118, the accumulation buffer 17 accumulates the difference image as a compressed image. The compressed image stored in the storage buffer 17 is appropriately read out and transmitted to the decoding side via the transmission path.

In step S119, the rate control unit 27 controls the quantization operation rate of the quantization unit 15 based on the compressed image stored in the storage buffer 17 so that overflow or underflow does not occur.

When the process of step S119 ends, the encoding process ends.

[Flow of adaptive offset processing]
Next, an example of the flow of the adaptive offset process executed in step S114 in FIG. 17 will be described with reference to the flowchart in FIG.

17, the TU size determination unit 136 of the orthogonal transform unit 111 supplies information related to the TU size to the class classification unit 112 by the process of step S108 in FIG. In step S151, according to the method of the present technology described above according to the supplied TU size, the class classification unit 112 determines whether the block (for example, TU) has an edge offset (EO) or a band offset (BO). Set whether to apply. The class classification unit 112 supplies the set offset type information to the on / off determination unit 141.

In step S152, the on / off determination unit 141 determines on / off of the adaptive offset processing for this TU using the pixel value after the deblocking filter processing from the deblocking filter 21.

For example, the on / off determination unit 141 calculates the cost function value in the adaptive offset process according to the type of offset set by the class classification unit 112 and the cost function value without the adaptive offset process. Then, the on / off determination unit 141 determines on / off of the adaptive offset process based on the calculated cost function value. The on / off determination unit 141 supplies the pixel value after the deblocking filter processing from the deblocking filter 21 and the determined on / off information to the category classification unit 142.

In step S153, the on / off determination unit 141 supplies the determined on / off information to the lossless encoding unit 16, and encodes the on / off information.

In step S154, the category classification unit 142 determines whether the adaptive offset filter is on for the TU based on on / off information from the on / off determination unit 141. If it is determined in step S154 that the adaptive offset filter is on, the process proceeds to step S155.

In step S155, the category classification unit 142 uses the pixel value after the deblocking filter processing from the on / off determination unit 141 to classify the category in the offset type set by the class classification unit 112. The category classification unit 142 supplies information indicating the classified category and the pixel value after deblocking filter processing to the offset processing unit 143.

In step S156, the offset processing unit 143 obtains an offset value and performs an offset process. That is, the offset processing unit 143 uses the pixel value from the screen rearrangement buffer 12 and the pixel value after the deblocking filter processing from the category classification unit 142 for the set offset type and the classified category. Find the offset value.

Then, the pixel value after deblocking filter processing from the category classification unit 142 is subjected to offset processing with the set offset type, the classified category, and the obtained offset value. The offset processing unit 143 supplies the pixel value after adaptive offset processing to the adaptive loop filter 71.

In step S157, the offset processing unit 143 supplies the obtained offset value to the lossless encoding unit 16, and encodes information indicating the offset value.

On the other hand, if it is determined in step S154 that the adaptive offset filter is off, the adaptive offset processing ends. That is, in this case, the category classification unit 142 supplies the pixel value after the deblocking filter processing from the on / off determination unit 141 to the offset processing unit 143 as it is. Further, the offset processing unit 143 supplies the pixel value after the deblocking filter processing from the category classification unit 142 to the adaptive loop filter 71 as it is (without performing the offset processing).

As described above, in the image encoding device 101, whether it is an edge offset or a band offset is set according to the size of a block (for example, TU). Therefore, since it is not necessary to send the information indicating whether it is a region division and edge offset or band offset (that is, the map information of the quad-tree structure described above with reference to FIG. 8) to the decoding side, encoding is performed. Efficiency can be improved.

<3. Second Embodiment>
[Image decoding device]
FIG. 19 illustrates a configuration of an embodiment of an image decoding device as an image processing device to which the present disclosure is applied. An image decoding apparatus 201 shown in FIG. 19 is a decoding apparatus corresponding to the image encoding apparatus 101 in FIG.

Assume that encoded data encoded by the image encoding device 101 is transmitted to an image decoding device 201 corresponding to the image encoding device 101 via a predetermined transmission path and decoded.

The image decoding device 201 in FIG. 19 is common to the image decoding device 31 in FIG. 2 in that the storage buffer 41, the lossless decoding unit 42, the inverse quantization unit 43, the calculation unit 45, and the deblocking filter 46 are provided. . The image decoding apparatus 201 in FIG. 19 includes a screen rearrangement buffer 47, a D / A conversion unit 48, a frame memory 49, a selection unit 50, an intra prediction unit 51, a motion compensation unit 52, and an image selection unit 53. It is common with the image decoding apparatus 31 of FIG.

19 is different from the image decoding device 31 in FIG. 2 in that an adaptive loop filter 91 in FIG. 4 is added.

Furthermore, the image decoding apparatus 201 in FIG. 19 is different from the image decoding in FIG. 2 in that the inverse orthogonal transform unit 44 is replaced with the inverse orthogonal transform unit 211 and that the class classification unit 212 and the adaptive offset unit 213 are added. Different from the device 31.

That is, the lossless decoding unit 42 converts the information supplied from the accumulation buffer 41 and encoded by the lossless encoding unit 16 of FIG. 12 into the code of the lossless encoding unit 16 as in the case of the lossless decoding unit 42 of FIG. Decoding is performed using a method corresponding to the conversion method. At this time, in the example of FIG. 21, motion vector information, reference frame information, prediction mode information (information indicating an intra prediction mode or an inter prediction mode), information on a TU (orthogonal transform) size, an adaptive offset parameter, and the like are also decoded. Is done.

As described above, the adaptive offset parameter includes information indicating on / off of the adaptive offset processing (hereinafter, also referred to as an on / off flag) encoded by the lossless encoding unit 16 in FIG. 12, information indicating an offset value, and the like. Consists of. The adaptive offset parameter is supplied to the adaptive offset unit 213. Further, the lossless decoding unit 42 supplies information regarding the TU size to the inverse orthogonal transform unit 211.

The inverse orthogonal transform unit 211 performs inverse orthogonal transform corresponding to the TU size from the lossless decoding unit 42, and the decoded residual data corresponding to the residual data before being orthogonally transformed by the image encoding device 101 in FIG. obtain. The decoded residual data obtained by the inverse orthogonal transform is supplied to the calculation unit 45. Further, the inverse orthogonal transform unit 211 supplies the class classification unit 212 with information regarding the TU size from the lossless decoding unit 42.

The class classification unit 212 is configured similarly to the class classification unit 112 of FIG. That is, the class classification unit 212 performs class classification according to the TU size from the inverse orthogonal transform unit 211. The class classification in the class classification unit 212 is setting (classification) of the type of offset. That is, the class classification unit 212 sets whether to apply edge offset (EO) or band offset (BO) according to the TU size from the inverse orthogonal transform unit 211. The class classification unit 212 supplies information indicating the set offset type to the adaptive offset unit 213.

The deblock filter 46, the adaptive offset unit 213, and the adaptive loop filter 91 are provided in the motion compensation loop in that order. The motion compensation loop is a block composed of a calculation unit 45, a frame memory 49, a selection unit 50, a motion compensation unit 52, and an image selection unit 53. Hereinafter, the filter processing performed by the deblock filter 46, the adaptive offset unit 213, and the adaptive loop filter 91 in the motion compensation loop is also collectively referred to as in-loop filter processing.

The adaptive offset unit 213 is supplied with information indicating the on / off flag of the adaptive offset process and the offset value, which are the adaptive offset parameters from the lossless decoding unit 42. The adaptive offset unit 213 performs offset processing on the pixel value of the decoded image from the deblocking filter 46 using the information, and supplies the pixel value after the offset processing to the adaptive loop filter 91.

That is, whether to apply edge offset (EO) or band offset (BO) is set by the class classification unit 212 according to the TU size from the inverse orthogonal transform unit 211. Therefore, when the on / off flag from the lossless decoding unit 42 indicates “on”, the adaptive offset unit 213 performs category classification based on the offset type set by the class classification unit 212 using the pixel value after the deblocking filter. . Then, the adaptive offset unit 213 performs offset processing on the pixel value after the deblocking filter with the offset value from the lossless decoding unit 42 in the classified category with the set offset type. The adaptive offset unit 213 supplies the post-offset pixel value to the adaptive loop filter 91.

Although the illustration of the adaptive loop filter 91 is omitted, the adaptive loop filter coefficient decoded by the lossless decoding unit 42 and extracted from the header is supplied. The adaptive loop filter 91 performs a filtering process on the decoded image from the adaptive offset unit 213 using the supplied filter coefficient.

Note that the basic operation principles related to the present technology in the class classification unit 212 and the adaptive offset unit 213 are the same as those of the class classification unit 112 and the adaptive offset unit 113 in FIG. However, in the image encoding device 101 shown in FIG. 12, the TU size (or PU size) is determined by the mode determination, and thereby classifying the adaptive offset processing.

On the other hand, in the image decoding apparatus 201 shown in FIG. 19, information on the TU size (or PU size) is added to the encoded stream and sent from the encoding side. Therefore, the image decoding apparatus 201 obtains the information by decoding the information, and classifies the adaptive offset process based on the obtained size information.

[Configuration example of inverse orthogonal transform unit and adaptive offset unit]
Next, each unit of the image decoding device 201 will be described. FIG. 20 is a block diagram illustrating a configuration example of the inverse orthogonal transform unit 211 and the adaptive offset unit 213.

In the example of FIG. 20, the inverse orthogonal transform unit 211 includes a TU size buffer 231, a 4 × 4 inverse orthogonal transform unit 232, an 8 × 8 inverse orthogonal transform unit 233, a 16 × 16 inverse orthogonal transform unit 234, and a 32 × 32 inverse. An orthogonal transform unit 235 is included.

20, the adaptive offset unit 213 is configured to include an on / off flag buffer 241, a category classification unit 242, and an offset processing unit 243.

Information on the TU size of the block (TU) from the lossless decoding unit 42 is supplied to the TU size buffer 231. The TU size buffer 231 includes information on the TU size among the 4 × 4 inverse orthogonal transform unit 232, the 8 × 8 inverse orthogonal transform unit 233, the 16 × 16 inverse orthogonal transform unit 234, and the 32 × 32 inverse orthogonal transform unit 235. , To the inverse orthogonal transform unit of the corresponding size. As a result, the inverse orthogonal transform unit having the corresponding size becomes enable, and the orthogonal transform coefficient from the inverse quantization unit 43 is inversely orthogonal transformed to correspond to the residual data before being orthogonally transformed by the image coding apparatus 101. Decoding residual data is obtained.

That is, the orthogonal transform coefficients from the inverse quantization unit 43 are 4 × 4 inverse

orthogonal transform unit

232, 8 × 8 inverse

orthogonal transform unit

233, 16 × 16 inverse

orthogonal transform unit

234, and 32 × 32 inverse orthogonal transform unit 235. To be supplied.

The 4 × 4 inverse orthogonal transform unit 232 is enabled when the TU size from the TU size buffer 231 indicates 4 × 4, and the decoding residual corresponding to the residual data before being subjected to orthogonal transform in the image encoding device 101. Get the data. The 4 × 4 inverse orthogonal transform unit 232 supplies the obtained decoded residual data (difference value) to the calculation unit 45.

The 8 × 8 inverse orthogonal transform unit 233 is enabled when the TU size from the TU size buffer 231 indicates 8 × 8, and the decoding residual corresponding to the residual data before being subjected to orthogonal transform in the image encoding device 101. Get the data. The 8 × 8 inverse orthogonal transform unit 233 supplies the obtained decoded residual data (difference value) to the calculation unit 45.

The 16 × 16 inverse orthogonal transform unit 234 is enabled when the TU size from the TU size buffer 231 indicates 16 × 16, and the decoding residual corresponding to the residual data before being subjected to orthogonal transform in the image encoding device 101. Get the data. The 16 × 16 inverse orthogonal transform unit 234 supplies the obtained decoded residual data (difference value) to the calculation unit 45.

The 32 × 32 inverse orthogonal transform unit 235 is enabled when the TU size from the TU size buffer 231 indicates 32 × 32, and the decoding residual corresponding to the residual data before being orthogonally transformed in the image encoding device 101. Get the data. The 32 × 32 inverse orthogonal transform unit 235 supplies the obtained decoded residual data (difference value) to the calculation unit 45.

Information regarding the TU size from the TU size buffer 231 is also supplied to the class classification unit 212. The class classification unit 212 is basically configured similarly to the class classification unit 112 of FIG. That is, the class classification unit 112 determines the type of offset to be applied to the block (whether it is an edge offset or a band offset) based on the TU size from the TU size buffer 231 by the above-described method according to the present technology. Set. The class classification unit 212 supplies the set offset type information to the category classification unit 242.

The on / off flag from the lossless decoding unit 42 is supplied to the on / off flag buffer 241. The offset value from the lossless decoding unit 42 is supplied to the offset processing unit 243.

The on / off flag buffer 241 temporarily stores the on / off flag from the lossless decoding unit 42 and supplies it to the category classification unit 242 at a predetermined timing. The category classification unit 242 is further supplied with information indicating which of the edge offset / edge offset is applied and the pixel value after deblocking filter processing from the deblocking filter 46.

When the on / off flag from the on / off flag buffer 241 indicates “on”, the category classification unit 242 classifies the category in the offset type set by the class classification unit 212 using the pixel value after the deblocking filter processing. To do. The category classification unit 242 supplies information indicating the classified category and the pixel value after deblocking filter processing to the offset processing unit 243.

When the on / off flag from the on / off flag buffer 241 indicates off, the category classification unit 242 supplies the pixel value after the deblocking filter processing from the on / off flag buffer 241 to the offset processing unit 243 as it is. To do.

When the on / off information from the on / off flag buffer 241 indicates on, the offset processing unit 243 performs adaptive offset processing on the pixel value after the deblocking filter processing from the category classification unit 242. The offset processing unit 243 supplies the pixel value after adaptive offset processing to the adaptive loop filter 91.

That is, the offset processing unit 243 performs an offset process on the post-deblock filter-processed pixel value from the category classification unit 242 with the set offset type, the classified category, and the offset value from the lossless decoding unit 42. Apply.

When the on / off information from the on / off flag buffer 241 indicates “off”, the offset processing unit 243 uses the pixel value after the deblocking filter processing from the category classification unit 242 as it is (without performing offset processing). To the adaptive loop filter 91.

In the above description, an example in which category classification is performed on each of the encoding side and the decoding side has been described. However, the present invention is not limited to this, and category classification is performed on the encoding side and the information may be sent to the decoding side. Good.

[Decoding process flow]
Next, the flow of each process executed by the image decoding apparatus 201 as described above will be described. First, an example of the flow of decoding processing will be described with reference to the flowchart of FIG.

When the decoding process is started, in step S201, the accumulation buffer 41 accumulates the transmitted encoded data. In step S <b> 202, the lossless decoding unit 42 decodes the encoded data supplied from the accumulation buffer 41. That is, the I picture, P picture, and B picture encoded by the lossless encoding unit 16 of FIG. 12 are decoded.

At this time, motion vector information, reference frame information, prediction mode information (intra prediction mode or inter prediction mode), information on the TU size, and information on adaptive offset parameters are also decoded.

When the prediction mode information is intra prediction mode information, the prediction mode information is supplied to the intra prediction unit 51. When the prediction mode information is inter prediction mode information, motion vector information corresponding to the prediction mode information is supplied to the motion compensation unit 52. Information about the TU size is supplied to the inverse orthogonal transform unit 211. The on / off flag information, which is an adaptive offset parameter, and information indicating the offset value are supplied to the adaptive offset unit 213.

In step S203, the intra prediction unit 51 or the motion compensation unit 52 performs a prediction image generation process corresponding to the prediction mode information supplied from the lossless decoding unit 42, respectively.

That is, when the intra prediction mode information is supplied from the lossless decoding unit 42, the intra prediction unit 51 generates Most Probable 並列 Mode and generates an intra prediction image of the intra prediction mode by parallel processing. When the inter prediction mode information is supplied from the lossless decoding unit 42, the motion compensation unit 52 performs a motion prediction / compensation process in the inter prediction mode, and generates an inter prediction image.

Through this process, the prediction image (intra prediction image) generated by the intra prediction unit 51 or the prediction image (inter prediction image) generated by the motion compensation unit 52 is supplied to the image selection unit 53.

In step S204, the image selection unit 53 selects a predicted image. That is, the prediction image generated by the intra prediction unit 51 or the prediction image generated by the motion compensation unit 52 is supplied. Therefore, the supplied predicted image is selected and supplied to the calculation unit 45, and is added to the output of the inverse orthogonal transform unit 44 in step S208 described later.

In step S202 described above, the transform coefficient decoded by the lossless decoding unit 42 is also supplied to the inverse quantization unit 43. In step S205, the inverse quantization unit 43 inverts the transform coefficient decoded by the lossless decoding unit 42 with the quantization parameter decoded by the lossless decoding unit 42 with the characteristic corresponding to the characteristic of the quantization unit 15 of FIG. Quantize.

In step S206, the TU size buffer 231 of the inverse orthogonal transform unit 211 receives information regarding the TU (orthogonal transform) size supplied in step S202. This information about the TU size corresponds to the TU size among the 4 × 4 inverse orthogonal transform unit 232, the 8 × 8 inverse orthogonal transform unit 233, the 16 × 16 inverse orthogonal transform unit 234, and the 32 × 32 inverse orthogonal transform unit 235. To the inverse orthogonal transform unit. Note that the information regarding the TU size is also supplied to the class classification unit 212.

In step S207, the inverse orthogonal transform unit corresponding to the TU size in the inverse orthogonal transform unit 211 converts the transform coefficient inversely quantized by the inverse quantization unit 43 with characteristics corresponding to the characteristics of the orthogonal transform unit 111 in FIG. Perform inverse orthogonal transform. As a result, the difference information corresponding to the input of the orthogonal transform unit 111 in FIG. 12 (the output of the calculation unit 13) is decoded.

In step S208, the calculation unit 45 adds the predicted image selected in the processing in step S204 described above and input via the image selection unit 53 to the difference information. As a result, the original image is decoded.

In step S209, the deblock filter 46 performs deblock filter processing on the image from the calculation unit 45. Thereby, block distortion is suppressed. The deblock filter 46 supplies the post-deblock filter processed pixel value to the adaptive offset unit 213.

In step S210, the class classification unit 212 and the adaptive offset unit 213 perform adaptive offset processing according to the orthogonal transform size received in step S206. Details of the adaptive offset processing will be described later with reference to FIG. Thus, the adaptive offset process is performed to remove ringing and the like.

The pixel value after adaptive offset processing is supplied to the adaptive loop filter 91 by the processing in step S210. In step S211, the adaptive loop filter 91 performs an adaptive loop filter process on the pixel value after the adaptive offset process, and supplies the pixel value after the adaptive loop filter to the frame memory 49 or the screen rearrangement buffer 47.

In step S212, the frame memory 49 stores the adaptively filtered image.

In step S213, the screen rearrangement buffer 47 rearranges the images after the adaptive loop filter 91. That is, the order of frames rearranged for encoding by the screen rearrangement buffer 12 of the image encoding device 101 is rearranged to the original display order.

In step S214, the D / A converter 48 D / A converts the image from the screen rearrangement buffer 47. This image is output to a display (not shown), and the image is displayed.

When the process of step S214 is completed, the decoding process is terminated.

[Flow of adaptive offset processing]
Next, an example of the flow of adaptive offset processing executed in step S210 in FIG. 21 will be described with reference to the flowchart in FIG.

21, the TU size buffer 231 of the inverse orthogonal transform unit 211 supplies information on the TU size to the class classification unit 212 by the process of step S206 in FIG. In step S251, the class classification unit 212 determines whether the edge offset (EO) or the band offset (BO) is applied to the block (eg, TU) according to the method of the present technology described above according to the supplied TU size. Set whether to apply. The class classification unit 212 supplies the set offset type information to the category classification unit 242.

In step S252, the on / off flag buffer 241 receives the on / off flag supplied from the lossless decoding unit 42 by the processing in step S202 of FIG.

In step S253, the category classification unit 242 determines whether the adaptive offset filter is on for the TU based on the on / off flag from the on / off flag buffer 241. If it is determined in step S253 that the adaptive offset filter is on, the process proceeds to step S254.

In step S254, the category classification unit 242 classifies the category in the offset type set by the class classification unit 212 using the pixel value after the deblocking filter processing from the deblocking filter 46. The category classification unit 242 supplies information indicating the classified category and the pixel value after deblocking filter processing to the offset processing unit 243.

In step S255, the offset processing unit 243 receives the offset value supplied from the lossless decoding unit 42 by the process in step S202 of FIG.

In step S256, the offset processing unit 243 performs an offset process. That is, the offset processing unit 243 performs an offset process on the post-deblock filter processed pixel value from the category classification unit 242 with the set offset type, the classified category, and the received offset value. The offset processing unit 243 supplies the pixel value after adaptive offset processing to the adaptive loop filter 91.

On the other hand, if it is determined in step S253 that the adaptive offset filter is off, the adaptive offset processing ends. That is, in this case, the category classification unit 242 supplies the pixel value after the deblocking filter processing from the on / off flag buffer 241 to the offset processing unit 243 as it is. Further, the offset processing unit 143 supplies the pixel value after the deblocking filter processing from the category classification unit 242 as it is (without performing the offset processing) to the adaptive loop filter 91.

As described above, also in the image decoding apparatus 201, whether it is an edge offset or a band offset is set according to the size of a block (for example, TU). Therefore, since it is not necessary to send the information indicating whether it is a region division and edge offset or band offset (that is, the map information of the quad-tree structure described above with reference to FIG. 8) to the decoding side, encoding is performed. Efficiency can be improved.

In the above description, the case of conforming to the HEVC method has been described as an example. However, the present technology can be applied to a device using another coding method as long as the device performs adaptive offset processing.

Note that this disclosure includes, for example, MPEG, When receiving image information (bitstream) compressed by orthogonal transform such as discrete cosine transform and motion compensation, such as 26x, via network media such as satellite broadcasting, cable television, the Internet, or mobile phones. The present invention can be applied to an image encoding device and an image decoding device used in the above. In addition, the present disclosure can be applied to an image encoding device and an image decoding device that are used when processing on a storage medium such as an optical disk, a magnetic disk, and a flash memory. Furthermore, the present disclosure can also be applied to motion prediction / compensation devices included in such image encoding devices and image decoding devices.

<4. Third Embodiment>
[Application to multi-view image coding and multi-view image decoding]
The series of processes described above can be applied to multi-view image encoding / multi-view image decoding. FIG. 23 shows an example of a multi-view image encoding method.

23, the multi-viewpoint image includes a plurality of viewpoint images, and a predetermined one viewpoint image among the plurality of viewpoints is designated as the base view image. Each viewpoint image other than the base view image is treated as a non-base view image.

When performing multi-view image coding as shown in FIG. 23, in each view (same view), information about TU size, and parameters such as adaptive offset parameters such as on / off information and offset values (hereinafter simply referred to as parameters). ) Can also be set. Each view (different view) can also share parameters set in other views.

In this case, the parameters set in the base view are used in at least one non-base view. Alternatively, for example, a parameter set in the non-base view (view_id = i) is used in at least one of the base view and the non-base view (view_id = j).

This can reduce the information sent to the decoding side and improve the encoding efficiency.

[Multi-view image encoding device]
FIG. 24 is a diagram illustrating a multi-view image encoding apparatus that performs the multi-view image encoding described above. As illustrated in FIG. 24, the multi-view image encoding device 600 includes an encoding unit 601, an encoding unit 602, and a multiplexing unit 603.

The encoding unit 601 encodes the base view image and generates a base view image encoded stream. The encoding unit 602 encodes the non-base view image and generates a non-base view image encoded stream. The multiplexing unit 603 multiplexes the base view image encoded stream generated by the encoding unit 601 and the non-base view image encoded stream generated by the encoding unit 602 to generate a multi-view image encoded stream. To do.

The image encoding device 101 (FIG. 12) can be applied to the encoding unit 601 and the encoding unit 602 of the multi-view image encoding device 600. In this case, the multi-view image encoding apparatus 600 sets and transmits the parameters set by the encoding unit 601 and the parameters set by the encoding unit 602.

Note that the parameters set by the encoding unit 601 as described above may be set and transmitted so as to be shared and used by the encoding unit 601 and the encoding unit 602. Conversely, the parameters set by the encoding unit 602 may be set and transmitted so as to be shared by the encoding unit 601 and the encoding unit 602.

[Multi-viewpoint image decoding device]
FIG. 25 is a diagram illustrating a multi-view image decoding apparatus that performs the above-described multi-view image decoding. As illustrated in FIG. 25, the multi-view image decoding device 610 includes a demultiplexing unit 611, a decoding unit 612, and a decoding unit 613.

The demultiplexing unit 611 demultiplexes the multi-view image encoded stream in which the base view image encoded stream and the non-base view image encoded stream are multiplexed, and the base view image encoded stream and the non-base view image The encoded stream is extracted. The decoding unit 612 decodes the base view image encoded stream extracted by the demultiplexing unit 611 to obtain a base view image. The decoding unit 613 decodes the non-base view image encoded stream extracted by the demultiplexing unit 611 to obtain a non-base view image.

The image decoding device 201 (FIG. 19) can be applied to the decoding unit 612 and the decoding unit 613 of the multi-view image decoding device 610. In this case, the multi-view image decoding apparatus 610 performs processing using the parameters set by the encoding unit 601 and decoded by the decoding unit 612 and the parameters set by the encoding unit 602 and decoded by the decoding unit 613.

Note that the parameters set by the encoding unit 601 (or encoding 602) as described above may be set and transmitted so as to be shared by the encoding unit 601 and the encoding unit 602. In this case, in the multi-viewpoint image decoding apparatus 610, processing is performed using the parameters set by the encoding unit 601 (or encoding 602) and decoded by the decoding unit 612 (or decoding unit 613).

<5. Fourth Embodiment>
[Application to hierarchical image coding / hierarchical image decoding]
The series of processes described above can be applied to hierarchical image encoding / hierarchical image decoding. FIG. 26 shows an example of the multi-view image encoding method.

As shown in FIG. 26, a hierarchical image includes images of a plurality of layers (resolutions), and an image of a predetermined one layer among the plurality of resolutions is designated as a base layer image. Images in each layer other than the base layer image are treated as non-base layer images.

In the case of performing hierarchical image coding (spatial scalability) as shown in FIG. 26, in each layer (same layer), information on the TU size, and parameters such as adaptive offset parameters such as on / off information and offset values (hereinafter simply referred to as “only”). (Referred to as parameters) can also be set. In addition, each layer (different layers) can share a buffer index set in another view.

In this case, the parameters set in the base layer are used in at least one non-base layer. Alternatively, for example, a parameter set in a non-base layer (layer _id = i) is used in at least one of the base layer and the non-base layer (layer_id = j).

[Hierarchical image encoding device]
FIG. 27 is a diagram illustrating a hierarchical image encoding apparatus that performs the above-described hierarchical image encoding. As illustrated in FIG. 27, the hierarchical image encoding device 620 includes an encoding unit 621, an encoding unit 622, and a multiplexing unit 623.

The encoding unit 621 encodes the base layer image and generates a base layer image encoded stream. The encoding unit 622 encodes the non-base layer image and generates a non-base layer image encoded stream. The multiplexing unit 623 multiplexes the base layer image encoded stream generated by the encoding unit 621 and the non-base layer image encoded stream generated by the encoding unit 622 to generate a hierarchical image encoded stream. .

The image encoding device 101 (FIG. 12) can be applied to the encoding unit 621 and the encoding unit 622 of the hierarchical image encoding device 620. In this case, the hierarchical image encoding device 620 sets and transmits the parameters set by the encoding unit 621 and the parameters set by the encoding unit 622.

Note that the parameters set by the encoding unit 621 as described above may be set and transmitted so as to be shared by the encoding unit 621 and the encoding unit 622. Conversely, the parameters set by the encoding unit 622 may be set and transmitted so as to be shared by the encoding unit 621 and the encoding unit 622.

[Hierarchical image decoding device]
FIG. 28 is a diagram illustrating a hierarchical image decoding apparatus that performs the hierarchical image decoding described above. As illustrated in FIG. 28, the hierarchical image decoding device 630 includes a demultiplexing unit 631, a decoding unit 632, and a decoding unit 633.

The demultiplexing unit 631 demultiplexes the hierarchical image encoded stream in which the base layer image encoded stream and the non-base layer image encoded stream are multiplexed, and the base layer image encoded stream and the non-base layer image code Stream. The decoding unit 632 decodes the base layer image encoded stream extracted by the demultiplexing unit 631 to obtain a base layer image. The decoding unit 633 decodes the non-base layer image encoded stream extracted by the demultiplexing unit 631 to obtain a non-base layer image.

The image decoding device 201 (FIG. 19) can be applied to the decoding unit 632 and the decoding unit 633 of the hierarchical image decoding device 630. In this case, in the hierarchical image decoding apparatus 630, the parameter set by the encoding unit 621, the parameter decoded by the decoding unit 632, and the encoding unit 622 are set, and the decoding unit 633 performs processing using the parameter.

Note that, as described above, the parameter set by the encoding unit 621 (or encoding 622) may be set and transmitted so as to be shared by the encoding unit 621 and the encoding unit 622. In this case, in the hierarchical image decoding apparatus 630, processing is performed using the parameters set by the encoding unit 621 (or encoding 622) and decoded by the decoding unit 632 (or decoding unit 633).

<6. Fifth embodiment>
[Computer]
The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes a computer incorporated in dedicated hardware, a general-purpose personal computer capable of executing various functions by installing various programs, and the like.

FIG. 29 is a block diagram illustrating a configuration example of hardware of a computer that executes the above-described series of processes by a program.

In the computer 800, a CPU (Central Processing Unit) 801, a ROM (Read Only Memory) 802, and a RAM (Random Access Memory) 803 are connected to each other by a bus 804.

Further, an input / output interface 805 is connected to the bus 804. An input unit 806, an output unit 807, a storage unit 808, a communication unit 809, and a drive 810 are connected to the input / output interface 805.

The input unit 811 includes a keyboard, a mouse, a microphone, and the like. The output unit 812 includes a display, a speaker, and the like. The storage unit 813 includes a hard disk, a nonvolatile memory, and the like. The communication unit 814 includes a network interface or the like. The drive 815 drives a removable medium 821 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 801 loads the program stored in the storage unit 83 into the RAM 803 via the input / output interface 810 and the bus 804 and executes the program, for example. Is performed.

The program executed by the computer 800 (CPU 801) can be provided by being recorded on a removable medium 821 as a package medium, for example. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the program can be installed in the storage unit 813 via the input / output interface 810 by attaching the removable medium 821 to the drive 815. Further, the program can be received by the communication unit 814 via a wired or wireless transmission medium and installed in the storage unit 813. In addition, the program can be installed in the ROM 802 or the storage unit 813 in advance.

The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

Further, in the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but may be performed in parallel or It also includes processes that are executed individually.

In addition, in this specification, the system represents the entire apparatus composed of a plurality of devices (apparatuses).

Also, in the above, the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, the configurations described above as a plurality of devices (or processing units) may be combined into a single device (or processing unit). Of course, a configuration other than that described above may be added to the configuration of each device (or each processing unit). Furthermore, if the configuration and operation of the entire system are substantially the same, a part of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or other processing unit). . That is, the present technology is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present technology.

An image encoding device and an image decoding device according to the above-described embodiments include a transmitter or a receiver in optical broadcasting, satellite broadcasting, cable broadcasting such as cable TV, distribution on the Internet, and distribution to terminals by cellular communication, etc. The present invention can be applied to various electronic devices such as a recording device that records an image on a medium such as a magnetic disk and a flash memory, or a playback device that reproduces an image from these storage media. Hereinafter, four application examples will be described.

<7. Application example>
[First application example: television receiver]
FIG. 30 illustrates an example of a schematic configuration of a television device to which the above-described embodiment is applied. The television apparatus 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing unit 905, a display unit 906, an audio signal processing unit 907, a speaker 908, an external interface 909, a control unit 910, a user interface 911, And a bus 912.

Tuner 902 extracts a signal of a desired channel from a broadcast signal received via antenna 901, and demodulates the extracted signal. Then, the tuner 902 outputs the encoded bit stream obtained by the demodulation to the demultiplexer 903. In other words, the tuner 902 serves as a transmission unit in the television apparatus 900 that receives an encoded stream in which an image is encoded.

The demultiplexer 903 separates the video stream and audio stream of the viewing target program from the encoded bit stream, and outputs each separated stream to the decoder 904. Further, the demultiplexer 903 extracts auxiliary data such as EPG (Electronic Program Guide) from the encoded bit stream, and supplies the extracted data to the control unit 910. Note that the demultiplexer 903 may perform descrambling when the encoded bit stream is scrambled.

The decoder 904 decodes the video stream and audio stream input from the demultiplexer 903. Then, the decoder 904 outputs the video data generated by the decoding process to the video signal processing unit 905. In addition, the decoder 904 outputs audio data generated by the decoding process to the audio signal processing unit 907.

The video signal processing unit 905 reproduces the video data input from the decoder 904 and causes the display unit 906 to display the video. In addition, the video signal processing unit 905 may cause the display unit 906 to display an application screen supplied via a network. Further, the video signal processing unit 905 may perform additional processing such as noise removal on the video data according to the setting. Furthermore, the video signal processing unit 905 may generate a GUI (Graphical User Interface) image such as a menu, a button, or a cursor, and superimpose the generated image on the output image.

The display unit 906 is driven by a drive signal supplied from the video signal processing unit 905, and displays an image on a video screen of a display device (for example, a liquid crystal display, a plasma display, or an OELD (Organic ElectroLuminescence Display) (organic EL display)). Or an image is displayed.

The audio signal processing unit 907 performs reproduction processing such as D / A conversion and amplification on the audio data input from the decoder 904, and outputs audio from the speaker 908. The audio signal processing unit 907 may perform additional processing such as noise removal on the audio data.

The external interface 909 is an interface for connecting the television apparatus 900 to an external device or a network. For example, a video stream or an audio stream received via the external interface 909 may be decoded by the decoder 904. That is, the external interface 909 also has a role as a transmission unit in the television apparatus 900 that receives an encoded stream in which an image is encoded.

The control unit 910 includes a processor such as a CPU and memories such as a RAM and a ROM. The memory stores a program executed by the CPU, program data, EPG data, data acquired via a network, and the like. For example, the program stored in the memory is read and executed by the CPU when the television apparatus 900 is activated. The CPU executes the program to control the operation of the television device 900 according to an operation signal input from the user interface 911, for example.

The user interface 911 is connected to the control unit 910. The user interface 911 includes, for example, buttons and switches for the user to operate the television device 900, a remote control signal receiving unit, and the like. The user interface 911 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 910.

The bus 912 connects the tuner 902, the demultiplexer 903, the decoder 904, the video signal processing unit 905, the audio signal processing unit 907, the external interface 909, and the control unit 910 to each other.

In the thus configured television apparatus 900, the decoder 904 has the function of the image decoding apparatus according to the above-described embodiment. Thereby, when the image is decoded by the television apparatus 900, information to be sent to the decoding side is reduced, and the encoding efficiency can be improved.

[Second application example: mobile phone]
FIG. 31 shows an example of a schematic configuration of a mobile phone to which the above-described embodiment is applied. A mobile phone 920 includes an antenna 921, a communication unit 922, an audio codec 923, a speaker 924, a microphone 925, a camera unit 926, an image processing unit 927, a demultiplexing unit 928, a recording / reproducing unit 929, a display unit 930, a control unit 931, an operation A portion 932 and a bus 933.

The antenna 921 is connected to the communication unit 922. The speaker 924 and the microphone 925 are connected to the audio codec 923. The operation unit 932 is connected to the control unit 931. The bus 933 connects the communication unit 922, the audio codec 923, the camera unit 926, the image processing unit 927, the demultiplexing unit 928, the recording / reproducing unit 929, the display unit 930, and the control unit 931 to each other.

The mobile phone 920 has various operation modes including a voice call mode, a data communication mode, a shooting mode, and a videophone mode, and is used for sending and receiving voice signals, sending and receiving e-mail or image data, taking images, and recording data. Perform the action.

In the voice call mode, the analog voice signal generated by the microphone 925 is supplied to the voice codec 923. The audio codec 923 converts an analog audio signal into audio data, A / D converts the compressed audio data, and compresses it. Then, the audio codec 923 outputs the compressed audio data to the communication unit 922. The communication unit 922 encodes and modulates the audio data and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal. Then, the communication unit 922 demodulates and decodes the received signal to generate audio data, and outputs the generated audio data to the audio codec 923. The audio codec 923 decompresses the audio data and performs D / A conversion to generate an analog audio signal. Then, the audio codec 923 supplies the generated audio signal to the speaker 924 to output audio.

Further, in the data communication mode, for example, the control unit 931 generates character data constituting the e-mail in response to an operation by the user via the operation unit 932. In addition, the control unit 931 causes the display unit 930 to display characters. In addition, the control unit 931 generates e-mail data in response to a transmission instruction from the user via the operation unit 932, and outputs the generated e-mail data to the communication unit 922. The communication unit 922 encodes and modulates email data and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal. Then, the communication unit 922 demodulates and decodes the received signal to restore the email data, and outputs the restored email data to the control unit 931. The control unit 931 displays the content of the electronic mail on the display unit 930 and stores the electronic mail data in the storage medium of the recording / reproducing unit 929.

The recording / reproducing unit 929 has an arbitrary readable / writable storage medium. For example, the storage medium may be a built-in storage medium such as a RAM or a flash memory, or an externally mounted type such as a hard disk, magnetic disk, magneto-optical disk, optical disk, USB (Universal Serial Bus) memory, or memory card. It may be a storage medium.

In the shooting mode, for example, the camera unit 926 images a subject to generate image data, and outputs the generated image data to the image processing unit 927. The image processing unit 927 encodes the image data input from the camera unit 926 and stores the encoded stream in the storage medium of the storage / playback unit 929.

Further, in the videophone mode, for example, the demultiplexing unit 928 multiplexes the video stream encoded by the image processing unit 927 and the audio stream input from the audio codec 923, and the multiplexed stream is the communication unit 922. Output to. The communication unit 922 encodes and modulates the stream and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal. These transmission signal and reception signal may include an encoded bit stream. Then, the communication unit 922 demodulates and decodes the received signal to restore the stream, and outputs the restored stream to the demultiplexing unit 928. The demultiplexing unit 928 separates the video stream and the audio stream from the input stream, and outputs the video stream to the image processing unit 927 and the audio stream to the audio codec 923. The image processing unit 927 decodes the video stream and generates video data. The video data is supplied to the display unit 930, and a series of images is displayed on the display unit 930. The audio codec 923 decompresses the audio stream and performs D / A conversion to generate an analog audio signal. Then, the audio codec 923 supplies the generated audio signal to the speaker 924 to output audio.

In the mobile phone 920 configured as described above, the image processing unit 927 has the functions of the image encoding device and the image decoding device according to the above-described embodiment. Accordingly, when encoding and decoding an image with the mobile phone 920, information to be sent to the decoding side is reduced, and encoding efficiency can be improved.

[Third application example: recording / reproducing apparatus]
FIG. 32 shows an example of a schematic configuration of a recording / reproducing apparatus to which the above-described embodiment is applied. For example, the recording / reproducing device 940 encodes audio data and video data of a received broadcast program and records the encoded data on a recording medium. In addition, the recording / reproducing device 940 may encode audio data and video data acquired from another device and record them on a recording medium, for example. In addition, the recording / reproducing device 940 reproduces data recorded on the recording medium on a monitor and a speaker, for example, in accordance with a user instruction. At this time, the recording / reproducing device 940 decodes the audio data and the video data.

The recording / reproducing apparatus 940 includes a tuner 941, an external interface 942, an encoder 943, an HDD (Hard Disk Drive) 944, a disk drive 945, a selector 946, a decoder 947, an OSD (On-Screen Display) 948, a control unit 949, and a user interface. 950.

Tuner 941 extracts a signal of a desired channel from a broadcast signal received via an antenna (not shown), and demodulates the extracted signal. Then, the tuner 941 outputs the encoded bit stream obtained by the demodulation to the selector 946. That is, the tuner 941 has a role as a transmission unit in the recording / reproducing apparatus 940.

The external interface 942 is an interface for connecting the recording / reproducing apparatus 940 to an external device or a network. The external interface 942 may be, for example, an IEEE1394 interface, a network interface, a USB interface, or a flash memory interface. For example, video data and audio data received via the external interface 942 are input to the encoder 943. That is, the external interface 942 serves as a transmission unit in the recording / reproducing device 940.

The encoder 943 encodes video data and audio data when the video data and audio data input from the external interface 942 are not encoded. Then, the encoder 943 outputs the encoded bit stream to the selector 946.

The HDD 944 records an encoded bit stream in which content data such as video and audio is compressed, various programs, and other data on an internal hard disk. Further, the HDD 944 reads out these data from the hard disk when reproducing video and audio.

The disk drive 945 performs recording and reading of data to and from the mounted recording medium. The recording medium mounted on the disk drive 945 is, for example, a DVD disk (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD + R, DVD + RW, etc.) or a Blu-ray (registered trademark) disk. It may be.

The selector 946 selects an encoded bit stream input from the tuner 941 or the encoder 943 when recording video and audio, and outputs the selected encoded bit stream to the HDD 944 or the disk drive 945. In addition, the selector 946 outputs the encoded bit stream input from the HDD 944 or the disk drive 945 to the decoder 947 during video and audio reproduction.

The decoder 947 decodes the encoded bit stream and generates video data and audio data. Then, the decoder 947 outputs the generated video data to the OSD 948. The decoder 904 outputs the generated audio data to an external speaker.

OSD 948 reproduces the video data input from the decoder 947 and displays the video. Further, the OSD 948 may superimpose a GUI image such as a menu, a button, or a cursor on the video to be displayed.

The control unit 949 includes a processor such as a CPU and memories such as a RAM and a ROM. The memory stores a program executed by the CPU, program data, and the like. The program stored in the memory is read and executed by the CPU when the recording / reproducing apparatus 940 is activated, for example. The CPU controls the operation of the recording / reproducing apparatus 940 in accordance with an operation signal input from the user interface 950, for example, by executing the program.

The user interface 950 is connected to the control unit 949. The user interface 950 includes, for example, buttons and switches for the user to operate the recording / reproducing device 940, a remote control signal receiving unit, and the like. The user interface 950 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 949.

In the thus configured recording / reproducing apparatus 940, the encoder 943 has the function of the image encoding apparatus according to the above-described embodiment. The decoder 947 has the function of the image decoding apparatus according to the above-described embodiment. Thereby, when encoding and decoding an image in the recording / reproducing apparatus 940, information to be sent to the decoding side is reduced, and encoding efficiency can be improved.

[Fourth Application Example: Imaging Device]
FIG. 33 illustrates an example of a schematic configuration of an imaging apparatus to which the above-described embodiment is applied. The imaging device 960 images a subject to generate an image, encodes the image data, and records it on a recording medium.

The imaging device 960 includes an optical block 961, an imaging unit 962, a signal processing unit 963, an image processing unit 964, a display unit 965, an external interface 966, a memory 967, a media drive 968, an OSD 969, a control unit 970, a user interface 971, and a bus. 972.

The optical block 961 is connected to the imaging unit 962. The imaging unit 962 is connected to the signal processing unit 963. The display unit 965 is connected to the image processing unit 964. The user interface 971 is connected to the control unit 970. The bus 972 connects the image processing unit 964, the external interface 966, the memory 967, the media drive 968, the OSD 969, and the control unit 970 to each other.

The optical block 961 includes a focus lens and a diaphragm mechanism. The optical block 961 forms an optical image of the subject on the imaging surface of the imaging unit 962. The imaging unit 962 includes an image sensor such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor), and converts an optical image formed on the imaging surface into an image signal as an electrical signal by photoelectric conversion. Then, the imaging unit 962 outputs the image signal to the signal processing unit 963.

The signal processing unit 963 performs various camera signal processing such as knee correction, gamma correction, and color correction on the image signal input from the imaging unit 962. The signal processing unit 963 outputs the image data after the camera signal processing to the image processing unit 964.

The image processing unit 964 encodes the image data input from the signal processing unit 963 and generates encoded data. Then, the image processing unit 964 outputs the generated encoded data to the external interface 966 or the media drive 968. The image processing unit 964 also decodes encoded data input from the external interface 966 or the media drive 968 to generate image data. Then, the image processing unit 964 outputs the generated image data to the display unit 965. In addition, the image processing unit 964 may display the image by outputting the image data input from the signal processing unit 963 to the display unit 965. Further, the image processing unit 964 may superimpose display data acquired from the OSD 969 on an image output to the display unit 965.

The OSD 969 generates a GUI image such as a menu, a button, or a cursor, and outputs the generated image to the image processing unit 964.

The external interface 966 is configured as a USB input / output terminal, for example. The external interface 966 connects the imaging device 960 and a printer, for example, when printing an image. Further, a drive is connected to the external interface 966 as necessary. For example, a removable medium such as a magnetic disk or an optical disk is attached to the drive, and a program read from the removable medium can be installed in the imaging device 960. Further, the external interface 966 may be configured as a network interface connected to a network such as a LAN or the Internet. That is, the external interface 966 has a role as a transmission unit in the imaging device 960.

The recording medium mounted on the media drive 968 may be any readable / writable removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory. In addition, a recording medium may be fixedly mounted on the media drive 968, and a non-portable storage unit such as an internal hard disk drive or an SSD (Solid State Drive) may be configured.

The control unit 970 includes a processor such as a CPU and memories such as a RAM and a ROM. The memory stores a program executed by the CPU, program data, and the like. The program stored in the memory is read and executed by the CPU when the imaging device 960 is activated, for example. For example, the CPU controls the operation of the imaging device 960 according to an operation signal input from the user interface 971 by executing the program.

The user interface 971 is connected to the control unit 970. The user interface 971 includes, for example, buttons and switches for the user to operate the imaging device 960. The user interface 971 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 970.

In the imaging device 960 configured as described above, the image processing unit 964 has the functions of the image encoding device and the image decoding device according to the above-described embodiment. Thereby, when encoding and decoding an image in the imaging device 960, information to be sent to the decoding side is reduced, and encoding efficiency can be improved.

In this specification, various types of information such as syntax elements such as information on TU size, adaptive offset parameters such as on / off information and offset values, etc. are multiplexed into the encoded stream, and from the encoding side to the decoding side. An example of transmission has been described. However, the method for transmitting such information is not limited to such an example. For example, these pieces of information may be transmitted or recorded as separate data associated with the encoded bitstream without being multiplexed into the encoded bitstream. Here, the term “associate” means that an image (which may be a part of an image such as a slice or a block) included in the bitstream and information corresponding to the image can be linked at the time of decoding. Means. That is, information may be transmitted on a transmission path different from that of the image (or bit stream). Information may be recorded on a recording medium (or another recording area of the same recording medium) different from the image (or bit stream). Furthermore, the information and the image (or bit stream) may be associated with each other in an arbitrary unit such as a plurality of frames, one frame, or a part of the frame.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present disclosure belongs can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present disclosure.

In addition, this technique can also take the following structures.
(1) a decoding unit that generates an image by decoding an encoded stream encoded in units having a hierarchical structure;
An offset setting unit that sets an offset type of adaptive offset processing according to the size or area of the block of the image generated by the decoding unit;
An image processing apparatus comprising: an adaptive offset processing unit that performs the adaptive offset processing on an image generated by the decoding unit with an offset type set by the offset setting unit.
(2) The offset setting unit sets a band offset for the block when the size or area of the block is large, and sets an edge offset for the block when the size or area of the block is small. Set The image processing apparatus according to (1).
(3) The image processing apparatus according to (1) or (2), wherein the block is a TU (Transform Unit).
(4) a receiving unit that receives the encoded stream and on / off information indicating on or off of the adaptive offset processing;
The decoding unit decodes the encoded stream received by the receiving unit to generate the image,
When the on / off information received by the receiving unit indicates that the adaptive offset processing is on, the adaptive offset processing unit targets the image generated by the decoding unit with the type of offset set by the offset setting unit. The image processing apparatus according to any one of (1) to (3), wherein the adaptive offset processing is performed.
(5) The offset setting unit sets a band offset when the size or area of the block is equal to or larger than the first size or the first area, and the size or area of the block is the first size or the first size. The image processing apparatus according to any one of (1) to (3), wherein an edge offset is set when the area is smaller than 1.
(6) When the size or area of the block is equal to or larger than the first size or the second size or the second area larger than the first area, the offset setting unit sets the adaptive offset processing to be off. The image processing apparatus according to (5).
(7) When the NSQT (Non-Square Quadtree Transform) is applied to the block, the offset setting unit sets an edge offset for the block. Any one of (1) to (6) An image processing apparatus according to 1.
(8) When NSQT (Non-Square Quadtree Transform) is applied to the block, the offset setting unit determines the type of the offset in the adaptive offset processing according to the size or area of the short side of the block. The image processing apparatus according to any one of (1) to (6) to be set.
(9) The block is an LCU (Largest Coding Unit),
The image processing apparatus according to any one of (1) and (2), wherein the offset setting unit sets an offset type of the adaptive offset processing according to integration of areas of subblocks included in the LCU.
(10) The image processing apparatus is
An image is generated by decoding an encoded stream encoded in units having a hierarchical structure,
Set the offset type of adaptive offset processing according to the size or area of the block of the generated image,
An image processing method for performing the adaptive offset processing on a generated image with a set type of offset.
(11) an offset setting unit that sets an offset type of adaptive offset processing according to the size or area of a block of an image subjected to local decoding processing when encoding an image;
An offset type set by the offset setting unit, an adaptive offset processing unit that performs the adaptive offset processing for the image, and
An image processing apparatus comprising: an encoding unit that encodes the image in units having a hierarchical structure using the image on which the adaptive offset processing has been performed by the adaptive offset processing unit.
(12) The offset setting unit sets a band offset for the block when the size or area of the block is large, and sets an edge offset for the block when the size or area of the block is small. Setting The image processing apparatus according to (11).
(13) The image processing device according to (11) or (12), wherein the block is a TU (Transform Unit).
(14) A transmission unit that transmits the image encoded by the encoding unit is further provided,
The adaptive offset processing unit determines whether to turn on or off the adaptive offset processing. When the adaptive offset processing is on, the adaptive offset processing unit sets the adaptive offset for the image with the type of offset set by the offset setting unit. Process,
The image processing apparatus according to any one of (10) to (13), wherein the transmission unit transmits on / off information indicating on or off of the adaptive offset processing.
(15) When the size or area of the block is equal to or larger than the first size or the first area, the offset setting unit sets a band offset, and the size or area of the block is the first size or the first size. The image processing apparatus according to any one of (11) to (13), wherein an edge offset is set when the area is smaller than 1.
(16) When the size or area of the block is equal to or larger than the first size or the second size or the second area larger than the first area, the offset setting unit sets the adaptive offset processing to be off. The image processing apparatus according to (15).
(17) The image processing device according to (16), wherein the offset setting unit sets an edge offset for the block when NSQT (Non-Square Quadtree Transform) is applied to the block.
(18) When NSQT (Non-Square Quadtree Transform) is applied to the block, the offset setting unit determines the type of offset of the adaptive offset processing according to the size or area of the short side of the block. Set The image processing device according to (16).
(19) The block is an LCU (Largest Coding Unit),
The image processing apparatus according to (11), wherein the offset setting unit sets an offset type of the adaptive offset process according to an integration of areas of subblocks included in the LCU.
(20) The image processing apparatus is
Set the offset type of adaptive offset processing according to the block size or area of the locally decoded image block when encoding the image,
Performing the adaptive offset process for the image with the set offset type,
An image processing method for encoding the image in units having a hierarchical structure using the image on which the adaptive offset processing has been performed.

16 lossless encoding unit, 42 lossless decoding unit, 101 image encoding device, 111 orthogonal transform unit, 112 class classification unit, 113 adaptive offset unit, 131 4 × 4 orthogonal transform unit, 132 8 × 8 orthogonal transform unit, 133 16 × 16 orthogonal transform unit, 134 32 × 32 orthogonal transform unit, 135 cost function calculation unit, 136 TU size determination unit, 141 on / off determination unit, 142 category classification unit, 143 offset processing unit, 201 image decoding device, 211 reverse Orthogonal transform unit, 212 class classification unit, 213 adaptive offset unit, 231 TU size buffer, 232 4 × 4 inverse orthogonal transform unit, 233 8 × 8 inverse orthogonal transform unit, 234 16 × 16 inverse orthogonal transform unit, 235 32 × 32 Inverse orthogonal transform unit, 241 on / off flag Gbuffer, 242 Category classification part, 243 Offset processing part

Claims

A decoding unit that decodes an encoded stream encoded in units having a hierarchical structure to generate an image;
An offset setting unit that sets an offset type of adaptive offset processing according to the size or area of the block of the image generated by the decoding unit;
An image processing apparatus comprising: an adaptive offset processing unit that performs the adaptive offset processing on an image generated by the decoding unit with an offset type set by the offset setting unit.
The offset setting unit sets a band offset for the block when the size or area of the block is large, and sets an edge offset for the block when the size or area of the block is small. Item 8. The image processing apparatus according to Item 1.
The image processing apparatus according to claim 1, wherein the block is a TU (Transform Unit).
A receiving unit for receiving the encoded stream and on / off information indicating on / off of the adaptive offset processing;
The decoding unit decodes the encoded stream received by the receiving unit to generate the image,
When the on / off information received by the receiving unit indicates that the adaptive offset processing is on, the adaptive offset processing unit targets the image generated by the decoding unit with the type of offset set by the offset setting unit. The image processing apparatus according to claim 1, wherein the adaptive offset processing is performed.
The offset setting unit sets a band offset when the size or area of the block is equal to or larger than the first size or the first area, and the block size or area is the first size or the first area. The image processing apparatus according to claim 1, wherein an edge offset is set when the value is smaller.
The offset setting unit sets OFF of the adaptive offset processing when the size or area of the block is equal to or larger than the first size or the second size or the second area larger than the first area. 5. The image processing apparatus according to 5.
The image processing apparatus according to claim 1, wherein the offset setting unit sets an edge offset for the block when NSQT (Non-Square Quadtree Transform) is applied to the block.
When the NSQT (Non-Square Quadtree Transform) is applied to the block, the offset setting unit sets the type of offset of the adaptive offset processing according to the size or area of the short side of the block. Item 8. The image processing apparatus according to Item 1.
The block is an LCU (Largest Coding Unit),
The image processing apparatus according to claim 1, wherein the offset setting unit sets an offset type of the adaptive offset processing according to integration of areas of subblocks included in the LCU.
The image processing device
An image is generated by decoding an encoded stream encoded in units having a hierarchical structure,
Set the offset type of adaptive offset processing according to the size or area of the block of the generated image,
An image processing method for performing the adaptive offset processing on a generated image with a set type of offset.
An offset setting unit that sets the type of offset of adaptive offset processing according to the size or area of the block of the image subjected to local decoding processing when the image is encoded;
An offset type set by the offset setting unit, an adaptive offset processing unit that performs the adaptive offset processing for the image, and
An image processing apparatus comprising: an encoding unit that encodes the image in units having a hierarchical structure using the image on which the adaptive offset processing has been performed by the adaptive offset processing unit.
The offset setting unit sets a band offset for the block when the size or area of the block is large, and sets an edge offset for the block when the size or area of the block is small. Item 12. The image processing apparatus according to Item 11.
The image processing apparatus according to claim 11, wherein the block is a TU (Transform Unit).
A transmission unit that transmits the image encoded by the encoding unit;
The adaptive offset processing unit determines whether to turn on or off the adaptive offset processing. When the adaptive offset processing is on, the adaptive offset processing unit sets the adaptive offset for the image with the type of offset set by the offset setting unit. Process,
The image processing device according to claim 11, wherein the transmission unit transmits on / off information indicating whether the adaptive offset processing is on or off.
The offset setting unit sets a band offset when the size or area of the block is equal to or larger than the first size or the first area, and the block size or area is the first size or the first area. The image processing apparatus according to claim 11, wherein an edge offset is set when the value is smaller.
The offset setting unit sets OFF of the adaptive offset processing when the size or area of the block is equal to or larger than the first size or the second size or the second area larger than the first area. 15. The image processing device according to 15.
The image processing apparatus according to claim 11, wherein the offset setting unit sets an edge offset for the block when NSQT (Non-Square Quadtree Transform) is applied to the block.
When the NSQT (Non-Square Quadtree Transform) is applied to the block, the offset setting unit sets the type of offset of the adaptive offset processing according to the size or area of the short side of the block. Item 12. The image processing apparatus according to Item 11.
The block is an LCU (Largest Coding Unit),
The image processing apparatus according to claim 11, wherein the offset setting unit sets an offset type of the adaptive offset process according to an integration of areas of subblocks included in the LCU.
The image processing device
Set the offset type of adaptive offset processing according to the block size or area of the locally decoded image block when encoding the image,
Performing the adaptive offset process for the image with the set offset type,
An image processing method for encoding the image in units having a hierarchical structure using the image on which the adaptive offset processing has been performed.