JP2016082304A - Image processor, image processing method and image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent degradation of subjective image quality of a decoded image by adaptively reducing the effect of SAO processing.SOLUTION: The image processor includes: an off set correction section that corrects an off-set value to be applied to a reconstruct image based on an image feature amount of each block in the reconstruct image; and an off-set section that offsets a pixel value of the reconstruct image using the off-set value corrected by the off set correction.SELECTED DRAWING: Figure 4

Description

  The present disclosure relates to an image processing apparatus, an image processing method, and an image processing system.

  H. High Efficiency Video Coding (HEVC) by JCTVC (Joint Collaboration Team-Video Coding), a joint standardization organization of ITU-T and ISO / IEC, with the goal of further improving coding efficiency over H.264 / AVC Standardization of an image coding method called “A” has been promoted, and the first edition of the standard specification was published at the beginning of 2013 (for example, see Non-Patent Document 1). Since then, the HEVC specification has been continuously expanded from various viewpoints such as enhancement of encoding tools (see Non-Patent Document 2, for example).

  Sample Adaptive Offset (SAO) is one of the technologies newly introduced in HEVC. The SAO filter is a kind of in-loop filter, and an offset is applied to each pixel value of a decoded image (reconstructed image) in order to mainly remove noise such as ringing noise or mosquito noise. The offset value of SAO is determined for each CTB (Coding Tree Block) from the viewpoint of RD (Rate-Distortion) optimization.

Benjamin Bross, el. Al, “High Efficiency Video Coding (HEVC) text specification draft 10 (for FDIS & Last Call)” (JCTVC-L1003_v34, January 14-23, 2013) David Flynn, el. Al, “High Efficiency Video Coding (HEVC) Range Extensions text specification: Draft 5” (JCTVC-O1005_v2, October 23-11, 2013)

  However, RD optimization does not consider subjective image quality (image quality perceived through the user's vision). This can lead to undesirable results in SAO processing, for example, excessive or unnecessary offsets are applied and the texture is blurred (discolored) on the contrary.

  Therefore, it is beneficial to provide a technique that can adaptively reduce the action of the SAO processing and prevent subjective degradation of the decoded image.

  According to the present disclosure, the offset value to be applied to the reconstructed image is corrected based on the image feature amount for each block of the reconstructed image, and the offset value corrected by the offset correcting unit, An image processing apparatus is provided that includes an offset unit that offsets a pixel value of the reconstructed image.

  According to the present disclosure, the offset value to be applied to the reconstructed image is corrected based on the image feature amount for each block of the reconstructed image, and the reconstructed image is corrected with the corrected offset value. Offsetting the pixel values of the image processing method.

  According to the present disclosure, an offset value to be applied to the reconstructed image is corrected based on an image feature amount for each block of the reconstructed image, and the offset value corrected by the offset correcting unit. The reconstructed image with the encoding unit for encoding the decoding unit, the decoding unit for decoding the corrected offset value encoded by the encoding unit, and the corrected offset value decoded by the decoding unit. And an offset unit that offsets the pixel values of the image processing system.

According to the technology according to the present disclosure, it is possible to adaptively reduce the action of the SAO processing and prevent subjective image quality degradation of the decoded image.
Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

It is the 1st explanatory view for explaining edge offset. It is the 2nd explanatory view for explaining edge offset. It is explanatory drawing for demonstrating band offset. It is a block diagram which shows an example of a schematic structure of an image coding apparatus. It is a block diagram which shows an example of a detailed structure of a SAO filter. It is a graph which shows the 1st example of the relationship between a feature-value and correction offset value. It is a graph which shows the 2nd example of the relationship between a feature-value and a correction offset value. It is a flowchart which shows an example of the schematic flow of the SAO process which concerns on one Embodiment. It is a flowchart which shows an example of the flow of the offset determination process shown in FIG. It is a flowchart which shows an example of the flow which concerns on 1st Example of the offset correction process shown in FIG. It is a flowchart which shows an example of the flow which concerns on 2nd Example of the offset correction process shown in FIG. It is a flowchart which shows the other example of the flow which concerns on 2nd Example of the offset correction process shown in FIG. It is a flowchart which shows an example of the flow which concerns on a 3rd Example of the offset correction process shown in FIG. It is a flowchart which shows the other example of the schematic flow of the SAO process which concerns on one Embodiment. It is a block diagram which shows an example of a structure of the image processing system which concerns on one Embodiment. It is a block diagram which shows an example of the hardware constitutions of an apparatus. It is a block diagram which shows an example of a schematic structure of a mobile telephone. It is a block diagram which shows an example of a schematic structure of a recording / reproducing apparatus. It is a block diagram which shows an example of a schematic structure of an imaging device.

  Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be given in the following order.
1. 1. Basic mechanism of sample adaptive offset (SAO) 2. Configuration example of encoder 3. Detailed configuration example of SAO filter Flow of processing 4-1. Outline of SAO processing 4-2. Offset determination processing 4-3. 4. Offset correction process Image processing system 6. Hardware configuration example Application example 8. Summary

<1. Basic mechanism of sample adaptive offset (SAO)>
Sample adaptive offset (SAO) is one of the newly introduced techniques in HEVC. The SAO filter is a kind of in-loop filter like the deblocking filter. The SAO filter applies an offset to each pixel value of the reconstructed image in order to remove noise such as ringing noise or mosquito noise. In HEVC, two types of edge offset and band offset are defined as SAO filter offset types. The offset type is specified by parameters sao_type_idx_luma and sao_type_idx_chroma encoded in CTU (Coding Tree Unit). For example, if the value of these parameters is 1, the band offset is used, and if it is 2, the edge offset is used in the corresponding CTB (Coding Tree Block).

(1) Edge offset Edge offset processing is offset processing for removing edge-like noise such as ringing noise that may occur in a decoded image due to quantization of orthogonal transform coefficients. In the edge offset process, for each CTU, an edge offset class corresponding to the angle of the edge is specified from the four candidate classes CL0 to CL3 as shown in FIG. 1A. Pixel P _c in the figure is a target pixel, the pixel P _a and the pixel P _b are adjacent pixels adjacent to the pixel of interest. Further, categories corresponding to the pixel value patterns of these three pixels are determined for each pixel of interest from among the four candidate categories CG1 to CG4 as shown in FIG. 1B. The encoder determines an optimum offset value for each of these four candidate categories for each color component of each CTB, and encodes the determined offset value. The identified edge offset class is also encoded.

(2) Band offset The band offset process compensates for the deviation from the original pixel value of the decoded pixel value that appears in a specific band of pixel values. In the band offset process, the pixel value range is divided into, for example, 32 bands, and the offset is applied to pixels having pixel values belonging to the band specified by the band position. In the example of FIG. 2, the pixel value range of 0 to 255 is divided into 32 bands B00 to B31, and an offset is applied to the pixel values belonging to the bands B03, B04, B05, and B06. Is done. The encoder determines an optimum offset value for each band to which the offset is to be applied, and encodes the determined offset value. The band position of the band to which the offset is to be applied is also encoded.

  In such SAO processing, an offset value (and other parameters such as an offset type) is determined for each block (typically CTB) from the viewpoint of RD (Rate-Distortion) optimization. According to RD optimization, the relationship between PSNR (Peak Signal-to-Noise Ratio) representing image quality and code amount can be optimized. However, PSNR does not necessarily match subjective image quality, and subjective image quality is not considered in RD optimization. Therefore, in the existing SAO processing, for example, as a result of applying an excessive or unnecessary offset to the decoded image, the texture may be blurred and the subjective image quality may be impaired.

  In order to eliminate or at least mitigate the drawbacks of the existing mechanisms described above, the technology according to the present disclosure adaptively reduces the effect of SAO processing. Thereby, subjective deterioration of the decoded image is prevented. In an embodiment described later, an offset value determined according to RD optimization is corrected based on the image feature amount for each block, and each pixel value in the block is offset by the corrected offset value. Such an embodiment will be described more specifically in the following sections.

<2. Example of encoder configuration>
FIG. 3 is a block diagram illustrating an example of a schematic configuration of the image encoding device 10. Referring to FIG. 3, the image encoding device 10 includes a rearrangement buffer 11, a subtraction unit 13, an orthogonal transformation unit 14, a quantization unit 15, a lossless encoding unit 16, an accumulation buffer 17, a rate control unit 18, and an inverse quantization. Unit 21, inverse orthogonal transform unit 22, addition unit 23, deblock filter 24, SAO filter 25, frame memory 26, switch 27, mode setting unit 28, intra prediction unit 30, and inter prediction unit 35.

  The rearrangement buffer 11 rearranges the images included in the series of image data. The rearrangement buffer 11 rearranges the images according to the GOP (Group of Pictures) structure related to the encoding process, and then subtracts the rearranged image data, the SAO filter 25, the intra prediction unit 30, and the inter prediction. To the unit 35.

  The subtraction unit 13 calculates prediction error data that is a difference between the image data input from the rearrangement buffer 11 and the prediction image data, and outputs the calculated prediction error data to the orthogonal transformation unit 14.

  The orthogonal transform unit 14 performs an orthogonal transform process on each of one or more transform blocks (TBs) set in each CTB. The orthogonal transform here may be, for example, discrete cosine transform (DCT) or Karoonen-Loeve transform. More specifically, the orthogonal transform unit 14 transforms the prediction error data input from the subtraction unit 13 from a spatial domain image signal to frequency domain transform coefficient data for each TB. Then, the orthogonal transform unit 14 outputs transform coefficient data to the quantization unit 15. Note that not all TBs have conversion coefficient data. A TB in which a prediction error is not encoded (for example, an image is reconstructed only by motion compensation) does not have transform coefficient data, and a coded block flag (CBF) indicating zero is provided for such TB. Can be encoded.

  The quantization unit 15 is supplied with transform coefficient data input from the orthogonal transform unit 14 and a rate control signal from a rate control unit 18 described later. The quantization unit 15 quantizes the transform coefficient data in a quantization step determined according to the rate control signal. Further, the quantization unit 15 generates a quantization parameter indicating the quantization step used for each TB. The quantization unit 15 outputs the quantized transform coefficient data (hereinafter referred to as quantized data) to the lossless encoding unit 16 and the inverse quantization unit 21. Further, the quantization unit 15 outputs the generated quantization parameter to the lossless encoding unit 16.

  The lossless encoding unit 16 generates an encoded stream by encoding the quantized data input from the quantizing unit 15 for each of one or more encoded blocks (CB). The lossless encoding unit 16 encodes various parameters referred to by the decoder, and inserts the encoded parameters into the header area of the encoded stream. Parameters encoded by the lossless encoding unit 16 are block information indicating how CTB, CB, TB, and PB should be set in an image, a quantization parameter, an encoded block flag, and SAO-related parameters described later. , Information regarding intra prediction and information regarding inter prediction may be included. Then, the lossless encoding unit 16 outputs the generated encoded stream to the accumulation buffer 17.

  The accumulation buffer 17 temporarily accumulates the encoded stream input from the lossless encoding unit 16 using a storage medium such as a semiconductor memory. Then, the accumulation buffer 17 outputs the accumulated encoded stream to a transmission unit (not shown) (for example, a communication interface or a connection interface with a peripheral device) at a rate corresponding to the bandwidth of the transmission path.

  The rate control unit 18 monitors the free capacity of the accumulation buffer 17. Then, the rate control unit 18 generates a rate control signal according to the free capacity of the accumulation buffer 17 and outputs the generated rate control signal to the quantization unit 15. For example, the rate control unit 18 generates a rate control signal for reducing the bit rate of the quantized data when the free capacity of the storage buffer 17 is small. For example, when the free capacity of the accumulation buffer 17 is sufficiently large, the rate control unit 18 generates a rate control signal for increasing the bit rate of the quantized data.

  The inverse quantization unit 21, the inverse orthogonal transform unit 22, and the addition unit 23 constitute a local decoder. The inverse quantization unit 21 inversely quantizes the quantized data in the same quantization step as that used by the quantization unit 15 to restore transform coefficient data. Then, the inverse quantization unit 21 outputs the restored transform coefficient data to the inverse orthogonal transform unit 22.

  The inverse orthogonal transform unit 22 restores the prediction error data by executing an inverse orthogonal transform process on the transform coefficient data input from the inverse quantization unit 21. Similar to the orthogonal transform, the inverse orthogonal transform is performed for each TB. Then, the inverse orthogonal transform unit 22 outputs the restored prediction error data to the addition unit 23.

  The adder 23 adds the decoded prediction error data input from the inverse orthogonal transform unit 22 and the predicted image data input from the intra prediction unit 30 or the inter prediction unit 35, thereby obtaining decoded image data (reconstruction). Image). Then, the addition unit 23 outputs the generated decoded image data to the deblock filter 24 and the frame memory 26.

  Each of the deblock filter 24 and the SAO filter 25 is an in-loop filter for the purpose of improving the image quality of the reconstructed image. The deblocking filter 24 removes block distortion by filtering the decoded image data input from the adding unit 23, and outputs the decoded image data after filtering to the SAO filter 25.

  The SAO filter 25 removes noise by applying edge offset processing or band offset processing to the decoded image data input from the deblocking filter 24, and outputs the processed decoded image data to the frame memory 26. More specifically, the SAO filter 25 corrects the offset value determined according to RD optimization in each block in the image based on the image feature amount for each block, and the pixel value in each block with the corrected offset value. Offset each. A more detailed configuration of the SAO filter 25 will be further described later.

  The frame memory 26 stores the decoded image data before filtering input from the adder 23 and the decoded image data after application of the in-loop filter input from the SAO filter 25 using a storage medium.

  The switch 27 reads decoded image data before filtering used for intra prediction from the frame memory 26 and supplies the read decoded image data to the intra prediction unit 30 as reference image data. Further, the switch 27 reads out the decoded image data after filtering used for inter prediction from the frame memory 26 and supplies the read out decoded image data to the inter prediction unit 35 as reference image data.

  The mode setting unit 28 determines an optimal block division and prediction mode for each CTB based on a comparison of costs input from the intra prediction unit 30 and the inter prediction unit 35. For the block for which the intra prediction mode is selected, the mode setting unit 28 outputs the prediction image data generated by the intra prediction unit 30 to the subtraction unit 13 and outputs information related to intra prediction to the lossless encoding unit 16. Further, the mode setting unit 28 outputs the prediction image data generated by the inter prediction unit 35 to the subtraction unit 13 and outputs information related to the inter prediction to the lossless encoding unit 16 for the block for which the inter prediction mode is selected. To do.

  The intra prediction unit 30 executes an intra prediction process for each of one or more prediction blocks (PB: Prediction Blocks) set in each CTB based on the original image data and the decoded image data. For example, the intra prediction unit 30 evaluates the prediction result of each candidate mode in the prediction mode set using a predetermined cost function. Next, the intra prediction unit 30 selects the prediction mode with the lowest cost, that is, the prediction mode with the highest compression rate, as the optimum mode. The intra prediction unit 30 generates predicted image data according to the optimal mode. Then, the intra prediction unit 30 outputs information related to intra prediction representing the optimal mode, cost, and predicted image data to the mode setting unit 28.

  The inter prediction unit 35 performs an inter prediction process for each of one or more PBs based on the original image data and the decoded image data. For example, the inter prediction unit 35 evaluates the prediction result of each candidate mode in the prediction mode set using a predetermined cost function. Next, the inter prediction unit 35 selects the prediction mode with the lowest cost, that is, the prediction mode with the highest compression rate, as the optimum mode. Further, the inter prediction unit 35 generates predicted image data according to the optimum mode. Then, the inter prediction unit 35 outputs information related to inter prediction representing the optimal mode, cost, and predicted image data to the mode setting unit 28. The information regarding the inter prediction here may include motion vector information. The motion vector information represents the motion vector to be applied to each PB with integer sample accuracy or fractional sample accuracy. When the motion vector is expressed with decimal sample accuracy, the reference pixel value of the decimal sample position interpolated using the interpolation filter is used for motion compensation.

<3. Detailed configuration example of SAO filter>
FIG. 4 is a block diagram showing an example of a detailed configuration of the SAO filter 25 shown in FIG. Referring to FIG. 4, the SAO filter 25 includes an offset determination unit 41, a feature amount acquisition unit 43, an offset correction unit 45, and an offset unit 47.

(1) Offset determination unit The offset determination unit 41 optimizes an offset value to be applied to a reconstructed image by rate distortion (RD) optimization, similar to the existing SAO processing described with reference to FIGS. 1A, 1B, and 2. Based on the above, the determination is made for each block. The block here can typically correspond to a CTB (Coding Tree Block). More specifically, the offset determination unit 41 sets an appropriate offset value to be applied to the _{reconstructed} image I _rec input from the _deblocking filter 24 for each offset type that may include an edge offset and a band offset. judge. In the case of edge offset, the set of offset values includes an offset value corresponding to each category and can be determined for each candidate class. Further, the offset determination unit 41 calculates a cost for each offset type (and candidate class) from the error between the original image I _org and the processed reconstructed image and the expected code amount. Then, the offset determination unit 41 selects the offset type indicating the lowest calculated cost and the class in the case of the edge offset. The set of offset values corresponding to the selection result here is an offset value to be applied to the reconstructed image (but before correction). When the edge determination unit 41 selects an edge offset as the offset type, the offset value V corresponding to each of the four candidate categories CG1 to CG4 as shown in FIG. 1B and related parameters (offset type, edge) Offset class) is output to the offset correction unit 45. On the other hand, when the band offset is selected as the offset type, the offset determination unit 41 sends the offset value V corresponding to each target band and related parameters (offset type, band position, etc.) to the offset correction unit 45. Output. Note that the offset determination unit 41 may select that neither edge offset processing nor band offset processing is performed on a certain CTB. In this case, the offset determination unit 41 outputs an offset type indicating a value (for example, zero) meaning no offset to the offset correction unit 45.

(2) Feature Quantity Acquisition Unit The feature quantity acquisition unit 43 acquires the image feature quantity referred to when correcting the offset value determined by the offset determination unit 41 for each block of the reconstructed image. Here, although not limited, three examples relating to the image feature amount will be described. In the first embodiment, the image feature amount acquired by the feature amount acquisition unit 43 includes an index related to the flatness of each block. For example, the flatness of each block can be expressed by one or more of pixel value variance, standard deviation, and first-order differential summation. The feature amount acquisition unit 43 calculates, for example, the variance (mean square error) VAR _i of the pixel value for each block of the _{reconstructed} image I _rec input from the deblock filter 24 as in the following equation (1). May be. In Expression (1), N represents the number of pixels belonging to the block B _i , P _k represents the pixel value of the k th pixel belonging to the block B _i , and P _{AVE — i} represents the average of the pixel values of the pixel group belonging to the block B _i. .

The feature quantity acquisition unit 43 may calculate a standard deviation equal to the positive square root of the variance VAR _i . Further, the feature amount acquisition unit 43 may calculate the primary differential sum DESUM _i as in the following equation (2). In Expression (2), P _{x, y} represents the pixel value at the pixel position (x, y) belonging to the block B _i .

  These variances, standard deviations, and first-order differential sums indicate zero when a block is completely flat (that is, the pixel values of the pixel groups belonging to the block are all equal). A large value of these indices suggests that the block image contains a lot of high frequency components. Some portion of such high frequency components may be due to noise such as ringing noise or mosquito noise that should be removed by offset. On the other hand, the smaller values of these indices suggest that there are fewer high frequency components in the block image and therefore less noise to remove. For such blocks, the effect of subjective image quality degradation due to image blurring is likely to exceed the noise removal gain of SAO processing, so the intensity of SAO processing is adaptively reduced. Is beneficial.

In the second embodiment, the image feature amount acquired by the feature amount acquisition unit 43 includes an index related to the loss of the high-frequency component of each block. For example, when a coding block flag (CBF) indicates that a certain TB does not include a transform coefficient, the TB is a quantization error (an error caused by truncation of a frequency component when quantizing the transform coefficient). Is not included. This means that there is no loss of high frequency components and there is no or small noise contained in TB. TB (transform block) is a sub-block of CTB. Therefore, for example, the feature amount acquisition unit 43 counts the CBF values of the sub-blocks belonging to the block as an index related to the loss of high-frequency components for each block of the _{reconstructed} image I _rec input from the deblock filter 24. By doing so, the proportion of sub-blocks whose transform coefficients are zero (hereinafter referred to as zero coefficient ratio) may be calculated. For blocks with a higher zero coefficient ratio, the effect of deterioration in subjective image quality due to image blurring is more likely than the gain of noise removal by SAO processing, so the strength of SAO processing is adapted. It is beneficial to reduce it.

  Alternatively or additionally, the feature amount acquisition unit 43 acquires a quantization parameter (for example, an average value in the block) used for each block or sub-block as an index related to the loss of high-frequency components. May be. A large quantization parameter value means a large loss of high-frequency components due to quantization, and conversely a small quantization parameter value means a small loss of high-frequency components due to quantization. To do. Therefore, a block having a smaller quantization parameter value is more likely to be affected by subjective image quality degradation due to image blurring than the noise removal gain by SAO processing. It is beneficial to adaptively reduce the intensity of the. Further, the degree of reduction in the strength of the SAO processing based on the above-described zero coefficient rate (or based on the flatness of the block in the first embodiment) may be weakened as the value of the quantization parameter increases.

  Alternatively or additionally, the feature quantity acquisition unit 43 may acquire the proportion of sub-blocks whose motion compensation sample accuracy is a specific accuracy as an index related to the loss of high-frequency components. For example, when the sample accuracy of motion compensation is a decimal accuracy (such as 1/2 accuracy or 1/4 accuracy) in a certain PB (prediction block), an interpolation filter can be used in that PB. PB is a sub-block of CTB. An interpolation filter having a long tap length (the maximum tap length of the interpolation filter in HEVC is 8 taps) acts as a substantial low-pass filter and has an effect of smoothing the image. Therefore, when the proportion of sub-blocks whose motion compensation sample precision is decimal precision (hereinafter referred to as the decimal precision sub-block ratio) is large, there is a high possibility that the image will be excessively blurred by applying an offset. . On the other hand, when the sample accuracy of motion compensation is only integer accuracy in a certain PB, the interpolation filter is not used in that PB. Therefore, in this case, even if the offset is further applied, the possibility that the image is excessively blurred is low. The sample accuracy of motion compensation can be ascertained from motion vector information.

In the third embodiment, the image feature amount acquired by the feature amount acquisition unit 43 includes a ratio of sub-blocks in which the motion vector in each block is zero (hereinafter referred to as zero vector rate). For example, when the subject is stationary within the angle of view, the motion vector can be zero in the PB that reflects the subject. In this case, when the SAO process is executed, the image may become blurred as a result of cumulatively adding the offset to the predicted pixel value of the inter prediction. Therefore, the feature amount acquisition unit 43 may calculate the zero vector rate for each block of the _{reconstructed} image I _rec input from the deblock filter 24 using, for example, motion vector information. For blocks with a higher zero vector rate, the effect of deterioration in subjective image quality due to image blurring is more likely than the gain of noise removal by SAO processing, so the strength of SAO processing is adapted. It is beneficial to reduce it. For sub-blocks with a motion vector of zero, the application of offset may be skipped to avoid cumulative addition of offset.

  The feature amount acquisition unit 43 outputs the image feature amount R for each block acquired according to any of the above-described methods or other methods to the offset correction unit 45.

(3) Offset Correction Unit The offset correction unit 45 corrects the offset value to be applied to the reconstructed image determined by the offset determination unit 41 based on the image feature amount for each block input from the feature amount acquisition unit 43. To do. For example, the offset correction unit 45 may correct the offset value so that a smaller offset value is applied to a flatter block based on the above-described index related to flatness. Further, the offset correction unit 45 may correct the offset value so that a smaller offset value is applied to a block with a smaller loss of the high frequency component based on the above-described index related to the loss of the high frequency component. Blocks with lower loss of high frequency components may include, for example, blocks with higher zero coefficient rates, lower quantization parameters, or higher decimal precision sub-block rates. Further, the offset correction unit 45 may correct the offset value so that a smaller offset value is applied to a block having a higher zero vector rate.

  The processing executed by the offset correction unit 45 can be conceptually expressed in the form of the following function.

Here, V and V _mod represent offset values before and after correction, respectively, and R _i represents an i-th feature amount (i is an integer of 1 or more). The function F may be in a linear form such as the following equation (4). The coefficient α is typically a correction coefficient not less than zero and not more than 1 depending on the feature amounts R ₁ , R ₂ ,. The operation Clip (X) is an operation for limiting the output value of the function to a predetermined offset range. For example, when X exceeds the upper limit _Xmax , Clip (X) = _Xmax .

FIG. 5A is a graph illustrating a first example of the relationship between the feature amount R and the corrected offset value V _mod . The horizontal axis of the graph represents the feature amount R, and the vertical axis represents the corrected offset value V _mod . According to the illustrated graph G11, the corrected offset value V _mod draws a straight line that monotonously increases according to the feature amount R. The maximum value of the correction offset value V _mod is equal to the offset value V before correction. Such a graph G11 is suitable for a scenario in which the effect of the SAO processing is further reduced when the image feature value is smaller. For example, the case where the image feature amount R corresponds to the above-described variance, standard deviation, first-order differential sum, or quantization parameter applies to the scenario of the graph G11. On the other hand, according to the graph G12, the corrected offset value V _mod draws a straight line that decreases monotonously according to the feature amount R. Such a graph G12 is suitable for a scenario in which the action of the SAO processing is more strongly reduced when the image feature value is larger. For example, the case where the image feature amount R corresponds to the above-described zero coefficient rate, decimal precision sub-block rate, or zero vector rate applies to the scenario of the graph G12.

FIG. 5B is a graph for explaining a second example of the relationship between the feature amount R and the corrected offset value V _mod . According to the illustrated graph G2, the corrected offset value V _mod draws a polygonal line that increases stepwise according to the feature amount R. For example, when the feature amount R belongs to the _first sub-range [0, r ₁ ], the correction coefficient α = 1/4 is multiplied by the offset value V. When the feature amount R belongs to the second subrange [r ₁ , r ₂ ], the correction coefficient α = 1/3 is multiplied by the offset value V. When the feature amount R belongs to the third subrange [r ₂ , r ₃ ], the correction coefficient α = ½ is multiplied by the offset value V. When the feature amount R belongs to the fourth subrange [r ₃ , r ₄ ], the corrected offset value V _mod is equal to the offset value V before correction. Such correction factors may be predetermined, for example, through pre-tuning and stored in memory in association with the corresponding feature sub-range. Then, the offset correction unit 45 can acquire a correction coefficient corresponding to the feature value from the memory when correcting the offset. The lossless encoding unit 16 may additionally encode a parameter that defines such a correction coefficient.

The graphs shown in FIGS. 5A and 5B are merely examples for explanation. The offset correction unit 45 may correct the offset value so that the correction offset value V _mod draws a straight line having various inclinations and intercepts, or other trajectories such as various curves or polygonal lines. In one variation, the offset correction unit 45 can adjust the correction coefficient α such that the correction value of the offset value is weakened as the quantization parameter value increases. The correction coefficient α _adj in the following equation (5) represents the adjusted correction coefficient, and is equal to the product of the correction coefficient α and the adjustment factor w within a range not exceeding 1.0.

  The adjustment factor w is, for example, 1.0 when the quantization parameter belongs to the subrange [0, 20], 1.2 when the quantization parameter belongs to the subrange [20, 30], and the quantization parameter is the subrange. If it belongs to [30, 40], it can be defined in advance in association with the sub-range of the quantization parameter, such as equal to 1.6. In this way, by setting a larger correction coefficient for correcting the offset value as the quantization parameter value is larger, the strength of correction of the offset value can be reduced. As a result, while reducing the effect of the SAO processing, it is possible to incorporate a mechanism that maintains the effect of removing noise caused by high-frequency component loss in the case where the quantization error is large.

  As described above, the edge offset processing is processing for removing edge noise. Therefore, reducing the edge offset leads to preventing the edge from being excessively blurred. On the other hand, the band offset process is a process for compensating for a deviation of pixel values appearing in a specific band. Therefore, reducing the band offset does not necessarily prevent the image from becoming blurred. Therefore, the offset correction unit 45 corrects the offset value for the block whose offset type is determined as the edge offset by the offset determination unit 41, but does not need to correct the offset value for the block whose offset type is the band offset. Good. Accordingly, it is possible to effectively prevent image blurring due to edge offset without reducing the effect of band offset.

The offset correction unit 45 outputs the corrected offset value V _mod to the offset unit 47 when the offset value is corrected according to any of the methods described above or other methods. When the offset value is not corrected, the original offset value V is output to the offset unit 47. Further, the offset correction unit 45 outputs SAO related parameters corresponding to the values output to the offset unit 47 to the lossless encoding unit 16 for encoding. The SAO-related parameters here may be parameters that specify, for example, an offset type, a set of offset values, and an edge offset class or a band position as necessary.

(4) Offset unit When the offset value is corrected by the offset correction unit 45, the offset unit 47 offsets the pixel value of the reconstructed image with the corrected offset value V _mod . The offset unit 47 can offset the pixel value with the offset value V determined by the offset determination unit 41 when the offset value is not corrected. For the block whose offset type is edge offset, the offset unit 47 determines the category of the pixel of interest from among the four candidate categories CG1 to CG4, and pays attention to the offset value corresponding to the determined category (corrected). Add to the pixel value of the pixel. For the block whose offset type is band offset, the offset unit 47 identifies the band to which the pixel value of the pixel of interest belongs from among the 32 bands, and focuses on the (corrected) offset value corresponding to the identified band. Add to the pixel value of the pixel. The offset unit 47 may skip the offset of the pixel value when the offset type indicates no offset, for example.

  When a block belongs to a B picture or a B slice, bi-prediction can be performed in the block during inter prediction. Bi-prediction includes calculation of a weighted average of a plurality of reference pixel values for deriving a predicted pixel value, which acts as a substantial low-pass filter and has an effect of smoothing an image. Therefore, the offset unit 47 may avoid that the image is excessively blurred by skipping the offset of the pixel value of the block belonging to the B picture or the B slice. In addition, the offset unit 47 is a pixel of a block or sub-block whose motion vector is zero in order to avoid blurring the image through cumulative addition of offset when the subject is stationary within the angle of view. The value offset may be skipped.

<4. Process flow>
[4-1. Outline of SAO processing]
FIG. 6 is a flowchart illustrating an example of a schematic flow of SAO processing executed by the SAO filter 25 according to an embodiment. The process shown in FIG. 6 is repeated for each block in the image.

  Referring to FIG. 6, first, the offset determination unit 41 determines an offset type (SAO type) and an offset value to be applied to the block of interest and related parameters based on RD optimization (step S10). Subsequent processing branches depending on the determined offset type (step S20).

  When the determined offset type is an edge offset, the feature amount acquisition unit 43 acquires the image feature amount of the block of interest referred to when correcting the offset value (step S30). Next, the offset correction unit 45 corrects the offset value input from the offset determination unit 41 based on the image feature amount of the target block input from the feature amount acquisition unit 43 (step S40). Next, the offset unit 47 offsets each pixel value of the pixel group belonging to the block of interest with the offset value corrected by the offset correction unit 45 (step S50).

  When the determined offset type is a band offset, the offset unit 47 offsets each pixel value of the pixel group belonging to the block of interest with an uncorrected offset value (that is, determined by the offset determination unit 41) (Step S1). S55).

  Next, the lossless encoding unit 16 encodes SAO related parameters including a parameter for specifying the offset value corrected by the offset correcting unit 45 (step S60). If the SAO-related parameter of the target block is the same as the SAO-related parameter of the adjacent block that has been encoded, a copy flag that enables the copy mode can be encoded instead of the SAO-related parameter. However, since the copy mode is not directly related to the technology according to the present disclosure, the description thereof is omitted here.

  Next, when an unprocessed block remains, the process returns to step S10, and the above-described process is repeated with the next block as a target block (step S70). If no unprocessed block remains, the SAO process shown in FIG. 6 ends.

[4-2. Offset judgment processing]
FIG. 7 is a flowchart showing an example of the flow of the offset determination process corresponding to step S10 in FIG. The process shown in FIG. 7 is repeated for each color component of the block of interest (step S11).

  First, the offset determination unit 41 determines the optimum offset value of each of the four candidate categories for each class of edge offset (step S12). Next, the offset determination unit 41 calculates a cost for RD optimization for each class (step S13).

  Moreover, the offset determination part 41 determines the optimal offset value for every band of a band offset (step S14). Next, the offset determination part 41 calculates the cost for RD optimization about each band position (step S15).

  Next, the offset determination unit 41 compares the calculated costs with each other, so that the optimum offset type (SAO type) for the block of interest, the corresponding set of offset values, and related parameters (such as edge offset class or band position) ) Is determined (step S16). Note that some parameters such as the edge offset class may be determined in common for the two color difference components.

[4-3. Offset correction process]
(1) First Embodiment FIG. 8 is a flowchart showing an example of a flow according to the first embodiment of offset correction processing corresponding to steps S30 and S40 of FIG.

  First, the feature amount acquisition unit 43 calculates an index related to the flatness of the block of interest using the reconstructed image input from the deblocking filter 24 (step S31). The index related to flatness here may be, for example, the variance of pixel values, the standard deviation, or the first derivative sum, or some combination thereof.

  Next, the offset correction unit 45 acquires a correction coefficient corresponding to the index value related to the flatness of the block of interest calculated by the feature amount acquisition unit 43 (step S41). For example, the offset correction unit 45 may calculate the correction coefficient using the calculated index, or may acquire the correction coefficient associated with the index value from the memory.

  Next, the offset correction unit 45 corrects the offset value to be applied to the block of interest using the acquired correction coefficient, for example, according to the method described with reference to FIG. 5A or 5B (step S47).

(2) Second Example FIG. 9A is a flowchart illustrating an example of a flow according to a second example of offset correction processing corresponding to steps S30 and S40 of FIG.

  First, the feature quantity acquisition unit 43 acquires an index related to the loss of the high-frequency component of the block of interest by referring to an encoding parameter such as an encoding block flag, a quantization parameter, or motion vector information (step S32). ). The index related to the loss of the high-frequency component here is, for example, a fraction of the sub-block where the transform coefficient is zero (zero coefficient ratio), a quantization parameter (average value in the block), or a motion compensation sample accuracy is a decimal It may be the ratio of sub-blocks that is precision (decimal precision sub-block ratio), or some combination thereof.

  Next, the offset correcting unit 45 acquires a correction coefficient corresponding to the index value related to the loss of the high frequency component acquired by the feature amount acquiring unit 43 (step S42). For example, the offset correction unit 45 may calculate a correction coefficient from the degree of high-frequency component loss, or may acquire a correction coefficient associated with the index value from the memory.

  Next, the offset correction unit 45 corrects the offset value to be applied to the block of interest using the acquired correction coefficient, for example, according to the method described with reference to FIG. 5A or 5B (step S47).

  FIG. 9B is a flowchart showing another example of the flow according to the second embodiment of the offset correction process corresponding to steps S30 and S40 of FIG.

  First, the feature amount acquisition unit 43 acquires a zero coefficient rate and a quantization parameter based on the summation of the encoded block flags as an index related to the loss of the high frequency component of the block of interest (step S32a).

  Next, the offset correction unit 45 acquires a correction coefficient corresponding to the value of the zero coefficient rate acquired by the feature amount acquisition unit 43 (step S42a). Here, for example, the correction coefficient acquired by the offset correction unit 45 shows a smaller value as the zero coefficient ratio is larger.

  Next, the offset correction unit 45 adjusts the acquired correction coefficient using the quantization parameter (step S43). As a result, the larger the quantization parameter value, the more the correction coefficient value can be increased in a range not exceeding 1 (ie, the correction strength can be reduced).

  Next, the offset correction unit 45 corrects the offset value to be applied to the block of interest using the correction coefficient adjusted using the quantization parameter (step S47a).

  As shown in FIG. 9B, instead of adjusting the correction coefficient before the correction of the offset value, the primary correction of the offset value based on the zero coefficient rate and the secondary correction of the offset value based on the quantization parameter are performed. It may be done. Further, instead of the zero coefficient rate, another index such as a decimal precision sub-block rate may be used.

(3) Third Example FIG. 10 is a flowchart showing an example of a flow according to a third example of offset correction processing corresponding to steps S30 and S40 of FIG.

  First, the feature quantity acquisition unit 43 calculates the ratio (zero vector rate) of sub-blocks in which the motion vector of the block of interest is zero by referring to, for example, motion vector information (step S33).

  Next, the offset correction unit 45 acquires a correction coefficient corresponding to the value of the zero vector rate calculated by the feature amount acquisition unit 43 (step S44). For example, the offset correction unit 45 may calculate the correction coefficient using the zero vector rate, or may acquire the correction coefficient associated with the value of the zero vector rate from the memory.

  Next, the offset correction unit 45 corrects the offset value to be applied to the block of interest using the acquired correction coefficient, for example, according to the method described with reference to FIG. 5A or 5B (step S47).

(4) Modification FIG. 11 is a flowchart showing another example of the schematic flow of the SAO process executed by the SAO filter 25 according to an embodiment. The process shown in FIG. 11 is repeated for each block in the image.

  Referring to FIG. 11, first, the offset unit 47 determines whether the block of interest belongs to a B picture or a B slice (step S5). Here, when it is determined that the target block belongs to the B picture or the B slice, the processes of steps S10 to S60 are skipped. If it is determined that the target block does not belong to the B picture or B slice, the process proceeds to step S10.

  Since the processing in steps S10 to S60 is the same as that described with reference to FIG. 6, the description thereof is omitted here. If an unprocessed block remains in step S70, the process returns to step S5, and the above-described process is repeated with the next block as the target block. If no unprocessed block remains, the SAO process shown in FIG. 11 ends.

<5. Image processing system>
In one embodiment, an image processing system including the above-described image encoding device 10 and an image decoding device that decodes an image encoded by the image encoding device 10 may be provided. FIG. 12 is a block diagram showing an example of the configuration of such an image processing system 1. Referring to FIG. 12, the image processing system 1 includes an image encoding device 10 and an image decoding device 60. The image encoding device 10 corrects an offset value to be applied to a reconstructed image reconstructed from an encoded stream based on an image feature amount for each block of the image, and specifies an encoding parameter for specifying the corrected offset value. Insert into encoded stream. The image decoding device 60 receives the encoded stream and decodes the corrected offset value encoded by the image encoding device 10. Then, the image decoding device 60 offsets the pixel value of the reconstructed image with the decoded corrected offset value.

  More specifically, the image decoding device 60 includes an accumulation buffer 61, a lossless decoding unit 62, an inverse quantization unit 63, an inverse orthogonal transform unit 64, an addition unit 65, a deblock filter 66, an SAO filter 67, and a rearrangement buffer 68. , A D / A (Digital to Analogue) conversion unit 69, a frame memory 70, selectors 71a and 71b, an intra prediction unit 80, and an inter prediction unit 85.

  The accumulation buffer 61 temporarily accumulates the encoded stream using a storage medium.

  The lossless decoding unit 62 decodes the quantized data from the encoded stream input from the accumulation buffer 61 according to the encoding method used at the time of encoding. In addition, the lossless decoding unit 62 decodes information inserted in the header area of the encoded stream. The information decoded by the lossless decoding unit 62 may include, for example, SAO related parameters, information related to intra prediction, and information related to inter prediction. The lossless decoding unit 62 outputs the quantized data to the inverse quantization unit 63. Further, the lossless decoding unit 62 outputs SAO related parameters to the SAO filter 67. Further, the lossless decoding unit 62 outputs information related to intra prediction to the intra prediction unit 80. In addition, the lossless decoding unit 62 outputs information on inter prediction to the inter prediction unit 85.

  The inverse quantization unit 63 inversely quantizes the quantized data input from the lossless decoding unit 62 in the same quantization step as that used for encoding, and restores transform coefficient data. The inverse quantization unit 63 outputs the restored transform coefficient data to the inverse orthogonal transform unit 64.

  The inverse orthogonal transform unit 64 generates prediction error data by performing inverse orthogonal transform on the transform coefficient data input from the inverse quantization unit 63 according to the orthogonal transform method used at the time of encoding. The inverse orthogonal transform unit 64 outputs the generated prediction error data to the addition unit 65.

  The adding unit 65 generates decoded image data by adding the prediction error data input from the inverse orthogonal transform unit 64 and the predicted image data input from the selector 71b. Then, the adding unit 65 outputs the generated decoded image data to the deblock filter 66 and the frame memory 70.

  The deblocking filter 66 removes block distortion by filtering the decoded image data input from the adding unit 65, and outputs the decoded image data after filtering to the SAO filter 67.

  The SAO filter 67 removes noise by applying edge offset processing or band offset processing to the decoded image data input from the deblocking filter 66, and the decoded image data after processing is sent to the rearrangement buffer 68 and the frame memory 70. Output. More specifically, the SAO filter 67 typically sets an offset type, a set of offset values, and an edge offset class for each block that can correspond to a CTB from SAO-related parameters decoded by the lossless decoding unit 62. Or identify other parameters such as band position. The offset value identified here may indicate a value corrected by the image encoding device 10 based on the image feature amount for each block. Then, the SAO filter 67 offsets each pixel value in each block with the decoded offset value. The SAO filter 67 may skip the offset for the block when the offset type indicates no offset, when the block belongs to a B picture or B slice, or when the motion vector is zero.

  The rearrangement buffer 68 rearranges the images input from the SAO filter 67 to generate a series of time-series image data. Then, the rearrangement buffer 68 outputs the generated image data to the D / A conversion unit 69.

  The D / A converter 69 converts the digital image data input from the rearrangement buffer 68 into an analog image signal. The D / A conversion unit 69 displays the decoded video by outputting an analog image signal to a display (not shown) connected to the image decoding device 60, for example.

  The frame memory 70 stores the decoded image data before filtering input from the adding unit 65 and the decoded image data after filtering input from the SAO filter 67 using a storage medium.

  The selector 71a switches the output destination of the image data from the frame memory 70 between the intra prediction unit 80 and the inter prediction unit 85 for each block in the image according to the mode information acquired by the lossless decoding unit 62. . For example, when the intra prediction mode is designated, the selector 71a outputs the decoded image data before filtering supplied from the frame memory 70 to the intra prediction unit 80 as reference image data. Also, when the inter prediction mode is designated, the selector 71a outputs the decoded image data after filtering to the inter prediction unit 85 as reference image data.

  The selector 71 b switches the output source of the predicted image data to be supplied to the adding unit 65 between the intra prediction unit 80 and the inter prediction unit 85 according to the mode information acquired by the lossless decoding unit 62. For example, the selector 71b supplies the prediction image data output from the intra prediction unit 80 to the addition unit 65 when the intra prediction mode is designated. Further, when the inter prediction mode is designated, the selector 71b supplies the predicted image data output from the inter prediction unit 85 to the addition unit 65.

  The intra prediction unit 80 performs intra prediction processing based on the information related to intra prediction input from the lossless decoding unit 62 and the reference image data from the frame memory 70, and generates predicted image data. Then, the intra prediction unit 80 outputs the generated predicted image data to the selector 71b.

  The inter prediction unit 85 performs inter prediction processing based on the information related to inter prediction input from the lossless decoding unit 62 and the reference image data from the frame memory 70, and generates predicted image data. Then, the inter prediction unit 85 outputs the generated predicted image data to the selector 71b.

<6. Hardware configuration example>
The above-described embodiments may be realized using any of software, hardware, and a combination of software and hardware. When the image encoding device 10 or the image decoding device 60 uses software, a program constituting the software is stored in advance in, for example, a storage medium (non-transitory media) provided inside or outside the device. Stored. Each program is read into a RAM (Random Access Memory) at the time of execution and executed by a processor such as a CPU (Central Processing Unit).

  FIG. 13 is a block diagram illustrating an example of a hardware configuration of an apparatus to which the above-described embodiment can be applied. Referring to FIG. 13, the image processing apparatus 800 includes a system bus 810, an image processing chip 820, and an off-chip memory 890. The image processing chip 820 includes n (n is 1 or more) processing circuits 830-1, 830-2,..., 830-n, a reference buffer 840, a system bus interface 850, and a local bus interface 860.

  The system bus 810 provides a communication path between the image processing chip 820 and an external module (for example, a central control function, an application function, a communication interface, or a user interface). The processing circuits 830-1, 830-2,..., 830-n are connected to the system bus 810 via the system bus interface 850 and to the off-chip memory 890 via the local bus interface 860. The processing circuits 830-1, 830-2,..., 830-n can also access a reference buffer 840 that can correspond to an on-chip memory (eg, SRAM). The off-chip memory 890 may be a frame memory that stores image data processed by the image processing chip 820, for example.

  As an example, the processing circuit 830-1 may correspond to the above-described SAO filter 25 or SAO filter 67, and the processing circuit 830-2 may correspond to the lossless encoding unit 16 or the lossless decoding unit 62. Note that these processing circuits may be formed not on the same image processing chip 820 but on separate chips. When the offset value is corrected in the encoder-side SAO filter according to the above-described method, the syntax of the SAO-related parameters is not affected. Thus, there is no need to alter the decoder logic depending on whether the offset value is modified from a value based on RD optimization.

<7. Application example>
In the above-described embodiment, a transmission device that transmits an encoded video stream using a satellite line, a cable TV line, the Internet, a cellular communication network, or the like, or an encoded video stream such as an optical disk, a magnetic disk, or a flash memory The present invention can be applied to various electronic devices such as a recording device for recording on a medium. Hereinafter, three application examples will be described.

(1) First Application Example FIG. 14 shows an example of a schematic configuration of a mobile phone to which the above-described embodiment is applied. A cellular phone 920 includes an antenna 921, a communication unit 922, an audio codec 923, a speaker 924, a microphone 925, a camera unit 926, an image processing unit 927, a demultiplexing unit 928, a recording / reproducing unit 929, a display unit 930, a control unit 931, an operation Part 932, sensor part 933, bus 934, and battery 935.

  The antenna 921 is connected to the communication unit 922. The speaker 924 and the microphone 925 are connected to the audio codec 923. The operation unit 932 is connected to the control unit 931. The bus 934 connects the communication unit 922, the audio codec 923, the camera unit 926, the image processing unit 927, the demultiplexing unit 928, the recording / reproducing unit 929, the display unit 930, the control unit 931, and the sensor unit 933 to each other.

  The mobile phone 920 has various operation modes including a voice call mode, a data communication mode, a shooting mode, and a videophone mode, and is used for sending and receiving voice signals, sending and receiving e-mail or image data, taking images, and recording data. Perform the action.

  In the voice call mode, an analog voice signal generated by the microphone 925 is supplied to the voice codec 923. The audio codec 923 converts an analog audio signal into audio data, A / D converts the compressed audio data, and compresses it. Then, the audio codec 923 outputs the compressed audio data to the communication unit 922. The communication unit 922 encodes and modulates the audio data and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal. Then, the communication unit 922 demodulates and decodes the received signal to generate audio data, and outputs the generated audio data to the audio codec 923. The audio codec 923 expands the audio data and performs D / A conversion to generate an analog audio signal. Then, the audio codec 923 supplies the generated audio signal to the speaker 924 to output audio.

  Further, in the data communication mode, for example, the control unit 931 generates character data that constitutes an e-mail in response to an operation by the user via the operation unit 932. In addition, the control unit 931 causes the display unit 930 to display characters. In addition, the control unit 931 generates e-mail data in response to a transmission instruction from the user via the operation unit 932, and outputs the generated e-mail data to the communication unit 922. The communication unit 922 encodes and modulates email data and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal. Then, the communication unit 922 demodulates and decodes the received signal to restore the email data, and outputs the restored email data to the control unit 931. The control unit 931 displays the content of the electronic mail on the display unit 930 and stores the electronic mail data in the storage medium of the recording / reproducing unit 929.

  The recording / reproducing unit 929 has an arbitrary readable / writable storage medium. For example, the storage medium may be a built-in storage medium such as a RAM or a flash memory, or an externally mounted storage medium such as a hard disk, a magnetic disk, a magneto-optical disk, an optical disk, a USB memory, or a memory card. May be.

  In the shooting mode, for example, the camera unit 926 captures an image of a subject, generates image data, and outputs the generated image data to the image processing unit 927. The image processing unit 927 encodes the image data input from the camera unit 926 and stores the encoded stream in the storage medium of the recording / playback unit 929.

  Further, in the videophone mode, for example, the demultiplexing unit 928 multiplexes the video stream encoded by the image processing unit 927 and the audio stream input from the audio codec 923, and the multiplexed stream is the communication unit 922. Output to. The communication unit 922 encodes and modulates the stream and generates a transmission signal. Then, the communication unit 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies a radio signal received via the antenna 921 and performs frequency conversion to acquire a received signal. These transmission signal and reception signal may include an encoded bit stream. Then, the communication unit 922 demodulates and decodes the received signal to restore the stream, and outputs the restored stream to the demultiplexing unit 928. The demultiplexing unit 928 separates the video stream and the audio stream from the input stream, and outputs the video stream to the image processing unit 927 and the audio stream to the audio codec 923. The image processing unit 927 decodes the video stream and generates video data. The video data is supplied to the display unit 930, and a series of images is displayed on the display unit 930. The audio codec 923 decompresses the audio stream and performs D / A conversion to generate an analog audio signal. Then, the audio codec 923 supplies the generated audio signal to the speaker 924 to output audio.

  The sensor unit 933 includes a sensor group such as an acceleration sensor and a gyro sensor, and outputs an index representing the movement of the mobile phone 920. The battery 935 includes a communication unit 922, an audio codec 923, a camera unit 926, an image processing unit 927, a demultiplexing unit 928, a recording / reproducing unit 929, a display unit 930, and a control via a power supply line which is omitted in the drawing. Power is supplied to the unit 931 and the sensor unit 933.

  In the mobile phone 920 configured as described above, the image processing unit 927 has the function of the image encoding device 10 according to the above-described embodiment. Thereby, in the mobile phone 920, the action of the SAO processing can be adaptively reduced based on the image feature amount.

(2) Second Application Example FIG. 15 shows an example of a schematic configuration of a recording / reproducing apparatus to which the above-described embodiment is applied. For example, the recording / reproducing device 940 encodes audio data and video data of a received broadcast program and records the encoded data on a recording medium. In addition, the recording / reproducing device 940 may encode audio data and video data acquired from another device and record them on a recording medium, for example. In addition, the recording / reproducing device 940 reproduces data recorded on the recording medium on a monitor and a speaker, for example, in accordance with a user instruction. At this time, the recording / reproducing device 940 decodes the audio data and the video data.

  The recording / reproducing apparatus 940 includes a tuner 941, an external interface 942, an encoder 943, an HDD (Hard Disk Drive) 944, a disk drive 945, a selector 946, a decoder 947, an OSD (On-Screen Display) 948, a control unit 949, and a user interface. 950.

  The tuner 941 extracts a signal of a desired channel from a broadcast signal received via an antenna (not shown), and demodulates the extracted signal. Then, the tuner 941 outputs the encoded bit stream obtained by the demodulation to the selector 946. That is, the tuner 941 has a role as a transmission unit in the recording / reproducing apparatus 940.

  The external interface 942 is an interface for connecting the recording / reproducing apparatus 940 to an external device or a network. The external interface 942 may be, for example, an IEEE 1394 interface, a network interface, a USB interface, or a flash memory interface. For example, video data and audio data received via the external interface 942 are input to the encoder 943. That is, the external interface 942 serves as a transmission unit in the recording / reproducing device 940.

  The encoder 943 encodes video data and audio data when the video data and audio data input from the external interface 942 are not encoded. Then, the encoder 943 outputs the encoded bit stream to the selector 946.

  The HDD 944 records an encoded bit stream in which content data such as video and audio is compressed, various programs, and other data on an internal hard disk. Also, the HDD 944 reads out these data from the hard disk when playing back video and audio.

  The disk drive 945 performs recording and reading of data with respect to the mounted recording medium. The recording medium mounted on the disk drive 945 may be, for example, a DVD disk (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD + R, DVD + RW, etc.) or a Blu-ray (registered trademark) disk. .

  The selector 946 selects an encoded bit stream input from the tuner 941 or the encoder 943 when recording video and audio, and outputs the selected encoded bit stream to the HDD 944 or the disk drive 945. In addition, the selector 946 outputs the encoded bit stream input from the HDD 944 or the disk drive 945 to the decoder 947 during video and audio reproduction.

  The decoder 947 decodes the encoded bit stream and generates video data and audio data. Then, the decoder 947 outputs the generated video data to the OSD 948. The decoder 904 outputs the generated audio data to an external speaker.

  The OSD 948 reproduces the video data input from the decoder 947 and displays the video. Further, the OSD 948 may superimpose a GUI image such as a menu, a button, or a cursor on the video to be displayed.

  The control unit 949 includes a processor such as a CPU and memories such as a RAM and a ROM. The memory stores a program executed by the CPU, program data, and the like. The program stored in the memory is read and executed by the CPU when the recording / reproducing apparatus 940 is activated, for example. The CPU controls the operation of the recording / reproducing device 940 according to an operation signal input from the user interface 950, for example, by executing the program.

  The user interface 950 is connected to the control unit 949. The user interface 950 includes, for example, buttons and switches for the user to operate the recording / reproducing device 940, a remote control signal receiving unit, and the like. The user interface 950 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 949.

  In the recording / reproducing apparatus 940 configured as described above, the encoder 943 has the function of the image encoding apparatus 10 according to the above-described embodiment. Thereby, the recording / reproducing apparatus 940 can adaptively reduce the action of the SAO processing based on the image feature amount.

(3) Third Application Example FIG. 16 illustrates an example of a schematic configuration of an imaging apparatus to which the above-described embodiment is applied. The imaging device 960 images a subject to generate an image, encodes the image data, and records it on a recording medium.

  The imaging device 960 includes an optical block 961, an imaging unit 962, a signal processing unit 963, an image processing unit 964, a display unit 965, an external interface 966, a memory 967, a media drive 968, an OSD 969, a control unit 970, a user interface 971, and a sensor 972. , A bus 973 and a battery 974.

  The optical block 961 is connected to the imaging unit 962. The imaging unit 962 is connected to the signal processing unit 963. The display unit 965 is connected to the image processing unit 964. The user interface 971 is connected to the control unit 970. The bus 973 connects the image processing unit 964, the external interface 966, the memory 967, the media drive 968, the OSD 969, the control unit 970, and the sensor 972 to each other.

  The optical block 961 includes a focus lens and a diaphragm mechanism. The optical block 961 forms an optical image of the subject on the imaging surface of the imaging unit 962. The imaging unit 962 includes an image sensor such as a CCD or a CMOS, and converts an optical image formed on the imaging surface into an image signal as an electrical signal by photoelectric conversion. Then, the imaging unit 962 outputs the image signal to the signal processing unit 963.

  The signal processing unit 963 performs various camera signal processes such as knee correction, gamma correction, and color correction on the image signal input from the imaging unit 962. The signal processing unit 963 outputs the image data after the camera signal processing to the image processing unit 964.

  The image processing unit 964 encodes the image data input from the signal processing unit 963, and generates encoded data. Then, the image processing unit 964 outputs the generated encoded data to the external interface 966 or the media drive 968. The image processing unit 964 also decodes encoded data input from the external interface 966 or the media drive 968 to generate image data. Then, the image processing unit 964 outputs the generated image data to the display unit 965. In addition, the image processing unit 964 may display the image by outputting the image data input from the signal processing unit 963 to the display unit 965. Further, the image processing unit 964 may superimpose display data acquired from the OSD 969 on an image output to the display unit 965.

  The OSD 969 generates a GUI image such as a menu, a button, or a cursor, and outputs the generated image to the image processing unit 964.

  The external interface 966 is configured as a USB input / output terminal, for example. The external interface 966 connects the imaging device 960 and a printer, for example, when printing an image. Further, a drive is connected to the external interface 966 as necessary. For example, a removable medium such as a magnetic disk or an optical disk is attached to the drive, and a program read from the removable medium can be installed in the imaging device 960. Further, the external interface 966 may be configured as a network interface connected to a network such as a LAN or the Internet. That is, the external interface 966 has a role as a transmission unit in the imaging device 960.

  The recording medium mounted on the media drive 968 may be any readable / writable removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory. Further, a recording medium may be fixedly attached to the media drive 968, and a non-portable storage unit such as a built-in hard disk drive or an SSD (Solid State Drive) may be configured.

  The control unit 970 includes a processor such as a CPU and memories such as a RAM and a ROM. The memory stores a program executed by the CPU, program data, and the like. The program stored in the memory is read and executed by the CPU when the imaging device 960 is activated, for example. The CPU controls the operation of the imaging device 960 according to an operation signal input from the user interface 971, for example, by executing the program.

  The user interface 971 is connected to the control unit 970. The user interface 971 includes, for example, buttons and switches for the user to operate the imaging device 960. The user interface 971 detects an operation by the user via these components, generates an operation signal, and outputs the generated operation signal to the control unit 970.

  The sensor 972 includes a sensor group such as an acceleration sensor and a gyro sensor, and outputs an index representing the movement of the imaging device 960. The battery 974 supplies power to the imaging unit 962, the signal processing unit 963, the image processing unit 964, the display unit 965, the media drive 968, the OSD 969, the control unit 970, and the sensor 972 via a power supply line that is omitted in the drawing. Supply.

  In the imaging device 960 configured as described above, the image processing unit 964 has the function of the image encoding device 10 according to the above-described embodiment. Thereby, the imaging device 960 can adaptively reduce the action of the SAO processing based on the image feature amount.

<8. Summary>
So far, the embodiments of the technology according to the present disclosure have been described in detail with reference to FIGS. According to the above-described embodiment, in the sample adaptive offset (SAO) process, the offset value to be applied to the reconstructed image is corrected based on the image feature amount for each block of the image, and the reconstruct is performed using the corrected offset value. The pixel value of the image is offset. Accordingly, it is possible to prevent the inconvenience that the texture of the image becomes unclear and the subjective image quality of the decoded image is deteriorated as a result of applying an excessive or unnecessary offset in the SAO processing.

  According to one embodiment, when the image feature amount indicates that the image of a certain block is relatively flat, the offset applied to the block in the SAO process is reduced. As a result, it is possible to prevent deterioration of subjective image quality that does not match the gain of noise removal for a block that includes fewer high-frequency components (that is, less noise to be removed).

  According to another embodiment, when the image feature amount indicates that the loss of high-frequency components in a block is relatively small, the offset applied to the block in the SAO process is reduced. As a result, it is possible to prevent deterioration in subjective image quality that does not match the gain of noise removal for a block that is expected to generate less noise due to loss of high-frequency components.

  According to another embodiment, when the motion vector is zero in all or a relatively large number of sub-blocks in a block, the offset applied to the block in SAO processing is reduced or the application of the offset is reduced. Skipped. As a result, it is possible to prevent the image from being blurred due to the cumulative addition of the offset in the image area having no motion.

  Further, according to the above-described embodiment, the corrected offset value obtained by correcting the offset determined based on the rate distortion (RD) optimization can be used in the SAO process. According to such a configuration, the subjective image quality of the decoded image can be improved as compared with an existing method in which an offset value based only on RD optimization that does not consider subjective image quality is used.

  The terms CTB, CB, PB, and TB described herein may be replaced by the terms CTU, CU, PU, and TU, respectively. CTB, CB, PB, and TB primarily mean individual blocks (or sub-blocks) as part of an image, while CTU, CU, PU, and TU are logical units that also contain associated syntax. Mainly means.

  Further, in this specification, the example in which the SAO related parameters are multiplexed in the header of the encoded stream and transmitted from the encoding side to the decoding side has been mainly described. However, the method for transmitting such information is not limited to such an example. For example, these pieces of information may be transmitted or recorded as separate data associated with the encoded bitstream without being multiplexed into the encoded bitstream. Here, the term “associate” means that an image (which may be a part of an image such as a slice or a block) included in the bitstream and information corresponding to the image can be linked at the time of decoding. Means. That is, information may be transmitted on a transmission path different from that of the image (or bit stream). Information may be recorded on a recording medium (or another recording area of the same recording medium) different from the image (or bit stream). Furthermore, the information and the image (or bit stream) may be associated with each other in an arbitrary unit such as a plurality of frames, one frame, or a part of the frame.

  The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

  In addition, the effects described in the present specification are merely illustrative or illustrative, and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification, together with the above effects or instead of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
An offset correction unit that corrects an offset value to be applied to the reconstructed image based on an image feature amount for each block of the reconstructed image;
An offset unit for offsetting a pixel value of the reconstructed image with an offset value corrected by the offset correcting unit;
An image processing apparatus comprising:
(2)
The image feature amount includes an index related to flatness of each block,
The offset correction unit corrects the offset value so that a smaller offset value is applied to a flatter block;
The image processing apparatus according to (1).
(3)
The index related to the flatness is the image processing apparatus according to (2), including one or more of a variance of pixel values, a standard deviation, and a first derivative sum.
(4)
The image feature amount includes an index related to a loss of high frequency components in each block,
The offset correction unit corrects the offset value so that a smaller offset value is applied to a block with a smaller loss of high-frequency components.
The image processing apparatus according to (1).
(5)
The index related to the loss of the high-frequency component may be one or more of a ratio of sub-blocks whose transform coefficient is zero, a quantization parameter, and a ratio of sub-blocks whose motion compensation sample accuracy is a specific accuracy. The image processing apparatus according to (4), including:
(6)
The image processing apparatus according to (5), wherein the offset correction unit corrects the offset value so that a smaller offset value is applied to a block having a larger ratio of sub-blocks having the transform coefficient of zero.
(7)
The image processing apparatus according to (6), wherein the strength of the correction performed by the offset correction unit is reduced as the value of the quantization parameter increases.
(8)
The image processing apparatus according to (5), wherein the offset correction unit corrects the offset value so that a smaller offset value is applied to a block having a smaller quantization parameter.
(9)
The image according to (5), wherein the offset correction unit corrects the offset value so that a smaller offset value is applied to a block having a larger proportion of sub-blocks in which the motion compensation sample accuracy is decimal accuracy. Processing equipment.
(10)
The image processing device according to (1), wherein the image feature amount includes a ratio of sub-blocks having a motion vector of zero.
(11)
The image processing apparatus according to (10), wherein the offset correction unit corrects the offset value so that a smaller offset value is applied to a block having a larger proportion of sub-blocks in which the motion vector is zero.
(12)
The image processing apparatus further includes an offset determination unit that determines an offset value to be applied to the reconstructed image for each block based on rate distortion optimization,
The offset correction unit corrects the offset value determined by the offset determination unit based on the image feature amount;
The image processing apparatus according to any one of (1) to (11).
(13)
The offset determination unit selects an offset type for each block,
The offset correction unit corrects the offset value for a block in which the selected offset type is an edge offset;
The image processing apparatus according to (12).
(14)
The image processing apparatus according to any one of (1) to (13), wherein the offset unit skips offsetting a pixel value of a block belonging to a B picture or a B slice.
(15)
The image processing apparatus according to any one of (1) to (14), wherein the offset unit skips offsetting a pixel value of a block or sub-block whose motion vector is zero.
(16)
The image processing apparatus according to any one of (1) to (15), wherein the block corresponds to a CTB (Coding Tree Block).
(17)
Correcting an offset value to be applied to the reconstructed image based on an image feature amount for each block of the reconstructed image;
Offsetting the pixel value of the reconstructed image with the modified offset value;
An image processing method including:
(18)
An offset correction unit that corrects an offset value to be applied to the reconstructed image based on an image feature amount for each block of the reconstructed image;
An encoding unit for encoding the offset value corrected by the offset correction unit;
A decoding unit for decoding the modified offset value encoded by the encoding unit;
An offset unit for offsetting a pixel value of the reconstructed image with the corrected offset value decoded by the decoding unit;
An image processing system comprising:

DESCRIPTION OF SYMBOLS 1 Image processing system 10 Image processing apparatus (image coding apparatus)
DESCRIPTION OF SYMBOLS 16 Lossless encoding part 25 SAO filter 41 Offset determination part 43 Feature-value acquisition part 45 Offset correction part 47 Offset part 60 Image processing apparatus (image decoding apparatus)
62 Lossless decoding unit 67 SAO filter

Claims (18)

An offset correction unit that corrects an offset value to be applied to the reconstructed image based on an image feature amount for each block of the reconstructed image;
An offset unit for offsetting a pixel value of the reconstructed image with an offset value corrected by the offset correcting unit;
An image processing apparatus comprising: The image feature amount includes an index related to flatness of each block,
The offset correction unit corrects the offset value so that a smaller offset value is applied to a flatter block;
The image processing apparatus according to claim 1.   The image processing apparatus according to claim 2, wherein the index relating to flatness includes one or more of a variance of pixel values, a standard deviation, and a first-order differential sum. The image feature amount includes an index related to a loss of high frequency components in each block,
The offset correction unit corrects the offset value so that a smaller offset value is applied to a block with a smaller loss of high-frequency components.
The image processing apparatus according to claim 1.   The index related to the loss of the high-frequency component is one or more of a ratio of sub-blocks having a transform coefficient of zero, a quantization parameter, and a ratio of sub-blocks having a specific accuracy of motion compensation sampling The image processing apparatus according to claim 4, further comprising:   The image processing apparatus according to claim 5, wherein the offset correction unit corrects the offset value so that a smaller offset value is applied to a block having a larger ratio of sub blocks in which the transform coefficient is zero.   The image processing apparatus according to claim 6, wherein the correction strength by the offset correction unit is weakened as the value of the quantization parameter increases.   The image processing apparatus according to claim 5, wherein the offset correction unit corrects the offset value so that a smaller offset value is applied to a block having a smaller quantization parameter.   The image processing according to claim 5, wherein the offset correction unit corrects the offset value so that a smaller offset value is applied to a block having a larger proportion of sub-blocks in which the motion compensation sample accuracy is decimal accuracy. apparatus.   The image processing apparatus according to claim 1, wherein the image feature amount includes a proportion of sub-blocks having a motion vector of zero.   The image processing apparatus according to claim 10, wherein the offset correction unit corrects the offset value so that a smaller offset value is applied to a block having a larger proportion of sub-blocks in which the motion vector is zero. The image processing apparatus further includes an offset determination unit that determines an offset value to be applied to the reconstructed image for each block based on rate distortion optimization,
The offset correction unit corrects the offset value determined by the offset determination unit based on the image feature amount;
The image processing apparatus according to claim 1. The offset determination unit selects an offset type for each block,
The offset correction unit corrects the offset value for a block in which the selected offset type is an edge offset;
The image processing apparatus according to claim 12.   The image processing apparatus according to claim 1, wherein the offset unit skips offsetting pixel values of blocks belonging to a B picture or a B slice.   The image processing apparatus according to claim 1, wherein the offset unit skips offsetting a pixel value of a block or sub-block whose motion vector is zero.   The image processing apparatus according to claim 1, wherein the block corresponds to a CTB (Coding Tree Block). Correcting an offset value to be applied to the reconstructed image based on an image feature amount for each block of the reconstructed image;
Offsetting the pixel value of the reconstructed image with the modified offset value;
An image processing method including: An offset correction unit that corrects an offset value to be applied to the reconstructed image based on an image feature amount for each block of the reconstructed image;
An encoding unit for encoding the offset value corrected by the offset correction unit;
A decoding unit for decoding the modified offset value encoded by the encoding unit;
An offset unit for offsetting a pixel value of the reconstructed image with the corrected offset value decoded by the decoding unit;
An image processing system comprising:

Images