JP6200220B2 - Image processing apparatus, encoding apparatus, decoding apparatus, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an encoding apparatus, a decoding apparatus, and a program that adjust a quantization step using depth information of an image.

  In recent years, a camera equipped with a depth sensor capable of acquiring depth information has been actively researched and developed. For example, there is Microsoft's Kinect (registered trademark) as an example of a camera equipped with an RGB-D sensor for acquiring image information obtained by adding D (depth) to RGB. From such a background, it is expected that the use of moving image information having a two-dimensional image signal and depth information will increase.

  In addition, H.264, which is a standard for efficiently processing moving image information. H.264 / AVC (Advanced Video Coding) and H.264. In the H.265 / HEVC (High Efficiency Video Coding) video compression encoding method, each frame of the video is divided into rectangular areas called blocks, and encoding is performed.

  As one of the video compression methods, quantization processing is performed when an image is encoded. Quantization is a process of roughly re-dividing the fineness of data originally possessed. That is, the process of rounding the result obtained by dividing each frequency component subjected to orthogonal transformation by a predetermined quantization step to an integer value is called “quantization”. Conversely, the process of multiplying the integer value obtained by this quantization by a quantization step and returning it to the orthogonally transformed component is called “inverse quantization”.

  Quantization reduces the number of bits required for encoding compared to direct encoding of orthogonally transformed components by encoding smaller integer values and smaller variations of quantized values than orthogonally transformed components (For example, see Non-Patent Document 1 and Non-Patent Document 2).

  FIG. 2 is a diagram illustrating an example of a quantization matrix of an inter block (inter-coded block) in H.264 / AVC. FIG. 1A shows a 4 × 4 16 inter-block quantization matrix 110. FIG. 1B shows an 8 × 8 64 inter-block quantization matrix 120.

  For example, the quantization step value of the block 101 of the quantization matrix 120 in FIG. 1B indicates “9”. When the image is DCT (Discrete Cosine Transform), the DCT coefficient corresponding to the block 101 corresponds to a DC (direct current) component. Therefore, the quotient value obtained by dividing the DC component of DCT by 9 and discarding the remainder is the value after quantization of the DC component.

  When the quantization step is small, the frequency component of the DCT coefficient is divided by the quantization step, and the remainder that is discarded is generally small. For this reason, an error between a value obtained by dequantizing the quantized value and a value before quantization is small, and deterioration in image quality due to quantization is small. However, since the value of the quotient generally increases and the variation that the value of the quotient can take increases, the coding efficiency (compression rate) decreases. Therefore, it is possible to adjust the compression rate, the image quality, and the like by changing the quantization step.

  In general, the quantization step is set to a small value with respect to a low-frequency component (upper left portion of the quantization matrix in FIG. 1) that is easily recognized by human eyes. In many cases, a large value is set for a high-frequency component (lower right portion of the quantization matrix in FIG. 1) in which image degradation is difficult to be recognized by human eyes.

  As described above, if the quantization step is increased, the encoding efficiency of the image can be increased. However, there is a high tendency for pseudo contours such as contour lines to occur in an image region having a low spatial frequency.

  Conversely, if the quantization step is reduced, the compression ratio of the image is reduced, but an image with improved image expression and definition can be reproduced. However, for example, even in an image region that is generally not important by humans, such as outside the depth of field, the low compression process is performed, so that the overall compression rate is reduced.

  Therefore, it is important to appropriately select the value of the quantization step according to the nature of the image in order to ensure the quality of the mutually contradictory image and ensure the compression rate.

Wataru Kameyama, Takeshi Hanamura, “Digital Broadcast Textbook (above) MPEG-1 / MPEG-2 / MPEG-4”, Impress, 2003 Supervised by Satoshi Okubo, “Revised third edition H.264 / AVC textbook”, Impress R & D, p124, January 1, 2009

  As described above, the quantization step is closely related to the image quality such as the generation of pseudo contours and the compression rate of the image data. Since the image area within the depth of field is likely to be a gaze area, visual image degradation due to pseudo contours is easily noticeable.

  In addition, noise generated in an image pickup device such as a CMOS is uniformly included in / out of the depth of field, which reduces the encoding efficiency. However, noise is important for improving the image expression and definition, and it does not have to be removed uniformly within / out of the depth of field.

  Therefore, the present invention uses the depth information to obtain information inside / outside the depth of field of the image, and adjusts the value of the quantization step using this information, thereby suppressing image degradation, or The purpose is to improve the coding efficiency.

An image processing apparatus according to an aspect of the present invention is an image processing apparatus that adjusts a quantization step of an image by using information on the depth of the image, and calculates a spatial frequency power of each of a plurality of blocks included in the image. A spatial frequency acquisition unit to acquire, and a depth information clustering unit that clusters information on the depth of each pixel position or each predetermined region in the image and determines each cluster of the plurality of blocks based on the result of the clustering; a power of one or more spatial frequencies of the block belonging to a given cluster, based on the power of the spatial frequency of a plurality of blocks belonging to other than the given cluster, the one or more blocks belonging to the given cluster And a determination result of the depth of field determination unit, the determination result of the depth of field determination unit, A quantization matrix including a quantization step including a quantization step for each of a plurality of frequency components orthogonally transformed with respect to a predetermined block of the plurality of blocks based on a compression rate of the recorded image; A matrix correction unit.

A compression rate determination unit that determines that the compression rate of the image is higher than a predetermined first compression rate as a high compression rate, wherein the depth-of-field determination unit includes at least one of the predetermined clusters The proportion of the spatial frequency power exceeding the predetermined spatial frequency among the spatial frequencies of the blocks is higher than the proportion of the power of the spatial frequency exceeding the predetermined spatial frequency among the spatial frequencies of the plurality of blocks belonging to other than the predetermined cluster. The one or more blocks belonging to the predetermined cluster are determined to belong to a depth of field, and the quantization matrix correction unit determines the depth of field when the compression rate is a high compression rate. Among the one or more blocks determined to belong to, a predetermined area of the quantization matrix for a block in which the power ratio of the low frequency component of the spatial frequency is higher than a predetermined ratio , It may be modified to a smaller value.

  In addition, the quantization matrix correction unit, for the blocks other than the one or more blocks determined to belong within the depth of field when the compression rate is lower than a predetermined second compression rate, The value of the quantization matrix may be modified to a larger value.

  Further, in the block in which the ratio of the power of the low frequency component of the spatial frequency is higher than a predetermined ratio, the value obtained by dividing the power of the spatial frequency below the first threshold by the power of the spatial frequency exceeding the first threshold is: The block may exceed the second threshold.

  An encoding apparatus according to another aspect of the present invention may include the image processing apparatus.

  A program according to another aspect of the present invention causes a computer to function as the encoding device.

  According to the present invention, since the quantization step is adjusted by performing the inside / outside determination of the depth of field in the image using the depth information, it is possible to suppress the image deterioration or improve the encoding efficiency. .

The figure which shows the example of the quantization matrix of an inter block (inter-screen coding block). 1 is a schematic block diagram of an image processing apparatus in Embodiment 1. FIG. The figure which shows the example of the area | region of the high band in the spatial frequency of an orthogonal transformation block. 3 is a flowchart illustrating an example of processing performed by the image processing apparatus according to the first exemplary embodiment. 6 is a flowchart illustrating an example of depth of field determination in the first embodiment. 9 is a flowchart illustrating an example of correcting a quantization matrix value according to the first embodiment. FIG. 6 is a block diagram illustrating an example of a schematic configuration of an image processing apparatus according to a second embodiment. FIG. 9 is a block diagram illustrating an example of a schematic configuration of an image processing apparatus according to a third embodiment. FIG. 10 is a block diagram illustrating an example of a schematic configuration of an image processing apparatus according to a fourth embodiment.

  Hereinafter, embodiments will be described with reference to the drawings.

[Example 1]
The image processing apparatus 10 according to the first embodiment is an apparatus that determines a modified quantization matrix. This image processing apparatus may function by executing a program, or may be implemented by an integrated circuit or the like.

<Outline and Position of Example 1>
Example 1 uses depth-of-field information based on image depth information to determine a modified quantization matrix (or quantization parameter). As already described, this quantization matrix (or quantization parameter) is used to compress information in the process of image encoding. Therefore, Example 1 described in detail below can be used in, for example, an image encoding device. A specific example of the image encoding device will be described in the second embodiment.

  In order to obtain information on the depth of field used in the first embodiment, it is desirable to acquire information for one frame of the image regarding the depth information in each divided block and the spatial frequency information. For this reason, the first embodiment described below shows a specific example in which image processing for one frame is performed in two passes.

  That is, in the first pass, depth-of-field information of each block in one frame is obtained. In the second pass, processing for correcting the quantization matrix (or quantization parameter) of each block in one frame is performed using this depth of field information.

  Therefore, those skilled in the art will understand that the entire image encoding apparatus including the first embodiment operates in both the first pass and the second pass. It is desirable to use the block division information in the first pass as it is in the second pass. For this reason, the block division information in the first pass is saved, and the saved block division information is used in the process of the second pass. The first embodiment is not limited to the two-pass process.

  In the first embodiment, reference is also made to a preprocessing unit, an entropy coding unit, a prediction error signal, post-block orthogonal transform information, post-quantization information, and the like in the image coding apparatus. These details will be described in detail in a second embodiment showing a specific example of an image encoding device. In order to facilitate understanding of the first embodiment, the outline thereof will be briefly described below.

  The preprocessing unit rearranges the pictures in accordance with the picture type of the original image, and sequentially outputs the picture type and the frame image for each frame. The preprocessing unit also performs block division and the like.

  The entropy encoding unit assigns a variable-length code according to the appearance frequency of the symbol, and encodes the quantized signal output and the like.

  The prediction error signal means a difference value between block data of the original image and block data of the prediction image. The amount of information of an image to be encoded can be compressed by predicting an image from other frame information or peripheral information in the same frame, obtaining a difference from the original image, and obtaining a prediction error signal. .

  The post-block orthogonal transformation information means information obtained by orthogonal transformation of the prediction error signal. For example, DCT (Discrete Cosine Transform) is used as the orthogonal transform method. Information can be further compressed by orthogonally transforming the prediction error signal.

  The post-quantization information means information obtained by quantizing information after orthogonal transformation. By quantizing the information after the orthogonal transformation, the code amount of the output signal can be reduced.

Based on the above assumptions, the configuration and operation of the first embodiment will be described in detail below. The first embodiment is not limited to these assumptions.
<Configuration of Example 1>
The processing for one frame in the first embodiment is roughly divided into the first pass and the second pass. When the first pass processing for the specific frame is completed, the second pass processing for the specific frame is executed.

  FIG. 2 is a schematic block diagram of the image processing apparatus 10 according to the first embodiment. The image processing apparatus illustrated in FIG. 2 includes a first pass operation unit 140, a 1 pass / 2 pass switching instruction unit 130, and a quantization matrix correction unit 160. The 1-pass / 2-pass switching instruction unit 130 activates the first-pass operation instruction signal 132 and operates the first-pass operation unit 140. Further, the 1-pass / 2-pass switching instruction unit 130 gives the 1-pass / 2-path switching instruction 162 to the pre-processing unit 200, and recognizes that the pre-processing unit 200 is the first pass process. The operation of the preprocessing unit will be described later. The 1-pass / 2-pass switching instruction unit 130 activates the second-pass operation instruction signal 184 and operates the quantization matrix correction unit 160. Further, the 1-pass / 2-pass switching instruction unit 130 activates the second-pass operation instruction signal 184 and operates an entropy encoding unit described later.

  The first-pass operation unit 140 includes a block division information storage unit 111, a prediction error signal information amount accumulation unit 112, a depth information clustering unit 113, a depth of field determination unit 114, a spatial frequency acquisition unit 115, a quantum The information amount integrating unit 116 after conversion and the compression rate determining unit 117 are included. The depth of field determination unit 114 includes a depth of field storage unit 118.

  The first pass operation unit 140 operates in the first pass, and the second pass has a capability of providing necessary information to other processing units.

  The block division information storage unit 111 acquires the block division information 164 from the preprocessing unit 200 and sequentially stores it. The block division information storage unit 111 can store block division information for at least one frame. The block division information for one frame is provided to the depth of field storage unit 118. Further, the block division information 164 for one frame is provided to the preprocessing unit, so that the preprocessing unit can perform the same block division in the second pass as in the first pass.

  The prediction error signal information amount integration unit 112 acquires an integrated value a of the information amount of the prediction error signal 168 for one frame.

  The post-quantization information amount integration unit 116 integrates post-quantization information 182 for one frame and obtains an integrated value b.

  The compression rate determination unit 117 obtains the compression rate c of the information amount for one frame achieved by orthogonal transform and quantization by calculating b / a = c. When the compression rate c is higher than the predetermined first compression rate, it is determined as “high compression rate”, and when the compression rate c is lower than the predetermined second compression rate, it is determined as “low compression rate”. The result of this determination is provided to the quantization matrix correction unit 160 in the second pass.

  For example, the depth information clustering unit 113 obtains a histogram of depth information for one frame, equally divides the range of values taken by the histogram into the number of clusters n (for example, n = 2), and classifies the depth information. The depth information clustering unit 113 assigns numbers (for example, cluster numbers = 1, 0) in descending depth order as the clustering result of the depth information of each pixel position or each predetermined region, and obtains this clustering result. As a clustering method, a K-means method can be used.

  The K-means method is a technique for performing clustering by giving a prototype that is a representative to each of a predetermined number of clusters (for example, n) and assigning each data to the nearest prototype. The first embodiment is not limited to a specific clustering method.

  The depth of field determination unit 114 uses, for example, post-block orthogonal transformation information 172 as clustering results and spatial frequency power information. The post-block orthogonal transform information 172 is acquired via the spatial frequency acquisition unit 115.

  In FIG. 2, the spatial frequency acquisition unit 115 may directly obtain the spatial frequency power from the original image instead of the post-block orthogonal transformation information 172. In this case, an original image is given to the spatial frequency acquisition unit 115. Then, the spatial frequency acquisition unit 115 calculates the power of the spatial frequency for the corresponding block. As a method for acquiring the power of the spatial frequency, DCT (Discrete Cosine Transform), Hadamard Transform, Discrete Fourier Transform, or the like can be used.

  First, the depth-of-field determination unit 114 calculates a ratio of power in a high band among spatial frequencies of orthogonal transform blocks clustered in the same cluster. Instead of power, the absolute value of the spatial frequency level, the intensity of each spatial frequency component, or the like may be used.

  In this specification, the term “spatial frequency power” is used, but this term also means “absolute value of spatial frequency level”, “intensity for each component of spatial frequency”, and the like. Use.

  FIG. 3 shows an example in which orthogonal transform coefficients (spatial frequencies) obtained by orthogonal transform of one block of an image are arranged in blocks. The spatial frequency component 300 is obtained by arranging 64 DCT coefficients obtained by DCT transforming 64 pixels of one orthogonal transform block (8 pixels × 8 lines) into a horizontal component and a vertical component.

For example, the region of the spatial high frequency component is a frequency region 390 indicated by hatching. The frequency region 390 is a region exceeding 1/2 of the maximum spatial frequency in the horizontal direction and the vertical direction. Assuming that the spatial frequency power in the frequency domain 390 is H ₁ and the spatial frequency power of the entire spatial frequency component 300 is W ₁ , the spatial high frequency power ratio α ₁ in the frequency domain 390 can be obtained by the following equation. .

α ₁ = H ₁ / W ₁
The number of clusters is set to two, and α ₁ obtained here is obtained for each of a plurality of blocks belonging to cluster 0 to obtain an average value α of the ratio of spatial high frequency power. Similarly, let β be the average value of the ratio of the power of the spatial high frequency obtained for all the blocks belonging to cluster 1.

  Then, α and β are compared, and if α> β, it can be determined that a plurality of blocks belonging to cluster 0 from which α is derived are within the depth of field region. The reason is that the region within the depth of field is based on an empirical rule that the ratio of the power of the spatial high frequency is large because there are many clear images without blur. As described above, it is possible to specify a region within / out of the depth of field.

  In addition, the method of taking the area | region of the spatial high frequency component shown in FIG. 3 is only an example, and Example 1 is not limited to this. In the above example, the number of clusters is 2. However, an area composed of a plurality of blocks belonging to a cluster having the highest spatial high frequency power ratio is divided into three or more clusters. You may determine with the area | region within the depth.

In the above example, the direct current (DC) component of the DCT coefficient at the upper left (0, 0) position of the spatial frequency component 300 in FIG. 3 is also used when obtaining the average value of the power of the entire spatial frequency. . However, since the DC component is a coefficient related to the brightness of the block, it may be excluded from the calculation when obtaining the average value of the power of the spatial frequency. By excluding the DC component, it is possible to obtain the spatial high frequency power ratio α ₁ regardless of the brightness of the image.

  The depth of field storage unit 118 included in the depth of field determination unit 114 is included in one frame using information from the block division information storage unit 111 and information from the depth of field determination unit 114. Store information inside / outside the depth of field of each block.

  Through the above processing, the first-pass operation unit 140 identifies whether the frame currently being processed is a high compression rate, a low compression rate, or otherwise. In addition, the first-pass operation unit 140 specifies inside / outside of the depth of field for each block of one frame.

  The 1-pass / 2-pass switching instruction unit 130 notifies the preprocessing unit 200 that the process is the second pass. In response to this transmission, the preprocessing unit 200 performs block division according to the block division information obtained from the block division information storage unit 111, and sequentially outputs information about the blocks so that each block is processed. Further, the 1-pass / 2-pass switching instruction unit 130 provides the second-pass operation instruction signal 184 to the quantization matrix correction unit 160 and the entropy encoding unit 204. The entropy encoding unit 204 will be described later with reference to FIG.

  The quantization matrix correction unit 160 includes a Qp value 174 that is a quantization matrix or a quantization parameter once created for encoding, information from the depth-of-field determination unit 114, information from the spatial frequency acquisition unit 115, And the information from the compression rate determination unit 117 is received. The quantization matrix correction unit 160 can correct the Qp value 174 that is a quantization matrix or a quantization parameter once created for encoding.

  First, a case where the compression rate determination unit 117 determines that the compression is high will be described. In this case, the quantization matrix correction unit 160 performs processing for suppressing the occurrence of pseudo contours on the blocks belonging to the depth of field.

  The quantization matrix correction | amendment part 160 specifies the block in which the ratio of the power of spatial low frequency is higher than a predetermined threshold among the blocks which belong within the depth of field. In general, a block within the depth of field has a high definition, so that the ratio of the power of the spatial high frequency is often higher than the power of the spatial low frequency. However, when the texture of the object itself has a spatial low frequency, there may be a block having a high spatial low frequency power ratio even within the depth of field. In such a block, the probability that a pseudo contour will occur as described above is high. Moreover, since the block exists within the depth of field, it is determined that the pseudo contour is conspicuous.

  Therefore, it is desirable that the quantization matrix correction unit 160 reduce the value of the quantization matrix for such a block in order to suppress the occurrence of pseudo contours. Therefore, the quantization matrix correction unit 160 identifies such a block.

  The quantization matrix correction unit 160 corrects the quantization step of the identified block to a smaller value. For example, the quantization matrix correction unit 160 may correct the value of each element of the quantization matrix to a value halved. Alternatively, the quantization matrix correction unit 160 may subtract 6 from the Qp value, for example. When the Qp value is subtracted by 6, each value of the quantization matrix is set to ½.

  Alternatively, for example, only the value of the quantization matrix corresponding to the spatial high frequency component region of the quantization matrix indicated by the frequency region 390 in FIG. 3 may be corrected to a value halved. Alternatively, in FIG. 3, only the value of the quantization matrix corresponding to the frequency region where the row number is 4 or more and the column number is 4 or more may be corrected to a value halved. It should be noted that the value of the coefficient 1/2 for correction is an example, and it goes without saying that other count values may be multiplied. Further, which spatial frequency band is adopted as the spatial high-frequency component is not limited to the above example.

  As described above, the reason for reducing the value of the quantization matrix corresponding to the spatial high-frequency component is that the high-frequency component is increased in the striped pattern because the pseudo contour exhibits a striped pattern. Therefore, in order to suppress the pseudo contour, it is often effective to reduce the quantization error of the high frequency component.

  Alternatively, the quantization matrix correction unit 160 individually corrects each value of the quantization matrix to generate a quantization matrix having a new value.

  Next, a case where the compression rate determination unit 117 determines that the compression is low will be described. In this case, it is possible to perform processing for improving the encoding efficiency. The quantization matrix correction unit 160 corrects the quantization matrix value for the orthogonal transform block determined by the depth of field determination unit 114 to be outside the depth of field to a larger value. This is because an area outside the depth of field is often not a gaze area, and even if the compression rate in this area is increased, it is difficult to recognize image degradation as an experience side. The quantization matrix correction unit 160 may correct the quantization matrix value for the orthogonal transform block determined by the depth of field determination unit 114 to be outside the depth of field, for example, to a double value. . In addition, Example 1 is not limited to 2 times.

  Alternatively, in the above case, the quantization matrix correction unit 160 may add, for example, 6 to the Qp value. As described above, when 6 is added to the Qp value, each value of the quantization matrix is doubled. Alternatively, the quantization matrix modification unit 160 may individually modify each value of the quantization matrix to generate a modified quantization matrix 186 having a new value.

  The quantization matrix information modified by the quantization matrix modification unit 160 as described above is transmitted to the decoding side through entropy coding (186). In addition, information within / outside of the depth of field may also be transmitted to the decoding side.

  In the above description, when the Qp value is corrected, the corrected Qp value may be transmitted to the decoding side. For example, H.M. In the H.264 / AVC standard, the Qp value is to be sent to the decoding side according to the standard. For this reason, if the corrected Qp value is sent to the decoding side, the decoding side does not need to newly provide an additional decoding function related to the correction of the quantization matrix.

<Operation>
Next, the operation of the image processing apparatus 10 in the first embodiment will be described. FIG. 4 is a diagram illustrating an example of processing of the image processing apparatus 10. In step S410 illustrated in FIG. 4, the spatial frequency acquisition unit 115 acquires post-block orthogonal transform information 172. Note that, as described above, the spatial frequency acquisition unit 115 may independently perform spatial frequency analysis for each block (orthogonal transform block) region in which orthogonal transform is performed on the frame (image) of the original image. .

  In step S420, the depth-of-field determination unit 114 performs inner / outer determination of the depth of field for each block of the image. In the determination, for example, 256-step depth information acquired for each pixel or for each predetermined region may be clustered by the K-means method or the like. As the number of clusters n, for example, n = 2. Details of the operation will be described with reference to FIG. The information on the depth of field for each block is stored in the depth of field storage unit 118 in association with the block division information in the block division information storage unit 111.

  In step S430, the compression rate determination unit 117 determines whether the compression rate of the entire image is high or low. For example, the compression rate determination unit 117 calculates the compression rate of the frame by calculating the ratio between the information of the prediction error signal information amount integration unit 112 and the information of the post-quantization information amount integration unit 116 as described above. be able to.

  In step S440, the quantization matrix correction unit 160 corrects the quantization matrix or Qp value 174 once created for encoding, and obtains a corrected quantization matrix or Qp value 186. Details of this operation will be described with reference to FIG.

  FIG. 5 is a flowchart illustrating an example of determination of the depth of field in the first embodiment.

  In step S510, based on the depth information 170 for one frame, the depth information clustering unit 113 clusters each block. An example of clustering is shown below.

  First, the depth information clustering unit 113 obtains a histogram of depth information, equally divides the range of values taken by the histogram into the number of clusters n (for example, n = 2), and classifies the depth information.

  The depth information clustering unit 113 assigns numbers (cluster number = 1, 0) in descending depth order as the clustering result of the depth information of each pixel position or each predetermined region, and obtains this clustering result.

  The depth information clustering unit 113 acquires a representative value of clustering information for each block. For example, the depth information clustering unit 113 obtains one depth information (representative value of cluster number = 1 or 0) for each block by averaging the clustering information for each block and performing rounding to an integer. . For example, if the value obtained by the above integer conversion is 1, the representative value of the cluster number = 1. If the integer value is 0, the representative value of the cluster number may be 0. The depth information clustering unit 113 may use the median value or mode value of the clustering information as a representative value of the cluster number for each block.

  In step S520, the depth-of-field determination unit 114 uses the information from the depth information clustering unit 113 to calculate the ratio of spatial high frequency power for each block belonging to the same cluster. Since the example of the calculation of the ratio of the spatial high frequency power in the block has already been described with reference to FIG.

  In step S530, the depth-of-field determining unit 114 identifies the cluster having the highest spatial high frequency power ratio. It can be estimated that a plurality of blocks belonging to the specified cluster are highly likely to exist within the depth of field.

  In step S <b> 540, the depth of field determination unit 114 sends information on the blocks belonging to the identified cluster to the depth of field storage unit 118. The information stored in the depth of field storage unit 118 is used in the quantization matrix correction unit 160 in the second pass.

  FIG. 6 is a flowchart illustrating an example in which the quantization matrix correction unit 160 corrects the value of the quantization matrix.

  In step S <b> 610, the quantization matrix correction unit 160 determines the processing branch based on the compression rate information from the compression rate determination unit 117 (information about whether the compression rate is high or low). I do. If the determination result is “Yes”, since the compression is high, the process proceeds to step S620. If the determination result is “No”, the process proceeds to step S611.

  In step S611, the quantization matrix correction unit 160 determines the processing branch based on the compression rate information from the compression rate determination unit 117 (information about whether the compression rate is high or low). I do. If the determination result is “Yes”, since the compression is low, the process proceeds to step S640. If the determination result is “No”, the processing is terminated because neither high compression nor low compression is performed.

  In step S620, the quantization matrix correction unit 160 identifies blocks having a spatial low frequency power ratio higher than, for example, a predetermined threshold among blocks belonging to the depth of field. In general, a block within the depth of field has a high definition, so that the ratio of the power of the spatial high frequency is often higher than the power of the spatial low frequency. However, when the texture of the object itself has a spatial low frequency, there may be a block having a high spatial low frequency power ratio even within the depth of field. In such a block, the probability that a pseudo contour will occur as described above is high. Moreover, since the block exists within the depth of field, it is determined that the pseudo contour is conspicuous. For such a block, for example, a block in which the value obtained by dividing the power of the spatial frequency below the first threshold by the power of the spatial frequency exceeding the first threshold in the block exceeds the second threshold may be found. The first threshold and the second threshold can be appropriately set by those skilled in the art when implementing this embodiment.

  Therefore, it is desirable that the quantization matrix correction unit 160 reduce the value of the quantization matrix for such a block in order to suppress the occurrence of pseudo contours. Therefore, the quantization matrix correction unit 160 identifies such a block.

  In step S630, the quantization matrix correction unit 160 sets the quantization step of the identified block to a smaller value. An example of setting the value of the quantization matrix has already been described in detail in the description of the quantization matrix correction unit 160, and is omitted here.

  In step S640, the quantization matrix correction unit 160 identifies a block outside the depth of field. As a rule of thumb, it is determined that blocks outside the depth of field are not easily caught by the eyes. Therefore, even if the compression rate of the block outside the depth of field is increased, it is predicted that the deterioration of the image is not noticeable. For this reason, the quantization matrix correction unit 160 increases the value of the quantization matrix corresponding to the block outside the depth of field so as to increase the compression rate of the block outside the depth of field. An example of setting the value of the quantization matrix has already been described in detail in the description of the quantization matrix correction unit 160, and is omitted here.

  As described above, in the first embodiment, information on the depth of field can be acquired from the depth information. In addition, it is possible to improve the image quality or increase the compression rate using the information on the depth of field. More specifically, during high compression, it is possible to sufficiently suppress the occurrence of pseudo contours that can occur within the depth of field. Further, at the time of low compression, the compression rate can be increased by increasing the quantization step in the region outside the depth of field so that the deterioration of the image is not as conspicuous as possible.

[Example 2]
In the second embodiment, an image processing apparatus (image encoding apparatus) including the image processing apparatus 10 in the first embodiment in the image processing unit (1) 11 will be described.

<Configuration>
FIG. 7 is a block diagram illustrating an example of a schematic configuration of the image processing apparatus 20 according to the second embodiment. In the example illustrated in FIG. 7, the image processing apparatus 20 includes a preprocessing unit 200, a prediction error signal generation unit 201, an orthogonal transformation unit 202, a quantization unit 203, an entropy coding unit 204, and an inverse quantization unit. 205, an inverse orthogonal transform unit 206, a decoded image generation unit 207, a loop filter unit 209, a decoded image storage unit 210, an intra prediction unit 211, an inter prediction unit 212, a motion vector calculation unit 213, and a prediction An image selection unit 215 and an image processing unit (1) 11 are included. An outline of each part will be described below.

  The preprocessing unit 200 rearranges the pictures according to the picture type, and sequentially outputs the picture type and the frame image for each frame. The preprocessing unit 200 also performs block division and the like, and outputs block division boundary information to the image processing unit (1) 11 and the loop filter unit 209. The pre-processing unit 200 acquires a 1-pass / 2-pass switching instruction 162 from the image processing unit (1) 11. In addition, the preprocessing unit 200 exchanges block division information with the image processing unit (1) 11 through the signal line 164. The pre-processing unit 200 gives block division information to the image processing unit (1) 11 in the first pass. The preprocessing unit 200 acquires block division information from the image processing unit (1) 11 in the second pass. In this way, the preprocessing unit 200 can use the same block division as the first pass block division in the second pass.

  The prediction error signal generation unit 201 obtains block data obtained by dividing the encoding target image of the input original image data into blocks such as 32 × 32, 16 × 16, and 8 × 8 pixels, for example.

  The prediction error signal generation unit 201 generates a prediction error signal based on the block data and the block data of the prediction image output from the prediction image selection unit 215. The prediction error signal generation unit 201 outputs the generated prediction error signal to the image processing unit (1) 11 and the orthogonal transformation unit 202.

  The orthogonal transform unit 202 performs an orthogonal transform process on the input prediction error signal. The orthogonal transform unit 202 outputs a signal indicating the transformed coefficient value to the image processing unit (1) 11 and the quantization unit 203.

  The quantization unit 203 quantizes the output signal from the orthogonal transform unit 202. The quantization unit 203 reduces the code amount of the output signal by performing quantization, and outputs the output signal to the image processing unit (1) 11, the entropy encoding unit 204, and the inverse quantization unit 205.

  The image processing unit (1) 11 can include the image processing apparatus 10 described in the first embodiment. The image processing unit (1) 11 receives the quantization parameter and the like from the quantization unit 203 through the signal line 174. Then, the information of the quantization parameter corrected by the image processing unit (1) 11 is returned to the quantization unit 203 through the signal line 186. The quantization unit 203 uses the modified quantization parameter to perform quantization. Note that the image processing unit (1) 11 may modify the quantization matrix itself instead of the quantization parameter, and give it to the quantization unit 203. Depth information is input to the image processing unit (1) 11 from the preprocessing unit 200 via the signal line 170. Note that the preprocessing unit 200 may provide the original image to the image processing unit (1) 11 as described above. The original image can be used in the spatial frequency acquisition unit 115 shown in FIG. The depth information is used in the depth information clustering unit 113 shown in FIG. Further, an orthogonal transform coefficient is input to the image processing unit (1) 11 from the orthogonal transform unit 202 via the signal line 710. This orthogonal transform coefficient is used by the spatial frequency acquisition unit 115 shown in FIG.

  The entropy encoding unit 204 recognizes that it is the second pass by receiving the second pass operation instruction signal 184 from the image processing unit (1) 11, and operates. The entropy encoding unit 204 entropy-encodes and outputs the output signal from the quantization unit 203, the motion vector information output from the motion vector calculation unit 213, the filter coefficient from the loop filter unit 209, and the like.

  Also, the entropy encoding unit 204 entropy encodes the difference value of the intra prediction direction acquired from the intra prediction unit 211, the difference value of the motion vector and the prediction vector acquired from the inter prediction unit 212, and the like.

  The entropy encoding unit 204 encodes information of the quantization matrix corrected by the image processing unit (1) 11. Entropy coding is a method of assigning variable-length codes according to the appearance frequency of symbols.

  The inverse quantization unit 205 performs inverse quantization on the output signal from the quantization unit 203 and outputs the result to the inverse orthogonal transform unit 206. The inverse orthogonal transform unit 206 performs an inverse orthogonal transform process on the output signal from the inverse quantization unit 205 and then outputs the output signal to the decoded image generation unit 207. By performing decoding processing by the inverse quantization unit 205 and the inverse orthogonal transform unit 206, a signal having the same level as the prediction error signal before encoding is obtained.

  The decoded image generation unit 207 is decoded by the block data of the image predicted in the screen by the intra prediction unit 211 or the motion compensated image by the inter prediction unit 212, and the inverse quantization unit 205 and the inverse orthogonal transform unit 206. Add the prediction error signal. The decoded image generation unit 207 outputs the decoded image block data generated by addition to the loop filter unit 209.

  The loop filter unit 209 is, for example, an ALF (Adaptive Loop Filter) or a deblocking filter. The loop filter unit 209 outputs the filter processing result to the decoded image storage unit 210, and stores the accumulated filter processing result for one image as a reference image.

  The decoded image storage unit 210 stores the input block data of the decoded image as new reference image data, and outputs the data to the intra prediction unit 211, the inter prediction unit 212, and the motion vector calculation unit 213.

  The intra prediction unit 211 generates block data of the predicted image from the already-encoded reference pixels for the processing target block of the encoding target image. The intra prediction unit 211 performs prediction using a plurality of prediction directions, and determines an optimal prediction direction. With respect to the prediction direction, the difference value is output to the entropy encoding unit 204 in order to include the difference value with the prediction direction of the encoded block in the bitstream.

  The inter prediction unit 212 performs motion compensation on the reference image data acquired from the decoded image storage unit 210 with the motion vector provided from the motion vector calculation unit 213. Thereby, block data as a motion-compensated reference image is generated. For the motion vector, the difference value with the motion vector (predicted vector) of the encoded block is output to the entropy encoding unit 204 in order to include the difference value in the bitstream.

  The motion vector calculation unit 213 obtains a motion vector using the block data in the encoding target image and the reference image acquired from the decoded image storage unit 210.

  The motion vector calculation unit 213 outputs the obtained motion vector to the inter prediction unit 212 and outputs motion vector information including information indicating the reference image to the entropy encoding unit 204.

  The block data output from the intra prediction unit 211 and the inter prediction unit 212 is input to the predicted image selection unit 215.

  The predicted image selection unit 215 selects one of the block data acquired from the intra prediction unit 211 and the inter prediction unit 212 as a predicted image. The selected prediction image is output to the prediction error signal generation unit 201.

  The configuration of the image processing apparatus 20 illustrated in FIG. 7 is an example, and the configurations may be combined or the configurations may be appropriately changed as necessary.

  As described above, according to the second embodiment, depth information is obtained at the time of image coding, and based on the obtained depth information, pseudo contour can be suppressed at the time of high compression or the compression rate can be improved at the time of low compression. .

[Example 3]
The image processing device (image decoding device) according to the third embodiment is a device that decodes the bitstream encoded by the image processing device 20 according to the second embodiment.

<Configuration>
FIG. 8 is a block diagram illustrating an example of a schematic configuration of the image processing apparatus 30 according to the third embodiment. As illustrated in FIG. 8, the image processing device 30 includes an entropy decoding unit 301, an inverse quantization unit 302, an inverse orthogonal transform unit 303, an intra prediction unit 304, a decoded information storage unit 305, and an inter prediction unit 306. A predicted image selection unit 307, a decoded image generation unit 308, a loop filter unit 310, and a frame memory 311. An outline of each part will be described below.

  When a bit stream is input, the entropy decoding unit 301 performs entropy decoding corresponding to the entropy encoding of the image processing device 20. The prediction error signal decoded by the entropy decoding unit 301 is output to the inverse quantization unit 302. In addition, information regarding modification of the quantization matrix used in the first or second embodiment, a difference value of a decoded motion vector when inter prediction is performed, and the like are output to the decoded information storage unit 305.

  In the case of intra prediction, the entropy decoding unit 301 notifies the intra prediction unit 304 to that effect. In addition, the entropy decoding unit 301 notifies the prediction image selection unit 307 whether the decoding target image is inter predicted or intra predicted.

  The inverse quantization unit 302 performs an inverse quantization process on the output signal from the entropy decoding unit 301 in consideration of information regarding modification of the quantization matrix used in the first or second embodiment as necessary. . The inversely quantized output signal is output to the inverse orthogonal transform unit 303.

  The inverse orthogonal transform unit 303 performs an inverse orthogonal transform process on the decoded block of the output signal from the inverse quantization unit 302 to generate a residual signal. The residual signal is output to the decoded image generation unit 308.

  The intra prediction unit 304 generates a prediction image using a plurality of prediction directions from the peripheral pixels already decoded of the decoding target image acquired from the frame memory 311.

  The decoding information storage unit 305 stores decoding information such as a division mode.

  The inter prediction unit 306 acquires the reference image data acquired from the frame memory 311 from the decoding information storage unit 305 and the difference value of the motion vector. The inter prediction unit 306 determines a prediction vector, adds the determined prediction vector and the difference value of the motion vector, and generates a motion vector. The inter prediction unit 306 performs motion compensation using the generated motion vector. Thereby, block data as a motion-compensated reference image is generated.

  The predicted image selection unit 307 selects either the intra predicted image or the inter predicted image. The selected block data is output to the decoded image generation unit 308.

  The decoded image generation unit 308 adds the predicted image output from the predicted image selection unit 307 and the residual signal output from the inverse orthogonal transform unit 303 to generate a decoded image. The generated decoded image is output to the loop filter unit 310.

  The loop filter unit 310 applies a filter for reducing block distortion to the decoded image output from the decoded image generation unit 308, and outputs the decoded image after the loop filter processing to the frame memory 311.

  Note that the decoded image after the loop filter is output to a display device or the like.

  The frame memory 311 stores a decoded image that serves as a reference image. The decoded information storage unit 305 and the frame memory 311 are configured separately, but they may be the same storage unit.

  As described above, according to the third embodiment, it is possible to appropriately decode the bitstream encoded by the image processing device 20 using the correction information related to the quantization matrix used at the time of encoding according to the second embodiment. .

[Example 4]
FIG. 9 is a block diagram illustrating an example of a schematic configuration of the image processing device 40 according to the fourth embodiment. An image processing apparatus 40 illustrated in FIG. 9 is an example of an apparatus in which the image processing described in the first to third embodiments is implemented by software.

  As shown in FIG. 9, the image processing apparatus 40 includes a control unit 401, a main storage unit 402, an auxiliary storage unit 403, a drive device 404, a network I / F unit 406, an input unit 407, and a display unit. 408. These components are connected to each other via a bus so as to be able to transmit and receive data.

  The control unit 401 is a CPU (Central Processing Unit) that controls each device, calculates data, and processes in a computer. The control unit 401 is an arithmetic device that executes an image processing program stored in the main storage unit 402 or the auxiliary storage unit 403. The control unit 401 receives data from the input unit 407 and the storage device, calculates and processes the data, and then outputs the data to the display unit 408 and the storage device.

  Also, the control unit 401 can realize the processing described in the first to fourth embodiments by executing an image processing program.

  The main storage unit 402 is a ROM (Read Only Memory), a RAM (Random Access Memory), or the like. The main storage unit 402 is a storage device that stores or temporarily stores programs and data such as OS (Operating System) and application software that are basic software executed by the control unit 401.

  The auxiliary storage unit 403 is an HDD (Hard Disk Drive) or the like, and is a storage device that stores data related to application software or the like.

  The drive device 404 reads the program from the recording medium 405, for example, a flexible disk, and installs it in the storage unit.

  A predetermined program is stored in the recording medium 405, and the program stored in the recording medium 405 is installed in the image processing apparatus 40 via the drive device 404. The installed predetermined program can be executed by the image processing apparatus 40.

  The network I / F unit 406 is a peripheral having a communication function connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network) constructed by a data transmission path such as a wired and / or wireless line. This is an interface between the device and the image processing apparatus 40.

  The input unit 407 includes a keyboard having cursor keys, numeric input, various function keys, and the like, and a mouse and a slide pad for performing key selection on the display screen of the display unit 408. The display unit 408 is configured by an LCD (Liquid Crystal Display) or the like, and performs display according to display data input from the control unit 401.

  The units of the image processing apparatus 10 in FIG. 2, the image processing apparatus 20 in FIG. 7, and the image processing apparatus 30 in FIG. 8 can be realized by, for example, the control unit 401 and the main storage unit 402 as a work memory.

  Further, the decoded image storage unit 210 illustrated in FIG. 7 is realized by, for example, the main storage unit 402 or the auxiliary storage unit 403, and the configuration other than the decoded image storage unit 210 illustrated in FIG. This can be realized by the main storage unit 402.

  Further, the decoded information storage unit 305 and the frame memory 311 illustrated in FIG. 8 can be realized by the main storage unit 402 or the auxiliary storage unit 403, for example. A configuration other than the decoded information storage unit 305 and the frame memory 311 illustrated in FIG. 8 can be realized by the control unit 401 and the main storage unit 402 as a work memory, for example.

  The program executed by the image processing apparatus 40 has a module configuration including each unit described in the first to third embodiments. As actual hardware, when the control unit 401 reads out and executes a program from the auxiliary storage unit 403, one or more of the above-described units are loaded onto the main storage unit 402, and one or more of the respective units are main. It is generated on the storage unit 402.

  As described above, the image processing described in the first to third embodiments can be realized as a program for causing a computer to execute the image processing. The processing described in the first to third embodiments can be realized by installing this program from a server or the like and causing the computer to execute the program.

  It is also possible to record the program in the recording medium 405 and cause the processing unit such as a computer or a portable terminal to read the recording medium 405 on which the program is recorded, thereby realizing the above-described image processing.

  Note that the recording medium 405 is a recording medium that records information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk, and information that is electrically stored such as a ROM or a flash memory. Various types of recording media such as a semiconductor memory for recording can be used.

  In addition, the image processing described in each embodiment described above can be implemented in one or a plurality of integrated circuits. Note that the image processing device 40 according to the fourth embodiment has a function as at least one of the image processing devices 10, 20, and 30 as described above.

  Further, the image processing apparatus in each of the embodiments described above can be applied to an encoding technique that outputs depth-of-field information using depth information or modifies a quantization matrix. H.264 / AVC and H.264 It is not limited to 265 / HEVC.

  Each embodiment has been described in detail above. However, the present invention is not limited to the specific embodiment, and various modifications and changes other than the above-described modification are possible within the scope described in the claims. .

10, 20, 30, 40 Image processing apparatus 11 Image processing unit (1)
111 Block division information storage unit 112 Prediction error signal information amount integration unit 113 Depth information clustering unit 114 Depth of field determination unit 115 Spatial frequency acquisition unit 116 Information amount integration unit after quantization 117 Compression rate determination unit 118 Preservation of depth of field Unit 130 1-pass / 2-pass switching instruction unit 160 quantization matrix correction unit

Claims (6)

An image processing apparatus that adjusts the quantization step of an image using information on the depth of the image,
A spatial frequency acquisition unit that acquires the power of each spatial frequency of the plurality of blocks included in the image;
A depth information clustering unit that clusters information on the depth of each pixel position or each predetermined region in the image, and determines each cluster of the plurality of blocks based on a result of the clustering;
The power of one or more spatial frequencies of the block belonging to a given cluster, based on the power of the spatial frequency of a plurality of blocks belonging to other than the given cluster, the one or more blocks belonging to the given cluster A depth-of-field determination unit that determines to belong to the depth of field;
Based on the determination result of the depth-of-field determination unit and the compression rate of the image, each quantization step of the plurality of frequency components orthogonally transformed with respect to a predetermined block among the plurality of blocks is performed. A quantization matrix correction unit that corrects the value of the quantization matrix including,
An image processing apparatus. A compression rate determination unit that determines a high compression rate when the compression rate of the image is higher than a predetermined first compression rate;
The depth of field determination unit
The ratio of the power of the spatial frequency exceeding a predetermined spatial frequency among the spatial frequencies of one or more blocks belonging to the predetermined cluster is the predetermined spatial frequency among the spatial frequencies of a plurality of blocks belonging to other than the predetermined cluster. Determining that the one or more blocks belonging to the predetermined cluster belong within a depth of field if the power ratio is higher than a spatial frequency power ratio exceeding
The quantization matrix correction unit, when the compression ratio is a high compression ratio, among the one or more blocks determined to belong to the depth of field, the ratio of the power of the low frequency component of the spatial frequency For a block that is higher than a predetermined percentage, the predetermined region of the quantization matrix is modified to a smaller value.
The image processing apparatus according to claim 1. The quantization matrix correcting unit applies the quantization to a block other than the one or more blocks determined to belong to the depth of field when the compression rate is lower than a predetermined second compression rate. Modify the value of the quantization matrix to a larger value,
The image processing apparatus according to claim 2.   In the block in which the ratio of the power of the low frequency component of the spatial frequency is higher than a predetermined ratio, a value obtained by dividing the power of the spatial frequency below the first threshold by the power of the spatial frequency exceeding the first threshold is the second. The image processing apparatus according to claim 2, wherein the image processing apparatus is a block exceeding a predetermined threshold value.   An encoding device comprising the image processing device according to any one of claims 1 to 4.   A program for causing a computer to function as the encoding device according to claim 5.

Images