JP5705391B1 - Image processing apparatus and image processing method - Google Patents

Abstract

Provided are an image processing apparatus and an image processing method capable of reducing noise amplified by super-resolution processing while preserving edge components with a configuration in which circuit scale and memory capacity are suppressed. In the image processing apparatus of the present invention, the gradient direction detection unit 302 outputs a pixel in the direction with a small gradient among the pixels existing in the periphery for each pixel of the first image D101 as the minute gradient pixel position D302, and the smoothing unit 303. Takes each pixel of the second image D102 as a target pixel, and obtains each pixel value of the output image DOUT from a pixel existing around the target pixel by performing smoothing processing using the minute gradient pixel position D302. .

Description

  The present invention relates to an image processing apparatus and an image processing method for reducing noise in an image subjected to super-resolution processing.

  There is an imaging apparatus provided with a function (digital zoom function) for electronically zooming by cutting out and enlarging only a desired range of images from a captured image. In general digital zoom, image enlargement processing is performed using the bilinear method, bicubic method, etc., but in recent years, with the aim of generating a digital zoom image with higher resolution, high resolution that does not exist in the input image A technique called super-resolution processing that can generate the above information has been used.

  In such super-resolution processing, super-resolution processing using a learning database is known as a technique for further increasing the quality and resolution of each input image. This method refers to, for example, a database in which an example of a correspondence relationship between a high resolution image and a low resolution image is learned for each patch divided in an interpolated image. In this way, high resolution information that does not exist in the input image is predicted, and the resolution is increased (see, for example, Patent Document 1). The “patch” is a rectangular area composed of a plurality of pixels.

  As a problem of super-resolution processing using such a learning database, there is a low resistance to noise. That is, when noise is mixed in the patch of interest, the noise is also used in the database, so that the noise is sharpened (emphasized) as an image feature.

  Therefore, as a method for suppressing the influence of noise, there is a method of measuring the amount of noise for each partial region of an image and selecting a super-resolution processed image and a filtered image according to the amount of noise (for example, Patent Document 2). reference). A super-resolution processed image is selected when the noise amount is small in the high frequency region, and a filtered image is selected and output when the noise amount is large in the high frequency region.

JP 2003-018398 A (page 4, FIG. 1) WO2012 / 176367 (3rd page, Fig. 1)

  When super-resolution processing as in Patent Document 2 is performed, if the amount of noise is large, high-resolution information such as edges is not generated by the filter processing, and the effect of super-resolution is reduced.

  The present invention has been made to solve the above-described problems, and an image processing apparatus and an image processing method capable of reducing noise amplified by super-resolution processing while preserving edge components. Is to provide.

An image processing apparatus according to the present invention estimates an high-frequency component included in the initial interpolation enlarged image from the image enlargement unit that generates an initial interpolation enlarged image by interpolating between pixels of the input image, and the initial interpolation enlarged image. A super-resolution processing unit that generates a super-resolution processing image by synthesizing the initial interpolation enlarged image and the high-frequency component; and the initial interpolation enlarged image is input, and a pixel of interest in the initial interpolation enlarged image And a filter processing unit that performs a smoothing process on the super-resolution processing image with reference to position information of pixels in a direction in which a fluctuation amount between pixels in the vicinity is small.

  According to the present invention, for each pixel of the interpolated enlarged image, the smoothing process using the pixels in the direction with the small gradient is performed with reference to the position information of the pixels in the direction with the small gradient among the pixels existing in the vicinity. By this method, it is possible to reduce noise amplified by a prediction error of super-resolution processing while preserving edge components.

1 is a block diagram illustrating an image processing apparatus according to Embodiment 1. FIG. 3 is a block diagram illustrating a configuration example of a super-resolution processing unit according to Embodiment 1. FIG. It is a figure which shows the low-resolution enlarged image obtained by the expansion of an input image and an input image. It is a figure which shows the arrangement | sequence of the patch in the initial interpolation expansion image, and the pixel contained in each patch. It is a figure which shows the correspondence of the arrangement | sequence of the pixel in each patch of an initial stage interpolation expansion image, and the component of the column vector by the difference value ( dl1- dl25 ) about each pixel. It is a figure which shows the arrangement | sequence of the patch in a super-resolution process image, and the pixel contained in each patch. It is a figure which shows the arrangement | sequence of the pixel of the super-resolution process image corresponding to the component of a column vector by the feature-value ( dh1- dh9 ) of the super-resolution process image obtained by conversion. 3 is a block diagram illustrating a configuration example of a super-resolution processing unit according to Embodiment 1. FIG. 3 is a block diagram illustrating a configuration example 1 of a filter processing unit according to Embodiment 1. FIG. 4 is a block diagram illustrating a configuration example of a gradient direction detection unit according to Embodiment 1. FIG. 6 is a block diagram illustrating a configuration example 2 of the filter processing unit according to Embodiment 1. FIG. FIG. 10 is a flowchart illustrating a processing procedure in the image processing apparatus according to the second embodiment. FIG. 10 is a flowchart illustrating a procedure of an example of a super-resolution processing step in the image processing apparatus according to the second embodiment.

FIG. 1 is a block diagram showing an image processing apparatus 100 according to Embodiment 1 of the present invention. The image processing apparatus 100 according to the first embodiment includes an image enlargement unit 101 that generates an enlarged input image D in the initial interpolation enlarged image D101, from the initial interpolation enlarged image D101, included in the initial interpolation enlarged image D101 A super-resolution processing unit 102 that generates a super-resolution processing image D102 by estimating and synthesizing a high-frequency component that is not present, and a filter process that smoothes the super-resolution processing image D102 with reference to the initial interpolation enlarged image D101 Part 103. The image enlargement unit 101 outputs the initial interpolation enlarged image D101 to the super-resolution processing unit 102 and the filter processing unit 103. The super-resolution processing unit 102 outputs the super-resolution processing image D102 to the filter processing unit 103. The filter processing unit 103 outputs the output image Dout .

Image enlarging unit 101, an input image D in, and expands interpolating between pixels by a linear filter operation, such as a bilinear method or bicubic method. The image enlargement unit 101 outputs an initial interpolation enlarged image D101 obtained by enlargement such as digital zoom.

The super-resolution processing unit 102 performs super-resolution processing using the input image D in . The super-resolution processing unit 102 outputs a super-resolution processed image D102 obtained by performing the super-resolution processing. An example of the configuration of the super-resolution processing unit 102 will be described below.

  FIG. 2 is a block diagram showing the super-resolution processing unit 102. The super-resolution processing unit 102 separates the initial interpolation enlarged image D101 into a low-resolution feature component D201H and a non-feature component D201L for each patch that is a rectangular region, and a low-resolution feature component D201H. A high-resolution conversion unit 202 that converts a high-resolution feature component D202H, a feature-component synthesis unit 203 that generates a high-resolution patch D203 by synthesizing the high-resolution feature component D202H and the non-feature component D201L, A coefficient that stores a plurality of sets of coefficient data D205 respectively corresponding to a plurality of different patterns, and a patch averaging unit 204 that generates a super-resolution processed image D102 by averaging the values of one or more high-resolution patches And a data storage unit 205. Further, the high resolution conversion unit 202 uses the coefficient data D205 selected from the coefficient data storage unit 205 to convert the low resolution feature component D201H into the high resolution feature component D202H. The feature component separation unit 201 outputs the low-resolution feature component D201H to the high-resolution conversion unit 202. Also, the feature component separation unit 201 outputs the non-feature component D201L to the feature component synthesis unit 203. The coefficient data storage unit 205 outputs the coefficient data D205 to the high resolution conversion unit 202. The high resolution conversion unit 202 outputs the high resolution feature component D202H to the feature component synthesis unit 203. The feature component synthesis unit 203 outputs the high resolution patch D203 to the patch averaging unit 204. The patch averaging unit 204 outputs a super-resolution processed image D102.

Hereinafter, the process of each part is demonstrated in detail.
Image enlarging unit 101 may expand the input image D in shown in FIG. 3 (a), generates an initial interpolated enlarged image D101 shown in FIG. 3 (b). 3A and 3B, each of the circles represents one pixel. For example, the image is enlarged by the bicubic method. The enlargement ratio is, for example, 2 for each of the horizontal direction and the vertical direction. Of the pixels of the initial interpolation enlarged image D101 in FIG. 3B, the pixels indicated by black circles are the pixels at the same position as the pixels of the input image D in shown in FIG.

  The feature component separation unit 201 separates the initial interpolation enlarged image D101 (FIG. 3B) into a low-resolution feature component D201H and a non-feature component D201L for each pixel group of the patch that is the rectangular region.

The patches in the initial interpolation enlarged image D101 are formed so as to overlap each other. For example, as shown in FIG. 4, each patch LPc1, LPc2, LPc3, LPc4 is a rectangular area composed of NL × NL pixels. In FIG. 4, NL is 5. In this case, the pitch of the patch (the distance between the center of the patch and the center of the patch) is 2 pixels in both the horizontal direction and the vertical direction. Thereby, three pixels overlap in the horizontal direction between patches adjacent in the horizontal direction. In addition, three pixels overlap in the vertical direction between patches adjacent in the vertical direction.
The patch pitches LPPh and LPPv in the horizontal direction and the vertical direction are both 2. As a result, the pixels located at the center of the patch are arranged every other row and every other column.

  The feature component separation unit 201 obtains an average value lMean of NL × NL pixels for each of the patches LPc1, LPc2, LPc3, LPc4 composed of NL × NL pixels of the initial interpolation enlarged image D101. Further, the feature component separation unit 201 outputs the obtained average value lMean as the non-feature component D201L. Further, the feature component separation unit 201 uses NL × NL difference values obtained by subtracting the average value lMean from the pixel value of each pixel as a feature amount. Further, the feature component separation unit 201 outputs a column vector obtained by arranging the feature amounts in a predetermined order as a feature component D201H.

For example, the formation of a column vector is as follows. As shown in FIG. 5A, the pixels P (1) to P (25) are taken out in the order indicated by their numbers (numerical values in parentheses). Then, the extracted pixel difference values dl 1 to dl 25 are used as feature amounts. As shown in FIG. 5B, the feature amounts are arranged in order from the top.
As described above, the characteristic component D201H is obtained by subtracting the average value from the pixel value of each pixel. Therefore, it can be said that the feature component D201H is a high-frequency component of the patch. Further, it can be said that the non-feature component D201L is a low-frequency component of the patch.

  The feature component separation unit 201 separates or extracts only the feature component (high frequency component) used for prediction in the high resolution conversion unit 202 from the image components of the initial interpolation enlarged image D101. Thus, the feature component separation unit 201 normalizes the input signal to the high resolution conversion unit 202.

  The high resolution conversion unit 202 converts the low resolution feature component D201H into the high resolution feature component D202H. The coefficient data D205 selected from the coefficient data storage unit 205 is used for the conversion to the feature component D202H. This conversion is a conversion from the feature component of the patch of the low-resolution image to the feature component of the corresponding patch of the super-resolution processed image. Here, “corresponding” with respect to “position” means that the pixel position at the center of the patch of the high-resolution image is the same as the pixel position at the center of the patch of the initial interpolation enlarged image. In other words, it means that the positions match when the high-resolution image is superimposed on the initial interpolation enlarged image. A patch pair is composed of a patch of an initial interpolation enlarged image and a patch of a high resolution image having centers at positions corresponding to each other.

Patches definitive super resolution processing image D 1 0 2 is also formed as well as overlap each other and the patch in the initial interpolation enlarged image D101.
For example, as shown in FIG. 6, each patch HPc1, HPc2, HPc3, HPc4 is a rectangular area composed of, for example, NH × NH pixels. In FIG. 6, NH is 3. In this case, the patch pitch HPPh in the horizontal direction and the patch pitch HPPv in the vertical direction are both two pixels. Thereby, one pixel overlaps in the horizontal direction between patches adjacent in the horizontal direction. In addition, one pixel overlaps in the vertical direction between patches adjacent in the vertical direction.

In the example shown in FIG. 6, the pixels situated super resolution processing image D 1 0 2 to definitive patch HPc1, HPc2, HPc3, HPc4 center, in that it is arranged in and every other column in every other row, This is the same as the patches LPc1, LPc2, LPc3, LPc4 in the initial interpolation enlarged image D101. Also, patch HPc1 the definitive super resolution processing image D 1 0 2, HPc2, HPc3 , the center of HPc4 coincides with the center of the patch LPC1, LPC2, LPC3, LPC4 in the initial interpolation enlarged image D101.

  The coefficient data D205 gives the relationship between each patch of the high resolution image and the low resolution image and the corresponding patch of the high resolution image, and is supplied from the coefficient data storage unit 205.

Here, if the feature component D201H of each patch of the initial interpolation enlarged image is defined by a matrix M that is a column vector nlp, a column vector nhp, and coefficient data D205, the conversion by the high resolution conversion unit 202 is expressed by Expression (1). Is done. Note that the column vector nlp has (NL × NL) values, and the feature component values of the pixels constituting each patch of the initial interpolation enlarged image are dl1 to dl (NL × NL). The column vector nhp has (NH × NH) values, and the characteristic component values of the pixels constituting the corresponding patch of the high resolution image are dh1 to dh (NH × NH).
The matrix M is composed of (NH × NH) rows (NL × NL) columns of components.

ＮＬ＝５、ＮＨ＝３の場合について式（１）を、行列を用いて書き直すと式（２）のようになる。列ベクトルｎｌｐは、２５個の値をもち、初期補間拡大画像の各パッチを構成する画素の特徴成分値は、ｄｌ１〜ｄｌ２５である。また、列ベクトルｎｈｐは、９個の値をもち、高解像度画像の対応するパッチを構成する画素の特徴成分値は、ｄｈ１〜ｄｈ９である。

When NL = 5 and NH = 3, equation (1) can be rewritten using a matrix as equation (2). The column vector nlp has 25 values, and the characteristic component values of the pixels constituting each patch of the initial interpolation enlarged image are dl1 to dl25. The column vector nhp has nine values, and the characteristic component values of the pixels constituting the corresponding patch of the high resolution image are dh1 to dh9.

  For example, the left side of the expression (2) is obtained by taking out the pixels Q (1) to Q (9) shown in FIG. 7 in the order indicated by the numbers (numerical values in parentheses) and calculating their difference values dh1 to dh9. They are arranged in order from the top. However, each of the pixels Q (1) to Q (9) shown in FIG. 7 is a center 3 × 3 position among the pixels shown in FIG. 5A, that is, P (7), P ( 8), P (9), P (12), P (13), P (14), P (17), P (18), and P (19).

Equations (1) and (2) indicate that the value of each pixel of the high-resolution feature component D202H is given by weighted addition to all (NL × NL) pixels of the low-resolution feature component D201H. ing.
The matrix M is a matrix composed of the coefficient data D205 selected from the coefficient data storage unit 205. The coefficient data storage unit 205 selects one set of coefficient data from the plurality of sets of stored coefficient data D205 and supplies the selected coefficient data to the high resolution conversion unit 202. Each of a plurality of sets of coefficient data stored in the coefficient data storage unit 205 is a linear regression model that approximates the relationship between a pair of a low resolution image feature component D201H and a high resolution image feature component D202H generated from a teacher image. And obtained by performing learning in advance.

  Note that the conversion from the low-resolution feature component D201H to the high-resolution feature component D202H may be nonlinear. In that case, as the coefficient data D205, data giving a coefficient of a nonlinear function is used.

  The feature component synthesis unit 203 synthesizes the high resolution feature component D202H and the non-feature component D201L to generate a high resolution patch D203.

For example, the feature component synthesis unit 203 uses the average value lMean (= D201L) of (NL × NL) pixels of the patch of the initial interpolation enlarged image as each component (value for each pixel) of the high-resolution feature component D202H. Add to dh1 to dh9.
The feature component synthesizer 203 cancels the normalization performed by the feature component separator 201 by adding the non-feature component (low frequency component) lMean separated by the feature component separator 201 to the high-resolution feature component. It has.

  The synthesis by the feature component synthesis unit 203 is expressed by Expression (3).

式（３）におけるｈｐは、高解像度パッチＤ２０３を構成する画素の画素値を表すＮＨ×ＮＨ個の値ｈ１〜ｈ（ＮＨ×ＮＨ）をもつ列ベクトルである。また、式（３）におけるｎｈｐは、高解像度パッチの特徴成分Ｄ２０２Ｈを構成するＮＨ×ＮＨ個の画素の特徴成分値ｄｈ１〜ｄｈ（ＮＨ×ＮＨ）をもつ列ベクトルである。さらに、式（３）におけるｌＭｅａｎは、非特徴成分Ｄ２０１Ｌを表すスカラー値である。また、式（３）におけるｃは、高解像度の特徴成分（高周波数成分）に対するゲインを調整するスカラー値（定数）である。
ＮＨ＝３の場合について、式（３）を、行列を用いて書き直すと式（４）のようになる。

In Expression (3), hp is a column vector having NH × NH values h1 to h (NH × NH) representing pixel values of the pixels constituting the high resolution patch D203. In addition, nhp in Equation (3) is a column vector having feature component values dh1 to dh (NH × NH) of NH × NH pixels constituting the feature component D202H of the high resolution patch. Furthermore, lMean in Equation (3) is a scalar value representing the non-feature component D201L. In Expression (3), c is a scalar value (constant) for adjusting the gain for the high-resolution feature component (high frequency component).
When NH = 3, when Equation (3) is rewritten using a matrix, Equation (4) is obtained.

In Equation (3) or (4), the estimated result as learned can be obtained by setting c = 1.
By setting c > 1, it is possible to intentionally enhance (enhance) the high-resolution feature component (high frequency component) and enhance the resolution of the output image.
The calculation according to Equation (3) or Equation (4) is performed for each of the high resolution patches.

As described above, since the patches are formed so as to overlap each other, some pixels belong to a plurality of patches. In the example of FIG. 6, the pixel Pxc located at the center of each patch among the pixels of the high resolution image belongs to one patch HPc1. A pixel Pxa adjacent in the horizontal direction to the center pixel Pxc belongs to two patches HPc1 and HPc2. A pixel Pxb adjacent in the vertical direction to the center pixel Pxc belongs to two patches HPc1 and HPc3. A pixel Pxd that is adjacent to the central pixel Pxc in an oblique direction belongs to four patches HPc1, HPc2, HPc3, and HPc4.
For the pixels belonging to two patches, two calculation results (pixel values for the pixels constituting the characteristic component hp obtained for each patch) are obtained according to Equation (3) or Equation (4). For the pixels belonging to one patch, four calculation results are obtained according to Equation (3) or Equation (4). For a pixel belonging to only one patch, only one calculation result according to Expression (3) or Expression (4) is obtained.

The patch averaging unit 204 averages the values of one or more high-resolution patches (pixel values for the pixels constituting the characteristic component hp determined for each patch) for each pixel of the high-resolution image. A super-resolution processed image D102 is generated. That is, for pixels belonging to two patches, the values of two patches are averaged, for pixels belonging to four patches, the values of four patches are averaged, and for pixels belonging to only one patch, one The patch value is output as it is.

  It can be said that the calculation according to Expression (3) or Expression (4) is estimation of pixel value candidates using one or more high-resolution patches for each pixel of the super-resolution processed image D102. It can be said that the averaging of the values of one or more high-resolution patches is a process for obtaining an average (simple average or weighted average) of one or more pixel value candidates. By this averaging, a final output pixel value is obtained.

  The coefficient data storage unit 205 stores a plurality of coefficient data for each pattern that gives the correspondence between the high resolution image and the low resolution image. The pattern referred to here is a pattern of change in pixel value in each part in the image.

In the case where the low resolution features Ingredient D 201H entrained noise in high resolution conversion unit 202 in the super-resolution processing section 102 is inputted, mixed noise becomes part of the feature components. As a result, coefficient data different from the coefficient data originally desired to be referenced is referred to, and noise is amplified. The amplified noise is recognized as image degradation.

Further, data stored in the coefficient data storage section 205 in the super-resolution processing section 102 is obtained by the relationship between the characteristic components of the pair of characteristic components and a high-resolution image images of the low-resolution image is approximated by a linear regression model. When a feature component having a small learning amount is increased in resolution, a prediction error may be generated from an error included in the coefficient data due to calculation with the coefficient data that is an approximate value, and noise and ringing may occur.

  2 superimposes the high-resolution feature component D202H and the non-feature component D201L to generate a high-resolution patch D203. The non-feature component D201L is advantageous in terms of circuit scale or data transfer speed because the data amount of the low-resolution feature component D201H is reduced with respect to the initial interpolation enlarged image D101. However, with respect to the effect of enhancing the sense of resolution, the high-resolution patch D203 is generated by synthesizing the high-resolution feature component D202H and the initial interpolation enlarged image D101 as in the super-resolution processing unit 102B of FIG. Good. The feature component separation unit 201, the resolution enhancement conversion unit 202, the feature component synthesis unit 203, the patch averaging unit 204, and the coefficient data storage unit 205 in the super resolution processing unit 102B in FIG. The same operation as the corresponding block in 102 is performed.

Next, the filter processing unit 103 in the image processing apparatus 100 will be described. The filter processing unit 103 refers to the initial interpolation enlarged image D101 to smooth the super-resolution processed image D102. FIG. 9 is a block diagram illustrating a configuration example of the filter processing unit 103. The filter processing unit 103 includes a gradient direction detection unit 302 and a smoothing unit 303. The gradient direction detection unit 302 receives the initial interpolation enlarged image D101, and outputs to the smoothing unit 303 a minute gradient pixel position D302 that is position information of pixels in a direction with a small gradient. Smoothing unit 303 may further enter the super-resolution processing image D102, and outputs the output image D out.
The gradient direction detection unit 302 detects the position of a pixel with a small gradient among the pixels existing around each pixel of the initial interpolation enlarged image D101 and outputs it as a minute gradient pixel position. The smoothing unit 303 performs smoothing by selecting a predetermined number of pixels at the minute gradient pixel position D302 determined by the gradient direction detection unit 302 from each pixel of the initial interpolation enlarged image D101 and performing weighted addition. .

An example of the configuration of the gradient direction detection unit will be described below. FIG. 10 is a block diagram illustrating a configuration example of the gradient direction detection unit 302. The gradient direction detection unit 302 includes a difference absolute value calculation unit 401 and a pixel extraction unit 402. The difference absolute value calculation unit 401 receives the initial interpolation enlarged image D101 and outputs the difference absolute value D401 to the pixel extraction unit 402. The pixel extraction unit 402 outputs a minute gradient pixel position D302.
The difference absolute value calculation unit 401 calculates each pixel of the initial interpolation enlarged image D101 as a first pixel of interest, and calculates a difference absolute value D401 from the pixel of interest for pixels existing around the first pixel of interest.
The pixel extraction unit 402 extracts a predetermined number of pixels in order from the pixel having the smallest difference absolute value D401, and outputs the pixel position as the minute gradient pixel position D302.

  The difference absolute value calculation unit 401 calculates the difference absolute value D401 using each pixel (X, Y) of the initial interpolation enlarged image D101 as a target pixel. The difference absolute value D401 is an absolute value of the difference value of pixel luminance data. The difference absolute value D401 is calculated as the difference absolute values ABS1 to ABS8 between the pixel existing around the pixel of interest and the pixel of interest. In other words, the difference absolute value D401 is a gradient from the target pixel. An example of the calculation for obtaining the difference absolute value D401 is shown in Expression (5).

ここで、ＩＭＢＣ（Ｘ，Ｙ）は、（Ｘ，Ｙ）における初期補間拡大画像Ｄ１０１の画素値であり、ここでは例えば輝度データの値である。

Here, IMBC (X, Y) is a pixel value of the initial interpolation enlarged image D101 at (X, Y), and here is, for example, a value of luminance data.

The pixel extraction unit 402 includes N pixels in order from a pixel having a small difference absolute value D401 among pixels existing in the periphery for each pixel (X, Y) of the initial interpolation enlarged image D101, in other words, a pixel having a small gradient from the target pixel. The small gradient pixels (X ^1st , Y ^1st ) to (X ^Nth , Y ^Nth ) are obtained. N is, for example, 2. Minute gradient pixel when N is ^2, the ^{(X 1st, Y 1s t)} , (X 2nd, Y 2nd). Minute gradient pixel ^{(X 1st, Y 1st),} (X 2n d, Y 2nd) , in the above equation (5), when the sort ABS8 from ABS1 in ascending order, to a first and second in the ABS1 the ABS8 A coordinate giving a small value.

As the minute gradient pixel, when the target pixel (X, Y) exists on the edge in the image, the pixel existing in the edge direction is selected. Therefore, the edge information in the image is transmitted to the subsequent process. By this sort processing, the edge direction can be detected with a small amount of calculation.
The pixel extraction unit 402 outputs the coordinates of the minute gradient pixel as the minute gradient pixel position D302.

  The detection of the minute gradient pixel position is not limited to sorting the absolute difference values as described above. Of the pixels present around each pixel of the initial interpolation enlarged image D101, the positions of pixels having a small gradient up to a predetermined number N may be detected as minute gradient pixel positions.

The smoothing unit 303 performs a process of reducing noise included in the super-resolution processing image D102 and erroneously emphasized by the super-resolution processing unit 102. The noise erroneously emphasized by the super-resolution processing unit 102 is a prediction error of the super-resolution process. The smoothing unit 303 uses the minute gradient pixel position D302 obtained from the gradient direction detection unit 302 to perform the smoothing filter processing based on the sort order shown in Expression ( 6 ) on the super-resolution processing image D102. Do it. That is, the smoothing unit 303 uses each pixel of the super-resolution processed image D102 as the second target pixel, and is located at the pixel position determined by the pixel extraction unit 402 from the pixels existing around the second target pixel. A predetermined number of pixels are selected and weighted. By this operation, the smoothing unit 303 calculates the output image Dout by diffusing noise included in the super-resolution processed image D102 in the edge direction.

ここで、ｉｍＳＲ（Ｘ，Ｙ）は、（Ｘ，Ｙ）における超解像処理画像Ｄ１０２の画素値を表す。また、ｉｍＬＰＦ（Ｘ，Ｙ）は、（Ｘ，Ｙ）における出力画像Ｄｏｕｔの画素値を表す。平滑化部３０３では、定数ＬＰＦＳＴＲによって平滑化の強度を変化させることができる。なお、定数ＬＰＦＳＴＲは、０以上で２以下の実数である。

Here, imSR (X, Y) represents the pixel value of the super-resolution processed image D102 at (X, Y). Further, imLPF (X, Y) represents the pixel value of the output image D out in (X, Y). In the smoothing unit 303, the smoothing intensity can be changed by a constant LPFSTR. The constant LPFSTR is a real number not less than 0 and not more than 2.

The above is the description of the operation of the image processing apparatus 100 according to the first embodiment.
Next, effects of the image processing apparatus 100 according to the first embodiment will be described.
In the image processing apparatus according to the first embodiment, in the filter processing unit 103 that performs the super-resolution processing image after the super-resolution processing, the gradient direction detection unit 302 performs sorting processing and exists in the super-resolution processing image D102. The direction of the edge to be detected can be detected.

  As a result, since the smoothing unit 303 can perform the LPF process in the edge direction, the edge component of the super-resolution processing image predicted by the super-resolution processing unit as in the conventional LPF processing is prevented from being diffused. Therefore, it is possible to prevent the super-resolution effect from being reduced.

  In addition, since the sort process can detect the edge direction with a small amount of calculation, the circuit scale and the memory capacity can be suppressed in the hardware implementation.

  Also, the edge direction is detected without being affected by the prediction error of the super-resolution processing unit 102 due to noise mixed in the image by detecting the edge direction with reference to the initial interpolation enlarged image in the filter processing unit 103. I can do it.

  In addition, in the case of smoothing performed by a known LPF processing method (for example, a median filter), the high frequency components in the image are uniformly diffused as a smoothing method, so that edge components included in the high frequency components are also diffused. As a result, although the effect of super-resolution is reduced, the first embodiment makes it possible to preserve the edge component while diffusing noise by performing LPF processing in the edge direction.

  The gradient direction detection unit 302 may derive an edge component image by differential processing (for example, a Laplacian filter) and detect the gradient direction therefrom.

  In addition, the smoothing unit 303 is not limited to the process of Expression (6), and it is only necessary to perform the smoothing process using the pixel corresponding to the minute gradient pixel position D302 in the interpolation enlarged image.

Regarding the configuration of the filter processing unit 103, as a process for preventing the LPF process from being applied to the component of the initial interpolation enlarged image D101,
A subtraction unit 301 for obtaining the difference image D 3 01 from a difference between the super-resolution processing image D102 and the initial interpolation enlarged image D101,
An adder 304 that obtains an output image Dout by adding the value of a pixel at the same position of the output image of the smoothing unit to the value of each pixel of the initial interpolation enlarged image D101; May be added to A configuration example at the time of addition is shown in FIG. The filter processing unit 103 in FIG. 11 includes a subtraction unit 301, a gradient direction detection unit 302, a smoothing unit 303, and an addition unit 304. The subtraction unit 301 receives the initial interpolation enlarged image D101 and the super-resolution processed image D102, and outputs the subtraction result D301 to the smoothing unit 303. The gradient direction detection unit 302 receives the initial interpolation enlarged image D101 and outputs the minute gradient pixel position D302 to the smoothing unit 303. The smoothing unit 303 outputs the smoothing result D303 to the adding unit 304. Adding section 304 further inputs the initial interpolation enlarged image D101, and outputs the output image D out.

Subsequently, the operation of the image processing method according to the second embodiment will be described.
FIG. 12 illustrates a processing procedure in the image processing method according to the second embodiment. The operation of the image processing method of the second embodiment, first, in the image enlarging step S11, the input image D in expanded by the image expansion section 101 generates an initial interpolated enlarged image D101.

Next, in the super-resolution processing step S12, a super-resolution processing image (or super-resolution) that is a high frequency component is estimated from the initial interpolation enlarged image D101 by estimating a frequency component higher than the frequency component included in the initial interpolation enlarged image. (Enlarged image ) is generated.

  Finally, in the filter processing step S13, the initial interpolated enlarged image D101 is referred to and the super-resolution processed image D102 is smoothed.

FIG. 13 shows an example of a processing procedure in the super-resolution processing step S12. In the patch selection step S21, one patch is selected from the initial interpolation enlarged image D101. The selection of patches is performed in raster order, for example, in the order of images from top left to bottom right (from top to bottom, and from left to right at the same height position). Along with the selection of the patch, the corresponding local region in the input image D in is selected.

  Next, in step S22 to step S25, processing by the feature component separation unit 201, the coefficient data storage unit 205, the high resolution conversion unit 202, and the feature component synthesis unit 203 is performed for each patch of the initial interpolation enlarged image D101.

In the processing for each patch (from step S22 to step S25), first, in the feature component separation step S22, each patch is separated into a low-resolution feature component D201H and a non-feature component D201L.
In parallel with this, in the coefficient data selection step S2 3, selects and outputs the coefficient data corresponding to each patch from the coefficient data memory unit 205.

After step S22 and step S23, in the high resolution conversion step S24, the coefficient data D205 selected in step S23 is used for the feature component D201H of each patch of the low resolution enlarged image obtained in step S22. An operation is performed to convert the low-resolution feature component D201H to the high-resolution feature component D202H.
Next, in step S25, the non-feature component D201L separated in step S22 is combined with the high-resolution feature component D202H that is the output of step S25 to generate a high-resolution patch.

  Next, in determination step S26, it is determined whether or not the processing in steps S21 to S25 has been performed for all patches in the image. If there is a patch that has not yet been processed (NO in step S26), the process returns to step S21 to select the next patch.

  If the processing for all the patches has been completed in step S26, the process proceeds to patch averaging step S27, where the high-resolution patch values obtained by the processing for each patch for each pixel are averaged by the patch averaging unit 204, and the super solution is obtained. An image processed image D102 is generated.

  Even if all the patches in the image are not finished, the processing of step S27 may be performed on the pixel immediately after the processing on the patch to which the pixel belongs is completed. In this case, the process of steps S21 to S27 and the process of step S27 are performed in parallel.

  Part or all of the elements constituting the image processing apparatus or part or all of the processing in the image processing method can be realized by software, that is, by a programmed computer.

  As described above, according to the present invention, smoothing amplified by the super-resolution processing unit can be performed while preserving the edge component with a configuration suitable for hardware while suppressing the circuit scale and memory capacity. High image quality due to possible super-resolution can be used.

  Moreover, although the embodiments of the present invention have been described above, the present invention is not limited to these embodiments.

  DESCRIPTION OF SYMBOLS 100 Image processing apparatus, 101 Image expansion part, 102 Super-resolution processing part, 103 Filter processing part, 201 Feature component separation part, 202 High resolution conversion part, 203 Feature component synthesis part, 204 Patch averaging part, 205 Coefficient data Storage unit, 301 subtraction unit, 302 gradient direction detection unit, 303 smoothing unit, 304 addition unit, 401 difference absolute value detection unit, 402 pixel extraction unit.

Claims (9)

An image enlargement unit that interpolates between pixels of the input image and generates an initial interpolation enlarged image;
Super-resolution processing for estimating a high-frequency component included in the initial interpolation enlarged image from the initial interpolation enlarged image and generating a super-resolution processing image by combining the initial interpolation enlarged image and the high-frequency component And
The initial interpolation enlarged image is input, and the super-resolution processing image is referred to with reference to the position information of the pixel in the direction in which the amount of variation between the target pixel and the neighboring pixels in the initial interpolation enlarged image is small. An image processing apparatus comprising: a filter processing unit that performs a smoothing process. The variation amount is an absolute value of a difference between pixel data of the target pixel and pixel data of pixels existing around the target pixel,
The filter processing unit
A gradient direction detection unit that detects, as the gradient, a position of a pixel in a small gradient direction as a minute gradient pixel position among pixels existing around each pixel of the initial interpolation enlarged image with the variation amount as a gradient;
The image processing apparatus according to claim 1, further comprising: a smoothing unit that performs a smoothing process using a pixel at the minute gradient pixel position for each pixel of the super-resolution processing image. The gradient direction detector
A difference absolute value calculation unit that calculates a difference absolute value between a pixel existing around the first pixel of interest and the first pixel of interest using each pixel of the initial interpolation enlarged image as a first pixel of interest;
A pixel extraction unit that extracts a predetermined number of pixels in order from the pixel having the smallest difference absolute value and outputs the pixel position as the minute gradient pixel position;
The smoothing unit
Using each pixel of the super-resolution processed image as a second pixel of interest, selecting the predetermined number of pixels at the pixel position determined by the pixel extraction unit from pixels existing around the second pixel of interest The image processing apparatus according to claim 2, wherein smoothing is performed by weighted addition. The super-resolution processing unit performs super Zosho management based on the correspondence relationship between the low-resolution image and a high resolution image which has previously been learned from claim 1, characterized in that to obtain a super-resolution processing image 4. The image processing apparatus according to any one of items 3. The super-resolution processor
A coefficient data storage unit that stores coefficient data obtained by previously learning an example of a correspondence relationship between a low-resolution feature component and a high-resolution feature component;
Using said coefficient data, the image processing according the any one of claims 1 to 3, the characteristic components of low resolution and having a high resolution converter for converting the feature components of the high resolution apparatus. The super-resolution processing unit, for each a rectangular area comprising the input image from a plurality of pixels patches, super Zosho management based on the correspondence between the low-resolution image and a high resolution image which has previously been learned 4. The image processing apparatus according to claim 1, wherein a super-resolution processed image is obtained by averaging the obtained high-resolution patches. 5. The super-resolution processor
A feature component separation unit that separates the initial interpolation enlarged image into low-resolution feature components and non-feature components for each patch;
A high-resolution conversion unit that converts the low-resolution feature component into a high-resolution feature component;
A feature component synthesis unit that synthesizes the high resolution feature component and the low resolution non-feature component or the initial interpolation enlarged image to generate a high resolution patch;
4. The patch averaging unit according to claim 1, further comprising: a patch averaging unit that generates a super-resolution processing image by averaging values of one or more high-resolution patches for each pixel. 5. The image processing apparatus described. The super-resolution processor
A feature component separation unit that separates low resolution feature components for each patch of the initial interpolation enlarged image;
A high-resolution conversion unit that converts the low-resolution feature component into a high-resolution feature component;
A feature component synthesis unit that synthesizes the high resolution feature component and the initial interpolation enlarged image to generate a high resolution patch;
A patch averaging unit that generates a super-resolution image by averaging the values of one or more high-resolution patches for each pixel;
A coefficient data storage unit for storing coefficient data obtained by learning in advance an example of a correspondence relationship between the low-resolution feature component and the high-resolution feature component;
It said high resolution conversion unit uses the coefficient data stored in the coefficient data storage unit, the low-resolution characteristic component from claim 1, characterized in that converting the feature components of the high resolution of the 3 The image processing apparatus according to any one of the above. An image enlargement step of interpolating between pixels of the input image to generate an initial interpolation enlarged image;
Super-resolution processing for estimating a high-frequency component included in the initial interpolation enlarged image from the initial interpolation enlarged image and generating a super-resolution processing image by combining the initial interpolation enlarged image and the high-frequency component Steps,
The initial interpolation enlarged image is input, and the super-resolution processing image is referred to with reference to the position information of the pixel in the direction in which the amount of variation between the target pixel and the neighboring pixels in the initial interpolation enlarged image is small. An image processing method comprising: a filter processing step for performing a smoothing process.

Images