JP2016071808A - Image processor, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the image quality of images after image processing to remove noises.SOLUTION: The image processor includes: a difference acquisition part; and a smoothing processing part. The difference acquisition part sequentially acquires a difference of pixel value as pixels of interest in a pair of input pixels with respect to each of plural input pixels which are disposed in a two dimensional grid, and which are in a point-symmetrical relationship with respect to the pixel of interest within a predetermined range of a certain distance from the pixel of interest. A smoothing processing part performs a smoothing processing, within the predetermined range, in which the smaller difference, the higher degree of smoothing.SELECTED DRAWING: Figure 3

Description

  The present technology relates to an image processing apparatus, an image processing method, and a program for causing a computer to execute the method. Specifically, the present invention relates to an image processing apparatus, an image processing method, and a program for causing a computer to execute the method.

  2. Description of the Related Art Conventionally, in an image processing apparatus such as an imaging apparatus, various noise removal filters have been used to improve image quality. For example, an imaging apparatus using a bilateral filter as a noise removal filter has been proposed (see, for example, Patent Document 1). In this bilateral filter, the larger the difference between the pixel values of the focused pixel of interest and the surrounding pixels around the focused pixel, the smaller the weight coefficient is calculated, and the pixel value is weighted and added by the weight coefficient.

JP 2006-180268 A

  However, in the above-described conventional technique, there is a possibility that bleeding or blurring occurs at the edge. For example, when the density gradient of the edge passing through the target pixel is small, the difference in pixel value between the peripheral pixel and the target pixel does not become so large, and as a result, the weighting factor for the peripheral pixel becomes relatively high, and the edge Bleeding and blurring will occur. In order to suppress the occurrence of blurring and blurring, the setting of the filter may be changed so that the weighting coefficient for the surrounding pixels is small. However, with such a setting, there is a possibility that the noise removal rate may be reduced. Therefore, with the above-described conventional technology, it is difficult to improve image quality while achieving both noise removal and suppression of blurring.

  The present technology has been created in view of such a situation, and an object thereof is to improve the image quality of an image subjected to image processing for removing noise.

  The present technology has been made to solve the above-described problems. The first aspect of the present technology is that a plurality of input pixels arranged in a two-dimensional grid are sequentially set as target pixels in order from the target pixel. A difference acquisition unit that obtains a difference between pixel values of a pair of input pixels that are point-symmetric with respect to the target pixel within a predetermined range of distance, and a smoothing process that has a higher degree of smoothing as the difference is smaller An image processing apparatus including a smoothing processing unit that performs the processing within the predetermined range, an image processing method thereof, and a program for causing a computer to execute the method. This brings about the effect that a smoothing process with a higher degree of smoothing is performed as the difference between the pixel values of the pair of input pixels located in point symmetry with respect to the target pixel is smaller.

  Further, in the first aspect, the difference acquisition unit obtains the difference and generates a peripheral pixel weight coefficient generation unit based on the difference using a smaller coefficient as the peripheral pixel weight coefficient as the difference is larger. A target pixel weight coefficient generation unit that generates a coefficient as a target pixel weight coefficient from the peripheral pixel weight coefficient, and the smoothing processing unit includes the pair of input pixels based on the peripheral pixel weight coefficient and the target pixel weight coefficient; You may perform the weighting addition with respect to the said attention pixel as the said smoothing process. This brings about the effect that the weighted addition for the pair of input pixels and the target pixel by the peripheral pixel weight coefficient and the target pixel weight coefficient is performed as a smoothing process.

  In the first aspect, the peripheral pixel weight coefficient generation unit may increase the coefficient as the peripheral pixel decreases as the difference increases and as the distance between the target pixel and one of the pair of input pixels decreases. It may be generated as a weighting coefficient. Accordingly, there is an effect that a larger coefficient is generated as a peripheral pixel weight coefficient as the difference is larger and smaller as the distance between the target pixel and one of the pair of input pixels is shorter.

  Further, in the first aspect, a wide-ranging smoothing processing unit that performs smoothing processing as wide-ranging smoothing processing within the range of a distance longer than the certain distance from the target pixel as the target pixel in order for each of the plurality of input pixels. The difference obtaining unit obtains a difference between the pixel values of the pair of input pixels after the wide range smoothing process, and the smoothing unit is configured to obtain the difference after the wide range smoothing process. The smoothing process may be performed within a predetermined range. This brings about the effect that the smoothing process is performed within a predetermined range after the wide area smoothing process.

  In the first aspect, the pixel value may include a luminance signal and a color difference signal. As a result, the smaller the difference between the pixel values including the luminance signal and the color difference signal, the more smoothing processing is performed with a higher degree of smoothing.

  In the first aspect, the pixel value may include a plurality of color information. As a result, the smaller the difference between pixel values including a plurality of color information is, the more smoothing processing is performed with a higher degree of smoothing.

  In this first aspect, a color information interpolation unit that interpolates missing color information among the plurality of color information in each of the plurality of input pixels having one of the plurality of color information as the pixel value. The weight coefficient generation unit may sequentially set each of the plurality of input pixels interpolated with the color information as a target pixel. This brings about the effect that the missing color information among the plurality of color information is interpolated in each of the plurality of input pixels.

  In addition, according to a second aspect of the present technology, an imaging unit that captures an image including a plurality of input pixels arranged in a two-dimensional grid, and a fixed pixel from the target pixel as the target pixel in order for each of the plurality of input pixels. A difference acquisition unit that obtains a difference between pixel values of a pair of input pixels that are point-symmetric with respect to the target pixel within a predetermined range of distance, and a smoothing process that has a higher degree of smoothing as the difference is smaller And a smoothing processing unit that performs the processing within the predetermined range. This brings about the effect that a smoothing process with a higher degree of smoothing is performed as the difference between the pixel values of the pair of input pixels located in point symmetry with respect to the target pixel is smaller.

  According to the present technology, it is possible to achieve an excellent effect that the image quality of an image subjected to image processing for removing noise can be improved. Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

1 is a block diagram illustrating a configuration example of an imaging apparatus according to a first embodiment. It is a block diagram showing an example of 1 composition of an image sensor in a 1st embodiment. It is a block diagram which shows one structural example of the image process part in 1st Embodiment. It is a figure which shows an example of the input image data in 1st Embodiment. It is a block diagram which shows the example of 1 structure of the weighting coefficient calculation part in 1st Embodiment. It is a block diagram which shows the example of 1 structure of the surrounding pixel weighting coefficient calculation part in 1st Embodiment. It is a block diagram which shows one structural example of the attention pixel weighting coefficient calculation part in 1st Embodiment. It is a block diagram which shows one structural example of the weighting addition part in 1st Embodiment. It is a graph which shows an example of the edge before and behind smoothing in a 1st embodiment. It is a graph which shows an example of the noise before and behind smoothing in a 1st embodiment. 3 is a flowchart illustrating an example of an operation of the imaging apparatus according to the first embodiment. It is a block diagram which shows one structural example of the image process part in 2nd Embodiment. It is a figure which shows an example of the weighting coefficient set to the low pass filter in 2nd Embodiment. 10 is a flowchart illustrating an example of an operation of the imaging apparatus according to the second embodiment. It is a block diagram which shows one structural example of the image pick-up element in 3rd Embodiment. It is a block diagram which shows one structural example of the image process part in 3rd Embodiment. It is a block diagram which shows one structural example of the demosaic process part in 3rd Embodiment. It is a figure which shows an example of the input image data in 3rd Embodiment. It is a figure which shows an example of the weighting coefficient set to the R low-pass filter, G low-pass filter, and B low-pass filter in 3rd Embodiment. It is a figure which shows an example of the weighting coefficient set to R periphery W low-pass filter, G periphery W low-pass filter, and B periphery W low-pass filter in 3rd Embodiment. It is a figure which shows an example of the image data after a demosaic process in 3rd Embodiment. It is a block diagram which shows the example of 1 structure of the weighting addition part in 3rd Embodiment. 14 is a flowchart illustrating an example of an operation of the imaging apparatus according to the third embodiment. It is a block diagram which shows the example of 1 structure of the image process part in the modification of 3rd Embodiment.

Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order.
1. First embodiment (an example of smoothing to a degree corresponding to the difference between a pair of pixels)
2. Second Embodiment (Example in which smoothing is performed to a degree corresponding to the difference between a pair of pixels after passing through a low-pass filter)
3. Third embodiment (smoothing to a degree corresponding to the difference between a pair of pixels after demosaic processing)
4). Modified example

<1. First Embodiment>
[Configuration example of imaging device]
FIG. 1 is a block diagram illustrating a configuration example of the imaging apparatus 100 according to the first embodiment. This imaging apparatus is an apparatus that captures an image, and includes an imaging lens 110, an imaging element 120, a control unit 130, a recording unit 140, and an image processing unit 200.

  The imaging lens 110 is a lens that collects light from the subject and guides it to the imaging device 120.

  The image sensor 120 converts the light from the imaging lens 110 into an electric signal according to a control signal, and generates image data. For example, each of the R (Red) pixel, the G (Green) pixel, the B (Blue) pixel, and the W (White) pixel is provided in the imaging element 120 in a two-dimensional lattice pattern. The image sensor 120 performs analog to digital (AD) conversion on an analog electrical signal photoelectrically converted by each of these pixels to generate image data. As the image sensor 120, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor is used. The image sensor 120 supplies the generated image data as input image data to the image processing unit 200 via the signal line 129. The imaging element 120 is an example of an imaging unit described in the claims. In addition, although the R pixel, the G pixel, the B pixel, and the W pixel are arranged on the image sensor 120, only the R pixel, the G pixel, and the B pixel may be arranged without providing the W pixel. The imaging element 120 is an example of an imaging unit described in the claims.

  The image processing unit 200 performs predetermined image processing including smoothing processing on input image data. In this smoothing process, the image processing unit 200 pays attention to the pixels in order according to the timing signal, and performs weighted addition within a range of a certain distance from the focused pixel (hereinafter referred to as “filter range”). The image processing unit 200 supplies the image data after the image processing to the recording unit 140 via the signal line 209 as output image data.

  The control unit 130 controls the entire imaging apparatus 100. The control unit 130 generates a control signal according to a user operation or the like, and supplies the control signal to the image sensor 120 via the signal line 139. This control signal includes, for example, a vertical synchronization signal. Further, the control unit 130 generates a timing signal indicating the timing at which each pixel is focused, and supplies the timing signal to the image processing unit 200 via the signal line 138. The recording unit 140 records output image data.

  Note that the imaging apparatus 100 may further include a display unit and display output image data on the display unit. The imaging device 100 may further include an interface, and output image data may be output to an external device via the interface.

  In addition, although the imaging lens 110, the imaging element 120, the control unit 130, the image processing unit 200, and the recording unit 140 are provided in the same device, these may be provided in separate devices. For example, the imaging device 120 or the like may be provided in the imaging device 100 and the image processing unit 200 may be provided in the information processing device or the like.

[Configuration example of image sensor]
FIG. 2 is a block diagram illustrating a configuration example of the image sensor 120 according to the first embodiment. The image sensor 120 includes a row scanning circuit 121, a pixel array unit 122, a timing control circuit 124, an AD conversion unit 125, a column scanning circuit 126, a demosaic processing unit 127, and a YUV conversion unit 128. The pixel array unit 122 is provided with a plurality of pixels 123 in a two-dimensional grid.

  The timing control circuit 124 controls the scanning timing of rows and columns. Here, a row is a pixel array unit 122 in which a plurality of pixels 123 are arranged in a certain direction, and is also called a line. The column is a pixel array unit 122 in which a plurality of pixels 123 are arranged in a direction orthogonal to the row.

  The timing control circuit 124 generates a horizontal synchronization signal HSYNC in synchronization with the vertical synchronization signal VSYNC from the control unit 130 and supplies it to the row scanning circuit 121. Further, the timing control circuit 124 generates a column scanning signal instructing the timing for scanning the column in synchronization with the horizontal synchronization signal HSYNC, and supplies it to the column scanning circuit 126.

  The row scanning circuit 121 sequentially selects each row (line) in synchronization with the horizontal synchronization signal HSYNC. The row scanning circuit 121 exposes the selected row over a set exposure period.

  The pixel 123 converts light into an electrical signal and outputs it as a pixel signal. The pixel 123 supplies the generated pixel signal to the AD conversion unit 125. Each pixel signal includes color information of any one of an R signal, a G signal, a B signal, and a W signal.

  The AD conversion unit 125 performs pixel conversion on the pixel signal to generate pixel data. The AD conversion unit 125 is provided for each column. The AD conversion unit 125 of the column selected by the column scanning circuit 126 supplies the generated pixel data to the demosaic processing unit 127. The column scanning circuit 126 selects each column according to the control of the timing control circuit 124.

  The demosaic processing unit 127 performs demosaic processing for interpolating a missing signal from the surrounding pixels among the R signal, the G signal, and the B signal for each pixel. The demosaic processing unit 127 supplies each of the interpolated pixel data to the YUV conversion unit 128.

The YUV conversion unit 128 performs YUV conversion processing for converting each of the R signal, the G signal, and the B signal into a luminance signal Y and color difference signals U and V for each pixel. The YUV conversion process is performed by the following equation, for example. The YUV conversion unit 128 supplies image data including pixel data after YUV conversion to the image processing unit 200 as input image data.

  Note that although the AD conversion unit 125, the demosaic processing unit 127, and the YUV conversion unit 128 are all provided in the image sensor 120, at least some of them may be provided outside the image sensor 120.

  The image sensor 120 is configured to perform YUV conversion. However, the image sensor 120 may be configured to supply the image data after demosaic processing to the image processing unit 200 as it is without performing YUV conversion.

  FIG. 3 is a block diagram illustrating a configuration example of the image processing unit 200 according to the first embodiment. The image processing unit 200 includes a data buffer 210, a target pixel selection unit 220, a weight coefficient calculation unit 230, and a weighting addition unit 260.

  The data buffer 210 holds a plurality of pixel data. For example, when the weighted addition unit 260 performs weighted addition within a 3 × 3 filter range centering on the pixel of interest, pixel data for at least three lines is held. Note that the filter range is not limited to the 3 × 3 range as long as it is within a certain distance from the target pixel, and may be, for example, a 5 × 5 range.

  The target pixel selection unit 220 sequentially selects each pixel in the input image data according to the timing signal. The selected pixel is hereinafter referred to as “target pixel”. The target pixel selection unit 220 supplies the coordinates of the selected target pixel as the target coordinates (x, y) to the weighting coefficient calculation unit 230 and the weighting addition unit 260 via the signal line 229.

  The weighting factor calculation unit 230 calculates a weighting factor for each of the pixels within a filter range of a certain distance from the target pixel. In the filter range, pixels other than the target pixel are hereinafter referred to as “peripheral pixels”. The weighting factor calculation unit 230 reads out the pixel values of the pair of neighboring pixels that are point-symmetric with respect to the target pixel in the filter range from the data buffer 210, and obtains a difference between them. Then, the weighting factor calculation unit 230 calculates a larger weighting factor as the surrounding pixel weighting factor as the difference is smaller. In addition, the weight coefficient calculation unit 230 calculates a difference between the total value of the peripheral pixel weight coefficients and “1” as the target pixel weight coefficient. The weighting factor calculation unit 230 supplies each of the surrounding pixel weighting factor and the target pixel weighting factor to the weighting addition unit 260 via the signal line 239. The weighting factor calculation unit 230 is an example of a difference acquisition unit described in the claims.

  The weighted addition unit 260 performs weighted addition as the smoothing process. The weighted addition unit 260 performs weighted addition on the respective pixel values of the peripheral pixel and the target pixel using the peripheral pixel weight coefficient and the target pixel weight coefficient. The total number of pixels in the filter range is called the filter size or the number of taps. As described above, a larger coefficient is set for the peripheral pixel weighting coefficient as the difference between the pixel values of the pair of peripheral pixels is smaller. For this reason, the smoothing process with a higher degree of smoothing is performed as the difference is smaller. The weighted addition unit 260 outputs the addition result of the weighted addition to the recording unit 140 as the pixel value of the target coordinate (x, y) in the output image data. The weighting addition unit 260 is an example of a smoothing processing unit described in the claims.

  Note that the image processing unit 200 may further perform various types of image processing such as white balance processing and gamma correction processing in addition to the smoothing processing (weighted addition).

  FIG. 4 is a diagram illustrating an example of input image data according to the first embodiment. In this input image data, a plurality of pixel data each including a luminance signal Y and color difference signals U and V are arranged. In the figure, pixels surrounded by a thick solid line are examples of a target pixel, and a range surrounded by a one-dot chain line is an example of a filter range in which weighted addition is performed. As illustrated in the figure, the image processing unit 200 performs weighted addition within a 3 × 3 filter range centering on the target pixel.

[Configuration example of weighting factor calculation unit]
FIG. 5 is a block diagram illustrating a configuration example of the weighting factor calculation unit 230 according to the first embodiment. The weight coefficient calculation unit 230 includes a peripheral pixel weight coefficient calculation unit 240 and a target pixel weight coefficient calculation unit 250.

The peripheral pixel weight coefficient calculation unit 240 calculates a peripheral pixel weight coefficient. The peripheral pixel weight coefficient calculation unit 240 reads out pixel data of each peripheral pixel from the data buffer 210. These pixel data include color difference signals U and V, respectively. Then, the peripheral pixel weight coefficient calculation unit 240 calculates a coefficient wgh using the following equation from the color difference signals of the pair of peripheral pixels that are point-symmetric with respect to the target pixel.

  In the above equation, u is one color difference signal U of the pair of peripheral pixels, and u ′ is the other color difference signal U of the pair of peripheral pixels. v is the color difference signal V of one of the pair of peripheral pixels, and v ′ is the color difference signal V of the other of the pair of peripheral pixels. For example, when the color difference signals of the pair of peripheral pixels are U (x−1, y) and U (x + 1, y), U (x−1, y) is set in u, and U (x + 1) is set in u ′. , Y) is set. K is a fixed parameter indicating how severely the similarity of pixel values is determined when calculating the coefficient wght. As k increases, the coefficient increases and the degree of smoothing increases. On the other hand, the smaller the k, the smaller the coefficient and the lower the degree of smoothing. Although the fixed value is set to k, the variance of the pixel values of the peripheral pixels may be set to k.

Then, the surrounding pixel weighting coefficient calculation unit 240 calculates the surrounding pixel weighting coefficient by multiplying the coefficient wght obtained by Equation 1 by a smaller predetermined coefficient as the distance between the target pixel and the surrounding pixel is longer. For example, when the Euclidean distance between the target pixel and one of the pair of surrounding pixels is “1”, the coefficient “1/8 (= 0.125)” is multiplied by wght. When the Euclidean distance is “2 ^1/2 ”, the coefficient “1/16 (= 0.0625)” is multiplied by wght.

  Here, when performing weighted addition within the 3 × 3 filter range, the peripheral pixel weighting coefficient is calculated for each of the pixel pairs located above, below, left, and right of the target pixel. The peripheral pixel weighting factors of the upper and lower pixel pairs of the target pixel are hereinafter referred to as “W1”, and the peripheral pixel weighting factors of the left and right pixel pairs are hereinafter referred to as “W2”. In addition, there are two pairs of pixels in the oblique direction. One of the peripheral pixel weighting coefficients is hereinafter referred to as “W3” and the other as “W4”. The surrounding pixel weight coefficient calculation unit 240 supplies each of these surrounding pixel weight coefficients to the target pixel weight coefficient calculation unit 250 and the weighting addition unit 260.

  The target pixel weight coefficient calculation unit 250 calculates the target pixel weight coefficient W0 from the peripheral pixel weight coefficients. The target pixel weight coefficient calculation unit 250 calculates a difference between the total value of the peripheral pixel weight coefficients and “1” as the target pixel weight coefficient W0. The same peripheral pixel weighting coefficient is applied to a pair of peripheral pixels that are point-symmetric with respect to the target pixel. For this reason, when the surrounding pixel weight coefficients of W1, W2, W3, and W4 are obtained, the pixel weight coefficient calculation unit 250 of interest adds each of them by doubling each other, and the difference between the added value and “1” is calculated. Find it. The target pixel weight coefficient calculation unit 250 supplies the calculated target pixel weight coefficient W0 to the weighting addition unit 260.

  Note that the weighting factor calculation unit 230 calculates the weighting factor by calculation, but is not limited to this configuration as long as the weighting factor can be generated. For example, for each possible value of the difference, the peripheral pixel weighting coefficient and the target pixel weighting coefficient are calculated in advance using Equation 1 and held in a table, and the weighting coefficient calculation unit 230 corresponds to the difference from the table. It is good also as a structure which reads a weighting coefficient.

[Configuration example of peripheral pixel weight coefficient calculation unit]
FIG. 6 is a block diagram illustrating a configuration example of the peripheral pixel weight coefficient calculation unit 240 according to the first embodiment. The peripheral pixel weighting factor calculating unit 240 includes a left / right paired weighting factor calculating unit 241, an upper / lower pairing weighting factor calculating unit 242, an oblique paired weighting factor calculating unit 243 and 244, and multipliers 245, 246, 247 and 248. Prepare.

  The left / right pair weight coefficient calculation unit 241 calculates a coefficient wght for the left and right peripheral pixel pairs of the target pixel. The left / right pair weight coefficient calculation unit 241 reads the left pixel value P (x−1, y) and the right pixel value P (x + 1, y) of the target pixel (x, y) from the data buffer 210. These pixel values include color difference signals U and V, respectively. That is, the pixel values of U (x−1, y), U (x + 1, y), V (x−1, y), and V (x + 1, y) are read out. The left / right pair weight coefficient calculation unit 241 calculates the coefficient wght from these pixel values using Equation 1 and supplies the coefficient wght to the multiplier 245.

  The upper / lower pair weight coefficient calculation unit 242 calculates the coefficient wght for the upper and lower peripheral pixel pairs of the target pixel. The upper / lower paired weight coefficient calculation unit 242 reads out the pixel value P (x, y−1) above the target pixel (x, y) and the lower pixel value P (x, y + 1) from the data buffer 210. The upper / lower pair weight coefficient calculation unit 242 calculates the coefficient wght from these pixel values using Equation 1 and supplies the coefficient wght to the multiplier 246.

  The diagonal pair weight coefficient calculation unit 243 calculates a coefficient wgh for the peripheral pixel pairs at the upper right and lower left of the target pixel. The diagonal pair weight coefficient calculation unit 243 obtains the upper right pixel value P (x + 1, y−1) and the lower left pixel value P (x−1, y + 1) of the target pixel (x, y) from the data buffer 210. read out. The diagonal pair weight coefficient calculation unit 243 calculates the coefficient wght from these pixel values using Expression 1 and supplies the coefficient wght to the multiplier 247.

  The diagonal pair weight coefficient calculation unit 244 calculates the coefficient wght for the peripheral pixel pairs at the upper left and lower right of the target pixel. The diagonal pair weight coefficient calculation unit 244 uses the upper left pixel value P (x−1, y−1) and the lower right pixel value P (x + 1, y + 1) of the target pixel (x, y) as the data buffer 210. Read from. The diagonal pair weight coefficient calculation unit 244 calculates the coefficient wght from these pixel values using Equation 1 and supplies the coefficient wght to the multiplier 248.

  The multiplier 245 multiplies the coefficient wght by a coefficient “0.125” corresponding to the distance. The multiplier 245 multiplies wght from the left / right paired weight coefficient calculation unit 241 by “0.125” and outputs the result as a peripheral pixel weight coefficient W1.

  The multiplier 246 multiplies the coefficient wght by a coefficient “0.125” corresponding to the distance. The multiplier 246 multiplies wght from the upper / lower pair weight coefficient calculation unit 242 by “0.125” and outputs the result as a peripheral pixel weight coefficient W2.

  The multiplier 247 multiplies the coefficient wght by a coefficient “0.0625” corresponding to the distance. The multiplier 247 multiplies wght from the diagonal pair weight coefficient calculation unit 243 by “0.0625”, and outputs the result as a peripheral pixel weight coefficient W3.

  The multiplier 248 multiplies the coefficient wght by a coefficient “0.0625” corresponding to the distance. The multiplier 248 multiplies wght from the diagonal pair weight coefficient calculation unit 244 by “0.0625” and outputs the result as a peripheral pixel weight coefficient W4.

  As described above, the peripheral pixel weighting coefficient calculation unit 240 calculates the weighting coefficient for each pair of peripheral pixels, so that the calculation amount and the circuit scale are reduced as compared with a bilateral filter that calculates the weighting coefficient for each peripheral pixel. can do. For example, in the bilateral filter, when there are eight neighboring pixels, eight weighting factors must be calculated, whereas the neighboring pixel weighting factor calculating unit 240 needs to calculate only four of W1 to W4. .

[Configuration Example of Pixel Weight Factor Calculation Unit of Interest]
FIG. 7 is a block diagram illustrating a configuration example of the pixel-of-interest weight coefficient calculation unit 250 according to the first embodiment. The target pixel weight coefficient calculation unit 250 includes multipliers 251, 252, 253, and 254, an adder 255, and a subtracter 256.

  The multiplier 251 multiplies the peripheral pixel weighting coefficient W1 by “2”. The multiplier 251 supplies the multiplication result to the adder 255. The multiplier 252 multiplies the peripheral pixel weighting coefficient W2 by “2”. The multiplier 252 supplies the multiplication result to the adder 255. The multiplier 253 multiplies the peripheral pixel weighting coefficient W3 by “2”. The multiplier 253 supplies the multiplication result to the adder 255. The multiplier 254 multiplies the peripheral pixel weighting coefficient W4 by “2”. The multiplier 254 supplies the multiplication result to the adder 255.

  The adder 255 adds all the multiplication results of the multipliers 251 to 254. The adder 255 supplies the addition result to the subtracter 256.

  The subtracter 256 subtracts the addition result of the adder 255 from “1”. The subtracter 256 outputs the subtraction result as a target pixel weight coefficient W0.

  Thus, by calculating the difference between the sum of the peripheral pixel weighting factors and “1”, the division is performed so that the sum of all weighting factors becomes “1” and the sum of the weighting factors becomes “1”. There is no need to do it. As a result, the amount of calculation and the circuit scale can be reduced as compared with a low-pass filter that performs division so that the sum of the weighting coefficients becomes “1”.

[Configuration example of weighted addition unit]
FIG. 8 is a block diagram illustrating a configuration example of the weighted addition unit 260 in the first embodiment. The weighting addition unit 260 includes a reading unit 261, adders 262, 263, 264, 265 and 266, multipliers 267, 268, 269 and 270, and an addition unit 271.

  The reading unit 261 reads out the color difference signals of the target pixel and the peripheral pixels. The reading unit 261 supplies the color difference signals U (x−1, y) and the color difference signals U (x−1, y) of the left and right peripheral pixel pairs of the target pixel to the adder 262, and then The color difference signal V (x−1, y) and the color difference signal V (x−1, y) are supplied to the adder 262.

  Further, the reading unit 261 supplies the color difference signals U (x, y−1) and the color difference signals U (x, y + 1) of the upper and lower peripheral pixel pairs of the target pixel to the adder 263, and then The color difference signal V (x, y−1) and the color difference signal V (x, y + 1) are supplied to the adder 263.

  Further, the reading unit 261 supplies the adder 264 with the color difference signal U (x + 1, y−1) and the color difference signal U (x−1, y + 1) of each of the upper right and lower left peripheral pixel pairs of the target pixel. Next, the reading unit 261 supplies the color difference signal V (x + 1, y−1) and the color difference signal V (x−1, y + 1) to the adder 264.

  Further, the reading unit 261 supplies the adder 265 with the color difference signals U (x−1, y−1) and the color difference signals U (x + 1, y + 1) of the upper left and lower right peripheral pixel pairs of the target pixel. Next, the reading unit 261 supplies the color difference signal V (x−1, y−1) and the color difference signal V (x + 1, y + 1) to the adder 265.

  Further, the reading unit 261 supplies the color difference signals U (x, y) and V (x, y) of the target pixel to the multiplier 266 in order.

  The adder 262 adds the color difference signals of the left and right peripheral pixel pairs. The adder 262 supplies the addition result to the multiplier 267. The adder 263 adds the color difference signals of the upper and lower peripheral pixel pairs. The adder 263 supplies the addition result to the multiplier 268. The adder 264 adds the color difference signals of the upper right and lower left peripheral pixel pairs. The adder 264 supplies the addition result to the multiplier 269. The adder 265 adds the color difference signals of the upper left and lower right peripheral pixel pairs. The adder 265 supplies the addition result to the multiplier 270.

  The multiplier 267 multiplies the addition result of the adder 262 by the peripheral pixel weight coefficient W1. The multiplier 267 supplies the multiplication result to the adder 271. The multiplier 268 multiplies the addition result of the adder 263 by the peripheral pixel weight coefficient W2. The multiplier 268 supplies the multiplication result to the adder 271. The multiplier 269 multiplies the addition result of the adder 264 by the peripheral pixel weight coefficient W3. The multiplier 269 supplies the multiplication result to the adder 271. The multiplier 270 multiplies the addition result of the adder 265 by the peripheral pixel weight coefficient W4. The multiplier 270 supplies the multiplication result to the adder 271.

  The multiplier 266 multiplies the color difference signal of the target pixel by the target pixel weight coefficient W0. The multiplier 266 supplies the multiplication result to the adder 271.

  The adding unit 271 adds the multiplication results of the multipliers 266, 267, 268, 269, and 270. The adder 271 outputs the addition result as color difference signals U ′ and V ′ at coordinates (x, y) in the output image data. On the other hand, the luminance signal Y (x, y) in the input image data is output as it is as the luminance signal Y (x, y) in the output image data.

  FIG. 9 is a graph illustrating an example of edges before and after smoothing according to the first embodiment. In the figure, the vertical axis indicates pixel values, and the horizontal axis indicates coordinates. A black circle indicates a target pixel, and a white circle indicates a peripheral pixel. A in the same figure is a graph which shows an example of the edge before smoothing. B in the figure is a graph showing an example of an edge after smoothing using Expression 1. C in the same figure is a graph which shows an example of the edge after smoothing by a bilateral filter.

  As illustrated in a in FIG. 9, when the edge density gradient is gentle, the pixel value P (x, y) of the target pixel and the pixel values P (x−1, y) and P ( The respective differences from x + 1, y) are relatively small. Here, if a bilateral filter is used, the smaller the difference, the greater the weight from the difference between the pixel value P (x, y) and the pixel values P (x-1, y) and P (x + 1, y). A coefficient is calculated. For this reason, in the bilateral filter, the gentler the edge density gradient, the larger the weighting factor for the surrounding pixels and the greater the degree of smoothing. Therefore, as illustrated in c in the figure, the gradient is further reduced, and blurring or blurring may occur at the edge.

  On the other hand, in the image processing unit 200, the smaller the difference from the difference between the pixel value P (x−1, y) and the pixel value P (x + 1, y) of the peripheral pixel, the larger the peripheral pixel, A weighting factor is calculated. When the edge passes through the target pixel, the difference between the pixel values of the peripheral pixel pair is larger than the difference between the pixel values of the target pixel and the peripheral pixels. For this reason, the peripheral pixel weighting coefficient according to Equation 1 is smaller than the weighting coefficient in the bilateral filter, and the degree of smoothing is low. Therefore, as illustrated in FIG. 9c, the gradient of the edge is maintained even after smoothing, and the occurrence of blurring and blurring of the edge can be suppressed.

  FIG. 10 is a graph illustrating an example of noise before and after smoothing in the first embodiment. In the figure, the vertical axis indicates pixel values, and the horizontal axis indicates coordinates. A black circle indicates a target pixel, and a white circle indicates a peripheral pixel. A in the same figure is a graph which shows an example of the noise before smoothing. B in the figure is a graph showing an example of noise after smoothing.

  As illustrated in a in FIG. 10, when noise occurs in the target pixel, the difference between the pixel values of the peripheral pixel pair is relatively small, and the peripheral pixel weighting coefficient is large. Therefore, the degree of smoothing is increased, and noise is removed as illustrated in FIG.

  As illustrated in FIGS. 9 and 10, the image processing unit 200 can remove noise while suppressing the occurrence of edge blurring and blurring. Thereby, the image quality of output image data can be improved. In addition, since the weighting coefficients for the pair of point-symmetrical pixels are the same, the weighting coefficients have symmetry, and the filter configured by the weighted addition unit 260 has linear phase characteristics. This makes it difficult for distortion to occur at the edge.

[Operation example of imaging device]
FIG. 11 is a flowchart illustrating an example of the operation of the imaging apparatus 100 according to the first embodiment. This operation is started, for example, when a predetermined operation for imaging (pressing a shutter button or the like) is performed.

  The image sensor 120 in the imaging device 100 generates image data (step S901), and performs demosaic processing (step S902) and YUV conversion processing (step S903).

  Then, the image processing unit 200 in the imaging apparatus 100 selects a target pixel that has not been selected (step S905), and calculates a weighting coefficient from the difference between the pixel values of peripheral pixel pairs that are point-symmetric with respect to the target pixel. Calculate (step S906). In addition, the image processing unit 200 performs weighted addition on the pixel values of the target pixel and the peripheral pixels (step S907). The image processing unit 200 determines whether all the pixels have been selected (step S908). If all the pixels have not been selected (step S908: No), the image processing unit 200 returns to step S905. On the other hand, when all the pixels have been selected (step S908: Yes), the image processing unit 200 ends the imaging and image processing operations.

  As described above, according to the first embodiment of the present technology, the image processing unit 200 performs smoothing as the difference between the pixel values of the pair of pixels located in point symmetry with respect to the target pixel increases. In order to increase the degree, the degree of smoothing decreases when the edge passes through the target pixel. For this reason, it is possible to suppress the occurrence of bleeding and blurring at the edge, and to improve the image quality of the image data.

<2. Second Embodiment>
In the first embodiment, the weighting factor calculation unit 230 and the weighting addition unit 260 perform weighting addition within a filter range of a certain distance from the target pixel to remove noise. However, a low-pass filter that performs weighted addition in a wider filter range than the weighted addition unit 260 may be further added. By adding this low-pass filter, noise in a wider frequency band can be removed and the image quality can be improved. The image processing unit 200 according to the second embodiment is different from the first embodiment in that a low-pass filter that performs weighted addition in a wider filter range than the weighted addition unit 260 is added.

  FIG. 12 is a block diagram illustrating a configuration example of the image processing unit 200 according to the second embodiment. The image processing unit 200 according to the second embodiment is different from the first embodiment in that it further includes a low-pass filter 280 and a data buffer 290.

  The low-pass filter 280 performs weighted addition in a wider filter range than the weighted addition unit 260. By this wide-ranging smoothing process (weighted addition), noise in a wider frequency band than the weighted addition unit 260 is removed. The low-pass filter 280 reads the color difference signals U and V from the data buffer 210 and performs a smoothing process, and causes the data buffer 290 to hold the processed signals (U ′ and V ′) and the luminance signal Y. The low-pass filter 280 is an example of a wide-range smoothing processing unit described in the claims.

  In addition, the weighting coefficient calculation unit 230 and the weighting addition unit 260 of the second embodiment read pixel data from the data buffer 290.

  FIG. 13 is a diagram illustrating an example of weighting factors set in the low-pass filter 280 according to the second embodiment. In the figure, pixels surrounded by a thick solid line indicate the target pixel. The numerical value described in the pixel indicates a weighting factor.

The low pass filter 280 sequentially selects the pixels in the input image data as the target pixel. For each pixel in the 5 × 5 filter range centered on this pixel of interest, a smaller weight coefficient is set as the distance from the pixel of interest is longer. A weighting factor of “4” is multiplied for the pixel of interest and its nine pixels, top, bottom, left, right, top left, top right, bottom left, and bottom right. For a pixel whose distance from the target pixel is 2 or more and less than 2 × 2 ^1/2 , a weighting factor of “2” is multiplied. A weighting factor of “1” is set for the remaining pixels. The low-pass filter 280 multiplies the corresponding pixel values (U and V) by these coefficients, divides the total value of the multiplication results by “64”, and stores the result in the data buffer 290. The luminance signal Y is held in the data buffer 290 as it is without passing through the low-pass filter 280.

  FIG. 14 is a flowchart illustrating an example of the operation of the imaging apparatus 100 according to the second embodiment. The operation of the imaging apparatus 100 of the second embodiment is different from that of the first embodiment in that step S904 is further executed.

  After the YUV conversion process (step S903), the image processing unit 200 removes noise by the low-pass filter 280 (step S904), and performs the processes after step S905.

  As described above, according to the second embodiment of the present technology, the image processing unit 200 performs weighted addition in a wider range than the weighting addition unit 260 in the low-pass filter 280, and thus removes noise in a wider frequency band. Thus, the image quality of the image data can be improved.

<3. Third Embodiment>
In the first embodiment, the demosaic process is performed in the image sensor 120, but the demosaic process may be performed by the image processing unit 200. The image processing unit 200 according to the third embodiment is different from the first embodiment in that demosaic processing is further performed.

  FIG. 15 is a block diagram illustrating a configuration example of the image sensor 120 according to the third embodiment. The image sensor 120 according to the third embodiment is different from the first embodiment in that the demosaic processing unit 127 and the YUV conversion unit 128 are not provided. The AD conversion unit 125 according to the third embodiment supplies image data before demosaic to the image processing unit 200 as input image data.

  FIG. 16 is a block diagram illustrating a configuration example of the image processing unit 200 according to the third embodiment. The image processing unit 200 according to the third embodiment is different from the first embodiment in that it further includes a demosaic processing unit 300 and a data buffer 290.

  The demosaic processing unit 300 performs demosaic processing on the input image data. The demosaic processing unit 300 reads out pixel data from the data buffer 210, performs demosaic processing, and stores the processed image data in the data buffer 290. The demosaic processing unit 300 is an example of a color information interpolation unit described in the claims.

In addition, the weighting coefficient calculation unit 230 and the weighting addition unit 260 of the third embodiment read pixel data from the data buffer 290. However, the weighting coefficient calculation unit 230 according to the third embodiment calculates the coefficient wght using the following expression instead of the expression 1.

  In the above equation, r is one r signal of the pair of peripheral pixels, and r ′ is the other r signal of the pair of peripheral pixels. g is a g signal of one of the pair of peripheral pixels, and g ′ is a g signal of the other of the pair of peripheral pixels. Further, b is one b signal of the pair of peripheral pixels, and b ′ is the other b signal of the pair of peripheral pixels.

  Note that the image processing unit 200 may further execute YUV conversion processing. In this case, the image processing unit 200 is further provided with a YUV conversion processing unit that performs YUV conversion on the image data after the demosaic processing.

  FIG. 17 is a block diagram illustrating a configuration example of the demosaic processing unit 300 according to the third embodiment. The demosaic processing unit 300 includes an R interpolation filter 301, a G interpolation filter 302, a B interpolation filter 303, and W interpolation filters 304, 305, and 306. The demosaic processing unit 300 includes subtracters 307, 308, and 309.

  The R interpolation filter 301 sequentially selects a plurality of pixels as target pixels, and interpolates R signals at the target pixels. For example, the R interpolation filter 301 performs weighted addition with a coefficient corresponding to the distance for each of the R signals within the 5 × 5 filter range centering on the target pixel. The R interpolation filter 301 supplies the addition result to the subtracter 307 as an R ′ signal.

  The G interpolation filter 302 interpolates the G signal in the target pixel. For example, the G interpolation filter 302 performs weighted addition with a coefficient corresponding to the distance for each of the G signals in the 5 × 5 filter range centering on the target pixel. The G interpolation filter 302 supplies the addition result to the subtracter 308 as a G ′ signal.

  The B interpolation filter 303 interpolates the B signal in the target pixel. For example, the B interpolation filter 303 performs weighted addition with a coefficient corresponding to the distance for each of the B signals in the 5 × 5 filter range centering on the target pixel. The B interpolation filter 303 supplies the addition result to the subtractor 309 as a B ′ signal.

  The W interpolation filter 304 interpolates the W signal for the target pixel from the upper, lower, left, and right W pixels of the R pixel within a certain range from the target pixel. For example, the W interpolation filter 304 performs weighted addition using a coefficient corresponding to the distance to each of the average values of the upper, lower, left, and right W signals of the R signal within the 5 × 5 filter range with the target pixel as the center. The W interpolation filter 304 supplies the addition result to the subtracter 307 as a W ′ signal.

  The W interpolation filter 305 interpolates the W signal for the target pixel from the upper, lower, left, and right W pixels of the G pixel within a certain range from the target pixel. For example, the W interpolation filter 305 performs weighted addition with a coefficient corresponding to the distance to each of the average values of the upper, lower, left, and right W signals of the G signal within the 5 × 5 filter range with the target pixel as the center. The W interpolation filter 305 supplies the addition result to the subtracter 308 as a W ′ signal.

  The W interpolation filter 306 interpolates the W signal for the target pixel from the upper, lower, left, and right W pixels of the B pixel within a certain range from the target pixel. For example, the W interpolation filter 306 performs weighted addition with a coefficient corresponding to the distance to each of the average values of the upper, lower, left, and right W signals of the B signal within the 5 × 5 filter range with the target pixel as the center. The W interpolation filter 306 supplies the addition result to the subtracter 309 as a W ′ signal.

  The subtractor 307 subtracts the W ′ signal from the W interpolation filter 304 from the R ′ signal from the R interpolation filter 301. The subtracter 307 outputs the subtraction result as an r signal. The subtracter 308 subtracts the W ′ signal from the W interpolation filter 305 from the G ′ signal from the G interpolation filter 302. The subtracter 308 outputs the subtraction result as a g signal. The subtractor 309 subtracts the W ′ signal from the W interpolation filter 306 from the B ′ signal from the B interpolation filter 303. The subtractor 309 outputs the subtraction result as a b signal.

  FIG. 18 is a diagram illustrating an example of the input image data 530 according to the third embodiment. As illustrated in the figure, in the input image data, a plurality of pixels whose pixel values are any of color information of R signal, G signal, B signal, and W signal are arranged in a two-dimensional lattice pattern.

  As described above, when demosaicing image data including W pixels, brightness is improved as compared to demosaicing Bayer array image data not including W pixels, but color noise and blurring are reduced due to less color information. Tends to occur. Generation of these noises can be effectively suppressed by the smoothing process according to Equation 1. Note that the image sensor 120 may generate Bayer array input image data, and the image processing unit 200 may demosaic the input image data.

  FIG. 19 is a diagram illustrating an example of weighting factors set in the R interpolation filter 301, the G interpolation filter 302, and the B interpolation filter 303 according to the third embodiment.

In FIG. 19, “a” is a diagram illustrating an example of a weighting coefficient set in the R interpolation filter 301. The R interpolation filter 301 sequentially selects pixels in the input image data as target pixels. For the R signal within the range of 5 × 5 with the target pixel as the center, a smaller weight coefficient is set as the distance from the target pixel is longer. The target pixel and the R pixels located in the upper, lower, left, right, upper left, upper right, lower left, and lower right are multiplied by a weighting factor of “4”. For R pixels whose distance from the target pixel is 2 or more and less than 2 × 2 ^1/2 , a weighting factor of “2” is multiplied. A weighting factor of “1” is set for the remaining R pixels. The R interpolation filter 301 multiplies the corresponding R signal by these coefficients, and generates an R ′ signal by dividing by a predetermined value so that the total value becomes “1”.

B in FIG. 19 is a diagram illustrating an example of a weighting coefficient set in the G interpolation filter 302. In this G interpolation filter 302, a smaller weighting factor is set for a G signal within a 5 × 5 range centering on the target pixel as the distance from the target pixel is longer. The target pixel and the G pixels located in the upper, lower, right, left, upper left, upper right, lower left, and lower right are multiplied by a weighting factor of “4”. For a G pixel whose distance from the target pixel is 2 or more and less than 2 × 2 ^1/2 , a weighting factor of “2” is multiplied. A weighting factor of “1” is set for the remaining G pixels. The G interpolation filter 302 multiplies the corresponding G signal by these coefficients and divides by a predetermined value so that the total value becomes “1” to generate a G ′ signal.

C in FIG. 19 is a diagram illustrating an example of a weighting factor set in the B interpolation filter 303. In the B interpolation filter 303, a smaller weighting factor is set for a B signal within a 5 × 5 range centering on the target pixel as the distance from the target pixel is longer. For the target pixel and its B pixels located at the top, bottom, left, right, top left, top right, bottom left and bottom right, a weighting factor of “4” is multiplied. For the B pixel whose distance from the target pixel is 2 or more and less than 2 × 2 ^1/2 , a weighting factor of “2” is multiplied. A weighting factor of “1” is set for the remaining B pixels. The B interpolation filter 303 multiplies the corresponding B signal by these coefficients and divides by a predetermined value so that the total value becomes “1” to generate a B ′ signal.

  FIG. 20 is a diagram illustrating an example of weighting factors set in the W interpolation filters 304, 305, and 306 according to the third embodiment.

In FIG. 20, “a” is a diagram illustrating an example of a weighting factor set in the W interpolation filter 304. In the W interpolation filter 304, a smaller weight coefficient is set for the upper, lower, left, and right W signals of the R pixel within the range of 5 × 5 around the target pixel as the distance from the target pixel of the R pixel is longer. The target pixel and the upper, lower, left, and right W pixels of the upper, lower, left, right, upper left, upper right, lower left, and lower right pixels are multiplied by a weighting factor of “4”. In this case, the filter range is a 7 × 7 range. In addition, a weighting factor of “2” is multiplied for the W pixels on the upper, lower, left, and right sides of the R pixel that is 2 or more and less than 2 × 2 ^1/2 from the target pixel. A weighting factor of “1” is set for the upper, lower, left, and right W pixels of the remaining R pixels. A hatched portion of a in the figure indicates an R pixel. The W interpolation filter 304 multiplies these coefficients by the average value of the corresponding W signal, and divides by a predetermined value so that the total value becomes “1” to generate a W ′ signal.

B in FIG. 20 is a diagram illustrating an example of a weighting factor set in the W interpolation filter 305. In this W interpolation filter 305, a smaller weighting factor is set for the upper, lower, left, and right W signals of the G pixel within the 5 × 5 range with the target pixel as the center, as the distance from the target pixel of the G pixel is longer. The target pixel and the upper, lower, left, and right W pixels of the G pixel that are in the upper, lower, left, right, upper left, upper right, lower left, and lower right are multiplied by a weighting factor of “4”. Further, a weighting factor of “2” is multiplied for the upper, lower, left, and right W pixels of the G pixel whose distance from the target pixel is 2 or more and less than 2 × 2 ^1/2 . A weighting factor of “1” is set for the upper, lower, left, and right W pixels of the remaining G pixels. A hatched portion of a in the figure indicates a G pixel. The W interpolation filter 305 multiplies these coefficients by the average value of the corresponding W signal, and divides by a predetermined value so that the total value becomes “1” to generate a W ′ signal.

20 is a diagram illustrating an example of a weighting factor set in the W interpolation filter 306. In the W interpolation filter 306, a lower weight coefficient is set as the distance from the target pixel of the B pixel to the upper, lower, left, and right W signals of the B pixel within the 5 × 5 range with the target pixel as the center is longer. A weighting factor of “4” is multiplied for the target pixel and the upper, lower, left, right, upper, lower left, upper right, lower left, and lower right B pixels of the B pixel. In addition, the W pixel on the top, bottom, left, and right of the B pixel whose distance from the target pixel is 2 or more and less than 2 × 2 ^1/2 is multiplied by a weighting factor of “2”. A weighting factor of “1” is set for the upper, lower, left, and right W pixels of the remaining B pixels. A hatched portion of a in the figure indicates a B pixel. The W interpolation filter 306 multiplies these coefficients by the average value of the corresponding W signal, and divides by a predetermined value so that the total value becomes “1” to generate a W ′ signal.

  FIG. 21 is a diagram illustrating an example of image data 540 after demosaic processing according to the third embodiment. As illustrated in the figure, as a result of interpolation of insufficient color information for each pixel by demosaic processing, image data 540 including an r signal, a g signal, and a b signal is generated for each pixel.

  FIG. 22 is a block diagram illustrating a configuration example of the weighted addition unit 260 according to the third embodiment. The reading unit 261 of the third embodiment is different from the first embodiment in that instead of the color difference signals U and V, the r signal, g signal, and b signal of the pixel of interest and the surrounding pixels are read.

  FIG. 23 is a flowchart illustrating an example of the operation of the imaging apparatus 100 according to the third embodiment. The operation of the imaging apparatus 100 of the third embodiment is different from that of the first embodiment in that the image processing unit 200 executes the demosaic process (step S902) and does not execute the YUV conversion process (step S903).

  As described above, according to the third embodiment of the present technology, the image processing unit 200 further performs the demosaic process, and thus it is not necessary for the image sensor 120 to perform the demosaic process. Thereby, the circuit scale of the image sensor 120 can be reduced.

[Modification]
In the third embodiment, the image processing unit 200 outputs the image data after the demosaic process and the smoothing process as it is, but takes out the high-frequency component removed by the demosaic process, The high frequency component may be restored. This further improves the image quality of the image data. The image processing unit 200 according to the modified example is different from the first embodiment in that the high frequency component is restored.

  FIG. 24 is a block diagram illustrating a configuration example of the image processing unit 200 according to the modification of the third embodiment. The image processing unit 200 according to the modified example is different from the third embodiment in that a W interpolation filter 310 and a high-frequency component restoration unit 320 are further provided.

  The W interpolation filter 310 is a filter that interpolates an insufficient W signal for each pixel with a filter having a smaller number of taps than the interpolation filter in the demosaic processing unit 300. For example, when an interpolation filter having a tap number of 7 × 7 is used in the demosaic processing unit 300, a W interpolation filter 310 having a tap number of 3 × 3 is provided. The W interpolation filter 310 supplies the interpolated W signal to the high frequency component restoration unit 320 as a high frequency component.

  The high frequency component restoration unit 320 restores a high frequency component in the image data after the smoothing process. The high frequency component restoration unit 320 synthesizes the high frequency component from the W interpolation filter 310 with the image data from the weighted addition unit 260 and outputs the resultant as output image data. Thereby, the high frequency component is restored.

  Thus, according to the modification, the high-frequency component removed by the demosaic process is restored after the smoothing process, so that the image quality of the image data can be further improved.

  The above-described embodiment shows an example for embodying the present technology, and the matters in the embodiment and the invention-specific matters in the claims have a corresponding relationship. Similarly, the invention specific matter in the claims and the matter in the embodiment of the present technology having the same name as this have a corresponding relationship. However, the present technology is not limited to the embodiment, and can be embodied by making various modifications to the embodiment without departing from the gist thereof.

Further, the processing procedure described in the above embodiment may be regarded as a method having a series of these procedures, and a program for causing a computer to execute these series of procedures or a recording medium storing the program. You may catch it. As this recording medium, for example, a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disc), a memory card, a Blu-ray disc (Blu-ray (registered trademark) Disc), or the like can be used.
Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

In addition, this technique can also take the following structures.
(1) A pair of input pixels that are point-symmetric with respect to the target pixel within a predetermined range from the target pixel as a target pixel in order for each of the plurality of input pixels arranged in a two-dimensional grid. A difference acquisition unit for obtaining a difference between the respective pixel values;
An image processing apparatus comprising: a smoothing processing unit that performs smoothing processing with a higher degree of smoothing within the predetermined range as the difference is smaller.
(2) The difference acquisition unit
A peripheral pixel weight coefficient generation unit that determines the difference and generates a smaller coefficient as a peripheral pixel weight coefficient from the difference as the difference is larger;
A target pixel weight coefficient generation unit that generates a coefficient for the target pixel from the peripheral pixel weight coefficient as a target pixel weight coefficient;
The image processing apparatus according to (1), wherein the smoothing processing unit performs weighted addition on the pair of input pixels and the target pixel based on the peripheral pixel weighting coefficient and the target pixel weighting coefficient as the smoothing process.
(3) The peripheral pixel weight coefficient generation unit generates a coefficient that is smaller as the difference is larger and larger as the distance between the target pixel and one of the pair of input pixels is shorter as the peripheral pixel weight coefficient. (2) The image processing apparatus according to (2).
(4) A wide-area smoothing processing unit that performs smoothing processing as wide-ranging smoothing processing within a range longer than the fixed distance from the target pixel as the target pixel in order for each of the plurality of input pixels,
The difference acquisition unit obtains a difference between the pixel values of the pair of input pixels after the wide area smoothing process,
The image processing apparatus according to any one of (1) to (3), wherein the smoothing processing unit performs the smoothing process within the predetermined range after the wide-area smoothing process.
(5) The image processing device according to any one of (1) to (4), wherein the pixel value includes a luminance signal and a color difference signal.
(6) The image processing device according to any one of (1) to (4), wherein the pixel value includes a plurality of pieces of color information.
(7) a color information interpolation unit that interpolates missing color information among the plurality of color information in each of the plurality of input pixels having one of a plurality of color information as the pixel value;
The image processing apparatus according to any one of (1) to (4), wherein the weight coefficient generation unit sequentially sets each of the plurality of input pixels interpolated with the color information as a target pixel.
(8) an imaging unit that captures an image including a plurality of input pixels arranged in a two-dimensional lattice;
Difference acquisition for obtaining a difference between respective pixel values of a pair of input pixels that are point-symmetric with respect to the target pixel within a predetermined distance from the target pixel as a target pixel in order for each of the plurality of input pixels. And
An imaging apparatus comprising: a smoothing processing unit that performs smoothing processing with a higher degree of smoothing within the predetermined range as the difference is smaller.
(9) The difference acquisition unit is in a point-symmetrical position with respect to the target pixel within a predetermined range of a certain distance from the target pixel as the target pixel in order for each of the plurality of input pixels arranged in a two-dimensional grid. A difference acquisition procedure for obtaining a difference between respective pixel values of the pair of input pixels;
An image processing method comprising: a smoothing processing unit that performs a smoothing process having a higher degree of smoothing within the predetermined range as the difference is smaller.
(10) The difference acquisition unit is in a point-symmetrical position with respect to the target pixel within a predetermined range of a certain distance from the target pixel as the target pixel in order for each of the plurality of input pixels arranged in a two-dimensional grid. A difference acquisition procedure for obtaining a difference between respective pixel values of the pair of input pixels;
A program for causing a computer to execute a smoothing processing unit that performs a smoothing process with a higher degree of smoothing within the predetermined range as the difference is smaller.

DESCRIPTION OF SYMBOLS 100 Image pick-up device 110 Imaging lens 120 Image pick-up element 121 Row scanning circuit 122 Pixel array part 123 Pixel 124 Timing control circuit 125 AD conversion part 126 Column scanning circuit 127 Demosaic processing part 128 YUV conversion part 130 Control part 140 Recording part 200 Image processing part 210 290 Data buffer 220 Target pixel selection unit 230 Weight coefficient calculation unit 240 Peripheral pixel weight coefficient calculation unit 241 Left / right pair weight coefficient calculation unit 242 Up / down pair weight coefficient calculation unit 243, 244 Diagonal pair weight coefficient calculation unit 245, 246, 247, 248, 251, 252, 253, 254, 266, 267, 268, 269, 270 Multiplier 250 Target pixel weight coefficient calculation unit 255, 262, 263, 264, 265 Adder 256, 307, 308, 309 Subtractor 260 Weight Only adding unit 261 reads unit 271 adding unit 280 lowpass filter 300 demosaic processing unit 301 R interpolation filter 302 G interpolation filter 303 B interpolation filters 304, 305, 306 W interpolation filter 310 W interpolation filter 320 the high frequency component restoring portion

Claims (10)

Each of a plurality of input pixels arranged in a two-dimensional grid as a target pixel in order, as a target pixel, each of a pair of input pixels that are point-symmetric with respect to the target pixel within a predetermined distance from the target pixel A difference acquisition unit for obtaining a difference between values;
An image processing apparatus comprising: a smoothing processing unit that performs smoothing processing with a higher degree of smoothing within the predetermined range as the difference is smaller. The difference acquisition unit
A peripheral pixel weight coefficient generation unit that determines the difference and generates a smaller coefficient as a peripheral pixel weight coefficient from the difference as the difference is larger;
A target pixel weight coefficient generation unit that generates a coefficient for the target pixel from the peripheral pixel weight coefficient as a target pixel weight coefficient;
The image processing apparatus according to claim 1, wherein the smoothing processing unit performs weighted addition on the pair of input pixels and the target pixel based on the peripheral pixel weight coefficient and the target pixel weight coefficient as the smoothing process. The peripheral pixel weight coefficient generation unit generates a coefficient that is smaller as the difference is larger and larger as the distance between the target pixel and one of the pair of input pixels is shorter as the peripheral pixel weight coefficient. Image processing apparatus. A wide range smoothing processing unit that performs smoothing processing as a wide range smoothing process within a range of a distance longer than the fixed distance from the target pixel as a target pixel in order for each of the plurality of input pixels,
The difference acquisition unit obtains a difference between the pixel values of the pair of input pixels after the wide area smoothing process,
The image processing apparatus according to claim 1, wherein the smoothing processing unit performs the smoothing process within the predetermined range after the wide area smoothing process. The image processing apparatus according to claim 1, wherein the pixel value includes a luminance signal and a color difference signal. The image processing apparatus according to claim 1, wherein the pixel value includes a plurality of pieces of color information. A color information interpolation unit that interpolates missing color information among the plurality of color information in each of the plurality of input pixels having one of a plurality of color information as the pixel value;
The image processing apparatus according to claim 1, wherein the weight coefficient generation unit sequentially sets each of the plurality of input pixels interpolated with the color information as a target pixel. An imaging unit that captures an image including a plurality of input pixels arranged in a two-dimensional lattice;
Difference acquisition for obtaining a difference between respective pixel values of a pair of input pixels that are point-symmetric with respect to the target pixel within a predetermined distance from the target pixel as a target pixel in order for each of the plurality of input pixels. And
An imaging apparatus comprising: a smoothing processing unit that performs smoothing processing with a higher degree of smoothing within the predetermined range as the difference is smaller. The difference acquisition unit sequentially inputs a pair of inputs that are point-symmetric with respect to the target pixel within a predetermined range from the target pixel as the target pixel for each of the plurality of input pixels arranged in a two-dimensional grid. A difference acquisition procedure for obtaining a difference between respective pixel values of the pixels;
An image processing method comprising: a smoothing processing unit that performs a smoothing process having a higher degree of smoothing within the predetermined range as the difference is smaller. The difference acquisition unit sequentially inputs a pair of inputs that are point-symmetric with respect to the target pixel within a predetermined range from the target pixel as the target pixel for each of the plurality of input pixels arranged in a two-dimensional grid. A difference acquisition procedure for obtaining a difference between respective pixel values of the pixels;
A program for causing a computer to execute a smoothing processing unit that performs a smoothing process with a higher degree of smoothing within the predetermined range as the difference is smaller.

Images