JP2012142747A - Signal processing apparatus, signal processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a signal processing apparatus, a signal processing method, and a program capable of suppressing an amount of operation of signal processing to compensate signal deterioration caused by convolution of impulse responses.SOLUTION: The signal processing apparatus comprises: a first processing part which thins data out of a data signal; a convolution part which convolves, into the data signal processed by the first processing part, a compensation filter compensating signal deterioration caused by an impulse response obtained by thinning data by a second processing part; and an interpolation processing part which interpolates the result of the convolution by the convolution part.

Description

  The present invention relates to a signal processing device, a signal processing method, and a program.

  2. Description of the Related Art Recently, an imaging apparatus that uses an imaging element such as a CCD or CMOS and picks up an optical image of a subject formed on a light receiving surface of an imaging element by an imaging lens has become widespread. An image obtained by such an imaging apparatus has a resolution corresponding to the number of light receiving pixels on the light receiving surface and the resolution of the imaging lens.

  For this reason, in order to improve the resolution of an image obtained by the imaging device, it is effective to increase the number of light receiving pixels or to increase the resolution of the imaging lens. For example, by increasing the density of light receiving pixels on the light receiving surface and increasing the resolving power of the imaging lens so that a point image projected on the light receiving surface through the imaging lens is within the range of one light receiving pixel, It is possible to improve the resolution of the image obtained.

  Here, it is relatively easy to increase the density of the light receiving pixels constituting the image sensor due to recent technological improvements. On the other hand, in order to increase the resolving power of the imaging lens, it is effective to reduce the shape error and assembly error of each lens that constitutes the imaging lens, but it further increases the manufacturing accuracy of processing, assembly, and adjustment systems. The difficulty is high.

  As another approach for increasing the resolution, Patent Document 1 discloses a method for increasing the resolution by performing signal processing on an image obtained by imaging with an imaging device based on a point spread function (PSF). Are listed.

JP 2009-141742 A

  However, in the image obtained by the Bayer array type image sensor, the number of pixels of the G component is twice the number of pixels of the RB component. For this reason, when the image obtained by the image sensor is subjected to signal processing, there is a problem that the amount of calculation regarding the G component is larger than that of the other components. In addition to the G component, there is a concern that the calculation load for image signal processing increases due to the recent increase in the number of pixels.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to reduce the amount of calculation of signal processing for compensating for signal degradation caused by convolution of impulse responses. To provide a new and improved signal processing apparatus, signal processing method, and program.

  In order to solve the above-described problem, according to an aspect of the present invention, signal degradation due to an impulse response in which data is thinned out by a first processing unit and a second processing unit that thin out data from a data signal is compensated. There is provided a signal processing device including a convolution unit that convolves a compensation filter into a data signal processed by the first processing unit, and an interpolation processing unit that interpolates a convolution result by the convolution unit.

  The first processing unit thins data from the data signal after filtering by the first filter processing unit, and the second processing unit thins data from an impulse response after filtering by the second filter processing unit. The signal processing apparatus may further include a third filter processing unit that filters an interpolation result by the interpolation processing unit.

  The second filter processing unit may perform filtering using a filter used in the first filter processing unit and a filter used in the third filter processing unit.

  The signal processing device may further include an adding unit that adds a filtering result obtained by the third filter processing unit to the data signal.

  The data signal is an image signal, and the signal processing device further includes an adjustment unit that adjusts a convolution result of a certain region by the convolution unit based on edge information of the region, and the interpolation unit includes the adjustment unit. The convolution result after adjustment according to may be interpolated.

  The adjustment unit may adjust a convolution result related to the region based on the edge information so that the gain becomes higher as the edge component of the region is weaker.

  The adjustment unit is configured to generate the region based on the edge information of the region in the data signal, the edge information of the region in the convolution result, or the edge information of the region in the data signal processed by the first processing unit. You may adjust the convolution result about.

  The data signal is a Bayer array type input image captured by an imaging optical system, the impulse response is a point spread function of the imaging optical system, and the second processing unit is a point spread distribution of a G component. Data is thinned out from the function, the first processing unit thins out the data from the input image of the G component, and other components are subjected to processing independent of the first processing unit and the second processing unit. Also good.

  In order to solve the above problem, according to another aspect of the present invention, a step of thinning out data from a data signal by a first processing unit and an impulse response in which data is thinned out by a second processing unit There is provided a signal processing method including a step of convolving a compensation filter that compensates for signal degradation into a data signal processed by the first processing unit and a step of interpolating a convolution result.

  In order to solve the above-described problem, according to another aspect of the present invention, a computer includes an impulse response in which data is thinned out by a first processing unit that thins data from a data signal and a second processing unit. There is provided a program for functioning as a convolution unit that convolves a compensation filter that compensates for signal degradation due to the data signal after processing by the first processing unit, and an interpolation processing unit that interpolates a convolution result by the convolution unit. The

  As described above, according to the present invention, it is possible to suppress the amount of calculation of signal processing for increasing the image resolution.

It is explanatory drawing which showed the structure of the image processing apparatus by embodiment of this invention. It is explanatory drawing which showed the pixel arrangement | sequence in the light-receiving surface of a Bayer arrangement type CMOS image sensor. It is explanatory drawing which showed the concept of the frequency band of a Bayer arrangement | sequence image. It is the functional block diagram which showed the structure of the signal processing part by a comparative example. It is the flowchart which showed operation | movement of the signal processing part by a comparative example. It is explanatory drawing which showed the Y-axis cross section of the two-dimensional frequency response of the inverse filter designed in the signal processing part by a comparative example. It is explanatory drawing which showed the cross section of the Y-axis of the two-dimensional frequency response of the image decompress | restored by the signal processing part by a comparative example. It is the functional block diagram which showed the structure of the signal processing part by the 1st Embodiment of this invention. It is explanatory drawing which showed the example of a process by a filter process part and a reduction process part. It is explanatory drawing which showed the example of a process by a filter process part and a reduction process part. It is explanatory drawing which showed the example of a process by an expansion process part and a filter process part. It is explanatory drawing which showed the example of a process by an expansion process part and a filter process part. 5 is a flowchart illustrating an operation according to the first embodiment of the present invention. It is the functional block diagram which showed the structure of the signal processing part by the 2nd Embodiment of this invention. It is explanatory drawing which showed the pixel position utilized for calculation of edge information. It is explanatory drawing which showed the relationship between edge information and a gain. 5 is a flowchart illustrating an operation according to the second exemplary embodiment of the present invention. It is explanatory drawing which showed the Y-axis cross section of the two-dimensional frequency response of an input image. It is explanatory drawing which showed the cross section of the Y-axis of the two-dimensional frequency response of the restoration result of the input image shown in FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  In the present specification and drawings, a plurality of components having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numeral. However, when it is not necessary to particularly distinguish each of a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are given.

Further, the “DETAILED DESCRIPTION OF THE INVENTION” will be described according to the following item order.
1. 1. Basic configuration of image processing apparatus 2. Image processing according to comparative example First embodiment of the present invention 3-1. Configuration of Signal Processing Unit according to First Embodiment 3-2. 3. Operation according to the first embodiment Second embodiment of the present invention 4-1. Configuration of signal processing unit according to second embodiment 4-2. 4. Operation according to the second embodiment Summary

<1. Basic Configuration of Image Processing Device>
The present invention can be implemented in various forms as described in detail in “3. First Embodiment” and “4. Second Embodiment” as an example. In addition, the image processing apparatus (signal processing apparatus) according to each embodiment is
A. A first processing unit (230A) for thinning data from a data signal;
B. A convolution unit (250) for convolving a compensation filter that compensates for signal degradation due to the impulse response with data thinned out by the second processing unit (230B) into the data signal processed by the first processing unit;
An interpolation processing unit (260) for interpolating a convolution result by the convolution unit;
Is provided.

  In the following, first, a basic configuration common to each of such embodiments will be described with reference to FIGS.

  FIG. 1 is an explanatory diagram showing a configuration of an image processing apparatus 1 according to an embodiment of the present invention. As shown in FIG. 1, the image processing apparatus 1 according to the embodiment of the present invention includes an imaging optical system 10, an iris 2, a CMOS imaging device 14, an S / H_AGC 16, an A / D conversion unit 18, and an AE. A detection block 24, a microcomputer 26, and an iris driver 28 are provided. The image processing apparatus 1 according to the embodiment of the present invention can be applied to various devices such as an imaging device, a portable device, an in-vehicle device, and a medical device.

  The imaging optical system 10 includes a plurality of imaging lenses, and forms an optical image of a subject on the light receiving surface of the CMOS imaging device 14 via the iris 12.

  A CMOS (Complementary Metal Oxide Semiconductor) image sensor 14 converts an optical image of a subject formed on the light receiving surface into an electrical signal. The CMOS image sensor 14 is merely an example of an image sensor, and the image processing apparatus 1 may include, for example, a CCD (Charge Coupled Device) image sensor instead of the CMOS image sensor 14.

  The S / H_AGC 16 performs sample hold (S / H) and AGC (Automatic Gain Control) on the output from the CMOS image sensor 14, and the A / D converter 18 converts the output from the S / H_AGC 16 into a digital format. Convert. The digital signal from the A / D conversion unit 18 is supplied to the signal processing unit 20 and the AE detection block 24.

  The signal processing unit 20 performs various types of signal processing on the digital output from the A / D conversion unit 18, and outputs the processed signal to the subsequent circuit via the output terminal 22. For example, the signal processing unit 20 performs signal processing for improving the resolution of an input image input as a digital output, as will be described later in detail as each embodiment.

  An AE (Auto Exposure) detection block 24 detects a digital signal input from the A / D conversion unit 18. The microcomputer 26 supplies a control signal for controlling the opening degree of the iris 12 to the iris driver 28 based on the detection value supplied from the AE detection block 24. The iris driver 28 drives the iris 12 in accordance with a control signal supplied from the microcomputer 26.

(Bayer array)
In the CMOS image sensor 14 described above, pixels are arranged according to a Bayer arrangement. Hereinafter, the Bayer arrangement will be described with reference to FIG.

  FIG. 2 is an explanatory diagram showing a pixel array on the light receiving surface of the Bayer array type CMOS image sensor 14. As shown in FIG. 2, in the Bayer array, pixels that detect the B component and pixels that detect the G component are alternately arranged on the first line (the uppermost line) along the horizontal direction. In the line, pixels for detecting the G component and pixels for detecting the R component are alternately arranged. Hereinafter, a plurality of lines having the same pixel arrangement are arranged in the vertical direction V, and a color image is output from the CMOS image sensor 14 by performing photoelectric conversion in each pixel.

  FIG. 3 is an explanatory diagram showing the concept of the frequency band of an image having a Bayer array. As shown in FIG. 3, an image having a Bayer array has a G component band equivalent to twice the R and B component bands from the viewpoint of sampling.

<2. Image restoration processing according to comparative example>
The basic configuration of the image processing apparatus according to the embodiment of the present invention has been described above. Next, an image restoration process according to a comparative example will be described with reference to FIGS.

  FIG. 4 is a functional block diagram showing the configuration of the signal processing unit 80 according to the comparative example. As illustrated in FIG. 4, the signal processing unit 80 according to the comparative example includes a PSF table 82, an inverse filter design unit 84, a convolution unit 86, and an addition unit 88.

  The PSF table 82 holds a point spread function P (z) defined for each RGB component and each pixel. Although the point spread function is a non-uniform distribution in the screen, the difference from the neighboring pixels is not necessarily large, so that the point spread function can be uniformly approximated by a block within a certain range. Note that the point spread function is acquired by a method described in, for example, Japanese Patent Application Laid-Open No. 2009-141742. Alternatively, a point spread function of an optical design value may be used.

  The inverse filter design unit 84 designs an inverse filter K (z) for the point spread function P (z) for each RGB component and for each pixel. For example, the inverse filter design unit 84 designs the inverse filter K (z) by the method described in Japanese Patent Application Laid-Open No. 2009-141742 or a method that minimizes the mean square error with the input image based on the Wiener filter.

  Specifically, the inverse filter design unit 84 calculates the frequency response K (w) of the inverse filter according to the following Equation 1, and performs inverse Fourier transform on the frequency response K (ω) of the inverse filter, thereby obtaining the inverse filter K ( w) can be designed. In Equation 1, P (w) indicates the frequency response of PSF, and N (w) indicates the frequency response of the noise component.

  The convolution unit 86 performs inverse filter Kr (z), G component for the R component calculated by the inverse filter design unit 84 for each of the R component, G component, and B component of the input image X (z) in the Bayer array. The inverse filter Kg (z) for the B component and the inverse filter Kb (z) for the B component are convolved. The calculation by the convolution unit 86 is shown in the following formula 2. In Equation 2, Xr (z) represents the R component of the input image, Xg (z) represents the G component of the input image, and Xb (z) represents the B component of the input image.

  The adder 88 adds the convolution result Y (z) obtained by the convolution unit 86 to the input image for each RGB component as shown in Equation 3. In Equation 3, Or (z) represents the R component of the output image, Og (z) represents the G component of the output image, and Ob (z) represents the B component of the output image.

  The configuration of the signal processing unit 80 according to the comparative example has been described above. Next, the operation of the signal processing unit 80 according to the comparative example will be described with reference to FIG.

-Step S94
First, as shown in FIG. 5, the inverse filter design unit 84 performs an inverse filter K (z) for the point spread function P (z) as shown in Equation 1, for each RGB component and for each pixel, for example. design.

-Step S96
Thereafter, the convolution unit 86 applies the inverse filter Kr (z) for the R component calculated by the inverse filter design unit 84 to each of the R component, G component, and B component of the input image X (z) in the Bayer array. Each of the inverse filter Kg (z) for the G component and the inverse filter Kb (z) for the B component is convolved (S96).

-Step S98
Then, the adding unit 88 adds the convolution result Y (z) obtained by the convolution unit 86 to the input image for each RGB component as shown in Equation 3, for example, and outputs an output image O (z).

  FIG. 6 is an explanatory diagram showing a Y-axis cross section of the two-dimensional frequency response of the inverse filter designed in the signal processing unit 80 according to such a comparative example. FIG. 7 is an explanatory diagram showing a Y-axis cross section of the two-dimensional frequency response of the image restored by the signal processing unit 80 according to the comparative example. In the drawings, the R component is indicated by a solid line, the B component is indicated by a broken line, and the G component is indicated by a thin line.

(Background to the embodiment of the present invention)
As shown in FIG. 6, in the inverse filter of the G component and the B component according to the comparative example, the frequency of 0.5 or more when the Nyquist frequency is 1 is also raised by 10 dB or more. However, most of the high frequency components that have passed through the imaging lens are not image components but noise components. For this reason, the restoration method according to the comparative example has a problem that not only the resolution of the image but also noise components are amplified.

  Further, when the restoration method according to the comparative example is applied to the Bayer array type image shown in FIG. 2, the G component has a band twice as large as the RB component, so that coloring occurs. Specifically, as shown in FIG. 7, the G component and the B component are restored to a high frequency, but the R component is rapidly reduced from a frequency of about 0.4. Then, red coloring occurs.

  Further, in the image obtained by the Bayer array type imaging device, the number of pixels of the G component is twice the number of pixels of the RB component. For this reason, in signal processing such as inverse filter design and convolution, there is a problem that the amount of calculation related to the G component is larger than that of the other components. In addition to the G component, there is a concern that the calculation load for image signal processing increases due to the recent increase in the number of pixels.

  The embodiment of the present invention is made with the above circumstances as a focus, and according to the embodiment of the present invention, the optical image projected on the light receiving surface of the image sensor is captured to obtain an image, It is possible to efficiently improve the quality of the image from the viewpoint of calculation amount. Hereinafter, each embodiment of the present invention will be described in detail.

<3. First embodiment of the present invention>
[3-1. Configuration of Signal Processing Unit According to First Embodiment]
FIG. 8 is a functional block diagram showing the configuration of the signal processing unit 20-1 according to the first embodiment of the present invention. As shown in FIG. 10, the signal processing unit 20-1 according to the first embodiment includes a PSF table 210, filter processing units 220A, 220B, and 220C, reduction processing units 230A and 230B, and an inverse filter design unit 240. And a convolution unit 250 and an enlargement processing unit 260.

  In the following, an example in which processing according to the present embodiment including reduction and enlargement is performed for the G component and processing according to a comparative example that does not include reduction and enlargement is performed for the RB component will be described. On the other hand, processing according to the present embodiment including reduction and enlargement may be performed.

  The PSF table 210 holds a point spread function P (z) defined for each RGB component and each pixel. Although the point spread function is a non-uniform distribution in the screen, the difference from the neighboring pixels is not necessarily large, so that the point spread function can be uniformly approximated by a block within a certain range. Note that the point spread function is acquired by a method described in, for example, Japanese Patent Application Laid-Open No. 2009-141742. Alternatively, a point spread function of an optical design value may be used.

The filter processing unit 220A (second filter processing unit) uses a low-pass filter to reduce the aliasing component with respect to the G component of the Bayer-array input image X (z). Filter the G component. When the filter characteristic of the low-pass filter is L _a (z), the filtering by the filter processing unit 220A is expressed as Equation 4.

The reduction processing unit 230A thins out the data from the low-frequency input image X _a (z) after filtering by the filter processing unit 220A, thereby reducing the input image X _a (z) with respect to each of the horizontal direction and the vertical direction, for example 1 / Reduce to 2. Hereinafter, with reference to FIG. 9 and FIG. 10, an example of processing by the filter processing unit 220A and the reduction processing unit 230A will be specifically described.

9 and 10 are explanatory diagrams showing an example of processing by the filter processing unit 220A and the reduction processing unit 230A. The filter processing unit 220A and the reduction processing unit 230A sequentially process each G component in the horizontal direction and the vertical direction. For example, consider the case where the G component pixels are distributed according to the Bayer array as shown in the upper part of FIG. In this case, the filter processing unit 220A first uses a low-pass filter such as ( _{1 , 3, 3, 3, 1} ) along the horizontal direction to input the G component (G _{1,1 to} G _{m, n} ).

When the filtering result (g _{1,1 to} g _{m, n} ) shown in the middle stage of FIG. 9 is obtained by the filter processing unit 220A, the reduction processing unit 230A displays the filtering result (g _{1,1 to} g _{m, n} ). Thin out data from each row. As a result, as shown in the lower part of FIG. 9, an input image whose horizontal width is reduced to ½ is obtained.

The filter processing unit 220A and the reduction processing unit 230A perform the same processing on the vertical direction of the input image whose horizontal width is reduced to ½. For example, the filter processing unit 220A uses a low-pass filter such as (1, 3, 3, 3, 1) along the vertical direction and uses the G component ( g1 _{, n-1 to} gm _{, n-1} ) are filtered.

When the filtering result (g ′ _{1, n−1 to} g ′ _{m, n−1} ) shown in the middle stage of FIG. 10 is obtained by the filter processing unit 220A, the reduction processing unit 230A receives the filtering result (g ′ _{1, n -1 to} g'm _{, n-1} ) data is thinned out from each column. As a result, as shown in the lower part of FIG. 10, an input image in which the horizontal width and the vertical width are reduced to ½ is obtained.

As shown in Formula 5, the filter processing unit 220B (first filter processing unit) uses the point spread function P (z), the low-pass filter L _a (z) used in the filter processing unit 220A, and the filter processing unit. Filter using the low pass filter L _b (z) used at 220C. As a result, a band-limited point spread function P _a (z) is obtained.

The reduction processing unit 230B thins out the data from the point spread function P _a (z) obtained by the filter processing unit 220B, so that the point spread function P _a (z) is, for example, 1 for each of the horizontal direction and the vertical direction. Reduce to / 2. Note that the filter processing unit 220B and the reduction processing unit 230B, as described with reference to FIG. 9 and FIG. 10 regarding the input image, sequentially process the respective G components in the horizontal direction and the vertical direction, thereby performing point image distribution. The function P _a (z) may be acquired.

The inverse filter design unit 240 designs an inverse filter having an inverse characteristic of the point spread function P _a (z) reduced by the reduction processing unit 230B. Specifically, the inverse filter design unit 230 calculates the frequency response K _a (w) of the inverse filter according to Equation 6 below, and performs inverse Fourier transform on the frequency response K _a (ω) of the inverse filter, thereby performing the inverse filter. K _a (z) can be designed. In Equation 6, P _a (w) represents the frequency response of the point spread function after the reduction process, P ^* _a (w) represents the complex conjugate of P _a (w), and N (w) represents the noise component. The frequency response of is shown. The inverse filter is not strictly necessary that the inverse characteristics of the point spread function P _{a (z),} if the compensation filter for compensating at least a portion of the signal degradation due to the point spread function P _{a (z)} Good.

As shown in Equation 7, the convolution unit 250 applies an inverse filter K _ag (designed for the G component by the inverse filter design unit 240 to the input image X _ag (z) after the reduction processing by the reduction processing unit 230A. z) Convolve.

  The enlargement processing unit 260 expands the convolution result to an image T (z) having the same size as the input image by inserting data in the horizontal direction and the vertical direction of the convolution result by the convolution unit 250 (for example, interpolation of 0). To do.

The filter processing unit 220C (third filter processing unit) uses the low-pass filter L _b (z) to suppress the imaging component of the image T (z) after the enlargement by the enlargement processing unit 260, as shown in Expression 8. Filter using. Thereby, an output image O _g (z) relating to the G component is obtained.

  Here, with reference to FIG. 11 and FIG. 12, a processing example by the enlargement processing unit 260 and the filter processing unit 220C will be specifically described.

11 and 12 are explanatory diagrams showing an example of processing by the enlargement processing unit 260 and the filter processing unit 220C. The enlargement processing unit 260 and the filter processing unit 220C sequentially process each G component in the horizontal direction and the vertical direction. For example, when the convolution result (G _{1,1 to} G _{m−1, n−1} ) shown in the upper part of FIG. 11 is obtained by the convolution unit 250, first, the enlargement processing unit 260 sets 0 to each row of the convolution result. insert. As a result, an image having the same horizontal width as the input image is obtained.

  When the enlargement processing unit 260 obtains the enlargement result shown in the middle of FIG. 11, the filter processing unit 220C uses a low-pass filter such as (1, 3, 3, 3, 1) along the horizontal direction. Filter the magnified result.

  The enlargement processing unit 260 and the filter processing unit 220C perform the same processing on the image shown in the lower part of FIG. 11 whose horizontal width is the same as that of the input image and whose vertical width is half that of the input image. For example, the enlargement processing unit 260 inserts 0 in each column of the image as shown in the middle part of FIG. Thereby, an image having the same horizontal width and vertical width as the input image is obtained.

The filter processing unit 220C filters this image using a low-pass filter such as (1, 3, 3, 3, 1) along the vertical direction. As a result, G ′ _{1,1 to} G ′ _{m, n} are obtained as shown in the lower part of FIG. Among these, the G component value according to the Bayer arrangement shown with colors in the lower part of FIG. 12 is output as an output image of the G component.

[3-2. Operation according to first embodiment]
The configuration of the signal processing unit 20-1 according to the first embodiment of the present invention has been described above. Next, the operation according to the first embodiment of the present invention will be described with reference to FIG.

FIG. 13 is a flowchart showing an operation according to the first embodiment of the present invention. As shown in FIG. 13, first, the filter processing unit 220B uses the point spread function P (z) as the low-pass filter L _a (z) used in the filter processing unit 220A and the low-pass used in the filter processing unit 220C. Filtering is performed using the filter L _b (z) (S300).

The reduction processing unit 230B is, by thinning out data from the point obtained by the filter processing unit 220B spread function P _{a (z),} for each of the horizontal and vertical point spread function P _{a (z)} For example, it is reduced to 1/2 (S310). Furthermore, the inverse filter design unit 240 designs an inverse filter K _a (z) having an inverse characteristic of the point spread function P _a (z) reduced by the reduction processing unit 230B (S320).

On the other hand, the filter processing unit 220A filters the G component of the input image X (z) using a low-pass filter in order to suppress the aliasing component with respect to the G component of the input image X (z) in the Bayer array. (S330). The reduction processing unit 230A thins out the data from the low-frequency input image X _a (z) after filtering by the filter processing unit 220A, thereby reducing the input image X _a (z) with respect to each of the horizontal direction and the vertical direction, for example 1 / It is reduced to 2 (S340).

Subsequently, the convolution unit 250 convolves the inverse filter K _ag (z) designed for the G component by the inverse filter design unit 240 with respect to the input image X _ag (z) after the reduction processing by the reduction processing unit 230A. (S350).

Thereafter, the enlargement processing unit 260 inserts data in the horizontal direction and the vertical direction of the convolution result by the convolution unit 250 (for example, interpolates 0), whereby the convolution result is an image T (z) having the same size as the input image. (S360). Further, the filter processing unit 220C filters the image T (z) that has been enlarged by the enlargement processing unit 260 using the low-pass filter L _b (z) in order to suppress the imaging component (S370). Thereby, an output image O _g (z) relating to the G component is obtained.

  As described above, according to the first embodiment of the present invention, since the input image reduced by the reduction processing unit 230A is convolved, it is possible to suppress the calculation load in the convolution unit 250. is there. In the above description, the example in which each of the horizontal width and the vertical width is reduced to ½ has been described. However, the present embodiment is not limited to such an example. For example, only the horizontal width or the vertical width may be reduced, the horizontal width and the vertical width may be reduced at different ratios, or the reduction width may be made smaller than 1/2. May be.

<4. Second embodiment of the present invention>
The first embodiment of the present invention has been described above. Subsequently, a second embodiment of the present invention will be described with reference to FIGS.

[4-1. Configuration of Signal Processing Unit According to Second Embodiment]
FIG. 14 is a functional block diagram showing the configuration of the signal processing unit 20-2 according to the second embodiment of the present invention. As shown in FIG. 14, the signal processing unit 20-2 according to the second embodiment includes a PSF table 210, filter processing units 220A, 220B, and 220C, reduction processing units 230A and 230B, and an inverse filter design unit 240. ', A convolution unit 250, an enlargement processing unit 260, an adjustment unit 270, and an addition unit 280.

  Since the PSF table 210, the filter processing units 220A and 220B, and the reduction processing units 230A and 230B have been described in the first embodiment, a detailed description thereof will be omitted.

The inverse filter design unit 240 ′ designs an inverse filter having an inverse characteristic of the point spread function P _a (z) reduced by the reduction processing unit 230B. Specifically, the inverse filter design unit 240 ′ calculates the inverse filter frequency response K _a (w) according to the following Equation 9 and performs inverse Fourier transform on the inverse filter frequency response K _a (ω). A filter K _a (z) can be designed. In Equation 9, P _a (w) represents the frequency response of the point spread function after the reduction process, P ^* _a (w) represents the complex conjugate of P _a (w), and N (w) represents the noise component. The frequency response of is shown.

As shown in Equation 10, the convolution unit 250 applies an inverse filter K _ag (designed for the G component by the inverse filter design unit 240 to the input image X _ag (z) after the reduction processing by the reduction processing unit 230A. z) Convolve.

The adjustment unit 270 adjusts the convolution result Y _a (z) of _a certain target pixel by the convolution unit 250 based on edge information in the vicinity of the target pixel. Specifically, the adjustment unit 270 first calculates edge information Diff in the vicinity of the target pixel shown in Expression 11. The positions of the pixels A, B, C, and D in Expression 11 with respect to the target pixel P are as shown in FIG.

  Note that the adjustment unit 250 may calculate edge information Diff near the target pixel in the input image, may calculate edge information Diff near the target pixel in the convolution result by the convolution unit 250, or may be a reduction processing unit. The edge information Diff near the target pixel in the input image after the reduction processing by 230A may be calculated.

  Then, the adjustment unit 270 calculates the gain G according to the edge information Diff. For example, the adjustment unit 270 may calculate the gain G according to the relationship illustrated in FIG. 16 between the edge information Diff and the gain G. According to the relationship shown in FIG. 16, when the edge information Diff is less than or equal to the threshold, 1.00 is calculated as the gain G, and when the edge information Diff becomes greater than or equal to the threshold, the gain G decreases stepwise.

Further, the adjustment unit 270 adjusts the convolution result Y _a (z) by the convolution unit 250 according to the calculated gain G, as shown in Expression 12.

  The enlargement processing unit 260 inserts data in the horizontal direction and the vertical direction of the convolution result adjusted by the adjustment unit 270 (for example, interpolates 0), thereby converting the convolution result into an image T (z having the same size as the input image. ).

The filter processing unit 220C filters the image T (z) enlarged by the enlargement processing unit 260 using a low-pass filter L _b (z) in order to suppress the imaging component, as shown in Expression 13. Thereby, the addition image Z _b (z) regarding the G component is obtained.

As shown in Formula 14, the adding unit 280 adds the addition image Z _b (z), which is the filtering result by the filter processing unit 220C, to the input image X (z). Thereby, an output image O _g (z) relating to the G component is obtained.

[4-2. Operation according to the second embodiment]
The configuration of the signal processing unit 20-2 according to the second embodiment of the present invention has been described above. Subsequently, an operation according to the second embodiment of the present invention will be described with reference to FIG.

FIG. 17 is a flowchart showing an operation according to the second embodiment of the present invention. As shown in FIG. 16, the filter processing unit 220B converts the point spread function P (z) into the low-pass filter L _a (z) used in the filter processing unit 220A and the low-pass filter L used in the filter processing unit 220C. Filtering is performed using _b (z) (S300).

The reduction processing unit 230B is, by thinning out data from the point obtained by the filter processing unit 220B spread function P _{a (z),} for each of the horizontal and vertical point spread function P _{a (z)} For example, it is reduced to 1/2 (S310). Furthermore, the inverse filter design unit 240 designs an inverse filter K _a (z) having an inverse characteristic of the point spread function P _a (z) reduced by the reduction processing unit 230B (S320).

On the other hand, the filter processing unit 220A filters the G component of the input image X (z) using a low-pass filter in order to suppress the aliasing component with respect to the G component of the input image X (z) in the Bayer array. (S330). The reduction processing unit 230A thins out the data from the low-frequency input image X _a (z) after filtering by the filter processing unit 220A, thereby reducing the input image X _a (z) with respect to each of the horizontal direction and the vertical direction, for example 1 / It is reduced to 2 (S340).

Subsequently, the convolution unit 250 convolves the inverse filter K _ag (z) designed for the G component by the inverse filter design unit 240 with respect to the input image X _ag (z) after the reduction processing by the reduction processing unit 230A. (S350).

Then, the adjustment unit 270 adjusts the convolution result Y _a (z) of _a certain target pixel by the convolution unit 250 based on the edge information near the target pixel (S360). Thereafter, the enlargement processing unit 260 inserts data in the horizontal direction and the vertical direction of the convolution result adjusted by the adjustment unit 270 (for example, interpolates 0), so that the convolution result is an image T having the same size as the input image. The image is enlarged to (z) (S370).

Further, the filter processing unit 220C filters the image T (z) that has been enlarged by the enlargement processing unit 260 using the low-pass filter L _b (z) in order to suppress the imaging component (S380). Thereafter, the addition unit 280 adds the addition image Z _b (z), which is the filtering result by the filter processing unit 220C, to the input image X (z). Thereby, an output image O _g (z) relating to the G component is obtained.

  Here, with reference to FIG. 18 and FIG. 19, the effect by the 2nd Embodiment of this invention is demonstrated.

  FIG. 19 is an explanatory diagram showing a Y-axis cross section of the two-dimensional frequency response of the restoration result of the input image shown in FIG. As shown in FIG. 19, according to the second embodiment of the present invention, it is possible to sufficiently restore the low frequency up to about 0.4. Further, according to the second embodiment of the present invention, as shown in FIG. 19, since each color falls from about 0.4, the problem of coloring is solved, and the high frequency is lowered by -10 dB or more. Therefore, the high frequency noise component is not amplified. In addition, by the adjustment based on the edge information by the adjustment unit 270, it is possible to improve the phenomenon that the pixels that occur at the black and white edge portion sink into black.

<5. Summary>
As described above, according to the embodiment of the present invention, since the convolution is performed on the input image reduced by the reduction processing unit 230A, it is possible to suppress the calculation load in the convolution unit 250. Further, according to the embodiment of the present invention, it is possible to sufficiently restore a low frequency up to about 0.4 while avoiding amplification of a high frequency noise component.

  Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

  For example, the image processing apparatus that compensates the input image in which the point spread function is convoluted has been described above, but the present invention is not limited to such an example. A point spread function is only an example of an impulse response, an input image is only an example of a data signal, and an image processing device is only an example of a signal processing device that compensates for a data signal in which the impulse response is convoluted. The present invention is applicable to all signal processing apparatuses that compensate for a data signal in which an impulse response is convoluted.

  Further, each step in the processing of the signal processing unit 20 of the present specification does not necessarily have to be processed in time series in the order described as a flowchart. For example, each step in the processing of the signal processing unit 20 may be processed in an order different from the order described as the flowchart, or may be processed in parallel.

  In addition, a computer program for causing hardware such as a CPU, a ROM, and a RAM built in the image processing apparatus 1 to perform the same functions as the components of the signal processing unit 20 described above can be created. A storage medium storing the computer program is also provided.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 10 Imaging optical system 20 Signal processing part 210 PSF table 220 Filter processing part 230 Reduction processing part 240,240 'Inverse filter design part 250 Convolution part 260 Enlargement processing part 270 Adjustment part 280 Addition part

Claims (10)

A first processing unit for thinning data from the data signal;
A convolution unit that convolves a compensation filter that compensates for signal degradation due to an impulse response with data thinned out by the second processing unit into the data signal processed by the first processing unit;
An interpolation processing unit for interpolating a convolution result by the convolution unit;
A signal processing apparatus comprising: The first processing unit thins data from the data signal after filtering by the first filter processing unit,
The second processing unit thins data from the impulse response after filtering by the second filter processing unit,
The signal processing device includes:
The signal processing apparatus according to claim 1, further comprising a third filter processing unit that filters an interpolation result obtained by the interpolation processing unit.   The signal processing apparatus according to claim 2, wherein the second filter processing unit performs filtering using a filter used in the first filter processing unit and a filter used in the third filter processing unit. The signal processing device includes:
The signal processing apparatus according to claim 1, further comprising an addition unit that adds a filtering result obtained by the third filter processing unit to the data signal. The data signal is an image signal;
The signal processing device includes:
An adjustment unit that adjusts a convolution result related to a certain region by the convolution unit based on edge information of the region;
The signal processing apparatus according to claim 4, wherein the interpolation unit interpolates a convolution result after adjustment by the adjustment unit.   The signal processing apparatus according to claim 5, wherein the adjustment unit adjusts a convolution result related to the region based on the edge information so that the gain becomes higher as the edge component of the region is weaker.   The adjustment unit is configured to generate the region based on the edge information of the region in the data signal, the edge information of the region in the convolution result, or the edge information of the region in the data signal processed by the first processing unit. The signal processing apparatus according to claim 6, wherein the convolution result is adjusted. The data signal is a Bayer array type input image captured by an imaging optical system,
The impulse response is a point spread function of the imaging optical system,
The second processing unit thins data from the point spread function of the G component,
The first processing unit thins data from an input image of G component,
The signal processing apparatus according to claim 7, wherein processing other than the first processing unit and the second processing unit is performed on the other components. Thinning out data from the data signal by the first processing unit;
Convolving a compensation filter that compensates for signal degradation due to an impulse response with data thinned out by the second processing unit into the data signal processed by the first processing unit;
Interpolating the convolution results;
Including a signal processing method. Computer
A first processing unit for thinning data from the data signal;
A convolution unit that convolves a compensation filter that compensates for signal degradation due to an impulse response with data thinned out by the second processing unit into the data signal processed by the first processing unit;
An interpolation processing unit for interpolating a convolution result by the convolution unit;
Program to function as

Images