JP2013055623A - Image processing apparatus, image processing method, information recording medium, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an apparatus and a method of achieving appropriate blur correction for the imaging data having an RGBW pixel arrangement.SOLUTION: The image processing apparatus includes a blur correction processing unit performing blur correction for the output signal from an image sensor having an RGBW arrangement consisting of RGB pixels and W (white) pixels. When performing blur correction processing for a W pixel signal, the blur correction processing unit creates a blur correction filter reflecting the color ratio of a local area including the W pixel signal to be corrected, and performs blur correction processing to which the filter reflecting the color ratio thus created is applied. For example, the low frequency components of RGB colors are calculated in the local area, the color ratio is calculated using these calculation values, and then blur correction is performed by creating a blur correction filter corresponding to the color ratio thus calculated.

Description

  The present disclosure relates to an image processing device, an image processing method, an information recording medium, and a program. In particular, the present invention relates to an image processing apparatus, an image processing method, an information recording medium, and a program that execute image blur correction.

An image captured by an imaging apparatus such as a digital camera has noise and blurring due to, for example, focus shift, lens aberration, noise generated by an imaging device such as a CCD or CMOS, and the like. .
In particular, in a camera using an inexpensive lens, image blur due to lens aberration increases as the image height (distance from the optical center) increases.

  For example, Patent Document 1 (Japanese Patent Application Laid-Open No. 2006-246080) and Patent Document 2 (Japanese Patent Application Laid-Open No. 2009-159603) are disclosed as related arts that disclose correction processing for image noise and blur based on various factors. And Japanese Patent Application Laid-Open No. 2010-081263.

Japanese Patent Application Laid-Open No. 2006-246080 discloses a correction processing configuration that uses different sharpening parameters for each region of a captured image. Specifically, the image correction is performed by increasing the sharpening parameter as the image height (distance from the optical center) increases.
However, the process of Patent Document 1 merely performs a process of increasing the intensity of the high-frequency component in accordance with the image height, and does not consider, for example, the change in the chromatic aberration of magnification or the frequency characteristic of the blur, and the appropriate correction. Has a problem that may not run

Further, Patent Document 2 (Japanese Patent Laid-Open No. 2009-159603) selects a filter corresponding to each lens characteristic and executes blur correction processing in order to cope with blurring that varies depending on lens differences or manufacturing errors. The configuration is disclosed.
However, if this correction processing is applied to blur in which the blur method changes continuously according to the image height, such as the above-described image height-dependent blur, the number of types of blur correction filters is enormous. The problem of becoming a number occurs.

Furthermore, Patent Document 3 (Japanese Patent Laid-Open No. 2010-081263) applies a filter in which the position of the center of gravity is shifted for each image height to a so-called one-side blur indicating how the blur occurs depending on the image height. A configuration for executing an appropriate correction according to the image height is disclosed.
More specifically, correction corresponding to blurring is performed by correcting the position of the center of gravity of the filter. However, the focus is on the correction of the center of gravity, the control of the correction strength is not shown, and individual processing corresponding to the strong or weak part is not executed, and appropriate correction is not made Will occur.

  Furthermore, as a filter attached to the image sensor, in addition to each color of RGB, correction processing for a captured image of an imaging device including a filter having an RGBW array including all-wavelength transmission type W (White) pixels is disclosed in Patent Document 4 ( JP, 2011-55038, A).

When a filter having an RGBW array including W (White) pixels is used, the W pixel transmits visible light having a wide wavelength range, and thus the blur is different depending on the color of the subject.
The process of Patent Document 4 is configured to perform uniform correction independent of the subject color, and there is a problem that a sufficient correction effect cannot be obtained for different blurring methods depending on the subject color.

  In addition, many of the above-described prior art configurations perform pixel value correction based on filtering by, for example, convolution calculation. In order to perform various corrections, a large number of filters for performing this filtering are stored. Therefore, there is a problem in mounting on a small and low-cost camera, such as requiring a memory with a large capacity.

JP 2006-246080 A JP 2009-159603 A JP 2010-081263 A JP 2011-055038 A

The present disclosure has been made in view of, for example, the above-described problems, and in an image blur correction configuration, an image processing apparatus, an image processing method, and an information recording medium that realize appropriate correction particularly according to an image height It aims at providing a program.
Further, in an embodiment of the present disclosure, the number of filters stored in advance in the memory is configured by appropriately using a blur correction filter corresponding to blur of various aspects based on a basic filter. Image processing apparatus, image processing method, information recording medium, and program are provided.

The first aspect of the present disclosure is:
A blur correction processing unit that performs blur correction processing on an output signal of an image sensor having an RGBW array including RGB pixels and W (white) pixels;
The blur correction processing unit
An image for generating a color ratio reflecting blur correction filter reflecting the color ratio of the local region including the W pixel signal to be corrected and executing the blur correction process using the color ratio reflecting filter in the blur correction process for the W pixel signal In the processing unit.

  Furthermore, in one embodiment of the image processing device according to the present disclosure, the blur correction processing unit performs interpolation having a W pixel signal corresponding to all pixels by an interpolation process using a W pixel signal included in an output signal of the image sensor. An image is generated, the color ratio reflection blur correction filter is generated for each W pixel constituting the generated interpolation image, and blur correction processing using the color ratio reflection filter is executed.

  Furthermore, in an embodiment of the image processing apparatus of the present disclosure, the blur correction processing unit calculates a low frequency component of each color of RGB in the local region, and calculates a ratio of the calculated values of the low frequency components of each color of RGB in the local region. The color ratio of the area.

  Furthermore, in an embodiment of the image processing apparatus according to the present disclosure, the blur correction processing unit is configured to perform the R corresponding blur correction previously stored in the memory according to a blend ratio determined according to each RGB color ratio of the local region. The color ratio reflecting blur correction filter is generated by blending the filter, the G corresponding blur correction filter, and the B corresponding blur correction filter.

  Furthermore, in an embodiment of the image processing device of the present disclosure, the image processing device further includes a data conversion unit that converts the RGBW array into an RGB array.

  Furthermore, in an embodiment of the image processing apparatus according to the present disclosure, the data conversion unit is a blur correction signal that is an RGB-compatible blur correction signal estimated from a blur correction signal corresponding to a W pixel signal generated by the blur correction processing unit. An RGB signal (Rd, Gd, Bd) is generated, and the RGB signal values (Rd, Gd, Bd) with blur correction are applied to determine the RGB signal values constituting the RGB array.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the image processing device further analyzes an output signal of an image sensor having an RGBW array including RGB pixels and W pixels to obtain edge intensity information corresponding to each pixel. An edge detection unit that generates edge information, and the data conversion unit calculates a non-blurred RGB signal (Rl, Gl, Bl) that is a non-applied signal of the blur correction process, and the blur correction processing unit Or an RGB array by blending the blur-corrected RGB signal (Rd, Gd, Bd) corresponding to RGB estimated from the generated blur correction signal corresponding to the W pixel signal and the RGB signal without blur correction. This is a configuration that determines the RGB signal values that make up a color image, and blur correction is performed at pixel positions that have been determined to have high edge strength based on the edge information. Ri by increasing the blending ratio of the RGB signal, the edge strength performs determination processing of RGB signal values by performing the blend processing set high blend ratio of blurring-uncorrected RGB signals in the pixel position is determined to be small.

  Furthermore, in one embodiment of the image processing device of the present disclosure, the blur correction processing unit performs blur correction processing by applying a W pixel signal to generate a blur correction W signal (Wd) corresponding to each pixel, The data converter is based on the assumption that there is a positive correlation between the W signal and each RGB signal in the local region of the image, and there is blur correction that is a blur correction signal corresponding to RGB from the blur correction W signal (Wd). RGB signals (Rd, Gd, Bd) are calculated.

  Furthermore, in an embodiment of the image processing device of the present disclosure, the blur correction processing unit includes the color ratio reflecting blur correction filter and a coordinate position corresponding high-pass filter that is generated according to an image height of a blur correction target pixel. The image processing apparatus according to claim 1, wherein blend processing is performed by applying a blend coefficient determined according to an image height of a blur correction target pixel, and blur correction processing is executed by applying a filter generated by the blend processing.

The blur correction processing unit
The color ratio reflecting blur correction filter and the coordinate position corresponding high pass filter are blended by applying a blend coefficient determined according to the image height and focus position information of the blur correction target pixel, and are generated by the blend processing. The blur correction process is executed by applying the filter.

  Furthermore, in an embodiment of the image processing apparatus of the present disclosure, the blur correction processing unit is configured to determine the coordinate position as the pixel position of the blur correction target pixel is farther from the image center and the image height is larger in the blending process. A blend process is performed in which the blend ratio of the corresponding high-pass filter is reduced.

  Furthermore, in one embodiment of the image processing device of the present disclosure, the blur correction processing unit includes an x-direction adjustment high-pass filter (HPF_x), a y-direction adjustment high-pass filter (HPF_y), which are stored in a memory in advance, and an image. The coordinate position-corresponding high-pass filter is generated by blending the center filter (HPF_center) corresponding to the corner center position [(x, y) = (0, 0)] according to the pixel position of the blur correction target pixel. .

  Furthermore, in an embodiment of the image processing apparatus according to the present disclosure, the blur correction processing unit further performs an intensity adjustment by performing an enhancement intensity adjustment on the coordinate position corresponding high-pass filter according to an image height of a blur correction target pixel. A filter is generated, and a blur correction process using the intensity adjustment filter is performed.

  Furthermore, in one embodiment of the image processing device according to the present disclosure, the blur correction processing unit separates the coordinate position corresponding high-pass filter into a DC component and an AC component, and an image of a blur correction target pixel with respect to the AC component. An adjustment process using an intensity adjustment parameter corresponding to the height is performed, and the intensity adjustment filter is generated by recombining the AC component subjected to the adjustment process and the DC component.

  Furthermore, in an embodiment of the image processing apparatus according to the present disclosure, the blur correction processing unit applies an intensity adjustment parameter corresponding to the image height of the blur correction target pixel and focus position information to the coordinate position corresponding high-pass filter. An enhancement intensity adjustment is performed to generate the intensity adjustment filter.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the image signal correction unit includes a saturation detection unit that detects whether or not a saturated pixel is included in a local region including a plurality of pixels including a blur correction target pixel. The blur correction processing unit inputs detection information from the saturation detection unit, and when the local region includes a saturated pixel, the blur correction target pixel is not subjected to blur correction, and the saturated pixel Only when it is not included, blur correction of the blur correction target pixel is executed.

Furthermore, the second aspect of the present disclosure is:
An image processing method for executing image blur correction processing in an image processing apparatus,
The blur correction processing unit executes a blur correction processing step for performing blur correction processing on an output signal of an image sensor having an RGBW array including RGB pixels and W (white) pixels,
The blur correction processing step includes
A step of generating a color ratio reflecting blur correction filter reflecting a color ratio of a local region including the W pixel signal to be corrected and executing the blur correction process using the color ratio reflecting filter in the blur correction processing for the W pixel signal; The image processing method is.

Furthermore, the third aspect of the present disclosure is:
A program for executing image blur correction processing in an image processing apparatus,
Causing the blur correction processing unit to execute a blur correction processing step for performing blur correction processing on an output signal of an image sensor having an RGBW array including RGB pixels and W (white) pixels;
In the blur correction processing step,
When blur correction processing is performed on a W pixel signal, a color ratio reflecting blur correction filter that reflects the color ratio of the local region including the W pixel signal to be corrected is generated, and blur correction processing using the color ratio reflecting filter is executed. In the program.

Furthermore, the fourth aspect of the present disclosure is:
A recording medium that records a program for executing image blur correction processing in an image processing apparatus,
Causing the blur correction processing unit to execute a blur correction processing step for performing blur correction processing on an output signal of an image sensor having an RGBW array including RGB pixels and W (white) pixels;
In the blur correction processing step,
When blur correction processing is performed on a W pixel signal, a color ratio reflecting blur correction filter that reflects the color ratio of the local region including the W pixel signal to be corrected is generated, and blur correction processing using the color ratio reflecting filter is executed. It is on the recording medium that records the program.

  Note that the program of the present disclosure is a program that can be provided by, for example, a storage medium or a communication medium provided in a computer-readable format to an information processing apparatus or a computer system that can execute various program codes. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the information processing apparatus or the computer system.

  Other objects, features, and advantages of the present disclosure will become apparent from a more detailed description based on embodiments of the present disclosure described below and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

According to the configuration of an embodiment of the present disclosure, an apparatus and a method for realizing appropriate blur correction for imaging data having an RGBW pixel array are realized.
Specifically, the image processing apparatus includes a blur correction processing unit that performs a blur correction process on an output signal of an image sensor having an RGBW array including RGB pixels and W (white) pixels. The blur correction processing unit generates a color ratio reflecting blur correction filter that reflects the color ratio of the local region including the W pixel signal to be corrected, and applies the generated color ratio reflecting filter when the blur correction process is performed on the W pixel signal. Perform blur correction processing. For example, low frequency components of each color of RGB are calculated in the local region, a color ratio is calculated using these calculated values, a blur correction filter corresponding to the calculated color ratio is generated, and blur correction is executed.
With these processes, an appropriate blur correction process using an optimal blur correction filter corresponding to the color ratio of the local region of the image is realized.

It is a figure explaining the aspect of the blur which generate | occur | produces according to image height. It is a figure explaining an example of the blur correction process by a Wiener filter. It is a figure explaining the structural example of the image processing apparatus of this indication. It is a figure explaining the structural example of the image processing apparatus of this indication. It is a figure which shows the flowchart explaining the whole sequence of the process which an image signal correction | amendment part performs. It is a figure explaining the blur correction | amendment with respect to the pixel area | region containing a saturated pixel, and false color generation. It is a figure explaining the process set as the setting which does not perform blur correction of a correction object pixel, when a saturation pixel is contained around the blur correction object pixel. It is a figure explaining flatness (weightFlat). It is a figure which shows the flowchart explaining the detailed sequence of a filter production | generation process. It is a figure which shows the flowchart explaining the sequence which produces | generates a coordinate position corresponding | compatible high-pass filter (HPF_dist). It is a figure explaining the example of the high pass filter applied in order to produce | generate a coordinate position corresponding | compatible high pass filter (HPF_dist). It is a figure which shows the enhancement characteristic of a filter. It is a figure explaining the example of a setting of (a) lens MTF characteristic, (b) Wiener filter characteristic, and (c) blend coefficient (alpha). It is a figure explaining the process which produces | generates the blurring correction filter (Ffinal) which adjusted the enhancement intensity | strength. It is a figure explaining the process which produces | generates the blurring correction filter (Ffinal) which adjusted the enhancement intensity | strength. It is a figure which shows the enhancement characteristic of a filter. It is a figure explaining the example of a setting of (a) lens MTF characteristic, (b) Wiener filter characteristic, and (c) enhancement intensity | strength adjustment parameter (beta). It is a figure which shows the spectral intensity characteristic of each color in an RGBW filter. It is a figure which shows the MTF characteristic of each color of RGBW. It is a figure explaining the conversion process to the RGB image of an RGBW image. It is a figure explaining one Example of an image processing apparatus. It is a figure explaining one Example of an image processing apparatus. It is a figure which shows the flowchart explaining the whole sequence of the process which an image signal correction | amendment part performs. It is a figure explaining the structural example of the low frequency component calculation filter which calculates low frequency component mR, mG, mB. It is a figure explaining the structural example of the low frequency component calculation filter which calculates low frequency component mR, mG, mB. It is a figure explaining the structural example of the low frequency component calculation filter which calculates low frequency component mR, mG, mB. It is a figure explaining the structural example of the low frequency component calculation filter which calculates low frequency component mR, mG, mB. It is a figure explaining the example of a calculation process of W corresponding blur correction filter. It is a figure explaining the example of the filter which calculates the low frequency component calculation filter which calculates the low frequency component mW, and the W signal Wn which removed the noise signal. It is a figure explaining the correspondence example of the pixel value ratio of W and G in a local region.

Hereinafter, the details of the image processing apparatus, the image processing method, the information recording medium, and the program of the present disclosure will be described with reference to the drawings. The description will be made according to the following items.
1. 1. About the characteristics of the blur that occurs in the captured image and the outline of the filter that can be applied to blur correction (Example 1) 2. Configuration example of image processing apparatus 3. Details of blur correction processing of the image processing apparatus according to the first embodiment (Embodiment 2) Configuration example of image processing apparatus that executes blur correction considering color ratio 5. Details of blur correction processing of image processing apparatus according to Embodiment 2 Summary of composition of this disclosure

[1. About the characteristics of blur that occurs in captured images and the outline of filters that can be applied to blur correction]
Prior to specific description of the image processing apparatus according to the present disclosure, characteristics of blur generated in a captured image and an outline of a basic filter applicable to blur correction will be described.

As described above, an image taken by an imaging device such as a digital camera is blurred according to lens characteristics and characteristics of an imaging element such as a CCD or CMOS.
In particular, in a camera using an inexpensive lens, image blur due to lens aberration increases as the image height (distance from the optical center) increases.

A mode of blur generated according to the image height will be described with reference to FIG.
FIG. 1 shows a captured image of a checker pattern in which rectangular white areas and black areas are alternately set.
As shown in FIG. 1, the center area of the image is less blurred. That is, the closer to the optical center (the smaller the image height), the smaller the blur.
On the other hand, the greater the distance from the optical center, that is, the greater the image height, the greater the blur.
As described above, the blur generated in the image taken by the camera tends to be smaller at the center of the image area near the optical center of the lens of the camera and larger at the peripheral area of the image far from the optical center of the lens.

Next, a Wiener filter as an example of a filter used for image blur correction will be briefly described.
(1) Ideal image (original image) without blur,
(2) Blurred shot image,
(3) a restored image restored by filter application processing on the captured image;
Assume these three images.
here,
(1) An ideal image (original image) without blur,
(3) a restored image restored by filter application processing on the captured image;
A filter that minimizes the square error between these two images is called a least square filter or a Wiener filter.

f (x, y) is an ideal image (original image) without blur,
g (x, y) is a captured image including blur,
h (x, y) is a degradation function due to lens aberration or camera shake,
n (x, y) is a noise component,
And
(X, y) is the pixel position of each image, and f (x, y) to n (x, y) may indicate the pixel value of the coordinate position (x, y) of each image.
Here, assuming that the degradation function h (x, y) due to lens aberration and camera shake is a fixed value, the following relationship (Equation 1) holds.

... (Formula 1)

When both sides of the above equation are Fourier transformed,
G (u, v) = H (u, v) × F (u, v) + N (u, v) (Equation 2)
It becomes.
Here, G (u, v), H (u, v), F (u, v), and N (u, v) are g (x, y), h (x, y), and f (x, y) represents the Fourier transform of n (x, y).
From this equation, when there is no zero point in the degradation function due to lens aberration or camera shake and the noise component is known, F (u, v) can be obtained from the following (Equation 3).

... (Formula 3)

  However, since the noise component is generally unknown, the above (Equation 3) cannot be solved exactly. Therefore, blur correction is performed using a Wiener filter K (u, v) of the following (Equation 4) that minimizes the error between the ideal image (F) and the restored image (F ′) corrected for blur. Done.

.... (Formula 4)
However, in the above (Formula 4),
(Sn (u, v) / Sf (u, v)): Indicates the power spectral density (Γ) between the ideal image (original image) F and noise N.
(Γ = Sn (u, v) / Sf (u, v))

  As shown in (Expression 5) below, k (x, y) obtained by inverse Fourier transform of this filter and an image f ′ (x, y) corrected for blur by convolving the observed image in real space. Can be obtained.

... (Formula 5)

An example of blur correction processing by the Wiener filter according to the above (Formula 5) is shown in FIG.
In FIG.
(A) Captured image G
(B) Wiener filter K
These are shown.

(A) A photographed image is a RAW image composed of RGBW pixels including W (White) pixels.
Here, the correction target pixel is a central W pixel of 7 × 7 pixels shown in FIG.
The Wiener filter K shown in (b) shows a multiplication coefficient for each of a plurality of W pixels included in 7 × 7 pixels. A convolution operation for multiplying and adding each pixel value to the pixel value of the W pixel included in the 7 × 7 pixel is executed to calculate a correction pixel value of the central W pixel.

The example shown in FIG. 2 shows an example of processing for W pixels of an RGBW-structured image, but the same filter processing is performed for each of R, G, and B pixels.
Also, blur correction by the same Wiener filter application processing is performed on an RGB configuration image instead of the RGBW configuration.

[2. (Example 1) Configuration example of image processing apparatus]
A configuration example of the image processing apparatus according to the present disclosure will be described with reference to FIGS. 3 and 4.
FIG. 3 is a diagram illustrating a configuration example of the imaging apparatus 100 which is an embodiment of the image processing apparatus of the present disclosure. The imaging apparatus 100 includes an optical lens 105, an imaging element (image sensor) 110, a signal processing unit 120, a memory 130, and a control unit 140. Note that the imaging device is an aspect of an image processing device.

  Note that the image processing apparatus of the present disclosure includes an apparatus such as a PC. An image processing apparatus such as a PC does not have the optical lens 105 and the image sensor 110 of the image pickup apparatus 100 shown in FIG. 3, and is composed of other components, and has an input unit for acquired data of the image sensor 100 or a storage unit. It becomes composition.

  Hereinafter, the imaging apparatus 100 illustrated in FIG. 3 will be described as a representative example of the image processing apparatus of the present disclosure. Note that the imaging apparatus 100 illustrated in FIG. 3 is, for example, a still camera or a video camera.

The imaging device (image sensor) 110 of the imaging apparatus 100 illustrated in FIG. 3 has a configuration including a color filter having a Bayer array composed of RGB arrays.
First, a configuration and processing using the image sensor 110 having an RGB array will be described.
A configuration and processing example using an image sensor having an RGBW array will be described later.

An imaging element (image sensor) 110 of the imaging apparatus 100 illustrated in FIG. 3 has an RGB array 181. That is,
Red (R) that transmits wavelengths near red,
Green (G) that transmits wavelengths in the vicinity of green,
Blue (B) that transmits wavelengths near blue,
It is an image sensor provided with a filter having these three types of spectral characteristics.

  The image sensor 110 having the RGB array 181 receives one of RGB light in units of pixels via the optical lens 105, and generates and outputs an electrical signal corresponding to the received light signal intensity by photoelectric conversion. With this image sensor 110, a mosaic image composed of three types of RGB spectra is obtained.

An output signal of the image sensor (image sensor) 110 is input to the image signal correction unit 200 of the signal processing unit 120.
The image signal correction unit 200 executes a blur correction process in consideration of, for example, a blur method that changes according to an image height (distance from the optical center) and a focus position. Details of the configuration and processing of the image signal correction unit 200 will be described later.

The RGB image subjected to the blur correction in the image signal correction unit 200 is output to the RGB signal processing unit 250.
The output of the image signal correction unit 200 is data having a Bayer array, similar to the output from the image sensor 110.

  The RGB signal processing unit 250 executes the same processing as the signal processing unit provided in a conventional camera or the like. Specifically, the color image 183 is generated by executing demosaic processing, white balance adjustment processing, γ correction processing, and the like. The generated color image 183 is recorded in the memory 130.

The control unit 140 executes control of these series of processes. For example, a program for executing a series of processes is stored in the memory 130, and the control unit 140 executes the program read from the memory 130 and controls the series of processes.
The memory 130 can be configured by various recording media such as a magnetic disk, an optical disk, and a flash memory.

A detailed configuration of the image signal correction unit 200 will be described with reference to FIG.
As illustrated in FIG. 4, the image signal correction unit 200 includes a line memory 201, an image height calculation unit 202, a saturation detection unit 203, a blur correction processing unit 210, an edge detection unit 221, and a blend processing unit 230.
The blend processing unit 230 includes a weighted addition processing unit 235.

The pixel value signal corresponding to each pixel output from the image sensor 110 is temporarily stored in the line memory 201.
Further, an xy address indicating the coordinate position of each pixel associated with the pixel value of each pixel is output to the image height calculation unit 202.
The line memory 201 has a line memory for seven horizontal lines of the image sensor. The line memory 201 sequentially outputs data for seven horizontal lines in parallel. The output destinations are the blur correction processing unit 210, the edge detection unit 221, and the saturation detection unit 203. The imaging data of the RGB array 181 is output in units of 7 lines to each of these processing units.

The edge detection unit 221 verifies the output signal from the line memory 201, generates edge information included in the image, for example, edge information including the edge direction and edge strength, and outputs the edge information to the blend processing unit 230.
Specifically, for example, the flatness (weightFlat) calculated from the pixel information of 7 × 7 pixels around the processing target pixel (center pixel of 7 × 7 pixels) is calculated and output to the blend processing unit 230.
This flatness (weightFlat) calculation process can be executed as a process similar to the process described in Japanese Patent Application Laid-Open No. 2011-55038, which is an earlier application of the present applicant.

  The blur correction processing unit 210 verifies the input signal (Rin, Gin, Bin) from the line memory 201, performs processing to reduce image blur, and blur correction signals (Rd, Gd, Bd) obtained as a result of the processing. ) Is calculated and output to the blend processing unit 230.

The blend processing unit 230 includes RGB signals (Rin, Gin, Bin) in the output signal from the line memory 201, edge information output from the edge detection unit 221, and a blur correction signal output from the blur correction processing unit 210. Input (Rd, Gd, Bd).
Using this information, the blend processing unit 230 generates an output signal RGB that has been subjected to blur correction in consideration of edges, and outputs the output signal RGB to the RGB signal processing unit 250.

The blend processing unit 230 includes a weighted addition processing unit 235, and the weighted addition processing unit 235 calculates the edge information of the processing target pixel calculated by the edge detection unit 221, that is, the processing target pixel (center pixel of 7 × 7 pixels). The weighted average processing of the blur correction signals (Rd, Gd, Bd) output from the blur correction processing unit 210 is executed according to the flatness (weightFlat) calculated from the pixel information of 7 × 7 pixels centering on , RGB pixel values of the RGB array 181 are calculated. Specifically, RGB pixel values are determined according to the following formula.
R = (weightFlat) × (Rd) + (1-weightFlat) × Rin
G = (weightFlat) × (Gd) + (1−weightFlat) × Gin
B = (weightFlat) × (Bd) + (1−weightFlat) × Bin
R, G, and B obtained as a calculation result of this expression are output to the RGB signal processing unit 250.

  The RGB signal processing unit 250 is the same as the signal processing unit for RGB array (Bayer array) signals of general cameras and image processing apparatuses. The RGB signal processing unit 250 performs signal processing on the RGB array (Bayer array) signal output from the blend processing unit 230 to generate a color image 183 (see FIG. 3). Specifically, the RGB signal processing unit 250 generates a color image 183 by executing, for example, white balance adjustment processing, demosaic processing, shading processing, RGB color matrix processing, γ correction processing, and the like.

FIG. 5 is a flowchart for explaining the overall sequence of processing executed by the image signal correction unit 200.
First, in step S101, a blur correction filter corresponding to the image height or the like is generated. This process is executed in the blur correction processing unit 210.
The blur correction processing unit 210 receives image height (distance from the image center (= optical center)) information calculated based on the xy address of the blur correction target pixel from the image height calculation unit 202, and sets the image height and the like. A corresponding blur correction filter is generated.
This specific process will be described later.

Next, in step S102, it is determined whether the blur correction target pixel is saturated.
Specifically, the saturation detection unit 203 shown in FIG. 4 performs saturation detection of the blur correction target pixel, and this detection information is input to the blur correction processing unit 210.
When the correction target pixel is saturated, there is a possibility that a false color is generated when blur correction is performed. This is because the blur correction process includes a process of setting the pixel value of the correction target pixel with reference to the pixel values of the pixels around the pixel that is the blur correction target.

For example, the pixel region 271a of the image shown in FIG. 6A is a pixel region including a highlight pixel, that is, a saturated pixel.
When blur correction processing is performed on a pixel area 271a including such saturated pixels, blur is reduced as shown in the pixel area 271b of FIG. 6B, but a color different from the original subject color is set. A false color is generated.
In order to prevent such false color generation, as shown in FIG. 7, when a saturation pixel is included around a blur correction target pixel, for example, a 7 × 7 pixel centering on the correction target pixel, the blur of the correction target pixel is detected. Set to not perform correction.

As described above, the blur correction processing unit 210 performs processing for replacing the saturation pixel region with a signal without blur correction processing according to the detection information of the saturation pixel (highlight) region in the saturation detection unit 203.
For example, the saturation detection unit 203 determines whether or not a predetermined threshold value is exceeded for a pixel in a 7 × 7 pixel region centered on the blur processing target pixel. If even one pixel in the 7 × 7 pixel region exceeds the threshold value, the saturation detection information is output to the blur correction processing unit 210.
The blur correction processing unit 210 performs processing of replacing with a signal without correction processing because there is a high possibility that a false color will occur if even one pixel in the 7 × 7 pixel region exceeds the threshold value.

This process corresponds to the process in the case where the determination in step S102 of the flow shown in FIG. 5 is Yes. In this case, the process proceeds to step S105 without executing steps S103 and S104.
That is, the blur correction process is not performed on the processing target pixel (the center pixel of 7 × 7 pixels).

If any of the pixels in the 7 × 7 pixel area centered on the blur processing target pixel does not exceed a predetermined threshold value and saturation is not detected, the determination in step S102 is No, and in step S103 The blur correction process with the filter applied is executed.
This blur correction process is a process executed in the blur correction processing unit 210 shown in FIG. 4, and the blur correction process is performed by applying the blur correction filter generated according to the image height generated in step S101.

Next, edge detection processing is executed in step S104.
This processing is executed in the edge detection unit 221 shown in FIG. The edge detection unit 221 verifies the output signal from the line memory 201 and generates edge information included in the image, for example, edge information including the edge direction and the edge strength.
Specifically, as described above, for example, the blend processing unit calculates the flatness (weightFlat) calculated from the pixel information of 7 × 7 pixels with the processing target pixel (the central pixel of 7 × 7 pixels) as the center. 230.

The flatness (weightFlat) has a value in the range of 0 to 1, as shown in FIG.
The closer to 1, the lower the flatness (the more texture)
The closer to 0, the higher the flatness (less texture)
This is an index value of flatness indicating such an image state.

As shown in FIG. 8, the flatness (weightFlat) is calculated as follows using two preset threshold values (Limit0, Limit1).
If 0 ≦ (ratioFlat) <Limit0,
Flatness (weightFlat) = 0
If Limit0 ≦ (ratioFlat) <Limit1,
Flatness (weightFlat) = 0-1
If Limit1 ≦ (ratioFlat),
Flatness (weightFlat) = 1
And

For example, the edge detection unit 221 outputs the above-described flatness (weightFlat) information to the blend processing unit 230 as edge information.
As described above, this flatness (weightFlat) calculation process can be executed as a process similar to the process described in Japanese Patent Application Laid-Open No. 2011-55038, which is an earlier application of the present applicant. See Japanese Patent Application Laid-Open No. 2011-55038.

Next, in step S105, a blend process is executed.
This process is a process executed by the blend processing unit 230 shown in FIG. The blend processing unit 230 includes RGB signals (Rin, Gin, Bin) in the output signal from the line memory 201, edge information output from the edge detection unit 221, and a blur correction signal output from the blur correction processing unit 210. Input (Rd, Gd, Bd).
Using this information, the blend processing unit 230 generates an output signal RGB that has been subjected to blur correction in consideration of edges, and outputs the output signal RGB to the RGB signal processing unit 250.

The blend processing unit 230 includes a weighted addition processing unit 235, and the weighted addition processing unit 235 calculates the edge information of the processing target pixel calculated by the edge detection unit 221, that is, the processing target pixel (center pixel of 7 × 7 pixels). The weighted average processing of the blur correction signals (Rd, Gd, Bd) output from the blur correction processing unit 210 is executed according to the flatness (weightFlat) calculated from the pixel information of 7 × 7 pixels centering on , RGB pixel values of the RGB array 181 are calculated. Specifically, RGB pixel values are determined according to the following formula.
R = (weightFlat) × (Rd) + (1-weightFlat) × Rin
G = (weightFlat) × (Gd) + (1−weightFlat) × Gin
B = (weightFlat) × (Bd) + (1−weightFlat) × Bin
R, G, and B obtained as a calculation result of this expression are output to the RGB signal processing unit 250.

Specifically, the weighted addition processing unit 235 of the blend processing unit 230 specifically outputs, for example, a blur correction signal at a pixel position where the edge strength is determined to be high based on the edge information of the processing target pixel calculated by the edge detection unit 221. Blend processing is performed in which the blend ratio is set high and the blend ratio of the signal without blur correction is set high at the pixel position where the edge strength is determined to be small.
This is because it is desirable that the signal is a sharp signal with a high-frequency component restored in the vicinity of the edge, but since the high-frequency component is originally not included in the flat portion, it is desirable that the signal be suppressed in noise. .

  However, as described above, when a saturated pixel is detected in a pixel unit of a predetermined unit, for example, a 7 × 7 pixel region centered on a pixel to be subjected to blur correction processing, blur correction is not performed for that pixel. The input pixel values Rin, Gin, Bin from the image sensor 110 are output as they are.

Finally, in step S106, it is determined whether or not processing for all input pixels has been completed. If there are unprocessed pixels, the processing from step S101 to step S105 is repeatedly executed for the unprocessed pixels.
If it is determined in step S106 that the processing has been completed for all input pixels, the processing of the image signal correction unit 200 ends.

[3. Details of blur correction processing of image processing apparatus according to first embodiment]
Next, a specific example of the blur correction process executed in the image signal correction unit 200 shown in FIG. 4 will be described.
The blur correction processing unit 210 of the image signal correction unit 200 shown in FIG. 4 verifies the input signal (Rin, Gin, Bin) from the line memory 201, performs processing to reduce image blur, and obtains the processing result. The blur correction signal (Rd, Gd, Bd) is calculated and output to the blend processing unit 230.

  The blur correction processing unit 210 generates a blur correction filter according to the image height or the like of the blur correction target pixel in step S101 of the flow illustrated in FIG. 5 and applies the generated filter to execute the blur correction process in step S103. .

FIG. 9 shows a flowchart for explaining a detailed sequence of the filter generation process in step S101 of FIG.
Details of the blur correction filter generation processing will be described with reference to the flowchart shown in FIG.

  First, in step S121, a high-pass filter corresponding to the xy address of the blur correction target pixel, that is, the coordinate (x, y), that is, a high-pass filter corresponding to the coordinate position (HPF_dist) is generated.

The blur correction processing unit 210 inputs the coordinates (x, y) and the image height (distance from the image center) of the pixel to be corrected from the image height calculation unit 202 shown in FIG. 4, and performs a coordinate position corresponding high pass filter (HPF_dist). ) Is generated.
This process will be described with reference to FIG.

  FIG. 10 shows a flowchart for explaining a generation processing sequence of the coordinate position corresponding high-pass filter (HPF_dist) and a diagram for explaining parameters such as parameters applied to this filter generation.

First, in step S131 of the flow illustrated in FIG. 10, the blur correction processing unit 210 determines according to the coordinates (x, y) of the blur correction target pixel.
(1) an x-direction adjusting high-pass filter (HPF_x);
(2) A process of blending the y-direction adjusting high-pass filter (HPF_y) is executed,
(3) Coordinate position reflection filter (HPF_xy)
Is generated.

In addition,
(1) x-direction adjusting high-pass filter (HPF_x),
(2) y-direction adjusting high-pass filter (HPF_y),
It is a filter held in a memory in the image processing apparatus, and is a filter that the blur correction processing unit 210 acquires from the memory.
Specific examples of these filters will be described with reference to FIG.

FIG. 11 shows the following three high-pass filters.
(A) High-pass filter that does not change the center of gravity (HPF_center)
(B) High-pass filter for adjusting the x direction (HPF_x) that changes the center of gravity in the x direction (horizontal direction)
(C) y-direction adjusting high-pass filter (HPF_y) for changing the center of gravity in the y-direction (vertical direction)

In step S131 of the flow shown in FIG.
Processing for blending (B) x-direction adjusting high-pass filter (HPF_x) and (C) y-direction adjusting high-pass filter (HPF_y) shown in FIG. 11 according to the coordinates (x, y) of the blur correction target pixel. Run
Coordinate position reflection filter (HPF_xy)
Is generated.

Specifically, according to the following formula, the coordinate position reflection filter (HPF_xy)
Is generated.
HPF_xy = (x × HPF_x + y × HPF_y) / (x + y)

  In step S131 in FIG. 10, the coordinate position reflection filter (HPF_xy) is generated in this manner according to the coordinates (x, y) of the blur correction target pixel.

Next, in step S132, based on the distance (r) between the coordinates (x, y) of the blur correction target pixel and the field angle center position [(x, y) = (0, 0)],
(1) a coordinate position reflection filter (HPF_xy);
(2) Center filter (HPF_center) corresponding to the view angle center position [(x, y) = (0, 0)]
Blend
(3) Coordinate position corresponding high pass filter (HPF_dist)
Is generated.

(1) The coordinate position reflection filter (HPF_xy) is a filter calculated in step S131.
(2) The center filter (HPF_center) corresponding to the center of view angle [(x, y) = (0, 0)] is the filter shown in FIG. 11A and is held in the memory in the image processing apparatus. The blur correction processing unit 210 acquires from the memory.

As shown in (a) parameter description in FIG.
When the coordinates of the blur correction target pixel are (x, y),
The distance r from the image center (0, 0) is calculated as the image height.
The maximum image height up to the field angle end (xmzx, ymax) is defined as rmax.

Under this setting, the blur correction processing unit 210
A coordinate position corresponding high pass filter (HPF_dist) is calculated according to the following equation.
HPF_dist = [(HPF_xy × r) + (HPF_center × (rmax−r))] / rmax
In step S132, the blur correction processing unit 210 calculates a coordinate position-corresponding high-pass filter (HPF_dist) according to the above processing.

The processing of steps S131 to S132 in the flow of FIG. 10 is the processing in step S121 of the flow shown in FIG.
By these processes, a coordinate position corresponding high pass filter (HPF_dist) is generated.

Next, in step S122 of the flow shown in FIG.
(1) a basic filter (default filter);
(2) The coordinate position corresponding high pass filter (HPF_dist) is
Blending according to the image height (r) and the blend coefficient α (focus, r) determined according to the focus position information (focus),
(3) Basic filter for coordinate position (Fbase),
Is generated.

(1) A basic filter (default filter) is a filter held in a memory in the image processing apparatus, and is a filter that the blur correction processing unit 210 acquires from the memory.
Specifically, a filter called the least square filter or the Wiener filter described above with reference to FIG. 2 can be applied.

The image processing apparatus holds a blur correction filter such as a Wiener filter in a memory as a basic filter (default filter).
In step S122 of the flowchart shown in FIG. 9, the basic filter (default filter) and the coordinate position corresponding high pass filter (HPF_dist) generated in step S121 are blended to generate a coordinate position corresponding basic filter (Fbase).

The coordinate position-corresponding basic filter (Fbase) is calculated by the following formula based on the blend height α determined according to the image height (r) and the focus position information (focus).
Fbase = (1.0−α) × (default filter) + α (HPF_dist)

Furthermore, in step 123, the intensity adjustment parameter β (focus, r) corresponding to the image height and the focus position is applied to the coordinate position corresponding basic filter (Fbase) generated in step S122, and the enhancement intensity adjustment is performed. A blur correction filter (Ffinal) to be applied to actual blur correction is generated.
The blur correction filter (Ffinal) generated in step S123 is a filter applied to the actual blur correction processing for the pixel of the processing target pixel (x, y).

First, the process executed in step S122, that is,
(1) a basic filter (default filter);
(2) The coordinate position corresponding high pass filter (HPF_dist) is
Blending according to the image height (r) and the blend coefficient α (focus, r) determined according to the focus position information (focus),
(3) Basic filter for coordinate position (Fbase),
Is generated.
This process will be described with reference to FIG.

(1) a basic filter (default filter);
(2) With the coordinate position corresponding high pass filter (HPF_dist),
The blending process based on the blending coefficient α determined according to the image height (r) and the focus position information (focus) is a process corresponding to the filter frequency characteristic adjustment (tuning) process.

That is, it is performed as a process of changing the frequency characteristics of the filter based on the image height (r) and the blend coefficient α determined according to the focus position information (focus).
The graph shown in FIG. 12 is a diagram showing the enhancement characteristics of a filter. The horizontal axis represents the spatial frequency, and the vertical axis represents the intensity (Magnitude) corresponding to the pixel value output after applying the filter.
It shows the enhancement characteristic of the filter generated at various settings (α = −0.5 to 1.5) of the blend coefficient α applied to the blending process described above.

As described above, the coordinate position-corresponding basic filter (Fbase) is calculated by the following equation based on the image height (r) and the blend coefficient α determined according to the focus position information (focus).
Fbase = (1.0−α) × (default filter) + α (HPF_dist)

In the above formula,
Blend coefficient: When α = 0,
Fbase = (default filter)
And
A line filter having the characteristic of α = 0.0 shown in FIG. 12 is obtained.

In the above formula,
Blend coefficient: When α = 1.0,
Fbase = (HPF_dist)
And
A line filter having the characteristic of α = 1.0 shown in FIG. 12 is obtained.

That is, by changing α to various values, for example, as shown in FIG.
A filter having each frequency characteristic of α = −0.5 to 1.5 can be generated.
Thus, the generation of the coordinate position corresponding basic filter (Fbase) executed in step S122 of the flow shown in FIG.
(1) a basic filter (default filter);
(2) With the coordinate position corresponding high pass filter (HPF_dist),
It is executed as a process corresponding to a filter frequency characteristic adjustment (tuning) process by blending.

The blend coefficient α is determined according to the image height (r) and the focus position information (focus).
A setting example of the blend coefficient α will be described with reference to FIG.

FIG. 13 shows the following figures.
(A) Lens MTF characteristics (b) Wiener filter characteristics (c) Setting example of blend coefficient α

As shown in FIG. 13A, in general, the MTF characteristics deteriorate as the image height increases (the distance from the image center increases). In particular, since the high frequency region is a region buried in noise, as shown in FIG. 13B, the enhancement characteristic of the Wiener filter is shifted to the low frequency region side.
Therefore, as shown in FIG. 13C, as the image height increases (the distance from the image center increases), the blend ratio of the high-pass filter is decreased. That is, it is preferable to reduce the blend coefficient α.
The same can be said about the focus shift. That is, the blend coefficient α is decreased as the focus shift increases.

  The focus shift is a value unique to the image processing apparatus (camera), and the shift amount generated corresponding to each focus position is a unique value in each camera. Accordingly, the focus shift amount can be calculated from the focus position information, and the blur correction unit 210 receives the focus position information (see FIG. 4) from the control unit 140 shown in FIG. A deviation amount is calculated, and a blend coefficient α (focus, r) is calculated using the focus deviation amount and the image height r acquired from the image height calculation unit 202.

As described above, the blur correction processing unit 210 in step S122 of the flow shown in FIG.
(1) a basic filter (default filter);
(2) The coordinate position corresponding high pass filter (HPF_dist) is
Blending according to the image height (r) and the blend coefficient α (focus, r) determined according to the focus position information (focus),
(3) Basic filter for coordinate position (Fbase),
Is generated.

Next, the process of step S123, that is,
A blur correction filter (Ffinal) whose enhancement intensity is adjusted by applying an intensity adjustment parameter β (focus, r) corresponding to the image height and focus position to the coordinate position corresponding basic filter (Fbase) generated in step S122. The process to generate will be described.

The process of step S123 is
(A) Multiplying the band separation process shown in FIG.
(B) Resynthesis process shown in FIG.
These processes are performed.

First, the band separation process shown in FIG. 14 will be described.
The coordinate position-corresponding basic filter (Fbase) shown in FIG. 14 is a filter generated in the process of step S122 of the flow shown in FIG.
First, the coordinate position corresponding basic filter (Fbase) is band-separated into a direct current component and an alternating current component and separated into two filters.
The DC component is a filter in which only the center of the filter of 7 × 7 pixels is set to coefficient = 1.0, and the coefficients of other pixels are all set to 0.
The AC component is a filter having a coefficient obtained by subtracting the coefficient of the DC component from the coefficient of the coordinate position corresponding basic filter (Fbase).

Further, the AC component is multiplied by an enhancement strength (= strength adjustment parameter β).
A blur correction filter (Ffinal) to be applied to the final blur correction shown in FIG. 15 is generated by the multiplication result of the AC component and the enhancement intensity and the recombining process of adding the DC component.
The enhancement strength corresponds to the strength adjustment parameter β.
In the example shown in FIGS.
An example of the enhancement strength adjustment parameter β = 1.5 is shown.

  The enhancement intensity adjustment is performed by a synthesis process in which the filter coefficient of the AC component generated by the band separation process shown in FIG. 14 is multiplied by the enhancement intensity adjustment parameter β = 1.5, and the corresponding coefficient and the corresponding coefficient for the DC property are added. The final blur correction filter (Ffinal) having the coefficients shown in FIG. 15 is generated.

The setting of the enhancement strength adjustment parameter β and the change in filter characteristics will be described with reference to FIG.
FIG. 16 is a diagram showing the enhancement characteristics of the filter as described above with reference to FIG. 12. The horizontal axis represents the spatial frequency, the vertical axis represents the intensity corresponding to the pixel value output after the filter application ( Magnitude).

It is assumed that the enhancement characteristic of the coordinate position-corresponding basic filter (Fbase) has a characteristic indicated by a dotted line shown in the figure, for example.
With respect to the coordinate position-corresponding basic filter (Fbase) having this characteristic, it has been described with reference to FIGS.
(S1) Band separation of the DC component and the AC component;
(S2) Multiplying the AC component to which the enhancement strength adjustment parameter β = 1.5 is applied;
(S3) Recombining process by adding the multiplication result of the AC component and the DC component,
As a result of the processing of (s1) to (s3), the blur correction filter (Ffinal) having the characteristics shown by the solid line in FIG.
Can be generated.

The example shown in FIG.
Enhancement strength adjustment parameter β = 1.5
Is an example of
By changing the value of β, it is possible to generate a blur correction filter (Ffinal) having various settings.

The enhancement intensity adjustment parameter β is determined according to the image height (r) and the focus position information (focus), as with the blend coefficient α described above.
A setting example of the enhancement strength adjustment parameter β will be described with reference to FIG.

FIG. 17 shows the following figures.
(A) Lens MTF characteristics (b) Wiener filter characteristics (c) Setting example of enhancement strength adjustment parameter β

As shown in FIG. 17A, generally, the MTF characteristics deteriorate as the image height increases (the distance from the image center increases). Therefore, as shown in FIG. 17B, the enhancement strength of the Wiener filter is also increased.
Therefore, as shown in FIG. 17C, it is preferable to increase the enhancement intensity adjustment parameter β as the image height increases (the distance from the image center increases).
The same can be said about the focus shift. That is, the enhancement intensity adjustment parameter β is increased as the focus shift increases.

  As described above, the focus shift is a value unique to the image processing apparatus (camera), and a shift amount generated corresponding to each focus position is a unique value in each camera. Accordingly, the focus shift amount can be calculated from the focus position information, and the blur correction unit 210 receives the focus position information (see FIG. 4) from the control unit 140 shown in FIG. The shift amount is calculated, and the enhancement intensity adjustment parameter β (focus, r) is calculated using the focus shift amount and the image height r acquired from the image height calculation unit 202.

As described above, the blur correction processing unit 210 in step S123 of the flow shown in FIG.
A blur correction filter (Ffinal) whose enhancement intensity is adjusted by applying an intensity adjustment parameter β (focus, r) corresponding to the image height and focus position to the coordinate position corresponding basic filter (Fbase) generated in step S122. Generate.

The blur correction filter (Ffinal) generated in step S123 of the flow shown in FIG. 9 is generated in step S101 of the overall processing flow shown in FIG. 5 described as the overall processing sequence of the image signal correction unit 200 shown in FIG. It is a correction filter.
This blur correction filter is a filter according to the image height and the focus position, and is a filter that is sequentially generated according to each correction pixel position and each focus position.
The blur correction filter (Ffinal) generated in step S101 of the flow shown in FIG. 5 is applied in step S103, and blur correction processing for each pixel is executed.
This filter application process is the same as the process described above with reference to FIG.
The blur correction filter (Ffinal) generated by the above-described processing is applied instead of the (b) Wiener filter shown in FIG.

That is, the pixel of a pixel region of a predetermined unit centered on the pixel to be corrected (for example, 7 × 7 pixels) is multiplied by the coefficient of the blur correction filter (Ffinal) generated by the above processing and added. A convolution operation is executed to calculate a corrected pixel value of the center pixel.
As described above, when saturation is detected in step S102 shown in the flow of FIG. 5, the blur correction process is omitted.
By pixel value correction using the blur correction filter (Ffinal) generated by the above-described process when a saturated pixel is not detected in a pixel area of a predetermined unit pixel area (for example, 7 × 7 pixels) centering on the correction target pixel A blur correction process is executed.

  As described above, the blur correction filter (Ffinal) generated in the present embodiment is a filter whose coefficient is adjusted in consideration of the pixel position to be corrected, specifically the image height and the focus position. That is, it is possible to perform optimum blur correction according to each pixel position and focus position by using a filter whose frequency characteristics and intensity characteristics are adjusted according to the image height and the focus position.

  In the above-described embodiment, the blur correction filter (Ffinal) finally generated according to the flow shown in FIGS. 8 and 9 is used as the blur correction filter generated in step S101 of the flow shown in FIG. The filter (Ffinal) has been described as a process executed by the blur correction process of each pixel by applying it in step S103 shown in FIG.

  However, for example, step S123 in the flow shown in FIG. 9 is omitted, and in step S122, the coordinate position corresponding basic filter (Fbase) generated based on the blend coefficient α (focus, r) is applied as the final blur correction filter. May be.

  Alternatively, step S121 and step S122 in the flow shown in FIG. 9 may be omitted, and the blur correction filter may be generated only by the process of step S123. That is, for the basic filter (default filter) held in the pre-memory, the process of step 123 of the flow shown in FIG. 9, that is, the band separation process and the enhancement strength adjustment parameter β described with reference to FIGS. A filter generated by executing the intensity adjustment of the alternating current component to which (focus, r) is applied and the resynthesis process may be applied to the blur correction.

  In the above embodiment, the configuration example in which the blend coefficient α and the enhancement intensity adjustment parameter β are determined by applying the image height (r) and the focus position information (focus) has been described. The intensity adjustment parameter β may be determined depending on only the image height. Alternatively, it may be determined depending only on the focus position information (focus). Furthermore, the configuration may be determined in consideration of other parameters such as a zoom position.

[4. Second Embodiment Regarding Configuration Example of Image Processing Device that Performs Blur Correction Considering Color Ratio]
In the embodiment described above, the image sensor has a configuration having an RGB array. In recent years, an image pickup device including a filter having an RGBW array in which W (white) that transmits visible light including RGB in addition to RGB is added is widely used.

As described above, when a filter having an RGBW arrangement including such W (White) pixels is used, the W pixel transmits visible light having a wide wavelength region, and thus the blur is different depending on the color of the subject.
The above-mentioned Patent Document 4 (Japanese Patent Laid-Open No. 2011-055038) filed earlier by the present applicant also discloses a blur correction processing configuration for a photographed image having an RGBW pixel array, but this is disclosed in this prior application. The processing is a configuration in which the correction is performed without considering the color ratio in the blur correction for the W (White) pixel, and a sufficient correction effect cannot be obtained for the W pixel that generates a different blur depending on the color ratio. There is a case.
Hereinafter, an embodiment that solves such a problem will be described.

FIG. 18 is a diagram showing the spectral intensity characteristics of each color in the RGBW filter.
R (red) efficiently transmits light having a wavelength corresponding to red near 600 nm.
G (green) efficiently transmits light having a wavelength corresponding to green around 530 nm.
B (blue) efficiently transmits light having a wavelength corresponding to blue around 430 nm.
For these RGB filters,
W (white) transmits all the R, G, and B wavelengths.

FIG. 19 is a diagram showing the MTF characteristics of each color of RGBW. The horizontal axis shows the spatial frequency, and the vertical axis shows the MTF. The incident light is assumed to be white.
The higher the MTF, the less blur, and the lower the MTF, the greater the degree of blur.
For example, green (G) has a high value of MTF over a wide spatial frequency and less blur for each of the other colors RG. This means that the blur is reduced when the G (green) component is large in the incident light, and the blur is increased when the R (red) and B (blue) are large.

  In the case of the configuration using the RGB array of the Bayer array, blur correction can be performed according to the individual RGB characteristics by using a filter corresponding to the MTF characteristics of each RGB color. That is, if the blur correction filters for R, G, and B are applied to the blur correction for each pixel of R, G, and B, appropriate correction corresponding to each MTF characteristic of RGB can be performed.

However, in the case of W (White), the wavelength light input through the W filter differs depending on the color of the subject. As a result, the MTF characteristics also change depending on the colors of the subject and the light source, and the blurring method varies depending on the colors of the subject and the light source. Therefore, even if one filter is applied, it is difficult to perform appropriate correction.
Note that the W (White) MTF characteristic shown in FIG. 19 is merely an example of the case where the incident light is white light, and the W (White) MTF characteristic changes greatly depending on the color of the subject.

Hereinafter, in order to solve this problem, a configuration example in which blur correction is performed in consideration of the color ratio will be described.
An image processing apparatus (imaging apparatus) described below executes processing on an input signal from an imaging element using a filter having an RGBW array.
In addition to the blur correction process, the image processing apparatus executes a process of converting an RGBW image into an RGB image.

First, conversion processing of an RGBW image into an RGB image will be described with reference to FIG.
The image processing apparatus according to this embodiment includes an RGBW color filter including white (W: White) that transmits all the RGB wavelength light in addition to the RGB filter that selectively transmits the wavelength light of each RGB color. Processing is performed on acquired data of (image sensor).

  Specifically, as shown in FIG. 20A, acquired data of an image sensor having, for example, an RGBW type color filter including white (W: White) is converted into an RGB array (Bayer array) shown in FIG. ) Is executed. Further, in this conversion process, a process for reducing the occurrence of blur and false colors is also executed.

Specifically, in the conversion process from the RGBW array to the RGB Bayer array, the following five conversions and correction processes are executed.
Convert W pixel position to G pixel (estimate G pixel value) = (GonW)
Convert G pixel position to R pixel (estimate R pixel value) = (RonG)
Convert G pixel position to B pixel (estimate B pixel value) = (BonG)
Convert R pixel position to R pixel (correct R pixel value) = (RonR)
Convert to B pixel at B pixel position (correct B pixel value) = (BonB)

  Each conversion process described above is performed as a pixel value estimation or correction process for converting each RGBW pixel in the RGBW array into an RGB pixel in the RGB array. By executing these processes, the RGB Bayer array shown in FIG. 20 (2) is generated from the RGBW color array shown in FIG. 20 (1).

Hereinafter, such color array conversion processing is referred to as re-mosaic processing.
In the following embodiment, a re-mosaic process for converting an RGBW color array having white (W) into an RGB-type color array (Bayer array) is executed, and in the re-mosaic process, blurring or false colors are generated. A configuration for executing the processing to be reduced will be described.

  FIG. 21 is a diagram illustrating a configuration example of an imaging apparatus 300 that is an example of the image processing apparatus according to the present embodiment. The imaging apparatus 300 includes an optical lens 3105, an imaging element (image sensor) 310, a signal processing unit 320, a memory 330, and a control unit 340. These basic configurations are the same as those described above with reference to FIG.

  Note that the imaging device is an aspect of an image processing device. The image processing apparatus of the present disclosure includes an apparatus such as a PC. The image processing apparatus such as a PC does not have the optical lens 305 and the imaging element 310 of the imaging apparatus 300 shown in FIG. It becomes composition.

An imaging device 300 illustrated in FIG. 21 is different from the imaging device 100 illustrated in FIG. 3 described above, and an imaging element (image sensor) 310 includes a filter having an RGBW array. That is,
Red (R) that transmits wavelengths near red,
Green (G) that transmits wavelengths in the vicinity of green,
Blue (B) that transmits wavelengths near blue,
In addition to these,
And white (W) that transmits all of RGB
This is an image sensor provided with filters having these four types of spectral characteristics.

  The image sensor 310 having the RGBW array 381 receives any of the RGBW light in units of pixels via the optical lens 305, and generates and outputs an electrical signal corresponding to the received light signal intensity by photoelectric conversion. A mosaic image composed of four types of RGBW spectra can be obtained by the image sensor 310.

An output signal of the image sensor (image sensor) 310 is input to the image signal correction unit 400 of the signal processing unit 320.
As in the previous embodiment, the image signal correction unit 400 considers, for example, the blurring that changes according to the image height (distance from the optical center) and the focus position, and further performs blur correction in consideration of the color ratio. .
The image signal correction unit 400 further executes a conversion process from the RGBW array 381 to the RGB array 382. Specifically, the following conversion process described above with reference to FIG. 20 is performed.
Convert W pixel position to G pixel (estimate G pixel value) = (GonW)
Convert G pixel position to R pixel (estimate R pixel value) = (RonG)
Convert G pixel position to B pixel (estimate B pixel value) = (BonG)
Convert R pixel position to R pixel (correct R pixel value) = (RonR)
Convert to B pixel at B pixel position (correct B pixel value) = (BonB)
These five conversions and correction processes are executed.
In this conversion / correction processing, processing for suppressing false color and blur is also executed.

The RGB image 382 that has been subjected to blur correction and data conversion in the image signal correction unit 400 is output to the RGB signal processing unit 450.
The RGB signal processing unit 450 performs the same processing as the signal processing unit provided in a conventional camera or the like. Specifically, a color image 383 is generated by executing demosaic processing, white balance adjustment processing, γ correction processing, and the like. The generated color image 383 is recorded in the memory 330.

The control unit 340 controls these series of processes. For example, a program for executing a series of processing is stored in the memory 330, and the control unit 340 executes the program read from the memory 330 and controls the series of processing.
The memory 330 can be configured by various recording media such as a magnetic disk, an optical disk, and a flash memory.

A detailed configuration of the image signal correction unit 400 will be described with reference to FIG.
As shown in FIG. 22, the image signal correction unit 400 includes a line memory 401, an image height calculation unit 402, a saturation detection unit 403, a blur correction processing unit 410, an edge detection unit 421, and a color correlation re-mosaic processing unit (data conversion unit). 430.
The color correlation re-mosaic processing unit (data conversion unit) 430 includes a W position G interpolation parameter calculation unit 431, a G position RB interpolation parameter calculation unit 432, an R position R interpolation parameter calculation unit 433, a B position B interpolation parameter calculation unit 434, and a weighting. An adder 435 is included.

The pixel value signal corresponding to each pixel output from the image sensor 310 is temporarily stored in the line memory 401.
Further, an xy address indicating the coordinate position of each pixel associated with the pixel value of each pixel is output to the image height calculation unit 402.
The line memory 401 has a line memory for seven horizontal lines of the image sensor. The line memory 401 sequentially outputs data for seven horizontal lines in parallel. Output destinations are the blur correction processing unit 410, the edge detection unit 421, and the saturation detection unit 403. The imaging data of the RGBW array 381 is output to each of these processing units in units of 7 lines.

Note that the edge detection unit 421 and the blur correction processing unit 410 execute processing on the white (W) signal in the imaging data of the RGBW array 381.
The color correlation re-mosaic processing unit 430 executes processing using all RGBW signals in the imaging data of the RGBW array 381.

The edge detection unit 421 verifies the discrete white (W) signal included in the output signal from the line memory 401 and generates edge information included in the image, for example, edge information including the edge direction and the edge strength. The result is output to the color correlation re-mosaic processing unit 430.
Specifically, for example, the flatness (weightFlat) calculated from the pixel information of 7 × 7 pixels with the processing target pixel (the center pixel of 7 × 7 pixels) as the center is calculated and output to the color correlation re-mosaic processing unit 430. To do.
This flatness (weightFlat) calculation process can be executed as a process similar to the process described in Japanese Patent Application Laid-Open No. 2011-55038, which is an earlier application of the present applicant.

  The blur correction processing unit 410 first applies white (W) pixels included in the output signal from the line memory 401, sets W pixels at RGB pixel positions by interpolation processing, and sets W pixels for all pixels. Generate an image. Further, a blur correction process is performed on each W pixel of an image in which W pixels are set for all pixels to generate a blur correction W signal for all pixels, that is, a blur correction W signal (Wd), and a color correlation re-mosaic processing unit Output to 430.

The blur correction processing unit 410 performs blur correction considering at least the color ratio.
Further, as in the above-described embodiment, for example, a blur correction method that takes into account the image height (distance from the optical center) and the focus position and the color ratio is also considered. Good.

The color correlation re-mosaic processing unit 430 includes the RGBW signal in the output signal from the line memory 401, the edge information output from the edge detection unit 421, and the blur correction W signal corresponding to all pixels output from the blur correction processing unit 410. Enter (Wd).
The color correlation re-mosaic processing unit 430 performs conversion processing from the RGBW color array to the RGB array 382 using these pieces of information.

Specifically, as described above with reference to FIG.
Convert W pixel position to G pixel (estimate G pixel value) = (GonW)
Convert G pixel position to R pixel (estimate R pixel value) = (RonG)
Convert G pixel position to B pixel (estimate B pixel value) = (BonG)
Convert R pixel position to R pixel (correct R pixel value) = (RonR)
Convert to B pixel at B pixel position (correct B pixel value) = (BonB)
These five conversions and correction processes are executed.

The W position G interpolation parameter calculation unit 431 calculates an interpolation parameter applied to calculation of a G pixel value set at the W pixel position of the RGBW array 381. Using the W signal and B signal in the input signal from the line memory 401 and the blur correction W signal (Wd) input from the blur correction processing unit 410, an interpolation parameter to be applied to the above (GonW) process is calculated.
The G position RB interpolation parameter calculation unit 432 calculates an interpolation parameter applied to the calculation of the R pixel value or the B pixel value set at the G pixel position of the RGBW array 381. Using the W signal and the B signal in the input signal from the line memory 401 and the blur correction W signal (Wd) input from the blur correction processing unit 410, parameters to be applied to the processing of the above (RonG) and (BonG) are set. calculate.

The R position R interpolation parameter calculation unit 433 calculates an interpolation parameter that is applied to calculation of a corrected R pixel value set at the R pixel position of the RGBW array 381. Using the W signal and the B signal in the input signal from the line memory 401 and the blur correction W signal (Wd) input from the blur correction processing unit 410, a parameter to be applied to the above (RonR) process is calculated.
The B position B interpolation parameter calculation unit 434 calculates an interpolation parameter to be applied to the calculation of the corrected B pixel value set at the B pixel position of the RGBW array 381. Using the W signal and B signal in the input signal from the line memory 401 and the blur correction W signal (Wd) input from the blur correction processing unit 410, parameters to be applied to the above (BonB) process are calculated.

The weighted addition processing unit 435 receives the interpolation parameters calculated by the interpolation parameter calculation units 431 to 434, and further applies the edge information input from the edge detection unit 421 to configure the RGB array (Bayer array) 382. The RGB signal value of each pixel is calculated.
The data conversion processing from the RGBW array to the RGB array performed by the color correlation re-mosaic processing unit (data conversion unit) 430 is basically described in Japanese Patent Application Laid-Open No. 2011-55038, which is a prior application of the present applicant. Processing is available. For details of this data conversion processing, refer to Japanese Patent Application Laid-Open No. 2011-55038.
In this way, an RGB array (Bayer array) 382 composed of Gr, Rr, and Br calculated by the weighted addition processing unit 435 is generated and provided to the RGB signal processing unit 450.

  The RGB signal processing unit 450 is the same as the signal processing unit for RGB array (Bayer array) signals of general cameras and image processing apparatuses. The RGB signal processing unit 450 performs signal processing on the RGB array (Bayer array) 382 output from the weighted average unit 435 to generate a color image 383 (see FIG. 20). Specifically, the RGB signal processing unit 450 generates a color image 383 by executing, for example, white balance adjustment processing, demosaic processing, shading processing, RGB color matrix processing, and γ correction processing.

[5. Details of blur correction processing of image processing apparatus according to embodiment 2]
Next, details of the blur correction processing of the image processing apparatus according to the second embodiment will be described.
FIG. 23 is a flowchart for explaining the overall sequence of processing executed by the image signal correction unit 400.
The main processing entities of the processing of each step in the flowchart shown in FIG. 23 are as follows.
The processing in steps S301 to S305 is mainly processing executed by the blur correction processing unit 410.
The process of step S306 is a process executed by the edge detection unit 421.
The process of step S307 is a process executed by the color correlation re-mosaic processing unit (data conversion unit) 430.

  Further, the processing of steps S301 to S305 executed by the blur correction processing unit 410 and the processing of step S305 executed by the edge detection unit 421 are executed using the W pixels included in the image data of the RGBW array as described above. Is done.

Details of the processing of each step will be described.
First, in step S301, the blur correction processing unit 410 calculates a color ratio of a predetermined local area centered on one W pixel that is a blur correction target.

As in the previous embodiment, the blur correction processing unit 410 inputs a predetermined pixel area (= local area) centered on a pixel that is a blur correction process target, for example, a rectangular pixel area of 7 × 7 pixels. Process.
When the central pixel of the 7 × 7 pixel area is a W pixel, blur correction processing is performed on the W pixel.
If the center pixel of the 7 × 7 pixel area is not a W pixel, a blur correction process is performed on the interpolated W pixel value set based on the pixel values of surrounding W pixels with respect to the center position.
That is, as the preprocessing in step S301, the blur correction processing unit 410 applies white (W) pixels included in the output signal from the line memory 401, and sets W pixels at RGB pixel positions other than W pixels by interpolation processing. Then, image data of all pixels W is generated. From the image data of all the W pixels, W pixels are sequentially selected one by one, and blur correction processing is executed.

In step S301, the blur correction processing unit 410 calculates the color ratio (R (red) ratio, G (green) ratio, B (blue) ratio) of the 7 × 7 pixel area centered on the W pixel that is the blur correction target. Calculated by the following formula.
R ratio = mR / (mG + mR + mB)
G ratio = mG / (mG + mR + mB)
B ratio = mB / (mG + mR + mB)
However,
mR is the low-frequency component of the R signal in the local region (eg, 7 × 7 pixel region) mG is the low-frequency component of the G signal in the local region (eg, 7 × 7 pixel region) mB is the local region (eg, 7 × 7 pixel region) This is the low frequency component of the B signal in the pixel area).

In addition, when calculating R (red) ratio, G (green) ratio, and B (blue) ratio according to the above formula,
It is necessary to calculate the low frequency components mR, mG, mB of the R, G, B signals in the local region (for example, 7 × 7 pixel region).
In order to calculate these low-frequency components corresponding to each color, for example, processing using a low-frequency component calculation filter shown in FIGS.

  The low-frequency component calculation filters corresponding to RGB colors shown in FIGS. 24 to 27 output images from the 7 × 7 pixel image sensor 310 centered on the W pixels (including the interpolated W pixels) to be blurred. It is a filter applied to the filter process with respect to data. The low-frequency components mR, mG, and mB of the R, G, and B signals are calculated by applying to the output image data from the image sensor 310, not the interpolated image in which all pixels are W pixels by the interpolation processing.

The low frequency component calculation filter has different settings depending on the configuration of 7 × 7 pixels.
24 to 27 show the following three filters to be selectively applied according to the following 7 × 7 pixel configurations.
A low frequency component (mG) calculation filter of the G signal,
A low frequency component (mR) calculation filter of the R signal;
A low frequency component (mB) calculation filter of the B signal,

FIG. 24 shows R, G, B when the output image data from the 7 × 7 pixel image sensor 310 centering on the W pixel (including the interpolated W pixel) to be blurred is set as follows. The structure of the low frequency component calculation filter which calculates the low frequency components mR, mG, mB of each signal is shown.
(A) Low frequency component calculation filter (mR calculation filter 511, mG calculation filter 512, mB calculation filter 513) applied when the center pixel W is a G pixel on the left and a B pixel on the right
(B) Low frequency component calculation filters (mR calculation filter 514, mG calculation filter 515, mB calculation filter 516) applied when the left side of the center pixel W is a G pixel and the right side is an R pixel

  The numerical value (coefficient) shown in each filter is set only at the setting position of the color (R, G, or B) that is a low frequency component calculation target. Each coefficient value is multiplied by the pixel value at each corresponding pixel position of 7 × 7 pixels input from the image sensor, and the multiplied value is added and divided by the values (32, 25, etc.) shown below each filter. Thus, the low frequency component is calculated.

FIG. 25 shows R, G, B when the output image data from the image sensor 310 of 7 × 7 pixels centering on W pixels (including interpolated W pixels) subject to blur correction is set as follows. The structure of the low frequency component calculation filter which calculates the low frequency components mR, mG, mB of each signal is shown.
(C) Low frequency component calculation filters (mR calculation filter 521, mG calculation filter 522, mB calculation filter 523) applied when the right side of the center pixel W is a G pixel and the upper side is a B pixel.
(D) Low frequency component calculation filters (mR calculation filter 524, mG calculation filter 525, mB calculation filter 526) applied when the right side of the center pixel W is a G pixel and the upper side is an R pixel

FIG. 26 shows R, G, B when the output image data from the 7 × 7 pixel image sensor 310 centering on the W pixel (including the interpolated W pixel) subject to blur correction is set as follows. The structure of the low frequency component calculation filter which calculates the low frequency components mR, mG, mB of each signal is shown.
(E) Low frequency component calculation filters (mR calculation filter 541, mG calculation filter 542, mB calculation filter 543) applied when there is an R pixel in the upper left to lower right diagonal line of the center pixel G
(F) Low frequency component calculation filter (mR calculation filter 544, mG calculation filter 545, mB calculation filter 546) applied when B pixel exists in the upper left to lower right diagonal line of the center pixel G

  The example shown in FIG. 26 is an example in which the central pixel of 7 × 7 pixels input from the image sensor is not a W pixel. In this case, the blur correction process is performed on the interpolated image set for all the pixels W described above.

FIG. 27 shows R, G, B when the output image data from the 7 × 7 pixel image sensor 310 centered on the W pixel (including the interpolated W pixel) to be corrected for blur is as follows. The structure of the low frequency component calculation filter which calculates the low frequency components mR, mG, mB of each signal is shown.
(G) Low frequency component calculation filters (mR calculation filter 551, mG calculation filter 552, mB calculation filter 553) applied when there is an R pixel at the center pixel
(H) Low frequency component calculation filters (mR calculation filter 561, mG calculation filter 562, mB calculation filter 563) applied when there is a B pixel at the center pixel

  The example shown in FIG. 27 is also an example in which the central pixel of 7 × 7 pixels input from the image sensor is not a W pixel. In this case, the blur correction process is performed on the interpolated image set for all the pixels W described above.

In step S301, the blur correction processing unit 410 selectively applies the low-frequency component calculation filters illustrated in FIGS. 24 to 27 in accordance with the pixel pattern of the 7 × 7 pixel region centered on the W pixel that is the blur correction target. Thus, the low frequency components mR, mG, and mB of the RGB signals in the local region (for example, 7 × 7 pixel region) are calculated.
Furthermore, using these low-frequency components of each color, a color ratio (R (red) ratio, G) corresponding to the W pixel (including the case of the interpolation pixel) at the center of the 7 × 7 pixel area that is the blur correction target. (Green) ratio and B (blue) ratio) are calculated according to the following equations.
R ratio = mR / (mG + mR + mB)
G ratio = mG / (mG + mR + mB)
B ratio = mB / (mG + mR + mB)

Next, in step S302, the blur correction processing unit 410 applies the color ratio calculated in step S301 to generate a blur correction filter corresponding to the color ratio.
This process will be described with reference to FIG.

The formula shown in FIG. 28 is a formula for generating a blur correction filter corresponding to the color ratio.
R correspondence blur correction filter,
G-adaptive blur correction filter,
B-adaptive blur correction filter,
These three blur correction filters are blur correction filters corresponding to the respective colors stored in advance in the memory of the image processing apparatus.
These are filters that can execute, for example, optimum processing for blur correction corresponding to each color in accordance with the MTF characteristics of each color described above with reference to FIG.

  In the blur correction process of the present embodiment, the blur correction filter applied to the W (White) pixel is the RGB-compatible blur correction filter and the color ratio calculated in step S301, that is, the R (red) ratio, G ( The calculation is performed by applying the green) ratio, the B (blue) ratio, and these color ratios.

Specifically, as shown in FIG. 28, a W-corresponding blur correction filter (Wdeblur) is calculated according to the following equation.
W _deblurr
= (MR / (mG + mR + mB)) / R _deblurr
+ (MG / (mG + mR + mB)) / G _deblurr
+ (MB / (mG + mR + mB)) / B _deblurr
In addition,
R _deblur : R correspondence blur correction filter G _deblurr : G correspondence blur correction filter B _deblurr : B correspondence blur correction filter mR / (mG + mR + mB): R (red) ratio mG / (mG + mR + mB): G (green) ratio mB / (mG + mR + mB ): B (blue) ratio.

  In this way, the blur correction filter corresponding to each color is blended according to the color ratio of a predetermined pixel region (in this example, 7 × 7 pixels) centered on the W pixel to be corrected, and the W (White) pixel is obtained. Generate a blur correction filter to apply.

  The blur correction processing unit 410 shown in FIG. 22 applies a blur correction filter corresponding to W (White) pixels reflecting the color ratio generated in step S302, and applies to the image data of all pixels W pixels generated by the interpolation processing. The blur correction W signal (Wd) generated by performing the blur correction processing and generating the blur correction W signal (Wd) corresponding to all pixels can be output to the color correlation re-mosaic processing unit (data conversion unit) 430. .

Further, the blur correction processing unit 410 performs a filter generation process in consideration of the image height and the like on the blur correction filter corresponding to the W (White) pixel reflecting the color ratio generated in step S302 as in the above-described embodiment. Execute and generate a blur correction filter corresponding to W (White) pixels reflecting not only the color ratio but also the image height, etc., and apply this to apply blur correction processing to the image data of all pixels W pixels generated by interpolation processing Can be done.
Hereinafter, this processing example will be described.

  In step S303, the blur correction processing unit 410 uses the blur correction filter corresponding to the W (White) pixel that reflects the color ratio generated in step S302 as a base, and generates a filter in consideration of the image height and the like as in the above-described embodiment. Execute.

The process of step S303 is the same as the process of step S101 shown in the flow of FIG. 5 described in the previous embodiment.
That is, it is executed as a process according to the flow shown in FIGS.
The color ratio reflection blur correction filter generated in step S302 is used as a basic filter (default filter) applied in step S122 of the flow of FIG.

Specifically, a final blur correction filter (Ffinal) is generated according to the following procedure.
First, in step S121 of the flow shown in FIG. 9, a high-pass filter (HPF_dist) corresponding to the image height calculated from the pixel position of the correction target pixel is generated.
The high-pass filter (HPF_dist) corresponding to the image height is generated by the processing of steps S131 to S132 in the flow shown in FIG.

Next, in step S122 of the flow shown in FIG.
(1) a basic filter (default filter);
(2) The coordinate position corresponding high pass filter (HPF_dist) is
Blending according to the image height (r) and the blend coefficient α (focus, r) determined according to the focus position information (focus),
(3) Basic filter for coordinate position (Fbase),
Is generated.

(1) The basic filter (default filter) is a color ratio reflection blur correction filter generated in step S302 of the flow shown in FIG.
In step S122 of the flowchart shown in FIG. 9, the color ratio reflecting blur correction filter (default filter) and the coordinate position corresponding high-pass filter (HPF_dist) generated in step S121 are blended to obtain a coordinate position corresponding basic filter (Fbase). Generate.

The coordinate position-corresponding basic filter (Fbase) is calculated by the following formula based on the blend height α determined according to the image height (r) and the focus position information (focus).
Fbase = (1.0−α) × (default filter) + α (HPF_dist)
In the above formula,
default filter = color ratio reflection blur correction filter.

The blend coefficient α is determined according to the image height (r) and the focus position information (focus) as described in the previous embodiment.
As described above with reference to FIG. 13, the MTF characteristics generally deteriorate as the image height increases (the distance from the image center increases). In particular, since the high frequency region is a region buried in noise, as shown in FIG. 13B, the enhancement characteristic of the Wiener filter is shifted to the low frequency region side.
Therefore, as shown in FIG. 13C, as the image height increases (the distance from the image center increases), the blend ratio of the high-pass filter is decreased. That is, it is preferable to reduce the blend coefficient α.
The same can be said about the focus shift. That is, the blend coefficient α is decreased as the focus shift increases.

  The focus shift is a value unique to the image processing apparatus (camera), and the shift amount generated corresponding to each focus position is a unique value in each camera. Accordingly, the focus shift amount can be calculated from the focus position information, and the blur correction unit 410 receives the focus position information (see FIG. 22) from the control unit 340 shown in FIG. The deviation amount is calculated, and the blend coefficient α (focus, r) is calculated using the focus deviation amount and the image height r acquired from the image height calculation unit 402.

Further, in step 123 of the flow of FIG. 9, the intensity adjustment parameter β (focus, r) corresponding to the image height and the focus position is applied to the coordinate position corresponding basic filter (Fbase) generated in step S122. A blur correction filter (Ffinal) that is applied to the actual blur correction with the enhancement intensity adjusted is generated.
The blur correction filter (Ffinal) generated in step S123 is a filter applied to the actual blur correction processing for the pixel of the processing target pixel (x, y).

Note that the processing to which the intensity adjustment parameter β (focus, r) according to the image height and the focus position in step S123 is applied is the same processing as that described above with reference to FIGS.
That is, the coordinate position-corresponding basic filter (Fbase) is band-separated into a DC component and an AC component, separated into two filters, the AC component is multiplied by an enhancement strength (enhance strength adjustment parameter β), and the multiplication result The final blur correction filter (Ffinal) with enhanced intensity adjustment is generated by a synthesis process of adding the corresponding coefficients for the DC component and the DC component.

  The enhancement intensity adjustment parameter β is determined according to the image height (r) and the focus position information (focus), as with the blend coefficient α described above. As described above with reference to FIG. 17, the enhancement intensity adjustment parameter β is increased as the image height increases (the distance from the image center increases), and the enhancement intensity adjustment parameter increases as the focus shift increases. Increase β.

  As described above, the blur correction processing unit 410 uses the blur correction filter corresponding to the color ratio generated in step S302 of the flow shown in FIG. 23 as a basic filter (default filter) shown in step S122 of the flow shown in FIG. 9 is executed to generate a final blur correction filter (Ffinal).

  This final blur correction filter (Ffinal) is the blur correction filter generated in step S303 of the flow shown in FIG.

In step S304, it is determined whether the blur correction target pixel is saturated.
Specifically, the saturation detection unit 403 illustrated in FIG. 22 performs saturation detection of the blur correction target pixel, and this detection information is input to the blur correction processing unit 410.
When the correction target pixel is saturated, there is a possibility that a false color is generated when blur correction is performed. As described with reference to FIG. 6 and FIG. 7 in the previous embodiment, the blur correction process refers to the pixel values of the pixels around the pixel to be corrected for the correction target pixel. This is because it includes processing for setting pixel values.

  In order to prevent such false color generation, as shown in FIG. 7, when a saturation pixel is included around a blur correction target pixel, for example, a 7 × 7 pixel centering on the correction target pixel, the blur of the correction target pixel is detected. Set to not perform correction.

As described above, the blur correction processing unit 410 illustrated in FIG. 22 performs a process of replacing the saturation pixel region with a signal without blur correction processing according to the detection information of the saturation pixel (highlight) region in the saturation detection unit 403.
For example, the saturation pixel (highlight) executed by the saturation detection unit 403 determines whether or not a predetermined threshold value is exceeded for a pixel in a 7 × 7 pixel region centered on the blur processing target pixel. The blur correction processing unit 410 performs processing for replacing with a signal without correction processing because there is a high possibility that a false color is generated if even one pixel in the 7 × 7 pixel region exceeds the threshold value.
This process corresponds to the process in the case where the determination in step S304 of the flow shown in FIG. 23 is Yes. In this case, the process proceeds to step S307 without executing steps S305 and S306.
That is, the blur correction process is not performed on the processing target pixel (the center pixel of 7 × 7 pixels).

If any of the pixels in the 7 × 7 pixel area centered on the blur processing target pixel does not exceed the predetermined threshold value and saturation is not detected, the determination in step S304 is No, and in step S305 The blur correction process with the filter applied is executed.
This blur correction process is a process executed in the blur correction processing unit 410 shown in FIG. 22, and the blur correction process is performed by applying the blur correction filter generated according to the color ratio, image height, etc. generated in step S303. Is called.

Next, edge detection processing is executed in step S306.
This processing is executed in the edge detection unit 421 shown in FIG. The edge detection unit 421 verifies the output signal from the line memory 401 and generates edge information included in the image, for example, edge information including the edge direction and edge strength.
Specifically, for example, a color correlation re-mosaic processing unit (data conversion) is calculated by calculating a flatness (weightFlat) calculated from pixel information of 7 × 7 pixels centering on a processing target pixel (center pixel of 7 × 7 pixels). Part) 430.

The flatness (weightFlat) has a value in the range of 0 to 1, as described above with reference to FIG.
The closer to 1, the lower the flatness (the more texture)
The closer to 0, the higher the flatness (less texture)
This is an index value of flatness indicating such an image state.

As shown in FIG. 8, the flatness (weightFlat) is calculated as follows using two preset threshold values (Limit0, Limit1).
If 0 ≦ (ratioFlat) <Limit0,
Flatness (weightFlat) = 0
If Limit0 ≦ (ratioFlat) <Limit1,
Flatness (weightFlat) = 0-1
If Limit1 ≦ (ratioFlat),
Flatness (weightFlat) = 1
And

For example, the edge detection unit 421 outputs the flatness (weightFlat) information as edge information to the color correlation re-mosaic processing unit 430.
As described above, this flatness (weightFlat) calculation process can be executed as a process similar to the process described in Japanese Patent Application Laid-Open No. 2011-55038, which is an earlier application of the present applicant. See Japanese Patent Application Laid-Open No. 2011-55038.

Next, in step S307, color correlation re-mosaic processing is executed.
This color correlation re-mosaic process is a process executed by the color correlation re-mosaic processing unit (data conversion processing unit) 430 in FIG.
The color correlation re-mosaic processing unit 430 performs conversion processing from the RGBW color array to the RGB array.

Specifically, as described above with reference to FIG.
Convert W pixel position to G pixel (estimate G pixel value) = (GonW)
Convert G pixel position to R pixel (estimate R pixel value) = (RonG)
Convert G pixel position to B pixel (estimate B pixel value) = (BonG)
Convert R pixel position to R pixel (correct R pixel value) = (RonR)
Convert to B pixel at B pixel position (correct B pixel value) = (BonB)
These five conversions and correction processes are executed.

  The color correlation re-mosaic processing unit 430 is set for each pixel of the RGBW array on the assumption that there is a positive correlation between the W signal that is the main component of luminance in the RGBW array and the G, R, and B signals that are color components. Estimate the target pixel value of any of RGB.

  As shown in FIG. 22, the color correlation re-mosaic processing unit 430 includes a W position G interpolation parameter calculation unit 431, a G position RB interpolation parameter calculation unit 432, an R position R interpolation parameter calculation unit 433, and a B position B interpolation parameter calculation unit 434. And a weighted addition unit 435.

  First, the processing of the W position G interpolation parameter calculation unit 431 that is an interpolation parameter calculation unit for converting the W pixel position of the RGBW array into the G pixel of the RGB array (estimating the G pixel value) will be described.

The W position G interpolation parameter calculation unit 431 calculates an interpolation parameter applied to the calculation of the G pixel value set at the W pixel position of the RGBW array. This is an interpolation parameter applied to the above-described (GonW) process.
Specifically, using the Win signal and the Gin signal in the input signal from the line memory 401, and the blur correction W signal (Wd) input from the blur correction processing unit 203,
G signal with blur correction: Gd,
G signal without blur correction: Gl,
Each of these signal values is calculated.
These signal values are interpolation parameters (GonW interpolation parameters) applied to G pixel value calculation for setting the W pixel position of the RGBW array as the G pixel of the RGB array.

The color correlation re-mosaic processing unit 430 also performs processing using a 7 × 7 pixel area as a processing unit. The W position G interpolation parameter calculation unit 431 first obtains the ratio of the W signal and the G signal in the 7 × 7 pixel area of the processing unit.
Specifically, the low frequency components mW and mG of the W signal and the G signal are calculated using a 7 × 7 pixel region having a W pixel as a central pixel to be converted to a G pixel as a processing unit.
Specifically, it can be calculated by applying a filter for calculating a low frequency component similar to that described with reference to FIG. FIG. 29 (1) shows an example of the mW calculation filter 601 for calculating the low frequency component mW.

  The W pixel value in the 7 × 7 pixel centered on the W pixel position of the conversion target pixel of the 7 × 7 pixel input signal is multiplied by the filter coefficient of the corresponding pixel position of the mW calculation filter 601, and each multiplication result The added value is calculated as the low frequency component mW of the W signal. The coefficients of the filter shown in FIG. 29 (1) are merely examples, and a configuration using other coefficients may be used.

  Assume that the calculated ratio of mW to mG is maintained in a local region in the image. According to this assumption, the correspondence relationship between the pixel value ratios of W and G in the local region is a correspondence relationship as shown in the graph of FIG. In the graph of FIG. 30, the horizontal axis represents the W pixel value, and the vertical axis represents the G pixel value. If the ratio of the W pixel value to the G pixel value is constant in a specific narrow local region of the image, it can be assumed that there is a proportional relationship such as a straight line shown in FIG.

In accordance with this assumption, the blur correction white (W) signal (= Wd) of the conversion target pixel position output from the blur correction processing unit 410 is used.
G signal with blur correction: Gd,
Is calculated. Calculate according to the following formula.
Gd = (mG / mW) × Wd
This signal [Gd] is a G signal containing a high frequency component.
Here, the estimation method assuming that the ratio between the W pixel value and the G pixel value is constant is illustrated here, but other estimation methods using the correlation between the W pixel value and the G pixel value may be used. .

As described above, the W position G interpolation parameter calculation unit 431 has, as the interpolation parameter applied to the calculation of the G pixel value set at the W pixel position of the RGBW array,
G signal with blur correction: Gd,
G signal without blur correction: Gl,
Each of these signal values is calculated. The weighted addition processing unit 435 blends these two signals, Gd and Gl, according to the edge information of the conversion target pixel, and determines the final G pixel value.

Processing for calculating the G signal Gl without blur correction will be described. When calculating the G signal Gl without blur correction,
First, a W signal Wn from which a noise signal has been removed is calculated using, for example, a Wn calculation filter 602 as a smoothing filter shown in FIG.
The W pixel value in 9 pixels centering on the W pixel position of the conversion target pixel 502 of the input signal 501 of 7 × 7 pixels is multiplied by the filter coefficient of the corresponding pixel position of the Wn calculation filter 602, and each multiplication result A noise removal white (W) signal Wn of the W signal is calculated as the added value. The filter coefficient shown in FIG. 29 (2) is an example, and a configuration using other coefficients may be used. Moreover, it is good also as a structure using the area | region wider than 9 pixels. Further, a configuration in which the W signal input from the line memory 401 is used as it is may be used to reduce the calculation cost.

Applying this noise-removed W signal [Wn] and the low-frequency components mW and mG of the aforementioned W signal and G signal,
G signal without blur correction: Gl,
Is calculated. Calculate according to the following formula.
Gl = (mG / mW) × Wn
This signal [Gl] does not apply the blur correction signal Wd, and corresponds to a G signal that does not emphasize the high-frequency component.
Here, the estimation method assuming that the ratio between the W pixel value and the G pixel value is constant is illustrated here, but other estimation methods using the correlation between the W pixel value and the G pixel value may be used. .

  The weighted addition processing unit 435 performs blend processing of these two signals, Gd and Gl, according to the edge information of the conversion target pixel, and determines the final G pixel value. Specifically, the blending process is executed in which the ratio of the G signal with the blur correction: Gd is increased at the edge portion having a high texture degree instead of the flat portion, and the ratio of the G signal without blur correction: Gl is increased at the flat portion. To determine the final G pixel value.

The G position RB interpolation parameter calculation unit 432
Convert G pixel position to R pixel (estimate R pixel value) = (RonG)
Convert G pixel position to B pixel (estimate B pixel value) = (BonG)
Interpolation parameters to be applied to these processes are calculated.

In particular,
R signal with blur correction: Rd,
R signal without blur correction: Rl,
B signal with blur correction: Bd,
B signal without blur correction: Bl,
Each of these signal values is calculated.
These signal values are interpolation parameters (RonG interpolation parameters, BonG interpolation parameters) applied to R pixel value or B pixel value calculation for setting the G pixel position of the RGBW array as an R pixel or B pixel of the RGB array. .

The calculation processing of these signals is the same as the processing of the W position G interpolation parameter calculation unit 431 described above.
The low frequency components mW, mR, and mB of each signal are calculated, and the following values are calculated using the blur correction white (W) signal (= Wd) of the conversion target pixel position output from the blur correction processing unit 410. This is performed as a process for calculating.
R signal with blur correction: Rd,
B signal with blur correction: Bd,
R signal without blur correction: Rl,
B signal without blur correction: Bl,
Is calculated.

Calculate according to the following formula.
Rd = (mR / mW) × Wd
Bd = (mB / mW) × Wd
These signals [Rd] and [Bd] are signals including high-frequency components.
Rl = (mR / mW) × Wn
Bl = (mB / mW) × Wn
These signals [Rl] and [Bl] do not apply the blur correction signal Wd, and correspond to signals that do not emphasize high-frequency components.

  Here, the estimation method assuming that the ratio between the W pixel value and the R pixel value or the B pixel value is constant is illustrated here, but other than the correlation between the W pixel value and the R pixel value or the B pixel value is used. The estimation method may be used.

  Next, an R position R interpolation parameter calculation unit 433 which is an interpolation parameter calculation unit for converting the R pixel position of the RGBW array into an R pixel of the RGB array (correcting the R pixel value), and a B position B interpolation parameter calculation unit The process 434 will be described.

The R pixel position of the RGBW array corresponds to the R pixel position of the RGB array (Bayer array), and the R signal as it is can be used.
Similarly, the B pixel position of the RGBW array corresponds to the B pixel position of the RGB array (Bayer array), and the B signal as it is can be used.
However, this information may have lost high frequency signals due to the effects of aberrations of the optical lens. In such a case, there is a possibility that characteristics may be different from those of the R signal and the B signal set at the G pixel position.

In order to prevent such a characteristic difference from occurring, the R position / R interpolation parameter calculation unit 433 calculates a parameter for correcting the R signal using the W signal and the R signal included in the input signal. In particular,
R signal with blur correction: Rd,
R signal without blur correction: Rl,
Each of these signal values is calculated.
These signal values are interpolation parameters (RonR interpolation parameters) applied to the R pixel value correction process for setting the R pixel position in the RGBW array as the R pixel corrected in the RGB array.

The R position R interpolation parameter calculation unit 433 is similar to the other interpolation parameter calculation units described above,
Low frequency component mW of the W signal,
Low frequency component mR of R signal,
Each of these low frequency components,
Noise removal of W signal White (W) signal Wn
These are calculated.

Furthermore, using these,
R signal with blur correction: Rd,
R signal without blur correction: Rl,
These are calculated according to the following formula.
Rd = (mR / mW) × Wd
Rl = (mR / mW) × Wn
Here, the estimation method assuming that the ratio between the W pixel value and the R pixel value is constant is illustrated here, but other estimation methods using the correlation between the W pixel value and the R pixel value may be used. .

The B-position B interpolation parameter calculation unit 434, like the other interpolation parameter calculation units described above,
Low frequency component mW of the W signal,
The low frequency component mB of the B signal,
Each of these low frequency components,
Noise removal of W signal White (W) signal Wn
These are calculated.

Furthermore, using these,
B signal with blur correction: Bd,
B signal without blur correction: Bl,
These are calculated according to the following formula.
Bd = (mB / mW) × Wd
Bl = (mB / mW) × Wn
Here, the estimation method assuming that the ratio between the W pixel value and the B pixel value is constant is illustrated here, but other estimation methods using the correlation between the W pixel value and the B pixel value may be used. .

As described above, each of the W position G interpolation parameter calculation unit 431, the G position RB interpolation parameter calculation unit 432, the R position R interpolation parameter calculation unit 433, and the B position B interpolation parameter calculation unit 434
G, R, B signals with blur correction: Gd, Rd, Bd,
G, R, B signals without blur correction: Gl, Rl, Bl,
These signals are calculated.

Next, the processing of the weighted addition unit 435 will be described.
The weighted addition processing unit 435 applies the edge information input from the edge detection unit 421, and
Blur correction signal: Gd, Rd, Bd,
No blur correction signal: Gl, Rl, Bl,
The weighted average values of these signals: Gr, Rr, Br are calculated. Gr, Rr, and Br calculated by the weighted addition processing unit 435 correspond to the RGB signal values of the respective pixels constituting the RGB array (Bayer array) 382 shown in FIG.

In each of the W position G interpolation parameter calculation unit 431, the G position RB interpolation parameter calculation unit 432, the R position R interpolation parameter calculation unit 433, and the B position B interpolation parameter calculation unit 434, the calculated Gd, Rd, and Bd are corrected for blur. The processing unit 410 is calculated using the blur correction W signal (Wd) generated as the W signal subjected to the blur correction process, and includes a high frequency signal component and a noise amplified signal.
On the other hand, Gl, Rl, and Bl are signals calculated from the W signal that has not been subjected to the blur correction process, and are signals that do not include high-frequency signal components but have low noise.

The weighted addition processing unit 435, according to the edge information of the conversion target pixel,
Blur correction signal: Gd, Rd, Bd,
No blur correction signal: Gl, Rl, Bl,
The blend ratio of these signals is determined and blend processing is performed to determine final G, R, and B pixel values. Specifically, the ratio of the blur correction signal: Gd, Rd, Bd is increased at the edge portion having a high texture degree instead of the flat portion, and the ratio of the blur correction-free signal: Gl, Rl, Bl is increased at the flat portion. A blend process is executed to determine final G, R, and B pixel values.

  This is because it is desirable that the signal is a sharp signal with a high-frequency component restored in the vicinity of the edge, but since the high-frequency component is originally not included in the flat portion, it is desirable that the signal be suppressed in noise. .

  Therefore, the weighted addition processing unit 435 calculates the signals Gd, Rd, and Bd subjected to the blur correction and the signals Gl, Rl, and B1 not subjected to the blur correction by the edge detection unit 421 as shown in the following formula. The weighted average process is executed according to the edge information of the processing target pixel, that is, the flatness (weightFlat), and the pixel values Gr, Rr, and BR set in the RGB array 382 are calculated.

Gr = (weightFlat) × (Gd) + (1−weightFlat) × Gl
Br = (weightFlat) × (Bd) + (1−weightFlat) × Bl
Rr = (weightFlat) × (Rd) + (1−weightFlat) × Rl
Gr, Br, and Rr obtained as a calculation result of this equation are set as signals of the RGB array 382 and output to the RGB signal processing unit 450. By this processing, high-resolution signals Gr, Rr, and Br can be obtained while suppressing noise amplification.

  However, as described above, when a saturated pixel is detected in a pixel unit of a predetermined unit, for example, a 7 × 7 pixel region centered on a pixel to be subjected to blur correction processing, blur correction is not performed for that pixel. No blur correction signals Rl, Gl and Bl are output.

Finally, in step S308, it is determined whether or not processing for all input pixels has been completed. If there are unprocessed pixels, the processing from step S301 to step S307 is repeatedly executed for the unprocessed pixels.
If it is determined in step S308 that the processing has been completed for all input pixels, the processing of the image signal correction unit 400 ends.

The RGB array (Bayer array) 382 composed of Gr, Rr, and Br generated by the weighted addition processing unit 435 is provided to the RGB signal processing unit 450 as shown in FIGS.
As described above, the RGBW array to RGB array data conversion process executed by the color correlation re-mosaic processing unit (data conversion unit) 430 is basically Japanese Patent Application Laid-Open No. 2011-55038, which is a prior application of the present applicant. Processing described in the publication can be used. For details of this data conversion processing, refer to Japanese Patent Application Laid-Open No. 2011-55038.

  The RGB signal processing unit 450 is the same as the signal processing unit for RGB array (Bayer array) signals of general cameras and image processing apparatuses. The RGB signal processing unit 450 performs signal processing on the RGB array (Bayer array) 382 output from the weighted average unit 435 to generate a color image 383 (see FIG. 21). Specifically, the RGB signal processing unit 450 generates a color image 383 by executing, for example, white balance adjustment processing, demosaic processing, shading processing, RGB color matrix processing, and γ correction processing.

  In the above-described embodiment, the processing example for the RGBW array illustrated in FIG. 20A has been described. However, the process of the present disclosure is not limited to this color array, and is a W pixel having a higher sampling rate than a color signal. The present invention can be applied to imaging data of an imaging device including filters having various color arrangements including

  In the above-described embodiment, the blur correction filter to be applied to the blur correction process in step S305 in the flow shown in FIG. 22 is based on the color ratio reflecting filter generated in step S302 in the flow shown in FIG. The example in which the filter is changed to reflect the image height has been described.

  However, for example, the blur correction filter applied to the blur correction process in step S305 of the flow shown in FIG. 22 may be the color ratio reflecting filter generated in step S302 of the flow shown in FIG. That is, a configuration in which the process of step S303 is omitted may be employed.

  Also, when executing both steps S302 and S303, as in the first embodiment described above, for example, step S123 of the flow shown in FIG. 9 which is a detailed flow of the filter generation processing of step S303 is omitted. In step S122, the coordinate position corresponding basic filter (Fbas) generated based on the blend coefficient α (focus, r) may be applied as a final blur correction filter.

  Alternatively, step S121 and step S122 in the flow shown in FIG. 9 may be omitted, and the blur correction filter may be generated only by the process of step S123. That is, using the color ratio reflecting filter generated in step S302 as a basic filter (default filter), the processing in step 123 of the flow shown in FIG. 9, that is, the band separation processing and enhancement strength described with reference to FIGS. The filter generated by executing the intensity adjustment for the postal service to which the adjustment parameter β (focus, r) is applied and the re-synthesis process may be applied to the blur correction.

  In the above embodiment, the configuration example in which the blend coefficient α and the enhancement intensity adjustment parameter β are determined by applying the image height (r) and the focus position information (focus) has been described. The intensity adjustment parameter β may be determined depending on only the image height. Alternatively, it may be determined depending only on the focus position information (focus). Furthermore, the configuration may be determined in consideration of other parameters such as a zoom position.

[6. Summary of composition of the present disclosure]
As described above, the embodiments of the present disclosure have been described in detail with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present disclosure, the claims should be taken into consideration.

The technology disclosed in this specification can take the following configurations.
(1) a blur correction processing unit that performs a blur correction process on an output signal of an image sensor having an RGBW array including RGB pixels and W (white) pixels;
A data converter for converting the RGBW array into an RGB array;
The blur correction processing unit
An image for generating a color ratio reflecting blur correction filter reflecting the color ratio of the local region including the W pixel signal to be corrected and executing the blur correction process using the color ratio reflecting filter in the blur correction process for the W pixel signal Processing equipment.

(2) The blur correction processing unit generates an interpolated image having a W pixel signal corresponding to all pixels by an interpolation process using a W pixel signal included in the output signal of the image sensor, and configures the generated interpolated image The image processing apparatus according to (1), wherein the color ratio reflecting blur correction filter is generated for each W pixel to be performed, and blur correction processing using the color ratio reflecting filter is executed.
(3) The blur correction processing unit calculates a low frequency component of each color of RGB in the local region, and uses the ratio of the calculated low frequency components of each color of RGB as the color ratio of the local region. The image processing apparatus according to (2).

(4) The blur correction processing unit includes an R corresponding blur correction filter, a G corresponding blur correction filter, which are stored in advance in memory according to a blend ratio determined according to each RGB color ratio of the local region, and B The image processing apparatus according to any one of (1) to (3), wherein the color ratio reflecting blur correction filter is generated by blending with a corresponding blur correction filter.
(5) The image processing device according to any one of (1) to (4), further including a data conversion unit that converts the RGBW array into an RGB array.
(6) The data conversion unit outputs the RGB signal (Rd, Gd, Bd) with blur correction, which is a blur correction signal corresponding to RGB estimated from the blur correction signal corresponding to the W pixel signal generated by the blur correction processing unit. The image processing according to any one of (1) to (5), wherein processing for generating and determining RGB signal values constituting an RGB array by applying the RGB signal (Rd, Gd, Bd) with blur correction is performed apparatus.

  (7) The image processing apparatus further includes an edge detection unit that analyzes an output signal of an image sensor having an RGBW array including RGB pixels and W pixels to generate edge information including edge intensity information corresponding to each pixel. The data conversion unit calculates a non-blurred RGB signal (Rl, Gl, Bl) which is a non-applied signal of the blur correction processing, and generates a blur correction signal corresponding to the W pixel signal generated by the blur correction processing unit. Processing for determining RGB signal values constituting an RGB array or color image by blending RGB signals (Rd, Gd, Bd) with blur correction, which are RGB-specific blur correction signals estimated from the above, and RGB signals without blur correction In the pixel position where the edge strength is determined to be high based on the edge information, the blur correction is performed and the RGB signal blend ratio is increased to In any one of the above (1) to (6), the blending process in which the blending ratio of the RGB signal without blur correction is set high is executed at the pixel position determined to have a small degree, and the RGB signal value determination process is performed. Image processing device.

(8) The blur correction processing unit performs blur correction processing by applying a W pixel signal to generate a blur correction W signal (Wd) corresponding to each pixel. The data conversion unit Based on the assumption that there is a positive correlation between the W signal and each RGB signal, the RGB signal (Rd, Gd, Bd) with blur correction, which is a blur correction signal corresponding to RGB, is calculated from the blur correction W signal (Wd). The image processing apparatus according to any one of (1) to (7).
(9) The blur correction processing unit determines, according to the image height of the blur correction target pixel, the color ratio reflecting blur correction filter and a high-pass filter corresponding to the coordinate position to be generated according to the image height of the blur correction target pixel. The image processing apparatus according to any one of (1) to (8), wherein blend processing is performed by applying a blend coefficient to be performed, and blur correction processing is performed by applying a filter generated by the blend processing.

(10) The blur correction processing unit applies a blend coefficient that determines the color ratio reflecting blur correction filter and the coordinate position corresponding high-pass filter according to an image height of the blur correction target pixel and focus position information. The image processing apparatus according to any one of (1) to (9), wherein a blending process is performed, and a blur correction process is performed by applying a filter generated by the blending process.
(11) In the blending process, the blur correction processing unit reduces the blend rate of the high-pass filter corresponding to the coordinate position as the pixel position of the blur correction target pixel is far from the center of the image and the image height increases. The image processing apparatus according to any one of (1) to (10), which executes processing.

(12) The blur correction processing unit includes an x-direction adjustment high-pass filter (HPF_x), a y-direction adjustment high-pass filter (HPF_y), and a field angle center position [(x, y) = ( 0,0)] is blended according to the pixel position of the blur correction target pixel to generate the coordinate position corresponding high-pass filter in any one of (1) to (11). The image processing apparatus described.
(13) The blur correction processing unit further generates an intensity adjustment filter by performing enhancement intensity adjustment on the coordinate position corresponding high-pass filter according to the image height of the blur correction target pixel, and applies the intensity adjustment filter. The image processing apparatus according to any one of (1) to (12), wherein the blur correction process is performed.

(14) The blur correction processing unit separates the high-pass filter corresponding to the coordinate position into a DC component and an AC component, and applies an intensity adjustment parameter corresponding to the image height of the blur correction target pixel to the AC component. The image processing apparatus according to any one of (1) to (13), wherein the intensity adjustment filter is generated by performing synthesis and recombining the AC component and the DC component that have undergone adjustment processing.
(15) The blur correction processing unit executes enhancement intensity adjustment using the coordinate position corresponding high-pass filter to which an intensity adjustment parameter corresponding to image height of the blur correction target pixel and focus position information is applied, and thereby the intensity adjustment filter. The image processing apparatus according to any one of (1) to (14).
(16) The image signal correction unit includes a saturation detection unit that detects whether or not a saturated pixel is included in a local region including a plurality of pixels including a blur correction target pixel, and the blur correction processing unit includes the saturation correction unit. When the detection information from the detection unit is input, and the saturated pixel is included in the local region, the blur correction target pixel is not subjected to blur correction, and only when the saturated pixel is not included, the blur correction target The image processing apparatus according to any one of (1) to (15), wherein pixel blur correction is executed.

  Furthermore, a configuration of the present disclosure includes a method of processing executed in the above-described apparatus and system, a program for executing the processing, and a recording medium on which the program is recorded.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet, and installed on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

As described above, according to the configuration of an embodiment of the present disclosure, an apparatus and a method for realizing appropriate blur correction for imaging data having an RGBW pixel array are realized.
Specifically, the image processing apparatus includes a blur correction processing unit that performs a blur correction process on an output signal of an image sensor having an RGBW array including RGB pixels and W (white) pixels. The blur correction processing unit generates a color ratio reflecting blur correction filter that reflects the color ratio of the local region including the W pixel signal to be corrected, and applies the generated color ratio reflecting filter when the blur correction process is performed on the W pixel signal. Perform blur correction processing. For example, low frequency components of each color of RGB are calculated in the local region, a color ratio is calculated using these calculated values, a blur correction filter corresponding to the calculated color ratio is generated, and blur correction is executed.
With these processes, an appropriate blur correction process using an optimal blur correction filter corresponding to the color ratio of the local region of the image is realized.

DESCRIPTION OF SYMBOLS 100 Imaging device 105 Optical lens 110 Imaging element (image sensor)
120 signal processing unit 130 memory 140 control unit 181 RGB array 183 color image 200 image signal correction unit 201 line memory 202 image height calculation unit 203 saturation detection unit 210 blur correction processing unit 221 edge detection unit 230 blend processing unit 235 weighted addition unit 250 RGB signal processing unit 300 imaging device 305 optical lens 310 imaging element (image sensor)
320 Signal processing unit 330 Memory 340 Control unit 381 RGBW array 382 RGB array 383 Color image 400 Image signal correction unit 401 Line memory 402 Image height calculation unit 403 Saturation detection unit 410 Blur correction processing unit 421 Edge detection unit 430 Color correlation re-mosaic processing unit 431 W position G interpolation parameter calculation unit 432 G position RB interpolation parameter calculation unit 433 R position R interpolation parameter calculation unit 434 B position B interpolation parameter calculation unit 435 Weighted addition unit 450 RGB signal processing unit

Claims (19)

A blur correction processing unit that performs blur correction processing on an output signal of an image sensor having an RGBW array including RGB pixels and W (white) pixels;
The blur correction processing unit
An image for generating a color ratio reflecting blur correction filter reflecting the color ratio of the local region including the W pixel signal to be corrected and executing the blur correction process using the color ratio reflecting filter in the blur correction process for the W pixel signal Processing equipment. The blur correction processing unit
An interpolation image having a W pixel signal corresponding to all the pixels is generated by interpolation processing using the W pixel signal included in the output signal of the image sensor, and the color ratio reflecting blur is applied to each of the W pixels constituting the generated interpolation image. The image processing apparatus according to claim 1, wherein a correction filter is generated, and blur correction processing using the color ratio reflection filter is executed. The blur correction processing unit
The image processing apparatus according to claim 1, wherein low frequency components of each color of RGB in the local area are calculated, and a ratio of the calculated low frequency components of each color of RGB is set as a color ratio of the local area. The blur correction processing unit
The R corresponding blur correction filter, the G corresponding blur correction filter, and the B corresponding blur correction filter previously stored in the memory are blended in accordance with a blend ratio determined according to each RGB color ratio of the local region. The image processing apparatus according to claim 1, wherein a color ratio reflecting blur correction filter is generated. The image processing apparatus further includes:
The image processing apparatus according to claim 1, further comprising a data conversion unit that converts the RGBW array into an RGB array. The data converter is
An RGB signal (Rd, Gd, Bd) with blur correction that is a blur correction signal corresponding to RGB estimated from the blur correction signal corresponding to the W pixel signal generated by the blur correction processing unit;
The image processing apparatus according to claim 5, wherein a process for determining RGB signal values constituting an RGB array is performed by applying the blur-corrected RGB signals (Rd, Gd, Bd). The image processing apparatus further includes:
An edge detection unit that analyzes an output signal of an image sensor having an RGBW array composed of RGB pixels and W pixels and generates edge information including edge intensity information corresponding to each pixel;
The data converter is
An RGB signal (Rl, Gl, Bl) without blur correction, which is a non-applied signal of blur correction processing, is calculated, and the RGB blur is estimated from the blur correction signal corresponding to the W pixel signal generated by the blur correction processing unit. It is a configuration that performs RGB signal value determination processing that constitutes an RGB array or a color image by blending the RGB signal (Rd, Gd, Bd) with blur correction that is a correction signal and the RGB signal without blur correction,
The blending ratio of the RGB signal with blur correction is increased at the pixel position where the edge intensity is determined to be high according to the edge information, and the blend ratio of the RGB signal without blur correction is increased at the pixel position where the edge intensity is determined to be small. The image processing apparatus according to claim 5, wherein the RGB signal value determination process is performed by executing the set blend process. The blur correction processing unit
A blur correction process is performed by applying the W pixel signal to generate a blur correction W signal (Wd) corresponding to each pixel,
The data converter is
Based on the assumption that there is a positive correlation between the W signal and each of the RGB signals in the local region of the image, the RGB signal (Rd, Gd) that is a blur correction signal corresponding to RGB from the blur correction W signal (Wd). , Bd) is calculated. The blur correction processing unit
The color ratio reflecting blur correction filter and the high-pass filter corresponding to the coordinate position generated according to the image height of the blur correction target pixel are applied with a blend coefficient that is determined according to the image height of the blur correction target pixel. The image processing apparatus according to claim 1, wherein the blur correction process is performed by applying the filter generated by the blend process. The blur correction processing unit
The color ratio reflecting blur correction filter and the coordinate position corresponding high pass filter are blended by applying a blend coefficient determined according to the image height and focus position information of the blur correction target pixel, and are generated by the blend processing. The image processing apparatus according to claim 9, wherein the blur correction process is executed by applying the filtered filter. The blur correction processing unit
10. The blend process according to claim 9, wherein in the blend process, a blend process is executed in which the blend rate of the high-pass filter corresponding to the coordinate position is decreased as the pixel position of the blur correction target pixel is far from the image center and the image height increases. Image processing device. The blur correction processing unit
X-direction adjusting high-pass filter (HPF_x), y-direction adjusting high-pass filter (HPF_y), and a center filter corresponding to the central position [(x, y) = (0, 0)]. The image processing apparatus according to claim 9, wherein the coordinate position corresponding high-pass filter is generated by blending (HPF_center) according to a pixel position of a blur correction target pixel. The blur correction processing unit
Furthermore, the coordinate position corresponding high-pass filter performs enhancement intensity adjustment according to the image height of the blur correction target pixel to generate an intensity adjustment filter, and performs blur correction processing using the intensity adjustment filter. The image processing apparatus described. The blur correction processing unit
The coordinate position-corresponding high-pass filter is separated into a direct current component and an alternating current component, and the alternating current component is subjected to an adjustment process in which an intensity adjustment parameter corresponding to the image height of the blur correction target pixel is applied. The image processing apparatus according to claim 13, wherein the intensity adjustment filter is generated by recombining a component and the DC component. The blur correction processing unit
14. The image according to claim 13, wherein the intensity adjustment filter is generated by executing enhancement intensity adjustment using the image position of the blur correction target pixel and an intensity adjustment parameter corresponding to focus position information for the high-pass filter corresponding to the coordinate position. Processing equipment. The image signal correction unit
A saturation detection unit that detects whether a saturated pixel is included in a local region including a plurality of pixels including a blur correction target pixel;
The blur correction processing unit
When detection information from the saturation detection unit is input and a saturated pixel is included in the local region, the blur correction of the blur correction target pixel is not performed, and the blur is only performed when the saturated pixel is not included. The image processing apparatus according to claim 1, wherein blur correction of a correction target pixel is executed. An image processing method for executing image blur correction processing in an image processing apparatus,
The blur correction processing unit executes a blur correction processing step for performing blur correction processing on an output signal of an image sensor having an RGBW array including RGB pixels and W (white) pixels,
The blur correction processing step includes
A step of generating a color ratio reflecting blur correction filter reflecting a color ratio of a local region including the W pixel signal to be corrected and executing the blur correction process using the color ratio reflecting filter in the blur correction processing for the W pixel signal; An image processing method. A program for executing image blur correction processing in an image processing apparatus,
Causing the blur correction processing unit to execute a blur correction processing step for performing blur correction processing on an output signal of an image sensor having an RGBW array including RGB pixels and W (white) pixels;
In the blur correction processing step,
When blur correction processing is performed on a W pixel signal, a color ratio reflecting blur correction filter that reflects the color ratio of the local region including the W pixel signal to be corrected is generated, and blur correction processing using the color ratio reflecting filter is executed. program. A recording medium that records a program for executing image blur correction processing in an image processing apparatus,
Causing the blur correction processing unit to execute a blur correction processing step for performing blur correction processing on an output signal of an image sensor having an RGBW array including RGB pixels and W (white) pixels;
In the blur correction processing step,
When blur correction processing is performed on a W pixel signal, a color ratio reflecting blur correction filter that reflects the color ratio of the local region including the W pixel signal to be corrected is generated, and blur correction processing using the color ratio reflecting filter is executed. A recording medium that records the program.