JP2013066157A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an apparatus and method capable of realizing accurate detection and correction of a defect in an image having variously configured pixel arrays.SOLUTION: An apparatus applies plural different methods to a pixel region including a target pixel to obtain plural pieces of gradient detection information. On the basis of weighted addition of the plural pieces of gradient detection information, a minimum gradient direction is detected. Then, the apparatus calculates Laplacian based on a pixel value of a reference pixel located in a detected minimum gradient direction and having the same color as that of the target pixel, and determines the presence or absence of a defect in the target pixel. Then, the apparatus calculates a corrected pixel value by applying the pixel value of the reference pixel located in the direction detected in the direction determination process, to a target pixel from which a defect has been detected.

Description

  The present disclosure relates to an image processing device, an image processing method, and a program. In particular, the present invention relates to an image processing apparatus, an image processing method, and a program that execute an image correction process.

An image sensor used in an image pickup apparatus such as a digital camera is provided with a color filter made of, for example, an RGB array, and has a configuration in which specific wavelength light is incident on each pixel.
Specifically, for example, a color filter having a Bayer array is often used.

  The Bayer array captured image is a so-called mosaic image in which only pixel values corresponding to any of RGB colors are set for each pixel of the image sensor. The signal processing unit of the camera performs various signal processing such as pixel value interpolation on the mosaic image, performs demosaic processing for setting all pixel values of RGB for each pixel, and generates and outputs a color image. .

  Many studies have already been made on signal processing for a captured image having a color filter according to this Bayer array, and it can be said that it has been technically established to some extent. However, the current situation is that sufficient studies have not yet been made on signal processing for images having an array different from the Bayer array.

  For example, as a filter attached to an image sensor, correction processing for a captured image of an imaging device including a filter having an RGBW array including all-wavelength transmission type W (White) pixels in addition to RGB colors is disclosed in Patent Literature 1 (Japanese Unexamined Patent Application Publication No. 2011-55038).

JP 2011-055038 A

  The present disclosure has been made in view of, for example, the above-described problems. For example, an image processing apparatus that performs image correction processing on an image captured by an imaging element including a color filter having an array different from a Bayer array And an image processing method and a program.

The first aspect of the present disclosure is:
An image signal correction unit that executes image correction processing;
The image signal correction unit
A direction determination process for detecting a direction having a minimum pixel value gradient as a pixel value gradient direction in a pixel region including the target pixel;
A defect detection process for calculating the Laplacian based on the pixel value of the reference pixel in the minimum gradient direction detected in the direction determination process for the target pixel, and determining the presence or absence of a defect in the target pixel;
Executing a defect correction process for calculating a correction pixel value by applying a pixel value of a reference pixel in a direction detected in the direction determination process to a target pixel in which a defect is detected in the defect detection process;
The direction determination process is in an image processing apparatus that executes by applying a weighted addition result of a plurality of pieces of gradient information calculated by a plurality of different gradient detection processes.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the image signal correction unit includes, in the direction determination process, pixel value gradient information corresponding to high frequency texture, pixel value gradient information corresponding to low frequency texture, and a luminance signal. The corresponding pixel value gradient information is calculated, and the direction having the minimum pixel value gradient is detected based on the weighted addition result of these three types of gradient information.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the image signal correction unit calculates the pixel value gradient information corresponding to the high frequency texture by applying a pixel value difference between adjacent pixels, and the low frequency texture correspondence Is calculated by applying the pixel value difference of the non-adjacent pixels.

  Furthermore, in an embodiment of the image processing device of the present disclosure, the image signal correction unit calculates a luminance signal based on a pixel value of each RGB pixel in a pixel region unit including each RGB pixel, and calculates the calculated region unit. Pixel value gradient information corresponding to the luminance signal is calculated by applying the luminance signal.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the image signal correction unit executes a process of changing a weight set in the weighted addition process of the three types of gradient information according to the resolution of the output image. If the output image has a high resolution, the weight for the pixel value gradient information corresponding to the high frequency texture is set higher than the other gradient information, and if the output image has a low resolution, the pixel value gradient corresponding to the low frequency texture is set. The weight for information is set higher than other gradient information.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the image signal correction unit sets a weight set in the weighted addition processing of the three types of gradient information according to a frequency band of an input image that is a processing target image. If the input image contains many high-frequency areas, the weight for the pixel value gradient information corresponding to the high-frequency texture is set higher than other gradient information, and the input image contains many low-frequency areas. If included, the weight for the pixel value gradient information corresponding to the low frequency texture is set higher than other gradient information.

  Furthermore, in an embodiment of the image processing apparatus according to the present disclosure, the image signal correction unit includes an image in which RGB colors are arranged in units of 4 pixels of 2 × 2, or RGBW colors are arranged in units of 4 pixels of 2 × 2. The pixel value correction of the processed image is executed.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the image signal correction unit selects, from the minimum gradient direction, a pixel having the same color as a target pixel to be detected as a reference pixel in the defect detection process. A comparison is made between a plurality of Laplacians calculated based on different combinations of the target pixel and the selected pixel and a predetermined threshold value, and based on the comparison result, it is determined whether or not the target pixel is a defective pixel.

  Furthermore, in an embodiment of the image processing apparatus of the present disclosure, the image signal correction unit selects four pixels from the minimum gradient direction as pixels of the same color as the target pixel to be detected as a reference pixel, A comparison between three Laplacians calculated based on different combinations of two selected pixels and a predetermined threshold is performed, and when all three Laplacians are larger than the threshold, the target pixel is a defective pixel judge.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the image signal correction unit is configured such that, in the defect detection process, a pixel having the same color as a target pixel that is a defect detection target is within the predetermined reference area. If it is not possible to select four from the direction, pixel interpolation based on the pixel value of the same color pixel as the target pixel around the different color pixel position is performed on the different color pixel position of the color different from the target pixel in the minimum gradient direction, Interpolated pixels generated by pixel interpolation are set as reference pixels.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the image signal correction unit calculates a corrected pixel value of the target pixel by weighted addition of the pixel values of the reference pixel in the defect correction processing.

  Furthermore, in an embodiment of the image processing apparatus according to the present disclosure, the image signal correction unit calculates a pixel value gradient between two reference pixels on both sides around the target pixel in the defect correction process. Then, the corrected pixel value of the target pixel is calculated by weighted addition of the pixel values of the two pixels in the direction in which the pixel value gradient is small.

  Furthermore, in an embodiment of the image processing apparatus of the present disclosure, the image signal correction unit performs a highlight error correction determination process that determines whether or not the correction process executed in the defect correction process is a highlight error correction. If it is determined that it is a highlight error correction, the original pixel value before correction is output, and if it is determined that it is not a highlight error correction, a correction pixel value is output.

Furthermore, the second aspect of the present disclosure is:
An image processing method executed in an image processing apparatus,
A direction determination process in which the image signal correction unit detects a direction having the minimum pixel value gradient as the pixel value gradient direction in the pixel region including the target pixel;
A defect detection process for calculating the Laplacian based on the pixel value of the reference pixel in the minimum gradient direction detected in the direction determination process for the target pixel, and determining the presence or absence of a defect in the target pixel;
Executing a defect correction process for calculating a correction pixel value by applying a pixel value of a reference pixel in a direction detected in the direction determination process to a target pixel in which a defect is detected in the defect detection process;
In the direction determination process, there is an image processing method for executing a direction determination process using a weighted addition result of a plurality of pieces of gradient information calculated by a plurality of different gradient detection processes.

Furthermore, the third aspect of the present disclosure is:
A program for executing image processing in an image processing apparatus;
A direction determination process for detecting a direction having a minimum pixel value gradient as a pixel value gradient direction in the pixel region including the target pixel in the image signal correction unit;
A defect detection process for calculating the Laplacian based on the pixel value of the reference pixel in the minimum gradient direction detected in the direction determination process for the target pixel, and determining the presence or absence of a defect in the target pixel;
For the target pixel in which a defect is detected in the defect detection process, a defect correction process is executed to calculate a correction pixel value by applying a pixel value of a reference pixel in the direction detected in the direction determination process,
The direction determination process is in a program for executing a direction determination process using a weighted addition result of a plurality of pieces of gradient information calculated by a plurality of different gradient detection processes.

  Note that the program of the present disclosure is a program that can be provided by, for example, a storage medium or a communication medium provided in a computer-readable format to an information processing apparatus or a computer system that can execute various program codes. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the information processing apparatus or the computer system.

  Other objects, features, and advantages of the present disclosure will become apparent from a more detailed description based on embodiments of the present disclosure described below and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

According to the configuration of an embodiment of the present disclosure, there is provided an apparatus and a method capable of realizing highly accurate defect detection and correction for an image having a pixel array with various settings.
Specifically, a plurality of gradient detection information is acquired by applying a plurality of different methods to the pixel region including the target pixel. Further, the minimum gradient direction is detected based on the weighted addition of a plurality of gradient detection information. Further, a Laplacian is calculated based on the pixel value of the reference pixel of the same color as the target pixel in the detected minimum gradient direction, and the presence / absence of a defect in the target pixel is determined. Further, the correction pixel value is calculated by applying the pixel value of the reference pixel in the direction detected in the direction determination process to the target pixel in which the defect is detected.
By this processing, an apparatus and a method capable of realizing highly accurate defect detection and correction for an image having a pixel array with various settings are realized.

It is a figure explaining the structural example of an image pick-up element. It is a figure explaining the structural example of an image processing apparatus. It is a figure which shows the flowchart explaining the process which the image processing apparatus of this indication performs. It is a figure which shows the flowchart explaining the process which the image processing apparatus of this indication performs. It is a figure explaining the direction determination process which the image processing apparatus of this indication performs. It is a figure explaining the direction determination process which the image processing apparatus of this indication performs. It is a figure explaining the direction determination process which the image processing apparatus of this indication performs. It is a figure explaining the average value calculation process with a gradient weight which the image processing apparatus of this indication performs. It is a figure explaining the example of a weight determination process in the gradient weighted average value calculation process which the image processing apparatus of this indication performs. It is a figure explaining the example of a weight determination process in the gradient weighted average value calculation process which the image processing apparatus of this indication performs. It is a figure explaining the example of a weight determination process in the gradient weighted average value calculation process which the image processing apparatus of this indication performs. It is a figure explaining the gradient calculation process which the image processing apparatus of this indication performs. It is a figure explaining the gradient calculation process which the image processing apparatus of this indication performs. It is a figure explaining the gradient calculation process which the image processing apparatus of this indication performs. It is a figure explaining the gradient calculation process which the image processing apparatus of this indication performs. It is a figure explaining the direction determination process which the image processing apparatus of this indication performs. It is a figure explaining the defect detection process which the image processing apparatus of this indication performs. It is a figure explaining the pixel interpolation process which the image processing apparatus of this indication performs. It is a figure explaining the pixel interpolation process which the image processing apparatus of this indication performs. It is a figure explaining the pixel interpolation process which the image processing apparatus of this indication performs. It is a figure explaining the pixel interpolation process which the image processing apparatus of this indication performs. It is a figure explaining the Laplacian calculation process which the image processing apparatus of this indication performs. It is a figure explaining the Laplacian comparison process which the image processing apparatus of this indication performs. It is a figure explaining the defect correction process which the image processing apparatus of this indication performs. It is a figure explaining the defect correction process which the image processing apparatus of this indication performs. It is a figure explaining the modification of the defect correction process which the image processing apparatus of this indication performs. It is a figure explaining the example of a process with the highlight erroneous correction detection which the image processing apparatus of this indication performs. It is a figure explaining the example of a process with the highlight erroneous correction detection which the image processing apparatus of this indication performs. It is a figure explaining the example of a process with the highlight erroneous correction detection which the image processing apparatus of this indication performs. It is a figure explaining the example of a process with the highlight erroneous correction detection which the image processing apparatus of this indication performs.

The details of the image processing apparatus, the image processing method, and the program of the present disclosure will be described below with reference to the drawings. The description will be made according to the following items.
1. 1. Configuration example of image sensor 2. Configuration example of image processing apparatus 3. Specific examples of image processing 4. Direction determination processing 5. Modification of direction determination process About defect detection processing Defect correction processing 8. Deformation sequence of defect correction processing 10. Example of processing with highlight error correction determination Summary of composition of this disclosure

[1. Example of image sensor configuration]
A configuration example of the image sensor will be described with reference to FIG. FIG. 1 shows a configuration example of the following three image sensors.
(1) Bayer array (2) Quadrant Bayer RGB array (3) RGBW array

(1) The Bayer array is an array adopted in many cameras, and signal processing for a captured image having a color filter having the Bayer array is almost established.
However, with regard to (2) the four-divided Bayer type RGB array and (3) the RGBW type array, it is still considered that signal processing for an image photographed by an image sensor provided with these filters has been sufficiently studied. The current situation is that I can't say that.
Note that (2) the four-divided Bayer type RGB array corresponds to an array in which one R, G, B pixel of the Bayer array shown in (1) is set as four pixels.

  Hereinafter, (2) an image processing apparatus that performs signal processing on an image photographed by an imaging device including a color filter having a four-divided Bayer RGB array will be described.

[2. Configuration example of image processing apparatus]
FIG. 2 illustrates a configuration example of the imaging apparatus 100 that is a configuration example of the image processing apparatus of the present disclosure.
As illustrated in FIG. 2, the imaging apparatus 100 includes an optical lens 105, an imaging element (image sensor) 110, an image processing unit 120, a memory 130, and a control unit 140.

  2 is an example of the image processing apparatus according to the present disclosure, and the image processing apparatus according to the present disclosure includes an apparatus such as a PC. The image processing apparatus such as a PC does not include the optical lens 105 and the image sensor 110 of the image capturing apparatus 100 illustrated in FIG. 2, but includes other components, and includes an acquisition data input unit or a storage unit of the image sensor 100. It becomes composition.

  Hereinafter, the imaging apparatus 100 illustrated in FIG. 2 will be described as a representative example of the image processing apparatus of the present disclosure. 2 is a still camera, a video camera, or the like, for example.

An imaging element (image sensor) 110 of the imaging apparatus 100 shown in FIG. 2 has a configuration including a color filter composed of a four-divided Bayer RGB array described with reference to FIG.
Red (R) that transmits wavelengths near red,
Green (G) that transmits wavelengths in the vicinity of green,
Blue (B) that transmits wavelengths near blue,
It is an image sensor provided with a filter having these three types of spectral characteristics.
As described above, the four-divided Bayer RGB array corresponds to an array in which one pixel of the Bayer array shown in FIG. 1A is set as four pixels.

  The image sensor 110 having the four-divided Bayer RGB array 181 receives RGB light in units of pixels via the optical lens 105, generates an electrical signal corresponding to the received light signal intensity by photoelectric conversion, and outputs it. To do. With this image sensor 110, a mosaic image composed of three types of RGB spectra is obtained.

An output signal of the image sensor (image sensor) 110 is input to the image signal correction unit 200 of the image processing unit 120.
The image signal correction unit 200 performs correction processing of an image having a four-divided Bayer RGB array 181, for example, correction of defective pixels.

  The corrected image in the image signal correction unit 200 is input to the signal processing unit 250. The signal processing unit 250 generates a color image 183 by executing processing similar to that of the signal processing unit in the existing camera, for example, WB (white balance) adjustment, demosaic processing for setting RGB pixel values for each pixel, and the like. Output. The color image 183 is stored in the memory 130.

  Note that control signals from the control unit 140 are input to the optical lens 105, the image sensor 110, and the image processing unit 120, and imaging processing control and signal processing control are executed. For example, according to a program stored in the memory 130, the control unit 140 executes various processes in addition to image capturing in accordance with a user input from an input unit (not shown).

[3. Specific examples of image processing]
Next, processing executed in the image signal correction unit 200 of the image processing unit 120 in FIG. 2 will be described with reference to FIG.
FIG. 3A is an entire signal processing sequence showing processing executed in the image signal correction unit 200.
First, in step S101, one pixel to be processed (target pixel) is selected from the captured image input from the image sensor 110, and a pixel value gradient (gradient (Gradient) is referenced with reference to a pixel region (for example, N × N pixels) in the vicinity of the target pixel. )) Direction determination.
That is, the direction in which the pixel value gradient (gradient) is minimized is generated as the direction determination result.

  The direction in which the pixel value gradient (gradient) is the minimum corresponds to the edge direction and is the direction in which the pixel value does not change much. On the other hand, the direction perpendicular to the edge direction is a direction in which the pixel value gradient (gradient) increases and the change in pixel value is large.

  For example, when performing processing on the target pixel, as illustrated in FIG. 3, the processing is performed with reference to a pixel region 300 of N × N pixels centering on the target pixel 301. The example shown in FIG. 3 is an example in which N = 11.

Next, in step S102, it is determined whether or not the pixel of interest is a defective pixel. If a defect is included, defect correction is performed in step S103.
Details of each of these processes will be sequentially described below.

[4. About direction determination processing]
First, the pixel value gradient (gradient) direction determination process in step S101 of the flow of FIG. 3 will be described.

A detailed flow of the direction determination process is shown in FIG.
As shown in FIG. 4, in the direction determination process, an N × N pixel region 300 centered on the target pixel 301 is input, and the direction determination result of the pixel value gradient (gradient) corresponding to the target pixel 301 is obtained. It is executed as a process of outputting sequentially.

Specifically, first,
Calculating the gradient of the high frequency texture in step S121;
Calculating the gradient of the low-frequency texture in step S122;
Gradient calculation of luminance signal in step S123,
These three modes of gradient calculation are executed.
Further, in step S124,
A weighted average value is calculated for these three gradient calculation results, and based on this, a direction determination result is output in step S125.
Hereinafter, specific examples of these processes will be described.

FIG. 5 is a diagram illustrating an example of the gradient calculation process of the high frequency texture in step S121.
In the pixel area 300 of N × N pixels centering on the target pixel 301, as shown in FIG. 5, gradients in the following directions are calculated using pixel values of G pixels adjacent to or near the target pixel 301. .
Horizontal gradient: gradH
Vertical gradient: gradV
Upper right gradient: gradA
Bottom right gradient: gradD
Is calculated.
Specifically, the gradient of the high frequency texture is calculated according to (Equation 1) shown below.

・・・・・（式１）

... (Formula 1)

Gx, y means the G pixel value at the coordinate position (x, y).
N is the number of gradients in each direction.

FIG. 6 is a diagram illustrating an example of gradient calculation processing of the low-frequency texture in step S122.
In the pixel area 300 of N × N pixels centered on the pixel of interest 301, as shown in FIG. 6, gradients in the following directions are calculated using pixel values of G pixels adjacent to or near the pixel of interest 301. .
Horizontal gradient: gradH
Vertical gradient: gradV
Upper right gradient: gradA
Bottom right gradient: gradD
Is calculated.
Specifically, the gradient of the low frequency texture is calculated according to (Equation 2) shown below.

・・・・・（式２）

... (Formula 2)

Gx, y means the G pixel value at the coordinate position (x, y).
N is the number of gradients in each direction.

FIG. 7 is a diagram illustrating a luminance signal gradient calculation process in step S123.
In the pixel region 300 of N × N pixels centering on the target pixel 301, as shown in FIG.
First, a luminance signal is calculated by averaging four pixels of RGGB in units of 2 × 2 pixel areas.
Luminance = (R + G + G + B) / 4
It is.

Next, the following gradients in each direction are calculated using these average values.
Horizontal gradient: gradH
Vertical gradient: gradV
Upper right gradient: gradA
Bottom right gradient: gradD
Is calculated.
Specifically, the gradient of the low frequency texture is calculated according to (Equation 3) shown below.

・・・・・（式３）

... (Formula 3)

Note that Lx and y are the above brightness calculation formulas,
Luminance = (R + G + G + B) / 4
Is the luminance calculated by.
N is the number of gradients in each direction.

Next, the process of step S124 will be described with reference to FIG.
In step S124,
The high frequency texture gradient calculated in step S121,
The gradient of the low frequency texture calculated in step S122;
The gradient of the luminance signal calculated in step S123;
A weighted average value is calculated for these three gradient calculation results.

The example shown in FIG. 8 shows an example of calculation processing of the weighted average value gH of the gradient in the horizontal direction.
The gradient obtained by each of the three methods is averaged with an arbitrary weight.
The gradient of the high frequency texture calculated in step S121: gradHh,
Gradient of the low frequency texture calculated in step S122: gradH1,
Gradient of the luminance signal calculated in step S123: gradHi,
A result obtained by multiplying and adding the weights wh, wl, and wi is calculated as a weighted average value gH.

That is,
gH = wh × gradHh + wl × gradHl + wi × gradHi
The gradient weighted average value gH is calculated according to the above formula.

  The example shown in FIG. 8 is an example of calculation processing of the weighted average value gH of the gradient in the horizontal direction, but the weighted average value gV of the gradient in each direction is similarly applied to the vertical direction, the upper right direction, and the lower right direction. , GA, gD.

Each of the weights wh, wl, and wi is a weight for the following gradient.
The weight wh is a weight for the gradient of the high frequency texture calculated in step S121.
The weight wl is a weight for the gradient of the low frequency texture calculated in step S122.
The weight w i is a weight for the gradient of the luminance signal calculated in step S123,
Each of these weights.

As the weight, a preset value, for example, a fixed value such as a weight of 1: 1: 1 may be applied. However, a weight corresponding to the image feature may be set.
An example of weight setting according to image characteristics will be described with reference to FIGS.

First, an example of weight setting according to the resolution of the output image will be described with reference to FIG.
In FIG.
(A) Weight setting example when output image is full resolution,
(B) Weight setting example when output image is 1/2 resolution,
An example of weight setting according to these two resolutions is shown.

The full resolution is a case where an image corresponding to the resolution of the image sensor, that is, the pixel configuration of the image sensor is output.
The 1/2 resolution is an image in which the four pixel values of the same-color four-pixel block of the image sensor are averaged to set one pixel value and the total number of pixels is ¼. The image is halved.

(A) The weight when the output image is full resolution is set as follows, for example.
Weight for high frequency texture gradient: wh = 0.6
Weight for low frequency texture gradient: wl = 0.1
Weight for gradient of luminance signal: wi = 0.3
This is the setting.

on the other hand,
(B) The weight when the output image is 1/2 resolution is set as follows, for example.
Weight for high frequency texture gradient: wh = 0.1
Weight for low frequency texture gradient: wl = 0.6
Weight for gradient of luminance signal: wi = 0.3
This is the setting.

Thus, in the case of full resolution, the weight of the high frequency gradient is set larger than the other weights.
On the other hand, when the output resolution is lowered, the specific gravity of the low-frequency gradient weight is increased.
By setting such weights, gradient direction determination suitable for the output image is realized.

Next, with reference to FIG. 10, an example of processing for setting a weight according to the frequency band of the input image to be processed will be described.
In the example described below, for example, an image with many textures, that is, an image with many high-frequency regions, or an image region is set with a high-frequency gradient weight greater than other weights. On the other hand, for a planar image with little change in pixel value, this is a processing example in which the weight of the low frequency gradient is set larger than the other weights.

Various methods can be applied to determine whether an image is an image with many high-frequency regions or an image with many low-frequency regions. An example using Fourier transform will be described below as an example. .
FIG. 10 shows a processing sequence and a specific processing example when processing for setting a weight according to the frequency band of the input image is performed.
First, a two-dimensional Fourier transform is executed on the input image in step S151.

The Fourier transform will be described with reference to FIG.
When Fourier transform is performed on the input image, Fourier coefficients F having the same two-dimensional array as the image array are calculated.
That is, when a Fourier transform is performed on a W × H image in which the number of pixels of the image is horizontal W and vertical H, W × H Fourier coefficients F (u, v) corresponding to the number of W × H constituent pixels are calculated. Is done.
The Fourier coefficient F (u, v) is calculated according to the following equation.

  The Fourier coefficient F (u, v) calculated according to the above equation includes low-frequency power (amplitude) at the center and stores high-frequency power in the periphery as shown in the lower part of FIG.

In step S151 shown in FIG. 10, the Fourier coefficient is calculated by Fourier transform on the input image, that is, the output image of the image sensor as described above. The Fourier transform is calculated up to a preset period N.
Next, in step S152 shown in FIG. 10, the weights according to the calculated Fourier coefficients, that is,
Weight for high frequency texture gradient: wh,
Weight for low frequency texture gradient: wl,
Weight for gradient of luminance signal: wi,
These weights wh, wl, and wi are determined.

As shown in FIG. 10 (2), each weight is determined from the calculated Fourier coefficient according to the size of the frequency band.
The coordinate axes of the Fourier coefficients F (u, v) are set as u and v as shown in FIG. 10 (2), and three rectangular areas are set in advance.

That is,
(A) Outside area of u> T1 and v> T1: A
(B) Region of T1 ≧ u> T2 and T1 ≧ v> T2: B
(C) T2 ≧ −u and T2 ≧ v] central region: C
Set these areas.
Region A is a high frequency region,
Region C is the low frequency region,
Region B corresponds to an intermediate frequency region.

Each of the Fourier coefficients F (u, v) calculated in the Fourier transform in step S151 is included in any of the above (a) to (c).
In step S152, for example, each weight is determined with the following settings.

The sum of the Fourier coefficients of region A (high frequency) is
Weight for high-frequency texture gradient: wh.
The sum of the Fourier coefficients of region B (medium frequency)
The weight for the gradient of the luminance signal is wi.
The sum of the Fourier coefficients of region C (low frequency)
Weight for low frequency texture gradient: wl.

  By executing such a process, a weight corresponding to the frequency band of the image is set, and the gradient direction determination can be performed as an optimal process corresponding to the feature of the image. Note that the above processing may be executed in units of images, or may be executed in units of predetermined areas of images.

In step S124 shown in FIG.
The high frequency texture gradient calculated in step S121,
The gradient of the low frequency texture calculated in step S122;
The gradient of the luminance signal calculated in step S123;
A weighted average value is calculated for these three gradient calculation results.

That is,
gH = wh × gradHh + wl × gradHl + wi × gradHi
The gradient weighted average value gH is calculated according to the above formula.

Next, the direction determination process in step S125 shown in FIG. 4 will be described.
In step S125, the following values calculated in step S124, that is,
Weighted average value gH of the horizontal gradient,
Weighted average value gV of the gradient in the vertical direction,
Gradient weighted average value gA in the upper right direction,
Weighted average value gD of the gradient in the lower right direction,
These four values are compared, and the direction having the minimum value is set as the direction of the direction determination result.

  As described above, the direction in which the pixel value gradient (gradient) is the minimum corresponds to the edge direction and is a direction in which the pixel value does not change much. On the other hand, the direction perpendicular to the edge direction is a direction in which the pixel value gradient (gradient) increases and the change in pixel value is large.

For example, when a defective pixel (error pixel) with an incorrect pixel value is included in the image output from the image sensor and the defective pixel is corrected, the pixel value of the reference pixel selected from the surrounding pixels is used. Thus, a process for calculating a corrected pixel value is executed.
In selecting a reference pixel, more natural pixel value correction can be performed by selecting a pixel in a direction in which the pixel value gradient (gradient) is minimized. This is because an unnatural pixel value setting that occurs by using the pixel value of a pixel in a direction in which there is a sudden pixel value change is prevented.

The direction determination result obtained in step S125 is used for such processing.
An example of processing for using the direction determination result will be described later.

  Thus, in the direction determination process of the present disclosure, a weighted average is performed using a plurality of types of gradients. By combining a plurality of different gradients, for example, there is an advantage that textures of various frequencies can be handled.

In particular, the four-divided Bayer RGB array as shown in FIG. 1 (2) has a problem that the frequency of the texture that can be acquired by the phase is biased because the sampling intervals of the same color pixels are not equal.
In the configuration of the present disclosure, both the high-frequency texture and the low-frequency texture are acquired, and these are combined to execute the direction determination process of the pixel value gradient (gradient). By applying this method, it is possible to determine the direction with high accuracy.
Further, by using the luminance signal, it is possible to acquire a gradient from a signal sampled at equal intervals.

In the pixel value gradient (gradient) direction determination processing of the present disclosure, as described above,
(A) high frequency texture gradient,
(B) a low frequency texture gradient;
(C) the gradient of the luminance signal,
These three different gradients are calculated and combined to determine the direction of the pixel value gradient (gradient).
By this process, for example, a highly accurate direction determination is realized while suppressing the occurrence of an erroneous pixel value gradient (gradient) direction determination caused by, for example, a deviation in pixel arrangement.

  Conventionally, a technique for determining the texture direction of an image and detecting and correcting defects using pixel signals along the determined direction has been proposed. For texture direction determination, a luminance signal is used. There are known a technique to be used and a technique to use pixels that are most densely arranged in a checkered pattern such as a Bayer array G pixel. The latter method is more widely used because it can also determine the direction of higher frequency texture.

However, in the four-divided Bayer RGB array as shown in FIG. 1 (2), the G pixels are not arranged in a checkered pattern, so that the frequency of the texture that can be acquired is biased compared to the Bayer array. Therefore, even if the texture is the same, the direction determination accuracy may be reduced, for example, the direction determination result varies depending on the image position. In addition, since the gradient is calculated using pixels sampled at regular intervals, the texture direction of a specific frequency may not be determined well.
The method of the present disclosure described above solves these problems and enables highly accurate direction determination.

[5. About modification of direction determination process]
Next, a modified example of the direction determination process will be described.
(Modification 1 of direction determination processing)
FIG. 12 shows a processing example of Modification 1 of the direction determination processing.
In FIG.
High frequency texture gradient calculation processing example,
Example of low frequency texture gradient calculation process,
These are shown.

In the pixel area 300 of N × N pixels centered on the target pixel, as shown in FIG. 12, the gradients in the following directions are obtained using the pixel values of G, R, and B pixels adjacent to or near the target pixel 301. Is calculated.
Horizontal gradient: gradH
Vertical gradient: gradV
Upper right gradient: gradA
Bottom right gradient: gradD
Is calculated.

This processing example is a processing example that uses R pixels and B pixels in addition to G pixels when acquiring gradients of high-frequency and low-frequency textures.
Specifically, the gradient of the high frequency texture is calculated according to (Equation 4) shown below.

・・・・・（式４）

(Formula 4)

  Further, the gradient of the low frequency texture is calculated according to the following (Equation 5).

・・・・・（式５）

... (Formula 5)

(Modification 2 of direction determination processing)
FIG. 13 shows a processing example of Modification 2 of the direction determination processing.
In FIG.
An example of using the direction determination result for re-mosaic processing,
An example of using the direction determination result for demosaic processing,
These are shown.
As described above, the direction determination result can be applied not only to the correction of the defective pixel but also to various processes using the direction determination result.

Note that the re-mosaic process is a process of generating a different pixel array by changing the RGB array set for each pixel output from the image sensor.
Specifically, for example, when the output from the image sensor is the four-divided Bayer type RGB array shown in FIG. 1B, this is a process of changing this array to the Bayer array shown in FIG.
For example, the signal processing unit 250 in the image processing unit 120 of the imaging apparatus 100 illustrated in FIG. 2 is generally configured to perform signal processing on the Bayer array image signal illustrated in FIG. Therefore, when the output from the image sensor 110 is a pixel array different from the Bayer array, the signal processor 250 executes the re-mosaic process to change the pixel array to the Bayer array and inputs the processed signal to the signal processor 250. It becomes possible to adopt a configuration of an existing general signal processing unit.

  The demosaic process is a process for setting all RGB pixel values for each pixel. In other words, the configuration of FIG. 2 corresponds to generating an image corresponding to the color image 183.

In both re-mosaic processing and demosaic processing, in order to set any pixel value of RGB at a certain target pixel position (interpolation pixel position), refer to pixels of the same color as the set color for the interpolation pixel position from the surroundings The pixel value is selected as a pixel, and the pixel value of the selected reference pixel is applied to determine the pixel value at the target pixel position.
In this interpolation processing, it is possible to calculate a more natural interpolation pixel value by setting a direction in which the pixel value gradient (gradient) is small as the reference pixel selection direction.

  The embodiment according to each flow shown in FIG. 13 is an embodiment in which the direction determination result of the pixel value gradient (gradient) is used as information for determining the direction of the reference pixel in such re-mosaic processing or demosaic processing. It is.

(Modification 3 of direction determination processing)
FIG. 14 shows a processing example of Modification 3 of the direction determination processing.
In the example shown in FIG. 14, in addition to steps S <b> 121 to S <b> 123 of the direction determination process described above with reference to FIG. 4, a 4-pixel addition average gradient calculation process is added as step S <b> 131.

  Specifically, as shown in FIG. 15, an average of pixel values is calculated in units of four pixel blocks of the same RGB color, and a gradient is calculated from the average.

That is, one pixel having an addition average value of pixel values of four pixels is set in units of four pixel blocks shown in FIG. 15, and an image in which the total number of pixels is set to ¼ pixels is applied to apply horizontal and vertical. The gradient in each of the upper right and lower right directions is calculated.
Note that these gradient calculation methods are the same as the luminance signal gradient calculation processing described above with reference to FIG. 7. In the above-described (Equation 3), pixel values (= addition of 4 pixels) are used instead of luminance. It is calculated by setting (average value).

In step S124 of FIG.
The high frequency texture gradient calculated in step S121,
The gradient of the low frequency texture calculated in step S122;
The gradient of the luminance signal calculated in step S123;
The gradient of the 4-pixel addition average calculated in step S131,
A weighted average value is calculated for these four gradient calculation results.
In this way, various settings can be made by increasing or decreasing the type of gradient that is the processing target of the weighted average value.

(Modification 4 of direction determination processing)
The processing example described above has been described as a processing example for the four-divided Bayer type RGB array described above with reference to FIG. 1 (2), but other arrays, for example, the Bayer array shown in FIG. The above-described processing of the present disclosure can also be applied to pixel data of various arrays such as the WRGB array shown in 1 (3) or the 4-split WRGB type array as shown in FIG.

[6. About defect detection processing]
Next, the defect detection process in step S102 of the flow of FIG. 3 will be described.
The process in step S102 is a process for determining whether or not the target pixel is a defective pixel. If a defect is included, defect correction is performed in step S103.

FIG. 17 shows a detailed flow of the defect detection process.
An N × N pixel region 500 centered on the target pixel 501 is input, and the direction information detected in step S101 in the flow of FIG.
Enter.

In the defect detection process,
Horizontal pixel interpolation in step S201;
Vertical pixel interpolation in step S202;
Pixel interpolation in the upper right direction in step S203,
Lower right pixel interpolation in step S204,
Any one of these processes is selectively executed.

That is, pixel interpolation is performed by selecting a direction corresponding to the direction with the smallest gradient obtained as the direction determination result in step S101 of the flow of FIG.
For example, when the direction in which the gradient is minimum is the horizontal direction, the horizontal pixel interpolation in step S201 is executed.

A specific example of pixel interpolation processing will be described with reference to FIG.
FIG. 18 shows a processing example in which the target pixel is an R pixel.
In FIG.
An example of horizontal interpolation processing (corresponding to S201),
An example of interpolation processing in the lower right direction (corresponding to S204) is shown.

Interpolation processing is executed by selecting pixels of the same color from each direction and using these as reference pixels.
However, as shown in FIG. 18, the number of reference pixels changes depending on the direction.
In the horizontal direction interpolation processing example shown in FIG. 18, there are 6 R pixels of the same color as the R pixel that is the central target pixel in the horizontal direction, including the target pixel itself, and these 6 pixels can be acquired as reference pixels. It is.

However, in the example of interpolation processing in the lower right direction shown in FIG. 18, there are only three R pixels having the same color as the R pixel that is the central target pixel, including the target pixel in the lower right direction, and only these three pixels. However, it can only be used as a reference pixel.
The number of pixels that can be referred to varies depending on the reference direction. In particular, when the target pixel is an R pixel or a B pixel, the decrease in the number of reference pixels according to the reference direction becomes significant.

  In this way, when only reference pixels less than a preset threshold value can be obtained in a certain reference direction, as shown in FIG. 19, pixels of the same color that are perpendicular to the reference direction are selected, Based on the pixel value of the selected pixel, an interpolation process for setting an interpolation pixel value in the reference direction is executed. After setting this interpolation pixel as a reference pixel and increasing the number of reference pixels, based on these reference pixels, interpolation processing is performed as a pixel value setting for the pixel of interest.

  In an array in which the sampling interval is not constant, such as the four-divided Bayer RGB array described above with reference to FIG. 1 (2), the number of pixels may be small depending on the direction. Increase pixels. By this processing, it is possible to increase the number of reference pixels, and it is possible to calculate a corrected pixel value with higher accuracy.

As described above, when the number of reference pixels is small in the reference direction, it is possible to set the same number of reference pixels in all directions and all colors of RGB by setting interpolation pixels in the reference direction.
With this process, in the process of setting the correction pixel value for the target pixel, not only correction with higher accuracy is possible, but the same process can be performed in all directions and in all colors, so that the same correction circuit can be used. It is also possible to reduce the circuit scale provided in the apparatus.

The process described with reference to FIG. 19 is an example of an interpolation process in which pixels of the same color that are perpendicular to the reference direction are selected and an interpolation pixel value is set in the reference direction based on the pixel value of the selected pixel. Met.
The process for setting the interpolation pixel value in the reference direction is not limited to the pixel in the direction perpendicular to the reference direction, and the same color pixels around the interpolation pixel position in any direction are applied. May be.
This processing example will be described with reference to FIGS.

FIG. 20 shows an example of processing when a reference pixel is set in the lower right direction, as in FIG.
The target pixel 501 is a center R pixel, and is an example of processing for setting an R pixel as a reference pixel in the lower right direction.
Processing when an R pixel is set at the reference pixel interpolation position 521 shown in the drawing will be described.
FIG. 20 shows images of (x, y) = (1, 1) to (11, 11) where X is the horizontal direction and Y is the vertical direction.
The target pixel 501 is an R pixel at the coordinate position (x, y) = (6, 6).
The reference pixel interpolation position 521 is the position of the B pixel at the coordinate position (8, 8).

The R pixel is interpolated at the position of the B pixel at the coordinate position (8, 8).
In this example, the same color pixels are selected not only from the vertical direction in the reference direction (in this example, the lower right direction) but also from all directions. Specifically, for example, a preset number of the same color pixels within a preset distance from the reference pixel interpolation position 521 are selected.
The example illustrated in FIG. 20 illustrates a processing example in which five R pixels around the reference pixel interpolation position 521 are selected as pixels to be applied to the interpolation pixel value calculation. This is an R pixel surrounded by a bold frame shown in the figure. Specifically, the following R pixels.
(1) R pixel at coordinate position (x, y) = (6, 7),
(2) R pixel at coordinate position (x, y) = (9, 7),
(3) R pixel at coordinate position (x, y) = (6, 10),
(4) R pixel at coordinate position (x, y) = (9, 10),
(5) R pixel at coordinate position (x, y) = (10, 10),
An average of the pixel values of these five R pixels is calculated and set as the interpolation pixel value of the R pixel at the reference pixel interpolation position 521.

  That is, the interpolation pixel value Ra of the R pixel at the reference pixel interpolation position 521 is calculated according to the following equation.

A specific example of the interpolation pixel value calculation process when the number of pixels applied to the interpolation process is 3 will be described with reference to FIG.
The example illustrated in FIG. 21 illustrates a processing example in which three R pixels around the reference pixel interpolation position 521 are selected as pixels to be applied to the interpolation pixel value calculation. This is an R pixel surrounded by a bold frame shown in the figure. Specifically, the following R pixels.
(1) R pixel at coordinate position (x, y) = (6, 7),
(2) R pixel at coordinate position (x, y) = (9, 7),
(3) R pixel at coordinate position (x, y) = (9, 10),
An average of the pixel values of these three R pixels is calculated and set as the interpolation pixel value of the R pixel at the reference pixel interpolation position 521.

  That is, the interpolation pixel value Ra of the R pixel at the reference pixel interpolation position 521 is calculated according to the following equation.

Thus, as a process of interpolating reference pixels in the reference direction,
A process of applying a pixel of the same color as the target pixel in a direction perpendicular to the reference direction described with reference to FIG.
A process of applying a pixel of the same color as the pixel of interest around the interpolation pixel position in the reference direction described with reference to FIGS.
Any of these processes can be applied.

Next, the Laplacian calculation process in step S205 of the flow of FIG. 17 will be described with reference to FIG.
As shown in FIG. 22A, five pixels of the same color including the target pixel 501 extracted in the interpolation process described above are selected.

Further, using these five pixels, as shown in FIG. 22B, they are arranged in a line in the order of pixel positions, and three types of combination arrays of pixels including the pixel of interest 501 are set. That is,
(B1) a three-pixel array in which the target pixel 501 is set to the right end;
(B2) a three-pixel array in which the target pixel 501 is set at the center;
(B3) a three-pixel array in which the target pixel 501 is set at the left end;
Set these three arrays.

Based on each of the three three-pixel arrays (b1) to (b3), three types of Laplacians L1, L2, and L3 are calculated. That is,
(L1) Laplacian L1, based on the three-pixel arrangement of (b1) above,
(L2) Laplacian L2 based on the three-pixel arrangement of (b2) above,
(L3) Laplacian L3 based on the three-pixel arrangement of (b3) above,
These Laplacians are calculated.

The Laplacian is calculated according to (Equation 6) below.
That is, the Laplacian L _i when the pixel position of each pixel in the three-pixel array is i−1, i, i + 1 from the left and the pixel values at each pixel position are G _i−1 , G _i , G _{i + 1} is as follows: (Equation 6)
L _i = G _i-1 + G _{i + 1} -2G _i
... (Formula 6)

Next, the Laplacian comparison and defect detection processing in step S206 of the flow of FIG. 17 will be described with reference to FIG.
As shown in FIG. 23, the defect detection is performed by comparing the Laplacians calculated in step S205.

In particular,
L1> τ and
L2> τ and
L3> τ,
At this time, it is determined that the target pixel is a defective pixel.
Note that τ is a preset threshold value.

For example,
Two examples shown in FIG.
L1> τ and
L2> τ and
L3> τ,
In this example, these conditions are satisfied and the target pixel 501 is determined to be a defective pixel.

On the other hand, two examples shown in FIG.
L1> τ and
L2> τ and
L3> τ,
In this example, these conditions are not satisfied and it is determined that the target pixel 501 is not a defective pixel.

[7. About defect correction processing]
Next, the defect correction process in step S103 of the flow of FIG. 3 will be described.
Details of the correction processing will be described with reference to FIGS.

  As shown in FIG. 24, five pixels of the same color including the target pixel 501 used for defect detection are applied to the correction process. The five pixels are arranged in the direction in which the pixel value gradient (gradient) is small in the direction determination processing in step S101 of the flow of FIG. That is, there are five pixels including the target pixel described with reference to FIGS. 18 to 22 and the reference pixel. Some of the pixels may be pixels generated by the interpolation processing described with reference to FIG.

As shown in FIG. 24, using the five pixels arranged in the direction of small pixel value gradient (gradient) including the target pixel 501, first, in step S301, a reference inter-pixel gradient is calculated.
This reference inter-pixel gradient is a pixel value gradient (gradient) between two pixels on both sides of the target pixel 501.
As shown in FIG.
A pixel value gradient (gradient) g1 of the reference pixel 511 and the reference pixel 512;
Pixel value gradient (gradient) g2 between the reference pixel 513 and the reference pixel 514,
These are calculated.

  Further, in step S302, the correction value, that is, the correction pixel value of the target pixel 501 is calculated by applying the reference inter-pixel gradient.

  A specific example of the reference inter-pixel gradient calculation process in step S301 and the corrected pixel value calculation process in step S302 will be described with reference to FIG.

  As shown in FIG. 25A, the reference pixel gradient calculation process in step S301 is a process of calculating a pixel value gradient (gradient) between two reference pixels on both sides sandwiching the target pixel. .

The example shown in FIG. 25A shows an example of R pixel processing. R1 to R5 are pixel values of each pixel.
The reference pixel gradient calculation process in step S301 is calculated by the following equation.
g1 = | R1-R2 |
g2 = | R4-R5 |
In step S301, the gradient between these two reference pixels is calculated.

Next, the process of step S302, that is, the corrected pixel value calculation process will be described with reference to FIG.
Let the correction pixel value be R3 ′.
The corrected pixel value R3 ′ is calculated according to the following formula.
When g1 ≦ g2,
R3 ′ = α × R1 + (1−α) × R2
When g1> g2,
R3 ′ = β × R4 + (1-β) × R5
However, α and β are preset parameters of 0 or more and 1 or less.

Through these processes, the corrected pixel value of the target pixel 501 determined as a defective pixel is calculated, and this corrected pixel value is set as the pixel value of the target pixel.
These processes are executed, for example, in the image signal correction unit 200 in the image processing unit 120 of the imaging apparatus 100 shown in FIG.
The corrected image in which the correction pixel value is set is output to the signal processing unit 250. In the signal processing unit 250, the same processing as the signal processing unit in the existing camera, for example, WB (white balance) adjustment, RGB for each pixel A demosaic process or the like for setting pixel values is executed to generate and output a color image 183.

As described above, in the processing of the present disclosure, whether or not the target pixel is a defective pixel is determined by using a plurality of pixels arranged in a direction with a small pixel value gradient (gradient) including the target pixel that is the processing target pixel. When the determination is made, the same color pixels in the direction are supplemented by interpolation, and a reference pixel of a predetermined number or more is set.
Furthermore, also in the correction process with respect to the target pixel determined to be a defective pixel, the correction process using the pixel values of the reference pixels exceeding the specified number set by the interpolation process executed as necessary is executed.

  Thus, in the processing of the present disclosure, detection / correction accuracy is improved by executing processing in which the number of reference pixels is set to a predetermined number or more. In addition, since the same number of pixels can be prepared in all directions and all colors of RGB, it is possible to execute processing according to the same algorithm in all directions and all colors, and processing using the same processing circuit is possible. Also, the circuit scale can be reduced. In addition, it is possible to reduce the superiority or inferiority of correction caused by the pixel arrangement, that is, the difference between the direction in which the correction is good and the direction in which the correction is not good. This is particularly effective for pixels with a wide sampling interval.

[8. Deformation sequence of defect correction processing]
In the above-described processing example, the processing example for the four-divided Bayer RGB array described above with reference to FIG. 1B has been described. However, the above-described defect detection and correction processing also includes the above-described direction determination processing. Similarly, for other types of pixel data such as a Bayer array shown in FIG. 1 (1), a WRGB array shown in FIG. 1 (3), or a 4-split WRGB type array shown in FIG. However, the processing of the present disclosure can be applied.

  Also. In the above-described embodiment, a process example in which pixel interpolation is performed in the defect detection process is described. However, pixel interpolation may be performed in the correction process without performing pixel interpolation in the defect detection. Moreover, it is good also as a structure which performs pixel interpolation in the case of direction determination.

[9. Example of processing with highlight correction correction]
Next, an example of processing accompanied with highlight error correction determination will be described with reference to FIG.
For example, when photographing a starry sky, the star is photographed as a bright spot in the dark. In a photographed image by a camera, high brightness pixels are set in a low brightness pixel, and it may be determined that such a high brightness pixel is a defective pixel. If the pixel is recognized as a defective pixel, the pixel indicating the original star is corrected to the same pixel value as that of the surrounding pixels, that is, a low luminance pixel. Such erroneous correction is called highlight erroneous correction.

  Hereinafter, a description will be given of an embodiment in which it is verified whether the correction executed as the correction of the defective pixel is a highlight error correction, and when it is determined that the correction is a highlight error correction, a pixel value before correction is output.

FIG. 27 shows a flowchart for explaining the processing sequence of this embodiment.
The process according to the flow shown in FIG. 27 is a process executed in the image signal correction unit 200 of the image processing unit 120 in FIG.
First, in step S401, one pixel to be processed (target pixel) is selected from the captured image input from the image sensor 110, and a pixel value gradient (gradient) is selected with reference to a pixel region (for example, N × N pixels) in the vicinity of the target pixel. )) Direction determination.
The input image is an image 600 shown in FIG. 27, for example, and is an N × N image centered on the pixel of interest 601.

The direction determination process in step S401 is the same process as the direction determination process in step S101 of the flow illustrated in FIG. 3 described above. That is, the direction in which the pixel value gradient (gradient) is minimized is generated as the direction determination result.
The direction in which the pixel value gradient (gradient) is the minimum corresponds to the edge direction and is the direction in which the pixel value does not change much. On the other hand, the direction perpendicular to the edge direction is a direction in which the pixel value gradient (gradient) increases and the change in pixel value is large.

  For example, when performing processing on the target pixel, as illustrated in FIG. 27, the processing is performed with reference to a pixel region 600 of N × N pixels centered on the target pixel 601. The example shown in FIG. 27 is an example in which N = 11.

Next, in step S402, a defect detection process for determining whether the target pixel is a defective pixel is executed.
The defect detection process in step S402 is the same process as the defect detection process in step S102 in the flow of FIG. 3 described above. That is, the defect detection by the Laplacian calculation described above with reference to FIGS. 17 to 23 is executed.

The next step S403 is a tree step according to the result of whether or not the target pixel is a defective pixel.
If it is determined that the pixel is not defective, the process proceeds to step S407 without executing the correction process, and the original pixel value before correction is output.
On the other hand, if it is determined that the pixel is a defective pixel, the process proceeds to step S404, and correction processing is executed.

  The correction process in step S404 is the same process as the defect detection process in step S103 in the flow of FIG. 3 described above. That is, the pixel value correction process described above with reference to FIGS. 24 to 25 is executed.

Next, the process proceeds to step S405, and a determination process is performed to determine whether the correction process executed in step S404 is a highlight error correction.
In steps S405 to S408, the output pixel value of the target pixel is determined and output as follows according to the determination result as to whether the correction process executed in step S404 is a highlight error correction.

That is,
If it is determined that the correction process executed in step S404 is not highlight error correction, the process proceeds to step S408, and a corrected pixel value is output.
When it is determined that the correction process executed in step S404 is highlight highlight correction, the process proceeds to step S407, and the original pixel value before correction is output.
Details of this highlight error correction determination processing and output pixel value determination processing will be described with reference to FIGS.

FIG. 28 is a diagram for explaining detailed processing of highlight erroneous correction determination processing and output pixel determination processing.
In the highlight error correction determination process,
A corrected pixel value 611 of a target pixel to be processed;
An original pixel value 612 before correction of a target pixel to be processed;
A neighboring region (for example, an N × N pixel region centered on the target pixel) information 613 of the target pixel to be processed;
Enter these.

First, in step S501, white balance calculation is performed on the neighboring area information 613 of the target pixel.
This process will be described with reference to FIG.

FIG. 29 shows a pixel region 600 of N × N pixels as neighboring region information centered on the pixel of interest 601.
The white balance of the pixel area 600 is calculated.
The white balance is calculated as a ratio of these average values by calculating an average pixel value: aveR, aveG, and aveB for each of R, G, and B in the vicinity of the region 600.
Specifically, the white balance is calculated according to the following formula.

In step S501 in FIG. 28, the white balance in the vicinity region of the target pixel is calculated according to the above formula.
Next, in step S502, a different color pixel average value is calculated. The different color pixel average value is an average value of colors different from the color of the target pixel as the highlight error correction determination target.
As shown in FIG. 30, the different color pixel average value P is calculated according to the following equation, for example.

  The above formula is an example of calculating the different color pixel average value P when the target pixel is a G pixel. As shown in FIG. 30, different color pixels B1, B2 and R1, R2 are selected from the eight pixels around the target pixel 601, and these are calculated with the previously calculated white balance values (aveG / aveR) and (aveG / aveB). ) To calculate the different color pixel average value P.

  The above formula is an example of calculation when the target pixel is a G pixel. However, in the case of other B and R pixels, a pixel of a different color from the target pixel is selected from around the target pixel. The respective pixel values are added for each color unit, white balance adjustment is performed, and the different color pixel average value P is calculated.

Next, in step S503, it is determined whether the defect correction executed in step S404 in FIG. 27 is a highlight error correction, and whether the output pixel is set as a correction pixel value or an original pixel value according to the determination result. decide.
This output pixel selection method is executed according to the following equation, as shown in FIG.

In the above formula,
Gorg: Original pixel value of the target pixel,
Gcor: correction pixel value of the target pixel,
P: transplanted pixel average value,
It is.

That is, the one closer to the different color pixel average value P before and after correction is output.
If the difference between the corrected pixel value after correction and the different color pixel average value P is greater than or equal to the difference between the original pixel value before correction and the different color pixel average value P, it is determined that the highlight is erroneously corrected. A process of outputting the original pixel value is performed.
When the difference between the corrected pixel value after correction and the different color pixel average value P is less than the difference between the original pixel value before correction and the different color pixel average value P, correct correction that is not highlight error correction has been performed. A process of determining and outputting a corrected pixel value is performed.

The process of steps S405 to S408 in the flow of FIG. 27 is executed as the process described with reference to FIGS.
With this process, when highlight error correction is performed, the original pixel value can be restored and output.

[10. Summary of composition of the present disclosure]
As described above, the embodiments of the present disclosure have been described in detail with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present disclosure, the claims should be taken into consideration.

The technology disclosed in this specification can take the following configurations.
(1) having an image signal correction unit for executing image correction processing;
The image signal correction unit
A direction determination process for detecting a direction having a minimum pixel value gradient as a pixel value gradient direction in a pixel region including the target pixel;
A defect detection process for calculating the Laplacian based on the pixel value of the reference pixel in the minimum gradient direction detected in the direction determination process for the target pixel, and determining the presence or absence of a defect in the target pixel;
Executing a defect correction process for calculating a correction pixel value by applying a pixel value of a reference pixel in a direction detected in the direction determination process to a target pixel in which a defect is detected in the defect detection process;
The image processing apparatus that executes the direction determination process by applying a weighted addition result of a plurality of pieces of gradient information calculated by a plurality of different gradient detection processes.

(2) In the direction determination process, the image signal correction unit calculates pixel value gradient information corresponding to a high frequency texture, pixel value gradient information corresponding to a low frequency texture, and pixel value gradient information corresponding to a luminance signal. The image processing apparatus according to (1), wherein a direction having a minimum pixel value gradient is detected based on a weighted addition result of these three types of gradient information.
(3) The image signal correction unit calculates the pixel value gradient information corresponding to the high frequency texture by applying the pixel value difference of the adjacent pixel, and calculates the pixel value gradient information corresponding to the low frequency texture to the non-adjacent pixel. The image processing apparatus according to (2), wherein calculation is performed by applying a pixel value difference.
(4) The image signal correcting unit calculates a luminance signal based on a pixel value of each RGB pixel in a pixel region unit including each RGB pixel, and applies the calculated luminance signal in each region unit to correspond to the luminance signal. The image processing apparatus according to (2) or (3), wherein pixel value gradient information is calculated.

(5) The image signal correction unit executes a process of changing a weight set in the weighted addition process of the three types of gradient information according to the resolution of the output image. Set the weight for pixel value gradient information corresponding to texture higher than other gradient information, and if the output image has a low resolution, prioritize the weight for pixel value gradient information corresponding to low frequency texture over other gradient information. The image processing apparatus according to any one of (1) to (4), wherein the image processing apparatus is set to be high.
(6) The image signal correction unit executes a process of changing the weight set in the weighted addition process of the three types of gradient information according to the frequency band of the input image that is the processing target image, and the input image is high If there are many frequency regions, set a higher weight for pixel value gradient information corresponding to high-frequency textures in preference to other gradient information, and if the input image contains many low-frequency regions, pixel values corresponding to low-frequency textures The image processing apparatus according to any one of (1) to (5), wherein a weight for gradient information is set higher than other gradient information.

(7) The image signal correction unit executes pixel value correction of an image in which RGB colors are arranged in units of 2 × 2 pixels or an image in which RGBW colors are arranged in units of 2 pixels of 2 × 2. The image processing apparatus according to any one of (1) to (6).
(8) In the defect detection processing, the image signal correction unit selects, from the minimum gradient direction, a pixel having the same color as the target pixel as a defect detection target as a reference pixel, and based on different combinations of the target pixel and the selected pixel. A comparison between a plurality of Laplacians calculated in advance and a predetermined threshold value is performed, and it is determined whether the target pixel is a defective pixel based on a comparison result. Image processing apparatus.

(9) The image signal correction unit selects four pixels from the minimum gradient direction using a pixel having the same color as the target pixel as a defect detection target as a reference pixel, and calculates based on different combinations of the target pixel and the two selected pixels. The image processing according to (8), wherein the comparison is performed between the three Laplacians and a predetermined threshold value, and the target pixel is determined to be a defective pixel when all three Laplacians are greater than the threshold value. apparatus.
(10) In the defect detection process, the image signal correcting unit may select the minimum gradient when four pixels having the same color as the target pixel as a defect detection target cannot be selected from the minimum gradient direction within a predetermined reference area. Pixel interpolation based on the pixel value of the same color pixel as the target pixel around the different color pixel position is executed at the different color pixel position of the color different from the target pixel in the direction, and the interpolation pixel generated by the pixel interpolation is used as a reference pixel The image processing device according to (8) or (9) to be set.

(11) The image signal correction unit according to any one of (8) to (10), wherein the image signal correction unit calculates a correction pixel value of the pixel of interest by weighted addition of pixel values of the reference pixel in the defect correction processing. Image processing device.
(12) In the defect correction process, the image signal correction unit calculates a pixel value gradient between two reference pixels on both sides centered on the target pixel, and calculates two pixel values in a direction in which the pixel value gradient is small. The image processing apparatus according to any one of (8) to (11), wherein a corrected pixel value of the target pixel is calculated by weighted addition of pixel values.
(13) The image signal correction unit performs a highlight error correction determination process for determining whether or not the correction process executed in the defect correction process is a highlight error correction, and determines that it is a highlight error correction. In such a case, the original pixel value before correction is output, and when it is determined that the highlight correction is not erroneous, the corrected pixel value is output, according to any one of (1) to (12).

  Furthermore, a configuration of the present disclosure includes a method of processing executed in the above-described apparatus and system, a program for executing the processing, and a recording medium on which the program is recorded.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet, and installed on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

As described above, according to the configuration of an embodiment of the present disclosure, it is possible to provide an apparatus and a method capable of realizing highly accurate defect detection and correction for an image having a pixel array with various settings. .
Specifically, a plurality of gradient detection information is acquired by applying a plurality of different methods to the pixel region including the target pixel. Further, the minimum gradient direction is detected based on the weighted addition of a plurality of gradient detection information. Further, a Laplacian is calculated based on the pixel value of the reference pixel of the same color as the target pixel in the detected minimum gradient direction, and the presence / absence of a defect in the target pixel is determined. Further, the correction pixel value is calculated by applying the pixel value of the reference pixel in the direction detected in the direction determination process to the target pixel in which the defect is detected.
By this processing, an apparatus and a method capable of realizing highly accurate defect detection and correction for an image having a pixel array with various settings are realized.

DESCRIPTION OF SYMBOLS 100 Imaging device 105 Optical lens 110 Imaging element (image sensor)
DESCRIPTION OF SYMBOLS 120 Image processing part 130 Memory 140 Control part 181 4 division | segmentation Bayer type RGB arrangement | sequence 183 Color image 200 Image signal correction | amendment part 250 Signal processing part

Claims (15)

An image signal correction unit that executes image correction processing;
The image signal correction unit
A direction determination process for detecting a direction having a minimum pixel value gradient as a pixel value gradient direction in a pixel region including the target pixel;
A defect detection process for calculating the Laplacian based on the pixel value of the reference pixel in the minimum gradient direction detected in the direction determination process for the target pixel, and determining the presence or absence of a defect in the target pixel;
Executing a defect correction process for calculating a correction pixel value by applying a pixel value of a reference pixel in a direction detected in the direction determination process to a target pixel in which a defect is detected in the defect detection process;
The image processing apparatus that executes the direction determination process by applying a weighted addition result of a plurality of pieces of gradient information calculated by a plurality of different gradient detection processes. The image signal correction unit
In the direction determination process,
Pixel value gradient information corresponding to high frequency texture,
Pixel value gradient information corresponding to low frequency texture,
Pixel value gradient information corresponding to the luminance signal,
The image processing apparatus according to claim 1, wherein a direction having a minimum pixel value gradient is detected based on a weighted addition result of the three types of gradient information. The image signal correction unit
The pixel value gradient information corresponding to the high frequency texture is calculated by applying a pixel value difference between adjacent pixels,
The image processing apparatus according to claim 2, wherein the pixel value gradient information corresponding to the low frequency texture is calculated by applying a pixel value difference between non-adjacent pixels. The image signal correction unit
The luminance signal corresponding to the luminance signal is calculated by calculating a luminance signal based on the pixel value of each RGB pixel in a pixel region unit including each RGB pixel and applying the calculated luminance signal in each region unit. An image processing apparatus according to 1. The image signal correction unit
Executing a process of changing the weight set in the weighted addition process of the three types of gradient information according to the resolution of the output image;
If the output image is high resolution, set the weight for the pixel value gradient information for high-frequency textures higher than other gradient information,
The image processing apparatus according to claim 1, wherein when the output image has a low resolution, the weight for the pixel value gradient information corresponding to the low-frequency texture is set higher with priority over the other gradient information. The image signal correction unit
Executing a process of changing the weight set in the weighted addition process of the three types of gradient information according to the frequency band of the input image that is the processing target image;
If the input image contains a lot of high frequency areas, set the weight for the pixel value gradient information corresponding to the high frequency texture higher than other gradient information,
The image processing apparatus according to claim 1, wherein when the input image includes many low frequency regions, the weight for the pixel value gradient information corresponding to the low frequency texture is set higher than other gradient information. The image signal correction unit
The image processing apparatus according to claim 1, wherein pixel value correction is performed on an image in which RGB colors are arranged in units of 2 × 2 pixels or an image in which RGBW colors are arranged in units of 2 × 2 pixels. The image signal correction unit
In the defect detection process,
A pixel having the same color as the target pixel to be detected as a defect is selected as a reference pixel from the minimum gradient direction, and a plurality of Laplacians calculated based on different combinations of the target pixel and the selected pixel are compared with a predetermined threshold value. The image processing apparatus according to claim 1, wherein it is determined whether or not the target pixel is a defective pixel based on a comparison result. The image signal correction unit
Four pixels are selected from the minimum gradient direction using a pixel of the same color as the target pixel as a defect detection target as a reference pixel, three Laplacians calculated based on different combinations of the target pixel and the two selected pixels, and a predetermined threshold value The image processing apparatus according to claim 8, wherein the pixel of interest is determined to be a defective pixel when all three Laplacians are greater than the threshold. The image signal correction unit
In the defect detection process,
In the case where four pixels having the same color as the target pixel as a defect detection target cannot be selected from the minimum gradient direction within the predetermined reference area,
Pixel interpolation based on the pixel value of the same color pixel as the target pixel around the different color pixel position is executed at the different color pixel position of the color different from the target pixel in the minimum gradient direction, and the interpolation pixel generated by the pixel interpolation is The image processing apparatus according to claim 8, wherein the image processing apparatus is set as a reference pixel. The image signal correction unit
In the defect correction process,
The image processing apparatus according to claim 8, wherein a corrected pixel value of the target pixel is calculated by weighted addition of pixel values of the reference pixels. The image signal correction unit
In the defect correction process,
Calculate a pixel value gradient between two reference pixels on both sides centered on the target pixel, and calculate a corrected pixel value of the target pixel by weighted addition of two pixel values in a direction in which the pixel value gradient is small The image processing apparatus according to claim 8. The image signal correction unit
Executing a highlight error correction determination process for determining whether the correction process performed in the defect correction process is a highlight error correction;
If it is determined that the highlight correction is incorrect, the original pixel value before correction is output.
The image processing apparatus according to claim 1, wherein when it is determined that the highlight correction is not erroneous correction, a corrected pixel value is output. An image processing method executed in an image processing apparatus,
A direction determination process in which the image signal correction unit detects a direction having the minimum pixel value gradient as the pixel value gradient direction in the pixel region including the target pixel;
A defect detection process for calculating the Laplacian based on the pixel value of the reference pixel in the minimum gradient direction detected in the direction determination process for the target pixel, and determining the presence or absence of a defect in the target pixel;
Executing a defect correction process for calculating a correction pixel value by applying a pixel value of a reference pixel in a direction detected in the direction determination process to a target pixel in which a defect is detected in the defect detection process;
In the direction determination process, an image processing method for executing a direction determination process using a weighted addition result of a plurality of pieces of gradient information calculated by a plurality of different gradient detection processes. A program for executing image processing in an image processing apparatus;
A direction determination process for detecting a direction having a minimum pixel value gradient as a pixel value gradient direction in the pixel region including the target pixel in the image signal correction unit;
A defect detection process for calculating the Laplacian based on the pixel value of the reference pixel in the minimum gradient direction detected in the direction determination process for the target pixel, and determining the presence or absence of a defect in the target pixel;
For the target pixel in which a defect is detected in the defect detection process, a defect correction process is executed to calculate a correction pixel value by applying a pixel value of a reference pixel in the direction detected in the direction determination process,
The direction determination process is a program for executing a direction determination process using a weighted addition result of a plurality of pieces of gradient information calculated by a plurality of different gradient detection processes.

Images