JP2011234225A - Image processing method and image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing method for achieving both excellent shading compensation and suppression of an increase in random noise.SOLUTION: An image processing method comprises: a step S53 for detecting a dark-time image shading component from a dummy image signal related to a dark-time image; a step S55 for compensating the dark-time image signal shading using the detected dark-time image shading component; a step S57 for smoothing the dark-time image signal of compensated shading; a step for detecting a light-time image shading component from a dummy image signal related to a light-time image; a step for compensating the light-time image signal shading using the detected light-time image shading component; and a step for subtracting the smoothed dark-time image signal from the light-time image signal of compensated shading.

Description

  The present invention relates to an image processing method and an image processing apparatus that remove noise contained in an image signal.

  In general, it is known that a signal read from a solid-state imaging device has shading due to a manufacturing process of the device as fixed pattern noise (FPN).

  Specifically, in the imaging device, a drive signal is applied from both left and right sides, but the length of the signal line until the drive signal is transmitted varies depending on the position in the left and right direction of the pixel (which column is) Therefore, the transmission time of the drive signal is different between the central pixel and the peripheral pixel in the left-right direction of the image sensor (specifically, the transmission of the drive signal to the central part is It will be later in time than the transmission of the drive signal). Furthermore, the length of the signal line until the signal is transmitted differs depending on the position in the left-right direction of the pixel, but the impedance of the signal line differs depending on the difference in length. Due to the difference in impedance or the like, the drive signal transmitted to the central pixel has a slightly different waveform (for example, the signal edge becomes smoother) than the drive signal transmitted to the peripheral pixel. Thus, the on / off timing of the transistor and the like is shifted by the amount that the waveform is rounded.

  For this reason, a difference in exposure time occurs between the central portion and the peripheral portion in the left-right direction of the image sensor, and this appears in the captured image as dark shading in the horizontal direction. Thus, various techniques for correcting such shading have been proposed.

  First, FIG. 10 is a diagram for explaining a first conventional technique for correcting shading.

  The first prior art takes two images, a light image and a dark image, and subtracts the dark image from the light image.

  That is, the bright image is an image including a horizontal shading component HS, a subsequent defective pixel DP, a random noise white float RW, a random noise black sink RB, a fixed vertical stripe FP, and the like, as illustrated.

  The dark image is also an image including a horizontal shading component HS, a subsequent defective pixel DP, a random noise white float RW, a random noise black sink RB, a fixed vertical stripe FP, and the like, as shown in the figure.

  Therefore, by subtracting the dark image from the bright image, it is possible to remove the horizontal shading component HS, the fixed vertical stripe FP, and the like. However, since the late defective pixel DP has a level difference between the bright image and the dark image, it may be reduced by subtraction, but it is considered that the remaining defective pixel DP remains. In addition, it is considered that random noise is not necessarily reduced even if subtraction is performed, but may increase in some cases. In the example shown in FIG. 10, the random noise white float RW included in the dark image appears as the random noise black sink RB in the subtracted image, and the random noise black float RB appears as the random noise white float RW in the subtracted image. As a result, random noise has increased.

  Next, FIG. 11 is a diagram for explaining a second prior art for correcting shading.

  This second conventional technique takes two images, a light image and a dark image, smoothes the dark image, and subtracts the smoothed dark image from the light image. It has become.

  That is, for example, when the dark image is smoothed by excluding the late defective pixel DP, the random noise white floating RW and the random noise black sink RB included in the dark image are flattened almost smoothly. However, the fixed vertical stripe FP changes to a state where sharpness is lowered, and the shading distribution of the horizontal shading component HS also slightly changes due to smoothing.

  When such a smoothed dark image is subtracted from the bright image, some components remain in the horizontal shading component HS, and the original sharp vertical streak and the vertical streak with reduced sharpness also in the fixed vertical streak FP. And the difference will remain. Further, the late defective pixel DP remains by the level difference between the bright image and the dark image as in the first conventional technique. On the other hand, it is considered that the random noise can be maintained at the same level as the original bright image even if subtracted.

  Next, as a third conventional technique for correcting shading, a technique related to a noise removal method for an imaging apparatus described in, for example, Japanese Patent Application Laid-Open No. 2007-180616 will be described. The gazette includes an imaging device including a plurality of pixels, a median filter processing circuit that smoothes a dark image signal acquired in a state where the imaging device is shielded from light, and a subject image signal obtained by imaging a subject with the imaging device. The subtraction processing circuit that subtracts the dark image signal after smoothing and the subtraction processing circuit subtracts the dark image signal after smoothing from the black pixel signal before smoothing for each pixel, and the difference obtained is preset. A system control circuit that extracts pixels that are out of the range as defective pixels of the image sensor, and an image signal of the defective pixel extracted by the system control circuit in the subject image signal after subtracting the dark image signal, An imaging apparatus is described that includes an image processing circuit that performs correction using an image signal of peripheral pixels of a defective pixel.

  The third prior art further subtracts the smoothed dark image signal from the dark image signal before smoothing, and maintains a level above a certain level even after the subtraction, compared to the second prior art described above. The detected pixel is detected as a defective pixel, and the defective pixel is corrected. Therefore, it is considered that the subsequent defective pixel DP and the fixed vertical stripe FP shown in FIG. 11 can be removed better than the second conventional technique.

JP 2007-180616 A

  As described above, in the first conventional technique, shading is corrected well, but random noise may increase. In the second and third conventional techniques, increase in random noise can be suppressed. Shading correction remains. Thus, the conventional technology cannot achieve both good correction of shading and suppression of increase in random noise.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an image processing method and an image processing apparatus capable of achieving both good correction of shading and suppression of an increase in random noise.

  In order to achieve the above object, an image processing method according to an aspect of the present invention is an image processing method for removing noise contained in an image signal, and detects a dark image shading component from a dummy image signal related to a dark image. A dark image shading detection step, a dark image shading correction step for correcting shading of the dark image signal using the detected dark image shading component, and a dark image for smoothing the dark image signal whose shading has been corrected A smoothing step, a bright image shading detection step for detecting a bright image shading component from a dummy image signal relating to a bright image, and correction of shading of the bright image signal using the detected bright image shading component Brightness image shading correction step and shading correction From the bright-field image signal, having a brightness subtraction step of subtracting the image signal when the dark which is the smoothed.

  An image processing apparatus according to another aspect of the present invention provides a dark image shading detection that detects a dark image shading component from a dummy image signal related to a dark image in an image processing apparatus that removes noise included in the image signal. A dark image shading correction unit that corrects shading of the dark image signal using the detected dark image shading component, and a dark image smoothing unit that smoothes the dark image signal whose shading is corrected A bright-time image shading detection unit for detecting a bright-time image shading component from a dummy image signal related to a bright-time image, and a bright-time image shading correction for correcting shading of a bright-time image signal using the detected bright-time image shading component And the smoothed dark image signal from the bright image signal with the shading corrected. And a brightness subtraction unit configured to subtract.

  According to the image processing method and the image processing apparatus of the present invention, it is possible to achieve both good correction of shading and suppression of increase in random noise.

In Embodiment 1 of this invention, the block diagram which shows the structure of the imaging device to which the image processing apparatus was applied. FIG. 6 is a diagram illustrating a state of image change when noise removal mode is set and noise removal shooting is performed in the first embodiment. 5 is a flowchart showing processing at the time of shooting by the imaging apparatus in the first embodiment. 6 is a flowchart illustrating noise removal shooting processing in the first embodiment. 5 is a flowchart showing a bright-time image capture / correction process in the first embodiment. 5 is a flowchart showing dark image capturing / correction processing in the first embodiment. 5 is a flowchart showing horizontal shading extraction processing in the first embodiment. FIG. 4 is a diagram illustrating an example in which pixel data is read from a memory in units of two-dimensional blocks and a two-dimensional smoothing filter is applied in the first embodiment. FIG. 4 is a diagram illustrating an example in which pixel data is read from a memory in units of a plurality of continuous pixels in one line and a one-dimensional smoothing filter is applied in the first embodiment. The figure for demonstrating the 1st prior art for correct | amending shading in the said Embodiment 1. FIG. The figure for demonstrating the 2nd prior art for correct | amending shading in the said Embodiment 1. FIG.

Embodiments of the present invention will be described below with reference to the drawings.
[Embodiment 1]

  FIGS. 1 to 9 show Embodiment 1 of the present invention, and FIG. 1 is a block diagram showing a configuration of an imaging apparatus to which an image processing apparatus is applied. Here, in the present embodiment, an imaging apparatus such as a digital camera having a function as an image processing apparatus will be described as an example.

  An imaging apparatus shown in FIG. 1 includes a lens 1, an imaging device 2, a synchronization signal generation unit 3, a drive unit 4, a temperature detection unit 5, a signal processing circuit 6, an image processing unit 8, and a recording unit 10. A display unit 11, an instruction unit 12, a memory 13, and a system control unit 14.

  The lens 1 is a photographing optical system for forming an optical subject image on the image pickup surface of the image pickup device 2, and includes a diaphragm, a focus lens, and the like.

  The imaging device 2 includes an imaging surface on which a plurality of pixels are arranged, and generates an electrical image signal by photoelectrically converting an optical subject image formed by the lens 1. A specific example of the image sensor 2 is a CMOS image sensor, but it is not limited to this.

  It is assumed that a mechanical shutter (not shown) is provided between the lens 1 and the image sensor 2 so that the light shielding / passing of light from the lens 1 toward the image sensor 2 can be controlled. However, this mechanical shutter is not limited to having both the front curtain and the rear curtain, and the front curtain may be an electronic shutter and only the rear curtain may be a mechanical shutter. Therefore, in the following, a case where the front curtain is an electronic shutter and the rear curtain is a mechanical shutter will be described as an example.

  The synchronization signal generation unit 3 generates a vertical synchronization signal under the control of the system control unit 14.

  The drive unit 4 drives the image sensor 2 by generating a read start pulse and an electronic shutter start pulse based on the vertical synchronization signal generated by the synchronization signal generation unit 3 under the control of the system control unit 14. is there.

  The temperature detector 5 is provided in the vicinity of the image sensor 2 and substantially measures the temperature of the image sensor 2 that affects the noise level of the captured image.

  The signal processing circuit 6 includes a gain adjustment circuit 7 that amplifies the analog image signal from the image sensor 2, and after performing various analog signal processing, converts the analog image signal into a digital image signal and outputs the digital image signal. is there. The image signal output from the signal processing circuit 6 is stored in the memory 13.

  The image processing unit 8 performs various types of image processing on the image signal output from the signal processing circuit 6 and stored in the memory 13, and includes a defect correction unit 9. The defect correction unit 9 will be described later along with the flow of processing along the flowchart.

  The recording unit 10 stores the image data processed for recording by the image processing unit 8 in a nonvolatile manner, and is configured as a removable memory that can be carried out of the imaging apparatus such as a memory card. Therefore, the recording unit 10 may not have a configuration unique to the imaging apparatus.

  The display unit 11 is a display device that displays image data processed for display by the image processing unit 8 and a menu related to the operation of the imaging apparatus.

  The instructing unit 12 is a user interface for performing an operation input to the imaging apparatus, and includes a power button for instructing power on / off, a release button for instructing start of photographing, and on / off of a noise removal mode. Shooting mode setting button for setting auto / manual when noise removal mode is on, shutter speed indicating exposure time when taking an image, shooting sensitivity related to image gain (gain), etc. It includes a shooting condition setting button and various other setting buttons.

  The system control unit 14 determines the entire imaging apparatus based on an instruction from the user input from the instruction unit 12, information on the captured subject obtained from the image processing unit 8 (for example, AF information or AE information), and the like. The control is performed. For example, when the automatic exposure control is being performed, the system control unit 14 determines a shutter speed for capturing an image based on the AE information, and the image sensor 2 via the synchronization signal generation unit 3 and the drive unit 4. Control and drive. Further, when the automatic focus adjustment control is performed, the system control unit 14 performs the focus adjustment by controlling and driving the lens 1 via a lens driving unit (not shown) based on the AF information.

  Further, the system control unit 14 includes a dark image shading correction unit, a light image shading correction unit, a subtraction processing circuit 15 serving as a light / dark subtraction unit, a smoothing processing circuit 16 serving as a dark image smoothing unit, and a dark image. It includes a shading detection unit and a horizontal shading extraction circuit 17 that is a bright-time image shading detection unit, and also serves as a defective pixel detection unit. These will be described later together with the flow of processing along the flowchart.

  Next, FIG. 2 is a diagram showing how the image changes when the noise removal mode is set and noise-removing shooting is performed, FIG. 3 is a flowchart showing processing at the time of shooting by the imaging apparatus, and FIG. FIG. 5 is a flowchart showing a light image capture / correction process, FIG. 6 is a flowchart showing a dark image capture / correction process, and FIG. 7 is a flowchart showing a horizontal shading extraction process.

  First, with reference to FIG. 3, the flow of processing at the time of shooting by the imaging apparatus will be described.

  When the power is turned on by the power button of the instruction unit 12, this process is started, and a through image that displays an image captured by the image sensor 2 on the display unit 11 in near real time based on the control of the system control unit 14. Display is performed (step S1).

  Then, the system control unit 14 waits for the release button of the instruction unit 12 to be turned on (step S2).

  When the release button is turned on, the system control unit 14 determines whether or not the noise removal mode is off (step S3), and when it is determined that the release button is off, imaging in the normal shooting mode is performed (step S4). ).

  When it is determined in step S3 that the noise removal mode is not off, the system control unit 14 determines whether or not the noise removal mode is auto (step S5).

  Here, when it is determined that the noise removal mode is auto, the system control unit further determines whether the exposure time T set by automatic exposure control or manually is equal to or less than a predetermined threshold time W. 14 further determines (step S6). Here, the threshold time W may be a fixed value, or may be a value that can be changed according to user settings, shooting conditions, subject conditions, and the like.

  If it is determined in step S5 that the noise removal mode is not auto, or if it is determined in step S6 that the exposure time T is greater than the threshold time W, imaging in the noise removal shooting mode is performed (step S7).

  On the other hand, if it is determined in step S6 that the exposure time T is equal to or less than the threshold time W, the process proceeds to step S4, and imaging in the normal shooting mode is performed.

  Thus, when an image is captured by the process of step S4 or step S7, the image data processed for recording by the image processing unit 8 is recorded in the recording unit 10 (step S8), and is displayed by the image processing unit 8 for display. The processed image data is displayed on the display unit 11 (step S9).

  Thereafter, the system control unit 14 determines whether or not the power is turned off by the power button of the instruction unit 12 (step S10). When it is determined that the power is not turned off, the process returns to step S1 and the processing as described above. Is repeated, and if it is determined that the signal has been turned off, the process is terminated.

  Next, the flow of the noise-removal shooting process in step S7 in FIG. 3 will be described along FIG. 4 with reference to FIG.

  When this process is started, the exposure time T is longer than a predetermined threshold time X (condition 1), and the temperature C in the vicinity of the image sensor 2 detected by the temperature detector 5 is a predetermined threshold temperature. More than Y (condition 2), and a gain G indicating a signal amplification factor in the gain adjustment circuit 7 is larger than a predetermined threshold gain Z (threshold amplification factor) (condition 3). The system control unit 14 further determines whether one is satisfied (step S21). Here, the threshold time X, the threshold temperature Y, and the threshold gain Z may be fixed values or values that can be changed according to user settings, shooting conditions, subject conditions, and the like.

  Here, it is determined whether or not at least one of the three conditions is satisfied, but the present invention is not limited to this. For example, the system control unit 14 determines whether at least one of the two conditions, that is, the conditions 1 and 2, the conditions 2 and 3, and the conditions 3 and 1 is satisfied. May be. Alternatively, when any one of the conditions 1 to 3 is an essential condition, for example, when the condition 1 related to the exposure time T is an essential condition, the conditions 1 and 2, the conditions 1 and 3, The system control unit 14 may determine whether any one of the above holds. Similarly, the system control unit 14 may determine whether all the three conditions 1 to 3 are satisfied. Then, the success or failure may be determined by adding another condition to the conditions 1 to 3, or the success or failure may be determined by replacing any of the conditions 1 to 3 with another condition. Absent.

  If it is determined in step S21 that the condition is satisfied, 0 is assigned to the flag Flag (step S22), and if it is determined that the condition is not satisfied, 1 is assigned to the flag Flag ( Step S23).

  When the processing of step S22 or step S23 is completed, the mechanical shutter is opened, and then the system control unit 14 controls the synchronization signal generation unit 3 and the driving unit 4 to drive the image pickup device 2, thereby A time image (an image obtained by receiving subject light and performing photoelectric conversion) is captured (step S24).

  Then, a bright-time image capture / correction process as described later with reference to FIG. 5 is performed (refer to the image P1 in FIG. 2 for the captured image and the image P2 in FIG. 2 for the image subjected to the correction process). (Refer to step S25), and the processed image (see image P2 in FIG. 2) is temporarily stored in the memory 13 (step S26).

  Subsequently, after the mechanical shutter is closed, the system control unit 14 drives the image sensor 2 in the same manner as described above, and the dark image (the same exposure as the light image when the image sensor 2 is shielded from light). An image obtained by maintaining a state in which photoelectric conversion is possible only for time T is performed (step S27), and dark image capture / correction processing as described later with reference to FIG. 6 is performed ( Refer to the image P3 in FIG. 2 for the captured image, and refer to the images P4 and P5 in FIG. 2 for the corrected image (step S28), and store the processed image (see the image P5 in FIG. 2) in the memory 13. It is temporarily stored (step S29).

  Thereafter, the system control unit 14 reads the light image and the dark image temporarily stored in the memory 13 (steps S30 and S31), and the subtraction processing circuit 15 performs the light image (image P2 in FIG. 2). Then, a process for subtracting the dark image (image P5 in FIG. 2) is performed (brightness / darkness subtraction step) (refer to image P6 in FIG. 2 for the image obtained as a result of this process) (step S32).

  Then, based on the control of the system control unit 14, the defect correction unit 9 in the image processing unit 8 performs a process of correcting the defective pixel (refer to an image P7 in FIG. 2 for an image obtained as a result of this process) (step S33) When this process is completed, the process returns to the process shown in FIG.

  Next, the flow of the bright-time image capturing / correcting process in step S25 of FIG. 4 will be described along FIG. 5 with reference to FIG.

  When this process is started, first, a dummy read out from the image sensor 2 is performed to capture a bright dummy image (light dummy image acquisition step) (step S41).

  Then, it is determined whether or not the flag Flag set in step S22 or step S23 of FIG. 4 is 0 (step S42).

  If it is determined that the flag Flag is 0, the horizontal shading extraction circuit 17 in the system control unit 14 performs horizontal shading as described later with reference to FIG. By performing the extraction process, a horizontal shading component for the light image is extracted (light image shading detection step) (step S43).

  Further, a bright image obtained by exposure for the exposure time T is captured from the image sensor 2 (light image capturing step) (step S44). The bright image captured here is an image including a horizontal shading component HS, a subsequent defective pixel DP, a random noise white float RW, a random noise black sink RB, a fixed vertical stripe FP, and the like, as shown in an image P1 in FIG. is there.

  Therefore, the subtraction processing circuit 15 included in the system control unit 14 subtracts the horizontal shading component extracted in step S43 from the bright image P1, thereby correcting the horizontal shading component as shown in FIG. An image P2 is obtained (light image shading correction step) (step S45). Since the horizontal shading component extracted in step S43 includes a component due to the fixed vertical stripe FP, the bright image P2 is obtained by removing not only the horizontal shading component HS but also the fixed vertical stripe FP. ing.

  If it is determined in step S42 that the flag Flag is not 0, the same process as in step S44 is performed to capture a bright image obtained by exposure from the image sensor 2 for the exposure time T (light time). Image capturing step) (step S46). That is, when the condition of step S21 in FIG. 4 is not satisfied, the horizontal shading component HS and the fixed vertical stripe FP are negligible levels. Here, the horizontal shading extraction process and the process of subtracting the horizontal shading from the bright image are performed here. Is omitted.

  Then, when the process of step S45 or step S46 is completed, the process returns to the process shown in FIG.

  Next, the flow of dark image capturing / correction processing in step S28 of FIG. 4 will be described along FIG. 6 with reference to FIG.

  When this process is started, first, dummy reading is performed from the image sensor 2 to capture a dark dummy image (dark dummy image acquisition step) (step S51).

  Then, it is determined whether or not the flag Flag set in step S22 or step S23 of FIG. 4 is 0 (step S52).

  Here, when it is determined that the flag Flag is 0, the horizontal shading extraction circuit 17 in the system control unit 14 performs horizontal shading as described later with reference to FIG. 7 on the dark dummy image. By performing the extraction process, a horizontal shading component for the dark image is extracted (dark image shading detection step) (step S53).

  Further, a dark image obtained by exposure for an exposure time T in a light-shielded state is captured from the image sensor 2 (dark image capturing step) (step S54). The dark image captured here is an image including a horizontal shading component HS, a subsequent defective pixel DP, a random noise white float RW, a random noise black sink RB, a fixed vertical stripe FP, and the like, as shown in an image P3 in FIG. is there.

  Therefore, the subtraction processing circuit 15 included in the system control unit 14 subtracts the horizontal shading component extracted in step S53 from the dark image P3, thereby correcting the horizontal shading component as shown in FIG. An image P4 is obtained (dark image shading correction step) (step S55). Since the horizontal shading component extracted in step S53 includes a component due to the fixed vertical stripe FP, the dark image P4 is obtained by removing not only the horizontal shading component HS but also the fixed vertical stripe FP. ing.

  Next, a defective pixel registration process is performed (defective pixel detection step) (step S56). This defective pixel registration process is performed when the system control unit 14 compares the pixel value x of each pixel of the dark image P4 after shading correction with a predetermined defect determination value B and x ≧ B. The pixel is registered as a defective pixel, and when x <B, it is determined that the pixel is not a defective pixel, and processing for determining the next pixel is performed for all pixels. Further, information on the registered defective pixels is stored in the memory 13, for example.

  Thereafter, dark image smoothing processing is performed (dark image smoothing step) (step S57). In this dark image smoothing process, pixels other than defective pixels are sequentially set as the target pixel, and smoothing is performed on pixels located in the spatial vicinity of the target pixel (pixels of the same color in the case of a color imaging device). This is a process for obtaining the pixel value of the pixel of interest by applying a normalization filter. A specific example of this process will be described with reference to FIGS.

  First, FIG. 8 is a diagram illustrating an example in which pixel data is read from the memory 13 in units of two-dimensional blocks and a two-dimensional smoothing filter is applied.

  In the example shown in FIG. 8, a pixel having a pixel value “a”, a pixel having a pixel value “b”, a pixel having a pixel value “c”, and a pixel having a pixel value “d” in the vicinity of the target pixel having a pixel value “x”. (The case where the number of neighboring pixels is four is illustrated here, but of course not limited to this).

  As the smoothing filter, for example, (x + a + b + c + d) / 5 is used when an averaging filter is used, (2x + a + b + c + d) / 6 is used when a Gaussian filter is used, and { An example is given in which the median value in x, a, b, c, d} is the new pixel value of the pixel of interest.

  However, when a defective pixel exists in the neighboring pixels of the target pixel, the defective pixel is not used for the smoothing process.

  Next, FIG. 9 is a diagram illustrating an example in which pixel data is read from the memory 13 in units of a plurality of continuous pixels in one line and a one-dimensional smoothing filter is applied.

  In the example shown in FIG. 9, a case where a pixel having a pixel value “a” and a pixel having a pixel value “b” of the same color as the pixel of interest exists in the vicinity of the pixel of interest “x” on the same line is illustrated. (The case where the number of neighboring pixels is 2 is shown here, but the present invention is not limited to this.)

  As the smoothing filter, for example, (x + a + b) / 3 is used when an averaging filter is used, (2x + a + b) / 4 is used when a Gaussian filter is used, and { An example is given in which the median value in x, a, b} is the new pixel value of the pixel of interest.

  Even in the process described with reference to FIG. 9, when a defective pixel exists in the vicinity pixel of the target pixel, the defective pixel is not used for the smoothing process.

  Since the smoothing process shown in FIG. 9 only needs a line buffer, the memory capacity is smaller than the smoothing process shown in FIG. 8, and pixel values that are sequentially read out by the raster scan method are processed. There are advantages that are suitable for.

  Here, although an averaging filter, a Gaussian filter, and a median filter have been described as examples of the smoothing filter, the present invention is not limited to these.

  Returning to the description of FIG. 6, when it is determined in step S52 that the flag Flag is not 0, the dark image obtained by exposing for the exposure time T in the light-shielding state, similar to step S54, is obtained. 2 is performed (dark image capturing step) (step S58).

  Then, a defective pixel registration process is performed in the same manner as in step S56 (defective pixel detection step) (step S59). As described above, when the condition of step S21 in FIG. 4 is not satisfied, the horizontal shading component HS and the fixed vertical stripe FP are negligible levels. Therefore, here, the horizontal shading extraction process and the horizontal shading from the bright image are performed. The subtraction process is omitted. Also, since random noise is at a level that can be ignored, dark image smoothing processing is also omitted.

  Thus, when the process of step S57 or step S59 is completed, the process returns to the process shown in FIG.

  Next, the flow of horizontal shading extraction processing in step S43 in FIG. 5 and step S53 in FIG. 6 will be described with reference to FIG.

  When this processing is started, first, initialization is performed by substituting 1 into a variable i indicating the number of loops (step S61).

  Next, it is determined whether i is equal to or less than the number of columns Vn of pixels arranged on the dummy image (step S62).

  Here, if it is determined that i ≦ Vn, the pixel data of i columns are read (step S63), and IIR (infinite impulse response) addition processing is performed (step S64).

  Then, the variable i is incremented (step S65), the process returns to step S62, and the processing as described above is repeated.

  In this way, the variable i is incremented by 1 for each loop. When it is finally determined in step S62 that i> Vn, the result of the IIR addition calculated for all the columns is used as the horizontal shading component. (Step S66), the process returns to the process of FIG. 5 or FIG.

  Next, the processing of step S32 and step S33 of FIG. 4 will be further described with reference to FIG.

  By performing the processing up to step S29 in FIG. 4, the bright image stored in the memory 13 is an image as shown in P2 of FIG. 2, and the dark image is an image as shown in P5 of FIG. Yes. Therefore, in step S30 and step S31, these bright image P2 and dark image P5 are read, and in step S32, the subtraction processing circuit 15 in the system control unit 14 subtracts the dark image P5 from the bright image P2. To obtain an image as shown in P6 of FIG. With this process, the level of the subsequent defective pixel DP is reduced to the extent that the difference between the pixel value of the subsequent defective pixel DP in the bright image and the pixel value of the subsequent defective pixel DP in the dark image remains. Further, this process absorbs the OB step and further reduces the horizontal shading component that may still remain in step S45 of FIG. 5 or step S55 of FIG.

Here, the absorption of the OB step will be further described.
Since a solid-state image sensor generates a dark current, it is known that the obtained image has black level floating. Therefore, in many image pickup devices, an optical black (OB) region in which a light shielding curtain such as aluminum is formed is provided on a part of the image pickup surface, and the signal level obtained in the light-shielded OB region is determined from the image signal of the effective image pickup region. By subtracting, a process for correcting the black level is performed. However, since the configuration of the OB area is different from that of the effective imaging area, for example, parasitic capacitance is generated between the aluminum light-shielding curtain and the photodiode, so that there is a step (so-called “black level”) between the OB area and the effective imaging area. It is known that an OB step) occurs. Therefore, the black level floating cannot be completely corrected only by subtracting the black level obtained from the signal in the OB area from the image signal in the effective imaging area (that is, only performing OB clamping).

  Since there may be a difference in the temperature of the image sensor 2 between the time when the bright image is captured and the time when the dark image is captured, the black level of the bright image and the black level of the dark image generally match. However, it may be considered that the OB level difference in the bright image and the OB level difference in the dark image are substantially the same level, that is, the OB level remaining after the OB clamping is the bright image and the dark image. You can think of it as almost the same.

  Therefore, in the present embodiment, by subtracting the dark image signal of the effective imaging area from the bright image signal of the effective imaging area, the OB level difference remaining after the OB clamping is removed well, and an accurate black level is obtained. There are benefits that can be obtained.

  Subsequently, in step S33, the defect correction unit 9 in the image processing unit 8 performs a process of correcting the defective pixel registered in the process of step S56 or step S59 in FIG.

  This defective pixel correction process is similar to the smoothing process described with reference to FIGS. 8 and 9, except that only the defective pixel is the target pixel. Since the defective pixel is the target pixel, the pixel value x of the target pixel itself is not used to obtain the correction value of the defective pixel.

  In the pixel arrangement example of FIG. 8 described above, for example, when an averaging filter or a Gaussian filter is used for correction processing of a defective pixel, (a + b + c + d) / 4 may be set as a correction value of the defective pixel. For example, when a median filter is used, an average of two pixel values taking a median value in {a, b, c, d} may be used as a correction value for a defective pixel.

  In the pixel arrangement example of FIG. 9, (a + b) / 2 is used as the defective pixel correction value, regardless of whether the averaging filter, the Gaussian filter, or the median filter is used for the defective pixel correction process. It will be.

  In any of these cases, when another defective pixel exists in the vicinity of the defective pixel of interest, the other defective pixel is not used for the defective pixel correction process as described above. It is the same.

  In this way, if the rule that the pixel value of the defective pixel is not used is applied, both the smoothing and the defect correction filter processing can be shared, so that the configuration and processing can be simplified. However, the present invention is not limited to this, and the filter used for smoothing may be different from the filter used for defect correction.

  By performing such defective pixel correction processing, the subsequent defective pixel DP remaining in the image P6 after the processing in step S32 is eliminated, and an image as shown in P7 of FIG. 2 is obtained. When this image P7 is compared with the bright image P1 acquired from the image pickup device 2, the horizontal shading component HS, the subsequent defective pixel DP, the fixed vertical stripe FP, and the OB step are corrected, and the remaining is random noise white. It can be seen that there are only floating RW and random noise black sink RB. Moreover, these random noises RW and RB existed in the first bright image P1, and the random noise does not increase as in the prior art described with reference to FIG.

  In the above description, the imaging apparatus is taken as an example of an apparatus to which the image processing apparatus is applied, and the image processing method performed in the imaging apparatus has been described. However, the present invention is not limited to this, and the image processing apparatus is recorded on a recording medium. The image processing program may be a computer that reads and executes the image processing program, and the image processing method may be a method achieved by executing the image processing program by the computer. Therefore, as an aspect of the present invention, any of an image processing method, an image processing apparatus, an imaging apparatus, an image processing program, and a computer-readable recording medium that records the image processing program can be taken.

  According to the first embodiment, the light image shading component is detected from the dummy image signal related to the light image, and the shading of the light image signal is corrected using the detected light image shading component. Therefore, the shading of the bright image can be corrected well.

  Also, since the dark image shading component is detected from the dummy image signal related to the dark image and the shading of the dark image signal is corrected using the detected dark image shading component, the shading is corrected well. A dark image can be obtained.

  Furthermore, since the dark image signal whose shading has been corrected is smoothed, a dark image signal in which random noise is suppressed can be obtained.

  Since the smoothed dark image signal is subtracted from the bright image signal with the shading corrected, the OB step can be absorbed and the random noise of the bright image increases. Can be suppressed.

  In this way, it is possible to achieve both good correction of shading and suppression of an increase in random noise, and it is also possible to absorb OB steps.

  In addition, since the defective pixel is detected based on the dark image signal after the shading correction, the defective pixel can be accurately detected without being affected by the shading.

  Furthermore, since the pixel value of the detected defective pixel is interpolated from the pixel values of neighboring pixels, defect correction can be performed satisfactorily. At this time, since the dark image is smoothed by removing the signal of the detected defective pixel, the information on the defective pixel is not mixed into other pixels.

  The shading correction for dark images and the shading correction for bright images are performed when the exposure time is equal to or less than a predetermined threshold time, and the temperature around the image sensor when generating an image signal is equal to or lower than the predetermined threshold temperature. Since it is omitted when all the conditions that the rate is less than or equal to the predetermined threshold amplification factor are satisfied, when shading correction is not required, unnecessary processing is not performed, reducing processing load and power consumption. Can be planned.

  In addition, for dark images, the exposure time is less than a predetermined threshold time, the temperature around the image sensor when generating an image signal is less than a predetermined threshold temperature, and the amplification factor of the image signal is less than a predetermined threshold amplification factor When all the conditions are satisfied, the smoothing process is also omitted, so that the processing load can be further reduced and the power consumption can be further reduced.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, you may delete some components from all the components shown by embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined. Thus, it goes without saying that various modifications and applications are possible without departing from the spirit of the invention.

DESCRIPTION OF SYMBOLS 1 ... Lens 2 ... Image pick-up element 3 ... Synchronization signal generation part 4 ... Drive part 5 ... Temperature detection part 6 ... Signal processing circuit 7 ... Gain adjustment circuit 8 ... Image processing part 9 ... Defect correction part 10 ... Recording part 11 ... Display part 12 ... Instruction unit 13 ... Memory 14 ... System control unit (defective pixel detection unit)
15. Subtraction processing circuit (dark image shading correction unit, bright image shading correction unit, bright / dark subtraction unit)
16. Smoothing processing circuit (dark image smoothing unit)
17 ... Horizontal shading extraction circuit (dark image shading detection unit, bright image shading detection unit)
S32 ... Brightness / darkness subtraction step S43 ... Bright image shading detection step S45 ... Bright image shading correction step S53 ... Dark image shading detection step S55 ... Dark image shading correction step S56, S59 ... Defective pixel detection step S57 ... Dark image Smoothing step

Claims (4)

In an image processing method for removing noise contained in an image signal,
A dark image shading detection step for detecting a dark image shading component from a dummy image signal related to the dark image;
A dark image shading correction step for correcting shading of the dark image signal using the detected dark image shading component;
A dark image smoothing step for smoothing the dark image signal whose shading is corrected;
A light image shading detection step for detecting a light image shading component from a dummy image signal related to the light image;
A light image shading correction step for correcting the shading of the light image signal using the detected light image shading component;
A light / dark subtraction step of subtracting the smoothed dark image signal from the bright image signal with the shading corrected;
An image processing method comprising: A defective pixel detection step of detecting a defective pixel included in the dark image signal after the shading correction,
The image processing method according to claim 1, wherein the dark image smoothing step is a step of performing smoothing by removing a signal of a detected defective pixel.   In the dark image shading correction step and the bright image shading correction step, the exposure time is longer than a predetermined threshold time, and the temperature around the image sensor when generating the image signal is lower than the predetermined threshold temperature. 2. The image processing method according to claim 1, wherein the image processing method is performed only when at least one of a large value and an amplification factor of the image signal is larger than a predetermined threshold amplification factor is satisfied. In an image processing apparatus for removing noise included in an image signal,
A dark image shading detection unit for detecting a dark image shading component from a dummy image signal related to the dark image;
A dark image shading correction unit that corrects shading of the dark image signal using the detected dark image shading component;
A dark image smoothing unit that smoothes a dark image signal with shading corrected; and
A bright-time image shading detection unit for detecting a bright-time image shading component from a dummy image signal related to the bright-time image;
A light image shading correction unit that corrects shading of a light image signal using the detected light image shading component;
A light / dark subtraction unit that subtracts the smoothed dark image signal from a light image signal with shading corrected;
An image processing apparatus comprising:

Images