JP2006041935A - Device and program for processing image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing device and an image processing program capable of appropriately correcting a local dark current component caused by the heat generator of an image pickup device. <P>SOLUTION: The image processing device comprises means 11-13 for estimating dark current distribution (D<SB>(m, n)</SB>) in the effective pixel region of the image pickup device, based on a plurality of first pixel signals outputted near the generator (for example, the output section of FDA, or the like) of at least the image pickup device in the light-shielding pixel region in the image pickup device 20; and a correction means 14 for correcting a plurality of second pixel signals (S<SB>(m, n)</SB>) outputted from the effective pixel region, based on the dark current distribution (D<SB>(m, n)</SB>). <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and an image processing program that perform correction processing of a pixel signal output from an effective pixel region of an image sensor.

As is well known, a dark current component is superimposed on the pixel signal output from the effective pixel region of the image sensor. For this reason, a correction process of subtracting the dark current component from the pixel signal is necessary. The following is known as a conventional correction process (see, for example, Patent Document 1). That is, a correction value is stored in advance for each pixel in the effective pixel region, the correction value is changed according to the operation condition (for example, imaging time) of the image sensor, and the corrected correction value is subtracted from the pixel signal. This is a correction process. As a coefficient for changing the correction value, a common value is used for each pixel in the effective pixel region.
Japanese Patent Laid-Open No. 7-236093

  However, inside the image sensor, there is a heat generating part such as an output amplifier (for example, FDA; Floating Diffusion Amplifier), and the distribution state of the local dark current component caused by the heat generating part depends on the operating condition of the image sensor (for example, It will change greatly according to the imaging time. In other words, the local dark current component is inconspicuous because the amount of heat generation is small under conditions where the imaging time is short, but local conditions are necessary under conditions where the amount of heat generation increases, such as long exposure (for example, imaging time of 30 seconds or more). The dark current component becomes prominent.

Therefore, as in the conventional correction method described above, even if the correction value of each pixel in the effective pixel region is changed by a common value according to the operating condition (for example, imaging time) of the image sensor, it is caused by the heat generating part. The local dark current component could not be corrected appropriately.
An object of the present invention is to provide an image processing apparatus and an image processing program capable of appropriately correcting a local dark current component caused by a heat generating portion of an image sensor.

  The image processing apparatus according to claim 1, in the effective pixel region of the image sensor, based on a plurality of first pixel signals output from at least the vicinity of the heat generating portion of the image sensor in the light-shielded pixel region of the image sensor. An estimation unit that estimates a dark current distribution; and a correction unit that corrects a plurality of second pixel signals output from the effective pixel region based on the dark current distribution estimated by the estimation unit. .

According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the estimating unit selects an average value by selecting a pixel signal having a small influence of the heat generating portion from the plurality of first pixel signals. The average value is subtracted from each of the plurality of first pixel signals, and the dark current distribution is estimated based on the plurality of first pixel signals after the subtraction.
According to a third aspect of the present invention, in the image processing apparatus according to the second aspect, the estimating means smoothes the plurality of first pixel signals by excluding protruding points before obtaining the average value. Neighbor calculation is performed.

According to a fourth aspect of the present invention, in the image processing apparatus according to the first aspect, the estimation unit estimates the dark current distribution in consideration of a gain depending on coordinates in the effective pixel region. is there.
According to a fifth aspect of the present invention, in the image processing apparatus according to the first aspect, the estimation unit includes the gain depending on the coordinates in the effective pixel region and the heat conduction characteristics of the imaging element, and This is to estimate the dark current distribution.

  The image processing device according to claim 6 is based on a plurality of first pixel signals output from each of a horizontal region and a vertical region located in the vicinity of the heat generating portion of the image sensor in the light-shielded pixel region of the image sensor. And estimating means for estimating the dark current distribution in the effective pixel area of the image sensor, and correcting the plurality of second pixel signals output from the effective pixel area based on the dark current distribution estimated by the estimating means. And a correcting means for performing.

  According to a seventh aspect of the present invention, in the image processing apparatus according to the sixth aspect, the estimating means is a dark current in the horizontal direction of the effective pixel region based on the first pixel signal output from the horizontal region. After obtaining the characteristics, a dark current characteristic in the vertical direction of the effective pixel area is obtained based on the first pixel signal output from the vertical area, and the dark current distribution is obtained by multiplying the two dark current characteristics. To be estimated.

  The image processing program according to claim 8, in the effective pixel region of the image sensor, based on a plurality of first pixel signals output from at least the vicinity of the heat generating portion of the image sensor in the light-shielding pixel region of the image sensor. To cause a computer to execute an estimation procedure for estimating a dark current distribution and a correction procedure for correcting a plurality of second pixel signals output from the effective pixel region based on the dark current distribution estimated by the estimation means. belongs to.

  The image processing program according to claim 9 is based on a plurality of first pixel signals output from each of a horizontal region and a vertical region located in the vicinity of the heat generating portion of the image sensor in the light-shielded pixel region of the image sensor. Then, an estimation procedure for estimating the dark current distribution in the effective pixel region of the image sensor and a plurality of second pixel signals output from the effective pixel region are corrected based on the dark current distribution estimated by the estimation unit. And a correction procedure to be executed by the computer.

  According to the image processing apparatus and the image processing program of the present invention, it is possible to appropriately correct the local dark current component caused by the heat generating portion of the image sensor.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
(First embodiment)
Here, an example will be described in which the image processing apparatus 10 (FIG. 1) of the first embodiment is dedicated hardware (LSI) arranged at the subsequent stage of the image sensor 20. In this case, the image processing apparatus 10 according to the present embodiment is incorporated in an imaging apparatus such as a digital camera together with the imaging element 20.

  The image sensor 20 is, for example, a CCD image sensor. As shown in FIG. 2, the imaging device 20 is provided with an effective pixel region 21, a light-shielding pixel region 22, and an output unit 23. The effective pixel area 21 is an area for accumulating signal charges corresponding to the subject image. The light shielding pixel region 22 is disposed adjacent to the periphery of the effective pixel region 21, and the surface thereof is covered with a light shielding film. The output unit 23 is arranged at the upper left in the drawing outside the light-shielding pixel region 22 and includes a plurality of output amplifiers (for example, FDA) (not shown).

  In the image sensor 20, the charges accumulated in each pixel of the effective pixel region 21 and the light-shielded pixel region 22 are guided to the output unit 23 through a vertical transfer path and a horizontal transfer path (not shown). The output amplifier of the output unit 23 is in an operating state by applying a bias current, converts the input charge from the horizontal transfer path into a voltage signal, and outputs it as a pixel signal to the outside. The pixel signal output from the image sensor 20 is guided to the image processing apparatus 10 (FIG. 1) of the present embodiment after passing through an AD converter (not shown) or the like.

The order of output from the image sensor 20 starts from the pixel P _{S at} the upper left in FIG. 2, outputs along the horizontal line, outputs while shifting the line in the vertical direction, and at the lower right pixel _PE . The order is over. That is, first, pixel signals are output from the respective lines at the top of the light-shielding pixel region 22 in the figure, and then the left part of the light-shielding pixel region 22, the effective pixel region 21, and the right part of the light-shielding pixel region 22. A pixel signal is output from each of the included lines, and finally, a pixel signal is output from each line below the light-shielding pixel region 22.

  The pixel signal output from the effective pixel area 21 includes a signal component corresponding to the subject image and also includes a dark current component. In order to subtract the dark current component from the pixel signal in the effective pixel region 21, the image processing apparatus 10 (FIG. 1) of the present embodiment uses the pixel signal output from the light-shielded pixel region 22. The pixel signal output from the light-shielded pixel region 22 is composed of only a dark current component and corresponds to the black level of the subject image. For this reason, the light-shielding pixel region 22 is generally called an optical black region.

  By the way, when the image pickup device 20 is in an operating state, a bias current is always applied to the output amplifier of the output unit 23 of the image pickup device 20, so that the output unit 23 generates heat due to the bias current. The amount of generated heat varies depending on the operating conditions of the image sensor 20, and for example increases as the imaging time increases. Further, the heat generated by the output unit 23 spreads radially. Then, a local dark current component resulting from the heat generation of the output unit 23 is superimposed on a part of pixel signals in the effective pixel region 21 and the light-shielding pixel region 22.

  In the present embodiment, since the output unit 23 is arranged at the upper left in the figure, as shown in FIG. 3, a local dark current is applied to the pixel signal of the partial area 24 positioned at the upper left of the effective pixel area 21 and the light-shielded pixel area 22. The components are superimposed. The local dark current component (in the partial region 24) is not noticeable because the amount of heat generated by the output unit 23 is small under conditions where the imaging time is short, but like long exposure (for example, imaging time of 30 seconds or more). This will become prominent under conditions where the amount of heat generation increases. The output unit 23 corresponds to a “heat generating unit” in the claims.

  The image processing apparatus 10 according to the present embodiment appropriately corrects a local dark current component (in the partial region 24) caused by a heat generating part of the image sensor 20 (that is, heat generated by the output part 23). As shown, the signal shaping units 11 and 12, the calculation unit 13, the correction unit 14, and the result output unit 15 are configured. Among these, the signal shaping sections 11 and 12 and the calculation section 13 correspond to “estimating means” in the claims. The correction unit 14 corresponds to “correction means”.

  In the signal shaping unit 11, pixel signals output from the horizontal region 25 (FIG. 4) located in the vicinity of the output unit 23 (heat generation unit) in the light-shielding pixel region 22 of the image sensor 20 are sequentially captured. The number of pixels in the horizontal direction of the horizontal region 25 is (W + A), where W is the number of pixels in the horizontal direction of the effective pixel region 21 and A is the number of pixels in the left direction of the light-shielding pixel region 22. . The number of pixels in the vertical direction of the horizontal region 25 (that is, the number of lines) is B. For example, A, B = 4.

On the other hand, the signal shaping unit 12 sequentially receives pixel signals output from the vertical region 26 (FIG. 5) located in the vicinity of the output unit 23 (heat generation unit) of the image sensor 20. The number of pixels in the vertical direction of the vertical region 26 is (B + H), where the number of pixels in the vertical direction of the effective pixel region 21 is H. The number of pixels in the horizontal direction of the vertical region 26 is A.
In the image processing apparatus 10 according to the present embodiment, the pixel signal after passing through the signal shaping units 11 and 12 is input to the calculation unit 13 to estimate the dark current distribution in the effective pixel region 21, and this dark current distribution. The correction value D _{(m, n)} (m = 1,..., W) (n = 1,..., H) for each pixel corresponding to is output to the correction unit 14 at the subsequent stage. In addition to the correction value D _{(m, n)} output from the calculation unit 13, the correction unit 14 outputs a pixel signal S _{(m, n)} (m = 1,..., Output from the effective pixel region 21 of the image sensor 20 ₎ . W) (n = 1,..., H) are taken in, and the pixel signal S _{(m, n)} is corrected based on the correction value D _{(m, n)} . Hereinafter, each part of the image processing apparatus 10 will be described.

Specific processing in the signal shaping unit 11 will be described. When the signal shaping unit 11 takes in the pixel signal output from the horizontal region 25 (FIG. 4) of the image sensor 20, the signal shaping unit 11 divides the horizontal region 25 into (W + 1) blocks shown in FIG. Average values T ₀ , T ₁ , T ₂ ,..., T _W−2 , T _W−1 , T _W are obtained. Of the (W + 1) blocks, the leftmost block has A pixels in the horizontal direction, and the remaining blocks have one pixel in the horizontal direction.

When the signal shaping unit 11 obtains the average values T ₀ ,..., T _W for each block, the signal shaping unit 11 performs processing according to the operation flowchart of FIG. In the following description, the average value T _0, ..., a T _W, the pixel signals T _0, ..., called T _W. The pixel signals T ₀ ,..., T _W are graphed as shown in FIG. Number m (= 0,1, ..., W -1, W) of horizontal axis pixel signal T _m of a FIGS. 7 (a) represents the vertical axis represents the magnitude of the pixel signal T _m.

7 (a) is an example of the output amplifier 6-stage configuration of the output unit 23 of the image pickup device 20, the pixel signals T _0, ..., of the T _W, is output from the vicinity of the output unit 23 (heat generating portion) Six peaks appear in the pixel signal. These peaks correspond to (local) dark current components in the partial region 24 shown in FIG. In addition, other pixel signals (pixel signals away from the vicinity of the output unit 23) are substantially stable in size because the influence of the output unit 23 is small.

The signal shaping unit 11 performs steps S1 to S9 in FIG. 6 using such pixel signals T ₀ ,..., T _W. That is, the pixel signal T _0, ..., perform neighborhood operations of smoothing to eliminate the salient points of the T _W. Specifically, first (step S2), and focused on one pixel signals T _m, the pixel signal T _m-2 of the two by two _{longitudinal, T m-1, T m} + 1, T m + 2 In addition, median processing is performed using five adjacent pixel signals T _m−2 , T _m−1 , T _m , T _{m + 1} , and T _{m + 2} , and the median pixel signal T _MEDI is selected.

Next (Step S3, S5), attention to the pixel signal T _m-1 of the previous one than the pixel signal T _m is, obtains the difference between the pixel signal T _MEDI median described above, the difference ({T _MEDI− T _m−1 } or {T _m−1 −T _MEDI }) is compared with a predetermined set value C ₁ .
When the relationship between steps S3 and S5 is not satisfied, that is, when the following equation (1) is satisfied, the median pixel signal T _MEDI is ± C _{1 with} respect to the previous pixel signal T _m−1 . Therefore, it is overwritten by replacing the pixel signal T _MEDI with the current pixel signal T _m of interest (step S7, T _m = T _MEDI ).

| T _MEDI −T _m−1 | ≦ C ₁ (1)
On the other hand, when the relationship of the step S3 (the following equation (2)) is satisfied, the median pixel signal T _MEDI is too large (protrusion point) with respect to the previous pixel signal T _m−1 . The value obtained by adding the set value C ₁ to the previous pixel signal T _m−1 without using the median pixel signal T _MEDI is replaced with the current pixel signal T _m of interest. overwrite (step _{S4, T m = T m-} 1 + C 1).

T _MEDI −T _m−1 > C ₁ (2)
When the relationship of step S5 (the following equation (3)) is satisfied, it is determined that the median pixel signal T _MEDI is too small (a protruding point) with respect to the previous pixel signal T _m−1 . Thus, without using the median pixel signal T _MEDI , the value obtained by subtracting the set value C ₁ from the previous pixel signal T _m-1 is replaced with the current pixel signal T _m to be overwritten. (step _{S6, T m = T m-} 1 -C 1).

T _m-1 −T _MEDI > C ₁ (3)
By repeatedly performing the neighborhood calculation (steps S1 to S9) as described above, the protruding points of the pixel signals T ₀ ,..., T _W shown in FIG. The pixel signals T ₀ ,..., T _W that are the results of the neighborhood calculation (steps S1 to S9) all satisfy the above equation (1) and are graphed as shown in FIG. From the comparison between FIGS. 7A and 7C, it can be seen that the protruding points can be eliminated and smoothed without breaking the peak waveforms of the pixel signals T ₀ ,..., T _W. In addition, when only the process of step S2 in FIG. 6 is performed (steps S3 to S7 are omitted), as can be seen from FIG. I can not say.

Next (step S10), an image signal in a stable region where the influence of the output unit 23 (heat generation unit) is small is selected from the smoothed pixel signals T ₀ ,..., T _W (FIG. 7C). an average value T _a. The stable region may be, for example, a predetermined number of pixels from the larger number m (= 0, 1,..., W−1, W) of the pixel signal T _m . Further (step S11), the largest value among the smoothed pixel signals T ₀ ,..., T _W (FIG. 7C) is obtained as the representative value T _B.

Then, in the next steps S12 to S15, the smoothed pixel signals T ₀ ,..., T _W are normalized using the average value T _A and the representative value T _B. That is, as shown in the following equation (4), the pixel signal T ₀ after the smoothing, ..., while subtracting the average value T _A from each of the T _W, by subtracting the average value T _A from the representative value T _B, Each pixel signal (T _m −T _A ) after subtraction is divided by the representative value (T _B −T _A ) after subtraction, and the result is replaced with the pixel signal T _m of interest and overwritten.

_{T m = (T m -T A} ) / (T B -T A) ... (4)
FIG. 8 is a graph of the pixel signals T ₀ ,..., T _W obtained as a result of the normalization process (steps S10 to S15). The pixel signals T ₀ ,..., T _W after normalization represent local dark current characteristics in the horizontal direction of the effective pixel region 21 of the image sensor 20, and the signal shaping unit 11 to the calculation unit 13 in the subsequent stage (FIG. 1). And stored in a memory (eg, SDRAM) of the arithmetic unit 13. The average value T _A and the representative value T _B obtained in steps S10 and S11 are output to the signal shaping unit 12 described below and the calculation unit 13 described later.

Specific processing in the signal shaping unit 12 will be described. When the signal shaping unit 12 takes in the pixel signal output from the vertical region 26 (FIG. 5) of the image sensor 20, the signal shaping unit 12 divides the vertical region 26 into (H + 1) blocks shown in FIG. Average values L ₀ , L ₁ , L ₂ ,..., L _H-2 , L _H−1 , L _H are obtained. Of the (H + 1) blocks, the uppermost block has B pixels (number of lines) in the vertical direction, and the remaining blocks have one line.

When the signal shaping unit 12 obtains the average values L ₀ ,..., L _H for each block, the signal shaping unit 12 performs processing according to the operation flowchart of FIG. In the following description, the mean value L _0, ..., a L _H, the pixel signals L _0, ..., called L _H. Processing in step S21~S29 in FIG. 9, the pixel signals L _0, ..., corresponds to neighborhood operations to smooth and eliminate the salient points of the L _H. Subsequent processing in steps S30 to S34 corresponds to normalization processing of the smoothed pixel signals L ₀ ,..., L _H.

Specifically, first (step S21), and focused on one pixel signal L _n, the pixel signal L _n-2 of the two by two _{longitudinal, L n-1, L n} + 1, L n + 2 In addition, median processing is performed using five adjacent pixel signals L _n−2 , L _n−1 , L _n , L _{n + 1} , and L _{n + 2} to select a median pixel signal L _MEDI . Then (step S23, S25), attention to the pixel signal L _n-1 of the previous one than the pixel signal L _n are, obtains the difference between the pixel signal L _MEDI median described above, the difference ({L _MEDI− L _n−1 } or {L _n−1 −L _MEDI }) is compared with a predetermined set value C ₂ .

When the relationship between steps S23 and S25 is not satisfied, that is, when the following equation (5) is satisfied, the median pixel signal L _MEDI is ± C _{2 with} respect to the previous pixel signal L _n−1 . Since the pixel signal L _MEDI is included in (not a protruding point), the pixel signal L _MEDI is overwritten by being replaced with the current pixel signal L _n of interest (step S27, L _n = L _MEDI ).
| L _MEDI− L _n−1 | ≦ C ₂ (5)
On the other hand, when the relationship of step S23 (the following equation (6)) is satisfied, the median pixel signal L _MEDI is too large (protrusion point) with respect to the previous pixel signal L _n−1 . The value obtained by adding the set value C ₂ to the previous pixel signal L _n−1 without using the median pixel signal L _MEDI is replaced with the current pixel signal L _n of interest. overwrite (step _{S24, L n = L n-} 1 + C 2).

L _MEDI− L _n−1 > C ₂ (6)
Further, when the relationship of the step S25 (the following equation (7)) is satisfied, it is determined that the median pixel signal L _MEDI is too small (a protruding point) with respect to the previous pixel signal L _n−1 . Te, without using the pixel signals L _MEDI median, a value obtained by subtracting the set value C ₂ from the previous pixel signal L _n-1, overwriting by replacing the pixel signal T _n which is current interest (step _{S26, L n = L n-} 1 -C 2).

L _n-1 −L _MEDI > C ₂ (7)
By repeatedly performing the neighborhood calculation as described above (steps S21 to S29), the protruding points of the pixel signals L ₀ ,..., L _H can be eliminated and smoothed. The pixel signals L ₀ ,..., L _H that are the results of the neighborhood calculation (steps S21 to S29) all satisfy the above equation (5) and are graphed as shown in FIG. Also in this case, it is possible to smooth out the protruding points by eliminating the peak waveforms of the pixel signals L ₀ ,..., L _H.

Next (step S30), the average value T _A and the representative value T _B (FIG. 7C) obtained by the signal shaping unit 11 are acquired. Average T _A are those obtained from the image signal of the impact is small stable region of the output unit 23 of the image pickup device 20 (heat generating portion). The representative value T _B is the largest value among the pixel signals T ₀ ,..., T _W after smoothing in the horizontal direction (FIG. 7C).
Then, in the next steps S31 to S34, the smoothed pixel signals L ₀ ,..., L _H (FIG. 10) in the vertical direction are normalized using the average value T _A and the representative value T _B. That is, as shown in the following equation (8), the pixel signal L ₀ after the smoothing, ..., while subtracting the average value T _A from each of L _H, subtracts the average value T _A from the representative value T _B, Each pixel signal (L _n −T _A ) after subtraction is divided by the representative value (T _B −T _A ) after subtraction, and the result is replaced with the pixel signal L _n of interest and overwritten.

L _n = (L _n −T _A ) / (T _B −T _A ) (8)
FIG. 11 shows a graph of the pixel signals L ₀ ,..., L _H obtained as a result of the normalization process (steps S30 to S34). The normalized pixel signals L ₀ ,..., L _H represent local dark current characteristics in the vertical direction of the effective pixel region 21 of the image sensor 20, and the signal shaping unit 12 to the subsequent calculation unit 13 (FIG. 1). And is stored in the memory (for example, SDRAM) of the arithmetic unit 13 together with the above-described local dark current characteristics in the horizontal direction (normalized pixel signals T ₁ ,..., T _W ).

Next, specific processing in the calculation unit 13 will be described. The calculation unit 13 includes the horizontal pixel signals T ₁ ,..., T _W (FIG. 8) after passing through the signal shaping unit 11, and the average value T _A and the representative value T obtained in steps S10 and S11 of FIG. _{After B} is inputted, the pixel signals L ₁ ,..., L _H (FIG. 11) in the vertical direction after passing through the signal shaping unit 12 are inputted. The waveforms of the pixel signals T ₁ ,..., T _W (FIG. 8) represent local dark current characteristics in the horizontal direction of the effective pixel region 21 of the image sensor 20. Further, the waveforms of the pixel signals L ₁ ,..., L _H (FIG. 11) represent local dark current characteristics in the vertical direction of the effective pixel region 21.

Each time the arithmetic unit 13 captures the vertical image signal L _n (n = 1, 2,..., H) from the signal shaping unit 12, the horizontal pixel signal T _m (m = 1, 2,..., W), the average value T _A, and the representative value T _B , multiplication is performed in order according to the following equation (9). This multiplication is a process of multiplying the local dark current characteristic (FIG. 8) in the horizontal direction of the effective pixel region 21 of the image sensor 20 and the local dark current characteristic (FIG. 11) in the vertical direction for each pixel. It is.

D _{(m, n)} = (T _B −T _A ) × T _m × L _n (9)
As a result, a matrix array calculation result D _{(m, n)} shown in FIG. 12A can be obtained. Further, when the calculation result D _{(m, n)} in FIG. 12A is represented by a three-dimensional graph, for example, it is as shown in FIG. “M (= 1,..., W)” of the calculation result D _{(m, n)} corresponds to the horizontal coordinate in the effective pixel region 21 and “n (= 1,..., H)” is an effective pixel. This corresponds to the vertical coordinate in the region 21. The calculation result D _{(m, n)} represents a local dark current distribution in the effective pixel region 21. As described above, the local dark current distribution in the effective pixel region 21 can be estimated by multiplying the local dark current characteristics in the horizontal direction and the vertical direction (FIGS. 8 and 11).

Then, the calculation result D _{(m, n)} of FIGS. 12A and 12B corresponding to the local dark current distribution is a correction value D _{(m, n)} (for each pixel in the effective pixel region 21. m = 1,..., W) (n = 1,..., H) are output to the subsequent correction unit 14 (FIG. 1). In addition to the correction value D _{(m, n)} for each pixel, the pixel signal S _{(m, n)} from the effective pixel region 21 of the image sensor 20 is input to the correction unit 14. As shown in FIG. 13A, the pixel signal S _{(m, n)} also has the same matrix arrangement as the correction value D _{(m, n)} . The input timing is roughly the pixel signal S _{(m, n)} from the effective pixel region 21 when the correction value D _{(m, n)} is equal to the processing time in the signal shaping units 11 and 12 and the calculation unit 13. Slightly slower than.

For this reason, the correction unit 14 waits for a while until the correction value D _{(m, n) of} the corresponding line is output from the calculation unit 13 after capturing the pixel signal S _{(m, n)} of a certain line. When the correction value D _{(m, n) of the line} is taken in, the pixel signal S _{(m, n)} is corrected based on the correction value D _{(m, n)} . That is, the correction value D _{(m, n)} is subtracted from the pixel signal S _{(m, n)} for each pixel according to the following equation (10). As a result, a matrix array calculation result O _{(m, n)} shown in FIG. 13B can be obtained. The calculation result O _{(m, n)} is a pixel signal after the local dark current component (the dark current component in the partial region 24 in FIG. 3) is corrected. The correction process in the image processing apparatus 10 according to the present embodiment is thus completed.

O _{(m, n)} = S _{(m, n)} −D _{(m, n)} (10)
As described above, in the image processing apparatus 10 according to the first embodiment, based on the pixel signal output from the vicinity of the output unit 23 (the horizontal region 25 and the vertical region 26) in the light-shielding pixel region 22 of the image sensor 20. The dark current distribution in the effective pixel region 21 is estimated (the correction value D _{(m, n)} in FIG. 12 is obtained), and the effective pixel region 21 is based on the dark current distribution (correction value D _{(m, n)} ). In order to correct the pixel signal S _{(m, n)} output from the local dark current component (dark current component of the partial region 24 in FIG. 3) caused by the heat generating portion (output unit 23) of the image sensor 20 It can be corrected appropriately.

  That is, even if the distribution state of the local dark current component changes greatly according to the operating conditions (for example, the imaging time) of the image sensor 20, the change follows the change in the “near the output unit 23 (perpendicular to the horizontal region 25). Since the waveform of the pixel signal “output from the region 26) also changes, the dark current distribution in the effective pixel region 21 can be accurately estimated, and appropriate correction processing can always be performed. Therefore, it is possible to avoid a situation where an image is floated even during long exposure.

Further, in the image processing apparatus 10 of the first embodiment, an average value T _A of the pixel signals in the stable region shown in FIG. 7C is obtained, and the average value T _A is subjected to the smoothed pixel signals T ₀ ,. Subtract from each of _W and pixel signals L ₁ ,..., L _H and estimate the local dark current distribution in the effective pixel region 21 based on the pixel signal after subtraction (FIGS. 8 and 11). The local dark current component in the pixel region 21 can be corrected without excess or deficiency. In the subtraction process using the average value T _A , a negative value may be clipped to “0”.

Further, in the image processing apparatus 10 of the first embodiment, before obtaining the average value T _A of the pixel signals in the stable region, the pixel signals T ₀ ,..., T _W output from the image sensor 20 (FIG. 7A). ) And the pixel signals L ₁ ,..., L _H are excluded and smoothed by smoothing them (S1 to S9 in FIG. 6) (S21 to S29 in FIG. 9). Distribution can be estimated with good reproducibility. Therefore, the reproducibility of the correction process by the image processing apparatus 10 is also improved.

In the image processing apparatus 10 according to the first embodiment, since the pixel signals output from the horizontal region 25 and the vertical region 26 located in the vicinity of the output unit 23 (heat generating unit) of the image sensor 20 are used, the effective pixel region 21 is used. The local dark current distribution at can be reliably estimated.
Furthermore, in the image processing apparatus 10 of the first embodiment, after obtaining the dark current characteristics in the horizontal direction of the effective pixel area 21, the dark current characteristics in the vertical direction are obtained, and local multiplication of the effective pixel area 21 is performed by multiplying them. Since the dark current distribution is estimated, the correction processing of the pixel signal S _{(m, n)} output from the effective pixel region 21 for each line can be performed in substantially real time.
(Second Embodiment)
As shown in FIG. 14, the image processing apparatus 30 according to the second embodiment includes a gain unit 31 and a second calculation unit 32 between the calculation unit 13 and the correction unit 14 of the image processing apparatus 10 described above. Is. In this case, the correction value D _{(m, n) in} FIG. 12 output from the calculation unit 13 is output to the second calculation unit 32. In addition to the correction value D _{(m, n)} , the calculation unit 32 receives the gain G _{1 (m, n)} generated by the gain unit 31. The gain G _{1 (m, n)} also has the same matrix arrangement as the correction value D _{(m, n)} (FIG. 15).

The generation of the gain G _{1 (m, n)} in the gain unit 31 will be described. The gain unit 31 substitutes the coordinates (m, n) in the horizontal direction and the vertical direction in the effective pixel region 21 of the image sensor 20 into the following equations (11) to (13), so that the coordinates (m, n) A gain G _{1 (m, n)} depending on the is generated. A _m is the initial value of the horizontal direction, A _n is the initial value of the vertical direction, B _m is the slope of the horizontal direction, B _n is the vertical gradient. When the gain G _{1 (m, n)} is graphed, for example, as shown in FIG.

G1 _{(m, n)} = G1 _(m) × G1 _(n) (11)
G _{1 (m)} = A _m −B _m × m (12)
G _{1 (n)} = A _n −B _n × n (13)
Note that the parameters A _m , A _n , B _m , and B _n in Expressions (11) to (13) are determined in advance based on the operating conditions (imaging time and ISO setting) of the image sensor 20, the temperature of the outside air, and the like. . In order to detect the temperature of the outside air, for example, it is necessary to arrange a sensor in the vicinity of the image sensor 20.

The gain G _{1 (m, n)} (FIG. 15) generated in this way by the gain unit 31 is output to the calculation unit 32 at the subsequent stage, and the local dark current distribution estimated by the calculation unit 13 at the preceding stage. This is used for fine adjustment of ₍ correction value D _{(m, n) in} FIG. 12). In the second embodiment, the signal shaping units 11 and 12, the calculation units 13 and 32, and the gain unit 31 generally correspond to “estimating means” in the claims.
In the calculation unit 32, when the correction value D _{(m, n)} of a certain line is fetched from the calculation unit 13, the gain G1 _{(m, n)} of the corresponding line is fetched from the gain unit 31, and the following equation (14) is obtained. Therefore, multiplication is performed for each pixel. That is, the correction value D _{(m, n)} is finely adjusted in consideration of the gain G _{1 (m, n)} . As a result, the matrix array calculation result E _{(m, n)} shown in FIG. 17 can be obtained. The calculation result E _{(m, n)} represents a local dark current distribution (after fine adjustment ₎ in the effective pixel region 21.

E _{(m, n)} = G _{1 (m, n)} × D _{(m, n)} (14)
In the image processing apparatus 30 according to the second embodiment, the calculation result E _{(m, n)} is output to the correction unit 14 at the subsequent stage as the correction value E _{(m, n)} for each pixel in the effective pixel region 21. (FIG. 14). Similar to the image processing apparatus 10 already described (FIG. 1), the correction unit 14, in addition to the correction value E for each pixel _{(m, n),} the pixel signal S _(m from the effective pixel area 21 of the image sensor 20 _{, n)} (FIG. 13A) is input. Therefore, the pixel signal S _{(m, n)} can be corrected based on the correction value E _{(m, n)} taking into account the coordinate-dependent gain G _{1 (m, n)} . In this case, the correction value E _{(m, n)} is subtracted from the pixel signal S _{(m, n)} for each pixel according to the following equation (15).

O _{(m, n)} = S _{(m, n)} −E _{(m, n)} (15)
As described above, in the image processing apparatus 30 according to the second embodiment, the gain G _{1 (m, n)} depending on the coordinates (m, n) in the effective pixel region 21 of the image sensor 20 is taken into account and the local area is considered. The dark current distribution is estimated (the correction value E _{(m, n)} in FIG. 17 is obtained), and the pixels output from the effective pixel region 21 based on the dark current distribution (correction value E _{(m, n)} ). In order to correct the signal S _{(m, n)} , the effective pixel region 21 is changed by changing the set values of the parameters A _m , A _n , B _m , and B _n when generating the gain G _{1 (m, n).} The amount of correction of the local dark current component at (a dark current component of the partial region 24 in FIG. 3) can be finely adjusted.
(Third embodiment)
As shown in FIG. 18, the image processing device 40 according to the third embodiment is provided with a gain unit 41 and a calculation unit 42 instead of the gain unit 31 and the calculation unit 32 of the image processing device 30 described above. In this case, the correction value D _{(m, n)} of FIG. 12 output from the calculation unit 13 is output to the calculation unit 42. In addition to the correction value D _{(m, n)} , the calculation unit 42 receives the gain G _{2 (m, n)} generated by the gain unit 41. The gain G _{2 (m, n)} has a similar matrix arrangement (FIG. 19).

The generation of the gain G _{2 (m, n)} in the gain unit 41 will be described. The gain unit 41 generates the gain G _{2 (m, n)} by substituting the horizontal and vertical coordinates (m, n) in the effective pixel region 21 of the image sensor 20 into the following equation (16). To do. F _{IM (m, n)} represents the heat conduction characteristic of the image sensor 20. A ₀ is an initial value, and K is a fixed value corresponding to the heat conduction characteristics. When the gain G _{2 (m, n)} is graphed, for example, as shown in FIG.

G _{2 (m, n)} = A ₀ × F _{IM (m, n)} (16)
Note that the parameter A ₀ in Expression (16) is determined in advance based on the operating conditions (imaging time and ISO setting) of the image sensor 20, the temperature of the outside air, and the like. In order to detect the temperature of the outside air, for example, it is necessary to arrange a sensor in the vicinity of the image sensor 20.
The gain G _{2 (m, n)} (FIG. 19) generated in this way by the gain unit 41 is output to the subsequent calculation unit 42 and is estimated by the previous calculation unit 13 in the local dark current distribution. This is used for fine adjustment of ₍ correction value D _{(m, n) in} FIG. 12). In the third embodiment, the signal shaping units 11 and 12, the calculation units 13 and 42, and the gain unit 41 generally correspond to “estimating means” in the claims.

In the calculation unit 42, when the correction value D _{(m, n)} of a certain line is fetched from the calculation unit 13, the gain G2 _{(m, n)} of the corresponding line is fetched from the gain unit 41, and the following equation (18) is obtained. Therefore, multiplication is performed for each pixel. That is, the correction value D _{(m, n)} is finely adjusted in consideration of the gain G _{2 (m, n)} . As a result, the matrix array calculation result F _{(m, n)} shown in FIG. 21 can be obtained. This calculation result F _{(m, n)} represents a local dark current distribution (after fine adjustment ₎ in the effective pixel region 21.

F _{(m, n)} = G _{2 (m, n)} × D _{(m, n)} (18)
In the image processing device 40 of the third embodiment, the calculation result F _{(m, n)} is output to the correction unit 14 at the subsequent stage as the correction value F _{(m, n)} for each pixel in the effective pixel region 21. (FIG. 18). Similar to the image processing apparatus 10 already described (FIG. 1), the correction unit 14, the correction value F _{(m, n)} for each pixel of the other pixel signals S _(m from the effective pixel area 21 of the image sensor 20 _{, n)} (FIG. 13A) is input. Therefore, the pixel signal S _{(m, n)} can be corrected based on the correction value F _{(m, n)} taking into account the temperature-dependent gain G _{2 (m, n)} described above. In this case, the correction value F _{(m, n)} is subtracted from the pixel signal S _{(m, n)} for each pixel according to the following equation (19).

O _{(m, n)} = S _{(m, n)} −F _{(m, n)} (19)
As described above, in the image processing apparatus 40 according to the third embodiment, the gain G _{2 (m, n} ) depending on the coordinates (m, n) in the effective pixel region 21 of the image sensor 20 and the heat conduction characteristics of the image sensor 20. _{) In} consideration of the local dark current distribution ₍ determining the correction value F _{(m, n)} in FIG. 21) and effective based on this dark current distribution (correction value F _{(m, n)} ). In order to correct the pixel signal S _{(m, n)} output from the pixel region 21, the set value of the parameter A ₀ when generating the gain G _{2 (m, n)} is changed, thereby changing the effective pixel region 21. The correction amount of the local dark current component (dark current component of the partial region 24 in FIG. 3) can be finely adjusted.
(Modification)
In the embodiment described above, based on the pixel signals output from the horizontal region 25 (FIG. 4) and the vertical region (FIG. 5) located in the vicinity of the output unit 23 in the light-shielding pixel region 22 of the image sensor 20. Although the dark current distribution in the effective pixel region 21 is estimated, the present invention is not limited to this. The present invention can also be applied to the case where pixel signals output from the partial region (corresponding to the partial region 24 shown in FIG. 3) of the horizontal region 25 on the output unit 23 side and the vertical region 26 on the output unit 23 side are used. .

In the above-described embodiment, when the horizontal area 25 and the vertical area 26 are divided into a plurality of blocks (FIGS. 4B and 5B), the width other than the upper left block is one pixel (or one line) wide. Although the example divided into blocks has been described, the present invention is not limited to this. The present invention can also be applied to a case where one block width is a plurality of pixels (or a plurality of lines).
Furthermore, in the above-described embodiment, when finely adjusting the local dark current distribution (the correction value D _{(m, n) in} FIG. 12 ₎ estimated by the calculation unit 13, the gain G _{1 (m, n in} FIG. ₎ Or the gain G _{2 (m, n)} in FIG. 19 has been described, but the present invention is not limited to this. Gain G _{1 (m, n)} in FIG. 15 the gain G _{2 (m, n)} of FIG. 19 also in the case of adding both the, the present invention can be applied.

In the above-described embodiment, when normalizing the smoothed pixel signals T ₀ ,..., T _W and the pixel signals L ₀ ,..., L _H , step S13 (formula (4)) in FIG. In step S32 (Equation (8)), the pixel signals T _m and L _n are divided by the subtracted representative value (T _B −T _A ), and the calculation process according to Equation (9) is performed again. Although an example of multiplying by the representative value (T _B −T _A ) has been described, the present invention is not limited to this. “(T _B −T _A )” in either step S13 (formula (4)) of FIG. 6 or step S32 (formula (8)) of FIG. 9 is omitted, and the following formula (9) The present invention can also be applied to the case where “(T _B −T _A )” is omitted.

Furthermore, in the above embodiment, the pixel signal L ₀ after the vertical smooth, ..., when normalizing the L _H, the pixel signal T ₀ after the smoothing in the horizontal direction, ..., T _W (FIG. 7 (c) The average value T _A and the representative value T _B obtained from the above are obtained from the signal shaping unit 11 (step S30 in FIG. 9) and used for normalization. However, the present invention is not limited to this. . A similar average value and representative value may be obtained in the vertical direction, and normalization in the vertical direction may be performed using the average value and the representative value. However, using the information in the horizontal direction (average value T _A and representative value T _B ) at the time of normalization in the vertical direction immediately after capturing the pixel signal S _{(m, n)} from the effective pixel region 21. This is preferable because the dark current component can be corrected.

In the above-described embodiment, an example in which the output unit 23 of the image sensor 20 is a heat generating unit has been described, but the present invention is not limited to this. The present invention can be applied even when a portion other than the output unit 23 is a heat generating unit.
Furthermore, in the above-described embodiment, the case where the image processing apparatus 10 is incorporated in an imaging apparatus such as a digital camera has been described as an example, but the present invention is not limited to this. The present invention is also applicable to the case where RAW data (including the pixel signal of the light-shielded pixel region 22) output from the image sensor 20 is taken into an external computer and local dark current component correction processing is performed by software on the computer. Can be applied. In this case, an image processing program recorded on a recording medium (CD-ROM or the like) may be installed in the computer. Alternatively, an image processing program downloaded via the Internet may be installed.

1 is a block diagram illustrating an overall configuration of an image processing apparatus 10 according to a first embodiment. 1 is a schematic diagram illustrating a configuration of an image sensor 20. 4 is a schematic diagram illustrating a partial region 24 in which a local dark current component is superimposed in the imaging element 20. FIG. It is a figure explaining the horizontal area | region 25 which takes in a pixel signal, when estimating a local dark current component. It is a figure explaining the vertical area | region 26 which takes in a pixel signal, when estimating a local dark current component. It is a flowchart which shows the operation | movement procedure of the signal shaping part 11 of the image processing apparatus 10 of 1st Embodiment. FIG. 6 is a graph of horizontal pixel signals T ₀ ,..., T _W (before smoothing (a), during smoothing (b), after smoothing (c)). FIG. 6 is a graph of pixel signals T ₀ ,..., T _W after normalization in the horizontal direction. It is a flowchart which shows the operation | movement procedure of the signal shaping part 12 of the image processing apparatus 10 of 1st Embodiment. FIG. 5 is a graph of vertical pixel signals L ₀ ,..., L _H (after smoothing). FIG. 6 is a graph of pixel signals L ₀ ,..., L _H after normalization in the vertical direction. It is a figure explaining the correction value D _{(m, n)} which the calculating part 13 calculated | required. FIG. 7A is a diagram for explaining the pixel signal S _{(m, n)} output from the effective pixel region 21, and FIG. 7B is a diagram for explaining the pixel signal O _{(m, n)} after the local dark current component is corrected. ). It is a block diagram which shows the whole structure of the image processing apparatus 30 of 2nd Embodiment. It is a figure explaining gain G1 _{(m, n)} produced | generated by the gain part. It is the figure which graphed gain G1 _{(m, n)} . It is a figure explaining the correction value E _{(m, n)} which the calculating part 32 calculated | required. It is a block diagram which shows the whole structure of the image processing apparatus 40 of 3rd Embodiment. It is a figure explaining the gain G2 _{(m, n)} produced | generated by the gain part 41. FIG. FIG. 6 is a graph showing gain G _{2 (m, n)} . It is a figure explaining the correction value F _{(m, n)} which the calculating part 42 calculated | required.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10, 30, 40 Image processing apparatus 11,12 Signal shaping part 13,32,42 Calculation part 14 Correction part 20 Imaging element 21 Effective pixel area 22 Light-shielding pixel area 23 Output part (heat generation part)
24 Partial region where local dark current components are superimposed 25 Horizontal region 26 Vertical region 31, 41 Gain unit

Claims (9)

Estimating means for estimating a dark current distribution in an effective pixel region of the image sensor based on a plurality of first pixel signals output from at least the vicinity of the heat generating portion of the image sensor in a light-shielding pixel region of the image sensor;
An image processing apparatus comprising: a correcting unit that corrects a plurality of second pixel signals output from the effective pixel region based on the dark current distribution estimated by the estimating unit. The image processing apparatus according to claim 1.
The estimating means selects a pixel signal having a small influence of the heat generating portion from the plurality of first pixel signals, obtains an average value, and subtracts the average value from each of the plurality of first pixel signals, The image processing apparatus, wherein the dark current distribution is estimated based on a plurality of first pixel signals after subtraction. The image processing apparatus according to claim 2,
The image processing apparatus according to claim 1, wherein the estimation unit performs a neighborhood calculation for smoothing the surface by excluding protruding points of the plurality of first pixel signals before obtaining the average value. The image processing apparatus according to claim 1.
The image processing apparatus, wherein the estimation unit estimates the dark current distribution in consideration of a gain depending on coordinates in the effective pixel region. The image processing apparatus according to claim 1.
The image processing apparatus characterized in that the estimation means estimates the dark current distribution in consideration of coordinates within the effective pixel region and a gain depending on a heat conduction characteristic of the image sensor. Based on a plurality of first pixel signals output from each of a horizontal region and a vertical region located in the vicinity of the heat generating portion of the image sensor in the light-shielding pixel region of the image sensor, darkness in the effective pixel region of the image sensor An estimation means for estimating a current distribution;
An image processing apparatus comprising: a correcting unit that corrects a plurality of second pixel signals output from the effective pixel region based on the dark current distribution estimated by the estimating unit. The image processing apparatus according to claim 6.
The estimating means obtains a dark current characteristic in the horizontal direction of the effective pixel area based on the first pixel signal output from the horizontal area, and then, based on the first pixel signal output from the vertical area. The dark current distribution in the vertical direction of the effective pixel region is obtained, and the dark current distribution is estimated by multiplying the two dark current characteristics. An estimation procedure for estimating a dark current distribution in an effective pixel region of the image sensor based on a plurality of first pixel signals output from at least the vicinity of the heat generating portion of the image sensor in a light-shielding pixel region of the image sensor;
An image processing program for causing a computer to execute a correction procedure for correcting a plurality of second pixel signals output from the effective pixel region based on the dark current distribution estimated by the estimation means. Based on a plurality of first pixel signals output from each of a horizontal region and a vertical region located in the vicinity of the heat generating portion of the image sensor in the light-shielding pixel region of the image sensor, darkness in the effective pixel region of the image sensor An estimation procedure for estimating the current distribution;
An image processing program for causing a computer to execute a correction procedure for correcting a plurality of second pixel signals output from the effective pixel region based on the dark current distribution estimated by the estimation means.

Images