JP4525198B2 - Noise removal device, electronic camera, and noise removal program. - Google Patents

Description

The present invention relates to a noise removal device for removing noise from image data, and a noise removal program.
The present invention also relates to an electronic camera equipped with this noise removing device.

In general, when long-time shooting is performed with an electronic camera, fixed pattern noise appears in image data. As a device for removing this type of noise, a conventional device of Patent Document 1 is known.
In this conventional apparatus, first, normally captured image data and dark image data captured with the shutter closed are prepared. The conventional apparatus subtracts the dark image data from the image data in units of pixels, thereby removing the in-phase fixed pattern noise.
JP 2000-125204 A (Claims 1 and 2 etc.)

  By the way, the dark image data taken for a long time includes random noise in addition to fixed pattern noise. In the subtraction processing of the conventional device described above, this random noise is reversed in phase and added to the image data, so that there is a problem that the noise of the image data is increased in reverse.

  Furthermore, in the conventional apparatus, in order to obtain dark image data, a charge accumulation time equal to that of the image data is required. That is, when long-time shooting for 10 minutes is performed, a charge accumulation time of 10 minutes is required to obtain dark image data with the same conditions. In the meantime, shooting using an electronic camera becomes impossible, and a problem arises that a valuable shutter opportunity is missed.

  Further, the conventional apparatus has a problem that the amount of memory consumption increases because the dark image data is temporarily stored.

Therefore, an object of the present invention is to provide a technique in which random noise does not increase in noise removal during long-time shooting.
Another object of the present invention is to provide a technique for shortening the charge accumulation time of dark image data.
It is another object of the present invention to provide a technique for estimating a noise occurrence location in image data from dark image data with a reduced charge accumulation time.

<Claim 1>
According to a first aspect of the present invention, there is provided a noise removal apparatus including an image storage unit, a dark current offset unit, a noise detection unit, and a noise removal unit.
The image storage unit stores image data obtained by imaging the scene with the imaging device.
The dark current offset unit calculates a dark current offset value indicating a level increase due to dark current from dark image data obtained by imaging with an image sensor in a shielded state, and subtracts the dark current offset value from the dark image data. .
The noise detection unit acquires dark image data from which the dark current offset is removed, and detects a noise position exceeding a predetermined threshold value from the dark image data.
The noise removing unit performs noise removal by replacing neighboring pixel values with respect to the noise position in the image data.

<Claim 2>
The noise removing device according to claim 2 is the noise removing device according to claim 1, wherein the dark image data is set such that the charge accumulation time of the image sensor is shortened compared to the image data. On the other hand, the noise detection unit estimates a noise position that can be visually recognized by extending the charge accumulation time from the dark image data with the shortened charge accumulation time by setting the threshold value low according to the reduction ratio of the charge accumulation time. .

<Claim 3>
According to a third aspect of the present invention, there is provided the noise removing apparatus according to the first aspect, wherein the dark image data is set with a shorter charge accumulation time of the image pickup device than the image data. On the other hand, the noise detector amplifies the signal level of the dark image data in accordance with the reduction rate of the charge accumulation time, and simulates the noise that appears due to the extension of the charge accumulation time. The noise detection unit estimates the noise position by determining the threshold value of the dark image data after amplification.

<Claim 4>
The noise removal device according to claim 4 is the noise removal device according to claim 3, wherein the noise detecting unit amplifies the signal level of the dark image data after subtracting the dark current offset from the dark image data.

<Claim 5>
A noise removal program according to a fifth aspect causes a computer to function as the image storage unit, the dark current offset unit, the noise detection unit, and the noise removal unit according to any one of the first to fourth aspects. And

<Claim 6>
According to a sixth aspect of the present invention, there is provided an electronic camera according to any one of the first to fourth aspects, the image pickup device, a shielding mechanism that shields the image pickup device, and the object field by controlling the image pickup device. And a controller that drives the shielding mechanism to shield the image sensor and generate dark image data.

<Claim 7>
An electronic camera according to a seventh aspect of the present invention includes an imaging device, a shielding mechanism that shields the imaging device, generates image data of the object scene by controlling the imaging device, drives the shielding mechanism to shield the imaging device, and darkens the image sensor. A dark current offset unit for calculating a dark current offset value indicating a level increase due to dark current from the dark image data, and a dark current offset unit for subtracting the dark current offset value from the dark image data; The image processing apparatus includes a noise detection unit that acquires dark image data from which an offset is removed, detects a noise position that exceeds a threshold from the dark image data, and a recording unit that records the data of the noise position in association with the image data.

(Claim 1)
According to another aspect of the present invention, there is provided a noise removal apparatus that captures dark image data captured by a shielded image sensor. This dark image data has no information inherent to the image, and mainly includes the following three types of noise.
(1) Fixed pattern noise (2) Random noise (3) Dark current offset

  Of these noises, the dark current offset is close to a direct current component and is easy to estimate. For example, the dark current offset can be estimated by obtaining the average signal level of the dark image data. For example, the dark current offset can be estimated from the conditions of the temperature of the image sensor and the charge accumulation time. Further, a dummy light receiving element can be provided in the imaging element, and the dark current offset can be estimated from the output of the dummy light receiving element.

  Therefore, the noise removing device removes the dark current offset that can be estimated in this way from the dark image data. As a result, fixed pattern noise and random noise mainly remain in the dark image data.

  Among them, the fixed pattern noise is almost constant noise for each pixel. Therefore, the fixed pattern noise increases almost in proportion to the charge accumulation time.

  On the other hand, with random noise, noise generation is random and not constant. Therefore, the average level of random noise is not directly proportional to the charge accumulation time. (For example, for complete random noise, the average level of random noise is proportional to the square root of the charge accumulation time.)

For this reason, as the charge accumulation time becomes longer, the signal levels of the fixed pattern noise and the random noise diverge more and more. Therefore, by setting an appropriate threshold between the deviated signal levels, it is possible to distinguish between “fixed pattern noise that occurs locally” and “random noise that occurs with a uniform variation across the entire screen”. Can do.
Through such processing, the present apparatus accurately detects the occurrence location (noise position) of the fixed pattern.

  Further, in the present apparatus, local noise removal is performed on the image data side of normal photographing at the obtained noise position. As a result, there is no adverse effect that random noise is added to the image data as in the conventional case, and a high-quality noise removal result can be obtained.

(Claim 2)
According to another aspect of the present invention, there is provided a noise removing apparatus for shortening a charge accumulation time of dark image data. Normally, the fixed pattern noise increases almost in proportion to the charge accumulation time. Therefore, if the charge accumulation time is shortened, the signal level of the fixed pattern noise is lowered in a form that is substantially proportional to the shortening ratio. Therefore, the present apparatus lowers the threshold value so as to follow the decrease in the signal level. As a result, it is possible to accurately detect the noise position of the fixed pattern noise even for dark image data with a reduced charge accumulation time.
As described above, since the charge accumulation time of the dark image data can be flexibly shortened, it becomes possible to shorten the time during which the electronic camera cannot be photographed, thereby realizing a more convenient electronic camera.

(Claim 3)
According to a third aspect of the present invention, there is provided a noise removing apparatus for shortening a charge accumulation time of dark image data. Normally, the fixed pattern noise increases almost in proportion to the charge accumulation time. Therefore, the fixed pattern noise in the dark image data is reduced in proportion to the reduction rate of the charge accumulation time. Therefore, the present apparatus amplifies the dark image data to compensate for the lower level of the fixed pattern noise. As a result, the noise position of the fixed pattern noise can be accurately detected even with dark image data created by shortening the charge accumulation time.
As described above, since the charge accumulation time of the dark image data can be flexibly shortened, it is possible to shorten the time during which the electronic camera cannot be photographed, thereby realizing an electronic camera that is more convenient to use.

(Claim 4)
The noise removal apparatus according to claim 4 amplifies the signal level of the dark image data after subtracting the dark current offset from the dark image data. In this operation, the level of dark image data before amplification can be lowered by subtracting the dark current offset in advance.
Therefore, there is little possibility that the dark image data reaches the saturation level after amplification, and the level difference between the random noise and the fixed pattern noise can be kept clear even after amplification. As a result, the noise position of the fixed pattern noise can be accurately detected without being affected by saturation.

(Claim 5)
By executing the noise removal program of claim 5 on a computer, the computer can function as the noise removal device of any one of claims 1 to 4.

(Claim 6)
An electronic camera according to a sixth aspect is equipped with the noise removing device according to any one of the first to fourth aspects. Therefore, the above-described noise removal can be executed in the electronic camera.

(Claim 7)
The electronic camera according to claim 7 detects a noise position of the fixed pattern noise. The noise position data thus detected is recorded and stored in association with the image data.
Therefore, it is possible to store the image data at the time of imaging as it is and perform noise removal based on the noise position later.
In particular, in such an operation, it is only necessary to record the data of the noise position, so that the recording size can be remarkably saved as compared with recording dark image data together. As a result, the number of recordable frames on the recording medium can be increased.

<< First Embodiment >>
The first embodiment is an embodiment corresponding to claims 1, 3, 4, 6, and 7.
FIG. 1 is a diagram illustrating a configuration of an electronic camera 11 according to the first embodiment.
A photographing lens 12 is attached to the electronic camera 11. In the image space of the photographing lens 12, a shutter 13 and an image sensor 14 are arranged. The shutter 13 and the image sensor 14 are controlled by the control unit 15.

The image data output from the image sensor 14 is recorded in the image data storage circuit 17 via the dark current offset circuit 16.
On the other hand, dark image data (described later) output from the image sensor 14 is supplied to the amplifier 18 via the dark current offset circuit 16. The amplification factor of the amplification unit 18 is adjusted by the gain setting circuit 22. The dark white point detection circuit 19 determines the threshold value of the output of the amplifier 18 and extracts fixed pattern noise exceeding the threshold value. The threshold value of the dark white point detection circuit 19 is adjusted by the threshold setting circuit 32. The address output circuit 20 converts the extracted noise position into screen coordinates (address information), outputs it, and temporarily stores it in the address storage circuit 21. The dark white point correction circuit 23 performs local noise removal on the image data in the image data storage circuit 17 in accordance with the noise position acquired from the address storage circuit 21. The card interface 24 records and saves image data, noise positions, and the like in the memory card 25.

[Correspondence with Invention]
The correspondence relationship between the invention and this embodiment will be described below. Note that the correspondence relationship here illustrates one interpretation for reference, and does not limit the present invention.
The image storage unit described in the claims corresponds to the image data storage circuit 17.
The dark current offset unit described in the claims corresponds to the dark current offset circuit 16.
The noise detection unit described in the claims corresponds to the amplification unit 18, the gain setting circuit 22, the threshold setting circuit 32, the dark white point detection circuit 19, the address output circuit 20, and the address storage circuit 21.
The noise removing unit described in the claims corresponds to the dark white point correction circuit 23.
The imaging device described in the claims corresponds to the imaging device 14.
The shielding mechanism described in the claims corresponds to the shutter 13.
The control unit described in the claims corresponds to the control unit 15.
The recording unit described in the claims corresponds to the card interface 24.

[Description of noise removal operation]
FIG. 2 is a flowchart for explaining the noise removal processing of the electronic camera 11.
FIG. 3 is a diagram illustrating a process of dark image data.
Hereinafter, description will be made along the step numbers shown in FIG.

Step S1: In response to the user's release operation, the control unit 15 opens the shutter 13 and forms a subject image by the photographing lens 12 on the imaging surface of the imaging element 14. In this state, the control unit 15 causes the image sensor 14 to perform photoelectric conversion and charge accumulation, thereby generating image data of the charge accumulation time Ta. The image sensor 14 outputs the image data in the order of screen scanning.

Step S2: The dark current offset circuit 16 detects an increase in the signal level due to the dark current from the output of the OB pixel (light-shielding dummy pixel provided around the effective screen) included in the image data, and dark current offset And The dark current offset circuit 16 subtracts the dark current offset from the image data, thereby suppressing the black level from rising due to the long-time shooting. The image data processed in this way is temporarily stored in the image data storage circuit 17.

Step S3: Next, the control unit 15 determines whether or not to perform long-second shooting noise removal based on the charge accumulation time Ta.
If it is determined that the charge accumulation time Ta is shorter than the predetermined time and the fixed pattern noise is negligible, a normal photographing process routine is executed.
On the other hand, when it is determined that the charge accumulation time Ta is longer than the predetermined time and the fixed pattern noise cannot be ignored, the operation is shifted to step S4 and subsequent steps in order to perform noise removal.

Step S4: The control unit 15 determines the charge accumulation time Tb based on the charge accumulation time Ta and the temperature. The charge accumulation time Tb is a value obtained by experimentally obtaining in advance a “lower limit charge accumulation time” in which the charge accumulation time Ta and the occurrence of fixed pattern noise are maintained in a substantially proportional relationship.
The control unit 15 causes the image sensor 14 to accumulate charges while closing the shutter 13 and maintaining the light shielding state, and generates dark image data of the charge accumulation time Tb.
FIG. 3A shows dark image data generated by the image sensor 14 in this way.

Step S5: The dark current offset circuit 16 obtains a dark current offset from the dark image data, and subtracts the obtained dark current offset from the dark image data.
FIG. 3B shows dark image data obtained by subtracting the dark current offset in this way.

Step S6: The gain setting circuit 22 acquires the charge accumulation times Ta and Tb from the control unit 15, determines the amplification magnification based on the following formula, and sets the amplification factor 18 in the amplification unit 18.
Amplification factor = (Ta / Tb) α
The proportionality constant α is the default amplification magnification of the dark white point detection circuit 19.
The amplifying unit 18 amplifies the dark image data in accordance with the set amplification magnification.
FIG. 3C shows dark image data amplified in this way.

Step S7: The threshold setting circuit 32 determines the threshold RN based on the charge accumulation times Ta and Tb and the temperature. This threshold value RN is a value obtained by experimentally obtaining the variation upper limit of the value of random noise generated in the dark image data during the charge accumulation time Ta.
Note that random noise occurs uniformly in the screen. On the other hand, white-point fixed pattern noise occurs only in specific pixels within the screen. From such a feature, the threshold value RN may be the last signal level when a predetermined limited number (for example, 100) is extracted in descending order of the signal level from the histogram distribution of the dark image data.
The threshold value setting circuit 32 sets the threshold value RN thus determined in the dark white point detection circuit 19. The dark white point detection circuit 19 determines a threshold value of the dark image data after amplification using the set threshold value RN, and extracts noise exceeding the threshold value RN.
The address output circuit 20 converts the position of the noise extracted by the dark white point detection circuit 19 into address information indicating screen coordinates and outputs it. This address information is temporarily stored in the address storage circuit 21.

Step S8: Here, when the electronic camera 11 is set to the RAW recording mode, the operation shifts to step S9. On the other hand, in other cases, the operation proceeds to step S10.

Step S9: The card interface 24 records and saves the image data in the image data storage circuit 17 (that is, RAW data) and the noise position information in the address storage circuit 21 in association with each other.
Note that it is preferable to store data obtained by compressing the coordinate value of the noise position and compressed data of map information (mask image) indicating the noise position in a predetermined data area in the RAW data.
Such RAW data can be subjected to noise removal processing based on the noise position in post-processing (so-called development processing) on a computer.
In this way, the electronic camera 11 completes the RAW data recording process.

Step S <b> 10: The dark white point correction circuit 23 reads address information indicating the noise position from the address storage circuit 21. The dark white point correction circuit 23 reads a pixel value near the noise position from the image data in the image data storage circuit 17 in accordance with the read address information. The dark white point correction circuit 23 performs a noise removal operation (for example, a weighted average operation or a median operation) on these neighboring pixel values, and creates a pixel value after noise removal. The dark white point correction circuit 23 replaces the pixel value after the noise removal with the pixel value at the noise position.
By performing such processing for each noise position recorded in the address storage circuit 21, fixed pattern noise is removed from the image data in the image data storage circuit 17.

Step S11: The electronic camera 11 performs processing such as color interpolation processing and color coordinate conversion on the image data after the noise removal. After such processing, the card interface 24 records and saves the image data in the memory card 25.

[Effects of First Embodiment]
As described above, in the first embodiment, by removing the dark current offset from the dark image data, as shown in FIG. 3B, the dark image data mainly includes fixed pattern noise and random noise. Remains. Next, according to the threshold determination shown in FIG. 3C, random noise that falls within a uniform variation width is cut out, and locally generated fixed pattern noise is detected.

  It is possible to remove the fixed pattern noise included in the image data by performing noise removal of the image data for the noise position of the fixed pattern noise detected in this way.

  Unlike conventional devices, image data is not affected by random noise included in dark image data. As a result, random noise does not increase, and a good noise removal result can be obtained.

  Furthermore, in the first embodiment, by shortening the charge accumulation time Tb of the dark image data, it is possible to shorten the waiting time for capturing the dark image data.

  At this time, in the first embodiment, the dark image data is amplified so as to cancel the reduction ratio (Tb / Ta) of the charge accumulation time. As a result, it is possible to accurately detect the noise position even if the charge accumulation time Tb is shortened.

  In the first embodiment, the signal level is reduced before amplification by preliminarily subtracting the dark current offset from the dark image data. Therefore, there is little possibility that the dark image data is saturated after amplification, and the noise position can be detected more accurately.

  In the first embodiment, when the RAW recording mode is set, the detected noise position and the RAW data are recorded in association with each other. As a result, noise removal using the noise position can be performed in the post-processing of the RAW data.

  Furthermore, the data size of the noise position is much smaller than the data size of the dark image data. Therefore, even if noise position information is added, the recording size of the RAW data does not increase so much. Therefore, the number of frames that can be captured on the recording medium can be increased as compared with the case where dark image data is recorded together with RAW data.

In the first embodiment, dark image data is sequentially processed in a pipeline manner to detect a noise position. Therefore, a memory space for temporarily storing dark image data is not necessary. Therefore, the memory usage can be saved.
Next, another embodiment will be described.

<< Second Embodiment >>
The second embodiment is an embodiment corresponding to claims 1, 2, 6, and 7.
FIG. 4 is a diagram illustrating a configuration of the electronic camera 31 according to the second embodiment.
A feature point of the configuration of the electronic camera 31 is that the amplification unit 18 is fixed at a default amplification factor α, and the threshold value setting circuit 32 is used to positively increase or decrease the threshold value of the dark white point detection circuit 19. is there. Since other configurations are the same as those in the first embodiment (FIG. 1), description thereof is omitted here.
FIG. 5 is a flowchart for explaining the noise removal processing of the electronic camera 31.
In the following, description will be given along the step numbers shown in FIG.

Steps S21 to S25: Same as steps S1 to S5 of the first embodiment.

Step S26: The threshold setting circuit 32 acquires the charge accumulation times Ta and Tb from the control unit 15, obtains a threshold proportional to the shortening ratio using the following equation, and sets it in the dark white point detection circuit 19.
Threshold = RN (Tb / Ta) β
The proportionality constant β is a correction coefficient that is experimentally determined based on the subjective evaluation of the noise removal effect. The proportionality constant RN is a default threshold value.

Step S27: The dark white point detection circuit 19 performs threshold determination on the dark image data using the set threshold, and extracts noise exceeding the threshold. The extracted noise position is converted into address information by the address output circuit 20 and then temporarily stored in the address storage circuit 21.

Steps S28 to S31: Same as steps S8 to S11 of the first embodiment.
The operation and effect close to those of the first embodiment can also be obtained by the above operation.

<< Additional items of embodiment >>
In the above-described embodiment, the case where noise removal is performed in the electronic camera has been described. However, the present invention is not limited to this. For example, the noise removal process shown in FIG. 2 or 5 may be programmed (corresponding to claim 5). By executing this noise removal program on a computer, the computer can function as a noise removal device.

  In the above-described embodiment, the threshold determination for noise position detection is performed after the dark current offset is subtracted from the dark image data. However, the present invention is not limited to this. For example, instead of subtracting the dark current offset from the dark image data, the threshold value may be increased by an amount corresponding to the dark current offset.

  In the first embodiment described above, amplification is performed after the dark current offset is subtracted from the dark image data. This avoids the phenomenon that white spot noise is saturated during amplification, and enables accurate detection of the noise position. However, the present invention is not limited to this. If there is no nonlinear phenomenon such as saturation, the dark image data is first amplified, and the amplified dark current offset is subtracted from the dark image data after amplification.

  As described above, the present invention is a technique that can be used for an apparatus or a program that removes noise from an image taken in a long time.

It is a figure which shows the structure of the electronic camera 11 in 1st Embodiment. 5 is a flowchart for explaining noise removal processing of the electronic camera 11; It is a figure which shows the process progress of dark image data. It is a figure which shows the structure of the electronic camera 31 in 2nd Embodiment. 5 is a flowchart illustrating noise removal processing of the electronic camera 31.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Electronic camera 12 Shooting lens 13 Shutter 14 Image pick-up element 15 Control part 16 Dark current offset circuit 17 Image data storage circuit 18 Amplification part 19 Dark white point detection circuit 20 Address output circuit 21 Address storage circuit 22 Gain setting circuit 23 Dark white Point correction circuit 24 Card interface 25 Memory card 31 Electronic camera 32 Threshold setting circuit

Claims (7)

An image storage unit for storing image data obtained by imaging the object scene with an imaging device;
A dark current offset unit that calculates a dark current offset value indicating a level increase due to dark current from dark image data obtained by imaging with the imaging element in a shielded state, and subtracts the dark current offset value from the dark image data ; ,
Obtaining the dark image data from which the dark current offset has been removed, and detecting a noise position exceeding a predetermined threshold from the dark image data;
A noise removing device, comprising: a noise removing unit that removes noise by replacing neighboring pixel values with respect to the noise position in the image data. In the noise removal apparatus of Claim 1,
The dark image data is set by shortening the charge accumulation time of the image sensor compared to the image data,
The noise detector is
Estimating the noise position that becomes visible by extending the charge accumulation time from the dark image data with the charge accumulation time shortened by setting the threshold low according to the shortening ratio of the charge accumulation time. A noise removing device characterized by the above. In the noise removal apparatus of Claim 1,
The dark image data is set by shortening the charge accumulation time of the image sensor compared to the image data,
The noise detector is
The signal level of the dark image data is amplified in accordance with the shortening ratio of the charge accumulation time, the noise that appears due to the extension of the charge accumulation time is simulated, and the dark image data after amplification is threshold-determined to determine the noise. A noise removing device characterized by estimating a position. In the noise removal apparatus of Claim 3,
The noise detecting unit amplifies a signal level of the dark image data after subtracting a dark current offset from the dark image data. A noise removal program for causing a computer to function as the image storage unit, dark current offset unit, noise detection unit, and noise removal unit according to any one of claims 1 to 4. The noise removal device according to any one of claims 1 to 4,
An image sensor;
A shielding mechanism for shielding the image sensor;
An electronic device comprising: a control unit that controls the image sensor to generate image data of an object scene; drives the shielding mechanism to shield the image sensor and generates the dark image data; camera. An image sensor;
A shielding mechanism for shielding the image sensor;
A control unit that controls the image sensor to generate image data of the object scene, drives the shielding mechanism to shield the image sensor and generates dark image data, and a dark current level from the dark image data. A dark current offset value that calculates a dark current offset value indicating an increase, and subtracts the dark current offset value from the dark image data ; and
Obtaining the dark image data from which the dark current offset has been removed, and detecting a noise position exceeding a predetermined threshold from the dark image data;
An electronic camera comprising: a recording unit that records the data of the noise position in association with the image data.

Images