JP2007336386A - Image pick-up device and image pick-up method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To keep a resolution feeling relating to a subject image and to suppress a noise of the photographed image. <P>SOLUTION: Image information of a plurality of images obtained by changing an image formation state in an image pick-up element having a plurality of pixels is compared for each pixel or each pixel region, and a content of a signal processing for the photographed image is changed in response to a change of the image information obtained as the comparison result. A noise suppression processing is weakened in a portion which is determined as a pattern portion of a subject based on a magnitude of the change of the image information, and the noise suppression processing is strengthened in the other portion, so that both noise suppression of the photographed image and retention of the resolution feeling of a pattern of the subject are achieved. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and an imaging method, and more particularly to an image processing technique for improving the quality of a captured image.

  An imaging apparatus such as a digital camera or a video camera forms an external subject image on an image sensor such as a CCD or CMOS through an optical system including a zoom lens, a focus lens, and an optical camera shake correction mechanism. An image signal is obtained by converting the optical image of the imaged subject into an electric signal, that is, by photoelectric conversion.

  In recent years, with the penetration of digital signal processing technology, image signals output from the image sensor are analog-digital (A / D) converted and handled as digital image signals in almost all imaging devices. After the output from the image sensor, various and advanced image processing can be performed by converting each signal processing into a digital signal processing except analog signal processing such as correlated double sampling. For this reason, the image quality of the image obtained by the imaging apparatus has been greatly improved due to a synergistic effect with the increase in the number of pixels in the image sensor.

  Among the image quality of images obtained by the image pickup apparatus, noise is one of the main ones, and it can be said that the image quality is so good that the formed optical image can be converted into an image signal with as little noise as possible.

  Here, when classifying the noise in the imaging device, there is noise due to the dark current of the image sensor as noise generated on the image sensor, and the dark current itself appears as fixed pattern noise. In addition, there is noise called shot noise, which appears as so-called random noise. This shot noise includes noise that is generated regardless of the charge generated by the image sensor called dark current shot noise and noise that varies depending on the charge generated by the image sensor called optical shot noise. In addition, what is called a defective pixel can be regarded as a kind of noise from a large classification, and can be regarded as fixed pattern noise like dark current.

  Next, noise generated when the output signal of the image sensor is digitally converted includes noise generated by charge induction from the periphery in an analog electric circuit such as a correlated double sampling circuit or an A / D conversion circuit. There is also quantization noise generated due to a quantization error when converting an analog signal that is a continuous quantity into a digital signal that is a discrete quantity. These are all random noises.

  Of these noises in the image pickup apparatus, those that could not be prevented by each source have been reduced by a digital signal processing circuit. For example, for a dark current or defective pixel having a high temporal correlation, a black image in a light-shielded state is taken in addition to the normal image, and is removed by subtracting the black image from the normal image. In addition, quantization noise can be reduced basically by increasing the resolution of quantization, and has been solved by recent improvements in circuit technology. Other random noises are reduced by limiting the frequency band using a filter and by a technique called NR (noise reduction) using randomness.

JP 2000-105815 A JP-A-2005-354130

Among the noise reduction techniques in these imaging apparatuses, the conventional techniques for reducing random noise have the following problems.
First, when noise is reduced by limiting the frequency band using a filter, the frequency band component of the subject image is simultaneously suppressed together with the frequency band component of the noise. Therefore, the resolution of the image, which is another main element of the image quality of the image obtained by the imaging apparatus, is poor, and it is difficult to achieve both the resolution and the noise.

  On the other hand, as disclosed in Patent Document 1, by using a filter having a nonlinear element, a high-frequency component with a small amplitude is regarded as noise and suppressed, and a high-frequency component with a large amplitude is regarded as a subject image. There is a technique to prevent suppression. However, there has been a problem that it is not possible to cope with fluctuations in the state of noise such as light shot noise that varies depending on the amount of light incident on the image sensor.

  Similarly, as a method for achieving both a sense of resolution and a sense of noise of a subject image, as disclosed in Patent Document 2, a specific subject is recognized, and image processing contents are changed according to the recognition result. Thus, there has been proposed a method for realizing both resolution and noise. However, this method has a problem that it is difficult to deal with a more general subject.

  In the NR technique, the longer the image change information in the time axis direction is, the more noise suppression effect can be obtained. However, in an imaging device that shoots a moving image such as a video camera, it is applied in a range where no afterimage occurs, and there is a limit in a digital still camera in view of the fact that the subject is not strictly stationary. There is a problem that the effect of noise suppression cannot be obtained.

  The present invention has been made in view of such circumstances, and it is an object of the present invention to maintain a sense of resolution related to a subject image and to suppress noise in a captured image.

The imaging device according to the present invention compares an image sensor having a plurality of pixels and image information of a plurality of images obtained by changing the imaging state of the image sensor for each pixel or each pixel region. And signal processing means for performing signal processing on an image obtained by the image sensor, wherein the signal processing means performs signal processing according to a change in the image information obtained as a comparison result by the comparison means. The content of is changed.
The imaging apparatus according to the present invention includes a value of a predetermined frequency band component in a first image captured while focusing on the subject, and the predetermined frequency in a second image captured without focusing on the subject. The content of the signal processing for the first image in the pixel or pixel region in which the change of the band component exceeds the predetermined value, the value of the predetermined frequency band component of the first image, and the second The value of the predetermined frequency band component of the image is changed in content of signal processing for the first image in a pixel or a pixel region in which a change exceeding the predetermined value does not occur.
The imaging method according to the present invention includes an imaging step of obtaining a plurality of images by changing a state of image formation on an imaging element having a plurality of pixels, and image information of the plurality of images obtained by the imaging step. A comparison step for comparing each pixel region or each pixel region, and a signal processing step for performing signal processing on the image obtained by the imaging step. In the signal processing step, the comparison result obtained in the comparison step is obtained. The content of the signal processing is changed according to the change in the image information.
The imaging method according to the present invention includes a value of a predetermined frequency band component in a first image captured while focusing on a subject and the predetermined frequency in a second image captured without focusing on the subject. The content of the signal processing for the first image in the pixel or pixel region in which the change of the band component exceeds the predetermined value, the value of the predetermined frequency band component of the first image, and the second The value of the predetermined frequency band component of the image is changed in content of signal processing for the first image in a pixel or a pixel region in which a change exceeding the predetermined value does not occur.

  According to the present invention, the content of the signal processing is changed according to the change in the image information of a plurality of images obtained by changing the image formation state for each pixel or each region. It is possible to achieve both the suppression of noise and the maintenance of the resolution of the subject image.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  In the embodiments of the present invention described below, attention is paid to the fact that noise in a captured image has not yet occurred at the time of an optical image formed on an image pickup device in an image pickup apparatus. For example, the subject is usually focused. In addition to the image, take a completely blurred image. Hereinafter, an image in which the subject is not focused and completely blurred is referred to as a “defocused image”.

  The defocus image is the same as the focus image in terms of the amount of incident light, and is almost the same as the focus image in the noise state including light shot noise, but can be said to be an image that does not include the high-frequency component of the subject image. The same applies to a motion blur image obtained in a state in which motion blur is forcibly generated using, for example, an optical path changing means for optical camera shake correction. In the following, for convenience of explanation, it is assumed that the motion blurred image is also a kind of defocused image. Further, defocus and motion blur may be generated at the same time.

  Compares the value of a predetermined frequency band component in the captured focus image and defocused image, and determines that the pixel or pixel area where the change in the value of the frequency band component is small is a portion with a small pattern of the subject. Enhance noise suppression processing for images. On the other hand, for a pixel or a pixel region having a large change, a process for increasing the sense of resolution by determining that the pattern of the subject exists is performed on the focus image. In this way, by changing the processing content for the focus image according to the change in the predetermined frequency band components of the focus image and defocus image, both the noise feeling for random noise and the resolution feeling for the subject's picture are realized. To do.

  Further, after subtracting a black image obtained by shielding the image sensor from the focus image or defocus image, comparison of frequency band component values and change of signal processing contents are performed. Thereby, after removing fixed pattern noise such as dark current unevenness and defective pixels, it is possible to realize both a sense of noise with respect to random noise and a sense of resolution with respect to the pattern of the subject.

(First embodiment)
An imaging apparatus according to a first embodiment of the present invention will be described.
FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an operation sequence according to the first embodiment.

  In FIG. 1, 101 is a focus lens, 102 is a zoom lens, 103 is a shift lens, 104 is an iris, and 105 is an image sensor (eg, a CCD image sensor, a CMOS image sensor, etc.) having a plurality of pixels. Here, the shift lens 103 implements a camera shake correction mechanism, and is controlled to move so as to cancel the movement according to the movement of the imaging apparatus. Reference numeral 106 denotes an analog signal processing circuit, 107 denotes a digital signal processing unit, 108 denotes a memory (for example, a large capacity DRAM), 109 denotes a memory recording medium, and 110 denotes a microcomputer (hereinafter referred to as a microcomputer).

  1 are components in the digital signal processing unit 107, 201 is a clamp circuit, 202 is a black image difference circuit, 203 is a YC separation circuit, and 204 is a memory interface circuit (memory IF circuit). is there. The clamp circuit 201 clamps an optical black (optical black) level (hereinafter referred to as an OB level) in the image sensor as a reference for the brightness of the image signal, and corrects the input image signal using the OB level. To do. The black image difference circuit 202 is a black image obtained by photographing in a state where the image sensor is not exposed (light shielding state, non-exposure state) from image data obtained by photographing the image sensor in an exposed state (exposure state). Subtract image data. The YC separation circuit 203 separates the output from the black image difference circuit 202 into a luminance signal component and a color signal component. The memory IF circuit 204 is for reading and writing data from and to the memory 108.

  Reference numeral 205 denotes a first bandpass filter, 206 denotes a first absolute value circuit, 207 denotes a first moving average filter, 208 denotes a second bandpass filter, 209 denotes a second absolute value circuit, and 210 denotes a first absolute value circuit. A moving average filter 2, 211 is a level comparison circuit. 212 is a first delay adjusting circuit, 213 is a first band limiting filter, 214 is a second delay adjusting circuit, 215 is a second band limiting filter, 216 is a weighted addition circuit, 217 is a luminance signal processing circuit, 218 Is a third delay adjustment circuit, and 219 is a color signal processing circuit. 220 is an image format generation circuit, 221 is a JPEG processing circuit, and 222 is a card interface circuit.

With reference to FIG. 1 and FIG. 2, the operation of the imaging apparatus in the first embodiment will be described.
The imaging apparatus according to the first embodiment has, as its operation mode, a first shooting mode, a second shooting mode, a third shooting mode, and a still image processing mode, and sequentially switches between these modes. Perform the action.

  In the first shooting mode, a black image is shot with the image sensor 105 in a light shielding state (non-exposure state). In the second shooting mode, the image sensor 105 is exposed and the main image (focus image) focused on the subject is shot. In the third shooting mode, the image sensor 105 is exposed and the subject is focused. Take a defocused image. In the still image processing mode, output still image data is generated using the black image data, the main image data, and the defocused image data obtained by the operations in the first to third imaging modes.

The operation will be described below based on the operation sequence shown in FIG.
The microcomputer 110 performs the first shooting mode setting on the digital signal processing unit 107 (step S1), and then controls the iris 104 to be completely closed (step S2) to drive the image sensor 105. (Step S3). As a result, a black image is captured with the image sensor 105 in a light-shielded state. The output (analog black image signal) of the image sensor 105 in this black image shooting is subjected to correlated double sampling and analog-digital conversion processing in an analog signal processing circuit 106, and then subjected to digital signal processing as a digital black image signal. Input to the unit 107.

  The digital signal processing unit 107 performs predetermined processing on the digital black image signal with the OB level being constant in the clamp circuit 201, and the obtained black image data is transferred to the black image data area ( (Not shown) (step S4). This black image data includes fixed pattern noise due to dark current and fixed pattern noise due to defective pixels.

  Next, the microcomputer 110 performs the second shooting mode setting (step S5), and then performs various controls such as zoom control and focus control to adjust zoom, focus, exposure, white balance, and the like ( Step S6). After completing these adjustments, the microcomputer 110 drives the image sensor 105 to capture the main image with the image sensor 105 in an exposure state and a state in which the image sensor 105 is focused (step S7). The output (analog main image signal) of the image sensor 105 in this main image photographing is processed in the analog signal processing circuit 106 in the same manner as the black image signal described above, and then the digital signal processing unit 107 as a digital main image signal. Is input.

  The digital signal processing unit 107 performs predetermined processing on the digital main image signal with the OB level being constant in the clamp circuit 201. Further, the black image difference circuit 202 reads out and subtracts the black image data stored in the black image data area of the memory 108 from the main image data processed by the clamp circuit 201 via the memory IF circuit 204. To do. As a result, dark pattern current and fixed pattern noise due to defective pixels in the main image are suppressed.

  Thereafter, the main image data in which the fixed pattern noise is suppressed as described above is separated into a luminance signal component and a color signal component by the YC separation circuit 203. The main image luminance data, which is a luminance signal component, is stored in a main image luminance data area (not shown) of the memory 108 via the memory IF circuit 204. Similarly, the main image color data, which is a color signal component, is stored in a main image color data area (not shown) of the memory 108 via the memory IF circuit 204 (step S8).

  Next, the microcomputer 110 performs the third shooting mode setting (step S9), and changes the position of at least one of the focus lens 101 and the shift lens 103 by a predetermined amount (step S10). This blurs the optical image formed on the light receiving surface of the image sensor 105 and creates a state in which the subject image is erased (the subject image becomes unclear).

  In this state, the microcomputer 110 drives the image sensor 105 to capture a defocused image in which the image sensor 105 is in an exposed state and is not focused on the subject (step S11). The output (analog defocus image signal) of the image sensor 105 in this defocus image shooting is processed in the analog signal processing circuit 106 in the same manner as described above, and is then sent to the digital signal processing unit 107 as a digital defocus image signal. Entered.

  The digital signal processing unit 107 performs predetermined processing on the digital defocused image signal with the OB level being constant in the clamp circuit 201. Further, the black image difference circuit 202 reads the black image data stored in the black image data area of the memory 108 from the defocused image data obtained by the processing by the clamp circuit 201 via the memory IF circuit 204. Subtract. This suppresses dark pattern current and fixed pattern noise due to defective pixels in the defocused image.

  Thereafter, only the luminance signal component is extracted by the YC separation circuit 203 from the defocus image data in which the fixed pattern noise due to the dark current and the defective pixel is suppressed. The extracted defocus image luminance data is stored in a defocus image luminance data area (not shown) of the memory 108 via the memory IF circuit 204 (step S12).

  Next, the microcomputer 110 performs still image processing mode setting (step S13). Subsequently, the digital signal processing unit 107 reads the main image luminance data, the main image color data, and the defocused image luminance data stored in each data area of the memory 108 via the memory IF circuit 204 (step S14). ). Next, the digital signal processing unit 107 executes random noise suppression processing using the read main image luminance data, main image color data, and defocused image luminance data (step S15).

Hereinafter, the random noise suppression process shown in step S15 will be described.
The main image luminance data read from the memory 108 is input to the first band-pass filter 205, the first delay adjustment circuit 212, and the second delay adjustment circuit 214. Input to the delay adjustment circuit 218. Further, the defocused image luminance data read from the memory 108 is input to the second band pass filter 208.

  Here, the first and second band pass filters 205 and 208 have the same frequency characteristics, and include frequency components including random noise such as shot noise included in the main image luminance data and the defocused image luminance data. It has a frequency characteristic that can be extracted mainly.

  The first band pass filter 205 to which the main image luminance data is input extracts a frequency component including random noise (hereinafter referred to as a first image component) of the main image luminance data. The first absolute value circuit 206 extracts the size data of the first image component extracted by the first band pass filter 205. The first moving average filter 207 spatially local averages the size data of the first image component extracted by the first absolute value circuit 206.

  Similarly, the second band pass filter 208 to which the defocus image luminance data is input extracts a frequency component (hereinafter referred to as a second image component) including random noise of the defocus image luminance data. The second absolute value circuit 209 extracts the size data of the second image component extracted by the second band pass filter 208. The second moving average filter 210 spatially local averages the size data of the second image component extracted by the second absolute value circuit 209.

  The level comparison circuit 211 compares the size of the first image component and the size of the second image component that are locally averaged by the first and second moving average filters 207 and 210, respectively. Here, since the first image component is extracted from the main image luminance data, the size locally changes in accordance with the pattern of the subject in the main image. On the other hand, since the second image component is extracted from the defocus image luminance data, the size hardly changes in the entire screen. Accordingly, a portion in which the size of the first image component is equal to the size of the second image component can be determined as a portion where the subject image does not exist (almost), and the size of the first image component is the second image. A portion that is clearly larger than the size of the component can be determined as a portion where the pattern of the subject exists.

  Therefore, the level comparison circuit 211 generates the mixture ratio data K based on the difference value D between the size of the first image component and the size of the second image component, and outputs the generated mixture ratio data K to the weighted addition circuit 216. To do. As shown in FIG. 3, the mixing ratio data K is 0 when the difference value D is equal to or smaller than the first predetermined value V1, 1 when the difference value D is equal to or larger than the second predetermined value V2, When it is intermediate between the first predetermined value V1 and the second predetermined value V2, it is set to an arbitrary value between 0 and 1 obtained by linear interpolation. The mixing ratio data K may be given other arbitrary characteristics. Note that the level comparison circuit 211 compares the size of the first image component with the size of the second image component for each pixel of the image or for each predetermined pixel area.

  On the other hand, the first delay adjustment circuit 212 and the second delay adjustment circuit 214 to which the main image luminance data is input delays the main image luminance data for a predetermined time, and thereby the first band limiting filter 213 and the second band limiting circuit. Output to the filter 215. That is, with respect to the luminance data, the delay caused by the bandpass filter 205 (208), the absolute value circuit 206 (209), the moving average filter 207 (210) and the level comparison circuit 211, the delay adjustment circuit 212 (214) and the band limiting filter The delay amount is adjusted so that the delay caused by 213 (215) coincides with the weighted addition circuit 216.

  The first band limiting filter 213 is constituted by, for example, a low-pass filter, and strongly suppresses a random noise component that is a first image component included in the main image luminance data. The second band limiting filter 215 is configured by, for example, a low-pass filter, and weakly suppresses the random noise component that is the first image component included in the main image luminance data. Therefore, the output Y1 of the first band limiting filter 213 is luminance data in which random noise is strongly suppressed while sacrificing the resolution in the image, and the output Y2 of the second band limiting filter 215 decreases the resolution. It can be said that the luminance data leaves some random noise.

The weighted addition circuit 216 uses the mixing ratio data K shown in FIG. 3 and the outputs Y1 and Y2 of the band limiting filters 213 and 215 to generate weighted addition luminance data Yout by the following equation.
Yout = (1−K) × Y1 + K × Y2

  Thereby, the weighted addition luminance data Yout becomes luminance data related to the main image obtained by locally comparing certain frequency components of the main image data and the defocused image data and performing signal processing according to the comparison result. Specifically, in the main image, the noise suppression process is strengthened for the part where the subject's picture does not exist, and the random noise is strongly suppressed, and the high-frequency component is emphasized for the part where the subject's picture exists. The luminance data retains the components and maintains the resolution. The weighted addition luminance data Yout is subjected to predetermined luminance signal processing by the luminance signal processing circuit 217 and output to the image format generation circuit 220.

  The main image color data read from the memory 108 is adjusted by the third delay adjustment circuit 218 with the delay in the series of signal processing related to the luminance data described above. The color signal processing circuit 219 performs predetermined color signal processing such as color separation, matrix calculation, and white balance, and outputs the result to the image format generation circuit 220.

  The image format generation circuit 220 performs a predetermined uncompressed image based on the weighted addition luminance data Yout processed by the luminance signal processing circuit 217 and the main image color data processed by the color signal processing circuit 219. Generate data. The generated uncompressed image data is stored in a camera processed image data area (not shown) of the memory 108 via the memory IF circuit 204.

  The image data stored in the camera-processed image data area is then subjected to compression processing or the like by the JPEG processing circuit 221 as necessary, and recorded as compressed image data on the memory recording medium 109 via the card interface circuit 222. it can.

  According to the first embodiment, the magnitudes of frequency components including random noise in the captured focus image and the defocused image are compared, and a portion with a small difference is subjected to noise suppression processing as a portion where the subject image does not exist. Strongly suppresses random noise components. On the other hand, the part where the difference is large is considered as the part where the subject's picture exists, weakening the noise suppression processing to suppress the random noise component weakly, leaving the high frequency component of the picture, maintaining the resolution feeling, and the relative resolution feeling Strengthen.

  As a result, a camera-processed image that is finally obtained is an image in which the resolution of the pattern is maintained in the portion where the subject pattern is present, and the random noise is strongly suppressed in the portion where the subject pattern is not present. Of course, the fixed pattern noise is also removed from the camera processed image. Therefore, for example, the S / N of a flat portion with a small amount of high-frequency components such as the sky and human skin is improved, and for example, a fine pattern portion such as a tree or a contour is maintained in a sense of resolution, and noise in a captured image, particularly random noise, is maintained. It is possible to satisfactorily achieve both suppression and maintaining the resolution of the subject's picture.

  In addition, the noise part and the pattern part of the subject can be identified each time shooting without changing the amount of incident light, so that the noise state varies depending on the amount of incident light, such as light shot noise. Even if it responds, it can respond appropriately and does not cause a problem. Therefore, there is an effect that it is possible to cope with a wider photographing state than the noise removal technique using the conventional nonlinear filter.

  In this embodiment, it is possible to deal with any subject, not just a specific subject, and it is possible to deal with a more general subject. In addition, since it is not necessary to perform processing in the time axis direction on an image to be displayed or recorded, noise can be sufficiently suppressed compared to the conventional NR technique in which an afterimage or the like may occur, which is effective.

(Second Embodiment)
Next, an imaging apparatus according to the second embodiment of the present invention will be described.
FIG. 4 is a block diagram illustrating a configuration example of the imaging apparatus according to the second embodiment. FIG. 5 is a diagram illustrating an operation sequence according to the second embodiment.

  In FIG. 4, 301 to 310 are the same as the respective components 101 to 110 shown in FIG. 4, 401, 402, and 404 to 416 are components in the digital signal processing unit 307, 401 is a clamp circuit, 402 is a black image difference circuit, and 404 is a memory interface circuit (memory IF circuit). The clamp circuit 401, the black image difference circuit 402, and the memory IF circuit 404 correspond to the clamp circuit 201, the black image difference circuit 202, and the memory IF circuit 204 shown in FIG.

  Reference numeral 405 denotes a bandpass filter, 406 denotes an absolute value circuit, 407 denotes a moving average filter, and 408 denotes a level comparison circuit. 409 is a first delay adjustment circuit, 410 is a noise removal filter, 411 is a second delay adjustment circuit, 412 is a third delay adjustment circuit, 413 is an aperture addition circuit, 414 is a weighted addition circuit, 415 is a JPEG processing circuit Reference numeral 416 denotes a card interface circuit.

With reference to FIGS. 4 and 5, the operation of the imaging apparatus in the second embodiment will be described.
The imaging apparatus according to the second embodiment has, as its operation mode, a first shooting mode, a second shooting mode, a third shooting mode, a first black image difference processing mode, and a second black image difference processing mode. , A random noise extraction mode, and an output still image processing mode. The image pickup apparatus according to the second embodiment operates while sequentially changing these modes.

  In the first shooting mode, the image sensor 305 is in an exposure state, and a main image is shot with the subject focused. In the second shooting mode, a black image is shot with the image sensor 305 in a light-shielded state. In the third shooting mode, the defocus image is shot with the image sensor 305 in the exposure state.

  In the first black image difference processing mode, first fixed pattern noise-suppressed image data is generated using main image data and black image data obtained by the operations in the first and second imaging modes. In the second black image difference processing mode, second fixed pattern noise-suppressed image data is generated using the black image data and the defocused image obtained by the operations in the second and third shooting modes.

  In the random noise extraction mode, random noise component image data is generated from the second fixed pattern noise suppression image data. In the output still image processing mode, output still image data is generated from the first fixed pattern noise-suppressed image data and random noise component image data.

The operation will be described below based on the operation sequence shown in FIG.
The microcomputer 310 performs the first shooting mode setting on the digital signal processing unit 307 (step S21), and then performs various controls such as zoom control and focus control, and performs zoom, focus, exposure, white balance, and the like. Is adjusted (step S22). After completing these adjustments, the microcomputer 310 drives the image sensor 305 to capture the main image with the image sensor 305 in an exposed state and in a state in which the subject is focused (step S23). The output (analog main image signal) of the image pickup device 305 in this main image photographing is subjected to correlated double sampling and analog-digital conversion processing in an analog signal processing circuit 306, and then subjected to digital signal processing as a digital main image signal. This is input to the unit 307.

  The digital signal processing unit 307 performs predetermined processing on the digital main image signal with the OB level being constant in the clamp circuit 401, and the obtained main image data is transferred to the main image data area ( (Step S24). This main image data includes random noise, fixed pattern noise due to dark current, and fixed pattern noise due to defective pixels.

  Next, the microcomputer 310 performs the second shooting mode setting (step S25), and then controls the iris 304 to be in a completely closed state (step S26), drives the image sensor 305, and drives the image sensor 305. A black image is taken in a non-exposed state (step S27). The output (analog black image signal) of the image pickup element 305 in this black image shooting is processed in the analog signal processing circuit 106 in the same manner as the analog main image signal described above, and is then converted into a digital signal processing unit as a digital black image signal. 307 is input.

  The digital signal processing unit 307 performs predetermined processing on the digital black image signal with the OB level being constant in the clamp circuit 401, and the obtained black image data is transferred to the black image data area ( (Not shown) (step S28). This black image data also includes random noise, fixed pattern noise due to dark current, and fixed pattern noise due to defective pixels.

  The iris 304 is completely closed in step S26, and the position of the focus lens 301 is changed by a predetermined amount until the black image data is stored in the black image data area of the memory 308 in step S28 (step S29). As a result, the optical image formed on the light receiving surface of the image sensor 305 is blurred to create a state in which light is incident but the subject image is optically erased.

  Next, the microcomputer 310 performs the third shooting mode setting (step S30), returns the iris 304 to the state at the time of setting the first shooting mode (step S31), and drives the image sensor 305 (step S32). . The iris 304 returns when the first shooting mode is set, that is, in the state of the same optical aperture as the main image shooting in the first shooting mode, and in step S29, the optical image on the light receiving surface of the image sensor 305 is defocused in advance. ing. Therefore, the captured image obtained in step S32 is a defocused image in which the image sensor 305 is in an exposure state and is not in focus on the subject. The output (analog defocus image signal) of the image sensor 305 in this defocus image shooting is processed in the analog signal processing circuit 306 in the same manner as described above, and then is output to the digital signal processing unit 307 as a digital defocus image signal. Entered.

  The digital signal processing unit 307 performs predetermined processing on the digital defocused image signal with the OB level being constant in the clamp circuit 401. The obtained defocused image data is stored in a defocused image data area (not shown) of the memory 308 via the memory IF circuit 404 (step S33). This defocused image data also includes random noise, fixed pattern noise due to dark current, and fixed pattern noise due to defective pixels.

  Next, the microcomputer 310 performs the first black image difference processing mode setting (step S34). Subsequently, the digital signal processing unit 307 passes the main image data stored in the main image data area of the memory 308 and the black image data stored in the black image data area of the memory 308 via the memory IF circuit 404. Read out. Then, the black image difference circuit 402 subtracts the black image data from the read main image data, and the obtained first difference image data is sent to the first difference image data area of the memory 308 via the memory IF circuit 404. (Not shown) (step S35). Accordingly, the first differential image data is image data in which fixed pattern noise due to dark current and fixed pattern noise due to defective pixels are removed, but random noise is not statistically removed.

  Next, the microcomputer 310 performs the second black image difference processing mode setting (step S36). Subsequently, the digital signal processing unit 307 converts the defocus image data stored in the defocus image data area of the memory 308 and the black image data stored in the black image data area of the memory 308 into the memory IF circuit 404. Read through. Further, the black image difference circuit 402 subtracts the black image data from the read defocus image data, and the obtained second difference image data is supplied to the second difference image data in the memory 308 via the memory IF circuit 404. Store in an area (not shown) (step S37). Therefore, the second differential image data is image data in which fixed pattern noise due to dark current and fixed pattern noise due to defective pixels are removed, but random noise is not statistically removed.

  Next, the microcomputer 310 performs random noise extraction mode setting (step S38). Subsequently, the digital signal processing unit 307 reads out the second difference image data stored in the second difference image data area of the memory 308 via the memory IF circuit 404 and inputs it to the bandpass filter 405.

  The band-pass filter 405 to which the second difference image data is input extracts a frequency component mainly including random noise (hereinafter referred to as a second image component) from the second difference image data. The absolute value circuit 406 calculates second image component absolute value data representing the size of the second image component extracted by the bandpass filter 405, and the moving average filter 407 calculates the calculated second image component absolute value. The second image component evaluation data is calculated by spatially averaging the data locally.

  The calculated second image component evaluation data is written into a second image component evaluation value data area (not shown) of the memory 308 via the memory IF circuit 404 (step S39). Since the second image component evaluation data is calculated based on the defocused image, it represents the local spatial distribution of the random noise that has not been removed.

  Next, the microcomputer 310 performs output still image processing mode setting (step S40). Subsequently, the digital signal processing unit 307 reads out the first difference image data stored in the first difference image data area of the memory 308 via the memory IF circuit 404. In addition, the digital signal processing unit 307 reads out the second image component evaluation data stored in the second image component evaluation value data area of the memory 308 via the memory IF circuit 404 (step S41).

  The first difference image data read from the memory 308 is input to the band-pass filter 405, the first delay adjustment circuit 409, the second delay adjustment 411, and the third delay adjustment circuit 412 to obtain the second image. The component evaluation data is input to the level comparison circuit 408.

  The band-pass filter 405 to which the first difference image data is input extracts frequency components (hereinafter referred to as the first image component) mainly representing random noise and the subject image from the first difference image data. Thereafter, in the same manner as in the random noise extraction mode described above, the first image component evaluation data is calculated by the absolute value circuit 406 and the moving average filter 407 and input to the level comparison circuit 408 (step S42). Since the first image component evaluation data is calculated based on the main image, the first image component evaluation data represents a local spatial distribution of the size of the frequency component due to the pattern of the subject superimposed on the random noise that has not been removed. It has become.

  The statistical random noise amount included in the first image component evaluation data is substantially equal to the statistical random noise amount included in the second image component evaluation data. Therefore, in the level comparison circuit 408, the first image component evaluation data and the second image component evaluation data are compared to determine whether or not a high frequency component due to the subject image exists in the main image locally. It becomes possible.

  Therefore, the level comparison circuit 408 generates the mixture ratio data K1 and K2 and the mixture ratio selection flag Kf based on the difference value D between the first image component evaluation data and the second image component evaluation data, and the weighted addition circuit 414. (Step S43). The mixing ratio data K1, K2 and the mixing ratio selection flag Kf will be described later. Note that the comparison between the first image component evaluation data and the second image component evaluation data in the level comparison circuit 408 is performed for each pixel of the image or for each predetermined pixel region.

  The first differential image data input to the first delay adjustment circuit 409, the second delay adjustment circuit 411, and the third delay adjustment circuit 412, respectively, adjusts the amount of delay caused by pipeline processing using flip-flops. Is done. The weighting addition circuit 414 determines the timing of each data from the noise removal filter 410, the aperture addition circuit 413, and the memory IF circuit 404 and the mixing ratio data K1, K2 from the level comparison circuit 408 and the mixing ratio selection flag Kf. Adjusted to match. The noise removal filter 410 performs band limiting processing and has frequency characteristics that strongly suppress random noise components. The aperture adding circuit 413 performs so-called aperture correction processing and emphasizes the contour component of the subject.

  Accordingly, the weighted addition circuit 414 receives the first processed image data S1 in which the random noise of the first difference image data is significantly reduced via the first delay adjustment circuit 409 and the noise removal filter 410. . Further, the weighted addition circuit 414 receives the second processed image data S2 including the random noise of the first difference image data as it is via the second delay adjustment circuit 411. In addition, the weighted addition circuit 414 receives the third processed image data S3 obtained by amplifying the high-frequency component representing the subject outline of the first difference image data via the third delay adjustment circuit 412 and the aperture addition circuit 413. Is entered. Further, the mixture ratio data K1 and K2 and the mixture ratio selection flag Kf are input to the weighted addition circuit 414.

  As shown in FIG. 6, the mixing ratio data K1 is 0 when the difference value D is less than or equal to the first predetermined value V1, 1 when the difference value D is greater than or equal to the second predetermined value V2, and the difference value D is first. When it is intermediate between the predetermined value V1 and the second predetermined value V2, it is set to an arbitrary value between 0 and 1 obtained by linear interpolation. As shown in FIG. 7, the mixing ratio data K2 is 0 when the difference value D is equal to or smaller than the third predetermined value V3, 1 when the difference value D is equal to or larger than the fourth predetermined value V4, and the difference value D is equal to the third value. When it is intermediate between the predetermined value V3 and the fourth predetermined value V4, it is set to an arbitrary value between 0 and 1 obtained by linear interpolation. The mixing ratio data K1 and K2 may be given other arbitrary characteristics.

  The mixing ratio selection flag Kf is set to 0 when the difference value D is equal to or smaller than the second predetermined value V2, and is set to 1 when the difference value D is equal to or larger than the second predetermined value V2. Alternatively, the mixture ratio selection flag Kf may be set to 0 when the difference value D is equal to or smaller than the third predetermined value V3, and may be set to 1 when equal to or greater than the third predetermined value V3.

The weighted addition circuit 412 generates the final image data Smix by the following expression using the first to third processed image data S1, S2, S3, the mixture ratio data K1, K2, and the mixture ratio selection flag Kf. (Step S44).
Smix = (1−K1) × S1 + K1 × S2 (where Kf = 0)
Smix = (1−K2) × S2 + K2 × S3 (where Kf = 1)

  As a result, the final image data Smix is subjected to image processing according to the comparison result by comparing the local frequency components of the main image data and the defocused image data. Specifically, in the main image, the noise suppression process is strengthened for the part where the subject's picture does not exist, and the random noise is strongly suppressed.For the part where the subject's picture exists, the high-frequency component of the picture is increased to enhance the contour. You can hang it. Further, the original image that is not subjected to noise suppression or contour enhancement can be used for a portion in an intermediate state as the strength of the subject image, and a very natural final image can be generated.

  The generated final image data is stored in a camera processing image data area (not shown) of the memory 308 via the memory IF circuit 404 (step S45). Thereafter, the JPEG processing circuit 415 can perform compression processing or the like as necessary, and can record the compressed image data on the memory recording medium 309 via the card interface circuit 416.

  According to the second embodiment, the same effect as that of the first embodiment described above can be obtained, and as the final image data, the sense of resolution of the pattern is maintained in the portion where the pattern of the subject exists, and there is no pattern of the subject. In the portion, an image in which random noise is strongly suppressed can be obtained. Of course, it goes without saying that fixed pattern noise is also removed from the final image data.

  The above-described embodiment is suitable for use in, for example, a digital still camera or a digital video camera.

  It should be noted that the above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. . That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

It is a figure which shows the structural example of the imaging device by 1st Embodiment. It is a figure which shows the operation | movement sequence in 1st Embodiment. It is a figure which shows the relationship between the difference value (D) of the magnitude | size of the 1st image component and 2nd image component in 1st Embodiment, and a mixing ratio (K). It is a figure which shows the structural example of the imaging device by 2nd Embodiment. It is a figure which shows the operation | movement sequence in 2nd Embodiment. It is a figure which shows the relationship between the difference value (D) of the 1st image component evaluation data and 2nd image component evaluation data in 2nd Embodiment, and a mixture ratio (K1). It is a figure which shows the relationship between the difference value (D) of 1st image component evaluation data and 2nd image component evaluation data in 2nd Embodiment, and a mixture ratio (K2).

Explanation of symbols

101, 301 Focus lens 103, 303 Shift lens 104, 304 Iris 105, 305 Image sensor 107, 307 Digital signal processor 108, 308 Memory 110, 310 Microcomputer (microcomputer)
205, 208, 405 Band pass filter 206, 209, 406 Absolute value circuit 207, 210, 407 Moving average filter 211, 408 Level comparison circuit 213, 215 Band limiting filter 410 Noise removal filter 413 Aperture addition circuit 216, 414 Weighted addition circuit

Claims (14)

An imaging device having a plurality of pixels;
Comparison means for comparing image information of a plurality of images obtained by changing the state of image formation on the image sensor for each pixel or each pixel region;
Signal processing means for performing signal processing on an image obtained by the imaging device,
The image processing apparatus, wherein the signal processing means changes the content of signal processing in accordance with a change in the image information obtained as a comparison result by the comparison means.   The imaging apparatus according to claim 1, wherein signal processing corresponding to a change in the image information is performed for each pixel or each pixel region to which the image information is compared. The comparison means determines whether the target pixel or pixel region is a pixel or pixel region related to the subject image by comparing the image information,
The signal processing means performs first signal processing on the pixel or the pixel area that is determined not to be the pixel or the pixel area related to the subject image by the comparison means, and is determined to be the pixel or the pixel area related to the subject image. The imaging apparatus according to claim 1, wherein the second signal processing is performed on the pixel or the pixel region.   The signal processing means performs first signal processing in which noise suppression processing is strengthened in a pixel or a pixel region in which the change in the image information is smaller than a predetermined change amount, and the pixel in which the change in the image information is larger than the change amount. The image pickup apparatus according to claim 1, wherein the second signal processing with enhanced high-frequency component enhancement processing is performed in the pixel region.   The imaging apparatus according to claim 4, wherein the first signal processing includes band limitation processing, and the second signal processing includes aperture correction processing.   6. The image information according to claim 1, wherein the image information is a value of a predetermined frequency band component in each image or a value obtained by processing the value of the predetermined frequency band component. The imaging device described in 1.   The imaging apparatus according to claim 1, wherein the imaging state is a focus state of an optical image on the imaging element.   The imaging apparatus according to claim 1, wherein the imaging state is a motion blurring state of an optical image on the imaging element.   The plurality of images are a plurality of images obtained by subtracting a black image captured with the image sensor being in a light-shielded state from a plurality of captured images captured while changing the imaging state of the image sensor. The imaging device according to any one of claims 1 to 8.   10. The signal processing unit according to claim 1, wherein the signal processing unit performs signal processing only on a part of a plurality of images obtained by changing a state of image formation on the imaging element. The imaging apparatus of Claim 1.   The predetermined frequency band component value in the first image shot with the subject focused and the predetermined frequency band component value in the second image shot without the subject focused Content of signal processing for the first image in a pixel or pixel region in which a change exceeding 1 is generated, a value of the predetermined frequency band component of the first image, and the predetermined frequency band component of the second image An image pickup apparatus characterized by changing the content of signal processing on the first image in a pixel or a pixel region in which a change exceeding the predetermined value does not occur in the value of. The first image is an image obtained by subtracting a black image captured with the imaging element in a light-shielded state from a captured image captured with the subject focused.
The imaging according to claim 11, wherein the second image is an image obtained by subtracting a black image captured with the imaging element being shielded from a captured image captured without focusing on the subject. apparatus. An imaging process for obtaining a plurality of images by changing the state of image formation on an imaging device having a plurality of pixels;
A comparison step of comparing the image information of the plurality of images obtained by the imaging step for each pixel or each pixel region;
A signal processing step of performing signal processing on the image obtained by the imaging step,
In the signal processing step, the content of the signal processing is changed according to a change in the image information obtained as a comparison result in the comparison step.   The predetermined frequency band component value in the first image shot with the subject focused and the predetermined frequency band component value in the second image shot without the subject focused Content of signal processing for the first image in a pixel or pixel region in which a change exceeding 1 is generated, a value of the predetermined frequency band component of the first image, and the predetermined frequency band component of the second image The imaging method is characterized in that the content of signal processing for the first image in a pixel or a pixel region in which a change exceeding the predetermined value does not occur is changed.

Images