JP4696812B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus using a solid-state imaging element having a plurality of different photoelectric conversion characteristics, and more particularly to an imaging apparatus capable of removing noise from a captured image obtained by the solid-state imaging element.

  2. Description of the Related Art Conventionally, an imaging apparatus such as a digital camera includes a predetermined solid-state imaging device (imaging sensor), and acquires a captured image by imaging using the solid-state imaging device. In general, a captured image (image signal) includes a noise component. However, with the recent demand for higher image quality, it is required to suitably remove the noise component. By the way, some solid-state image sensors are composed of a plurality of different photoelectric conversion characteristics. For example, a linear photoelectric conversion characteristic (linear characteristic) in which an electric signal is linearly converted with respect to an incident light amount and output, and a logarithm of an electric signal that is converted logarithmically with respect to the incident light amount and output. A solid-state imaging device (referred to as a linear log sensor) having a photoelectric conversion property composed of a photoelectric conversion property (referred to as a logarithmic property) is known. A photographed image by a linear log sensor having the linear characteristic and logarithmic characteristic (referred to as linear / logarithmic characteristic) is referred to as a linear / logarithmic image.

As a method of performing the above-described noise removal for an imaging apparatus including a plurality of solid-state imaging elements having different photoelectric conversion characteristics as described above, for example, Patent Document 1 discloses a parameter (coring coefficient, etc.) in a region having different photoelectric conversion characteristics. A technique is disclosed in which noise is removed using different parameters. For example, in Patent Document 2, noise removal is performed before gradation conversion processing of a captured image, and an illumination component and a reflectance component are extracted from the image after the noise removal, and gradation conversion is performed (the illumination component is compressed). By doing so, a technique of performing DR compression that compresses the entire dynamic range (DR) while preserving local contrast is disclosed.
JP 2005-110235 A JP 2001-275015 A

  When tone conversion is performed on an image obtained by imaging using an imaging device having a plurality of photoelectric conversion characteristics such as the above-described solid-state imaging device, in a pixel in which the degree of image data expansion (data compression) by the tone conversion is large Not only the contrast component but also the noise component is expanded, and noise in a region with a large expansion degree becomes conspicuous only by performing uniform noise removal.

  In Patent Document 1, as described above, noise is reduced compared with the case of uniform noise removal by changing parameters in regions with different photoelectric conversion characteristics and performing noise removal using these different parameters. The noise component that was noticeable due to the difference in the degree of expansion due to the gradation conversion could not be removed. Further, in Patent Document 2, although the illumination component and the reflectance component are extracted for the gradation conversion process, only uniform noise removal is performed before the gradation conversion. Different noise removal processes are not performed, and noise components that are conspicuous due to differences in the degree of expansion due to gradation conversion have not been completely removed.

  The present invention has been made in view of the above circumstances, and it is possible to prevent noise from being noticeable due to a difference in the degree of expansion due to gradation conversion, and thus to obtain a high-quality captured image. The purpose is to provide.

According to a first aspect of the present invention, there is provided an imaging apparatus having a plurality of solid-state imaging elements having different photoelectric conversion characteristics and a noise removal amount of the captured image according to a degree of expansion in gradation conversion of the captured image by the solid-state imaging element. A noise removing unit that removes noise from the captured image by changing a parameter related to the captured image , the noise removing unit serving as a parameter related to the noise removal amount according to a luminance level corresponding to the degree of expansion of the captured image. Coefficient calculating means for calculating a coring coefficient; and coring processing means for performing a coring process for removing a noise component from the photographed image based on the coring coefficient calculated by the coefficient calculating means. In the different photoelectric conversion characteristics, the coefficient calculation means is capable of coring differently between the high luminance side characteristic and the low luminance side characteristic. Calculating a number, and calculates, based coring coefficients of the high-luminance side to the decompression degree in the grayscale conversion of the captured image.

  According to the above configuration, the noise removal unit performs noise removal on the photographed image by changing the parameter relating to the noise removal amount of the photographed image in accordance with the degree of expansion in gradation conversion of the photographed image by the solid-state imaging device. .

  According to this, since the degree of expansion at the time of gradation conversion changes according to the luminance level of the photographed image, the coefficient calculation means according to the luminance level of the photographed image, that is, for example, an image of a low luminance part or a high luminance The coring coefficient as a parameter relating to the amount of noise removal is calculated according to whether the image is a part image, and the noise component is removed from the captured image based on the coring coefficient calculated by the coefficient calculating means by the coring processing means. Coring processing is performed.

  In the above configuration, the noise removing unit further includes a low-frequency component extracting unit that extracts a low-frequency component from the captured image, and a high-frequency component extracting unit that extracts a high-frequency component from the captured image, and the coefficient calculation The means calculates the coring coefficient according to the level of the low frequency component corresponding to the luminance level, and the coring processing means is based on the coring coefficient calculated according to the level of the low frequency component. It is preferable to perform a coring process on the high-frequency component.

  According to this, the low frequency component is extracted from the photographed image by the low frequency component extracting means, and the high frequency component is extracted from the photographed image by the high frequency component extracting means. Then, the coring coefficient is calculated according to the level of the low frequency component corresponding to the luminance level by the coefficient calculating means, and the high frequency component based on the coring coefficient calculated according to the level of the low frequency component by the coring processing means. A coring process is performed on.

  Further, in the above configuration, the apparatus further includes an illumination component extraction unit that extracts an illumination component from the captured image, and a reflectance component extraction unit that extracts a reflectance component from the captured image, and the coefficient calculation unit includes the luminance level. The coring coefficient is calculated according to the level of the illumination component corresponding to the coring coefficient, and the coring processing means coring the reflectance component based on the coring coefficient calculated according to the level of the illumination component. It is preferable to carry out the treatment.

  According to this, the illumination component is extracted from the photographed image by the illumination component extracting means, and the reflectance component is extracted from the photographed image by the reflectance component extracting means. Then, the coefficient calculating means calculates a coring coefficient according to the level of the illumination component corresponding to the luminance level, and the coring processing means calculates the reflectance based on the coring coefficient calculated according to the level of the illumination component. A coring process is performed on the components.

  In the above-described configuration, in the plurality of different photoelectric conversion characteristics, the coefficient calculation means may use the coring coefficient for the high luminance side characteristic as the coring coefficient so that the noise amount after the gradation conversion is a low luminance part in the high luminance side characteristic. It is preferable to calculate a coring coefficient that is equal between the high luminance portion and the high luminance portion.

  According to this, the coring coefficient is calculated by the coefficient calculating means so that the noise amount after gradation conversion is equal in the low luminance part and the high luminance part in the high luminance side characteristic.

  In the above configuration, the solid-state imaging device has a linear photoelectric conversion characteristic in which an electric signal is linearly converted with respect to an incident light amount and is output, and an electric signal is logarithmically converted with respect to the incident light amount. It can have a photoelectric conversion characteristic composed of an output logarithmic photoelectric conversion characteristic.

  Further, in the above configuration, the solid-state imaging device has a first linear photoelectric conversion characteristic that is output by linearly converting an electrical signal with respect to an incident light amount, the first linear photoelectric conversion characteristic, and a graph. It can have a photoelectric conversion characteristic composed of a second linear photoelectric conversion characteristic having a different inclination.

  The above configuration further includes a predetermined information processing device configured to be able to exchange captured image data with the imaging device main body using a predetermined data transfer unit, and the information processing device removes the noise from the captured image. It can be performed.

  According to this, noise removal is performed on the photographed image by the predetermined information processing apparatus configured to be able to exchange the captured image data with the imaging apparatus main body using the predetermined data transmission / reception means.

  According to the imaging device of the first aspect, in noise removal, it is possible to change the amount of noise removal according to the degree of expansion in gradation conversion (set different noise removal amount according to the degree of expansion). For example, the noise (noise component) should not be noticeable due to the difference in the degree of expansion, for example, the noise is conspicuous in the image of the high luminance part with a high degree of expansion compared to the image of the low luminance part with a low degree of expansion. As a result, a high-quality captured image can be obtained.

According to the imaging apparatus of the first aspect , the general noise removal technique of performing the coring process by the coring processing unit based on the coring coefficient calculated by the coefficient calculating unit is used, that is, by a simple configuration. Therefore, it is possible to easily perform noise removal in which the amount of noise removal is changed according to the degree of expansion.

According to the imaging device of the second aspect of the present invention, the coring coefficient is calculated using the information (level information) of the low frequency component extracted from the photographed image using the low frequency component extraction means such as a general low pass filter. The calculation can be easily performed, and the noise removal based on the calculated coring coefficient can be easily performed by performing the coring process on the high-frequency component extracted from the photographed image.

According to the imaging device of the third aspect , information on the illumination component and the reflectance component is used for noise removal, that is, a wide dynamic range image is captured using a sensor having a plurality of photoelectric conversion characteristics as in the present invention. Since the illumination component and the reflectance component generated in the process of the DR compression process, which is a gradation conversion process suitably used for the imaging apparatus, are used, noise is included in the gradation conversion processing means (gradation conversion process). Removal means (noise removal processing) can be included, and the processing configuration can be simplified without separately providing a processing device dedicated to noise removal. In addition, the amount of calculation (calculation amount) in the noise removal processing is reduced, and the processing time can be shortened.

According to the imaging device of the fourth aspect , the coring coefficient is calculated so that the noise amounts of the low luminance portion and the high luminance portion in the high luminance side characteristic are equal. The coring coefficient for the side characteristics can be obtained easily (with a simpler calculation method), and based on this calculated coring coefficient, the noise amount does not differ between the low luminance part and the high luminance part. That is, it is possible to prevent the noise amount in the high luminance part from being noticeable because the noise amount in the high luminance part is larger than the noise amount in the low luminance part.

According to the imaging apparatus according to claim 5 , the captured image (linear / logarithmic image) obtained by the solid-state imaging device having the linear photoelectric conversion characteristic and the logarithmic photoelectric conversion characteristic, according to the expansion degree in the gradation conversion, that is, Noise can be easily removed by changing the amount of noise removal according to the degree of expansion in the gradation conversion of the logarithmic photoelectric conversion characteristic image, and as a result, a high-quality captured image can be obtained.

According to the imaging device according to claim 6 , expansion in gradation conversion is performed on a captured image (linear / linear image) obtained by a solid-state imaging device having the first linear photoelectric conversion characteristic and the second linear photoelectric conversion characteristic. Noise removal that changes the amount of noise removal according to the degree, that is, the degree of expansion in the gradation conversion of the second linear photoelectric conversion characteristic, can be easily performed, so that a high-quality captured image can be obtained. .

According to the imaging apparatus of the seventh aspect , the noise removal can be performed in the information processing apparatus configured to be able to exchange the captured image data with the imaging apparatus body, and the degree of freedom regarding the place where the noise removal is performed increases. In addition, a configuration capable of performing higher-speed arithmetic processing (noise removal processing) using an information processing apparatus such as a PC can be easily realized.

(Embodiment 1)
FIG. 1 shows a digital camera 1 which is an example of an imaging apparatus according to the first embodiment, and a block diagram of the digital camera 1 mainly related to imaging processing. As shown in FIG. 1, the digital camera 1 includes a lens unit 2, a solid-state imaging device 3, an amplifier 4, an A / D conversion unit 5, a noise removal unit 6, a gradation conversion unit 7, an image memory 8, an overall control unit 9, A monitor unit 10 and an operation unit 11 are provided.

  The lens unit 2 functions as a lens window for capturing subject light (light image), and constitutes an optical lens system for guiding the subject light to the solid-state imaging device 3 disposed inside the camera body. For example, a lens group including, for example, a zoom lens, a focus lens, and other fixed lens blocks arranged in series along the optical axis L of the subject light. The lens unit 2 includes a diaphragm (not shown) and a shutter (not shown) for adjusting the amount of light transmitted through the lens, and the overall control unit 9 controls driving of the diaphragm and shutter. Yes.

  The solid-state image sensor 3 is an image sensor that photoelectrically converts the image signal of each color component of R, G, and B into an image signal that is output to the subsequent amplifier 4 in accordance with the amount of light of the subject light image formed by the lens unit 2. In the present embodiment, as the solid-state imaging device 3, when the input luminance (sensor incident luminance) as shown in FIG. 4 is low (in the dark), the output pixel signal (output electric signal generated by photoelectric conversion) is linear. A photoelectric conversion characteristic composed of a linear characteristic that is converted and output and a logarithmic characteristic that is output by logarithmically converting the output pixel signal when the luminance is high (during light), in other words, the low luminance side is linear A photoelectric conversion characteristic having a logarithmic photoelectric conversion characteristic on the high luminance side is used. However, the solid-state imaging device 3 can be arbitrarily switched not only to the photoelectric conversion characteristics including the linear / logarithmic characteristics but also to the photoelectric conversion characteristics including only the linear characteristics or only the logarithmic characteristics. Further, the switching point between the linear characteristic and the logarithmic characteristic (hereinafter, this switching point is referred to as “inflection point”) is arbitrarily changed based on a predetermined control signal for each pixel circuit of the solid-state imaging device 3. Is possible. In the photoelectric conversion characteristics, the low luminance side characteristic as shown in the above linear characteristic is expressed as “low luminance side characteristic”, and the high luminance side characteristic as shown in the logarithmic characteristic is expressed as “high luminance side characteristic”. .

  FIG. 2 is a schematic configuration diagram of a two-dimensional MOS solid-state imaging device (CMOS image sensor) which is an example of the solid-state imaging device 3. In the figure, G11 to Gmn indicate pixels arranged in a matrix. A vertical scanning circuit 301 and a horizontal scanning circuit 302 are disposed in the vicinity of the outer peripheral edge of the pixel unit composed of the pixels G11 to Gmn. The vertical scanning circuit 301 sequentially scans row lines (signal lines) 304-1, 304-2,... 304-n (collectively referred to as row lines 304). The horizontal scanning circuit 302 applies the photoelectric conversion signal derived from each pixel to the output signal lines 306-1, 306-2,... 306-m (collectively referred to as the output signal line 306) in the horizontal direction for each pixel. Read sequentially. Each pixel is supplied with power by the power supply line 305. Each pixel is connected not only to the above lines and output signal lines but also to other lines (for example, a clock line), but is not shown in FIG.

  The output signal lines 306-1, 306-2,... 306-m are constant current sources 307-1, 307-2,. 307-m (these are collectively referred to as a constant current source 307) is provided. However, a resistor or a transistor may be provided as the amplifier circuit instead of the constant current source 307. Image data at the time of imaging and correction data at the time of reset of each pixel output via the output signal line 306 are sequentially selected as selection circuits (sample hold circuits) 308-1, 308-2,... 308-m. (They are collectively referred to as a selection circuit 308). Image data and correction data are output to the selection circuit 308 for each row and sampled and held. The sampled and held image data and correction data are output to the correction circuit 309 for each column, and the correction circuit 309 corrects the image data based on the correction data so that noise components due to sensitivity variations are removed. Is called. Then, the image data in which the sensitivity variation of each pixel is corrected is output serially from the correction circuit 309 for each pixel.

  FIG. 3 is a circuit diagram illustrating a configuration example of each of the pixels G11 to Gmn illustrated in FIG. As shown in the figure, each pixel of the solid-state imaging device 3 includes a photodiode PD1, transistors T10 to T13 as MOSFETs (Metal Oxide Semiconductor Field Effect Transistors), and FD (Floating Diffusion). Here, N-channel MOSFETs are employed as the transistors T10 to T13. VDD, φRSB, φRST, φTX, and φV indicate signals (voltages) for the respective transistors, and GND indicates ground.

  The photodiode PD1 is a photosensitive portion (photoelectric conversion portion), and outputs an electrical signal (photocurrent IPD1) corresponding to the amount of incident light from the subject. The transistor T12 is paired with the constant current source 307 shown in FIG. 2 to form an amplification circuit (source follower amplifier) for source follower amplification, and performs amplification (current amplification) on a voltage V1OUT described later. The transistor T13 is a signal reading transistor (row selection transistor) that operates as a switch that is turned on and off in accordance with a voltage (signal φV) applied to the gate. The source of the transistor T13 is connected to the output signal line 306 shown in FIG. 2. When the transistor T13 is turned on, the current amplified by the transistor T12 is led to the output signal line 306 as an output current.

  The transistor T10 operates as a switch that is turned on and off in accordance with a voltage applied to the gate of the transistor, and is generated in the photodiode PD1 in response to on / off switching depending on the level of the gate potential. This is a so-called transfer gate that switches between transfer and non-transfer of the photocurrent IPD1 (charge) to the FD. The photocurrent IPD1 generated in the photodiode PD1 flows through the parasitic capacitance of the photodiode PD1, the charge is accumulated, and a voltage corresponding to the amount of accumulated charge is generated. At this time, if the transistor T10 is in an on state, the charge (negative charge) accumulated in the parasitic capacitance moves toward the FD. The FD is a charge holding unit that temporarily holds a charge (signal charge), and plays a role of a so-called capacitor that changes the held charge into a voltage (performs charge-voltage conversion).

  The transistor T11 (reset gate transistor) switches between application and non-application of a reset bias to the FD according to on / off switching depending on the gate voltage of the transistor (for example, when the transistor T11 is in an on state) The transistor T10 is also turned on, and a reset bias is applied between φRSB and GND across the transistors T11, FD, transistor T10, and the photodiode PD1), and the gate voltage is set to the Mid potential (intermediate level). By doing so, linear conversion and logarithmic conversion are performed by FD of the charge (current flowing through the FD) moving from the photodiode PD1 to the FD and charge-voltage conversion by the transistor T11, respectively.

  In this case, a current (reset current) corresponding to the Mid potential flows through the transistor T11, and the source of the transistor T11 becomes a potential corresponding to the reset current. When the potential due to the charge moving from the photodiode PD1 is smaller than the source potential of the transistor T11 corresponding to the Mid potential, that is, the luminance of the subject to be imaged is low, that is, the subject is dark and incident on the photodiode PD1. When the amount of incident light is small, charge-voltage conversion as linear conversion is performed in the FD. On the other hand, when the source potential is exceeded, that is, the luminance of the subject to be imaged is high, that is, the subject is bright, and the photodiode PD1. When the amount of incident light incident on is large, charge-voltage conversion as logarithmic conversion is performed in the transistor T11.

  Thus, a voltage as a linear output based on an integral value of the photocurrent IPD1 in the FD or a logarithmic output by current-voltage conversion corresponding to the photocurrent IPD1 in the transistor T11 is applied to a connection node (output V1OUT) between the FD and the transistor T12. Appears as a voltage. That is, the output value in the linear characteristic region in the photoelectric conversion characteristic is an integral value of the photocurrent IPD1 in the FD, but in the logarithmic characteristic region, the potential due to the charge stored in the FD portion is the source of the transistor T11 (reset gate). In a region exceeding the current, a current equal to the photocurrent IPD1 flows to the transistor T11, and a value (voltage) obtained by current-voltage conversion of the photocurrent IPD1 in the transistor T11 appears in the FD section as the output value (signal charge is logarithmically). Compressed charge is accumulated in the parasitic capacitance). The current-voltage conversion in the transistor T11 corresponds to the logarithmic conversion. When the transistor T13 is turned on, the amplified current by the transistor T12 corresponding to these voltages is led to the output signal line 306 as an output current through the transistor T13. As described above, the solid-state imaging device 3 obtains an output signal that is linearly converted or logarithmically converted in accordance with the subject luminance (sensor incident luminance).

  Here, the N-channel MOSFET is used in each pixel of the solid-state imaging device 3, but a P-channel MOSFET may be used. Further, the solid-state imaging device 3 is not limited to a CMOS image sensor that performs linear conversion and logarithmic conversion using FD as described above. For example, a so-called subthreshold characteristic of a MOSFET (P-type or N-type) (a gate voltage is below a threshold value). At this time, a CMOS image sensor that performs the linear conversion and logarithmic conversion by using a characteristic that a minute current called a subthreshold current flows) may be employed. However, it is not limited to a CMOS image sensor, and may be a VMIS image sensor, a CCD image sensor, or the like.

  The amplifier 4 includes, for example, an AGC (auto gain control) circuit, and amplifies the analog image signal input from the solid-state imaging device 3 with an appropriate amplification factor so as to match the input voltage range of the A / D conversion unit 5. It is. In addition to the AGC circuit, the amplifier 4 may include a CDS (correlated double sampling) circuit that reduces sampling noise of an image signal. The gain value for the AGC circuit is set by the overall control unit 9.

  The A / D conversion unit 5 converts the analog image signal amplified (gain adjustment) by the amplifier 4 into a digital image signal, and each pixel signal obtained by receiving light at each pixel of the solid-state imaging device 3 is received. For example, it is converted into 12-bit pixel data.

  The noise removal unit 6 removes noise from the image signal input from the A / D conversion unit 5. Details of the noise removal will be described later.

  The gradation conversion unit 7 performs a predetermined gradation conversion process on the image signal so that the image signal input from the noise removal unit 6 has an appropriate output level, and changes the gradation characteristic of the image signal. It is something to be made.

  The image memory 8 includes a memory such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The image memory 8 is RAW image data before being processed by the noise removing unit 6 or the gradation converting unit 7 or the noise removing unit 6. The image data at the time of or after various data processing in the gradation conversion unit 7 or the overall control unit 9 is stored.

  The overall control unit 9 includes a ROM that stores each control program, a RAM that temporarily stores various data, a central processing unit (CPU) that reads and executes the control program from the ROM, and the like. It is responsible for overall operation control. Based on information (signals) transmitted from each processing unit such as the solid-state imaging device 3 and the operation unit 11 and the operation state of the apparatus, the overall control unit 9 obtains operation information such as parameters required by each processing unit. The operation of each processing unit is controlled by calculating and transmitting this.

  The monitor unit 10 includes, for example, a liquid crystal display (LCD) that includes a color liquid crystal display element disposed on the back of the camera. The image processed by the unit 7 and the image stored in the image memory 8 are displayed. The monitor unit 10 may display an image in which various processing results such as noise removal processing (coring processing) described later are reflected.

  The operation unit 11 is used to give an operation instruction (instruction input) to the digital camera 1 by a user. For example, various operation switch groups (operation button group) such as a power switch, a release switch, and a mode setting switch for setting various shooting modes. ). For example, when the release switch is pressed (turned on), the subject light is imaged by the imaging operation, that is, the solid-state imaging device 3, and the required image processing is performed on the image data obtained by this imaging. A series of photographing operations such as recording in the image memory 8 or the like is executed.

  Here, the noise removal unit 6 will be described in detail. The noise removing unit 6 corrects the sharpness of the image to a good state by performing a coring process for removing a noise component from the image signal. At this time, the noise removing unit 6 performs appropriate noise removal by changing a coring coefficient (a coefficient for removing the noise component of the image signal). Here, in particular, in a coring process for an image (logarithmic image) in a logarithmic characteristic region in which the degree of data compression by logarithmic compression differs depending on the high luminance side characteristic, that is, the input luminance level, an image using a low-pass filter (LPF) is used. The degree of expansion by gradation conversion of the logarithmic image (difference in local expansion; difference in data compression by logarithmic compression) by changing the coring coefficient (adaptively) according to the level of the low-frequency component extracted from In other words, noise removal is performed after setting the amount of noise removal corresponding to the degree of expansion of the noise component accompanying the expansion of the logarithmic image.

  FIG. 5 is a block configuration diagram of the noise removing unit 6. As shown in the figure, the noise removal unit 6 includes an area determination unit 61, an LPF unit 62, a coring processing unit 63, and a coefficient calculation unit 64. The area determination unit 61 is a low luminance side characteristic and a high luminance side characteristic of an image input from the A / D conversion unit 5 (this input image is expressed as a base image I), that is, an image of a linear characteristic region or a logarithm. It is determined whether the image is a characteristic region. The area determination unit 61 determines from the pixel values of the input image whether or not the photoelectric conversion characteristic at the time of imaging, that is, the characteristic indicated by the expression (1-1) or (1-2) described later, and the determination result ( Region determination information) is output to the coefficient calculation unit 64.

  The LPF unit 62 performs filtering processing using a linear LPF such as a Gaussian filter on the base image I, extracts (generates) low frequency components from the base image I, and information on the low frequency components (low frequency component information). ) Is output to the coefficient calculation unit 64. However, the LPF is not limited to the linear filter but may be a non-linear filter such as a median filter.

  The coring processor 63 performs a coring process based on coring coefficient information from a coefficient calculator 64 described later. The coring processing unit 63 has a characteristic (hereinafter referred to as a coring characteristic 630) having an input / output relationship shown in FIG. 6 with respect to the high frequency component obtained by subtracting the low frequency component extracted by the LPF unit 62 from the base image I. Coring processing based on “Th” in the coring characteristic 630 indicates the coring coefficient th calculated by the coefficient calculation unit 64. In this coring process, when the input is larger than -th and smaller than th, the output is zero (0). When the input is -th or less, the output is a value obtained by adding th to the input, and the input is th. When it is above, the conversion process is performed so that the output becomes a value obtained by subtracting th from the input. The subtraction of the low frequency component from the base image I, that is, the extraction of the high frequency component from the base image I is performed by a subtraction process by the subtraction unit 651. Further, the high frequency component after the coring process and the low frequency component from the LPF unit 62 are synthesized by the addition process by the addition unit 652 and output to the subsequent processing unit as a synthesized image O.

  The coefficient calculation unit 64 calculates (sets) a coring coefficient (th) in the coring process of the coring processing unit 63 based on the region determination information from the region determination unit 61 and the low frequency component information from the LPF unit 62. To do. In the final output image, here, the image output after the gradation conversion processing in the gradation conversion unit 7, the coefficient calculation unit 64 causes the high luminance part of the image to have a noise amount equivalent to that of the low luminance part. Calculate the coring coefficient. Specifically, when the region determination unit 61 determines that the image has a low luminance side characteristic, the coefficient calculation unit 64 performs low noise processing to perform uniform noise removal processing on the low luminance side characteristic image. The luminance-side characteristic coring coefficient (this coring coefficient value is set to “th0”) is output. However, the coring coefficient th0 is a certain value (for example, a fixed value) that is uniformly given to the entire image of the low luminance side characteristic. The coring coefficient th0 may be stored (set) in advance in the coring processing unit 63. The coring coefficient information (coring coefficient information) calculated by the coefficient calculating unit 64 is output to the coring processing unit 63.

  When the region determination unit 61 determines that the image has high luminance side characteristics, the coefficient calculation unit 64 calculates a coring coefficient corresponding to the degree of image expansion. By the way, the amount of noise before gradation conversion by the gradation conversion unit 7 is the same level in the high luminance part and the low luminance part in the high luminance side characteristic area. However, when the tone conversion returns to the linear characteristic, in this high luminance side characteristic area, the amount of noise in the high luminance part becomes larger than the amount of noise in the low luminance part, and the noise in the high luminance part (noise component) ) Will stand out. Therefore, after gradation conversion, the coring coefficient (noise removal) is set so that the noise level of the high-brightness part is equal to the noise level of the low-brightness part so that the amount of noise does not differ between the low-brightness part and the high-brightness part. The amount of noise in the high luminance portion is controlled to be equal to the amount of noise in the low luminance portion (high luminance noise amount = low luminance noise amount). Specifically, after tone conversion (after DR compression), the amount of noise after noise removal by the noise removing unit 6 is desired to be constant in the high luminance side characteristic region. Focusing on the luminance portion and the high luminance portion, the coring coefficient is calculated so that the respective noise amounts are the same (constant).

(Concept of coring coefficient calculation)
Here, the concept of calculating the coring coefficient will be specifically described.
The photoelectric conversion characteristic of the linear / logarithmic image by the solid-state image sensor 3 is a relational expression of the pixel value Y (output) with respect to the input luminance X (not logarithmic value) shown in the following formulas (1-1) and (1-2). It is represented by
Y = a * X + b (0 ≦ X ≦ Xth) (1-1)
Y = α * Ln (X) + β (X> Xth) (1-2)
However, the symbols “a, b, α, and β” indicate coefficients, and the symbol “*” indicates multiplication (the same applies hereinafter). Xth (and Yth corresponding to Xth) indicates the X coordinate (and Y coordinate) of the inflection point. A captured image having such a photoelectric conversion characteristic has a different coring coefficient calculation method in each case of 0 ≦ X ≦ Xth and X> Xth.

[When 0 ≦ X ≦ Xth]
The coring coefficient is calculated in the low luminance side characteristic region. Here, as described above, the value th0 uniformly set for the low luminance side characteristic region is used as the coring coefficient th in the low luminance side characteristic region (refer to equation (1-3) below).
th = th0 (1-3)

[When X> Xth]
The coring coefficient is calculated in the high luminance side characteristic region. Here, a condition is derived in which the amount of noise after gradation conversion is equal in the low luminance part and the high luminance part in the high luminance side characteristic area. Considering the case where noise is added to the image signal during imaging, the following equation (1-4) is obtained from the above equation (1-2).
Y = α * Ln (X) + β + N (1-4)
However, N: Noise included in the image signal (same for both high-luminance part and low-luminance part)

When the above equation (1-4) is converted into a linear characteristic (Y = X), the following equation (1-5) is obtained. However, the symbol “/” indicates division (and so on).
Y_LIN = exp ((Y−β) / α)
= Exp ((α * Ln (X) + β + N−β) / α)
= Exp (Ln (X) + N / α)
= Exp (Ln (X)) * exp (N / α)
= X * exp (N / α) (1-5)
Assuming that the low frequency component is L, the high frequency component H for performing noise removal processing is expressed by the following equation (1-6) from the above equation (1-5).
H = Y_LIN-L
= X * exp (N / α) −L (1-6)

When the noise removal amount ΔN is removed from the high frequency component, the high frequency component H ′ after noise removal is expressed by the following equation (1-7) from the above equation (1-6).
H ′ = X * exp (N / α) −L−ΔN (1-7)
By adding the low frequency component L to the high frequency component H ′, the linear characteristic output Y′_LIN after noise removal is calculated. That is, Y′_LIN is given by the following equation (1-8).
Y'_LIN = L + H '
= L + X * exp (N / α) −L−ΔN
= X * exp (N / α) −ΔN (1-8)

The linear characteristic output Y′_LIN is divided into an illumination component B and a reflectance component D to calculate a gradation conversion output Y′_drc.
Y'_LIN = B * D
Therefore, D = Y′_LIN / B
Further, gradation conversion is performed by compressing the illumination component B. If the illumination component after gradation conversion is B ′,
Y'_drc = B '* D
= B '* Y'_LIN / B (Substituting the above equation (1-8) into this equation)
= B ′ * (X * exp (N / α) −ΔN) / B
Therefore, the noise amount N ′ after gradation conversion is given by the following equation (1-9).
N ′ = Y′_drc−B ′ * X / B
= B ′ * (X * exp (N / α) −ΔN) / B−B ′ * X / B
= B ′ / B * (X * exp (N / α) −ΔN−X)
= B ′ / B * (X * (exp (N / α) −1) −ΔN) (1-9)

Here, the noises of the low luminance part and the high luminance part after gradation conversion are respectively N1 ′ and N2 ′, and the illumination components of the low luminance part and the high luminance part in the linear characteristic output are L1 and L2, respectively. Assuming that the illumination components after gradation conversion (after DR compression) for the components L1 and L2 are L1 ′ and L2 ′, and the noise removal amounts of the low luminance portion and the high luminance portion by coring processing are ΔN1 and ΔN2, respectively ( From the formula (1-9), the following formulas (1-10-1) and (1-10-2) are obtained.
N1 ′ = L1 ′ / L1 * ((exp (N / α) −1) −ΔN1) (1-10-1)
N2 ′ = L2 ′ / L2 * ((exp (N / α) −1) −ΔN2) (1-10-2)

From these, the relationship of ΔN1 and ΔN2 that leads to N1 ′ = N2 ′ is derived. That is, the following expression (1-11) is obtained from the relations (1-10-1) and (1-10-2) and N1 ′ = N2 ′.
L1 ′ / L1 * ((exp (N / α) −1) −ΔN1) = L2 ′ / L2 * ((exp (N / α) −1) −ΔN2) (1-11)
If this is transformed,
(L1 ′ / L1) / (L2 ′ / L2) * ((exp (N / α) −1) −ΔN1) = (exp (N / α) −1) −ΔN2
Further, when ΔN2 is arranged, the following equation (1-12) is obtained.
ΔN2 = (exp (N / α) −1) / ((L1 ′ / L1) / (L2 ′ / L2) * ((exp (N / α) −1) −ΔN1)) (1-12)

Therefore, when this concept is applied to the calculation of the coring coefficient of this embodiment, the coring coefficient th in the high luminance side characteristic region is given by the following expression (1-13) according to the above expression (1-12). It will be a thing.
th = (exp (N / α) −1) / ((L1 ′ / L1) / (L2 ′ / L2) * ((exp (N / α) −1) −th0)) (1-13)

  In this way, the digital camera 1 uses the noise removal unit 6 to change the parameter (coring coefficient) regarding the amount of noise removal according to the degree of expansion in the gradation conversion of the captured image (high luminance side characteristic image), That is, as shown in the above equation (1-13), for example, information on the low-luminance portion and the high-luminance portion (illumination components L1, L2, L1 ′, L2 ′ of the low-luminance portion and the high-luminance portion) corresponding to the difference in the degree of expansion. ) Is used to remove noise from the captured image. Thus, in noise removal, the amount of noise removal is changed according to the degree of expansion in gradation conversion (a different amount of noise removal is set according to the degree of expansion. Specifically, the larger the degree of expansion, the larger the degree of expansion. Therefore, it is possible to set the degree of expansion so that the noise is conspicuous in the image of the high luminance part with a high degree of expansion compared to the image of the low luminance part with a low degree of expansion. Noise (noise component) can be made inconspicuous due to the difference.

  FIG. 8 is a flowchart showing an example of an operation mainly related to noise removal of the digital camera 1 in the first embodiment. First, a captured image is obtained by imaging with the solid-state imaging device 3 (step S1). This captured image (base image I) is input to the noise removing unit 6 (step S2). The region determination unit 61 determines whether the input base image I is a low luminance side characteristic region (linear characteristic region) or a high luminance side characteristic region (logarithmic characteristic region) (step S3). . Next, a low frequency component is extracted from the base image I by the LPF process by the LPF unit 62 (step S4), and the coefficient calculation unit 64 uses the region determination information from the region determination unit 61 and the low frequency component information from the LPF unit 62. Based on the above, a coring coefficient corresponding to the degree of expansion in the logarithmic image gradation conversion is calculated (step S5). Based on the calculated coring coefficient, the coring processing unit 63 performs coring processing on the high frequency component obtained by subtracting the low frequency component from the base image I (step S6). The high-frequency component subjected to the coring process and the low-frequency component from the LPF unit 62 are combined by the adding unit 652 and output as a combined image O to the subsequent gradation converting unit 7 (step S7). Then, a predetermined gradation conversion process is performed on the composite image O by the gradation conversion unit 7 (step S8).

(Embodiment 2)
The noise removing unit 6 in the first embodiment may be configured as a functional block diagram of the noise removing unit 6a shown in FIG. That is, the noise removal unit 6a according to the second embodiment changes the region determination unit 61 to the region division extraction unit 66, and performs photoelectric conversion of low luminance side characteristics and high luminance side characteristics (linear characteristics and logarithmic characteristics). Different processing is performed for each characteristic. The noise removing unit 6a includes an area division extracting unit 66, a high luminance side characteristic processing unit 610m, a low luminance side characteristic processing unit 610n, and an image composition unit 67. The area division extraction unit 66 extracts and extracts a logarithmic image (referred to as an image I1) and a linear image (referred to as an image I2) from the base image I. The divided and extracted images I1 and I2 are input to the high luminance side characteristic processing unit 610m and the low luminance side characteristic processing unit 610n, respectively.

  The high luminance side characteristic processing unit 610m includes an LPF unit 62m, a coring processing unit 63m, and a coefficient calculation unit 64m. The LPF unit 62m performs LPF processing on the input image I1 to generate a low frequency component. Based on the low-frequency component information from the LPF unit 62m, the coefficient calculation unit 64m, as described above, in the high luminance side characteristics (image I1), the coring coefficient th that causes the high luminance portion to have a noise amount equivalent to that of the low luminance portion. Is calculated. The coring processing unit 63m is the coring coefficient information calculated by the coefficient calculating unit 64m (coring coefficient th in the high luminance side characteristic (including information on the coring coefficient th0 in the low luminance part of the high luminance side characteristic)) Based on the above, a coring process is performed on the high frequency component obtained by subtracting the low frequency component from the image I1 by the subtracting unit 651m. The image subjected to the coring process is synthesized with the low frequency component from the LPF unit 62m by the addition process by the addition unit 652m, and then input to the image synthesis unit 67 as an image I1 ′ (noise-removed image I1 ′). The

  On the other hand, the low luminance side characteristic processing unit 610n includes an LPF unit 62n and a coring processing unit 63n. The LPF unit 62n performs LPF processing on the input image I2 to generate a low frequency component. The coring processing unit 63n performs coring processing on the high frequency component obtained by subtracting the low frequency component from the image I2 by the subtraction unit 651n. However, in this case, coring coefficient information (coring coefficient th0 uniformly set for the low luminance side characteristics) used for the coring process is stored in, for example, the coring processing unit 63n. The image subjected to the coring process is synthesized with the low frequency component from the LPF unit 62n by the addition process by the addition unit 652n, and then input to the image synthesis unit 67 as an image I2 ′ (noise-removed image I2 ′). The The image synthesizing unit 67 synthesizes (adds) the noise-removed images I1 ′ and I2 ′ input from the high luminance side characteristic processing unit 610m and the low luminance side characteristic processing unit 610n, and outputs the synthesized image O to the subsequent processing unit. Output.

  As described above, the input base image I is first divided into regions, and the noise removal unit 6a configured to perform processing on each of the divided images makes the core in the coefficient calculation unit 64m. In the calculation of the ring coefficient, each time, without performing the case-by-case processing due to the difference in photoelectric conversion characteristics (low-luminance side characteristics and high-luminance side characteristics) for each pixel (saving the trouble of performing case-by-case each time) The coring coefficient can be calculated for all the pixels having the high luminance side characteristic as it is, and a higher speed calculation (noise removal processing) is possible.

  FIG. 9 is a flowchart illustrating an example of an operation mainly related to noise removal of the digital camera 1 in the second embodiment. First, a captured image is obtained by imaging with the solid-state imaging device 3 (step S21). This captured image (base image I) is input to the noise removing unit 6a (step S22). From the input base image I, the high luminance side characteristic region (image I1; logarithmic image) and the low luminance side characteristic region (image I2; linear image) are divided and extracted by the region division extraction unit 66 (step S23). When the divided and extracted image is an image of the high luminance side characteristic area (YES in step S24), a low frequency component is extracted from the image I1 by the LPF processing by the LPF unit 62m (step S25), and the LPF unit 62m. Based on the low-frequency component information from, the coefficient calculating unit 64m calculates a coring coefficient corresponding to the degree of expansion in logarithmic image gradation conversion (step S26). Based on the calculated coring coefficient, the coring processing unit 63m performs coring processing on the high frequency component obtained by subtracting the low frequency component from the image I1 (step S27), and this coring processing is performed. The high frequency component and the low frequency component from the LPF unit 62m are synthesized by the adding unit 652m to obtain an image I1 ′ (step S28).

  If the divided and extracted image is an image of the low luminance side characteristic area in step S24 (NO in step S24), a low frequency component is extracted from the image I2 by the LPF process by the LPF unit 62n (step S29). The coring processing unit 63n performs a coring process for a high frequency component obtained by subtracting the low frequency component from the image I2 (step S30). However, as the coring coefficient in the coring process here, for example, a value (th0 described above) stored in advance as a fixed value in the coring processing unit 63n is used. The high frequency component subjected to the coring process and the low frequency component from the LPF unit 62n are synthesized by the adding unit 652n to obtain an image I2 '(step S31). Then, the image I2 ′ in the low luminance side characteristic area and the image I1 ′ in the high luminance side characteristic area are synthesized by the image synthesizing unit 67 and output as the synthesized image O to the subsequent gradation converting unit 7 (Step S1). S32). Then, a predetermined gradation conversion process is performed on the composite image O by the gradation conversion unit 7 (step S33).

(Embodiment 3)
FIG. 10 is a block configuration diagram mainly relating to an imaging process of a digital camera 1a which is an example of an imaging apparatus according to the third embodiment. Compared with the digital camera 1 in the first embodiment, the digital camera 1a does not include the noise removal unit 6 described in the previous embodiment, but includes a gradation conversion unit 7a that also has the function of noise removal. Is different. Other configurations are the same as those of the first embodiment, and the description thereof is omitted.

  The gradation conversion unit 7a performs gradation conversion processing on a captured image (linear / logarithmic image) based on predetermined conversion characteristics. Specifically, in the gradation conversion processing, the gradation conversion unit 7a specifically improves the contrast in the linear / logarithmic image in the linear characteristic region so that the contrast can be improved locally while the contrast of the entire captured image is preserved. It has a function of performing DR compression so as to improve (improve) the contrast in the logarithmic characteristic region while maintaining the contrast (so that the contrast in the linear characteristic region is not overemphasized or lost). The gradation converting unit 7a divides the input linear / logarithmic image into image data of a linear image and a logarithmic image, extracts an illumination component and a reflectance component from the linear image and the logarithmic image, and then compresses the illumination component. Thus, the DR compression is performed. This DR compression technique is based on the Retinex theory (Multi-scale Retinex) focusing on human visual characteristics, that is, color constancy with respect to brightness. The gradation converting unit 7a performs noise removal on the reflectance component obtained in the process of the DR compression process. Hereinafter, the gradation converting unit 7a will be described in detail.

FIG. 11 is a functional block diagram relating to DR compression and noise removal in the gradation converting unit 7a. As shown in the figure, the gradation conversion unit 7a includes a region division extraction unit 71, a high luminance side characteristic conversion unit 72m, a low luminance side characteristic conversion unit 72n, and an image composition unit 73. The area division extraction unit 71 extracts (separates) a logarithmic image (image I1) and a linear image (image I2) from the base image I (linear / logarithmic image) input to the gradation conversion unit 7a. . Specifically, assuming that the base image I is represented by the above formulas (1-1) and (1-2), the region segmentation extraction unit 71 uses the relational expression shown in the following formula (2-0). Based on the pixel value of the base image I, the image I1 is divided into the image I2. The images I1 and I2 are input to the high luminance side characteristic conversion unit 72m and the low luminance side characteristic conversion unit 2n, respectively.
if (I ≧ Yth)
I1 = (I−β) / α, I2 = 0 (high luminance side characteristic)
else
I1 = 0, I2 = (Ib) / a (low luminance side characteristic) (2-0)

  The high luminance side characteristic conversion unit 72m performs gradation conversion (DR compression) of the image of the high luminance side characteristic region, and includes a first illumination component extraction unit 74m, a first illumination component compression unit 75m, and a first noise removal. Unit 76m and addition / linear conversion unit 77m.

By the way, according to the Retinex theory, the base image I is represented by the following equation (2-1), where the illumination component is the illumination component L and the reflectance component is the reflectance component R.
I = L * R (2-1)
However, the formula (2-1) is for the base image I as a linear image, and the base image I as a logarithmic image is expressed by a modified formula shown in the following formula (2-2). .
Log (I) = Log (L) + Log (R) (2-2)

The first illumination component extraction unit 74m extracts Log (L1) that is a logarithmic value of the illumination component L1 of the image I1 from the image I1. The extraction of Log (L1) is performed by the conversion shown in the following equation (2-3).
Log (L1) = Log (F (I1)) (2-3)
Log (F (I1)) may be approximately set to F (Log (I1)) (see the following formula (2-3) ′).
Log (L1) = F (Log (I1)) (2-3) '
However, “F” in each of the above formulas indicates conversion for extracting the illumination component. Since the illumination component can be approximated by a low-frequency component of the image, a low-frequency extraction filter such as a low-pass filter (LPF; linear filter) or a median filter (non-linear filter) is used in the conversion.

The first illumination component compression unit 75m performs a compression process on the illumination component Log (L1) extracted by the first illumination component extraction unit 74m, and compresses the illumination component L1 to a logarithmic value Log (L1) of the illumination component L1 ′. ') Is output. When a predetermined compression rate (DR compression rate) in DR compression is “c1”, Log (L1 ′) is expressed by the following equation (2-4).
Log (L1 ′) = Log (L1) * c1 (2-4)

  The first noise removal unit 76m performs noise removal processing on the logarithmic value Log (R1) of the reflectance component R1 of the image I1 based on the information on the illumination component (illumination component information) from the first illumination component extraction unit 74m. The logarithm value Log (R1 ′) of the reflectance component R1 ′ (noise-removed reflectance component image R1 ′) from which the noise has been removed is output. This Log (R1) is obtained by subtracting Log (L1) from the image I1 by the subtraction unit 781m. Details of the noise removal processing will be described later.

The addition / linear conversion unit 77m synthesizes Log (L1 ′) input from the first illumination component compression unit 75m and Log (R1 ′) input from the first noise removal unit 76m by addition processing, The combined logarithmic image is converted into a linear image and output. That is, assuming that an image after gradation conversion (DR compression and noise removal) for the image I1 is an image I1 ′ (noise removal image I1 ′), the addition / linear conversion unit 77m has the following relationship (2-5). The obtained image I1 ′ represented by the following expression (2-6) is output. The image I1 ′ is output to the image composition unit 73.
Log (I1 ′) = Log (L1 ′) + Log (R1 ′) (2-5)
I1 ′ = exp (Log (L1 ′) + Log (R1)) (2-6)

On the other hand, the low luminance side characteristic conversion unit 72n performs gradation conversion (DR compression) of the image of the low luminance side characteristic region. The second illumination component extraction unit 74n, the second illumination component compression unit 75n, and the second A noise removing unit 76n is provided. The second illumination component extraction unit 74n extracts the illumination component L2 from the image I2. The extraction process of the illumination component L2 from the image I2 is expressed by the following equation (2-7).
L2 = F (I2) (2-7)
However, “F” in the above expression indicates a conversion for extracting an illumination component, and a filter for low frequency extraction such as a low-pass filter or a median filter is used for this conversion as described above. It should be noted that the reflectance component R2 of the image I2 is obtained from the above equation (2-1) by R2 = I2 / L2.

The second illumination component compression unit 75n performs a compression process on the logarithmic value of the illumination component L2 extracted by the second illumination component extraction unit 74n, and outputs an illumination component L2 ′ obtained by compressing the illumination component L2. In this case, when the same compression rate “c1” as that of the first illumination component compression unit 75m is used in the DR compression, the illumination component L2 ′ is given by the following equation (2-8).
L2 ′ = exp (Log (L2) * c1) (2-8)

  The second noise removal unit 76n performs noise removal processing on the reflectance component R2 of the image I2 based on the illumination component information from the second illumination component extraction unit 74n, and the reflectance component R2 ′ ( The noise removal reflectance component image R2 ′) is output. The reflectance component R2 is obtained by dividing L2 from the image I2 by the divider 781n. Details of the noise removal processing will be described later.

  The image subjected to the noise removal process in the second noise removal unit 76n is combined with the illumination component L2 ′ from the second illumination component compression unit 75n by the multiplication process by the multiplication unit 782n, and then the image I2 ′ (noise removal image). I2 ′) is input to the image composition unit 73. The image synthesizing unit 73 synthesizes (adds) the noise-removed images I1 ′ and I2 ′ input from the high luminance side characteristic converting unit 72m and the low luminance side characteristic converting unit 72n, and outputs the synthesized image O to the subsequent processing unit. Output.

  Here, the first and second noise removing units 76m and 76n will be described in detail. In the present embodiment, noise removal is performed in the gradation conversion process. However, since a high frequency component necessary for the coring process has already been extracted as a reflectance component, noise removal may be performed on the reflectance component. Therefore, the first and second noise removal units 76m and 76n are configured only by functional parts that perform noise removal by coring processing, as shown in the block diagrams of FIGS. In the first and second noise removing units 76m and 76n, coring processing based on the coring characteristics 630 shown in FIG. 6 is performed by the coring processing units 762m and 762n, respectively, on the input reflectance components. However, the first and second noise removal units 76m and 76n have different coring coefficient setting methods in the coring process, and the first noise removal unit 76m includes a coefficient calculation unit 761m. The coring coefficient (th) calculated according to the degree of expansion of the gradation conversion by 761m is used, but the second noise removal unit 76n does not perform the calculation process according to the degree of expansion of the gradation conversion and does not perform the luminance process. The coring coefficient (th0) of the side characteristic is used.

  Further, in the coefficient calculation unit 64 of the noise removal unit 6 in the first embodiment (coefficient calculation unit 64m of the noise removal unit 6a in the second embodiment), the coring is performed using the low frequency component extracted by using the LPF as an input. The coefficient is calculated, but in the coefficient calculation unit 761m of the first noise removal unit 76m, the logarithmic value Log (L1) of the illumination component L1 corresponds to the low frequency component. The ring coefficient is calculated.

(Concept of coring coefficient calculation)
Here, the concept of calculating the coring coefficient in the third embodiment will be specifically described. The photoelectric conversion characteristics of the linear / logarithmic image by the solid-state imaging device 3 are expressed by the following equations (3-1) and (3-2) as in the first embodiment, and 0 ≦ X ≦ Xth as described above. And X> Xth, the coring coefficient calculation method is different.
Y = a * X + b (0 ≦ X ≦ Xth) (3-1)
Y = α * Ln (X) + β (X> Xth) (3-2)
However, the coordinates (Xth, Yth) are inflection points of this photoelectric conversion characteristic.

[When 0 ≦ X ≦ Xth]
The coring coefficient is calculated in the low luminance side characteristic region. Here, as the coring coefficient th in the low luminance side characteristic region, a value th0 that is uniformly set for the low luminance side characteristic region is used as in the first embodiment (refer to equation (3-3) below). .
th = th0 (3-3)

[When X> Xth]
The coring coefficient is calculated in the high luminance side characteristic region. Here, a condition is derived in which the amount of noise after gradation conversion is equal in the low luminance part and the high luminance part in the high luminance side characteristic area. Considering the case where noise is added to the image signal during imaging, the following equation (3-4) is obtained from the above equation (3-2).
Y = α * Ln (X) + β + N (3-4)
However, N: Noise included in the image signal (same for both high-luminance part and low-luminance part)

When the above equation (3-4) is converted into a linear characteristic (Y = X), the following (3-5) is obtained.
Y_LIN = exp ((Y−β) / α)
= Exp ((α * Ln (X) + β + N−β) / α)
= Exp (Ln (X) + N / α)
= Exp (Ln (X)) * exp (N / α)
= X * exp (N / α) (3-5)
When the low frequency component is L, the reflectance component R for performing noise removal processing is given by the following equation (3-6) from the above equation (3-5).
R = Y_LIN / L
= X / L * exp (N / α) (3-6)

In the noise removal processing here, noise removal is performed on the logarithmic image. Therefore, when the above equation (3-6) is converted into logarithmic characteristics, the following equation (3-7) is obtained.
Ln (R) = Ln (X / L * exp (N / α))
= Ln (X / L) + Ln (exp (N / α))
= Ln (X / L) + N / α (3-7)
Further, noise removal is performed with the noise removal amount ΔN, and the reflectance component R ′ after the noise removal is obtained. The reflectance component R ′ after noise removal is given by the following equation (3-8) from the above (3-7).
Ln (R ′) = Ln (X / L) + N / α−ΔN (3-8)

When the above equation (3-8) is converted into linear characteristics, the following equation (3-9) is obtained.
R ′ = exp (Ln (X / L) + N / α−ΔN)
= Exp (Ln (X / L) * exp (N / α-ΔN)
= (X / L) * exp (N / α-ΔN) (3-9)
The reflectance component R ′ and the illumination component L ′ obtained by compressing the illumination component L are combined to calculate an output Y ′ after gradation conversion. The output Y ′ is given by the following equation (3-10) from the above equation (3-9).
Y '= L' * R '
= L ′ * (X / L) * exp (N / α−ΔN) (3-10)
Therefore, the noise amount N ′ after gradation conversion is given by the following equation (3-11) from the above equation (3-10).
N ′ = Y′−L ′ * (X / L)
= L ′ * (X / L) * exp (N / α−ΔN) −L ′ * (X / L)
= L ′ * (X / L) * (exp (N / α−ΔN) −1) (3-11)

Here, in order to consider only the noise component, if the reflectance component other than noise is “1”, the value “X / L” of the term corresponding to the reflectance component other than noise in the above equation (3-11) is “ Substituting “1”, the following equation (3-12) is obtained.
N ′ = L ′ * (exp (N / α−ΔN) −1) (3-12)

Assume that the noise removal amounts of the low luminance portion and the high luminance portion by coring processing are ΔN1 and ΔN2, and the noise amount when noise removal is performed with these values. The noises of the low luminance part and the high luminance part after gradation conversion (after DR compression) are N1 ′ and N2 ′, respectively, and the illumination components of the low luminance part and the high luminance part after gradation conversion (after DR compression) Are L1 and L2, respectively, the following equations (3-13-1) and (3-13-2) are obtained from the above equation (3-12).
N1 ′ = L1 * (exp (N / α−ΔN1) −1) (3-13-1)
N2 ′ = L2 * (exp (N / α−ΔN2) −1) (3-13-2)

From these, the relationship of ΔN1 and ΔN2 that leads to N1 ′ = N2 ′ is derived. That is, the following expression (3-14) is obtained from the relations (3-13-1) and (3-13-2) and N1 ′ = N2 ′.
L1 * (exp (N / α−ΔN1) −1) = L2 * (exp (N / α−ΔN2) −1) (3-14)
If this equation (3-14) is transformed,
L1 / L2 * (exp (N / α−ΔN1) −1) = exp (N / α−ΔN2) −1
L1 / L2 * (exp (N / α−ΔN1) −1) + 1 = exp (N / α) / exp (ΔN2)
exp (ΔN2) = exp (N / α) / (L1 / L2 * (exp (N / α−ΔN1) −1) +1)
By arranging ΔN2, the following equation (3-15) is obtained.
ΔN2 = Ln (exp (ΔN2))
= Ln (exp (N / α) / (L1 / L2 * (exp (N / α−ΔN1) −1) +1)) (3-15)

Therefore, when this concept is applied to the calculation of the coring coefficient of this embodiment, the coring coefficient th in the high luminance side characteristic region is given by the following expression (3-16) according to the above expression (3-15). It will be a thing.
th = Ln (exp (N / α) / (L1 / L2 * (exp (N / α−th0) −1) +1)) (3-16)

  FIG. 14 is a flowchart illustrating an example of an operation mainly related to noise removal of the digital camera 1a according to the third embodiment. First, a captured image is obtained by imaging with the solid-state imaging device 3 (step S41). This captured image (base image I) is input to the gradation converting unit 7a (step S42). From the input base image I, a high luminance side characteristic region (image I1; logarithmic image) and a low luminance side characteristic region (image I2; linear image) are divided and extracted by the region division extraction unit 71 (step S43). When the divided and extracted image is an image of the high luminance side characteristic region (YES in Step S44), the first illumination component extraction unit 74m extracts the illumination component (Log (L1)) from the image I1 (Step S44). S45) Based on the illumination component information from the first illumination component extraction unit 74m, the coefficient calculation unit 761m in the first noise removal unit 76m calculates a coring coefficient corresponding to the degree of expansion in the logarithmic image gradation conversion. (Step S46). Based on the calculated coring coefficient, the coring processing unit 762m performs a coring process on the reflectance component (Log (R1)) obtained by subtracting the illumination component from the image I1 (step S47). The reflectance component (Log (R1 ′)) subjected to the coring process and the illumination component (Log) formed by compressing the illumination component extracted by the first illumination component extraction unit 74m by the first illumination component compression unit 75m. (L1 ′)) is synthesized by the addition / linear transformation unit 77m, and the logarithmic image is linearly transformed to obtain an image I1 ′ as a linear image (steps S48 and S49). However, either of the operations in steps S48 and S49 may be performed first.

  In step S44, when the divided and extracted image is an image of the low luminance side characteristic region (NO in step S44), the illumination component (L2) is extracted from the image I2 by the second illumination component extraction unit 74n ( In step S50), the coring processing for the reflectance component (R2) obtained by dividing the illumination component from the image I2 is performed by the coring processing unit 762n in the second noise removal unit 76n (step S51). However, as the coring coefficient in the coring process here, for example, a value (th0 described above) stored in advance as a fixed value in the coring processing unit 762n is used. The reflectance component (R2 ′) subjected to the coring process and the illumination component (L2 ′) obtained by compressing the illumination component extracted by the second illumination component extraction unit 74n by the second illumination component compression unit 75n. Are combined by the multiplier 782n to obtain an image I2 ′ (step S52). Then, the image I2 ′ in the low luminance side characteristic region and the image I1 ′ in the high luminance side characteristic region are combined by the image combining unit 73 and output as the combined image O from the gradation converting unit 7a (step S53). .

(Embodiment 4)
The digital camera 1 and the digital camera 1a are configured to perform various signal processing such as noise removal and gradation conversion (DR compression) in the digital camera, but these signal processing is not performed in the digital camera but in the digital camera. It is good also as a structure performed within information processing apparatuses (external apparatus with respect to a digital camera), such as PC (personal computer) and PDA (personal digital assistant) to which the imaging data obtained by this is given. An imaging system having such a configuration will be described. As illustrated in FIG. 15, the imaging system 100 includes a digital camera 1 b and an information processing device 200.

  Compared to the digital camera 1, the digital camera 1 b does not have the noise removing unit 6 and includes an input / output I / F unit 12. Other configurations are the same, and the description thereof is omitted. The input / output I / F unit 12 exchanges (transfers and receives) various data with the information processing apparatus 200 connected via a predetermined wired or wireless network (for example, LAN). The input / output I / F unit 12 performs a line securing process with the information processing apparatus 200 and reads an image signal stored in the image memory 8 to the information processing apparatus 200 (an input / output I / F unit 260 described later). Send. The image memory 8 is used as a place for temporarily storing imaging data (image signal from the A / D conversion unit 5) obtained by the solid-state imaging device 3 before transmission to the information processing apparatus 200.

  The information processing apparatus 200 includes a CPU and the like, and temporarily stores data such as imaging data transmitted from the digital camera 1b and an overall control unit 210 that controls operation control (data calculation processing) of the entire information processing apparatus 200. An image memory 220, an HDD (hard disk) 230 storing predetermined applications and various data, a display 240 for displaying images (video), etc., and a keyboard and a mouse for inputting operation instructions (data input). An operation unit 250 and an input / output I / F unit 260 for exchanging data with the outside, that is, the digital camera 1b (input / output I / F unit 12) are provided. Such an information processing apparatus 200 performs the above-described processing on the image data transmitted from the digital camera 1b by the overall control unit 210 (for example, a noise removal unit or a gradation conversion unit provided in the overall control unit 210). Similar to the embodiment, noise removal processing (coring processing) and gradation conversion processing (DR compression) are performed. Information (image) of the processing result is recorded on the HDD 230 and displayed on the display 240. In addition, it is good also as a structure which performs a noise removal process and a gradation conversion process, displaying the imaging data on the display 240 and confirming the processing content sequentially. The configuration for exchanging data between the digital camera 1b and the information processing apparatus 200 is not limited to the above configuration, and may be a configuration performed via a recording medium (storage medium) such as a memory card.

  FIG. 16 is a flowchart illustrating an example of operations mainly related to noise removal of the imaging system 100 according to the fourth embodiment. First, an instruction signal for image acquisition is transmitted from the information processing apparatus 200 to the digital camera 1b (conversely, the instruction signal may be transmitted from the digital camera 1b to the information processing apparatus 200) (step S61). The information processing apparatus 200 and the digital camera 1b (each input / output I / F unit) start processing for securing mutual lines (step S62). If the line is not secured (NO in step S63), an error is displayed on the display 240 (or monitor unit 10) (step S64), and the process returns to step S62 to continue the line securing process. When the line is secured (YES in step S63), the information processing apparatus 200 transmits a signal requesting signal output of the captured image to the digital camera 1b (or vice versa) (step S65), and the digital camera 1b. Image data is transmitted from the information processing apparatus 200 to the information processing apparatus 200 (step S66). The information processing apparatus 200 receives the image data and performs various parameter setting processing such as a coring coefficient (step S67). Based on the parameters, the noise removal processing (coring processing) and tone conversion are performed. Various signal processing such as processing (DR compression) is performed (step S68). The processing result is displayed on the display 240 (or the monitor unit 10), and the processed image data is stored in the HDD 230 (step S69). When various parameters such as the coring coefficient are changed (YES in step S70), a parameter resetting process is performed (step S71), and the process returns to step S68 based on the reset parameter. Signal processing is performed. If the parameter is not changed (NO in step S70), the flow ends.

  By the way, in each of the above-described embodiments, as the solid-state imaging device 3 having a plurality of different photoelectric conversion characteristics, an image sensor having photoelectric conversion characteristics (linear / logarithmic characteristics) in which the low luminance side has linear characteristics and the high luminance side has logarithmic characteristics. However, the present invention is not limited to this, and for example, a variable storage time type image sensor or a variable storage capacity type image sensor (hereinafter, these image sensors are referred to as adaptive image sensors) may be used. That is, this adaptive imaging sensor has linear photoelectric conversion characteristics (first linear characteristic 171 and second linear characteristic 172) on both the low luminance side and the high luminance side as shown in FIG. , Yth) is an inflection point), and the slope of each linear characteristic on the graph changes according to the luminance level. This change is performed at a predetermined output level, and a plurality of different photoelectric conversion characteristics are obtained in the same manner as the linear / logarithmic imaging sensor. Although this adaptive imaging sensor can perform noise removal processing similar to that in each of the above embodiments, the high luminance side characteristic is not a logarithmic characteristic but a linear characteristic. Accordingly, the first embodiment ( The following modifications are necessary for the third embodiment (including the second embodiment). This change will be described below in the fifth and sixth embodiments. Since the change point is similarly applied to the fourth embodiment, the description thereof is omitted.

(Embodiment 5)
First, changes to the first embodiment will be described. Here, as the high luminance side characteristic is changed from the logarithmic characteristic to the linear characteristic, it is necessary to change the way of thinking of the coring coefficient calculation in the noise removing unit 6 (6a). However, it is not necessary to change the configuration (functional block configuration) in the noise removing unit.

(Concept of coefficient calculation)
Here, the concept of calculating the coring coefficient in the fifth embodiment will be specifically described. Unlike the case of the first embodiment, the photoelectric conversion characteristic of the adaptive image sensor of this embodiment is a pixel value Y (output) with respect to the input luminance X shown in the following equations (4-1) and (4-2). (Corresponding to the photoelectric conversion characteristics of the base image I).
Y = a1 * X + b1 (0 ≦ X ≦ Xth) (4-1)
Y = a2 * X + b2 (X> Xth) (4-2)
However, the coordinates (Xth, Yth) are the inflection points of the photoelectric conversion characteristics. A photographed image obtained by the solid-state imaging device 3 having the photoelectric conversion characteristics of the equations (4-1) and (4-2) is referred to as a linear / linear image.

[When 0 ≦ X ≦ Xth]
The coring coefficient in the low luminance side characteristic region is calculated, and the coring coefficient th is the value th0 as in the first embodiment (refer to equation (4-3) below).
th = th0 (4-3)

[When X> Xth]
The coring coefficient is calculated in the high luminance side characteristic region. Also in this case, a condition is derived so that the amount of noise after gradation conversion is equal in the low luminance part and the high luminance part in the high luminance side characteristic area. Considering the case where noise is added to the image signal at the time of imaging, Equation (4-4) is obtained from Equation (4-2).
Y = a2 * X + b2 + N (4-4)
However, N: Noise included in the image signal (same for both high-luminance part and low-luminance part)

When the above equation (4-4) is converted into a linear characteristic (Y = X) and the linear characteristic output Y_LIN is calculated, the following equation (4-5) is obtained.
Y_LIN = (Y−b2) / a2
= (A2 * X + b2 + N−b2) / a2
= X + N / a2 (4-5)
When the low frequency component is L, the high frequency component H for performing noise removal processing is given by the following equation (4-6) from the above equation (4-5).
H = Y_LIN-L
= X + N / a2-L (4-6)

When the noise removal amount ΔN is removed from the high frequency component, the high frequency component H ′ after the noise removal is expressed by the following equation (4-7) from the above equation (4-6).
H ′ = X + N / a2−L−ΔN (4-7)
By adding the low frequency component L to the high frequency component H ′, the linear characteristic output Y′_LIN after noise removal is calculated. That is, Y′_LIN is given by the following equation (4-8).
Y'_LIN = L + H '
= L + X + N / a2-L-ΔN
= X + N / a2-ΔN (4-8)

The linear characteristic output Y′_LIN is divided into an illumination component B and a reflectance component D to calculate a gradation conversion output Y′_drc.
Y'_LIN = B * D
Therefore, D = Y′_LIN / B
Further, gradation conversion is performed by compressing the illumination component B. If the illumination component after gradation conversion is B ′,
Y'_drc = B '* D
= B '* Y'_LIN / B
Therefore, the noise amount N ′ after gradation conversion is given by the following equation (4-9).
N ′ = Y′_drc−B ′ * X / B
= B '* Y'_LIN / B-B' * X / B
= B '/ B * (Y'_LIN-X)
= B ′ / B * (X + N / a2−ΔN−X)
= B '/ B * (N / a2-ΔN) (4-9)

Here, the noises of the low luminance part and the high luminance part after gradation conversion (after DR compression) are N1 ′ and N2 ′, respectively, and the illumination components of the low luminance part and the high luminance part in the linear characteristic output are L1, respectively. L2, L1 ′ and L2 ′ are the illumination components after gradation conversion (after DR compression) for these illumination components L1 and L2, respectively, and ΔN1 is the noise removal amount of the low luminance part and the high luminance part by coring processing. Assuming ΔN2, the following equations (4-10-1) and (4-10-2) are obtained from the above equation (4-9).
N1 ′ = L1 ′ / L1 * (N / a2−ΔN1) (4-10-1)
N2 ′ = L2 ′ / L2 * (N / a2−ΔN2) (4-10-2)

From these, the relationship of ΔN1 and ΔN2 that leads to N1 ′ = N2 ′ is derived. That is, the following expression (4-11) is obtained from the relations (4-10-1) and (4-10-2) and N1 ′ = N2 ′.
L1 ′ / L1 * (N / a2−ΔN1) = L2 ′ / L2 * (N / a2−ΔN2) (4-11)
If this equation (4-11) is transformed,
(L1 ′ / L1) / (L2 ′ / L2) * (N / a2−ΔN1) = N / a2−ΔN2, and when ΔN2 is arranged, the following equation (4-12) is obtained.
ΔN2 = N / a2− (L1 ′ / L1) / (L2 ′ / L2) * (N / a2−ΔN1) (4-12)

Therefore, when this concept is applied to the calculation of the coring coefficient of this embodiment, the coring coefficient th in the high luminance side characteristic region is given by the following expression (4-13) according to the above expression (4-12). It will be a thing.
th = N / a2- (L1 ′ / L1) / (L2 ′ / L2) * (N / a2-th0) (4-13)

  Note that the flow of operations related to noise removal in the fifth embodiment is the same as the flow shown in FIG. 8, and a description thereof will be omitted.

(Embodiment 6)
Next, changes to the third embodiment will be described. Here, as the high luminance side characteristic is changed from the logarithmic characteristic to the linear characteristic, it is necessary to change the configuration of the gradation converting unit 7a and the concept of calculating the coring coefficient in the first noise removing unit 76m. Become. First, the gradation conversion unit 7b in which the configuration of the gradation conversion unit 7a is changed will be described.

  FIG. 18 is a functional block diagram of the gradation conversion unit 7b when the adaptive imaging sensor is used in the third embodiment. As shown in the figure, the gradation conversion unit 7b is different from the gradation conversion unit 7a in the third embodiment in that an area division extraction unit 71 ′, a high luminance side characteristic conversion unit 72m ′, and an image composition unit 73 ′ are included. Different. Other configurations are the same, and the description thereof is omitted.

The base image I input to the gradation converting unit 7b here is an image (linear / linear) having photoelectric conversion characteristics in the relations of equations (4-1) and (4-2), as in the fifth embodiment. Linear image). The area division extraction unit 71 ′ extracts and separates (separates) a linear image (image I1) having a high luminance side characteristic and a linear image (image I2) having a low luminance side characteristic from the base image I. The area division extraction unit 71 ′ discriminates the pixel value of the base image I based on the relational expression shown in the following expression (5-1) to divide the image into an image I1 and an image I2. The images I1 and I2 are input to the high luminance side characteristic conversion unit 72m ′ and the low luminance side characteristic conversion unit 2n, respectively.
if (I ≧ Yth)
I1 = (I−b2) / a2, I2 = 0 (high luminance side characteristic)
else
I1 = 0, I2 = (I−b1) / a1 (low luminance side characteristic) (5-1)

The high luminance side characteristic conversion unit 72m ′ includes a first illumination component extraction unit 74m ′, a first illumination component compression unit 75m ′, and a first noise removal unit 76m ′. The first illumination component extraction unit 74m ′ extracts the illumination component L1 from the image I1 as a linear image. The extraction process of the illumination component L1 from the image I1 is expressed by the following equation (5-2).
L1 = F (I1) (5-2)
However, “F” in the above expression is a conversion for extracting an illumination component as in the third embodiment, and a filter for low frequency extraction is used.

The first illumination component compression unit 75m ′ performs a compression process on the illumination component L1 extracted by the first illumination component extraction unit 74m ′, and outputs an illumination component L1 ′ obtained by compressing the illumination component L1. When the predetermined compression rate in the DR compression in this case is represented by “c3”, the illumination component L1 ′ is given by the following equation (5-3).
L1 ′ = exp (Log (L1) * c3) (5-3)

  The first noise removal unit 76m ′ performs noise removal processing on the reflectance component R1 of the image I1 based on the illumination component information from the first illumination component extraction unit 74m ′, and the reflectance component R1 from which the noise has been removed. '(Noise removal reflectance component image R1') is output. The reflectance component R1 is obtained by dividing L1 from the image I1 by the dividing unit 781m '. Details of the noise removal processing will be described later.

  The image subjected to the noise removal process in the first noise removal unit 76m ′ is combined with the illumination component L1 ′ from the first illumination component compression unit 75m ′ by the multiplication process by the multiplication unit 77m ′, and then the image I1 ′ ( The noise-removed image I1 ′) is input to the image composition unit 73 ′. The image synthesizing unit 73 ′ synthesizes (adds) the noise-removed images I1 ′ and I2 ′ as linear images, both of which are input from the high luminance side characteristic converting unit 72m ′ and the low luminance side characteristic converting unit 72n. The synthesized image O is output to a subsequent processing unit.

  Here, the first noise removing unit 76m 'will be described in detail. The concept of the noise removal process is the same as that of the first noise removal unit 76m in the third embodiment, but the reflectance component and the input for calculating the coring coefficient are linear characteristics. That is, the first noise removing unit 76m ′ has the configuration shown in FIG. 19 and uses the coring coefficient calculated by the coefficient calculating unit 761m ′ based on the information of the illumination component L1, and uses the coring processing unit 762m. In this configuration, the coring process is performed on the reflectance component R1.

(Concept of coefficient calculation)
Here, the concept of calculating the coring coefficient in the sixth embodiment will be specifically described. The photoelectric conversion characteristic of the adaptive image sensor of this embodiment is similar to that of the fifth embodiment in that the pixel value Y (output) with respect to the input luminance X shown in the following equations (6-1) and (6-2) It is expressed by a relational expression.
Y = a1 * X + b1 (0 ≦ X ≦ Xth) (6-1)
Y = a2 * X + b2 (X> Xth) (6-2)

[When 0 ≦ X ≦ Xth]
The coring coefficient in the low luminance side characteristic area is calculated, and the coring coefficient th is the value th0 as in the fifth embodiment (refer to equation (6-3) below).
th = th0 (6-3)

[When X> Xth]
The coring coefficient is calculated in the high luminance side characteristic region. Also in this case, a condition is derived so that the amount of noise after gradation conversion is equal in the low luminance part and the high luminance part in the high luminance side characteristic area. Considering the case where noise is added to the image signal at the time of imaging, Equation (6-4) is obtained from Equation (6-2).
Y = a2 * X + b2 + N (6-4)
However, N: Noise included in the image signal (same for both high-luminance part and low-luminance part)

When the above equation (6-4) is converted into a linear characteristic (Y = X) and the linear characteristic output Y_LIN is calculated, the following equation (6-5) is obtained.
Y_LIN = (Y−b2) / a2
= (A2 * X + b2 + N−b2) / a2
= X + N / a2 (6-5)
When the low frequency component is L, the reflectance component R for performing noise removal processing is given by the following equation (6-6) from the above equation (6-5).
R = Y_LIN / L
= (X + N / a2) / L (6-6)
When noise removal is performed with the noise removal amount ΔN, the reflectance component R ′ after noise removal is given by the following equation (6-7) from the above (6-6).
R ′ = (X + N / a2) / L−ΔN (6-7)

The reflectance component R ′ and the illumination component L ′ obtained by compressing the illumination component L are combined to calculate an output Y ′ after gradation conversion. The output Y ′ is given by the following equation (6-8) from the above equation (6-7).
Y '= L' * R '
= L ′ * ((X + N / a2) / L−ΔN) (6-8)
Therefore, the noise amount N ′ after gradation conversion is given by the following equation (6-9) from the above equation (6-8).
N ′ = Y′−L ′ * (X / L)
= L ′ * ((X + N / a2) / L−ΔN) −L ′ * (X / L)
= L ′ * (X / L) + L ′ * (N / a2) / L−L ′ * ΔN−L ′ * (X / L)
= L ′ * ((N / a2) / L−ΔN) (6-9)

Assume that the noise removal amounts of the low luminance portion and the high luminance portion by coring processing are ΔN1 and ΔN2, and the noise amount when noise removal is performed with these values. The low luminance portion and high luminance portion noise after gradation conversion (after DR compression) are N1 ′ and N2 ′, respectively, and the illumination components of the low luminance portion and high luminance portion in the linear characteristic output are L1 and L2, respectively. Assuming that the illumination components after gradation conversion (after DR compression) for these illumination components L1 and L2 are L1 ′ and L2 ′, respectively, from the above equation (6-9), the following equations (6-10-1) and ( 6-10-2) is obtained.
N1 ′ = L1 ′ * ((N / a2) / L1−ΔN1) (6-10-1)
N2 ′ = L2 ′ * ((N / a2) / L2−ΔN2) (6-10-2)

From these, the relationship of ΔN1 and ΔN2 that leads to N1 ′ = N2 ′ is derived. That is, the following formula (6-11) is obtained from the relations (6-10-1) and (6-10-2) and N1 ′ = N2 ′.
L1 ′ * ((N / a2) / L1−ΔN1) = L2 ′ * ((N / a2) / L2−ΔN2) (6-11)
If this equation (6-11) is transformed,
L1 ′ / L2 ′ * ((N / a2) / L1−ΔN1) = (N / a2) / L2−ΔN2, and when ΔN2 is arranged, the following equation (6-12) is obtained.
ΔN2 = (N / a2) / (L2 * (L1 ′ / L2 ′) * ((N / a2) / L1-ΔN1)) (6-12)

Therefore, when this concept is applied to the calculation of the coring coefficient of this embodiment, the coring coefficient th in the high luminance side characteristic region is given by the following expression (6-13) according to the above expression (6-12). It will be a thing.
th = (N / a2) / (L2 * (L1 ′ / L2 ′) * ((N / a2) / L1-th0)) (6-13)

  FIG. 20 is a flowchart illustrating an example of operations mainly related to noise removal of the digital camera 1a according to the sixth embodiment. First, a captured image is obtained by imaging with the solid-state imaging device 3 (step S81). This captured image (base image I) is input to the gradation converting unit 7b (step S82). A high luminance side characteristic region (image I1; linear image) and a low luminance side characteristic region (image I2; linear image) are divided and extracted from the input base image I by the region division extraction unit 71 ′ (step S83). . When the divided and extracted image is an image of the high luminance side characteristic area (YES in step S84), the illumination component (L1) is extracted from the image I1 by the first illumination component extraction unit 74m ′ (step S85). Based on the illumination component information from the first illumination component extraction unit 74m ′, the coefficient calculation unit 761m ′ in the first noise removal unit 76m ′ calculates a coring coefficient corresponding to the degree of expansion in the logarithmic image gradation conversion. (Step S86). Based on the calculated coring coefficient, the coring processing unit 762m ′ performs a coring process on the reflectance component (R1) obtained by dividing the illumination component from the image I1 (step S87). The processed reflectance component (R1 ′) and the illumination component (L1 ′) obtained by compressing the illumination component extracted by the first illumination component extraction unit 74m ′ by the first illumination component compression unit 75m ′. Are combined by the multiplier 77m ′ to obtain an image I1 ′ (step S88).

  In the above step S84, when the divided and extracted image is an image of the low luminance side characteristic region (NO in step S84), the illumination component (L2) is extracted from the image I2 by the second illumination component extraction unit 74n ( In step S89, the coring processing for the reflectance component (R2) obtained by dividing the illumination component from the image I2 is performed by the coring processing unit 762n in the second noise removal unit 76n (step S90). However, as the coring coefficient in the coring process here, for example, a value (th0 described above) stored in advance as a fixed value in the coring processing unit 762n is used. The reflectance component (R2 ′) subjected to the coring process and the illumination component (L2 ′) obtained by compressing the illumination component extracted by the second illumination component extraction unit 74n by the second illumination component compression unit 75n. Are combined by the multiplier 782n to obtain an image I2 ′ (step S91). Then, the image I2 ′ in the low luminance side characteristic region and the image I1 ′ in the high luminance side characteristic region are combined by the image combining unit 73 ′ and output as the combined image O from the gradation converting unit 7b (step S92). ).

  As described above, according to the imaging device (digital camera 1, 1a) (imaging system 100) of each of the above embodiments, the noise removal units 6 and 6a (first noise removal units 76m and 76m ′) have a plurality of differences. Noise removal is performed on the captured image by changing a parameter relating to a noise removal amount of the captured image in accordance with a degree of expansion in gradation conversion of the captured image by the solid-state imaging device 3 having photoelectric conversion characteristics. As a result, in noise removal, it is possible to change the amount of noise removal according to the degree of expansion in gradation conversion (set a different amount of noise removal depending on the degree of expansion). The noise (noise component) can be made inconspicuous due to the difference in the degree of expansion, such as when the noise is noticeable in the image of the high-brightness part where the degree of expansion is large compared to the image of the part. An image can be obtained.

  The noise removal units 6 and 6a (first noise removal units 76m and 76m ′) include coefficient calculation units 64, 64m, 761m, and 761m ′ and coring processing units 63, 63m, 762m, and 762m ′. In addition, since the degree of expansion at the time of gradation conversion changes according to the luminance level of the photographed image, the coefficient calculators 64, 64m, 761m, and 761m ′ use the coefficient calculation units 64, 64m, 761m, and 761m ′ according to the luminance level corresponding to the degree of decompression of the photographed image. That is, for example, the coring coefficient (th) as a parameter relating to the amount of noise removal is calculated according to whether the image is a low-luminance part image or a high-luminance part image, and the coring processing units 63, 63m, 762m, Based on the coring coefficient calculated by the coefficient calculating unit, a coring process for removing noise components from the captured image is performed by 762m ′. Therefore, a general noise removal technique is used in which coring processing is performed by the coring processing units 63, 63m, 762m, and 762m ′ based on the coring coefficients calculated by the coefficient calculation units 64, 64m, 761m, and 761m ′. That is, with a simple configuration, noise removal that changes the amount of noise removal according to the degree of expansion can be easily performed.

  The noise removing units 6 and 6a include LPF units 62 and 62m (low frequency component extracting unit) and subtracting units 651 and 651m (high frequency component extracting unit). Low frequency components are extracted, and high frequency components are extracted from the captured image by the subtracting units 651 and 651m. Then, the coring coefficient (th) is calculated according to the level of the low frequency component corresponding to the luminance level by the coefficient calculating units 64 and 64m, and calculated according to the level of the low frequency component by the coring processing units 63 and 63m. Since the coring process is performed on the high frequency component based on the coring coefficient, the low frequency component information (level information) extracted from the photographed image using a low frequency component extracting means such as a general LPF is used. Thus, the coring coefficient can be easily calculated, and noise removal based on the calculated coring coefficient can be easily performed by performing a coring process on the high-frequency component extracted from the captured image. it can.

  The imaging apparatus also includes first illumination component extraction units 74m and 74m ′ (illumination component extraction means) that extract illumination components from the captured image, a subtraction unit 781m and a division unit 781m ′ (extraction of the reflectance component from the captured image. The coefficient calculation units 761m and 761m ′ included in the first noise removal units 76m and 76m ′ have a coring coefficient (th) according to the level of the illumination component corresponding to the luminance level. The coring processing for the reflectance component based on the coring coefficient calculated according to the level of the illumination component is performed by the coring processing units 762m and 762m ′ included in the first noise removing units 76m and 76m ′. Therefore, when removing noise, information on the illumination component and the reflectance component is used, that is, as in the present invention, a plurality of photoelectric conversion characteristics are used. Since the configuration uses the illumination component and the reflectance component generated in the process of the DR compression process, which is a gradation conversion process suitably used for an imaging apparatus that captures a wide dynamic range image using one imaging sensor, It is possible to include the first noise removal units 76m and 76m ′ (noise removal processing) in the conversion units 7a and 7b (gradation conversion processing), and a processing configuration can be provided without separately providing a processing device dedicated to noise removal. It can be simplified. In addition, the amount of calculation (calculation amount) in the noise removal processing is reduced, and the processing time can be shortened.

  Further, in a plurality of different photoelectric conversion characteristics in the solid-state imaging device 3, the coefficient calculation units 64, 64m, 761m, and 761m ′ have a noise amount after gradation conversion as the coring coefficient for the high luminance side characteristics. The coring coefficient (th) is calculated so that the low luminance part and the high luminance part in the characteristic are equal. In this way, the coring coefficient is calculated so that the amount of noise in the low luminance part and the high luminance part in the high luminance side characteristic is equal. It can be easily obtained (with a simpler calculation method), and based on the calculated coring coefficient, the difference in noise amount between the low luminance portion and the high luminance portion is prevented, that is, the high luminance portion. Therefore, the noise amount of the high luminance portion is not conspicuous because the noise amount is larger than the noise amount of the low luminance portion.

  Further, the solid-state imaging device 3 has a linear characteristic (linear photoelectric conversion characteristic) in which an electric signal is linearly converted with respect to an incident light amount and is output, and an electric signal is logarithmically converted with respect to the incident light amount. Since it has a photoelectric conversion characteristic composed of an output logarithmic characteristic (logarithmic photoelectric conversion characteristic), with respect to a captured image (linear / logarithmic image) obtained by the solid-state imaging device 3 having the linear characteristic and the logarithmic characteristic, Noise removal that changes the amount of noise removal according to the degree of expansion in the gradation conversion, that is, the degree of expansion in the gradation conversion of the logarithmic image, can be easily performed, so that a high-quality captured image can be obtained. .

  In addition, the solid-state imaging device 3 includes a first linear characteristic 171 (first linear photoelectric conversion characteristic) in which an electrical signal is linearly converted with respect to an incident light amount, and the first linear characteristic 171 and the graph. The second linear characteristic 172 (second linear photoelectric conversion characteristic) having a different slope has a photoelectric conversion characteristic, so that the solid-state imaging device 3 having the first linear characteristic 171 and the second linear characteristic 172 It is easy to remove noise by changing the noise removal amount of the obtained captured image (linear / linear image) in accordance with the degree of expansion in gradation conversion, that is, in accordance with the degree of expansion in gradation conversion of the second linear characteristic 172. As a result, a high-quality captured image can be obtained.

  Furthermore, in the imaging system 100, the information processing apparatus 200 configured to be able to exchange the captured image data with the digital camera 1b using the data transmission / reception means (input / output I / F units 12, 260) causes the above-described noise to the captured image. Since removal processing can be performed, the degree of freedom regarding the place where noise removal is performed is increased, and a configuration capable of performing higher-speed arithmetic processing (noise removal processing) using the information processing apparatus 200 such as a PC is easily realized. It becomes possible to do.

In addition, this invention can take the following aspects.
(A) In the first embodiment, the noise removal processing by the noise removal unit 6 and the gradation conversion processing by the gradation conversion unit 7 are performed in this order. Processing may be performed in the order of processing and noise removal processing. However, in this case, the noise component included in the image input to the noise removal unit 6 varies according to the gradation conversion process (the amount of noise varies depending on the degree of expansion due to gradation conversion). At this time, it is necessary to set the noise removal amount in consideration of the variation due to the gradation conversion and perform the processing.

  (B) In each of the above embodiments, the captured image is divided into images of the low luminance side characteristic region and the high luminance side characteristic region corresponding to different photoelectric conversion characteristics, and the low luminance and high luminance in the high luminance side characteristic region are divided. The coring coefficient is calculated according to the degree of expansion of each image (partial image) at the luminance level of the luminance so that no difference occurs in the amount of noise between the low luminance portion and the high luminance portion. First, the high luminance side characteristic area is set to an image having three luminance levels of more than the above two luminance portions, for example, the low luminance portion, the middle luminance portion, and the high luminance portion, or in all three luminance portions or the middle luminance portion and the high luminance portion. The coring coefficient may be calculated so as not to cause a difference in the amount of noise in the luminance part. Also, in the low luminance side characteristic area, for example, when noise is conspicuous due to the difference in the degree of expansion due to gradation conversion, the same noise removal method as the above high luminance side characteristic area can be applied (high luminance side characteristic This may be applied to both the area and the low luminance side characteristic area).

  (C) In each of the above embodiments, the photoelectric conversion characteristics of the solid-state imaging device 3 are composed of two different photoelectric conversion characteristics (linear characteristics and logarithmic characteristics, or first linear characteristics and second linear characteristics). You may consist of the above different photoelectric conversion characteristics. In this case, the same noise removal method as described above is also applied to each photoelectric conversion characteristic (for example, a photoelectric conversion characteristic in which noise is conspicuous due to a difference in the degree of expansion due to gradation conversion among the photoelectric conversion characteristics).

  (D) In each of the above embodiments, the amount of noise in the high luminance part having a high degree of expansion is made equal to the amount of noise in the low luminance part having a lower degree of expansion than that of the high luminance part. Although the configuration is such that the coring coefficient (noise removal amount) is changed (set), the present invention is not limited to this, and the noise amount (noise removal amount) corresponding to each part, that is, the local expansion degree (for example, the expansion rate; expansion amount). In other words, the coring coefficient may be changed in a so-called stepwise manner in accordance with the degree of expansion.

1 is a block configuration diagram mainly relating to imaging processing of a digital camera which is an example of an imaging apparatus according to a first embodiment. FIG. It is a schematic block diagram of the two-dimensional MOS type solid-state imaging device which is an example of the said solid-state image sensor. FIG. 3 is a circuit diagram illustrating a configuration example of each pixel G11 to Gmn illustrated in FIG. 2. It is a graph which shows an example of the photoelectric conversion characteristic of the said solid-state image sensor. It is a block block diagram of the noise removal part shown in FIG. It is a graph which shows the coring characteristic in a coring process. It is a block block diagram of the noise removal part which concerns on 2nd Embodiment. 6 is a flowchart illustrating an example of an operation mainly related to noise removal of the digital camera according to the first embodiment. It is a flowchart which shows an example of the operation | movement mainly regarding noise removal of the digital camera in 2nd Embodiment. It is a block block diagram mainly regarding an imaging process of the digital camera which is an example of the imaging device which concerns on 3rd Embodiment. FIG. 11 is a functional block diagram related to DR compression and noise removal in the gradation conversion unit shown in FIG. 10. It is a block block diagram of the 1st noise removal part shown in FIG. It is a block block diagram of the 2nd noise removal part shown in FIG. It is a flowchart which shows an example of the operation | movement mainly regarding noise removal of the digital camera in 3rd Embodiment. It is a block block diagram mainly regarding an imaging process of the imaging system which is an example of the imaging device which concerns on 4th Embodiment. It is a flowchart which shows an example of the operation | movement mainly regarding noise removal of the imaging system in 4th Embodiment. It is a graph which shows an example of the photoelectric conversion characteristic of the adaptive imaging sensor in 5th and 6th embodiment. It is a block block diagram of the gradation conversion part which concerns on 6th Embodiment. It is a block block diagram of the 1st noise removal part shown in FIG. It is a flowchart which shows an example of the operation | movement mainly regarding noise removal of the digital camera in 6th Embodiment.

Explanation of symbols

1, 1a Digital camera (imaging device)
1b Digital camera 3 Solid-state image sensor 6, 6a Noise removal unit (noise removal means)
62, 62m LPF (low frequency component extraction means)
63, 63m, 762m, 762m ′ coring processing unit (coring processing means)
64, 64 m, 761 m, 761 m ′ coefficient calculation unit (coefficient calculation means)
651, 651m subtraction unit (high frequency component extraction means)
7, 7a, 7b Gradation conversion unit 74m, 74m ′ first illumination component extraction unit (illumination component extraction means)
76m, 76m ′ first noise removing unit (noise removing means)
781m subtraction unit (reflectance component extraction means)
781m 'division unit (reflectance component extraction means)
12, 260 Input / output I / F section (data exchange means)
171 First linear characteristic (first linear photoelectric conversion characteristic)
172 Second linear characteristic (second linear photoelectric conversion characteristic)
100 Imaging system (imaging device)
200 Information processing device 630 Coring characteristic

Claims (7)

A solid-state imaging device having a plurality of different photoelectric conversion characteristics;
A noise removing unit that performs noise removal on the photographed image by changing a parameter relating to a noise removal amount of the photographed image according to a degree of expansion in gradation conversion of the photographed image by the solid-state imaging device ;
The noise removing means is
Coefficient calculating means for calculating a coring coefficient as a parameter relating to the amount of noise removal according to a luminance level corresponding to the degree of expansion of the captured image;
Coring processing means for performing coring processing for removing noise components from the captured image based on the coring coefficient calculated by the coefficient calculating means;
In the plurality of different photoelectric conversion characteristics,
The coefficient calculating means calculates different coring coefficients for the high luminance side characteristic and the low luminance side characteristic, and calculates the high luminance side coring coefficient based on a degree of expansion in gradation conversion of the captured image. An imaging device that is characterized. The noise removing means is
Low frequency component extraction means for extracting a low frequency component from the captured image;
A high frequency component extracting means for extracting a high frequency component from the photographed image;
The coefficient calculating means calculates the coring coefficient according to a level of the low frequency component corresponding to the luminance level,
The coring processing unit, the image pickup apparatus according to claim 1, characterized in that the coring processing for the high frequency component based on the coring coefficient calculated according to the level of the low frequency components. Illumination component extraction means for extracting an illumination component from the captured image;
A reflectance component extracting means for extracting a reflectance component from the photographed image;
The coefficient calculating means calculates the coring coefficient according to a level of the illumination component corresponding to the luminance level,
The coring processing unit, an imaging apparatus according to claim 1, characterized in that the coring processing for the reflectance component based on the coring coefficient calculated in accordance with the level of the illumination component. In the plurality of different photoelectric conversion characteristics,
The coefficient calculating means calculates a coring coefficient as the coring coefficient for the high luminance side characteristic so that the noise amount after the gradation conversion is equal in the low luminance part and the high luminance part in the high luminance side characteristic. The imaging device according to any one of claims 1 to 3 , wherein The solid-state imaging device has a linear photoelectric conversion characteristic in which an electric signal is linearly converted with respect to an incident light amount, and a logarithmic photoelectric conversion in which the electric signal is logarithmically converted with respect to the incident light amount. the imaging apparatus according to any one of claims 1 to 4, characterized in that one having a photoelectric conversion characteristic comprising a characteristic. The solid-state imaging device includes a first linear photoelectric conversion characteristic in which an electrical signal is linearly converted with respect to an incident light amount, and a second linear photoelectric conversion characteristic that is different in slope from the first linear photoelectric conversion characteristic. the imaging apparatus according to any one of claims 1 to 5, characterized in that one having a photoelectric conversion characteristic composed of a linear photoelectric conversion characteristics. A predetermined information processing apparatus configured to be able to exchange image data with the imaging apparatus main body using predetermined data transfer means;
The imaging apparatus according to any one of claims 1 to 6, characterized in that the noise removal with respect to the photographed image by the information processing apparatus.

Images