JP2002112108A - Image processing unit - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing unit, having a wide dynamic range that can obtain sharp images with less noise. SOLUTION: The image processing unit 10, including an image pickup section 5 that picks up an image of an object for generating an imaged signal is provided with a sharpness emphasis section 8, that emphasis the sharpness of the image picked up signal, an exposure ratio detecting section 7 that detects the exposure ratio among picked-up images and a compositing section 9 that composites the image pickup signals, the sharpness of which is emphasized by the sharpness emphasis section 8, depending on the exposure ratio.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image processing apparatus constituting a video camera, a digital still camera or the like using a solid-state imaging device such as a CCD.

[0002]

2. Description of the Related Art Hitherto, highly reliable image pickup devices using a solid-state image pickup device such as a CCD have been widely used at low cost, and in recent years, attention has been paid to a device that takes a silver halide photograph. However, a solid-state imaging device such as a CCD has a narrow dynamic range with respect to incident light as compared with a silver halide film or the like. Therefore, when a bright subject and a dark subject are mixed in the angle of view, a bright subject is overexposed, A dark subject may be crushed.

[0003] As described above, in a device using a solid-state image sensor, the dynamic range is not as high as that of a device for taking a silver halide photograph. Here, JP-A-1-93967 and JP-A-7-1317 are disclosed.
Japanese Patent Application Publication No. 99-1999 discloses a method of enlarging a dynamic range by photographing a subject a plurality of times with different exposure amounts using a shutter, an aperture, and the like, and generating a composite image using a plurality of images obtained by the photographing. Is disclosed.

[0004] Japanese Patent Application Laid-Open No. 5-64070 discloses that
A method of synthesizing images with different exposure amounts using a beam splitter is also disclosed. And Japanese Patent Laid-Open No. 11-155
No. 108 discloses an application example of a three-panel camera.

[0005] These are techniques for creating an image with a wide dynamic range by synthesizing images captured at a plurality of different exposures. Specifically, an image is captured at a standard exposure using an imaging unit. After obtaining a standard captured image and a non-standard captured image captured with an exposure of 1 / K (K is an arbitrary natural number), the exposure ratio K
Is multiplied as a gain to create a normalized non-standard captured image.

In this technique, a wide dynamic range image is created by replacing a portion of the standard captured image where the image signal is saturated with the same portion of the normalized non-standard captured image. . As a result, even in a luminance portion saturated with a standard captured image captured with a standard exposure amount, the image can be reproduced based on the information of the replaced normalized non-standard captured image. To achieve a wider dynamic range.

Although the above example is a method of combining a plurality of captured images in principle, various problems arise in practice. That is, since it is difficult to obtain an appropriate exposure amount or exposure ratio when capturing individual images, it may not be possible to combine these images neatly. In addition, there is a problem that noticeable noise is generated in the obtained composite image.

To solve such a problem, Japanese Patent Laid-Open Publication No.
Japanese Patent Application Publication No. 708 discloses a technique in which an exposure ratio K for normalizing a non-standard captured image is calculated from the standard captured image and the non-standard captured image and multiplied by the non-standard captured image. In this technique, in order to accurately obtain the value of K, an imaging signal having a non-saturation level between a low level (noise level) and a high level (saturation level) is used.

Japanese Patent Application Laid-Open No. 11-191860 discloses that a saturated portion is not simply replaced with a normalized non-standard captured image, but a standard captured image using a weighting function corresponding to a light level. A method is disclosed in which the weighted average of the normalized non-standard captured image and the normalized non-standard captured image is used to smooth the appearance of the switching portion between the standard captured image and the normalized non-standard captured image.

In this method, by adjusting the mixing ratio (synthesis ratio) between the standard captured image and the normalized non-standard captured image in accordance with the level of the captured signal, the standard captured image is used in the dark portion, and the intermediate portion is used. Creates an image to be synthesized with an internally dividing ratio, and uses a normalized non-standard captured image in the bright part.

In general, the purpose of creating a wide dynamic range image from a plurality of captured images is to generate an image with less noise in a wide captured luminance range. At this time, if it is simply taken without saturating a subject in a wide luminance range, it is considered that it is sufficient to take an image with a low exposure amount. However, since the signal level is small at a low exposure amount, the S / N ratio is low in a dark part. Is reduced.

On the other hand, Japanese Patent Application Laid-Open No. H11-155098 discloses a method of adjusting an aperture gain and a coring level in a composite image based on a mixture ratio of a standard captured image and a non-standard captured image. That is, in this method, the coring level in the non-standard captured image and the standard captured image is set on the assumption that the non-standard captured image captured with a small exposure amount contains more noise than the standard captured image, By changing the coring level based on the mixture ratio of the non-standard captured image and the standard captured image, the sharpness is enhanced without enhancing the noise level.

In general, the signal level increases as the exposure amount increases, but the absolute level of noise also increases as the exposure amount increases.
The / N ratio is not improved even when the exposure amount increases.

Here, noise generated in the image sensor is divided into random noise and fixed pattern noise, and random noise is classified into dark noise and light shot noise.
The dark noise does not depend on the exposure, but the light shot noise increases in proportion to the square root of the exposure.

The fixed pattern noise is divided into dark current unevenness and sensitivity unevenness, and dark current unevenness does not depend on the exposure amount, but sensitivity unevenness increases in proportion to the exposure amount. Also, the absolute value of amplifier noise generated in an amplifier that amplifies an image signal obtained from an image sensor generally increases as the image signal increases, but the quantization noise when quantizing the image signal is large. It does not depend on the magnitude of the imaging signal.

That is, in general, the absolute amount of noise of the imaging information in each pixel included in the imaging element increases as the level of the imaging signal increases, but on the other hand, there is also noise independent of the level of the imaging signal. Therefore, the S / N ratio increases as the level of the imaging signal increases.

When a wide dynamic range image is created based on imaging signals constituting a plurality of images, a standard exposure amount and an image captured with a smaller exposure amount are different from each other. Although the absolute amount of noise is larger in the image captured according to the amount, the amount of increase in the imaging signal is larger than the amount of increase in noise.
The / N ratio is larger for images obtained with the standard exposure.

In general, the deterioration perceived by human eyes due to image noise is determined by the S / N ratio rather than the absolute amount of the noise. The larger the S / N ratio, the less noticeable the noise. In order to correct blurring of an image caused by an imaging optical system and deterioration of contrast caused by a light receiving area of an imaging element, a general imaging apparatus performs image processing called aperture correction, which emphasizes image sharpness. Have been done. However, when the sharpness is enhanced, there is a problem that noise is also enhanced at the same time.

For this reason, when the change in the imaging information is close to the noise level, sharpness is not emphasized, and a process called a coring process is performed so as to make noise in a flat portion of the image where there is no change in the imaging information stand out. Are executed simultaneously.

When the coring process is executed in an image pickup apparatus for creating a wide dynamic range image, it is necessary to appropriately grasp the noise level and determine the coring level. 15509
Japanese Patent Publication No. 8 discloses a configuration for changing the coring level, but does not mention a method for setting the coring level appropriately.

[0021]

SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing apparatus which has a wide dynamic range and can obtain a clear image with little noise.

[0022]

SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing apparatus including an image pickup means for picking up an image of a subject and generating an image pickup signal. An exposure ratio detecting unit that detects an exposure ratio in a plurality of imaging based on a signal, and an image combining unit that combines a plurality of imaging signals whose sharpness has been enhanced by the sharpness enhancing unit in accordance with the exposure ratio. This is achieved by providing an image processing apparatus characterized by the above.

According to such a means, an appropriate composite image can be obtained irrespective of the exposure ratio of a plurality of imaging signals.

Further, if the sharpness enhancing means enhances the sharpness of the image signal based on the noise level corresponding to the magnitude of the image signal, unnecessary amplification of noise can be avoided. .

It is another object of the present invention to provide an image processing apparatus including an image pickup means for picking up an image of a subject and generating an image pickup signal, wherein an exposure ratio detection for detecting an exposure ratio in a plurality of image pickups based on the image pickup signal. Means, an image synthesizing means for synthesizing the plurality of imaging signals according to the exposure ratio to generate a synthesized image, and detecting a mixing ratio between the plurality of imaging signals based on the synthesized image,
This is attained by providing an image processing apparatus comprising: a sharpness enhancement unit that enhances the sharpness of a composite image according to the detected mixture ratio.

According to such a means, since the sharpness is enhanced by the mixture ratio in the composite image, the sharpness of the composite image can be easily and appropriately enhanced.

If the sharpness enhancement means enhances the sharpness of the composite image based on the noise level corresponding to the composite image and the exposure ratio, the sharpness of the composite image is enhanced without amplifying the noise. can do.

Further, the image processing apparatus further includes an averaging means for averaging a plurality of imaging signals obtained by imaging the subject a plurality of times under a predetermined exposure amount for each pixel to generate an average imaging signal. If the combining means combines a plurality of average image signals generated by the averaging means based on the image signals captured under different exposure amounts, it is possible to suppress noise that occurs randomly.

Further, if the image pickup means is further provided with an image pickup number determining means for determining the number of times the image pickup means takes an image of a subject under a predetermined exposure amount, the influence of noise generated at random is suppressed. An image can be obtained efficiently.

[0030]

Embodiments of the present invention will be described below in detail with reference to the drawings. In the drawings, the same reference numerals indicate the same or corresponding parts.

There have been proposed various methods for creating a wide dynamic range image based on image signals obtained by imaging a plurality of times with different exposure amounts or using a plurality of image sensors. Neither method mentions a change in a noise component included in the composite image.

Therefore, by paying attention to the change of the noise component with respect to the exposure amount and adopting a method of enhancing the sharpness according to the noise component, a wide dynamic range image in which the noise is inconspicuous can be created.

In the following, a case will be described in which a wide dynamic range image is created based on a plurality of imaging signals obtained by imaging with basically different exposure amounts. First, a first imaging signal captured with a certain exposure amount and a second imaging signal captured with an exposure amount of 1 / K (K is a natural number) with respect to the imaging time of the first imaging signal are obtained.

Next, the second image pickup signal is multiplied by K and normalized. Thereby, as shown in FIG. 11, the unsaturated portion of the graph G1 showing the first imaging signal and the unsaturated portion of the graph G3 showing the normalized second imaging signal are linearly shown on the diagram. Overlap. Here, a case in which a wide dynamic range image is created from the two imaging signals will be considered.

At a point Pa in FIG. 11 where the first image signal and the second image signal overlap, if the pure signal level of the first image signal that has no noise component is Sa, the second image signal Is Sa / K. However, since a noise component actually exists, the magnitude of the first image pickup signal Va1 is expressed as N1 (Sa).
+ N1), and the magnitude of the second imaging signal Va2 is (Sa / K + N2), where N2 is the noise component.

Next, the magnitude of the second image signal Van2 normalized at the point Pa is (Sa + K × N2) by normalizing the second image signal by K times. That is, while the magnitude of the noise in the first imaging signal is N1, the magnitude of the noise in the normalized second imaging signal is (K × N2).

Accordingly, when it is assumed that the amount of noise due to imaging does not change with respect to the amount of light incident on the imaging device, the amount of noise in the second imaging signal is K times larger than that in the first imaging signal.

However, in general, the absolute amount of noise tends to increase with respect to the amount of incident light. Assuming that the function of noise with respect to the incident light amount I is N (I), FIG.
The noise amount of the first image pickup signal at the point Pa shown in (2) is N (Ia), and the noise amount of the normalized second image pickup signal is K × N (Ia / K).

On the other hand, when the amount of noise is smaller than the amount of incident light, the amount of incident light is proportional to the generated image signal (output signal V).

As described above, assuming that the function of the noise amount with respect to the output signal V is N (V), the noise amount of the first image pickup signal at the point Pa shown in FIG. The noise amount of the second imaging signal is K × N (Va / K).

That is, when the noise amount function N (V) for the output signal V is obtained, the noise amount can be estimated according to the magnitude of the output signal V. Here, as described above, noise is divided into fixed pattern noise that does not change over time and random noise, but random noise is not uniquely determined, so a statistical distribution is used as a function representing the amount of noise, Determine the range of variation. As described above, the maximum noise amount due to the output signal can be estimated by measuring the characteristic of the noise amount according to the output signal.

That is, the coring level is determined from the function N (V) for the noise amount in the first imaging signal, and K × N ( V / K), the coring level can be determined. Then, by determining the coring level at the time of sharpness enhancement based on the estimated maximum noise amount, it is possible to enhance the sharpness without enhancing the noise in the flat portion where the change of the imaging signal is small.

Thus, according to the present invention, by setting the optimum coring level by estimating the noise level, the sharpness is appropriately enhanced to obtain a good wide dynamic range image. Hereinafter, the image processing apparatus according to the embodiment of the present invention will be described more specifically. [First Embodiment] FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to a first embodiment of the present invention. As shown in FIG. 1, an image processing apparatus 10 according to the first embodiment
Includes an imaging unit 5, an averaging unit 6, an exposure ratio detection unit 7, a sharpness enhancement unit 8, a synthesis unit 9, and a switching unit SW.

Here, the switching unit SW is connected to the imaging unit 5, and the averaging unit 6, the exposure ratio detection unit 7, and the sharpness enhancement unit 8 are connected to the switching unit SW. Further, the synthesizing unit 9 is connected to the exposure ratio detecting unit 7 and the sharpness enhancing unit 8.

The image processing apparatus 10 having the above configuration can be realized, for example, by the circuit configuration shown in FIG. That is, the image processing apparatus 10 includes the lens 11 and the imaging element 3 that constitute the imaging unit 5, an amplifier (AGC) 12, a sample and hold circuit (CDS) 13, an A / D converter 14, and an image processing C
It can be constituted by a PU 15, a random access memory (RAM) 16 and a card RAM 17 connected to the image processing CPU 15.

FIG. 3 is a diagram showing a configuration of the image pickup section 5 shown in FIG. As shown in FIG. 3, the imaging unit 5 includes a lens 1, a divider (subject image divider) 2, and two imaging elements 3 and 4.

In the image pickup section 5 having such a configuration, the light incident through the lens 11 forming the photographing optical system is divided into two paths by the divider 2, and the object is respectively transferred to the two image pickup devices 3 and 4. An image is formed.

On the other hand, the image pickup section 5 shown in FIG. 1 can be composed of the lens 1 and the image pickup element 3 as shown in FIG. In this case, the image pickup device 3
The pulse S in the exposure timing signal shown in FIG.
The exposure operation is performed a plurality of times in accordance with the edges of S1, S2 (time T1, T2), and as shown in FIG.
And the imaging signals VS1 and VS2 at time T2, respectively.
Generate

Next, an imaging method by the imaging unit 5 will be described. As shown in FIG. 6, when a general solid-state imaging device such as a CCD is used, the output level of the imaging signal is determined according to the amount of incident light within a certain range.
When the amount of incident light exceeds a certain value, the output level is saturated and exhibits a non-linear characteristic. A signal level indicating such non-linear characteristics is called a saturation level, and a level indicating other linear characteristics is called a non-saturation level.

Here, in a general image pickup apparatus, it is necessary to limit the amount of incident light using a diaphragm, a shutter, or the like so that the output level falls within the range of the non-saturation level.
That is, for example, as shown in FIG. 7A, when the incident light amount from the subject to be imaged exceeds the non-saturation level in a part of the subject image and becomes equal to or more than the critical incident light amount IPc. As shown in FIG. 7B, a portion 21 exceeding the critical output signal Sc corresponding to the critical incident light amount IPc.
However, it is distorted due to saturation.

Next, considering the noise included in the image pickup signal, when a solid-state image pickup device or the like is used as the image pickup section 5, various noises are generated in the process of photoelectrically converting the subject image formed on the image pickup device. Is included. Here, these noises are roughly classified as shown in FIG. Here, the fixed pattern noise is noise that causes a constant error, and can be reduced by subtracting these errors from the output signal in normal image processing.

The quantization noise is a quantization error that occurs when an obtained analog image signal is converted into a digital signal, and can be reduced by increasing the quantization accuracy.

The random noise is classified into light shot noise that depends on the illuminance of the subject and dark noise that does not depend on the illuminance of the subject. Here, the light shot noise Ns that depends on the subject illuminance is proportional to the square root of the amount of charge (signal magnitude) S generated in the image sensor 3, and when a constant independent of the subject illuminance is Nsc, the following equation ( Indicated by 1). Ns = Nsc × S ^1/2 (1) From this, the following equation (2) is obtained. S / Ns = S ^1/2 / Nsc (2) As described above, the light shot noise Ns increases as the signal increases, but the S / N ratio increases in proportion to the square root of the signal. Incidentally, the noise N _D dark that is independent of the object illuminance is to become constant regardless of the magnitude of the signal, S / N ratio increases in proportion to the signal.

FIG. 9 is a graph showing the relationship between the signal magnitude S and the S / N ratio for dark noise and light shot noise. Here, the S / N ratio S / N for dark noise
N _D is indicated by graph 22, S / N ratio S / Ns to optical shot noise is indicated by the graph 23.
Note that the noise is less noticeable as the S / N ratio is larger,
The smaller the N ratio, the more noticeable, and basically, the larger the signal amount, the less noticeable the noise.

Here, as shown in FIG. 9, when the signal is smaller than (N _D / Nsc) ² , the dark noise N _D
Is dominant, and when large, the light shot noise Ns is dominant.

As described above, with respect to noise in an image, it is preferable that the magnitude of the signal be increased by accumulating electric charges in the image sensor as much as possible. However, in consideration of the above-described saturation, the output signal is saturated. Since it is necessary to perform exposure without the need, an optimum range of the amount of incident light on the image sensor is determined.

Next, a method of creating a new image based on a plurality of imaging signals obtained by imaging with different exposure amounts will be described. The imaging unit 5 guides light, which is obtained by restricting light from a subject using a shutter or an aperture, to an image sensor, and forms an image of the subject on the image sensor.

At this time, generally, when an object having a luminance I is picked up, an electric charge accumulated in the image sensor is equal to an exposure amount E obtained by multiplying the illuminance L of the object image by the exposure time T in an unsaturated region.
, And the output signal V is proportional to the exposure E, so that the following relational expression (3) holds. V∝L × T (= E) (3) In the light from the subject having the luminance I, the illuminance L of the subject image is controlled by the aperture, and the exposure amount T is controlled by the shutter.

Here, FIG. 10A shows the output signal from the image pickup section 5 when the light incident from the object to the image pickup element is controlled so that the exposure amount becomes E1 by using a shutter or a diaphragm. This shows the object luminance dependency. next,
FIG. 10B shows the subject luminance dependence of the output signal when the exposure is performed with the exposure amount E2 of E1 / K.

As shown in FIG. 10A, the exposure amount E
At the time of 1, the imaging signal output is saturated at V1 or more,
When the brightness of the subject is I1, the imaging output signal is V1. Similarly, when the exposure amount is E2, the imaging signal output is saturated at V2 or higher, and when the luminance of the subject is I2, the imaging signal output becomes V2. At this time, the subject luminance I2 is K times the subject luminance I1. That is, naturally, when the exposure amount is set to 1 / K, the saturation does not occur to a range where the subject brightness becomes K times.

Next, FIG. 11 shows a graph G1 shown in FIG. 10A and a graph G3 obtained by amplifying the imaging signal output by K times in the graph G2 shown in FIG. 10B. Is done. As shown in FIG. 11, the graphs G1 and G3 overlap up to the subject brightness I1, the output signal at the exposure E1 is saturated above the subject brightness I1, and the output signal at the exposure E2 is the subject brightness. Does not saturate until I2.

In the above description, the generation of noise in the process of photoelectric conversion of light from a subject is not described, but noise is generated due to the photoelectric conversion as described above.

Here, fixed pattern noise and quantization noise
Considering random noise N excluding noise,
Is given by the following equation (4). N = (N_D ²+ Ns²)^1/2 (4) Then, the following equation (5) can be obtained from the above equation (1).
Wear. N = (N_D ²+ Nsc²・ S)^1/2 (5) Also, when the subject luminance is Ia,
Signal (excluding noise) is Sa
The random noise N1 at the time of E1 is expressed by the following equation (6).
As shown. N1 = (N_D ²+ Nsc²・ Sa)^1/2 (6) Similarly, random noise at the exposure amount E2
N2 is represented by the following equation (7). N2 = K · (N_D ²+ Nsc²・ (Sa / K))^1/2  = (K²・ N_D ²+ K · Nsc²・ Sa)^1/2 (7) From the above equation (7), when K> 1, the noise is higher than the noise N1.
It can be seen that N2 is larger. In other words,
When the degree is equal to or less than I1, the image is captured with the exposure amount E2.
Exposure than the signal obtained by multiplying the signal obtained by
The signal obtained by imaging with the amount E1 is more noise.
Is small.

Therefore, in the image pickup signal obtained by imaging with the exposure amount E1 and the exposure amount E2, when the object luminance is equal to or less than I1, the image pickup signal obtained under the exposure amount E1 is used. Is equal to or greater than I1, the exposure amount E
By using the imaging signal obtained under the condition 2, it is possible to obtain the subject luminance information of a wide dynamic range and obtain a good image with little noise in a dark part.

Here, the two imaging signals obtained under the exposure amounts E1 and E2 have different noise magnitudes. The noise n is represented by a function n (S) that depends on the magnitude of the imaging signal and a constant n0 that does not depend on the function.
It can be expressed by the following equation (8). n = n0 + n (S) (8) The output signal V is expressed by the following equation (9). V = S + n0 + n (S) (9) The signal Va1 captured at the point Pa shown in FIG. 11 under the exposure amount E1 is represented by the following equation (10). Va1 = Sa + n0 + n (Sa) (10) On the other hand, the signal Va2 captured at the point Pa under the exposure amount E2 is Va2 = Sa / K + n0 + n (Sa / K) (11), and the normalized signal Van2 is It is shown by equation (12). Van2 = Sa + K × n0 + K × n (Sa / K) (12) That is, as shown in FIG. 12, between the image (dark portion) captured by the exposure E1 and the image captured by the exposure E2, At the critical luminance Ic, a discontinuity occurs in the possible noise width 24. That is, when the image captured simply by the exposure amount E1 and the image captured by the normalized exposure amount E2 are combined as in the related art, the magnitude of the noise is large at the switching point of the combination (critical luminance Ic). Will change.

Therefore, in the image processing apparatus according to the present invention, in order to solve such a problem, a good wide dynamic range image is created by basically performing two approaches. A first approach is a method of enhancing sharpness in accordance with a change in noise, and a second approach is a method of reducing the magnitude of noise of an image signal to be synthesized.

Here, the first approach will be described first. As described above, when a digital image is generally created, a sharpness enhancement process called aperture correction is performed on a captured image.

For example, a signal obtained by multiplying the image pickup signal of the pixel of interest by three times the image pickup signals of the adjacent pixels on both sides of the pixel of interest (−1) is added to the image pickup signal of the pixel of interest. When sharpness is enhanced by using a so-called (-1,3, -1) filter, for example, in an image (imaging signal) shown in FIG. 13A, the sharpness of an edge portion of the image is enhanced. 13 (b) can be obtained.

However, in the case of a flat image with little change in the magnitude of the imaging signal as shown in FIG. 14A, even if the sharpness is enhanced as shown in FIG. Although a flat image is obtained, when noise is included in the flat image as shown in FIG. 15A, the noise is emphasized as shown in FIG. I will. Therefore, a process called coring is executed.

In this processing, first, the aperture level AP is obtained by the following equation (13) based on the imaging signal level D _{0 of the} pixel of interest and the imaging signal levels D ₋₁ and D ₁ of the pixels before and after it. AP = −D ₋₁ + 2D ₀ −D ₁ (13) If the absolute value of the aperture level AP obtained by the above equation (13) is smaller than the set coring level C, the aperture is set to the target pixel. If the absolute value of the aperture level AP is equal to or higher than the set coring level C without adding the level AP, a process of adding the aperture level AP to the target pixel is executed.

Thus, as shown in FIG. 16, in a flat portion where the aperture level AP is smaller than the coring level C, the increase Da of the imaging signal level is set to 0, so that noise enhancement is avoided and the aperture is reduced. Level AP
Since the increase Da is proportional to the aperture level AP at an edge portion where is larger than the coring level C, so-called image edge enhancement can be realized. Note that even if a noise component is included in the edge portion, the noise does not stand out because the signal change is large.

That is, in the above processing, the noise level is grasped by obtaining the aperture level AP, and when the imaging signal changes below the set coring level, the sharpness is not emphasized, Emphasizes sharpness when changing more than level.

However, the noise is generally classified into random noise and non-random noise, and the noise is not constant, and the magnitude of the noise depends on the magnitude of the imaging signal and does not depend on the magnitude of the image signal.

When an image with a wide dynamic range is created, it is necessary to combine signals obtained by imaging under different exposure amounts. That is, if aperture processing is simply performed by a conventional method on a synthesized image obtained by synthesizing signals of different exposure amounts, appropriate coring processing cannot be performed.

In other words, when combining the standard image information imaged with the first exposure amount and the non-standard image information imaged with a 1 / K exposure amount relative to the first exposure amount, After normalizing the non-standard image information by K times as described above, the non-standard image information is synthesized. When the conventional aperture processing is performed thereafter, the noise level of the standard image information obtained by n0 + n (Sa) and K × n0 + K × n (Sa / K)
The sharpness is enhanced under the same coring level as the noise level of the normalized non-standard image information obtained by

When K> 1, n0 + n
Since (Sa) <K × n0 + K × n (Sa / K), if the coring level is set in accordance with the noise level of the standard image information, the noise level of the normalized non-standard image information becomes the coring level. If it becomes larger, noise in the non-standard image information will be emphasized.

Therefore, in the image processing apparatus according to the first embodiment of the present invention, prior to the synthesis, the sharpness enhancement section 8
A method of enhancing sharpness is adopted.

FIG. 16 is a block diagram showing the configuration of the sharpness enhancement section 8 according to the first embodiment. As shown in FIG. 16, the sharpness enhancement unit 8 includes a color processing unit 81 and an aperture processing unit 82. Here, an imaging signal (imaging information) is supplied from the imaging unit 5 to the color processing unit 81, and RGB information is supplied from the color processing unit 81 to the aperture processing unit 82. Then, the signal processed by the aperture processing unit 82 is supplied to the synthesizing unit 9.

In general, when color information is obtained by the imaging unit 5, photoelectric conversion is performed with different spectral sensitivities for each of the RGB components as shown in FIG. 21, so that a CCD or the like as shown in FIG. A solid-state imaging device having a color filter that selectively transmits RGB components on the solid-state image sensor is used.

A signal obtained by using such an image sensor is converted into digital information corresponding to each pixel as shown in FIG. Then, the color processing unit 81 shown in FIG. 17 performs an interpolation process called a color process on the digital information, and performs an R process as shown in FIG.
The information of three planes for each GB component is created. Here, FIG. 19A illustrates information (R plane) indicating only the R component for each pixel, and FIG. 19B illustrates information (G plane) indicating only the G component for each pixel. c) B for each pixel
Represents information (B plane) indicating only components.

Next, the aperture processing unit 82 shown in FIG. 17 performs the above-described aperture correction on the created RGB information. Here, a plurality of pieces of imaging information captured under a plurality of different exposure amounts are respectively subjected to color processing, and aperture processing is performed on each of the obtained plurality of RGB information. The color processing may employ a conventional method. For example, when the imaging information of the target pixel is only the R component, the G component and the B component are interpolated from the G component and the B component of the surrounding pixels. Can be sought. The aperture correction is performed on each plane of the plurality of created RGB information.

For example, when creating a wide dynamic range image based on an image signal obtained by imaging twice with different exposure amounts, the synthesizing unit 9 is imaged with the first exposure amount and has a sharpness Standard image information with
When synthesizing the non-standard image information which is imaged at an exposure amount of 1 / K with respect to the first exposure amount and whose sharpness is emphasized, it is multiplied by K in order to normalize the non-standard image information as described above. And combine them.

When the sharpness is emphasized after simply combining a plurality of pieces of image information as in the prior art, the sharpness is emphasized after setting a constant coring level. Cannot be enhanced, but if the sharpness enhancement is performed prior to the image synthesis, n0
The sharpness is emphasized at the same coring level as the noise level of the standard image information determined by + n (Sa) and the noise level of the non-standard image information before normalization determined by n0 + n (Sa / K). n0 + n (S
a)> n0 + n (Sa / K), so that noise of non-standard image information is not emphasized.

More precisely, the sharpness can be more appropriately emphasized by setting the coring level according to the signal level. That is, as shown in FIG. 22, by changing the coring level C according to the level L of the signal actually output from the imaging unit 5, the sharpness can be appropriately emphasized. Here, the coring level determined by the noise level no which does not depend on the magnitude of the output signal is C0, and a function empirically set by the noise level n (S) which depends on the magnitude of the output signal is fc
Assuming (L), the coring level C is expressed by the following equation (14). C = C0 + fc (L) (14) The aperture level AP is the imaging signal D ₀ of the target pixel.
And the imaging signals D ₋₁ and D ₁ of the pixels before and after the pixel, the level L of the output signal corresponding to each pixel is set to (2 ×
D ₀ + D ₋₁ + D ₁ ) / 4, or the imaging signal D ₀ can be used as a representative value.

Next, a description will be given of the second approach for reducing the magnitude of noise in the image signal to be synthesized. When combining information obtained by imaging under different exposures, a non-standard image is captured under an exposure of 1 / K with respect to a standard image. Thus, n0 + n (Sa) <K × n0 + K × n (Sa /
K).

Therefore, for non-random noise such as fixed pattern noise, the noise component can be recorded for each pixel of the CCD and corrected according to the value of the recorded noise component. Cannot be removed.

Here, the random noise of the standard image is n0
+ N (Sa), normalized random noise is K × n0
+ K × n (Sa / K), m
The value obtained by averaging the imaging signals obtained by performing the imaging twice for each pixel is Sa + (n0 + n (Sa)) / m1 / ² , and the noise component is 1 / ^m1 / ² . Similarly, 1 / K
The average value of the imaging signals obtained by imaging n times with the exposure amount of each pixel is Sa + (K × n0 + K × n (S
a / K)) / n ^{1/2 and the} noise component is 1 / n ^{1
/ 2} .

Then, the condition that both noises are the same
Is represented by the following equation (15). (N0 + n (Sa)) / m^1/2= (K × n0 + K × n (Sa / K)) / n^{1 / 2}  .. (15) Then, the following equation (16) is established from equation (15). m / n = (K × n0 + K × n (Sa / K)) / (n0 + n (Sa))²  (16) Here, as described above, noise depending on the magnitude of the imaging signal
Is the light shot noise, the magnitude of which is the square root of the amount of light.
Since it is proportional, the following equation (17) holds. m / n = (K × n0 + K^1/2× n (Sa)) / (n0 + n (Sa))²  (17) Therefore, when the exposure ratio K is determined, m and n are calculated according to Expression (17).
Is determined. That is, the ratio of m and n is changed according to the exposure ratio K.
By determining, the randomness of the standard image and the non-standard image
The noise level can be the same. The above
The number of times of imaging m and n is determined by the image processing C shown in FIG.
It is calculated by the synthesis unit 9 included in the PU 15.

The above are the principles of the above two approaches. Hereinafter, the configuration and operation of the image processing apparatus according to the first embodiment shown in FIG. 1 will be described.

As shown in FIG. 1, a plurality of imaging signals (imaging information) generated by the imaging unit 5 are input to an exposure ratio detection unit 7 and a sharpness enhancement unit 8. Then, the exposure ratio detecting section 7 detects the exposure ratio K, and the sharpness emphasizing section 8 emphasizes the sharpness by the method described above.

Here, the exposure ratio detecting operation by the exposure ratio detecting section 7 will be described. The exposure ratio detection unit 7 calculates the average value of the imaging signal of the non-saturated portion in the standard image and the average value of the imaging signal in the corresponding portion of the non-standard image, and obtains the exposure ratio K by calculating the ratio of these average values. Can be detected.

Next, the synthesizing unit 9 selects the standard image for the non-saturated portion in the standard image, and converts the exposure ratio K calculated by the exposure ratio detecting unit 7 into the non-standard image for the saturated portion. Multiply and select this. Then, by performing such processing for each pixel, a wide dynamic range image can be created.

FIG. 1 shows a case where a wide dynamic range image is formed by an image obtained by imaging m times with a fixed exposure amount and an image obtained n times with different exposure amounts. By switching the switching unit SW according to the control signal, the imaging signal generated by the imaging unit 5 is supplied to the averaging unit 6. Then, the averaging unit 6
An average is obtained for each pixel in a plurality of pieces of imaging information, and an average image is created.

Then, a series of image processing as described above is performed by
This is executed using the image processing CPU 15 shown in FIG. That is, the image processing apparatus shown in FIG. 2 includes a lens 11 for forming a subject image, an image sensor 3 for photoelectrically converting the subject image, and an amplifier (AGC) for amplifying an image signal obtained by the image sensor 3. ) 12, an analog circuit including a sample hold circuit (CDS) 13 and an A / D converter 14, an image processing CPU 15 for digitally processing quantized information, and an image processing RAM 16 for temporarily storing an image. And a card RAM 17 for recording the created image.

Hereinafter, the operation of the image processing apparatus according to the first embodiment will be described with reference to the flowchart of FIG. First, in step S1, the imaging device 3 included in the imaging unit 5 generates a plurality of imaging signals (imaging information) by imaging under different exposure amounts, and the RAM 16 stores the imaging signals.

Next, in step S2, the image processing CPU
The exposure ratio detector 7 included in 15 calculates an exposure ratio between a plurality of images stored in the RAM 16 according to the ratio of the average value of the unsaturated portion. Then, in step S3,
The image processing CPU 15 executes a color interpolation process for enhancing the sharpness of the image. Then, in this color interpolation processing, based on digital information corresponding to each pixel as shown in FIG. 18, the RGB components shown in FIG.
Create a GB image.

Next, in step S4, image processing C
The sharpness enhancement unit 8 included in the PU 15 determines the coring level based on the imaging signal (imaging information) as described above, and enhances the sharpness. Then, in step S5, the synthesizing unit 9 included in the image processing CPU 15 selects the standard image in units of one pixel with respect to the non-saturated portion of the standard image,
In the saturated portion of the standard image, a synthesized image is created by selecting a non-standard image normalized by multiplying the non-standard image by the exposure ratio detected by the exposure ratio detection unit 7 in units of one pixel.

According to the operation as described above, the first embodiment
According to the image processing apparatus according to the above, after setting an appropriate coring level according to the exposure ratio in the plurality of captured images to enhance the sharpness, the plurality of captured images with the enhanced sharpness are combined. By doing so, a good wide dynamic range image can be generated.

[Second Embodiment] The image processing apparatus 10 according to the first embodiment emphasizes the sharpness of each image to be synthesized before the images are synthesized. May be emphasized. In the following, an image processing apparatus according to the second embodiment that emphasizes sharpness after combining the images will be described.

As shown in FIG. 24, the image processing apparatus 20 according to the second embodiment is different from the image processing apparatus 20 according to the first embodiment shown in FIG.
Except that the synthesizing unit 9 is connected to the switching unit SW and the exposure ratio detecting unit 7 and the sharpness enhancing unit 8 is connected to the synthesizing unit 9. is there.

In the image processing apparatus 20 shown in FIG. 24, first, a plurality of pieces of imaging information generated by the imaging section 5 are supplied to the synthesizing section 9 and synthesized. Here, the synthesizing unit 9 selects the standard image when the imaging signal is not saturated in the standard image (when smaller than the threshold), and when the imaging signal is saturated (when larger than the threshold), the exposure is performed. The non-standard image is multiplied by the exposure ratio K detected by the ratio detection unit 7 and selected. Then, a wide dynamic range image is created by performing such processing for each pixel.

Then, as shown in FIG. 25, the sharpness enhancement section 8 is provided with a color processing section 81 connected to the synthesis processing section 9.
And an aperture processing unit 82 connected to the color processing unit 81. The color processing unit 81 performs the same color processing as in the first embodiment on the synthesized information.

Further, from the signal shown in FIG. 26 (c) obtained by synthesizing the first and second image pickup signals shown in FIGS. 26 (a) and 26 (b),
As shown in FIGS. 26D to 26F, a total of three planes of RGB information are created for each of the RGB components, and aperture processing is performed on the created RGB information.

Here, as described above, when the aperture correction for enhancing the sharpness is performed simply after synthesizing the image, the setting of the coring level appropriate for the noise level in the non-standard image information portion multiplied by K is performed. There is a problem that can not be.

Therefore, it is necessary to check from which image the synthesized image was created in enhancing the sharpness. However, in the above-described color processing, RGs are obtained from a plurality of pieces of image information.
To create the B information, the emphasized information is a mixture of the standard image information and the non-standard image information.

That is, as shown in FIG. 28, when the target pixel 30 detects the signal of the B component, the shortage of R component
The component and the G component are interpolated from surrounding pixels. In this case, for example, the G component of the target pixel 30 is
Interpolation is performed by calculating the average of signals obtained from four pixels adjacent to 0 and detecting a G component signal.

Here, according to whether the G component detected by the surrounding pixels forms a part of the standard image or a part of the non-standard image, or according to a mixture ratio of the two images. , It is necessary to set the coring level appropriately.

If the pixels arranged around the target pixel 30 used for the interpolation are all standard images, the respective noise levels are n0 + n (Sa).
In the case of non-standard images, all of which have been normalized, the respective noise levels are K × n0 + K × n (Sa / K).
It is necessary to set a coring level corresponding to these noise levels.

However, in order to record the mixture ratio for each pixel, it is necessary to record the mixture ratio for a total of three planes for each of the RGB components. Absent. Therefore, the image processing apparatus according to the second embodiment predicts the mixture ratio and sets the coring level. Hereinafter, a method of setting the coring level will be described.

As shown in FIG. 29, in order to determine which information to use between the standard image information shown by the graph G1 and the normalized non-standard image information shown by the graph G3. Is the threshold Th.

When the information of the standard image is smaller than the threshold Th, the information of the standard image is adopted as the composite image information. When the information of the standard image is larger than the threshold Th, the information of the standard image is normalized. The information of the obtained non-standard image is adopted. At this time, the B component obtained at the target pixel 30 shown in FIG. 28 is created from a single piece of information. For example, when the B component is smaller than the threshold Th, the B component is used as information of a standard image. .

When the G component or the R component to be interpolated from the pixels around the target pixel 30 is considerably smaller than the threshold value Th, the interpolation is performed only with the standard image. Is likely to be information.
If the component takes a value that is considerably larger than the threshold value Th, it is highly likely that the information is information interpolated only with the information of the non-standard image.

Further, the coring process in the aperture correction aims at making the noise less noticeable in a flat portion of the image where there is little change in the image pickup signal. When the difference from the surrounding pixels is small, the coring value is reduced. It becomes a problem.

That is, in this case, since the pixel value (the magnitude of the imaging signal) around the target pixel 30 is an approximate value, the signal level containing no noise is temporarily set to the threshold value Th.
If all the information corresponding to the surrounding pixels is the standard image information, the information corresponding to the surrounding pixels includes the noise calculated by n0 + n (Th).

On the other hand, when the average value of n0 + n (Sa) is calculated and the maximum value of n0 + n (Sa) is Nn, the minimum value when all the information corresponding to the surrounding pixels is the standard image information Is (Th-Nn).

When the information corresponding to the surrounding pixels is all normalized non-standard image information, K × n0 + K
× n (Th / K)
The average value of K × n0 + K × n (Th / K) is obtained.
When the maximum value of the noise amount obtained by K × n0 + K × n (Th / K) is Nk, the maximum value when all information corresponding to surrounding pixels is normalized non-standard image information is ( Th + Nk).

Here, as shown in FIG. 27, when the coring level set for the standard image is Cn and the coring level set for the normalized non-standard image is Ck, The mixing ratio is predicted based on the level of the created image signal, and the coring level is determined.

That is, the value of the created image pickup signal is (T
h-Nn), the sharpness is emphasized under the coring level Cn set for the standard image when included in the area A1 smaller than the standard image, and normalized when the area A3 is larger than (Th + Nk). The sharpness may be enhanced under the coring level Ck set for the non-standard image.
When the value of the created imaging signal is included in the area A2 between (Th−Nn) and (Th + Nk), the value is obtained by internally dividing the coring level Cn and the coring level Ck at a predetermined ratio. The sharpness is emphasized below the required mixed coring level. The values of Nn and Nk are n0 + n (Th) and K × n0 + K × n (T
h / K) may be determined from the maximum possible value.

By adopting a method of estimating the mixture ratio in the above-described color processing, it is not necessary to record which image the synthetic image is created from.

It should be noted that even in such a method, FIG.
, The coring level C may be changed according to the actual output signal level L. In this case, first, according to the level of the imaging signal, FIG.
The coring level may be determined as shown in FIG. That is, when the image signal includes only the standard image and the level of the image signal is included in the area A1, a standard coring level determined as a function of the image signal is set, and the image signal is mixed by color processing. If it is included in A2, a mixed coring level that is also obtained as a function of the imaging signal is set, and if it is highly likely that it has been created from a normalized non-standard image and it is included in region A3, it is set as a function of the imaging signal. What is necessary is just to switch so as to set the required non-standard coring level.

In the above description, both the case of Embodiment 1 in which sharpness is enhanced prior to image synthesis and the case of Embodiment 2 in which sharpness is enhanced after image synthesis are used. In the above description, a method of simply combining one of the standard image and the non-standard image by selecting the same based on a certain threshold value Th has been described. May be used in combination with a conventional method of gradually changing the mixing ratio so as not to be noticeable.

That is, as shown in FIG. 31, for example, when the value of the image pickup signal forming the standard image is equal to or smaller than the threshold value Th0, only the standard image is used, and the value from the threshold value Th0 to the threshold value Th1 For this, a method is employed in which a composite image is synthesized using a mixture ratio uniquely determined as a linear function of the value of the imaging signal, and when the threshold value is equal to or greater than Th1, a composite image is generated using only the non-standard image. May be.

In this case, in the image processing apparatus according to the second embodiment in which the sharpness is enhanced after the images are synthesized, the images are synthesized while changing the mixing ratio as described above, and then mixed according to the value of the obtained synthesized image information. The coring level may be determined by simultaneously predicting the ratio and the mixture ratio by the color processing.

That is, as shown in FIG. 32, when the value of the image pickup signal is equal to or less than the threshold value (Th0-Nn), the value of the image pickup signal falls below the coring level set for the standard image. If the threshold value (Th1 + Nk) or more, the sharpness is enhanced under the coring level set for the normalized non-standard image, and the threshold value (Th0-N
When the magnitude is between n) and the threshold value (Th1 + Nk), the mixing ratio and the coring level may be set based on the mixing ratio.

Next, the operation of the image processing apparatus according to the second embodiment will be described with reference to the flowchart shown in FIG. As shown in FIG. 33, first, in step S
In 1, in the imaging unit 5 including the imaging element, a plurality of imaging information is obtained by performing imaging a plurality of times under different exposure amounts. Then, the imaging information is stored in the RAM 16 for image processing.

Next, in step S2, the exposure ratio detecting section 7
Calculates the exposure ratio among a plurality of images stored in the RAM 16 by the ratio of the average value of the imaging signals in the unsaturated portion of the image as described above.

In step S3, the synthesizing unit 9 selects the standard image for each non-saturated portion of the standard image in pixel units as described above, and the saturated portion is detected by the exposure ratio detecting unit 7. A composite image is created by selecting the normalized non-standard image by multiplying the exposure ratio K by the non-standard image.

In step S4, in order to enhance the sharpness of the composite image obtained in step S3, the color processing section 81 included in the sharpness enhancement section 8 executes a color interpolation process, as shown in FIG. By performing color processing on the information, FIG.
A plane image for each of the RGB components shown in FIG. 9 is created.

Next, in step S5, the sharpness enhancement section 8
The aperture processing unit 82 included in the above determines the coring level by predicting the mixture ratio based on the image information as described above, emphasizes the sharpness, and ends the operation.

As described above, the image processing apparatus according to Embodiment 2 of the present invention can also generate a wide dynamic range image with less noticeable noise.

As described above, according to the image processing apparatus of the present invention, regardless of the exposure ratio of a plurality of imaging signals,
Since an appropriate synthesized image can be obtained, a clear image having a wide dynamic range can be obtained.

If the sharpness enhancing means enhances the sharpness of the image signal based on the noise level corresponding to the magnitude of the image signal, unnecessary amplification of noise can be avoided. Therefore, an image with less noise can be obtained.

Further, according to the image processing apparatus of the present invention, the sharpness is enhanced by the mixture ratio in the synthesized image, and thus the sharpness of the synthesized image can be easily and appropriately enhanced, so that the dynamic range is wide. A clear image can be easily obtained.

If the sharpness enhancing means enhances the sharpness of the composite image based on the noise level corresponding to the composite image and the exposure ratio, the sharpness of the composite image is enhanced without amplifying the noise. Therefore, a higher quality image can be obtained.

Further, an averaging means for averaging a plurality of image pickup signals for each pixel to generate an average image pickup signal is provided. The image synthesizing means may generate a plurality of average image pickup signals at random. Noise can be suppressed, so that the quality of the obtained image can be further improved.

Further, if the image pickup means is further provided with an image pickup number determining means for determining the number of times the image pickup means picks up an image of an object under a predetermined exposure amount, the influence of noise generated at random is suppressed. Since an image can be obtained efficiently, an image having a wide dynamic range and little noise can be created at low cost.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to Embodiment 1 of the present invention.

FIG. 2 is a diagram illustrating a configuration of the image processing apparatus illustrated in FIG. 1;

FIG. 3 is a diagram illustrating a first configuration example of an imaging unit illustrated in FIG. 1;

FIG. 4 is a diagram illustrating a second configuration example of the imaging unit illustrated in FIG. 1;

FIG. 5 is a timing chart illustrating an operation of the imaging unit illustrated in FIG. 4;

FIG. 6 is a graph showing operating characteristics of the imaging unit shown in FIG.

FIG. 7 is a diagram illustrating an operation of the imaging unit illustrated in FIG. 1;

FIG. 8 is a diagram in which noise generated in the image processing apparatus shown in FIG. 1 is classified.

FIG. 9 is a graph showing characteristics of an S / N ratio in a signal generated by the imaging unit shown in FIG.

FIG. 10 is a graph showing a function of an image pickup signal generated in the image pickup section shown in FIG. 1 with respect to subject brightness;

FIG. 11 is a first graph illustrating an operation of the image processing apparatus illustrated in FIG. 1;

FIG. 12 is a second graph illustrating an operation of the image processing apparatus illustrated in FIG. 1;

FIG. 13 is a first graph for explaining the operation of the sharpness enhancing section shown in FIG. 1;

FIG. 14 is a second graph for explaining the operation of the sharpness enhancing section shown in FIG. 1;

FIG. 15 is a third graph illustrating the operation of the sharpness enhancement unit shown in FIG. 1;

FIG. 16 is a diagram for explaining setting of a coring level by a sharpness enhancement section shown in FIG. 1;

FIG. 17 is a block diagram illustrating a configuration of a sharpness enhancement unit illustrated in FIG. 1;

FIG. 18 is a first diagram illustrating the operation of the sharpness enhancement section shown in FIG. 17;

FIG. 19 is a second diagram illustrating the operation of the sharpness enhancing unit illustrated in FIG. 17;

FIG. 20 is a diagram illustrating a color filter provided in the imaging device according to the first embodiment;

FIG. 21 is a graph showing a change in sensitivity obtained by using the color filter shown in FIG. 20, depending on a wavelength.

FIG. 22 is a fourth graph for explaining the operation of the sharpness enhancing section shown in FIG. 1;

FIG. 23 is a flowchart illustrating an operation of the image processing apparatus according to the first embodiment of the present invention.

FIG. 24 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 2 of the present invention.

FIG. 25 is a diagram illustrating a configuration of a sharpness enhancement unit illustrated in FIG. 24;

FIG. 26 is a diagram for explaining the operation of the sharpness enhancement section shown in FIG. 25;

FIG. 27 is a first graph for explaining the operation of the sharpness enhancing section shown in FIG. 24;

FIG. 28 is a view for explaining the operation of the sharpness enhancement section shown in FIG. 24;

FIG. 29 is a first graph illustrating an operation of the combining unit illustrated in FIG. 24;

FIG. 30 is a second graph illustrating the operation of the sharpness enhancing unit shown in FIG. 24.

FIG. 31 is a second graph illustrating the operation of the synthesizing unit shown in FIG. 24.

FIG. 32 is a third graph illustrating the operation of the sharpness enhancing section shown in FIG. 24.

FIG. 33 is a flowchart showing an operation of the image processing apparatus according to the second embodiment of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1, 11 Lens (photography optical system) 2 Divider (subject image splitter) 3, 4 Image sensor 5 Imaging unit 6 Averaging unit 7 Exposure ratio detection unit 8 Sharpness enhancement unit 9 Combining unit 10, 20 Image processing device 12 Amplifier (AGC) 13 Sample hold circuit (CDS) 14 A / D converter 15 Image processing CPU 16 Random access memory (RAM) 17 Card RAM 81 Color processing unit 82 Aperture processing unit SW Switching unit

Claims (6)

[Claims] 1. An image processing apparatus that includes an imaging unit that captures an image of a subject and generates an imaging signal, comprising: a sharpness enhancement unit that enhances the sharpness of the imaging signal; An exposure ratio detecting unit that detects an exposure ratio in the imaging, and an image combining unit that combines a plurality of the imaging signals whose sharpness has been enhanced by the sharpness enhancing unit in accordance with the exposure ratio. Characteristic image processing device. 2. The image processing apparatus according to claim 1, wherein the sharpness enhancing means enhances the sharpness of the image signal based on a noise level corresponding to a magnitude of the image signal. 3. An image processing apparatus including an imaging unit configured to image a subject and generate an imaging signal, comprising: an exposure ratio detection unit configured to detect an exposure ratio in the plurality of imaging based on the imaging signal; Image synthesizing means for synthesizing the imaging signals according to the exposure ratio to generate a synthesized image; detecting a mixing ratio between the plurality of imaging signals based on the synthesized image; An image processing apparatus, comprising: a sharpness enhancement unit that enhances the sharpness of the composite image in response. 4. The image processing apparatus according to claim 3, wherein the sharpness enhancement means enhances the sharpness of the composite image based on a noise level corresponding to the composite image and the exposure ratio. 5. An averaging means for averaging a plurality of imaging signals obtained by the imaging means taking a plurality of images of the subject under a predetermined exposure amount for each pixel to generate an average imaging signal. The image combining unit combines the plurality of averaged image signals generated by the averaging unit based on the imaged signals captured under the different exposure amounts.
An image processing apparatus according to claim 1. 6. The image processing apparatus according to claim 5, further comprising: a number-of-imagings determining unit that determines the number of times the imaging unit images the subject under the predetermined exposure amount according to the exposure ratio.