JP2003101886A - Image pickup device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image pickup device at a low cost, for which a dynamic range and a spatial frequency band are wide, capable of obtaining images of excellent image quality. SOLUTION: The image pickup device such as an SL camera 1 is provided with a prism 3 for separating an image from an optical system 2 into two, a CCD 4a for picking up one of the separated images, a CCD 4b for picking up the other of the separated images, for which pixel size, a number of pixels and the dynamic range are different from the ones of the CCD 4a, an exposure control circuit 5 for controlling the exposures of the CCDs 4a and 4b so as to be different from each other, A/D conversion circuits 6a and 6b for respectively converting outputs of the CCDs 4a and 4b to digital images, color separation circuits 7a and 7b for respectively converting the digital images to color images, a magnification circuit 12 for magnifying one of the color images corresponding to the number of the pixels, and an SL synthesis circuit 11 for synthesizing the other of the color images and the color image after a magnification processing and generating one wide dynamic range image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup device, and more particularly to an image pickup device capable of obtaining a wide dynamic range image.

[0002]

2. Description of the Related Art The progress of digital cameras has been remarkable in recent years, but the problem is that the dynamic range of the image pickup device is narrower than that of silver halide photography, which has been widely used in the past.

Therefore, in order to obtain an image with an expanded dynamic range, in a digital camera equipped with a single image pickup device, two images are continuously taken with different shutter speeds, and these two images are taken. A technique for obtaining a wide dynamic range image by synthesizing images has been conventionally proposed.

However, in this case, if a subject moves while two images are continuously shot, the subject position changes between the images, so-called subject blurring occurs, so that the images are combined. May not be possible. Therefore, although this technique is effective for static subjects, it cannot cope with dynamic subjects.

As a technique for improving such a point, for example, in Japanese Patent Laid-Open No. 11-98418, the optical path of the image pickup system is divided to form the same optical image on the two CCDs to form two CCDs. An image pickup apparatus has been proposed in which two exposure images having different exposures can be obtained almost at the same time by making the exposure amount of 2 different by controlling the ND filter or the electronic shutter.

As means for calculating the gradation conversion characteristics relating to two images with different exposures, for example, Japanese Patent Laid-Open No. 2000-2000
There is a technique for calculating the gradation characteristic for expanding the dynamic range by using the histogram of the luminance edge of each image as described in JP-A-228747.

[0007]

However, in an image pickup apparatus for expanding the dynamic range by using a plurality of image pickup elements as described in Japanese Patent Laid-Open No. 11-98418, a plurality of image pickup elements are used. However, there is a drawback that the cost becomes high. In particular, in the case of using the same two image pickup devices with high pixel counts in recent years, the cost is significantly increased as compared with the case where only one image pickup device is used, and thus the product price is considerably affected. It will be.

Further, when a plurality of image pickup devices are used, it is possible that the spatial frequency characteristic can be widened by devising the processing, but in the above publication, a special technique such as pixel shifting is used. There is no particular description of a technique for achieving both wide band processing and wide dynamic range processing.

Further, when synthesizing images having different exposures, gradation conversion is usually performed by using different gradation conversion characteristics for each image. In particular, the noise characteristics sometimes deteriorated in a region using an image exposed for a short time.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a low-cost image pickup apparatus capable of obtaining an image having a wide dynamic range and a wide spatial frequency band and good image quality.

[0011]

In order to achieve the above object, the image pickup apparatus according to the first invention comprises an optical system for generating a plurality of optical images for the same subject and a plurality of optical images for the plurality of optical images. A plurality of image pickup devices having different characteristics for picking up images and outputting as analog images, exposure control means for controlling the exposure amounts of the plurality of image pickup devices to be different from each other, and an analog output from the plurality of image pickup devices A / D conversion means for converting an image into a digital image, and this A / D
And a synthesizing unit for synthesizing a plurality of digital images converted by the converting unit to generate one wide dynamic range image.

An image pickup apparatus according to a second aspect of the present invention is the image pickup apparatus according to the first aspect of the present invention, wherein the plurality of image pickup elements have different dynamic ranges from each other, thereby making the characteristics different from each other. is there.

An image pickup apparatus according to a third aspect of the present invention is the image pickup apparatus according to the second aspect of the present invention, wherein the plurality of image pickup elements further differ from each other in pixel size so that the characteristics are different from each other. It is a thing.

An image pickup device according to a fourth aspect is the image pickup device according to the third aspect, wherein the plurality of image pickup elements are
The angles of view that can be imaged are equal to each other.

According to a fifth aspect of the present invention, in the image pickup apparatus according to the first aspect, the exposure control means determines which of a dark portion and a bright portion in the synthesis result by the synthesizing means is important. When it is determined that the dark part is important, the exposure amount of at least one image sensor is controlled to be equal to or higher than the standard exposure amount,
When it is determined that the bright portion is important, the exposure amount of at least one image pickup device is controlled to be equal to or less than the standard exposure amount.

An image pickup apparatus according to a sixth aspect of the present invention is the image pickup apparatus according to the first aspect of the present invention, wherein at least one of the plurality of image pickup elements is an image pickup element of a type that can obtain only monochrome information. At least one is an image sensor of a type that can obtain color information.

An image pickup device according to a seventh aspect of the present invention is the image pickup device according to the first aspect, wherein the image pickup element has a plurality of color filters arranged in a mosaic pattern in order to obtain different color components for each pixel. It has a mosaic color filter, and the arrangement of the color filter is different for each of the plurality of image pickup devices.

An image pickup apparatus according to an eighth aspect of the present invention is the image pickup apparatus according to the seventh aspect, wherein the mosaic color filter is arranged such that color components obtained at the same pixel coordinates are different among the image pickup elements. It was done.

An image pickup apparatus according to a ninth aspect of the present invention is the image pickup apparatus according to the eighth aspect, wherein the mosaic color filter performs a linear operation on pixel values obtained by each image pickup element for the same pixel coordinates. Thus, the value of the color component common to all pixels can be calculated.

An image pickup apparatus according to a tenth aspect of the present invention is the image pickup apparatus according to the seventh aspect.
In the image pickup device according to the invention, the mosaic color filter is configured by a combination of coloring materials that can obtain a spatial frequency band characteristic of good luminance in an image pickup device among the plurality of image pickup devices, In the image pickup device having the above, it is configured by a combination of coloring materials that can obtain good color reproduction characteristics.

An image pickup apparatus according to an eleventh invention is the above-mentioned seventh invention.
In the image pickup device according to the invention, the mosaic color filter is composed of a combination of coloring materials that can obtain good sensitivity characteristics in an image pickup device among the plurality of image pickup devices, and another image pickup device In (1), it is composed of a combination of coloring materials that can obtain good color reproduction characteristics.

An image pickup apparatus according to a twelfth aspect of the present invention is the above-mentioned first aspect.
In the image pickup apparatus according to the first aspect of the present invention, a combination of color materials having the same average spectral transmittance of each color material is selected as a combination of the color materials that can obtain the above-described excellent sensitivity characteristics.

An image pickup apparatus according to a thirteenth invention is the above-mentioned first one.
In the image pickup device according to the invention, the A / D conversion means reduces the number of gradations of A / D conversion for each image pickup device according to the extent to which the exposure amount of the image pickup device falls below the standard exposure amount. .

An image pickup device according to the fourteenth invention is the above-mentioned third one.
In the image pickup apparatus according to the invention of claim 1,
It has a size unifying unit that performs at least one of enlarging and reducing so that the D-converted digital images from the respective image pickup devices become images of the same size. A single image is composed using a plurality of images including the image.

An image pickup device according to a fifteenth invention is the image pickup device according to the first invention.
In the image pickup device according to the invention, when the synthesizing means overlaps the brightness ranges in which the respective image pickup devices are not saturated, the overlapping brightness ranges in the digital image obtained from the plurality of image pickup devices. The image is synthesized by using the image with the least noise.

The image pickup device according to the 16th aspect of the present invention is the image pickup device according to the 7th aspect.
In the image pickup device according to the twelfth invention, the synthesizing unit further includes a multi-plate color separating unit that generates a digital image with a wider spatial frequency band by using digital images from a plurality of image pickup elements. When the brightness ranges in which the image pickup devices are not saturated overlap each other between the plurality of image pickup devices, the multi-plate color separation unit obtains the composite result of the subjects included in the overlapping brightness ranges from the digital images of all the related image pickup devices. Is generated by.

The image pickup apparatus according to the seventeenth aspect of the present invention is the above-mentioned first aspect.
In the image pickup device according to the sixth aspect of the present invention, the multi-plate color separation unit calculates a color component commonly obtained for all pixels in a digital image from the plurality of image pickup elements, and sets the value of the color component in the vicinity of the pixel of interest. And an edge discriminating means for discriminating an edge pattern in each pixel by comparing
The three primary color components are generated from the digital images from the plurality of image pickup devices based on the determination result by the edge determination means.

[0028]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. 1 to 8 show a first embodiment of the present invention. FIG. 1 is a block diagram showing a configuration of a camera system in which a display device is connected to an SL camera, and FIG. 2 is mounted on the SL camera. Showing the configuration of the two CCDs that are installed, and Fig. 3 shows the two CCs mounted on the SL camera.
A diagram showing the difference in the exposure amount of D, FIG. 4 shows 2 after A / D conversion.
6 is a diagram showing the relationship between the pixel value of one CCD and the gradation characteristic of the composite image and the absolute brightness of the scene. FIG. 5 is a diagram showing the gradation conversion characteristics when the pixels from each CCD are combined. Is two CCs when the standard exposure is set to dark
A diagram showing the relationship between the pixel value of D and the gradation characteristic of the combined image and the absolute brightness of the scene, FIG. 7 is a diagram showing the state of the combining process in the SL combining circuit of the SL camera, and FIG. 8 is based on the noise characteristic. It is a diagram which shows the modification which switches the CCD used for every brightness area.

In the first embodiment, the image pickup apparatus of the present invention is applied to an SL camera (super latitude camera).

As shown in FIG. 1, this camera system includes an SL camera 1 as an image pickup device for picking up a subject image in a wide dynamic range, and a display device for displaying an image based on an image pickup signal output from the SL camera 1. 14, and the signals captured by the SL camera 1 in a wide dynamic range are compressed and displayed in the gradation reproduction range of the display device 14.

The SL camera 1 is an optical system 2 for forming a subject image by condensing a light flux, and a prism which is an optical system for dividing the optical path of incident light through the optical system 2 into two. 3 and CCDs 4a and 4b as primary-color or complementary-color single-plate image pickup devices that pick up two optical images resulting from division by the prism 3 and output as analog electric signals, and exposure by these CCDs 4a and 4b. An exposure control circuit 5 that controls the time and outputs the exposure information to an SL synthesizing circuit 11, which will be described later, and an A / D converter that converts the analog signals output from the CCDs 4a and 4b into digital signals. The barrel A / D conversion circuits 6a and 6b and these A / D converters
Color separation circuits 7a, 7b for converting the digital signals output from the conversion circuits 6a, 6b into color images, and image buffers 8a, 8b for holding the images colored by the color separation circuits 7a, 7b, respectively. A motion detection circuit 9 which is a judgment means for performing motion detection based on the images held in these image buffers 8a and 8b and outputting the result to the exposure control circuit 5, and the image buffer 8 described above.
A histogram calculating circuit 10 as a synthesizing means for calculating a histogram based on the images held in a and 8b, and an enlarging circuit 12 as a synthesizing means and a size unifying means for enlarging the image held in the image buffer 8b.
And gradation conversion between the image in the image buffer 8a and the image enlarged by the enlargement circuit 12 based on the histogram calculated by the histogram calculation circuit 10 with reference to the exposure information output from the exposure control circuit 5. And an output port 13 for outputting the wide dynamic range image synthesized by the SL synthesizing circuit 11. It is configured.

The image output from the output port 13 is displayed on the external display device 14 as described above.

More specifically, the CCDs 4a and 4b are
As shown in FIGS. 2A and 2B, the area of the entire image pickup surface is the same, but the area of each pixel is the pixel p of the CCD 4b.
2 is 4 times the pixel p1 of the CCD 4a (the vertical pixel size and the horizontal pixel size are each 2 times), while the number of pixels is CC
D4a is four times as large as CCD4b (the number of pixels in the vertical direction and the number of pixels in the horizontal direction are twice each).

Therefore, the imaging angle of view of the CCD 4a and the CCD 4
b has the same angle of view, and these CCDs 4a, 4b
When a certain scene is photographed by, the images obtained by picking up the same scene at different resolutions are displayed in the image buffer 8a,
8b, respectively.

Next, the operation of such a camera system will be described.

When the image pickup operation by the SL camera 1 is started, two optical images of the same scene are formed by the optical system 2 and the prism 3, and the CCDs 4a, 4b are formed.
Each image is formed on the image pickup surface of. These primary color or complementary color single-plate CCDs 4a and 4b photoelectrically convert the formed optical image to accumulate charges for the charge accumulation time adjusted by the exposure control circuit 5, and then capture image signals. Is output.

At this time, the exposure control circuit 5 sets a standard exposure amount determined by AE for the CCD 4a having a small pixel size, and the CCD 4 having a large pixel size.
For b, as will be described later, an exposure amount of 1 / β times or γ times is set according to the shooting scene (here, β
> 1, γ> 1).

Hereinafter, of these, 1 / β times exposure will be referred to as short-time exposure, and γ times exposure will be referred to as long-time exposure.

Each C according to the exposure amount set in this way
When the saturated charge amounts Vmax of the CDs 4a and 4b are plotted on the same graph, it becomes as shown in FIG. That is, the signal output for the absolute brightness of the shooting scene is
The CCD 4a is indicated by the reference numeral 21, while the CCD 4b is indicated by the reference numeral 22 having a gentle inclination in the short-time exposure and is shown by the reference numeral 23 having a steep inclination in the long-time exposure. . The scene in which the output signal value V is obtained in the standard exposure is the output signal value V / β in the short-time exposure and the output signal value γV in the long-time exposure.

In this way, two CCs with different exposures are used.
By capturing the same subject at D almost simultaneously, an image signal can be obtained without saturation in a wide luminance range. Further, since the CCDs 4a and 4b have different pixel sizes, the noise level of the CCD 4a is relatively high as indicated by reference numeral 24, and the noise level of the CCD 4b is 25.
As shown in (4), it is considerably lower than that of the CCD 4a.

Next, the analog images output from the CCDs 4a and 4b are converted into digital images by the A / D conversion circuits 6a and 6b, respectively.

At this time, the bit width of the digital signal obtained by the A / D conversion circuit 6a is, for example, 10 bits, and A / D
The bit width of the digital signal obtained by the D conversion circuit 6b is 8 bits, which are different from each other.

The signals digitized by the A / D conversion circuits 6a and 6b in this manner are colorized by the color separation circuits 7a and 7b, respectively, and then the image buffers 8a and 8b.
Stored as a color image.

Of these, the color image stored in the image buffer 8b and having a small number of pixels from the CCD 4b is
The enlargement circuit 12 uses a known cubic interpolation method or the like to store the CC stored in the image buffer 8a.
Enlargement processing is performed so as to match the size of the image from D4a.

After that, the SL synthesizing circuit 11 exposes the image data stored in the image buffer 8a and the image data enlarged by the enlarging circuit 12 from the exposure control circuit 5 as described later. With reference to the information, gradation conversion is performed on the basis of the histogram calculated by the histogram calculation circuit 10 and synthesis is performed to generate one wide dynamic range image.

The wide dynamic range image thus generated is displayed on the display device 14.

Each of the image buffers 8a and 8b has a storage capacity capable of holding two pieces of image data, so that the image data picked up in the previous image pickup operation remains without being overwritten. Has become. This makes it possible to perform exposure control according to the result of the motion detection circuit 9, as will be described later.

Next, how the SL synthesizing circuit 11 performs gradation conversion and image synthesizing will be described with reference to FIGS. 4, 5 and 7. In addition, here, the case where short-time exposure is performed by the CCD 4b will be described as an example.

First, referring to FIG. 4, each CCD 4a, 4b
The relationship between the pixel value of the image obtained from the above and the brightness of the scene will be described.

The pixel value before gradation conversion of the image obtained from the CCD 4a is shown by reference numeral 26, and the pixel value before gradation conversion of the image obtained from the CCD 4b is shown by reference numeral 27. There is. Reference numeral 28 indicates the gradation characteristic of the composite image.

The SL synthesizing circuit 11 first sets the switching luminance L indicating the sharing of the synthesizing result of two images having different exposures. Then, the CCD 4a as shown in FIG.
The image obtained from the CCD 4 and the CCD 4 as shown in FIG.
Both the image obtained from b, the region Rd corresponding to the subject having the luminance equal to or lower than the switching luminance L and the region Rd not corresponding to the subject.
It is divided into two parts b and b as shown in FIG. 7 (C) and FIG. 7 (D).

The switching luminance L is output from the exposure control circuit 5 to the SL synthesizing circuit 11 so as to output the two CCDs 4a, 4a.
Based on the information of the exposure ratio of b, the brightness level at which the output from the CCD 4a is saturated is set to a level obtained by multiplying a predetermined coefficient. In the case as shown in FIG. 4, the level indicated by the vertical dotted line is the switching luminance L.

Next, the SL synthesizing circuit 11 includes the CCD 4a,
The image from 4b is subjected to gradation conversion as shown in FIGS. 7 (E) and 7 (F).

In the case of the example shown in FIG. 4, CC
For the standard exposure image by D4a, the region Rd is shown in FIG.
The conversion is performed with the gradation conversion characteristic as indicated by reference numeral 32 in (A),
For the short-time exposure image by the CCD 4b, the area Rb
Is converted with a gradation conversion characteristic as indicated by reference numeral 34 in FIG. Note that reference numerals u1 and u2 in FIG. 5 are pre-combination pixel values respectively corresponding to the switching luminance L as shown in FIG. 4, reference numeral 31 is a gradation characteristic before combination of the CCD 4a, and reference numeral 33 is. The gradation characteristics of CCD4b before composition are
Reference numeral 35 indicates the noise level after the CCD 4b is combined.

The gradation conversion characteristics as shown in FIGS. 5 (A) and 5 (B) are described in, for example, the above-mentioned Japanese Patent Laid-Open No. 2000-2287.
It is determined based on the histogram of the luminance edge of each image calculated by the histogram calculation circuit 10 using the gradation characteristic system calculation algorithm for expanding the dynamic range as described in Japanese Patent Publication No. 47. By using this algorithm, the total gradation characteristic, which is obtained by combining the gradation characteristics and represents the relationship of the pixel value with respect to the combined scene luminance, is continuous as shown by reference numeral 28 in FIG. Guaranteed to become.

After performing the gradation conversion as described above, the SL synthesizing circuit 11 sets the image area Rd from the CCD 4a and the image area Rb from the CCD 4b as shown in FIG. 7 (G). Combine to complete a single composite image.

Here, looking at the noise characteristics of the composite image, as shown in FIG.
In the vicinity of the pixel value corresponding to the switching brightness L, the gradation conversion characteristic has a considerably large slope, so that noise sharply increases as indicated by reference numeral 35. Therefore, in the present embodiment, the CCD used for the short-time exposure is a C with a large pixel size and less noise.
By adopting CD4b, noise is alleviated so as not to increase corresponding to this point.

Further, since the short-time exposure image is gradation-converted by using the gradation conversion curve as shown by reference numeral 34 in FIG. 5B, the composition result occupies a relatively small number of gradations. . Therefore, in the present embodiment, by reducing the bit length of the A / D conversion for the short-time exposure image output from the CCD 4b, the memory amount of the image buffer 8b can be reduced without substantially lowering the image quality. It is configured to be able to.

Next, the function of the exposure control using the difference from the image captured last time will be described with reference to FIG.

When the SL synthesizing circuit 11 completes the above-described image synthesizing process, the motion detecting circuit 9 operates as follows.

First, for each of the image buffers 8a and 8b, a difference is calculated from the image captured this time and the image captured last time, and the difference is not less than a predetermined threshold and the pixel value is the same as the previous image. This time, the area where both images are not saturated is determined. Then, a histogram of the subject brightness in this area is created, and the brightness with the most histogram and the histogram value at this brightness with the most histogram are obtained.

After performing such a calculation related to the histogram on each of the image data held in the image buffers 8a and 8b, the maximum values of the obtained histograms are compared and the image buffer of the one giving the maximum value is compared. For, the final luminance is the maximum luminance obtained in the histogram.

The brightness value thus obtained is output to the exposure control circuit 5.

The exposure control circuit 5 sets a standard exposure amount such that the brightness value output from the motion detection circuit 9 is a proper exposure amount. If the standard exposure amount is less than a certain threshold value, it is determined that the exposure amount is too small, and the exposure amount of the CCD 4b is set to γ times the standard exposure amount,
On the other hand, when it is larger than a certain threshold value, the exposure amount of the CCD 4b is set to 1 / β times the standard exposure amount.

The exposure amount of the CCD 4b is 1 of the standard exposure amount.
When set to / β times, the relationship between the pixel value of the image from the CCD 4b and the brightness is as shown by reference numeral 37 in FIG.
Further, the image from the CCD 4a is indicated by reference numeral 36, and further, the gradation characteristic after the combination is indicated by reference numeral 38. In this way, the switching brightness L is also the CCD 4b.
Is changed to a range where saturation does not occur, and the combined result is the Rd portion of the image from the CCD 4b for long-time exposure and the CC for standard exposure.
It becomes a combination with the Rb portion of the image from D4a.

Also in this case, since the pixel size of the CCD 4b is large, the noise characteristic in the dark portion of the combined result is improved, and the dynamic range seen in S / N is expanded.

In the above description, only the enlarging process is performed by the enlarging circuit 12, but the enlarging / reducing circuit may perform the reducing process as needed.

According to the first embodiment as described above, the characteristics of the two CCDs are made different according to the conversion by the different gradation conversion characteristics at the time of image combination, and the number of output gradations at the A / D conversion is set. By setting so that
A low-cost imaging device capable of improving the image quality of a composite image without using more expensive parts than necessary.

In particular, by making the pixel size of the plurality of image pickup devices different from each other to increase the pixel size of the CCD for short-time exposure, without using a special CCD or the like having a function such as noise suppression, A dynamic range characteristic corresponding to a difference in gradation conversion characteristic can be easily obtained, and a noise characteristic of a composite image can be improved.

At that time, CC having a large pixel size
Since the number of pixels of D is reduced to keep the angle of view the same as that of the other CCD and the enlargement processing is performed at the time of composition to match the sizes, it is possible to expand the dynamic range in the entire area of the composite image. Since the number of pixels at the time of composition is the same, the composition of the composition circuit is simple and the expandability is improved.

Further, since the plurality of image pickup elements have the same angle of view, it is not necessary to adopt an optical system having a different magnification for each image pickup element, and it is easy to divide the optical path using the same optical system. Since the configuration can be adopted, the cost can be reduced and the imaging device can be made small.

Since the number of output gradations (bit length) at the time of A / D conversion is reduced for a short-time exposure image pickup device having a narrow gradation range in the combined result, such a large number of gradations is originally required. There is no waste of giving unnecessary gradations to the data that should not be processed. As a result, it is possible to reduce the performance of the circuit that processes the output of the image sensor and the size of the image buffer required during the image processing, while substantially maintaining the image quality of the combined result without deteriorating. .

In addition, in the exposure control, the moving area is detected from each of the outputs of the CCDs having different exposure amounts, the exposure is set by judging which of the bright portion and the dark portion is important in the combined result. However, since the image pickup luminance range shared by each CCD is changed, it is possible to set an appropriate exposure for an important subject to be noticed so that it can be displayed clearly.

The present embodiment is not limited to the above-mentioned configuration, and various modifications and applications are possible.

For example, although the two-plate type image pickup system is used in the above description, even when three or more CCDs are used, the CCD having the optimum characteristic corresponding to the gradation conversion characteristic received at the time of combining is selected, and each CCD is selected. By capturing images with different exposures, it is possible to obtain a high-quality image with an expanded dynamic range, as described above.

Further, it is not always necessary to increase the pixel size of the CCD for short-time exposure, and even if the pixel size is the same, a large dynamic range can be obtained, and the characteristics of relatively low noise can be obtained. Any CCD will do. Also in this case, it is possible to improve the noise characteristic of the synthesis result.

When color information is not so important as in a surveillance camera, one of the CCDs, for example, a CCD for short-time exposure, is a monochrome CCD, and when synthesizing images, a subject having a switching brightness or higher is monochrome. By synthesizing, cost reduction of the entire system can be achieved while satisfying the required performance according to the application.

Even when images are combined, in the present embodiment, with the switching brightness as a boundary, the subject having a darker brightness is shared by the standard exposure CCD and the object having a brighter brightness is shared by the short exposure CCD. But both C
In consideration of the noise characteristics of the CD and the amplification amount of noise generated as a result of gradation conversion during synthesis, switching is performed so that the noise characteristics of the synthesis result are piecewise optimized as shown in FIG. 8, for example. It is also possible.

That is, in FIG. 8, the 8-bit output from the CCD 4a is as shown by reference numeral 41, and C
The 8-bit output from the CD 4b is shown by reference numeral 42. Further, reference numeral 43 indicates the gradation conversion characteristic of the combined image obtained by combining the results of gradation conversion on the images from these CCDs 4a and 4b.

Similar to the above, the region Rg1 is a region where the absolute luminance of the scene is equal to or higher than the switching luminance L in the composite image, and indicates the luminance range in which the image from the CCD 4b is used.

On the other hand, the area Rg2 is provided in each CCD 4
When gradation conversion is performed on the images from a and 4b, due to the noise characteristics and gradation conversion characteristics of the CCD, although the brightness level is equal to or lower than the switching brightness L, the image after gradation conversion has CC
The range in which the noise characteristic of the image from D4b is superior to that of the CCD 4a is shown. In this portion, the image quality is improved by using the image from the CCD 4b even if the luminance is the switching luminance L or less.

If the noise characteristic of each CCD is quantitatively grasped beforehand, such a range can be calculated by giving the gradation conversion characteristic received by the image from each CCD. .

In this way, the noise characteristic of the combined image can be further improved by synthesizing using the signal from the image pickup device having the best noise characteristic in each overlapping section.

FIGS. 9 to 15 show a second embodiment of the present invention, and FIG. 9 is a block diagram showing the structure of a camera system in which a display device is connected to an SL camera,
FIG. 10 is a diagram showing the configuration of two CCDs mounted on the SL camera, and FIG. 11 is two CCDs mounted on the SL camera.
12 is a diagram showing an effective luminance range in which the information from can be used, FIG. 12 is a flowchart showing the processing of the integrated color separation circuit, and FIG. 13 is a pair of neighboring pixels when the edge direction is determined in the integrated color separation circuit. FIG. 14 is a diagram showing four types of neighboring pixels of 3 × 3 pixels of M5 calculated in the integrated color separation circuit, and FIG. 15 is a diagram showing combinations of neighboring types and edge directions in the integrated color separation circuit. 6 is a chart showing calculation formulas for R component and B component.

In the second embodiment, the above-mentioned first embodiment is used.
The same parts as those of the embodiment are denoted by the same reference numerals, and the description thereof will be omitted. Only different points will be mainly described.

The SL camera 51, which is an image pickup device in the camera system of the second embodiment, is constructed in substantially the same manner as the SL camera 1 of the first embodiment described above.
Instead of the CCDs 4a and 4b, C which is an image pickup device having a different configuration
Using the CDs 54a and 54b, the color separation circuits 7a and 7b
Is not provided, and an integrated color separation circuit 52 that also serves as a synthesizing unit, a multi-plate color separating unit, and an edge determining unit is used instead of the motion detecting circuit 9.

The CCDs 54a and 54b have the same number of pixels and the same pixel size, but one CCD 5
4a is a CCD of a primary color single plate type having R, G, B mosaic color filters as shown in FIG.
CD 54b has Y, C, Wa, as shown in FIG.
Complementary color single plate type C with Wb mosaic color filter
It's a CD.

The spectral characteristics of the respective color filters forming the complementary color filter of the CCD 54b are the same when the pixel value V_x (x is Y, Wa, Wb or C) obtained at each pixel is the same for the same subject and the same exposure. The pixel value V obtained in the pixel having the primary color filters R, G, B of the CCD 54a at the time of photographing
_R, V_G, and V_B are adjusted so as to satisfy the following formula 1.

[Equation 1] V_Y = V_R + V_G V_C = V_B + V_G V_Wa = α × V_R + V_G + β × V_B V_Wb = β × V_R + V_G + α × V_B Here, a and β are predetermined constants.

Further, the integrated color separation circuit 52 performs a process of converting the single-plate image stored in the image buffers 8a and 8b into a color image, as described later.

Next, the operation of such a camera system will be described.

When the image pickup operation by the SL camera 51 is started, two optical images for the same scene are formed by the optical system 2 and the prism 3, and a single plate CCD is formed.
Each image is formed on the image pickup surfaces of 54a and 54b.

These single plate CCDs 54a and 54b are
The formed optical image is photoelectrically converted to the exposure control circuit 5 described above.
The charge is stored for the charge storage time adjusted by
After that, the image signal captured is output.

At this time, the exposure control circuit 5 sets a standard exposure amount determined by AE for the complementary color single plate type CCD 54b, and 1 / γ (here) for the primary color single plate type CCD 54a. Then, an exposure amount γ> 1) times (so-called short-time exposure amount) is set.

Each C according to the exposure amount set in this way
The saturation charge amounts Vmax of the CDs 54a and 54b are plotted on the same graph as shown in FIG. That is, the signal output with respect to the absolute brightness of the photographic scene becomes a short-time exposure for the CCD 54a, as shown by the reference numeral 61 with a gentle inclination, while the CCD 5a
For 4b, reference numeral 62 is used.

As described above, two CCs with different exposures are used.
By capturing the same subject at D almost simultaneously, a signal can be obtained without being saturated in a wide luminance range. Also,
Since the CCDs 4a and 4b have the same pixel size as described above, the noise levels are both 63 in this example.
It is the same as shown in.

Next, the analog signals output from the CCDs 54a and 54b are converted into digital signals having a bit width of 10 bits by the A / D conversion circuits 6a and 6b, respectively, and then converted into the image buffers 8a and 8b. The image is held as a single plate.

Next, the single plate images in the image buffers 8a and 8b are converted into color images by the integrated color separation circuit 52. In this integrated color separation circuit 52, both single-plate images are effectively used to perform colorization on each single-plate image.

That is, as shown in FIG. 11, which C
The CDs 54a and 54b also have an effective luminance range in which the signal output is equal to or higher than the noise level indicated by reference numeral 63 and is equal to or lower than the saturation level Vmax. For a subject in this effective brightness range, it is possible to use data of any single plate image. Therefore, the integrated color separation circuit 52
In any of the images, an area D composed of pixels in which all the pixels in the neighboring pixels of 3 × 3 pixels satisfy the condition of the noise level or more and the saturation level or less is obtained. The processing is performed according to such a flow chart to generate an RGB color image.

In the following, description will be given according to the flowchart shown in FIG. In this description, "M1"
Is a single plate image in the image buffer 8a corresponding to the CCD 54a, and "M2" is the image buffer 8 corresponding to the CCD 54b.
The single plate image in b and the other M3 to M6 represent images of intermediate results.

Further, the even-numbered pixel means a pixel at a position as hatched in FIG. 10 (C), and the odd-numbered pixel means another pixel in FIG. 10 (C). That is, when the pixel coordinates of pixels arranged in the vertical direction and the horizontal direction are represented by (k, l) (here, k, l is an integer of 1 or more), those in which k + 1 is an odd number are even pixels, and k + l is Those that are even are odd pixels. Specifically, (1,
1), (2,2), (1,3), (3,1), etc. are odd pixels, and (1,2), (2,1), (1,4),
(2, 3) and the like are even pixels.

When the processing is started, first, in order to compensate for the difference in the exposure amount from the image M2, the image M1 is multiplied by γ and
This is newly designated as M1 (step S1).

Next, from each pixel value of the image M2, the pixel value at the corresponding position of the exposure-compensated image M1 is subtracted,
The result is set as the pixel value of the image M3 (step S2).
As a result, in M3, the G component is obtained by calculating R + G-R = G or B + G-B = G in the odd-numbered pixel by the above-mentioned mathematical expression 1. In the even-numbered pixels, αR + βB + G−G =, depending on whether the color filter at the corresponding position of the CCD 54b is Wa or Wb.
αR + βB or βR + αB + G−G = βR + a
Either B or B is obtained.

Then, M4 is created by combining the odd pixels of M3 and the even pixels of M1 (step S3). M
Since the odd-numbered pixels of 3 and the even-numbered pixels of M1 both have G components, M4 is an image in which all pixels have only G components.

Then, in each pixel of M4, 3 ×
The direction of the edge in the neighboring pixels of 3 pixels is determined (step S4). That is, in a neighboring pixel composed of 3 × 3 pixels centered on a certain pixel as shown in FIG.
The absolute value of the difference between the pixel values of the neighboring pixel pairs with the same number is calculated in four ways, and the number of the pair that gives the minimum value of the difference is used as the label in the edge direction.

Further, M5 is created by combining the even pixels of M3 and the odd pixels of M1. This M5 is 3x3
If a neighboring pixel made up of pixels is exemplified, depending on its position,
There are four types as shown in FIG. And M
In the neighboring pixels of 3 × 3 pixels of each pixel of 5, the pairs of neighboring pixels in the edge direction labeled in step S4 are averaged to obtain the pixel value of M6 (step S6).
5). These M5 and M6 take values as shown in the table of FIG. 15 for the combination of the neighborhood type and the edge direction.

Next, the calculation is performed for each pixel value of M5 and M6 to obtain the R and B components. The calculation formula at this time is shown in FIG.
It is as described in the rightmost column in the chart shown in. In the formula, M5 and M6 represent the pixel values of M5 and M6 at a certain pixel position, and R and B represent the R and B components of that pixel position, respectively. Also, each coefficient C1 ~
C5 satisfies the following expression 2.

[Equation 2]

All of these calculation formulas have the form shown in the following formula 3, that is,

[Equation 3] To write in the form of, the matrix of coefficients P11-P22 is
Four types of neighborhoods (types I to IV) shown in FIG. 14 and four types of edge directions (types 1 to IV) shown in FIG.
ROM for a total of 16 combinations by combination with 4)
It is possible to easily realize the calculation of the R component and the B component by storing them in the memory and performing a matrix calculation which is a linear calculation. The G component is M4 in step S3.
As a result, the RGB components are obtained (step S6).

Image buffer 8b corresponding to CCD 54b
For the image inside, for the pixels belonging to the area D,
The result obtained by executing the processing shown in FIG. 12 can be used as it is. Further, for pixels that do not belong to the region D, the pixel value of the image buffer 8a is greatly affected by noise, so the integrated color separation circuit 52 interpolates only from the pixel values of the image buffer 8b. , RGB components are obtained.

As can be seen from the filter arrangement as shown in FIG. 10B, all the four types of filters are always included in the neighboring pixels of 3 × 3 pixels. The integrated color separation circuit 52 obtains the maximum value vi of the absolute value of the average value and the mutual difference for each of the same filter types for the neighboring pixels of 3 × 3 pixels of each pixel (where i = C, Y,
Wa, Wb). Then, three of the ones having the smallest vi are selected, and the corresponding three average values Ax, Ay, Az (x,
y and z are any three of C, Y, Wa and Wb),

[Equation 4] RG of each pixel by matrix operation which is a linear operation like
Find the B component. Here, P11 to P33 in Equation 4 are
Depending on which three were chosen from the four types of filters,
It is a constant calculated in advance.

After calculating the RGB values of all pixels inside and outside the area D in this way, the integrated color separation circuit 52
The result is output to the image buffer 8b and held therein.

On the other hand, for the image in the image buffer 8a corresponding to the CCD 54a, the results obtained by executing the processing shown in FIG. Therefore, it is necessary to multiply by 1 / γ.

For pixels not belonging to the area D,
The RGB components are obtained by interpolating only the pixel values of the image buffer 8a. This interpolation is performed by a known interpolation method for the Bayer array because the CCD 54a has a normal primary color Bayer array.

After calculating the RGB values of all pixels inside and outside the area D in this way, the integrated color separation circuit 52
The result is output to the image buffer 8a and held therein.

After the single color image of each image buffer is colorized by the integrated color separation circuit 52 in this way, the SL synthesizing circuit 11 performs the same processing as in the first embodiment described above.
The images in the image buffers 8a and 8b are combined to generate one wide dynamic range image.

The wide dynamic range image thus generated is output from the output port 13 and displayed on the display device 14.

According to the second embodiment as described above, substantially the same effect as that of the above-described first embodiment is obtained, and a plurality of CCDs having different color filter characteristics (color filter arrangement) from each other are used. When the CCDs are imaged with different exposures and the results are combined, color separation is performed by using the outputs of both CCDs in the overlapping effective luminance range where the CCDs are above the noise level and below the saturation level in both CCDs. As a result, the spatial frequency band is wide,
Moreover, a composite image having a wide dynamic range can be obtained.

Furthermore, when the calculation is performed for each pixel value on the images obtained from the two CCDs, the arrangement of color filters is devised so that the G component can be obtained for all the pixels. Easy color separation. In this way, the signal of the common color component obtained in all pixels has a wide spatial frequency band. Therefore, by performing color separation using this color component signal as a clue, the band of other color components can also be broadened. It becomes possible and the processing can be simplified.

Particularly, in the vicinity of each pixel, the edge direction of the G component, which is a common color component obtained in all pixels, is discriminated, and interpolation is performed based on that direction to obtain components other than the G component. Therefore, it is possible to easily perform broadband color separation of all color components including components other than the G component with a relatively simple circuit configuration.

The present embodiment is not limited to the above-mentioned configuration, and various modifications and applications are possible.

For example, the method of selecting the color filters of each CCD is not limited to the method of performing wideband color separation as in the present embodiment, and the color filter arrangements of both pixels are the same. Higher image quality can be achieved simply by making the positions different.

In this case, the difference is calculated after compensating the exposure ratio for the images obtained by both CCDs. Since there is a high possibility that color moire occurs due to the influence of the mosaic filter array in the portion where the difference is large, it is possible to reduce the color moire by reducing the saturation of that portion.

In this way, by setting the filter arrangement of the image pickup device so that different color information can be obtained at the same pixel position, it is possible to easily widen the band.

A CCD having a color filter in which one of the CCDs is made of a color material capable of obtaining a luminance signal in a wide band.
By specializing the other CCD as a CCD provided with a color filter composed of a color material having high color reproducibility, a luminance component is acquired from the CCD that can obtain a broadband luminance signal when performing color separation, and The color difference components may be calculated from the CCD having high reproducibility, and both may be combined. As a result, it becomes possible to obtain a high-quality image excellent in luminance characteristics and color reproduction characteristics by utilizing the characteristics of both CCDs.

Further, one of the CCDs is provided with a color filter composed of a color material capable of obtaining high sensitivity characteristics, and the other is a CC provided with a color filter composed of a color material having high color reproducibility.
You may comprise so that it may become D. In this case, by using a CCD provided with a color filter having high sensitivity characteristics for standard exposure, the shutter speed of standard exposure can be increased. Further, since the CCD provided with the color filter having high color reproducibility has relatively low sensitivity, the shutter speed for short-time exposure can be set relatively slow by using this CCD for short-time exposure. This is a preferable configuration from the viewpoint of both noise characteristics and shutter control. In this way, a low-sensitivity element for short-time exposure,
By using a high-sensitivity element for long-time exposure, it is possible to relatively narrow the range of shutter speeds required for capturing an image in a predetermined luminance range in the image pickup apparatus as a whole, and shutter control is facilitated.

In particular, in the case of this embodiment, the CCD 5
This condition is satisfied if the constants α and β are appropriately set in the color filter 4b so that the sensitivity difference between the color filters is reduced (the average spectral transmittance is substantially equal among the color filters). In addition, the phenomenon that only a specific pixel is saturated does not occur when the exposure amount is increased, and the sensitivity characteristic of the image sensor can be improved.

The present invention is not limited to the above-described embodiments, and it goes without saying that various modifications and applications are possible without departing from the spirit of the invention.

[0127]

As described above, according to the image pickup apparatus of the present invention, it is possible to obtain an image of good quality with a wide dynamic range and a wide spatial frequency band at low cost.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a camera system in which a display device is connected to an SL camera according to a first embodiment of the present invention.

FIG. 2 is a diagram showing a configuration of two CCDs mounted on the SL camera according to the first embodiment.

FIG. 3 is a diagram showing a difference in exposure amount between two CCDs mounted on the SL camera according to the first embodiment.

FIG. 4 is a diagram showing a relationship between pixel values of two CCDs after A / D conversion, gradation characteristics of a composite image, and absolute brightness of a scene in the first embodiment.

FIG. 5 is a diagram showing each gradation conversion characteristic when the pixels from each CCD are combined in the first embodiment.

FIG. 6 is a diagram showing a relationship between pixel values of two CCDs and absolute brightness of a scene when standard exposure is set dark in the first embodiment.

FIG. 7 is a diagram showing a state of a combining process in an SL combining circuit of the SL camera of the first embodiment.

FIG. 8 is a diagram showing a modified example of switching the CCD to be used for each luminance section based on a noise characteristic in the first embodiment.

FIG. 9 is a block diagram showing a configuration of a camera system in which a display device is connected to an SL camera according to a second embodiment of the present invention.

FIG. 10 is a diagram showing a configuration of two CCDs mounted on the SL camera according to the second embodiment.

FIG. 11 is a diagram showing an effective luminance range in which information from two CCDs mounted on the SL camera according to the second embodiment can be used.

FIG. 12 is a flowchart showing the processing of the integrated color separation circuit in the second embodiment.

FIG. 13 is a diagram showing a pair of neighboring pixels when determining an edge direction in the integrated color separation circuit of the second embodiment.

FIG. 14 is a diagram showing four types of neighboring pixels formed by 3 × 3 pixels of M5 calculated by the integrated color separation circuit of the second embodiment.

FIG. 15 is a table showing calculation formulas for R and B components for combinations of neighboring types and edge directions in the integrated color separation circuit of the second embodiment.

[Explanation of symbols]

1, 51 ... SL camera (imaging device) 2 ... Optical system 3 ... Prism (optical system) 4a, 4b, 54a, 54b ... CCD (imaging device) 5 ... Exposure control circuit (exposure control means) 6a, 6b ... A / D conversion circuit (A / D conversion means) 7a, 7b ... Color separation circuits 8a, 8b ... Image buffer 9 ... Motion detection circuit (determination means) 10 ... Histogram calculation circuit (synthesis means) 11 ... SL synthesis circuit (synthesis means) 12 ... Enlarging circuit (combining means, size unifying means) 13 ... Output port 14 ... Display device 52 ... Integrated color separation circuit (combining means, multi-plate color separating means, edge discriminating means)

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) H04N 9/04 H04N 9/04 B 9/07 9/07 D // H04N 101: 00 101: 00 F term (Reference) 5C022 AA00 AB06 AB14 AB37 AC42 AC69 CA00 5C024 AX01 BX01 BX04 CX46 DX04 EX17 EX18 EX48 EX52 GY01 HX23 HX50 HX51 HX58 5C065 AA03 BB48 CC01 DD02 EE01 EE05 GG18 GG21 GG30

Claims (17)

[Claims] 1. An optical system for generating a plurality of optical images for the same subject; a plurality of image pickup elements having different characteristics for picking up each of the plurality of optical images and outputting as an analog image; Exposure control means for controlling the exposure amounts of the image pickup elements to be different from each other, A / D conversion means for converting an analog image output from the plurality of image pickup elements into a digital image, and conversion by the A / D conversion means. An image pickup apparatus comprising: a synthesizing unit configured to synthesize a plurality of digital images thus generated to generate one wide dynamic range image. 2. The image pickup apparatus according to claim 1, wherein the plurality of image pickup devices have different characteristics by making the dynamic ranges different from each other. 3. The image pickup apparatus according to claim 2, wherein the plurality of image pickup elements further have different characteristics by making pixel sizes different from each other. 4. The image pickup apparatus according to claim 3, wherein the plurality of image pickup elements have mutually equal image pickup angles of view. 5. The exposure control means comprises a judging means for judging which of a dark portion and a bright portion in the synthesis result by the synthesizing means is important, and the dark portion is important. If it is determined, the exposure amount of at least one image sensor is controlled to be equal to or higher than the standard exposure amount, and if it is determined that the bright part is important, the exposure amount of at least one image sensor is controlled to be equal to or less than the standard exposure amount. The image pickup apparatus according to claim 1, wherein the image pickup apparatus is one. 6. At least one of the plurality of image pickup devices is an image pickup device of a type that can obtain only monochrome information, and at least another one is an image pickup device of a type that can obtain color information. The image pickup apparatus according to claim 1, wherein: 7. The image pickup device has a mosaic color filter in which color filters of a plurality of colors are arranged in a mosaic pattern in order to obtain different color components for each pixel, and the arrangement of the color filters is as described above. The image pickup apparatus according to claim 1, wherein the plurality of image pickup elements are different from each other. 8. The image processing according to claim 7, wherein the mosaic color filter has color filters arranged such that color components obtained at the same pixel coordinates are different between image pickup devices. apparatus. 9. The mosaic color filter can calculate a common color component value for all pixels by performing a linear operation on pixel values obtained by each image sensor for the same pixel coordinates. The image pickup apparatus according to claim 8, wherein the image pickup apparatus is configured as described above. 10. The mosaic color filter is composed of a combination of coloring materials capable of obtaining a spatial frequency band characteristic of good luminance in an image pickup device among the plurality of image pickup devices, and there are others. The image pickup apparatus according to claim 7, wherein the image pickup element is configured by a combination of color materials that can obtain good color reproduction characteristics. 11. The mosaic color filter is composed of a combination of color materials capable of obtaining good sensitivity characteristics in one of the plurality of image pickup elements, and in another certain image pickup element. The image pickup apparatus according to claim 7, wherein the image pickup apparatus is configured by a combination of color materials that can obtain good color reproduction characteristics. 12. The image pickup apparatus according to claim 11, wherein a combination of color materials having good sensitivity characteristics is selected so that the average spectral transmittance of each color material is the same. 13. The A / D conversion means reduces the number of gradations of A / D conversion for each image pickup device according to the extent to which the exposure amount of the image pickup device falls below the standard exposure amount. The image pickup apparatus according to claim 1, wherein the image pickup apparatus is provided. 14. The synthesizing means has a size unifying means for performing at least one of enlargement and reduction so that the digital images from the respective A / D-converted image pickup devices have the same size. 4. The image pickup apparatus according to claim 3, wherein one image is composed by using a plurality of images including an image whose size is adjusted by the size unifying unit. 15. The synthesizing means, when the luminance ranges in which the respective image pickup elements are not saturated overlap among a plurality of image pickup elements,
The image pickup apparatus according to claim 1, wherein the images are combined by using an image having the least noise in an overlapping luminance range among digital images obtained from a plurality of image pickup elements. 16. The synthesizing means further comprises multi-plate color separating means for generating a digital image having a wider spatial frequency band by using digital images from a plurality of image pickup elements, and each image pickup element is saturated. In the case where the brightness ranges that do not overlap are overlapped between the plurality of image pickup devices, the composite result for the subjects included in the overlapping brightness ranges is generated by the multi-plate color separation means from the digital images of all the related image pickup devices. The imaging device according to any one of claims 7 to 12, wherein the imaging device is present. 17. The multi-plate color separation means calculates a color component commonly obtained from all the pixels from digital images from the plurality of image pickup devices, and compares the values of the color components in the vicinity of the pixel of interest. It is characterized in that it comprises edge discrimination means for discriminating an edge pattern in each pixel, and generates three primary color components from digital images from a plurality of image pickup devices based on the discrimination result by the edge discrimination means. The imaging device according to claim 16.