JP2000299872A - Color image pickup unit - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a color image pickup unit where no color moire takes place even from an object with a periodic luminance change, color photographing with high image quality is attained with a small sized inexpensive device and an excellent picture is obtained by compensating a defective pixel even when there exists the defective pixel in an image pickup device. SOLUTION: The color image pickup unit consists of a CCD image pickup device 4 where a random arrangement is adopted for color coding arrangement of a pixel group, an EEPROM 17 that stores arrangement data with respect to color coding array of a color image pickup device and pixel defect data, and a digital process circuit 7 that generates and processes a color signal on the basis of the color coding arrangement data stored in the EEPROM 17, and the digital process circuit 7 conducts pixel defect correction processing on the basis of the pixel defect data stored in the EEPROM in the color signal generating processing process.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a color image pickup apparatus using a color image pickup device capable of preventing occurrence of color moire.

[0002]

2. Description of the Related Art Generally, an image pickup device represented by an image pickup tube and a solid-state image pickup device is widely used in an image pickup apparatus. In particular, a color image pickup device used for a single tube or a single plate (Single Sensor) color image pickup device can constitute a color image pickup device with one image pickup device, so that a color separation prism is not required and the lens can be miniaturized. In addition, there is no need for various adjustments of a multi-plate type represented by registration, and it has many features such as low power consumption, and has made many contributions to miniaturization and power consumption reduction of color imaging devices. A single-chip color camera using a color CCD image pickup device, which is a solid-state image pickup device, has become a mainstream imaging device.

In order to obtain color information on one light receiving surface, each of the above color image pickup devices performs color modulation (color coding) in a light receiving plane using a color filter called a stripe filter or a mosaic filter. ing. That is, for example, by attaching filters of three colors of RGB in a predetermined regular arrangement on each photoelectric conversion element (pixel), each pixel has a different spectral sensitivity. Therefore, since the video signal obtained by subject imaging contains dot-sequential color information according to the filter array, the video signal is separated into signals corresponding to the respective color filters in accordance with the above-described predetermined array. By extracting the separated signal, color information can be extracted. In order to obtain a luminance signal (Y signal), all of the RGB information is required. To obtain the luminance information of one pixel, at least three pixels (one for each of RGB) are required, and the luminance resolution is sacrificed. However, color imaging can be performed with one imaging device.

The above filter arrangement includes three primary color filters such as an RGB stripe, a Bayer type RGB mosaic (various types), a YeMgCy stripe, and a YeMgCyW4.
Various color coding patterns, such as complementary color filters such as a color mosaic and a YeMgCyG four-color mosaic, have been proposed and put to practical use.

[0005]

SUMMARY OF THE INVENTION The present invention relates to an electric configuration of the above-mentioned color image pickup device (such as an image pickup tube or a solid-state image pickup device, or a CCD or another type) and a type of color coding (primary color or complementary color). Or three or four colors), which point out the essential problems that they have, regardless of which one they have, and suggest solutions to them. In the following description, only one example will be used unless otherwise specified. , Will be described.

In the above conventional color coding arrangement, RG
An example of the B-Bayer array will be described with reference to FIGS. The RGB Bayer arrangement corresponds to the 2 shown in FIG.
The basic array is composed of four pixels of × 2, and the basic arrays are sequentially arranged as shown in FIG. 3B so as to fill up the plane, and the distribution ratio of the number of pixels to each color of RGB is determined. The characteristic is that the luminance resolution is increased by increasing the density of G that greatly contributes to the luminance signal as 1: 2: 1. In addition, since it is isotropically arranged in two directions, vertical and horizontal, an isotropic resolution can be obtained unlike a stripe filter. FIG. 3B illustrates an example of arbitrary 8 × 8 = 64 pixels.

However, since the Bayer array uses a regular array as described above, there is a serious problem that false resolution based on spatial sampling by the array, that is, so-called color moire occurs. . That is, when there is a periodic subject having a luminance change (monochrome pattern) having the same cycle as the above-described arrangement cycle in a black and white subject having no color, for example, paying attention to the RG row as one horizontal line, R is white and G is black. If so, a signal equivalent to a signal obtained from a red subject having no change in luminance is output, so that a color output that does not exist originally occurs. Since a false color signal, that is, a color moiré, generated by such a striped repetition pattern is generated in a low frequency region by so-called frequency aliasing (aliasing), the false color signal can also be obtained by an electrical filtering process including color band suppression in a subsequent stage. It cannot be removed.

For this reason, the conventional single-chip color image pickup device is
In order to ensure image quality, an optical low-pass filter such as quartz is essential for the optical system, which not only imposes a great restriction on miniaturization and cost reduction, but also inevitably causes image quality deterioration due to remaining color moiré.

SUMMARY OF THE INVENTION The present invention has been made to substantially solve the problem associated with the above-described periodic color coding arrangement in a conventional color image pickup apparatus.
Even with a subject having a periodic luminance change, color moiré does not occur and small-sized, low-cost, high-quality color imaging is possible, and even if a pixel defect exists in the image sensor, a good image can be obtained by supplementing the defective pixel. It is an object of the present invention to provide a color imaging device which can be used. The invention according to claim 2 provides a color image pickup apparatus that uses a color image pickup device that satisfies a predetermined color coding arrangement restriction condition, so that even if an isolated pixel defect exists, the resolution is not substantially reduced. The purpose is to provide. The invention according to claim 3 is
It is an object of the present invention to provide a color imaging apparatus that can easily cope with color separation processing or pixel defect correction processing of a color imaging element having a different color coding arrangement or pixel defect mode.

[0010]

According to a first aspect of the present invention, there is provided a pixel group including a plurality of pixels each including a photoelectric conversion element, and a color coding of the pixel group. A color image sensor having a random array,
A storage unit that stores array data and pixel defect data related to a color coding array of the color imaging element, and a color separation unit that generates a color signal based on the color coding array data stored in the storage unit. The color separation means constitutes a color image pickup apparatus by performing a predetermined pixel defect correction process based on the pixel defect data stored in the storage means in the color signal generation process.

[0011] In the color image pickup apparatus thus configured, a subject is imaged using a color image pickup element having a random color coding arrangement having no periodicity, and a color signal is generated based on the color coding arrangement data. As a result, high-quality color imaging can be performed without color moiré even in a subject having a periodic luminance change, and pixel defect correction is performed based on pixel defect data of the color image sensor. With such a configuration, even if a pixel defect exists in the color image sensor, a good image in which the defective pixel is compensated can be obtained.

According to a second aspect of the present invention, in the color image pickup apparatus according to the first aspect, the predetermined pixel defect correction processing by the color separation means is performed in the same color as the color of a signal to be complementarily generated for the defective pixel. The process is a process of complementing the defective pixel output signal by using an output signal of a pixel closest to the defective pixel among the pixels. With this configuration, when a color image pickup device that satisfies a predetermined color coding arrangement restriction condition is used, even if an isolated pixel defect is present, it is possible to prevent degradation in resolution from actually occurring.

According to a third aspect of the present invention, in the color imaging apparatus according to the first or second aspect, the storage means includes an EEP.
It is characterized by being constituted by a ROM.
With such a configuration, it is possible to easily cope with the color separation processing or the pixel defect correction processing of the color imaging elements having different color coding arrangements or pixel defect modes.

[0014]

Next, an embodiment will be described. FIG. 1 is a block diagram showing an embodiment of a color imaging device (digital camera) using a color imaging device according to the present invention. In FIG. 1, 1 is a lens system, 2
Is a lens driving mechanism, 3 is an exposure control mechanism, 4 is a CCD image sensor, 5 is a CCD driver, 6 is a pre-processing circuit including an A / D converter, 7 is a digital processing circuit, and includes a memory as hardware, It performs digital process processing. 8 is a memory card interface, 9 is a memory card, 10 is an LCD image display system, 11 is a system controller including a microcomputer as a main configuration,
12 is an operation switch system, 13 is an operation display system, 14 is a strobe,
15 is a lens driver, 16 is an exposure control driver, 17 is EE
PROM.

FIG. 2 shows an example of a random arrangement of color filters of the CCD image sensor 4 used as a color image sensor in the embodiment shown in FIG. The number of pixels of the CCD image pickup device is arbitrary, but is assumed to be about 1 million pixels. In FIG. 2, only the filter arrangement corresponding to 8 × 8 = 64 pixels at the center is shown. In the following description, a procedure for obtaining such a random sequence is specifically described, and the illustration is merely to assist the understanding, and the illustration of such a region is sufficiently understood. There will be. (In addition, since the random arrangement is the essence of the present invention, it is meaningless and impossible to exemplify the pattern of the entire region.) The type of the filter in the present embodiment uses so-called RGB three primary colors. , The coding is RGB random filter coding.

Next, an example of a procedure for obtaining such coding will be described. Since this coding is random coding, it is a matter of course that dice having two planes each assigned to RGB may be used to determine the color filter of each pixel. However, in order to reduce the complexity, spreadsheet software is used. A table array corresponding to the entire pixel array is prepared using the above method. Then, in each cell of the array, the formula MOD (RND / 3) (where RND is a random number function having an appropriate number of digits, and MOD (n / d) is a remainder function obtained by dividing n by d)
Is assigned to, for example, 0 → R, 1
→ G, 2 → B may be applied.

The sequences obtained in this way usually do not statistically have a particularly large bias, but those obtained by only one trial are extremely low in color, although they are stochastically low. There is a possibility that the number of pixels is large or small and a specific color is concentrated over a large area. Or,
The possibility of a pattern having periodicity as in the conventional example is extremely low, but is not zero. Therefore, after performing a plurality of trials by the above method to obtain a plurality of array samples, it is possible to perform an imaging test (actually, it is preferable to use a simulation) by a real photograph and adopt a sample having a good evaluation result. desirable.

However, even if such a trial method is unavoidable in the final sequence selection, all trials from the beginning of the design are based only on trials.
Generally, it is not preferable because it significantly lowers the design efficiency. Alternatively, even when evaluating an array obtained by trial, in order to obtain good image quality, there must be objective requirements required for the array itself which are essential (for extreme examples, all It is self-evident that the condition must not be a pixel of only one color.) It is extremely effective to specifically find such a condition and adopt this as a limiting condition (judgment criterion).

More specifically, the CCD in the present embodiment
The RGB three-color random arrangement of the image sensor is such that an arbitrary pixel of interest has two pixels other than the filter color (own color) of the pixel of interest.
The color is adjacent on any of the four sides of the pixel of interest, and the same color as the filter color (own color) of the pixel of interest is adjacent on any of the four sides or four corners of the pixel of interest Is adopted as a constraint. That is, if the filter of the target pixel is R, the constraint at the preceding stage is a condition that G and B are included in at least one pixel each of the four adjacent pixels on the upper, lower, left, and right sides. This condition guarantees that complementation with the nearest pixel information in the color separation processing described later is always performed by any of the adjacent pixels, either up, down, left, or right. Guarantee. In addition, according to the constraint conditions at the subsequent stage, even if a pixel defect (however, an isolated one is assumed) occurs,
Since the same color as the color of the filter of the target pixel (own color) always exists in the eight adjacent pixels in the vertical, horizontal, and oblique directions, if this pixel defect is complemented, the resolution will actually deteriorate. Not something. In the arrangement example shown in FIG. 2, some pixels in each peripheral row apparently do not satisfy the above-described arrangement constraint condition, but this condition is satisfied by the presence of a further outer pixel not shown. Things. Under such circumstances, the photoelectric conversion surface of the image sensor has a margin of one to several rows (columns) for each of the four turns than the effective image area, so that a so-called discarded pixel area (which is involved in image signal generation, (An area that is not an image area).

The sequence satisfying such a constraint condition can be prepared by preparing a large number of completely random sequences by trial and testing them under the above conditions, or by generating a sequence by software processing such as a spreadsheet. In this case, any of them can be obtained by imposing constraints beforehand and generating them.

In a color image pickup apparatus (digital camera) using the CCD image pickup device 4 having such a random color filter array, signals are read out and processed in the same manner as in a conventional camera, and the picked-up image is stored in a memory card. 9
Or displayed on the LCD image display system 10. An operation different from the conventional one is a color separation process, which is performed by the digital process circuit 7 under the control of the system controller 11. Needless to say, the color separation processing basically means that a pixel (for example, B
This is a signal complementing process using neighboring pixel information and the like for the R filter pixel in the signal generating process, and there is no difference in this point from the conventional one. However, in the conventional color separation, regular sampling based on permutation is performed in accordance with the regular color coding of the CCD image pickup device, and simple complementation using a hold circuit or the like, and addition and subtraction between pixels as necessary. (Specific processing includes various types such as analog processing, digital processing, and hybrid processing), but the random coding applied in the present invention has no regularity, and thus such processing cannot be performed. Therefore, the color separation process is performed with reference to the filter coding data (the filter table of all the pixels corresponding to FIG. 2) for each pixel of the CCD image sensor to be used. This coding data is stored in EEPROM 17
And can cope with the case where the color coding of the CCD image sensor used is different.
The EEPROM 17 also stores pixel defect data indicating which pixel is a defective pixel that cannot output information normally. Then, based on the pixel defect data, the pixel defect correction processing is performed together with the color separation processing in the digital process circuit.

The specific color separation (color signal generation) processing of this embodiment is performed as follows. That is,
"For the target pixel of interest, first determine whether the pixel is a defective pixel by referring to the pixel defect data. If the pixel is not a defective pixel, refer to the coding data. Then, the signal level of the pixel is output as it is as a signal, and for other color signals, the nearest corresponding pixel of the color filter (excluding the defective pixel) is searched for in the neighboring pixels based on the coding data, and The signal level of the pixel to be processed is output as the color signal.
Based on the coding data for all color signals, the nearest pixel of the corresponding color filter (excluding the defective pixel) is searched for in the neighboring pixels, and the signal level of the corresponding pixel is output as the color signal. Is performed.

An example of the color separation processing when the filter of the target pixel to be processed is R is as follows. (1) The output when the R pixel that is the target pixel to be processed is not a defective pixel is as follows. R signal: Since the reference result of the coding data is R, the signal level of the pixel is output as it is as an R signal. G signal: Since the reference result is not G, the nearest pixel (not a defective pixel) among neighboring pixels based on the coding data
A G pixel is searched, and the signal level of the corresponding pixel is output as a G signal. B signal: Since the reference result is not B, the nearest pixel (not a defective pixel) among neighboring pixels based on the coding data
The B pixel is searched, and the signal level of the corresponding pixel is output as a B signal. (2) The output when the R pixel, which is the target pixel to be processed, is a defective pixel is as follows. R signal: Searches for the nearest (non-defective) R pixel among neighboring pixels based on coding data, and outputs the signal level of the corresponding pixel as an R signal. G signal: A nearest G pixel (not a defective pixel) is searched for among neighboring pixels based on coding data, and the signal level of the corresponding pixel is output as a G signal. B signal: A closest B pixel (not a defective pixel) is searched for in a nearby pixel based on coding data, and the signal level of the corresponding pixel is output as a B signal.

In the above example of the color separation processing, for example, (R, G,
B, B) Assuming that filter pixels are arranged, two “nearest B” pixels exist on the left and right. In such a case, even if only one of them is adopted,
An average value of both may be adopted.

The color signals obtained as a result of the above-described color separation processing are processed as a synchronized RGB three-primary-color signal for all pixels by a subsequent circuit in the same manner as a conventional RGB three-primary-color signal, and finally are stored in a memory. Recorded on card 9 or L
The image is displayed on the CD image display system 10. The processing in the circuit at the subsequent stage is appropriately used according to the necessity, and is known per se, for example, by color balance processing, conversion into a luminance-color difference signal by matrix operation or inverse conversion processing thereof, band limitation, and the like. False color removal or reduction processing, various non-linear processing represented by γ conversion, various information compression processing, and the like.

Considering the false color generated at that time, regarding a subject such as a black-and-white knife edge or an isolated white point (line), the false color is naturally affected by the planar color coding as in the conventional coding. Occurs. However, these are all isolated (false) color points and color lines, and their main energy is distributed in a high frequency range. It is possible to reduce. In the imaging of a striped repetitive pattern, which has been the biggest problem in the past, since coding is random, at least a low-frequency false color (color moiré) folded to a low frequency does not occur. In the embodiment, only the generation of the isolated false color that can be removed or reduced is described above. Therefore, in the present embodiment, a high quality image can be obtained with virtually no false colors that are visually problematic despite the fact that an optical low-pass filter, which is conventionally required in this type of digital camera, is not used. .

Further, since the nearest pixel information in the color separation processing is always complemented by any of the adjacent pixels in the upper, lower, left and right directions, the maximum value of the pixel blur, that is, the PSF
The maximum value of the width of (point spread function) is limited to 3 pixels,
Compared to the conventional case where the average value of peripheral pixels is used instead of one pixel for complementation in color separation in the Bayer array, the same color resolution for G and about twice the color resolution for RB should be secured. Can be. Considering complementation in the case where an isolated pixel defect is present, in the case of the Bayer array, the width of the PSF must be allowed to be a maximum of 5 pixels. The isolated pixel defect can be complemented by the color (own color) of the pixel of interest existing in any of the eight adjacent pixels in the direction, so that the maximum width of the PSF is at least three pixels in the horizontal and vertical directions. It does not change. (With regard to the complement of the color of the pixel of interest, when the complement of other colors is considered, the maximum width is 7 pixels and 5 pixels, respectively.) Moreover, this is a comparison of only the effect of pure pixel sampling, Since the low-pass filter is not used, the response is not reduced, and the resolution can be increased to a frequency region where resolution cannot be achieved conventionally.

Although the present invention has been described based on the above embodiment, various modifications of the above embodiment are conceivable. First, in the above-described embodiment, the coding data and the pixel defect data are stored in the EEPROM 17, so that it is possible to cope with the case where the coding of the CCD image pickup device used is different. From the viewpoint of sharing the memory, this is one natural configuration, but E is a storage means for storing these two types of data.
The EPROM may be separate. At this time, considering only the coding data, only one type of imaging device (the same except for mass production variation) is usually applied to one imaging device body, and there is no need to change color coding. Can use the same data. Focusing on this point, the EEPROM can be replaced with, for example, a mask ROM. When replaced with a mask ROM, it can be configured at lower cost. Further, in either case, it goes without saying that the memory as the storage means for these two types of data can also be used as the program storage memory of the microcomputer of the system controller 11.

Also, as described at the beginning, the present invention relates to an electric configuration of the image pickup device (such as an image pickup tube or a solid-state image pickup device, or a CCD or another type) and a type of color coding (primary color or complementary color, Or, three colors or four colors) is intended to address and solve the essential problems irrespective of the color, and the present invention is not limited to the above-described embodiment, and all color imaging devices including these and the It can be applied to the used color imaging device. For example, multicolor random coding such as five colors or six colors may be used, or the invention may be applied to a multi-chip camera. For example, the present invention can be applied to an R / B sensor of a G, R / B type two-plate color camera as it is.

Further, the “procedure for obtaining random coding” described in the above embodiment is merely an example, and the random coding itself can be obtained by any method. That is, unlike the conventionally known regular arrangement, the color coding arrangement is noticeably regular (periodic) when focusing on a predetermined area of at least several pixels to several tens of pixels of the photoelectric conversion element arrangement. If it does not have a structure and consequently reduces the occurrence of low-frequency false colors for a striped repetitive pattern input caused by a conventional regular arrangement, that arrangement will be a random color coding arrangement in the present invention. .

[0031]

As described above with reference to the embodiment, according to the first aspect of the present invention, a subject is imaged using a color image pickup device having a random color coding array, and the color coding array data is stored. The color signal is generated based on the image data, so that high-quality color imaging can be performed without color moiré even in a subject having a periodic luminance change, and based on pixel defect data of the color imaging device. Therefore, even if a pixel defect exists in the color image pickup device, a good image in which the defective pixel is compensated can be obtained. According to the second aspect of the present invention, by using a color image pickup device satisfying a predetermined color coding arrangement restriction condition, even if an isolated pixel defect is present, it is possible to substantially prevent degradation in resolution. is there. According to the third aspect of the present invention, since the EEPROM is used as the storage means for storing the color coding arrangement data and the pixel defect data, the color separation processing of the color image pickup device having the different color coding arrangement or the pixel defect mode or the pixel separation processing is performed. It is possible to easily cope with defect correction processing.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of a color imaging apparatus according to the present invention.

FIG. 2 is a diagram showing an example of a random color filter array of the CCD image sensor in the embodiment shown in FIG.

FIG. 3 is a diagram showing an example of a basic array and an overall array of an RGB Bayer array.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Lens system 2 Lens drive mechanism 3 Exposure control mechanism 4 CCD image sensor 5 CCD driver 6 Pre-process circuit 7 Digital process circuit 8 Memory card interface 9 Memory card 10 LCD image display system 11 System controller 12 Operation switch system 13 Operation display system 14 Strobe 15 Lens driver 16 Exposure control driver 17 EEPROM

Claims (3)

[Claims] The present invention relates to a color image pickup device having a pixel group formed by arranging a plurality of pixels each formed of a photoelectric conversion element, wherein a color coding arrangement of the pixel group is randomly arranged, and a color coding arrangement of the color imaging device. Storage means for storing array data and pixel defect data; and color separation means for generating a color signal based on the color coding array data stored in the storage means, wherein the color separation means A color image pickup device configured to perform a predetermined pixel defect correction process based on the pixel defect data stored in the storage means in a generation process. 2. The predetermined pixel defect correction processing by the color separation unit uses an output signal of a pixel closest to the defective pixel among pixels of the same color as a color of a signal to be complementarily generated for the defective pixel. 2. The color imaging apparatus according to claim 1, wherein the processing is a processing for complementing the defective pixel output signal. 3. The color imaging apparatus according to claim 1, wherein said storage means is constituted by an EEPROM.