JP2001036809A - Solid-state image pickup device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a solid-state image pickup device capable of providing an image suitable to user's request without increasing capacity of a storage medium and an image processing method. SOLUTION: All primary colors are interpolated, generated in parallel based on a data correcting part 36a of a signal processing part 36 by RGB synchronizing parts 36A to 36N of an image generating part 36b, one of the pieces of obtained image data is selected via a selector 36c and supplied to a matrix processing part 36d by a digital camera 10. Luminance data and color difference data generated from the image data are transmitted to a monitor 42 by the matrix processing part 36d. Thus, a chance for an image processing which is not found in the conventional digital camera how to easy out composition at an optimal image composition part 36e is performed from the image to be displayed on the monitor 42 is provided to a user. An optimal image is provided by controlling replacement and synthesis with each image area selected according to an instruction from an operation part 14 by a system control part 18.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a solid-state imaging device and an image processing method, and performs image processing according to a plurality of specific images based on obtained pixel data. It is suitable for use in processing equipment and the like.

[0002]

2. Description of the Related Art A method of directly recording an electric signal obtained based on signal charges photoelectrically converted by an image pickup device as so-called raw data, a recording method of the raw data and compression processing by performing signal processing on the raw data, for example, JPEG (Joint Photog
raphic Experts Group) standard. Of the latter, in general,
For example, image creation is performed by one signal processing (including software processing) mounted on an image creation in a digital camera. Specific examples of these methods are given below.

For example, a display method and a display device of a digital electronic camera described in Japanese Patent Application Laid-Open No. H10-145719 are required to efficiently display a raw data image and a generated image. By easily distinguishing and displaying the thumbnails of the image data after the processing and the thumbnails of the image data before the signal processing in different modes, erroneous display operations can be reduced, and the image data before the signal processing can be compared with the image data before the signal processing. By performing the processing, high-speed switching display of thumbnail images is performed.

In addition, regarding the configuration and processing of obtaining interpolation by using pixel data of an image pickup signal to obtain an image and simultaneously obtaining plane pixel data of three primary colors RGB, see Morimoto et al., “3-CCD Digital Still Camera”, 70th Anniversary Fine Imaging Symposium Proceedings, D-5, No.
73-76, Photographic Society of Japan, (October 1995)
This digital camera is composed of two CCDs in which the relationship of the pixel positions of the color G is shifted by a pixel in the three-chip system,
Signal processing is performed on a signal obtained by alternately allocating R and B to a CCD having the relationship of the position of the vertical stripe, thereby achieving high resolution.

[0005] In such a digital image input device, an image is created by subjecting an input digital signal to signal processing by software, for example. The created image is recorded on a recording medium according to a standard format. The user views the captured image of the object scene on a monitor or the like after the signal processing directly or after the reproduction from the recording medium, for viewing. Further, the user may perform image processing on the reproduced image from the recording medium using a personal computer, for example, so as to bring a desired effect to the reproduced image.

[0006]

In the single-panel digital camera, a false signal is generated at a color boundary as compared with the above-described three-panel digital camera. Often obtained.

As described above, in a single-chip digital camera, if an image generated by signal processing or image processing is not an image satisfactory to the user, it is difficult to restore raw data in consideration of the structure of the generated image. . Therefore, once the signal processing or the image processing has been performed, it is impossible to make a new change by appropriate image processing. Such a problem of the latter method is caused by performing the processing in one of the methods as described above. To solve this problem, it is only necessary to record both the raw data and the image data after the signal processing.

However, recording both data on a recording medium mounted on a camera affects the number of images to be captured, resulting in extremely uneconomical recording. Further, for the restoration of the raw data described above, a dramatic improvement in the processing capability of a personal computer is desired.

An object of the present invention is to provide a solid-state imaging device and an image processing method capable of solving the above-mentioned drawbacks of the prior art and providing an image according to a user's request without increasing the capacity of a recording medium. With the goal.

[0010]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, the present invention provides a color separation for color-separating incident light from an object field via an optical system in accordance with the position of a light receiving element for photoelectrically converting the light. The color-separated light is imaged in color by imaging means in which light receiving elements are two-dimensionally arranged, and the obtained color imaging signal is converted into digital image data by digital conversion means, and recording means for recording the image data and / or In a solid-state imaging device that outputs the image data to a recording unit via a signal processing unit that performs signal processing on the image data,
Data correction means for correcting image color adjustment and gradation to digital image data, based on image data from this data correction means, at least interpolating and generating at least three primary colors R, G, B at the position of the light receiving element in parallel A plurality of interpolation generating means; a selecting means for selecting one of the plurality of interpolation generating means; generating luminance data and chrominance data based on image data supplied via the selecting means; Display image generation means for performing processing for preventing the occurrence of the image and performing contour enhancement processing on the processed luminance data, and outputting each of a plurality of interpolation generation means to one image data from the display image generation means. By replacing the image data with the optimal image area
A signal processing means including a synthesizing means for synthesizing an image, a display means for displaying the image data obtained by the display image generating means, and checking and evaluating the image displayed on the display means to obtain a result of the evaluation. And control means for controlling cut-out and replacement of an optimal image area in the synthesizing means in response to an instruction from the operating means for providing a corresponding instruction.

Here, the color separating means matches a color filter for separating the incident light into the three primary colors R, G, and B to the light receiving element of the imaging means, and the imaging means comprises a light receiving element and a light receiving element adjacent to the light receiving element. It is preferable that the elements are two-dimensionally arranged so as to be shifted from each other by a half pitch in the vertical and horizontal directions.

[0012] The color separation means is a color separation device for the three primary colors R, G, and B.
It is preferable that G is arranged in a square lattice pattern, and that the colors R and B are completely a checkered color filter pattern with respect to the color G.

[0013] The plurality of interpolation generating means may include at least three primary colors.
Among the R, G, and B, the pixel data of the color G is defined as high-range pixel data, and the color G interpolating at the light receiving element in the colors R and B different from the color G based on the high-range pixel data is performed. Interpolating means, R interpolating means for interpolating the color R using the pixel data of the color R which is correlated with the G interpolating means, and color interpolation of the color B using the pixel data of the color B which is correlated with the G interpolating means. First interpolation including B interpolation means for performing interpolation, and average interpolation means for interpolating and generating plane pixel data of each color by average interpolation using pixel data obtained by the G interpolation means, the R interpolation means and the B interpolation means, respectively. Generating means, luminance data generating means for obtaining luminance data at the position of the light receiving element from the pixel data of the three primary colors R, G, B obtained by the color separating means, and a light receiving element based on the luminance data obtained by the luminance data generating means. Brightness data at the gap position The required luminance data interpolation means, the R interpolation developing means for interpolating the color R using the luminance data interpolation means and the color R pixel data, the G interpolation for interpolating the color G using the luminance data interpolation means and the color G pixel data A second means including a developing means and a B interpolation developing means for interpolating the color B using the luminance data interpolating means and the color B pixel data;
It is desirable to include interpolation generating means.

The control means includes a correction control of the data correction means,
The image generation control of the display image generation unit and the selection switching control of the selection unit may be performed, and the generation of signals used in this device may be controlled.

In the solid-state image pickup device according to the present invention, the three primary colors R, R and R are respectively processed in parallel by a plurality of interpolation generating means based on the image data corrected by the data correcting means in the signal processing means.
G and B are interpolated and generated in parallel. Thus, the selecting means can immediately supply the image data subjected to the interpolation generation to the display image generating means when one of the image data already generated by the interpolation is selected. The display image generating means sends the luminance data and the color difference data generated based on the image data to the display means. The display means displays an image supplied by the selection. The combining means replaces each image area selected under the control of the control means in accordance with an instruction from the operating means and combines them to generate one high-quality image.

Further, in order to solve the above-mentioned problems, the present invention captures a color image of incident light from an object field via an optical system, and converts an image signal obtained by the color image into digital image data. In an image processing method for recording this image data and / or recording the image data through signal processing, a data correction step for correcting color and gradation of an image to a digital image signal, An interpolation generation step including a plurality of plane interpolation generations in which all three primary colors R, G, and B are interpolated and generated in parallel based on the obtained image data and / or at the position of the gap of the light receiving element adjacent to the light receiving element; An image selecting step of selecting one image interpolated and generated from a plurality of types of image data obtained in the step, and based on the image data obtained in the image selecting step. The signal processing step includes generating a degree data and a color difference data, performing a process for preventing the occurrence of aliasing distortion on these data, and performing a contour enhancement process on the processed luminance data in the signal processing step. An image display step of displaying image data obtained in the image generation step on a prepared display means; and an image area evaluated by the image displayed in the image display step and an image area of another image generated by another interpolation. A first determining step of making a determination to replace the image, an optimal combining step of performing an optimal image generation process in accordance with the first determining step, and selecting the other area after the optimal combining step. A second determining step of determining whether to repeat the processing using the image generated by the interpolation generation, and a third determining step of determining whether to use another interpolation generation when the processing is terminated in the second determining step. The signal processing step, and when repeating the processing in the second and third determination steps, performs a series of processing of selecting a new image and cutting out and replacing the selected image, respectively. I do.

Here, the optimal synthesizing step includes a reference image selecting step of selecting a reference image from the generated images, a cutout selecting step of selecting an image generated by a process different from the image of the reference image selecting step, An area specifying step of specifying an area to be cut out from the image selected in the cutout selecting step,
It is preferable to include a replacement step of replacing the area specified in the area specifying step with the corresponding area of the reference image.

Preferably, the optimum combining step includes an area matching step of matching the indicated cut-out area of the evaluated image and the replacement area of the image.

In this image processing method, when the image pickup signal is recorded as it is, it is desirable that a reproducing step of reproducing the recorded digital image data be performed before the signal processing step.

In the color imaging, the color G among the three primary colors R, G, and B is arranged in a square lattice, and the color R and the color G are photographed with a color filter pattern that is completely arranged in a checkered pattern. Preferably, a signal is used.

According to the image processing method of the present invention, the three primary colors R, at the position of the gap of the light receiving element and / or the light receiving element adjacent to the light receiving element are determined based on the image data corrected.
Since a plurality of plane interpolation generations are performed in parallel for all G and B, a plane image at the time of image selection can be output instantaneously. The luminance data and the color difference data obtained based on the image data supplied by the image selection are displayed on the display means. At this time, image evaluation is performed based on the displayed image, and a determination is made to replace one image with an image area by another interpolation generation, thereby performing an optimal image generation process. Further, thereafter, it is determined whether or not to repeat the process using the image generated by the interpolation generation selected for another region, and when ending this process, it is determined whether to use another interpolation generation, and further processing is performed. Then, each image area is combined to generate one image.

[0022]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, an embodiment of the solid-state imaging device according to the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 shows the configuration of a digital still camera 10 according to an embodiment to which the present invention is applied. The digital still camera 10 of FIG. 1 includes an optical lens system 12, an operation unit 14, a system control unit 18, a signal generation unit 20, a timing signal generation unit 22, a driver unit 24, an aperture adjustment mechanism 26, an optical low-pass filter 2
8, color separation unit CF, imaging unit 30, preprocessing unit 32, A / D conversion unit 3
4. A signal processing unit 36, a compression / expansion unit 38, a recording / reproducing unit 40, and a monitor 42 are provided. These components will be described sequentially. The optical lens system 12 is configured by combining a plurality of optical lenses, for example. Although not shown, the optical lens system 12 has a zoom mechanism that adjusts the positions at which these optical lenses are arranged and adjusts the angle of view of the screen according to an operation signal from the operation unit 14, and the distance between the subject and the camera 10 It includes an AF (Automatic Focus) adjustment mechanism that adjusts the focus accordingly. The operation signal is supplied to the system control unit 18 via the system bus 16. Optical lens system 12
The signal generator 20 and the timing signal generator 2 described later
2. A drive signal is supplied via the driver unit 24.

The operation unit 14 has a shutter switch (not shown) and a function of selecting an item displayed on a monitor screen, for example. In particular, the shutter switch outputs an operation signal to the system controller 18 via the system bus 16 so as to operate the camera 10 in each of a plurality of stages. Further, in the present embodiment, the operation unit 14 also operates a pointing device displayed on a monitor so that an optimum image generation mode can be selected and a partial area of an image can be specified. In this case, the operation of the operation unit 14 is also performed by the system control unit 18.
Is supplied as an operation signal.

The system control unit 18 includes, for example, a CPU (Cent
ral Processing Unit). The system control unit 18 has a ROM (Read Only Memory) in which the operation procedure of the digital still camera 10 is written. The system control unit 18, for example,
A control signal for controlling the operation of each unit is generated using the information supplied from the operation unit 14 in accordance with the operation of the user and the information of the ROM. The system control unit 18 transmits the generated control signal to the signal generation unit 20, although not explicitly showing the supply of the control signal, in addition to the timing signal generation unit 22, the preprocessing unit 32, and the A / D conversion unit 34, the system bus The signal is also supplied to a signal processing unit 36, a compression / expansion unit 38, a recording / reproducing unit 40, and a monitor 42 via 16. In particular, the system control unit 18 performs various controls on a signal processing unit 36 described later.

The signal generator 20 generates a system clock by an oscillator under the control of the system controller 18. The signal generator 20 supplies this system clock to the timing signal generator 22 and the signal processor 36. Also,
The system clock is also supplied via the system bus 16 as a reference for the operation timing of the system control unit 18, for example.

The timing signal generating section 22 includes a circuit for generating a timing signal for operating each section based on the supplied system clock based on a control signal. The timing signal generator 22 outputs the generated timing signal to each unit as shown in FIG. The driver unit 24 also supplies drive signals to the aperture adjustment mechanism 26 and the imaging unit 30 in addition to the zoom adjustment mechanism and the AF adjustment mechanism of the optical lens system 12 described above.

The aperture adjusting mechanism 26 is a mechanism for adjusting the cross-sectional area of the incident light beam (that is, the aperture opening area) so as to supply the optimum light beam of the incident light to the imaging section 30 in photographing the subject. A drive signal is also supplied from the driver unit 24 to the aperture adjustment mechanism 26. This drive signal is supplied to the system control unit described above.
This is a signal for operating according to the control from 18. In this case, although not shown, the system control unit 18
AE (Automatic Exposure)
: Automatic exposure) The aperture and exposure time are calculated as processing. A drive signal corresponding to the signal from the timing signal generation unit 22 to which the control signal corresponding to the calculated value is supplied is supplied from the driver unit 24 to the aperture adjustment mechanism 26.

The image pickup section 30 arranges the image pickup element for photoelectric conversion so that a plane perpendicular to the optical axis of the optical lens system 12 is formed. Also, on the incident light side of the image sensor, a color filter CF that performs color separation integrally with an optical low-pass filter 28 that limits the spatial frequency of an optical image to a Nyquist frequency or lower corresponding to each image sensor is arranged integrally. Is established. In this embodiment, imaging is performed using a single-plate type color filter. The type of the color filter CF will be described in more detail later. CCD (Charge Coupled Device) and M
There is an OS (Metal Oxide Semiconductor) type. The imaging unit 30 reads out all the pixels of the signal charges obtained by the photoelectric conversion according to the supplied drive signal. Further, the image pickup section 30 is not limited to reading out the image pickup signal, but may read out the image signal by interlace or X, Y address method.

Although not shown, the pre-processing unit 32 includes a CDS (Corr
elated Double Sampling: Correlated double sampling; below
CDS) section. The CDS unit uses, for example, a CCD type imaging device, and basically clamps various noises generated by the device by a timing signal from the timing signal generation unit 22, and holds a signal charge by the timing signal. It has a sample and hold circuit. The CDS section removes noise components
The signal is sent to the A / D converter 34. The A / D converter 34 has an A / D converter that quantizes a signal level of an analog signal, which is a supplied signal charge, with a predetermined quantization level and converts the signal level into a digital signal. The A / D converter 34 is a timing signal generator
A digital signal converted by a timing signal such as a conversion clock supplied from 22 is output to the signal processing unit 36.

The signal processing section 36 includes a data correction section 36a, an image generation section 36b, a selector 36c, a matrix processing section 36d, and an optimum image synthesis section 36e. Data correction unit 36a
A gamma correction circuit (not shown) for color correction and AWB (Automatic W
hite Balance) circuit. In particular, the gamma correction circuit uses a look-up table, which is a set of a plurality of data sets in which a digital signal supplied to a ROM (Read Only Memory) and correction data output corresponding to the digital signal are combined. The data correction unit 36a is not limited to this arrangement, and may be provided at a subsequent stage.
By arranging at this position, the number of lookup tables can be minimized. Also in these series of data correction, the data is supplied according to the timing signal from the timing signal generation unit 22. The data correction unit 36a outputs the processed correction data to the image generation unit 36b.

The image generating section 36b basically determines the position of a gap without a light receiving element, that is, the pixel data at the virtual light receiving element or virtual pixel from the surrounding pixel data with respect to the pixel data obtained from the supplied light receiving element. Interpolate and generate one of the three primary colors R, G, and B. The three primary colors R,
G and B plane image data can be obtained simultaneously. The image generating unit 36b includes a plurality of RGB synchronizing units 36A to 36N for simultaneously generating the three primary colors R, G, and B. In the present embodiment, two RGB synchronizing units are added to the image generating unit 36b.
The case where 36A and 36B are provided will be described.

As shown in FIG. 2, the RGB synchronization unit 36A includes a high-frequency G data generation unit 360A, an interpolation function unit 362A, and an average plane interpolation function unit 364A. The high-frequency G data generation unit 360A is an arithmetic function unit that generates pixel data of the color G at a position obtained by rotating the arrangement of the color filters by 45 °.
Needless to say, it also has a data arrangement conversion function of rotating the above arrangement by 45 °. High-frequency G data generator 360A
Outputs the generated pixel data of the color G to the interpolation function unit 362A.

The interpolation function unit 362A includes an R interpolation function unit 3620A
And B interpolation function unit 3622A. Pixel data of the color G is supplied to the R interpolation function unit 3620A and the B interpolation function unit 3622A, respectively. The R interpolation function unit 3620A and the B interpolation function unit 3622A have color R data and color
B Data is supplied. Each of the R interpolation function unit 3620A and the B interpolation function unit 3622A has a data arrangement conversion function for speeding up processing, and also converts the pixel data of the color R and the pixel data of the color B at all the positions where the light receiving elements exist. It has a calculation function to generate based on the correlation of G pixel data. When considering the correspondence between each color and the interpolation function unit, the high-frequency G data generation unit 360A described above is used.
May be used as the G interpolation function unit. Thus, the interpolation function unit 36
2A outputs the generated pixel data of R, (G), and B to the average plane interpolation function unit 364A. Average plane interpolation function unit 36
4A is supplied with the pixel data of the color G that has passed through the interpolation function unit 362A, and the rotation of the obtained pixel data is returned (−45 °). This makes the G square RB perfect checker pattern again. The average plane interpolation function unit 364A
An arithmetic function unit that interpolates and generates pixel data at a position of a virtual pixel having no light receiving element based on pixel data located in the periphery corresponding to the position of the light receiving element. The average plane interpolation function unit 364A supplies the generated three primary color RGB plane data to the selector 36c.

As shown in FIG. 3, the RGB synchronizing unit 36B includes a luminance data generation function unit 360B, a luminance data interpolation function unit 362B, and a high resolution plane interpolation function unit 364B. The luminance data generation function unit 360B is an arithmetic function unit having a function of generating luminance data at the position of the light receiving element based on pixel data obtained from the light receiving element. The luminance data generation function unit 360B calculates the luminance data Y (or the high-frequency luminance data) at the pixel where the light receiving element is located by arithmetic processing.
It generates a representative) with Y _h, and outputs the luminance data interpolation function unit 362B. The luminance data interpolation function unit 362B is an arithmetic function unit that performs interpolation generation of luminance data at the position of a virtual pixel located between the supplied luminance data Y 1. This arithmetic function part is
An LPF that provides an LPF (Low Pass Filter) effect by using a configuration of an arithmetic circuit that performs arithmetic processing for interpolation generation of the luminance data Y or a product sum of predetermined coefficients may be used. The LPF can be configured digitally as shown in FIG. The relationship of this interpolation generation will be described later. Luminance data interpolation function unit 362B generates the luminance data Y _h plane is supplied to the high-resolution plane interpolation function unit 364B.

[0036] High-resolution plane interpolation function unit 364B inputs the luminance data Y _h and data corrected three primary colors R plane, G, and pixel data of B, R plane data using these data, G plane data and B An arithmetic function unit that generates plane data. As shown in FIG. 5, the high-resolution plane interpolation function unit 364B includes an R interpolation development unit 3640B, a G interpolation development unit 3642B, and a B interpolation development unit 3644B corresponding to each color. R interpolation expansion unit 3640B, the G interpolation expansion unit 3642B and B interpolation expansion unit 3644B, a high frequency luminance data Y _h are supplied. Interpolation and development section for each color 3640B, 364
2B, 3644B generates plane data of each color from the high-frequency luminance data Y _h and the color of the pixel data of each interpolation. The high-resolution plane interpolation function unit 364B outputs the generated three-primary-color RGB plane data to the selector 36c. The arithmetic functions of these RGB synchronizing units 36A and 36B are performed under the control of the system control unit 18. Although not shown, the RGB synchronizing units 36A and 36B store image data obtained by performing the signal processing and have nondestructive readable memories. The arithmetic functions of the RGB synchronizing units 36A and 36B will be described later in detail.

Returning to FIG. 1, the selector 36c has a switch function for switching the output destination in units of three data supplied to each of the RGB synchronizing units of the image generating unit 36b, ie, R, G, B data. Switching of the switch is performed under the control of the system control unit 18. For example, when the system control unit 18 performs the automatic selection display, the switching is performed every elapse of a predetermined time. When the user selects an output source from the operation unit 14, the system control unit 18 performs control according to an operation signal that is an instruction from the operation unit 14. Selector 36c
Outputs the data of the three primary colors R, G, and B according to the control to the matrix processing unit 36d.

As shown in FIG. 6, the matrix processing section 36d includes a color difference matrix section 360d, an anti-aliasing filter section 362d, and an aperture adjusting section 364d. The color difference matrix unit 360d converts the supplied R plane data, G plane data, and B plane data into a format used for image display, that is, luminance data Y, color difference data (RY), (BY). The data in these output formats is obtained by multiplying each color by a determined mixing ratio and calculating. As a coefficient for determining the mixing ratio, a conventional value is used. The converted three data are output to the anti-aliasing filter unit 362d. The anti-aliasing filter unit 362d has a low-pass filter corresponding to each of these three data. Each data band is different, but low pass filter (LPF) 3620d, 3622d and 36
For 24d, a cutoff frequency that includes this band and does not cause aliasing is selected. Anti-aliasing filter section
362d outputs data without distortion. Among them, the luminance data Y is sent to the aperture adjustment unit 364d. The aperture adjustment unit 364d is a contour compensator having a function of raising the high frequency range of the luminance data Y. The contour compensator is configured using, for example, a transversal filter or the like so as to compensate for a decrease in response in a high frequency band. By raising the level of this data in the high-frequency range, effects such as contour enhancement of the image are obtained. The LPFs 3622d and 3624d may be provided with a delay element in consideration of the processing time of the aperture adjustment unit 364d. Thus, the matrix processing unit 36
d outputs the luminance data Y and the color difference data (RY), (BY) to the optimum image synthesizing unit 36e and the monitor 42.

The optimal image synthesizing unit 36e is
Has the function of synthesizing one image in accordance with the control of. This synthesis is performed by exchanging data at the designated address by effectively using the address of the memory. For this reason, it includes a memory for storing an image to be used as a reference for replacement and a memory for storing an image subjected to RGB synchronization processing different from the image of this reference. It is convenient to match the memory areas. When cutting out an image, for example, a function for detecting an edge is provided.
Confirm with 14. The data is supplied to a memory for storing an image based on the data at the address included in the determined area, and the data is replaced. When the system control unit 18 is in the image display mode, the optimum image combining unit 36e is set to the non-enabled state.
The 36e can be shut down to save power. When the optimum image combining unit 36e is enabled by the system control unit 18, the above-described processing is performed. A more detailed procedure will be described later.

The selected image is supplied from the matrix processing section 36d to the monitor 42 via the system bus 16. In this case, on the monitor 42, the user can determine on the monitor 42 what effect the image generated by the respective RGB synchronizing processes of the image generating unit 36b has on the image of the object scene, and which part of the image is affected. You can know if it is big.

With such a configuration, the signal processing unit 36 finally performs optimal image synthesis to output optimal image data, that is, luminance data Y and color difference data, to the compression / expansion unit 38.

Returning to FIG. 1, the compression / decompression unit 38 is, for example, a JPEG (Joint Photographic E) using orthogonal transform.
(xperts Group) standard, and a circuit for expanding the compressed image to original data again. The compression / decompression unit 38 supplies the compressed data to the recording / reproduction unit 40 via the system bus 16 during recording under the control of the system control unit 18. Further, the compression / decompression unit 38 may pass data from the signal processing unit 36 under the control of the system control unit 18 and supply the data to the monitor 42 via the system bus 16 as described above. When the compression / decompression unit 38 performs the decompression process, the data read from the recording / reproduction unit 40 is taken into the compression / decompression unit 38 via the system bus 16 and processed.
Here, the processed data is also supplied to the monitor 42 for display.

The recording / reproducing section 40 includes a recording processing section for recording on a recording medium, and a reproducing processing section for reading image data recorded from the recording medium (both not shown). The recording medium includes, for example, a semiconductor memory such as a so-called smart media, a magnetic disk, and an optical disk. When a magnetic disk or an optical disk is used, there is a head for writing the image data together with a modulator for modulating the image data. The monitor 42 has a function of considering the size of the screen and displaying the luminance data and the color difference data or the data of the three primary colors RGB supplied via the system bus 16 under the control of the system control unit 18 while adjusting the timing. Have.

The digital camera 10 shown in FIG. 1 is configured as described above to record an optimum image. By the way, before explaining the operation of the digital camera 10, the image pickup unit 30 and the color filter, which are one of the camera features, are further described.
The CF will be further described. FIG. 7 illustrates an imaging surface of the imaging unit 30. As shown in FIG. 7, the imaging unit 30 includes a light receiving unit 30a in which light receiving elements PD adjacent to a light receiving element PD for photoelectrically converting incident light are vertically and horizontally shifted and two-dimensionally arranged. An electrode EL arranged so as to bypass the opening AP formed on the front surface of the portion 30a, and for extracting a signal from the light receiving element PD, and a signal supplied via this electrode EL is provided in a direction perpendicular to the light receiving portion 30a. Vertical transfer registers VR1 to VR4 for sequentially transferring are provided.

The vertical transfer registers VR1 to VR4 transfer signals in accordance with the supplied vertical transfer drive signals V1 to V4. That is, the vertical transfer register has a four-electrode structure per light receiving portion. In addition, a horizontally adjacent region of one light receiving portion region has a two-electrode structure and has a pixel shift as described above. The opening AP formed in the imaging unit 30 of the present embodiment is formed in a hexagonal honeycomb shape. The opening shape is generally a square lattice, but this shape may satisfy the conditions for improving the sensitivity and keeping the width of the vertical transfer register the same so as not to lower the transfer efficiency. As can be seen from the above, the shape may be a polygon, and as another example, a square lattice may be used.
As the opening shape rotated by 45 °, there is, for example, a rhombus or the like, and may be an octagon or the like.

As shown in FIG. 7, each of the openings AP
When the interval between the light receiving elements PD arranged directly below the color filters CF covering the pixels is a pixel pitch PP in each direction, the arrangement of the apertures AP is vertical in each column or horizontal in each row. The two-dimensional arrangement is shifted in the direction by the pixel pitch PP. When using a polygon that is more than a rectangle,
The openings AP may be arranged in a dense arrangement with no gap between the openings AP in accordance with the shape of the openings. In such a case, the pixel pitch PP for arrangement may be shifted by a half pitch. In the case of a hexagon as shown in FIG. 7, a dense arrangement can be formed by displacing the pixel pitch PP by a half in the horizontal and vertical directions. Obtaining such a dense arrangement depends on the shape of the opening AP.

Here, the arrangement relationship between the case where the image pickup section 30 is arranged in a square lattice shape, which is generally used, and the above-described pixel shift, that is, the so-called honeycomb arrangement, will be compared. As shown in FIG. 8A, the arrangement of the honeycombs is such that the pixel pitch PP is
This is equivalent to the arrangement shown in FIG. 8B in which the square lattice arrangement of N (μm) is rotated by 45 °. Further, a color filter CF equivalent to the honeycomb arrangement is schematically represented as shown in FIG. 9, and the three primary colors R, G, and B primary color filters correspond to the shifted arrangement of the light receiving elements. It becomes a complete checkered pattern. This pattern is called G square lattice RB
Call it a complete checkered pattern. A dashed square indicates a virtual pixel without a light receiving element. In this pattern, R and B may be switched. The color filter CF is not limited to the three primary color RGB primary color filters, but may be a complementary color filter. However, in this case, a configuration for obtaining a primary color from a complementary color is added.

Here, the honeycomb arrangement is a distance | PP between adjacent pixels in the horizontal / vertical direction in a square lattice arrangement.
| = N * (2) ^-1/2 based on N (μm) and distance between adjacent pixels |
PP | (see Fig. 8 (b)). Therefore, in the honeycomb arrangement, the pixels are arranged more densely than in the square lattice arrangement, and in principle, the resolution in the horizontal and vertical directions is (2)
It can be improved by ^half . Further, in the case of expanding from a honeycomb-shaped arrangement to a square lattice-shaped arrangement suitable for the output form, the data of the virtual pixel is subjected to the interpolation processing in the above-described image generation unit 36b based on the adjacent pixel data.
It can be seen that if the image is developed in a square lattice while performing this interpolation processing, the resolution can be higher than when the light receiving elements PD are simply arranged in a square lattice.

An operation procedure for creating an optimum image by the digital camera 10 to which the imaging unit 30 having such features is applied will be described. When taking a single image according to the procedure of the main routine shown in FIG. 10, the digital camera 10 first takes an image at a timing considering the exposure conditions obtained by AE and AF by operating the shutter button of the operation unit 14 (Step S10). Into the digital camera 10, incident light impinges on the light receiving element PD of the imaging unit 30 via the two-dimensionally arranged color filters CF and the openings AP having different spectral sensitivity characteristics as shown in FIG. In the light receiving element PD, a signal obtained by photoelectrically converting incident light is read out from a plurality of horizontal transfer registers HR (not shown) via the electrode EL and the vertical transfer register VR, and output to the preprocessing unit 32.

Next, preprocessing in signal processing is performed on the read signal (step S12). As the pre-processing, the pre-processing unit 32 performs, for example, CDS (correlated double sampling) processing to remove noise components included in the signal from the imaging unit 30.

Next, the signal from which noise has been removed is converted to an A / D signal.
The conversion unit 34 converts the digital signal (step S14).
). By this conversion, the signal charge supplied from each light receiving element PD is converted into pixel data. Signal processing after this conversion is performed by digital processing. Although not shown in FIG. 1, a non-destructive buffer memory is preferably used particularly when a CCD image sensor is used. In this case, the buffer memory supplies pixel data of each color to the signal processing unit 36 according to control signals such as a write / read enable signal and an address signal supplied from the system control unit 18.

It is determined whether signal processing is performed on the digitized pixel data (step S16). This determination is made in advance by the user by operating the operation unit 14 of the digital camera 10.
May be set, or may be set via the operation unit 14 immediately before use. This setting is equivalent to selecting whether to record pixel data subjected to signal processing or to record raw pixel data after A / D conversion. If it is determined that the signal processing is to be performed (YES), the process proceeds to step S18. If it is determined that the pixel data is to be recorded immediately (NO), the process proceeds to step S28.

In step S18, for example, white balance and gamma correction are performed. This processing is performed by the data correction unit 36a of the signal processing unit 36. The pixel data corrected in this way is supplied to each of the RGB synchronizing units 36A to 36N of the image generating unit 36b.

Upon receiving the supply of the pixel data, the system control unit 18 determines whether or not to generate an optimal image (step S20). The determination in this case is set in advance similarly to the above-described determination, or is performed by setting each time. The direction in which the process proceeds is determined based on this setting. That is, when the optimum image is to be generated (YES), the process proceeds to the subroutine SUB1. If the signal processing is performed but the generation of the optimum image is not performed (NO), the process proceeds to step S22.

To generate an optimum image, the image generation unit 36
An image is generated in each of the RGB synchronization units 36A to 36N of b (subroutine SUB1). This generation procedure will be described later in detail, and the luminance data and color difference data of the obtained optimal image are sent to the compression / decompression unit 38 (step S26). On the other hand, when the generation of the optimum image is not performed, the system control unit 18 controls so as to generate a predetermined image according to the specification set in advance. Thereby, only one RGB synchronizing unit corresponding to the predetermined image generation is operated to generate one image data (step S22).

The generated image data is supplied to the selector 36c.
Is supplied to the matrix processing unit 36d via the. The matrix processing unit 36d basically converts the image data (plane pixel data) into a matrix of luminance data and color difference data and outputs the matrix data to the monitor 42, and also outputs it to the compression / expansion unit 38 (subroutine SUB2).

After the matrix processing or the generation of the optimum image, the compression / expansion unit is applied to the generated single image.
At 38, compression processing is performed (step S26). The compression processing here is performed by JPEG using orthogonal transform as described above in this embodiment.
Apply standard compression.

After the compression, the compressed image data is supplied to the recording / reproducing unit 40 via the system bus 16. Also, in a case where image data that has not been subjected to signal processing, that is, so-called raw data is recorded, although not shown in FIG.
The data after the A / D conversion is supplied via the system bus 16. The recording / reproducing unit 40 records / supplies the supplied image data.
Recording is performed according to the playback device. In the case of a semiconductor memory, digital data itself is recorded, and data is modulated and recorded. For example, in the case of magnetic modulation, optical modulation, or the like, analog recording is performed. The processing at the time of recording is performed as described above, and the processing for recording one image is completed.

By operating as described above, an area having a higher image quality than the area of the reference image is cut out of each generated image and synthesized with the reference image. It is possible to improve the image quality as compared with the case where the image is generated by using.

Next, a subroutine SUB1 for generating an optimum image will be described with reference to FIG. If the generation of the optimum image is selected, the data correction unit 36
The corrected data from a is supplied to each of the RGB synchronizing units 36A to 36N. Based on the supply of the pixel data of the three primary colors R, G, and B, an image composed of respective plane pixel data is generated (subroutine SUB3). The procedure for generating each image will be described later. In this manner, a plurality of types of images are generated for data captured from one field.

Although a plurality of images are generated, the images are displayed one by one on the monitor 42 in order to grasp the characteristics of each image. To perform this display, first, the created image data is selected (sub-step SS100).

Based on the selected image data (plane pixel data), the matrix processing unit 36d performs matrix processing to generate luminance data and color difference data for the pixels (subroutine SUB2). The generated luminance data and color difference data are supplied to the monitor 42 via the system bus 16 under the control of the system control unit 18. The monitor 42 displays an image using the luminance data and the color difference data (substep SS102).

According to this procedure, the selected image is displayed. It is determined whether all of the plurality of generated images have been selected (sub-step SS104). When it is determined that all images have been selected (YES), the flow proceeds to sub-step SS106. If all the images have not yet been selected (NO), the flow returns to sub-step SS100. By this series of processing, images having characteristics obtained by performing different image generation processing can be shown to the user. The user selects an image to be a reference of the optimum image based on the displayed image (sub-step SS106).

In response to the selection, an image to be cut out is selected (substep SS108). The image to be cut out is an image that has been subjected to image generation processing different from the reference image. At this time, although not shown in the flowchart, the monitor 42 displays both or alternately the reference image and the selected image. In the case of alternate display, the monitor 42 performs processing such as automatic switching at predetermined time intervals or switching in response to an instruction from the operation unit 14 of the user.
Has been done against.

While each image is displayed as described above, the cut-out image is displayed and compared with the reference image, for example,
An instruction to cut out an area with high image quality is issued by the operation unit 14 (substep SS110). The indication of the area to be cut out is
In addition to designating the image area, a finer area is designated by using this designation and the result of extracting the edge boundary. The region specified by this region specification is replaced with the corresponding region of the image based on the cut-out region (substep SS11
2). That is, partial replacement of an image. When the size of the memory for storing the reference image and the size of the memory for storing the selected other image are set to be the same in this replacement,
Since the correspondence is clear, the replacement procedure becomes easy.
Since the replacement area is not limited to one area, it is further determined whether or not there is a replacement area (sub-step SS114). The determination is made by the user via the operation unit 14. If it is OK to end the replacement process (YES
), And proceed to sub-step SS116. If the replacement process is to be continued (NO), the process returns to sub-step SS110 to repeat the above-described series of processes.

In sub-step SS116, it is determined whether there is a generated image other than the image just cut out and / or if the image is present, it is unnecessary for the user. If there is no image, or if there is an image, and no further processing is required (YES), the flow proceeds to subroutine SUB2. If there is an image and there is a request for processing (NO), the process returns to sub-step SS108.

The image synthesized in this way is converted into data that can be displayed on the monitor 42 by matrix processing (subroutine SUB2). This image data is displayed on the monitor 42. This image has a higher image quality than an image obtained by one image processing because the image is composed of a region whose partial portion has a higher image quality. After displaying this image for a predetermined time,
Move to return. Thus, the process of generating the optimum image ends.

Next, a subroutine SUB2 for forming a matrix of image data will be described with reference to FIG. First, matrix processing is performed in the color difference matrix unit 360d of the matrix processing unit 36d using the RGB data output via the selector 36c (substep SS200).
). By this matrix processing, luminance data Y and color difference data (RY), (BY) are generated. After this processing, the obtained luminance data Y and color difference data (RY), (BY) are subjected to LPF processing over a wide band (substep SS202). By this processing, generation of aliasing distortion is suppressed. This processing is performed by the anti-aliasing filter unit 362d. After this process, the color difference data (RY) = C _r , (BY) = C _b
Is obtained. Further, aperture adjustment is further performed on the luminance data Y (sub-step SS204). The aperture adjustment is performed by the aperture adjustment unit 364d in FIG. By performing such processing, luminance data Y having good high-frequency characteristics can be obtained. After this processing, the process proceeds to the subroutine SUB2
To end.

Next, a subroutine SU for generating a plurality of images.
Each process of B3 will be described. In generating a plurality of images, the subroutine SUB3 sequentially performs the subroutines SUB4 and SUB5 of the processing procedure of the RGB synchronizing units 36A and 36B in this embodiment (see FIG. 13). The subroutine SUB4 interpolates the color G that has undergone the rotation processing, and at the same time, corrects the high-frequency correction signal that contributes to contour enhancement (that is, a signal that emphasizes resolution).
Using the color G, taking into account the correlation of the color G, the color R and B are subjected to interpolation processing, and the obtained color RGB is subjected to average interpolation processing. This procedure is shown in FIG. 14 and will be described below.

First, the three supplied RGB data are in the high frequency range.
G The position is changed to a position rotated 45 ° by the data generation unit 360A. The R interpolation function unit 3620A and the B interpolation function unit 3622A also change the arrangement of the supplied pixel data to positions rotated by 45 ° (substep SS400). The relationship of the rotational arrangement of the light receiving element is that the region surrounded by the alternate long and short dash line 100, 102 in FIG. As a result of this rotation, for example, the area indicated by the alternate long and short dash line 102 is changed to the pixel arrangement shown in FIG.

Next, the high-frequency G data generation unit 360A
The G data _Gh is generated and the light receiving element positions corresponding to the colors R and B other than the color G are interpolated. These two processes may be performed by the high-frequency G data generation unit 360A and a G interpolation function unit (not shown). High-frequency G data G _h is not shown is provided with a contour enhancement data generation function unit, and a frequency duplication prevention function unit. The contour emphasis data generation function unit generates pixel data serving as a basis of a high-frequency component signal (Y _H ) which is developed in an oblique Bayer arrangement from the supplied three primary color RGB data and emphasizes resolution. The generation of the pixel data, for example, of Y _h · Y _low method, and generates the pixel data supplied using the method of calculating the Y _h. When generating the pixel data, the contour emphasis data generation function unit supplies the generated high frequency component signal (Y _H ) to the frequency duplication prevention function unit. The frequency duplication prevention function unit is supplied with, for example, a signal in which horizontal and vertical resolution is emphasized, and a common frequency band is present in one of the horizontal and vertical signals and the other signal. For example, a common frequency band is band-limited to one signal, and this signal and the other signal are combined and added. As a result, the output signal becomes a high-frequency component signal.
(Y _H ) becomes the high-frequency G data G _h for which the contour is emphasized.

For example, pixel data of color B rotated by 45 °
When considering the area surrounded by B ₀₂ , B ₂₀ , B ₄₁ , and B ₂₄ , pixel data Y _h22 is obtained by using the five pixel data B ₂₀ , R ₂₂ , and B ₂₄ obliquely to the left as the horizontal direction. )

[0073]

Y _h22 = 0.5 * R ₂₂ + 0.25 * (B ₂₀ + B ₂₄ ) (1) Using five pixel data B ₀₂ , R ₂₂ , and B ₄₁ obliquely to the right as the vertical direction (5 lines before rotation), the pixel data Y _h22 is expressed by the equation (2).

[0074]

Y _h22 = 0.5 * R ₂₂ + 0.25 * (B ₀₂ + B ₄₁ ) (2) When performing calculation in both directions with emphasis on the resolution, when a common frequency band exists in the calculated pixel data, the frequency duplication prevention function unit strictly passes only the frequency band in one direction and sets the horizontal / vertical Filter processing is performed so that abnormalities do not occur even if signals in the directions are combined. The filtering process is a high-pass filtering process. When used simply as high-frequency G data, this frequency duplication prevention function unit may be skipped.

As a second creation method, four colors G ₁₁ ,
Using G ₁₃ , G ₃₁ , and G ₃₃ , the pixel data G _h22 in the pixel data R ₂₂ is given by the equation (3)

[0076]

G _h22 = (G ₁₁ + G ₁₃ + G ₃₁ + G ₃₃ ) / 4 (3) As a third creation method, a color G
Of horizontal pixel data in this arrangement from the only data | Correlation between G ₁₁ -G ₃₃ | and vertical pixel data |
G ₁₃ -G ₃₁ | and the difference | G ₁₁ -G ₃₃ |-| G ₁₃ -G ₃₁ |, and when the calculation result is smaller than a predetermined value, it is determined that there is a correlation in the horizontal direction, and the equation (2) )

[0077]

G _h22 = (G ₁₁ + G ₃₃ ) / 2 (4) To check for vertical correlation,
_{G 13 -G 31 | - | G} 11 -G 33 | When is calculated and the calculation result is smaller than a predetermined value, as there is a correlation in the horizontal direction,
Equation (5)

[0078]

G _h22 = (G ₁₃ + G ₃₁ ) / 2 (5) As described above, the high band G data may be generated based on the correlation detection. One of these creation methods is selected to generate high-frequency G data. Generating while shifting the three rows and three columns of the oblique Bayer arrangement, it is possible to generate the pixel data G _h in the position of the light receiving element. The above-described first to third creation methods also perform interpolation on the color G (substep SS402).

Next, an interpolation process is performed on the pixel data of the color R (substep SS404). This interpolation process is shown in FIG.
This is performed by the R interpolation function unit 3620A. As shown in FIG. 2, the R interpolation function unit 3620a, R data and the high-frequency G data G _h is supplied. In this interpolation process, generally, the fact that a change in the level of the signal of the primary color G greatly affects a change in the luminance of the video signal is used. This method is based on the fact that this influence is reflected in the development of the other primary colors R and B. The principle will be briefly described with reference to FIG. This is because the unknown R shown in FIG.
Here is an example of finding ₁₁ . At this time, G ₂₀ , G ₁₁ , G ₂₂ , R ₂₀ ,
Using known signal level of R _22. The interpolation process, assume an approximation of a weighted average value delta _R of G _20, the weighted average delta _G and R ₂₀ of G _22, R ₂₂ are equal (Δ _G = Δ _R). Using this relationship, the weighting factor when the weighted average is also because it is known values, the unknown pixel data R ₁₁ is easily calculated. Using the high-frequency G data G _h to be supplied, colors G other than color R,
The pixel data of the color R in B is calculated.

Similarly, an interpolation process for the color B is performed using this principle (sub-step SS406). In this case also assumed approximation that is equal weighted average delta _B of the pixel data of the weighted average delta _G and the color B of the pixel data of the color _G (Δ G = Δ
_B ). Using this, unknown pixel data is calculated. Color G other than the color B is used to supply high frequency G data G _h, and calculates the pixel data of the color in the R B.

By this calculation, all three primary colors RGB at the position of the light receiving element are calculated. The calculated RGB data is supplied to the average plane interpolation function unit 364A. The average plane interpolation function unit 364A converts the diagonal Bayer arrangement to the original G square
A rotation process for returning to an RB complete checkerboard pattern is performed, and data of three primary colors RGB is interpolated at the position of a virtual pixel between the light receiving elements (substep SS408). This interpolation is calculated by averaging surrounding pixel data of the same color. Pixels used for calculation,
That is, the number of data uses two or more pixels. In this way, the pixel data of all the light receiving elements and the virtual pixels are calculated. Using the calculated pixel data, plane data of three primary colors RGB is generated. After this generation, the process proceeds to the return, and this subroutine SUB4 ends. By operating in such a procedure, as a result, the RGB
36A makes it possible to output a captured image of a subject as a high-resolution signal.

Next, the subroutine SUB5 for performing the checkerboard processing will be described. Subroutine SUB5 as shown in FIG. 17, the subroutine generates high band luminance data Y _h to the position of the light receiving element by the checkered treated with SUB6, high-frequency luminance at the position of the virtual pixel in the subroutine SUB7 the data based on data
The Y _h interpolation generates a further process for calculating the RGB plane data in the subroutine SUB9. Regarding this processing procedure, the operation procedure of each subroutine will be further described.

First, it is determined whether or not the mode is the adaptive processing mode (sub-step SS600). In the case of the adaptive processing mode (YES), the process proceeds to substep SS602 in FIG. If the mode is not the adaptive processing mode (N
O), and proceed to sub-step SS604 of FIG.

Next, in sub-step SS602, a selection is made as to whether to perform diagonal correlation processing. If the oblique correlation processing is to be performed (YES), the flow proceeds to sub-step SS606. If the oblique correlation process is not performed (NO), the process proceeds to sub-step SS608 via connector B. In sub-step SS608, it is determined whether or not to perform the correlation process.

In the above-described sub-step SS604, the calculation of the luminance data is performed irrespective of the adaptive processing mode. In performing this processing, the CCD image sensors of the imaging unit 30 are originally two-dimensionally arranged as shown in FIG.
Here, the suffix is a position when the position of each light receiving element as a pixel is represented by a matrix expression. The pixels of the actual light receiving element are indicated by solid lines, and the pixels corresponding to the virtual light receiving elements are indicated by broken lines. Basically, the luminance data Y is the pixel data G
It is known that the value can be calculated by (0.5 * R + 0.5B) using pixel data R and B. Also in this case, the pixel data G is treated as it is as luminance data (pixel data G = luminance data). Also, the luminance data based on the pixel data R and B is
If the color corresponding to the actual position of the light receiving element is R / B instead of G, for example, the luminance data Y ₂₂ corresponding to the position of the pixel data R _{22 in} FIG. 21 is the pixel data R ₂₂ and the pixels located therearound. Four pixels of data B, that is, pixel data B ₀₂ and B
₂₀ , B ₂₄ , B ₄₂

[0086]

[6] obtained from _{Y 22 = R 22/2 +} (B 02 + B 20 + B 24 + B 42) / 8 ··· (6). The luminance data Y ₂₄ corresponding to the position of the pixel data B ₂₄ is four pixels of the pixel data R which is located around the pixel data B _24, i.e. pixel data R _{0 4,}
Using R ₂₂ , R ₂₆ , R ₄₄

[0087]

Equation 7] obtained from _{Y 24 = B 24/2 +} (R 04 + R 22 + R 26 + R 44) / 8 ··· (7). The correction amount to be corrected using surrounding pixels is a number obtained by multiplying the sum of these four pixels by the number of pixels, ie, 4
The value obtained by dividing by × 2 = 8 is added to the half value of the pixel to be created. This is the same as multiplying the calculated average by a factor of 0.5. This calculation is performed for each pixel to obtain the luminance data Y. As a result, a checkerboard pattern of luminance data shown in FIG. 22 is obtained. Note that such calculation is also performed when there is no correlation in the oblique direction, the vertical direction, and the horizontal direction, as described later.

Next, in sub-step SS606, it is determined whether or not to perform the diagonal processing step by step.
When it is determined that the diagonal processing is performed based on a plurality of steps (YES), the flow proceeds to sub-step SS610. If the diagonal processing is not to be performed after a plurality of stages (NO), the flow proceeds to sub-step SS612.

Here, in sub-step SS610, comparison data is calculated. Pixel data having the same color as the color of the pixel data to be created is used as the pixel data used for the calculation. The comparison data ARS is, for example, the target pixel data is R
_In the case of ₂₂ , using the surrounding pixel data R ₀₀ , R ₄₄ , R ₀₄ , R ₄₀ ,

[0090]

ARS _L = | R ₀₀ -R ₄₄ | (8) ARS _R = | R ₀₄ -R ₄₀ | (9) The suffixes “L” and “R” indicate that the diagonal (S) is inclined leftward and rightward, respectively. When the array of FIG. 21 is rotated 45 ° counterclockwise, it corresponds to the horizontal direction and the vertical direction. Calculated comparison data ARS _L
Further correlation value using the values of _{ARS R (ARS L -ARS R)} , (AR
S _R -ARS _L ) is calculated.

Next, in sub-step SS614, when the calculated correlation value (ARS _L -ARS _R ) is larger than the newly provided predetermined criterion value J0 (YES), the value of ARS _R is smaller. This means that the values of the used pixel data are similar. Thereby, it is determined that there is a correlation in the diagonally right direction, and the flow advances to sub-step SS616. Also,
When the above condition is not satisfied (correlation value (ARS _L -ARS
_R ) <J0) (NO), it is determined that there is no right oblique correlation with respect to the pixel to be created, and the flow advances to substep SS618. In sub-step SS616, this case, the luminance data Y ₂₂

[0092]

Y ₂₂ = R _22/2 + (R ₀₀ + R ₄₄ ) / 4 (10)

In sub-step SS618, when the calculated correlation value (ARS _R -ARS _L ) is larger than the predetermined judgment reference value J0 (YES), it is determined that there is a correlation in the diagonally left direction and the sub-step SS618 is executed. Proceed to SS620. When the above condition is not satisfied (correlation value (ARS _R -ARS _L ) <J0) (NO), it is determined that there is no left oblique correlation with respect to the creation target pixel, and the flow advances to substep SS622. In sub-step SS620, this case, the luminance data Y ₂₂

[0094]

Equation 10] _{Y 22 = R 22/2 +} (R 04 + R 40) / 4 is obtained from (11). Sub-step SS616 and sub-step SS62
After calculating the luminance data of 0, the process proceeds to sub-step SS624 in FIG.

Next, in sub-step SS622, new comparison data is calculated. Here, the pixel data used for the calculation is different from the color of the pixel data to be created. For example, the comparison data is calculated using the color G. Comparison data AG
S, for example, if the object pixel data of R _22,
The comparison data AGS includes the surrounding pixel data G ₁₁ , G ₃₃ , G ₁₃ , G
_{Using 31} ,

[0096]

AGS _L = | G ₁₁ -G ₃₃ | (12) AGS _R = | G ₁₃ -G ₃₁ | (13) In this sub-step, a correlation value (AGS _L -AGS _R) is calculated using the calculated comparison data AGS _L and AGS _R.
R ), (AGS _R -AGS _L ) are also calculated. After this processing, the flow advances to sub-step SS626 in FIG. 19 via connector D.

Next, in sub-step SS626, when the calculated correlation value (AGS _L -AGS _R ) is larger than, for example, a newly provided predetermined judgment reference value J0a (YES)
), The value of the pixel data used is estimated to be similar because the value of AGS _R is small. As a result, it is determined that there is a correlation in the diagonally right direction, and the flow advances to sub-step SS628. If the above condition is not satisfied (correlation value (AGS _L -AGS _R ) <J0a) (NO), it is determined that there is no right diagonal correlation with respect to this creation target pixel, and the flow proceeds to substep SS630. In sub-step SS628, in this case,
Y ₂₂

[0098]

Y ₂₂ = R _22/2 + (G ₁₁ + G ₃₃ ) / 4 (14) The luminance data Y ₂₂ may be calculated from equation (10).

In sub-step SS630, when the calculated correlation value (AGS _R -AGS _L ) is larger than the predetermined judgment reference value J0a (YES), it is determined that there is a correlation in the diagonally left direction, and the sub-step SS630 is executed. Proceed to SS632. When the above condition is not satisfied (correlation value (AGS _R -AGS _L ) <J0a) (NO)
Then, it is determined that there is no left diagonal correlation with respect to the pixel to be created, and the process proceeds to substep SS608. In sub-step SS632,
In this case, the luminance data Y ₂₂

[0100]

Equation 13] _{Y 22 = R 22/2 +} (G 13 + G 31) / 4 is obtained from (15). The luminance data Y ₂₂ is may be a formula (11). After calculating the luminance data in the sub-step SS628 and the sub-step SS632, the sub-step SS in FIG.
Continue to 624.

By the way, when the simple diagonal processing is selected in the sub-step SS606, the process proceeds to the sub-step SS612 as described above. In this sub-step SS612,
The comparison data is calculated. The comparison data is used, for example, to determine in which direction the pixel data around the pixel data to be subjected to the adaptive processing is correlated. For example, if the object pixel data of R _22, comparison data AG, using the surrounding pixel data _{G 11, G 13, G 31} , G 33,

[0102]

AG = | G ₁₁ + G _33- (G ₁₃ + G ₃₁ ) | (16) Although the case where the pixel data is color R is described, the case of color B is also calculated from the surrounding pixel data G. By this calculation, the larger value having a slope on one of the left and right sides is obtained as the comparison data AG. After this calculation, the flow advances to sub-step SS634.

In sub-step SS634, it is determined whether or not there is a correlation (that is, a diagonal correlation) between pixel data positioned diagonally across the target pixel data. In this determination, J1 is newly set as a determination reference value. When the comparison data AG is larger than the criterion value J1 (YES), the process proceeds to sub-step SS636. When the comparison data AG is smaller than the criterion value J1 (NO), the comparison data AG shown in FIG.
To sub-step SS608.

In sub-step SS636, the four pixel data G used for calculating the comparison data AG are averaged to obtain the luminance data
Calculate Y. The determination of whether pixels arranged in a plurality of stages and simple oblique direction with respect to the pixels of the creation target are correlated, several patterns, for example, is determined for the pixel data R = R ₂₂ Will be. by the way,
In general, a false color may be generated at the boundary of the oblique line. However, when the luminance data Y in the pixel data R located near the boundary is calculated by the above-described calculation, it is possible to satisfactorily suppress the occurrence of a false color at the color boundary when the entire image is viewed. Substeps even for the pixel omitted specific explanation data B = B ₂₄ SS614
620SS620, SS626 SS632, and SS634, the comparison data is calculated in the same manner as SS636, and adaptive luminance data Y based on the presence or absence of the oblique correlation can be created.

After the processing in sub-step SS636, connector C
Through sub-step SS624 in FIG. Thus, a series of diagonal processing ends. If diagonal processing is not performed in sub-step SS634 (NO), the flow advances to sub-step SS608 via connector B. The processing performed after this sub-step is data processing in accordance with the presence / absence of a horizontal / vertical correlation with the creation target pixel. Sub-step SS608
Then, it is determined whether or not to perform the correlation processing. This is a determination in the case where correlation processing is performed in other directions, that is, horizontal and vertical directions, for a wide range of the light receiving element (or color filter). When making this determination (YES), the sub-step SS
Continue to 638. If this determination is not made (NO), the flow advances to sub-step SS604 in FIG.

In sub-step SS638, comparison data is calculated. Again By way of example for the pixel data R = R _22. In this processing, the vertical direction comparison data ABR _V and the horizontal direction comparison data ABR _H for the pixel data R = R ₂₂ are obtained by using the pixel data B of the other color arranged around, that is, the pixel data B, by using the equation (17). ), Equation (18)

[0107]

ABR _V = | B ₀₂ -B ₄₂ | (17) ABR _H = | B ₂₀ -B ₂₄ | (18) When the correlation values (ABR _H -ABR _V ) and (ABR _V -ABR _H ) are further calculated using the calculated comparison data ABR _V and ABR _H , a newly provided predetermined judgment reference value J2
A procedure for comparing the magnitude of the correlation value in each direction with respect to and determining the presence or absence of a correlation will be described.

In sub-step SS640, it is determined whether or not there is a correlation (that is, a vertical correlation) between pixel data positioned vertically across the target pixel data. In this judgment, J2a is set as a judgment reference value. When the difference between the comparison data ABR _H and the comparison data ABR _V is larger than the determination reference value J2a (YES), it is determined that there is a vertical correlation, and the flow advances to substep SS642. Also, the difference of the comparison data (ABR _H -ABR _V )
Is smaller than the determination reference value J2a (NO), it is considered that there is no vertical correlation, and the flow advances to sub-step SS644.

[0109] In sub-step SS642, it means that the value between the pixel data is short that there is a correlation, to calculate the luminance data Y with the pixel data B _02, B _42. In this case, the luminance data Y ₂₂ is

[0110]

Equation 16] _{Y 22 = R 22/2 +} (B 02 + B 42) / 4 obtained by ... (19). Thereafter, it is considered that the calculation of the luminance data Y in the pixel data has been completed, and the process proceeds to the sub-step SS624 in FIG.

Next, in sub-step SS644, it is determined whether or not there is a correlation (that is, a horizontal correlation) between pixel data positioned horizontally across the target pixel data. For this determination, J2b described above is used as a determination reference value. When the difference between the comparison data ABR _V and the comparison data ABR _H is larger than the determination reference value J2b (YES), it is determined that there is a horizontal correlation, and the flow advances to substep SS646. In addition, the difference (ABR _V -AB
If R _H ) is smaller than the determination reference value J2b (NO), it is determined that there is no horizontal correlation, and the flow advances to substep SS648.

In sub-step SS646, luminance data Y is calculated using pixel data B ₂₀ and B ₂₄ assuming that there is a correlation. In this case, the luminance data Y ₂₂ is

[0113]

Y ₂₂ = R _22/2 + (B ₂₀ + B ₂₄ ) / 4 (20) Thereafter, it is considered that the calculation of the luminance data Y in the pixel data has been completed, and the process proceeds to the sub-step SS624 via the connector C.

Next, in sub-step SS648, for example,
Selects whether or not to perform a correlation judgment of surrounding pixels of color B with pixels of color R, which is a pixel to be created. Since the pixel of the color R is disposed at the center position of the surrounding pixels of the color B, the distance between the pixels in the sub-steps SS640 and SS644 is short. That is, for example, in the vertical direction, the pixel R ₂₂ -pixel
B ₀₂ , pixel R ₂₂ -pixel B ₄₂ are half the distance of pixel B ₀₂ -pixel B ₄₂ . This relationship can be applied to a pixel located in the horizontal direction with respect to the creation target pixel. Therefore, it can be seen that the determination of the presence or absence of the correlation for a narrower range of the light receiving element (or the color filter) is performed in the subsequent processing as compared with the above-described horizontal / vertical direction correlation determination. If this correlation is to be determined (YES), the flow proceeds to sub-step SS650. If this correlation determination is not made (NO), the flow advances to substep SS604 via connector A. In this case, it is determined that one of the criterion values J2a and J2b different from the criterion value J2 is not satisfied. Note that a processing procedure that does not perform the subsequent processing may be used.

In sub-step SS650, comparison data is calculated again. In this case, the comparison data is calculated in the vertical direction and the horizontal direction by calculating each correlation between the target pixel data and the surrounding pixel data and adding the obtained correlation values. Calculation of the luminance data Y for the case as well as the pixel data R ₂₂ described above, the vertical direction of the comparison data AC
Using the pixel data of the other color arranged around R _V and the comparison data ACR _H in the horizontal direction, that is, pixel data G, Equations (21) and (22)

[0116]

Equation 18] _{ACR V = | G 11 -G 31} | + | G 13 -G 33 | ··· (21) ACR H = | G 11 -G 13 | + | G 31 -G 33 | ··· ( Calculated by 22). After this processing, the flow advances to substep SS612. By using this comparison data, the correlation value can be obtained by making the distance of the pixel data even closer to the pixel data to be created.
The presence or absence of a correlation can be checked in a range narrower than the range of the correlation determination in the procedure from 0 to SS646. After this calculation,
Proceed to sub-step SS652.

In sub-step SS652, it is determined whether or not there is a correlation (ie, vertical correlation) between pixel data positioned vertically across the target pixel data. In this determination, J3 is set as a determination reference value (here, the determination reference value J3 may be divided into J3a and J3b for horizontal and vertical use). When the difference between the comparison data ACR _H and the comparison data ACR _V is larger than the determination reference value J3 (YES), it is determined that there is a vertical correlation, and the flow advances to substep SS654. In addition, the difference
When (ACR _H -ACR _V ) is smaller than the judgment reference value J3 (NO),
It is determined that there is no vertical correlation, and the flow advances to substep SS656.

In sub-step SS654, the same processing as the above-described processing in sub-step SS642 is performed. Therefore,
Equation (19) is used for the calculation. Also, sub-step SS65
In step 6, it is determined whether or not there is a correlation (that is, a horizontal correlation) between pixel data positioned horizontally across the target pixel data. The determination reference value J3 is also used for this determination.

In the sub-step SS656, when the difference (ACR _V -ACR _H ) of the comparison data is equal to or larger than the judgment reference value J3 (YES),
It is determined that there is a horizontal correlation, and the flow advances to substep SS658.
In this case, the luminance data Y in sub-step SS658 uses pixel data as described above in sub-step SS646,
It is calculated based on equation (20). After this, sub-step SS
Continue to 624. When the difference (ACR _V -ACR _H ) of the comparison data is smaller than the judgment reference value J3 in sub-step SS656 (NO),
It is determined that there is no horizontal correlation, and the flow advances to sub-step SS604 in FIG. In sub-step SS604, the expression
According to (6), the target pixel data and the surrounding pixel data of the other color (in this case, pixel data B) are added and averaged to obtain 0.5.
To calculate the luminance data Y. After this calculation, the flow advances to sub-step SS624.

In sub-step SS624, it is determined whether the creation of the checkered luminance data has been completed for one frame. This determination can be easily made, for example, by counting the number of calculated luminance data Y and determining whether or not the counted value matches the number of light receiving elements. When the count value is smaller than the number of light receiving elements (N
O), it is determined that the processing has not been completed yet. As a result, the process of calculating the luminance data Y is returned to the sub-step SS600 of FIG. 18 via the connector E, and a series of processes up to this point are repeated. When the count value matches the number of light receiving elements (YES), the process shifts to return. After this return, the process is shifted to the subroutine SUB7.
By calculating the luminance data Y in this way,
Data is created at a position where a checkered light-receiving element actually exists as shown in FIG.

By the way, in an image including a color boundary in an oblique direction, the direction of the color boundary can be estimated from the direction of correlation. However, the pixel R for which the luminance data is calculated (for example,
R ₂₂ ) cannot be specified in the horizontal or vertical direction when calculating from the surrounding pixel data B ₀₂ , B ₂₀ , B ₂₄ , and B ₄₂ .

Next, the operation of the subroutine SUB7 will be described (see FIG. 23). The operation of the subroutine SUB7 is performed based on the configuration of the digital filter of the luminance data interpolation function unit 362B as described above. In sub-step SS70, low-pass filter processing, which is a characteristic of the digital filter, is performed, and pixel data at the position of the virtual light receiving element is generated to perform data interpolation. This relationship is briefly shown in FIG. Also in FIG. 4, the pixel d corresponding to the existing light receiving element
_(-3) , d _(-1) , d ₍₁₎ , d ₍₃₎ are indicated by solid lines, pixels corresponding to the virtual light receiving elements are indicated by broken lines, and between four existing light receiving elements (existing pixels) The relationship is arranged. The pixels dn _(-4) , dn _(-2) , dn ₍₀₎ , dn ₍₂₎ ,
The d _{n (4),} considering the correspondence between real light-
Treat as the same relationship as the state without any data.
That is, zero is set in advance for these pixels. For example, when the pixel dn ₍₀₎ is horizontally interpolated as shown in FIG. 4A, the tap coefficients of the digital filter are k ₀ , k ₁ , k ₂ , k ₃ , k ₄ ,. kn, the luminance data Yh ₍₀₎ including the high-frequency component is _expressed by equation (23).

[0123]

Equation 19] _{Y h (0) = k 0} * d n (0) + k 1 * (d (1) + d (-1)) + k 2 * (d n (-2) + d n (2 _{) + k 3 * (d (} -3) + d (3)) + k 4 * (d n (-4) + d n (4)) + ··· k n * (d n (-n) + d _{n (n)} )... (23) However, in this case, FIG.
As can be seen from (a), the coefficient is doubled because zero data is entered alternately. This relationship is based on the other pixels dn _(-4) , dn _(-2) , dn ₍₂₎ , dn _{(4) to be} interpolated in FIG.
Also apply to By performing these interpolation processes, the luminance data Yh _(-4) , Yh _(-2) ,
Yh ₍₂₎ and Yh ₍₄₎ are obtained (see FIG. 4 (b)).

In the vertical direction, low-pass filtering is performed by a digital filter provided in the luminance data interpolation function unit 362B. In this case, the pixel data corresponding to the virtual light receiving element has already been interpolated by the horizontal interpolation process, so that the pixel data is dense. Therefore, the coefficients of the low-pass filter can be made the same as usual. Expressing luminance data including high-frequency components obtained in this way in matrix representation as shown in FIG. 21, the luminance data Y _h including high-frequency component as shown in FIG. 24 is generated. Luminance data Y _h including high-frequency component is referred to as the high-frequency luminance data in the following description.

Next, the operation of the subroutine SUB8 will be described. The subroutine SUB8 is performed by the high resolution plane interpolation function unit 364B as shown in FIG. The high-resolution plane interpolation function unit 364B, a high frequency luminance data Y _h Toko of high-frequency luminance data Y color interpolated corresponding to _h of the pixel data generated by the subroutine SUB7 is supplied to each of the arithmetic processor . High frequency luminance data Y _h, as is clear from FIG. 5 R
It is supplied in common to the interpolation / expansion unit 3640B, the G interpolation / expansion unit 3642B, and the B interpolation / expansion unit 3644B. Using the supplied pixel data, the pixel data of the pixel of each virtual light receiving element is interpolated for each color according to the flowchart shown in FIG. In this case, the interpolation processing of the pixel data G is first performed in sub-step SS800. At this time, as shown in FIG. 26, since a single-plate pixel shift type G square RB perfect checkerboard pattern is used, pixels having existing pixel data G are represented by a solid square grid. Further, a pixel having no pixel data G, that is, a pixel having a color different from the color G while having a corresponding pixel of the virtual light receiving element and existing pixel data is represented by a square grid indicated by a broken line. This pixel data
Pixels without G are called virtual pixels. 4 for interpolation
Existing pixel data is used one by one.

FIG. 26 shows this relationship specifically. As the pattern of FIG. 26, the virtual pixel _{G 12, G 14, G 16} , G 21 ~ G
_In the case of interpolating one line of ₂₆ , G ₃₂ , G ₃₄ , G ₃₆ , the interpolation processing is performed by four adjacent pixel data G ₁₁ , G ₁₃ , G ₃₁ , G ₃₃ and pixel data G ₁₃ , G ₁₅ , G ₃₃ , G ₃₅ etc. are used. The calculation is also performed using the high-frequency luminance data of FIG. 24 corresponding to the pixel data G used for the interpolation. For example, interpolated pixel data G ₂₁ virtual pixel is interpolated using the high-frequency luminance data of existing data and high-luminance data and the position to be interpolated corresponding to two pixels of the same column, the formula ( twenty four)

[0127]

G ₂₁ = (G ₁₁ + G ₃₁ ) / 2− (Y _h11 + Y _h31 ) / 2 + Y _h21 (24) Using the calculation formula of Expression (24), the virtual pixel G
₂₃ can be interpolated. Also, the interpolation of the virtual pixel G _12, using the high-frequency luminance data of existing data and high-luminance data and the position to be interpolated corresponding to two pixels of the same row, the formula (25)

[0128]

G ₁₂ = (G ₁₁ + G ₁₃ ) / 2− (Y _h11 + Y _h13 ) / 2 + Y _h12 (25) Using the equation of Equation (25), the virtual pixel G
₃₂ can be interpolated. The pixel data located at the center of each of the four pixel data G ₁₁ , G ₁₃ , G ₃₁ , G ₃₃
G _22, using the pixel data and the high luminance data of these four positions, the formula (26)

[0129]

Obtained from Equation 22] _{G 22 = (G 11 + G} 13 + G 31 + G 33) / 4- (Y h11 + Y h13 + Y h31 + Y h33) / 4 + Y h22 ··· (26) . Using the equation of Equation (26), the virtual pixel G
₂₃ can be interpolated. Pixel data G ₁₃ , G ₁₅ ,
G _33, when the G ₃₅ the four regarded as a set of data interpolation, already because the pixel data G ₂₃ is calculated, may be calculated pixel data G _14, G _34, G ₂₅ remaining. By repeating this process, a plane image of the pixel data G is created. However, since the outermost edge of the plane image does not have such a relationship, it is preferable to set a boundary value when performing strict interpolation. In consideration of the effective screen, the data in the peripheral portion is out of the range of the effective screen, and thus need not be calculated.

Next, the calculation of the pixel data R is performed in the sub-step SS.
802. Also in this case, the pixels corresponding to the existing data and the pixel data calculated by the calculation are represented by a solid square grid, and the virtual pixels are represented by a broken square grid. The existing pixel data in the pixel data R is represented by R as shown in FIG.
_{00, R 04, R 22,} R 26, R 40, R 44 only. In this case, in the sub-step SS802, pixel data obliquely adjacent to the virtual pixel to be interpolated and the pixel data corresponding to this position shown in FIG.
Is used. For example, pixel data R _11
Are the pixel data R ₀₀ , R ₂₂ and the high-frequency luminance data Y _h00 ,
Using Y _h22 and Y _h11 ,

[0131]

R ₁₁ = (R ₀₀ + R ₂₂ ) / 2− (Y _h00 + Y _h22 ) / 2 + Y _h11 (27) Similarly, virtual pixels R ₁₃ , R ₃₁ , R _33
Are the pixel data R ₀₄ , R
₂₂ , the pixel data R ₄₀ , R ₂₂ and the pixel data R ₄₄ , R ₂₂ are used to calculate. If an existing pixel data R ₂₆ is also calculated in consideration, it is possible to also create the virtual pixel R _15, R ₃₅ by the adjacent diagonal interpolation process. The result is shown in FIG.

Next, in sub-step SS804, a pixel surrounded by the pixel calculated in the immediately preceding sub-step SS802 is set as a pixel to be interpolated, and the four pixel data calculated at the time of interpolation and the high frequency of that position are calculated. An interpolation process is performed using the luminance data. For example, as can be seen from FIG. 28 centering on the pixel data R ₂₄ , the surrounding pixel data R ₁₃ , R
_15, using the data of the position of R _33, R _35, formula (28)

[0133]

Equation 24] _{R 24 = (R 13 + R} 15 + R 33 + R 35) / 4- (Y h13 + Y h15 + Y h33 + Y h35) / 4 + Y h24 calculated by ... (28) You. When an arrangement relationship equivalent to the pixel data used in Expression (28) is obtained from peripheral pixels, by performing this interpolation, the pixel data R ₀₂ , R
₂₀ and R ₄₂ are obtained. In other words, from the viewpoint of the pixel to be interpolated, all the pixel data used for the interpolation are positioned obliquely.

Next, in sub-step SS806, the pixel data obtained so far is used, and among these pixels, interpolation is performed from the pixel data located at the top, bottom, left and right with respect to the interpolation target pixel. For example, using a high-frequency luminance data of four pixel data and its vertical and horizontal positions around the pixel data R _12, (29)

[0135]

Equation 25] _{R 12 = (R 02 + R} 11 + R 13 + R 22) / 4- (Y h02 + Y h11 + Y h13 + Y h22) / 4 + Y h12 calculated by ... (29) You. For example, pixel data R ₁₄ , R ₃₂ , and R ₃₄ having a similar positional relationship can be calculated by substituting data corresponding to the positional relationship of the pixel data used in Expression (28). Furthermore, if pixels continue on the right side of FIG. 29, pixel data R ₁₆ and R ₃₆ can also be calculated.

Since uninterpolated virtual pixels remain in the peripheral portion as shown in FIG. 30, the virtual pixels may be interpolated from, for example, three pixels surrounding the virtual pixels. In the case of this interpolation as well, if the above-described interpolation method is used, the pixel data R ₀₁ of the virtual pixel becomes

[0137]

Equation 26] _{R 01 = (R 00 + R} 02 + R 11) / calculated by _{3- (Y h00 + Y h02 +} Y h11) / 3 + Y h01 ··· (30). In this way, the pixel data R ₀₃ , R ₀₅ , R ₁₀ , R ₃₀ , R ₄₁ , R ₄₃ , and R ₄₅ are interpolated.
Finally, the entire plane screen for the pixel data R is interpolated.

Next, interpolation processing for the pixel data B is performed in sub-steps SS808, SS810, and SS812. The sub-steps SS808, SS810, and SS812 are the adjacent oblique interpolation processing on the pixel data B, the central interpolation processing using four pieces of interpolation data, and the central interpolation processing using four pixels of up, down, left and right. These interpolation processes are the above-described interpolation processes of the pixel data R (that is, the sub-steps SS802, SS804, SS806
). This corresponds to the pixel data R in FIG.
It can be seen from the relationship of the pixel arrangement of the pixel data B of one. That is, the pixel arrangement of the pixel data B in FIG. 31 is such that the pixel data R in FIG. 27 is shifted by two columns in the horizontal (that is, row) direction as a whole from the matrix display represented by the subscript of each color. I have. From this, when interpolating virtual pixels by applying the equations (27) to (30), the number of the column in each column of the subscript of each pixel data on the right side of two or more columns in the matrix display To
It is good to calculate in the relation which added +2. For example, the pixel data B ₁₃ and the pixel data B ₃₃ are obtained by converting the color R in the equation (27) into the color B
And replace the positional relationship between the pixel data R ₀₀ and R ₃₁ with the pixel data
B ₀₂ , B ₃₃

[0139]

## EQU27 ## B _{11 + 2} = (B _{00 + 2} + B _{22 + 2} ) / 2- (Y _{h00 + 2} + Y _{h22 + 2} ) / 2 + Y _{h11 + 2} B ₁₃ = (B ₀₂ + B ₂₄ ) / 2- (Y _h02 + Y _h24 ) / 4 + Y _h13・ ・ ・ (31) B _{31 + 2} = (B _{22 + 2} + B _{40 + 2} ) / 2- (Y _{h22 + 2} + Y _{h40 + 2} ) / 4 + Y _{h31 + 2} B ₃₃ = (B ₂₄ + B ₄₂ ) / 2- (Y _h24 + Y _h42 ) / 4 + Y _h33 ... (32) You. Further, when the number of columns in the matrix display of the pixel data to perform interpolation processing of each pixel data in less than 2 left, using the relation for calculating the pixel data R ₁₃ by using the pixel data R _04, R _22, It is better to subtract -2 from the subscript number.
For example, pixel data B ₁₁ is

[0140]

( _Equation 28) B _13-2 = (B _04-2 + B _22-2 ) / 2- (Y _h04-2 + Y _h22-2 ) / 2 + Y _h13-2 B ₁₁ = (B ₀₂ + B ₂₀ ) / 2- (Y _h02 + Y _h20 ) / 4 + Y _h11 (33) A similar relationship holds in other equations (28) to (30). Be aware of this relationship
When the interpolation processing is performed in 810 and SS812, plane interpolation development for the pixel data B can be performed. After this processing,
Proceed to sub-step SS814.

In sub-step SS814, it is determined whether or not the plane interpolation development has been completed for each color. If a series of processing has not been completed yet (NO), the process returns to sub-step SS800 and repeats the processing. Note that this confirmation process may be performed for each color. When a series of processing is completed (YES), the flow shifts to return. After this transition,
The processing of the subroutine SUB8 ends. By operating in such a procedure, as a result, the RGB
Makes the captured image of the subject high resolution,
This makes it possible to output a signal in which a false color generated in an image is suppressed.

As a more specific example, a digital camera
The case where one image 50 is photographed by 10 will be described. The image 50 by the digital camera 10 roughly includes two people, a landscape, and a signboard on which characters are written by imaging. The digital camera 10 performs A / D conversion of an image pickup signal into digital image data. As described above, when the image data is not subjected to signal processing, the digital image data (ie, raw data) is recorded as it is on the recording medium of the recording / reproducing unit 40. When performing signal processing, the image data is stored in the RGB data of the image generation unit 36b.
This is supplied to the synchronization units 36A and 36B, respectively. RGB synchronization unit 36
A and 36B generate image data by performing interpolation generation of pixel data in accordance with the above-described subroutine. The generated image data is displayed on the monitor 42 via the selector 36c, the matrix processing unit 36d, and the system bus 16. RGB synchronization unit 36
From A, for example, a high-resolution image suitable for the monochrome character mode can be obtained. The RGB synchronizing unit 36B generates an image in which the resolution is slightly lower than that of the image from the RGB synchronizing unit 36A, but in which a false color that may occur at a color boundary is suppressed. The user, based on the displayed image display on the monitor 42,
Select a reference image.

In this case, the image from the RGB synchronizing unit 36B is used as a reference image with emphasis on suppressing the generation of false colors.
This image is supplied to the optimum image synthesizing unit 36e. Also,
Since the RGB synchronizing unit 36A provides an image having a resolution suitable for displaying characters, for example, a setting is made to cut out an area 52 including a vertical signboard. This setting is specified by the operation unit 14 of the digital camera 10. The operation unit 14 selects a plurality of boundaries of the region to be extracted, and at the same time, detects a nearby edge selected by the system control unit 18, thereby performing a process of appropriately selecting a range to be cut out, and determines a designated range.

In response to this determination, the system control unit 18
The image of the region 52 obtained by performing the control of cutting out the region 52 of the B synchronization unit 36A is supplied to the optimum image combining unit 36e. The optimal image combining unit 36e replaces the image supplied on the memory with the corresponding area of the currently stored image and combines the images. As a result, an image in which the characteristics of both the landscape region including the person and the character display region are utilized is generated. If there is a region to be cut out, this process is repeated. Although only two RGB synchronizing units are provided in this embodiment, the image generating unit 36b further differs from the two image generating characteristics described above in the image generating unit 36b.
An RGB synchronization unit may be provided. At this time, the different RG
B It is possible to determine whether to perform replacement using the image generated by the synchronizing unit and generate a more optimal image.

When raw data is recorded on a recording medium, for example, an image of the raw data is read from the recording medium to a personal computer via a reproducing device. The personal computer has an RGB synchronization unit 36 of the image generation unit 36b.
Software corresponding to the processing in A and 36B is stored. In a personal computer, by performing the above-described signal processing on image data in a program procedure using software, an optimum image can be displayed on a monitor as a processing result, and this image can be recorded.

With the above configuration, a plurality of signal processes are performed in parallel on digital image data (ie, raw data), and an area which provides the optimum image characteristics obtained by each process is cut out and synthesized. Accordingly, it is possible to provide a user with an opportunity for image processing. By performing this process within the range that does not affect the tone characteristics at all, without deteriorating the consistency with the infrastructure such as the existing output device, that is, without affecting other devices, It is also possible to obtain an optimal image having many high-quality image areas in the screen area.

[0147]

As described above, according to the solid-state imaging device of the present invention, the three primary colors R, G, and B are all processed in parallel by the plurality of interpolation generating means based on the image data corrected by the data correcting means in the signal processing means. Interpolation is generated in parallel, and one of the plurality of interpolation generating means is selected via a selecting means and supplied to the display image generating means. The display image generating means sends the luminance data and the color difference data generated based on the image data to the display means. This provides the user with an opportunity to perform image processing such as how to perform synthesis by the synthesizing unit from an image displayed on the display unit without affecting other devices and circuits. The image processing is to replace one image area selected under the control of the control means in accordance with an instruction from the operation means, synthesize the image area, and generate one image, thereby affecting other devices. In addition, the image quality can be further improved as compared with the conventional captured image.

Further, according to the image processing method of the present invention, all three primary colors R, G, and B at the positions of the voids of the light receiving element and / or the light receiving element adjacent to the light receiving element are set in parallel on the basis of the corrected image data. Since a plurality of plane interpolation generations are performed, a plane image is immediately supplied at the time of image selection, and luminance data and color difference data obtained based on the supplied image data are displayed on a display unit, and based on the displayed image, Provide an opportunity for users to evaluate images. In this evaluation, an image is replaced according to a determination that one image serving as a reference is replaced with an image area generated by another interpolation, and an optimal image is generated, thereby achieving unprecedented shooting. It is possible to provide an opportunity for the user to perform signal processing on the captured image data at the time or after shooting, and to generate an optimal high-quality image that satisfies the user's requirements more than conventional captured images.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a schematic configuration of a digital camera to which a solid-state imaging device according to the present invention is applied.

FIG. 2 is a block diagram illustrating a schematic configuration of an RGB synchronizing unit used in the image generating unit of FIG. 1;

FIG. 3 is a block diagram illustrating a schematic configuration of an RGB synchronization unit that performs processing different from that of the RGB synchronization unit in FIG. 2;

FIG. 4 is a schematic diagram illustrating the principle of a low-pass filter used in the luminance data interpolation function unit in FIG.

FIG. 5 is a block diagram illustrating a schematic configuration of a high-resolution plane interpolation function unit of FIG. 3;

FIG. 6 is a block diagram illustrating a schematic configuration of a matrix processing unit in FIG. 1;

FIG. 7 is a schematic diagram illustrating the relationship between the arrangement of light receiving elements and the color filters corresponding to the respective light receiving elements in the imaging unit in FIG. 1;

8 is a schematic diagram illustrating a difference between a case where the arrangement of the light receiving elements is arranged in a square lattice and a case where the arrangement is a honeycomb arrangement in the imaging unit in FIG. 1;

FIG. 9 shows a color filter applied to the image pickup unit of FIG. 1, G square RB
It is a schematic diagram explaining a complete checkerboard pattern.

FIG. 10 is a main flowchart illustrating an operation procedure of the digital camera in FIG. 1;

FIG. 11 is a flowchart illustrating an operation procedure of a subroutine SUB1 applied to FIG. 10;

FIG. 12 is a flowchart illustrating an operation procedure of a subroutine SUB2 applied to FIG. 10;

FIG. 13 is a flowchart illustrating an operation procedure of a subroutine SUB3 applied in the subroutine SUB1 of FIG.

14 is a flowchart illustrating an operation procedure by a rotation process of a subroutine SUB4 applied in a subroutine SUB3 of FIG.

FIG. 15 is a schematic diagram illustrating the relationship of data arrangement change in subroutine SUB4 of FIG.

16 is a schematic diagram illustrating the principle of R interpolation and B interpolation in a subroutine SUB4 of FIG.

FIG. 17 is a schematic diagram illustrating an operation procedure for creating checkered data in a subroutine SUB5 applied in the subroutine SUB3 of FIG.

FIG. 18 is a flowchart illustrating a procedure for creating checkered data in a subroutine SUB7 of FIG. 17;

FIG. 19 is a flowchart illustrating a continued operation procedure from FIG. 18 in a subroutine SUB7 of FIG. 17;

20 and FIG. 19 in the subroutine SUB7 of FIG.
6 is a flowchart for explaining a continuous operation procedure from FIG.

FIG. 21 is a schematic diagram illustrating a positional relationship between pixel data and virtual pixels obtained from a light receiving element via a color filter.

FIG. 22 is a schematic diagram showing positions of high-frequency luminance data obtained by the processing of FIGS. 18 to 20;

FIG. 23 is a flowchart illustrating a procedure of an interpolation process of subroutine SUB7 in subroutine SUB5 of FIG. 17;

FIG. 24 is a schematic diagram illustrating positions of high-frequency luminance data obtained by the interpolation processing of FIG. 23;

FIG. 25 is a flowchart illustrating a procedure of an RGB plane interpolation process of a subroutine SUB8 in a subroutine SUB5 of FIG. 17;

26 is a schematic diagram illustrating a positional relationship between a pixel to be interpolated with respect to pixel data G and an existing pixel in the RGB plane interpolation processing of FIG. 25;

FIG. 27 is a schematic diagram showing a positional relationship between a pixel to be interpolated with respect to pixel data R and existing pixels in the RGB plane interpolation processing of FIG. 25;

28 is a schematic diagram illustrating a positional relationship when the result of the adjacent oblique interpolation processing is added to the positional relationship of FIG. 27;

29 is a schematic diagram showing a positional relationship when the result of the interpolation process using four pixel data obtained by the adjacent oblique interpolation process is added to the positional relationship of FIG. 28;

30 is a schematic diagram showing a positional relationship when a result of performing an interpolation process using pixel data positioned vertically, horizontally, and horizontally with respect to a pixel to be interpolated is added to the positional relationship of FIG. 29;

31 is a schematic diagram showing a positional relationship between a pixel to be interpolated with respect to pixel data B and an existing pixel in the RGB plane interpolation processing of FIG. 25;

FIG. 32 is a diagram illustrating an example of an image captured by the digital camera in FIG. 1;

[Explanation of symbols]

 10 Digital camera 14 Operation unit 18 System control unit 30 Imaging unit 34 A / D conversion unit 36 Signal processing unit 36a Data correction unit 36b Image generation unit 36c Selector 36d Matrix processing unit 36e Optimal image synthesis unit 36A to 36N RGB synchronization unit

Claims (10)

[Claims] 1. A light-separated light which is color-separated by color separation means for color-separating light incident from a field via an optical system in accordance with the position of the light-receiving element for photoelectrically converting the light-receiving element is two-dimensionally arranged. The color imaging is performed by the imaging means, the obtained color imaging signal is converted into digital image data by the digital conversion means, and the image data is recorded via the recording means for recording the image data and / or the signal processing means for performing signal processing on the image data. In a solid-state imaging device that outputs the image data to a recording unit, the device includes: a data correction unit that corrects a color adjustment and a gradation of an image to the digital image data; at least based on image data from the data correction unit. A plurality of interpolation generating means for performing parallel interpolation generation of all three primary colors R, G, and B at the position of the light receiving element; and selecting one of the plurality of interpolation generation means. A step of generating luminance data and color difference data based on the image data supplied via the selecting means, performing processing to prevent the occurrence of aliasing distortion on these data, and performing contour enhancement processing on the processed luminance data. Display image generating means to be applied, and one image data from the display image generating means,
A signal processing unit including a combining unit for combining an image by replacing the image data output by each of the plurality of interpolation generating units with an optimal image region; and an image obtained by the display image generating unit. Display means for displaying data; and the optimum image in the synthesizing means in response to an instruction from an operation means for providing an instruction in accordance with the result of the evaluation while confirming and evaluating the image displayed on the display means A solid-state imaging device comprising: control means for controlling extraction and replacement of an area. 2. The apparatus according to claim 1, wherein the color separation means matches a color filter for separating the incident light into three primary colors R, G, and B with a light receiving element of the imaging means, Wherein the light receiving element and the light receiving element adjacent to the light receiving element are two-dimensionally arranged so as to be shifted from each other by a half pitch in a vertical direction and a horizontal direction. 3. The apparatus according to claim 2, wherein the color separation means arranges the color G among the three primary colors R, G, and B in a square lattice, and the color R and the color R are completely different from the color G. A solid-state imaging device characterized by a color filter pattern arranged in a checkered pattern. 4. The apparatus according to claim 3, wherein said plurality of interpolation generating means sets pixel data of a color G of at least three primary colors R, G, and B as high-range pixel data, and G interpolating means for interpolating the color G at the light receiving element in the colors R and B different from the color G based on the data, and interpolating the color R using the pixel data of the color R which is correlated with the G interpolating means. R interpolation means, B interpolation means for performing color B interpolation using pixel data of color B having a correlation with the G interpolation means, and the B interpolation means, the R interpolation means and the B interpolation means, respectively. A first interpolation generating means including an average interpolation means for interpolating and generating plane pixel data of each color by an average interpolation using pixel data; and the light receiving means based on pixel data of three primary colors R, G, B obtained by the color separation means. Luminance to find luminance data at element position Data generating means, luminance data interpolating means for obtaining luminance data at the gap position of the light receiving element based on the luminance data obtained by the luminance data generating means, interpolation of color R using the luminance data interpolating means and the color R pixel data R interpolating and developing means, the luminance data interpolating means and the color G
A solid-state image sensor comprising: a G interpolation developing means for performing the interpolation of the color data; and a second interpolation generating means including a B interpolation developing means for performing the interpolation of the color B using the luminance data interpolation means and the color B pixel data. Imaging device. 5. The apparatus according to claim 1, wherein said control means also performs correction control of said data correction means, image generation control of said display image generation means, and selection switching control of said selection means. A solid-state imaging device for controlling generation of a signal used in the device. 6. A color image of incident light from an object scene via an optical system, and an image signal obtained by the color image is converted into digital image data, and the recording and / or recording of the image data is performed.
Or an image processing method for performing recording on the image data through signal processing, the method comprising: a data correction step of correcting color adjustment and gradation of an image to the digital image signal; An interpolation generation step including a plurality of plane interpolation generations in which all three primary colors R, G, and B at positions of gaps in the light receiving element and / or the light receiving element adjacent to the light receiving element are generated in parallel based on image data; An image selecting step of selecting one image interpolated and generated among a plurality of types of image data obtained in the generating step, and generating luminance data and color difference data based on the image data obtained in the image selecting step; The signal processing step includes a process of preventing the occurrence of aliasing in the data and a display image generating step of performing an edge enhancement process on the processed luminance data. Further, the method includes: an image display step of displaying image data obtained in the display image generation step on a prepared display unit; and performing image evaluation using the image displayed in the image display step, and forming one image. A first determining step of making a determination to perform replacement with an image area by another interpolation generation, an optimal combining step of performing an optimal image generating process according to the first determining step, Then, a second determining step of determining whether or not to repeat the process using the image generated by the interpolation generation selected for another region, and, in the case of ending the process in the second determining step, whether to use another interpolation generation Is included in the signal processing step. When the processing is repeated in the second and third determination steps, a series of processing of selecting a new image and cutting out and replacing the selected image is performed. Do each Image processing method comprising and. 7. The method according to claim 6, wherein the optimal synthesizing step is performed by a reference image selecting step of selecting an image to be a reference from the generated images, and is generated by a process different from the image of the reference image selecting step. An extraction selecting step of selecting an image, an area specifying step of specifying an area to be extracted from the image selected in the extraction selecting step, and replacing the area specified in the area specifying step with a corresponding area of the reference image. An image processing method comprising: 8. The image processing method according to claim 6, wherein the optimal synthesizing step includes an area matching step of matching the indicated cut-out area of the evaluated image and the replacement area of the image. Method. 9. The method according to claim 6, wherein, if the image signal is recorded as it is, a reproducing step of reproducing the recorded digital image data is performed before the signal processing step. Characteristic image processing method. 10. The method according to claim 6, wherein, in the color imaging, among the three primary colors R, G, B, the color G is arranged in a square lattice, and the color R, B is completely checkered with respect to the color G. An image processing method characterized by using an image pickup signal photographed by a color filter pattern arranged in the image processing apparatus.