JPH11215513A - Image-pickup device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain an image whose false color and false contour are reduced by calculating a signal at a prescribed pixel position based on an edge component at a prescribed pixel position of a chrominance signal, decoding a signal corresponding to pixel number of an image-pickup element and decoding the signal corresponding to the pixel number of the image-pickup element, based on the decoded chrominance signal and the edge component. SOLUTION: A separate means 4 separates R, G, B signals from a signal received by a frame memory 3, and the G signal is fed to an edge discrimination means 5 and a G component decoding means 6. The R, B signals are fed to the G component decoding means 6 and RB decoding means 7. The edge discrimination means 5 discriminates an edge component at a prescribed pixel position at a prescribed pixel position in the G signal and gives the discrimination result to the G component decoding means 6. The G component decoding means 6 decodes a pixel signal of the G component to obtain the G signal with a total pixel number in an image-pickup element 1, based on the discrimination result from the edge discrimination means 5. Then the result is fed to RB component decoding means 7, 9 and an edge discrimination means 8, where the pixel is decoded and generated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to high image quality and miniaturization of an image pickup apparatus such as an electronic still camera.

[0002]

2. Description of the Related Art Conventionally, in an image pickup apparatus such as an electronic still camera, a single image pickup element (hereinafter, referred to as a single image pickup element) has been used for miniaturization.
(Referred to as a single-chip image sensor). FIG. 19 is a diagram showing an example of a typical color filter array of a conventional image sensor. In the figure, R is an image sensor having a color filter having a spectral characteristic for transmitting light of R, and similarly, B and G are image sensors having respective color filters. As shown in FIG. 19, G,
R is arranged every two pixels, and B and G are 2 in the (n + 1) th line.
It is arranged for each pixel. Therefore, the R and B signals are obtained every four pixels in the upper and lower portions (hatched portions in the drawing), and the G signal is obtained every two pixels. Then, arithmetic processing is performed from the obtained pixel signals to generate R, G, and B signals of the number of imaging elements.

FIG. 20 is a block diagram showing an example of the configuration of a conventional image pickup apparatus that generates R, G, and B signals from signals from the image pickup element in the image pickup element using the primary color filters shown in FIG. FIG. In FIG. 20, 101
Is an image sensor, 102 is a frame memory, 103 is a separating means for separating the signals in the frame memory 102 into R, G, and B signals, and 104 is an interpolating means. , Pixels R, G,
B.

Next, the operation will be described. The respective pixel signals R, G, and B are read from the image sensor 101, and the frame memory 1
02 receives each pixel signal. The frame memory 10
Each signal is separated by the separating means 103 from the signal taken in by 2 and sent to the interpolating means 104. The interpolation means 104 interpolates and generates a signal of a pixel not obtained in each of the R, G, and B signals from a signal of an adjacent pixel, and calculates and outputs RGB signals of all the pixels of the image sensor.

Here, an example of an interpolation method in the interpolation means 104 will be described with reference to FIG. Separation means 1
The R, G, and B signals separated by H.03 are shown in FIG.
(B) As shown in (c), the pixels indicated by G, R, and B in the figure are the respective signals obtained from the image sensor 102, while the pixel signals from which no blank pixels are obtained It is. As for the G signal (FIG. 21A), in order to interpolate a signal g (hereinafter, referred to as a pixel position (n, m)) at a vertical n-th line and a horizontal m-th pixel position, adjacent pixels in a vertical direction are interpolated. (| G (n−1, m) −G (n + 1, m))
|) And the difference (| G (n, m-1)
−G (n, m + 1) |), and interpolates the pixel signals in the direction in which the difference is small. For example, if the difference between adjacent pixels in the left-right direction is small, g (n, m) = (G (n, m)
-1) + G (n, m + 1)) / 2, and when the difference between adjacent pixels in the vertical direction is small, g (n, m) =
It is calculated as (G (n-1, m) + G (n + 1, m)) / 2.

For the R and B signals (FIGS. 21 (b) and (c)), first, horizontal pixel interpolation is performed, and then vertical interpolation is performed. For example, for R in FIG. 21B, interpolation is performed on the vertical n-1 line and the n + 1 line, and the signal at the pixel position (n-1, m) (n + 1, m) is converted into r (n-1, m). ) = (R (n-1, m-1) + R (n-
1, m + 1)) / 2 r (n + 1, m) = (R (n + 1, m-1) + R (n +
1, m + 1)) / 2, and then the pixels of the n-th line are moved up and down (n-1).
And n + 1 lines), the signal of each horizontal pixel position m-1, m, m + 1 is obtained. B can be obtained in a similar manner.

By the above-described interpolation method, it is possible to calculate R, G, and B signals of the number of pixels of all the image pickup devices at the output of the interpolation means 104.

FIG. 22 shows an example of a conventional image pickup apparatus using a pixel mixing type image pickup device disclosed in Japanese Patent Application Laid-Open No. 6-178307, in which two upper and lower pixels are mixed and read. Similarly, FIG. 10 is a block diagram showing a case in which a signal from an image sensor is generated by interpolation from three horizontal scanning lines. In the figure, 105 is an image sensor, 106 is a frame memory, 107 is a signal selection circuit, 108 is a color interpolation circuit, and 109 is an RGB matrix. As shown in FIG. 22, the image sensor 105 is composed of four pixels A, B, C, and D (hereinafter, the number assigned to each pixel signal indicates a pixel position). Pixels A and B are alternately arranged for each line so that they can be generated.

Next, the operation will be described. Each pixel signal is read out from the image sensor 105 without pixel mixture readout, and each pixel signal is taken into the frame memory 106. From the signals fetched into the frame memory 106, a signal of adjacent three lines is selected by a signal selection circuit 107 and sent to a color interpolation circuit 108. In the color interpolation circuit 108, the color signals A, B, C,
After D is generated by interpolation, it is output as an RGB signal by the RGB matrix circuit 109.

Here, the color interpolation circuit 108 generates each color signal by interpolation. This interpolation method will be described. For example, in the color signal interpolation generation for the n2 line, the signal selection circuit 107 causes the three vertical lines n1, n2, n3
Are selected and sent to the color interpolation circuit 108. The color signals obtained in the n2 line are C and D pixels, and there are no A and B pixels. Therefore, the pixels of A and B are interpolated from the signals of the n1 and n3 lines in the vertical direction. However, since the positions of the pixels of A and B are different in the n1 and n3 lines, the interpolation coefficient in the horizontal direction must be changed. Become. Now, for the third pixel (n2, 3) of the n2 line of the color signal after interpolation, the respective color signals A ', B', C ', and D' are generated by interpolation from five horizontal pixels before interpolation. , C ', D' for interpolates generated weighted horizontally only _{center, C 23 '= (C 21} /2 + C 23 + C 25/2) / 2 D 23' = (D 22 + D ₂₄ ) / 2. On the other hand, A ', B' for _{the, A 23 '= (A 11} /4 + A 13/2 + A 15/4) / 2 +
_{(A 31/4 + A 33} /2 + A 35/4) / 2 B 23 '= (B 12/2 + B 14/2) / 2 + (B 32/2 +
From the formula B _34/2 ) / 2, horizontal pixels can be weighted and interpolated.

Next, in the interpolation generation of the color signal of the n3 line, the pixels of A and B are generated by interpolation from the n3 line, and C, D
Pixels are generated by interpolation from lines n2 and n4. That is, at the pixel position (3, 3), for example, A ₃₃ ′ = (A ₃₂ + A ₃₄ ) / 2 B ₃₃ ′ = (B _31/2 + B ₃₃ + B _35/2 ) / 2 C ₃₃ ′ = ( _{C 21/4 + C 23/} 2 + C 25/4) / 2 +
_{(C 41/4 + C 43} /2 + C 45/4) / 2 D 33 '= (D 22/2 + D 24/2) / 2 + (D 42/2 +
D _44/2 ) / 2.

In consideration of the fact that the pixel arrangements of A and B are interchanged for each line, the pixel positions (4,
3), A ₄₃ ′ = (A _31/4 + A _33/2 + A _35/4 ) / 2 + (A
_51/4 + _A53 / 2 + _A55 / 4) / 2 _B43 '= ( _B32 / 2 + _B34 / 2) / 2 + ( _B52 / 2 +
_{B 54/2) / 2 C} 43 In _{'= (C 41/2 +} C 43 + C 45/2) / 2 D 43' = (D 42 + D 44) / 2 n5 line, the pixel position (5,3) _A53 '= ( _A52 + _A54 ) / 2 _B53 ' = ( _B52 + _B54 ) / 2 _C53 '= ( _C41 / 4 + _C43 / 2 + _C45 / 4) / 2 + (C
_61/4 + _C63 / 2 + _C65 / 4) / 2 _D53 '= ( _D42 / 2 + _D44 / 2) / 2 + ( _D62 / 2 +
D _64/2 ) / 2. Thereafter, the color signals A ′,
B ', C', and D 'will be generated.

[0013] Therefore, since the color signal is interpolated and generated by three vertical lines, it is possible to obtain a signal of the number of pixels of the image sensor with little deterioration in the vertical resolution of the color signal. In the above description, the four color signals are described as A, B, C, and D. However, the four color signals are, for example, Mg (magenta), G (green), Cy (cyan), and Ye (yellow). A color can be considered, and the interpolation coefficient in the color interpolation circuit 108 may be any coefficient that can generate a color signal by interpolation.

[0014]

A conventional image pickup apparatus is configured to generate a color signal by interpolation between adjacent pixels in the horizontal direction and pixel signals in upper and lower lines.
It does not consider changes in spatial frequency in local areas such as edges in images from the obtained color signals,
There is a problem that false colors and false contours occur.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and has a first color filter having a spectral sensitivity characteristic for a first color signal in upper and lower four pixels in two vertical rows and two horizontal columns. And a second color filter having a spectral sensitivity characteristic for the second color signal are arranged in the first vertical line, and the second vertical line is a pixel position where the first color filter in the first vertical line is arranged. A third color filter having a spectral sensitivity characteristic for the third color signal is arranged in the same column as
An image sensor in which a first color filter having a spectral sensitivity characteristic for a first color signal is arranged in the same column as the color filters described above, and the color filters of the upper and lower four pixels are sequentially arranged vertically and horizontally repeatedly. In an imaging device provided with
Based on the edge component of the first color signal at the predetermined pixel position by the first color filter, a signal at the predetermined pixel position in the first color signal is calculated to restore the signal of the number of pixels of the image sensor, and the restoration is performed. By calculating the second and third color signals based on the obtained first color signal and the edge component and restoring the signals corresponding to the number of pixels of the image sensor, an image with reduced false colors and false contours can be obtained. It is an object to obtain an imaging device that can be obtained.

[0016]

An image pickup apparatus according to the present invention comprises a first color filter having a spectral sensitivity characteristic for a first color signal and a second color filter in four upper and lower pixels of two vertical rows and two horizontal columns. A second color filter having a spectral sensitivity characteristic with respect to a signal is arranged in a first row, and a second row is arranged in the same column as a pixel position where the first color filter is arranged in the first row. A third color filter having a spectral sensitivity characteristic for the third color signal is arranged, and has the same spectral sensitivity characteristic for the first color signal as the first color filter in the same column as the second color filter. In an imaging apparatus including an imaging element in which a first color filter is arranged and the upper and lower four-pixel color filters are sequentially arranged in a vertical and horizontal direction, a first color signal by the first color filter is provided. At a given pixel position Based on the sides pixel signal, and determines the edge determination means an edge component in a predetermined pixel position, based on the output of the edge determination means, said first, second, third
, Second and third read out by the color filters
A first calculating means for calculating a signal at the predetermined pixel position in the first color signal from the color signal of the first color signal, and an output of the first calculating means and a second signal from the color filter based on the output of the edge determining means. And the third and the third color signals
And second calculation means for calculating a signal of the color signal of the image sensor, and obtains first, second, and third color signals of the number of pixels in the image sensor.

Further, in the image pickup apparatus according to the present invention, the edge determination means detects a horizontal edge component by calculating an absolute value of a difference between left and right adjacent pixels at a predetermined pixel position of the first color signal. Horizontal edge detecting means for performing
Vertical edge detecting means for calculating an absolute value of a difference between upper and lower pixels at a predetermined pixel position of a color signal to detect a vertical edge component; and outputs from the horizontal edge detecting means and the vertical edge detecting means. And determining means for determining a horizontal or vertical edge component in the predetermined pixel based on

Also, in the imaging apparatus according to the present invention, when the judgment means in the edge judgment means determines that the output from the horizontal edge detection means or the output from the vertical edge detection means is larger than a predetermined value. , It is assumed that an edge component is detected in the peripheral pixels of the predetermined pixel,
If the output from the horizontal edge detecting means is larger than the output of the vertical edge detecting means, there is a vertical correlation.If the output from the horizontal edge detecting means is smaller than the output of the vertical edge detecting means, the correlation is horizontal. It is determined that there is a correlation depending on the direction, and when both the outputs from the horizontal edge detecting means and the vertical edge detecting means are smaller than a predetermined value, it is determined that no edge component is detected.

Further, in the image pickup apparatus according to the present invention, the first calculating means may determine that the second color signal B has a predetermined pixel l row m
At the position of column B (l, m), the first color signal A, the second color signal A
Ahlpf (l, m), Bhlpf for each of the color signals B through a horizontal low-pass filter
(L, m) is calculated, and Ahlpf (l, m) and Bhlpf which are output signals from the horizontal low-pass filter are calculated.
(L, m) and the pixel value B (l, m) at the pixel position
As a result, the pixel value A in the first color signal A in l rows and m columns
(L, m) is calculated by A (l, m) = B (l, m) × ｛Ahl
pf (l, m) / Bhlpf (l, m)}, and the horizontal direction signal calculation for calculating the pixel value of the first color signal A at another pixel position of the third color signal C similarly Means and a value Avl via a low-pass filter in the vertical direction for each of the first color signal A and the second color signal B at the position of the predetermined pixel l row m column B (l, m)
pf (l, m) and Bvlpf (l, m) are calculated, and Avl which is an output signal from the vertical low-pass filter is calculated.
Based on the ratio of pf (l, m) to Bvlpf (l, m) and the pixel value B (l, m) at the above pixel position, the first of l rows and m columns
The pixel value A (l, m) of the color signal A of A is represented by A (l,
m) = B (l, m) × ｛Avlpf (l, m) / Bvl
pf (l, m)}, a vertical signal calculating means for calculating a pixel value of the first color signal A at another pixel position of the third color signal C, and a first color signal. Average value calculating means for calculating a pixel value A (l, m) of the first color signal A in l rows and m columns from an average value of upper, lower, left and right adjacent pixels at the position of the predetermined pixel l rows and m columns in the signal A Based on the output of the edge judging means, selected from the output of the horizontal direction signal calculating means or the output of the vertical direction signal calculating means, or the output from the average value calculating means. Pixel value A of first color signal A
(L, m) to obtain a first color signal of the number of pixels in the image sensor.

Also, in the imaging apparatus according to the present invention, when the first calculating means determines that the output of the edge determining means does not detect an edge component at a position of a predetermined pixel at l rows and m columns, the average is calculated. The output of the value calculation means is selected, and when it is determined that there is a correlation in the vertical direction, the output of the vertical signal calculation means is selected. When it is determined that there is a correlation in the horizontal direction, the output of the horizontal signal calculation means is determined. Select the output,
This is for obtaining a first color signal of the number of pixels in the image sensor.

Further, in the imaging apparatus according to the present invention, the second calculating means may apply a low-pass filter in the horizontal direction to the output A from the first calculating means at the position of predetermined pixel l row and m column. Value A1hlpf (l, m) through the vertical low-pass filter and value A1vlpf (l,
m), and the value B1hlpf (l, m) of the second color signal B through a horizontal low-pass filter and the value C1vlpf (l) of the third color signal C through a vertical low-pass filter. , M) (or the value B1vlpf through the low-pass filter in the vertical direction with respect to the second color signal B)
(L, m) and the value C1hlpf (l, m)) of the third color signal C through a horizontal low-pass filter are calculated, and A1hlpf (l, m) and B1hlpf (l, m) are calculated.
m) (or the ratio with C1hlpf (l, m))
And A1vlpf (l, m) and C1vlpf (l, m)
(Or the ratio to B1vlpf (l, m)) and
From the pixel value A (l, m) at the predetermined pixel l row m column in the output A from the first calculating means, the pixel value at the second color signal B and the third color signal C at l row m column Let B (l, m) and C (l, m) be B (l, m) = A (l, m) x {B1
hlpf (l, m) / A1hlpf (l, m)｝, C
(L, m) = A (l, m) × ｛C1vlpf (l, m)
/ A1vlpf (l, m)}, (or B (l, m)
= A (l, m) × ｛B1vlpf (l, m) / A1vl
pf (l, m)｝ C (l, m) = A (l, m) × ｛C1
hlpf (l, m) / A1hlpf (l, m)｝)
At a predetermined pixel x row y column position different from the column position,
The value A2hlpf (x, y) of the output A from the first calculating means through a horizontal low-pass filter, and the horizontal low-pass filter of the second color signal B at the output from the signal calculating means. B2hlpf (x,
y) is calculated, and A2hlpf (x, y) and B2hlpf
The second color signal B at the position of x row and y column is obtained from the ratio of (x, y) and the pixel value A (x, y) at the pixel x row and y column at the output A from the first calculating means. (X, y) is converted to B (x,
y) = A (x, y) × ｛B2hlpf (x, y) / A2
hlpf (x, y)}, and a horizontal signal calculating means for calculating the C signal in the same manner also for the third color signal C, and a low-pass filter in the vertical direction with respect to the output A from the first calculating means. A2vlpf (x, y) via
The value B2vlpf of the second color signal B output from the signal calculation means through a vertical low-pass filter.
(X, y) is calculated, and A2vlpf (x, y) and B2v
By the ratio of lpf (x, y) and the pixel value A (x, y) at the pixel x row and y column at the output A from the first calculating means, x
The second color signal B (x, y) at the position of the row y column is represented by B
(X, y) = A (x, y) × ｛B2vlpf (x, y)
/ A2vlpf (x, y)}, and a vertical signal calculating means for calculating the C signal for the third color signal C in the same manner.
Average value calculation means for calculating an average value of obliquely adjacent pixels at a position of a predetermined pixel x row and y column in the third color signal, wherein the horizontal direction signal calculation is performed based on an output of the edge determination means. Means, the vertical direction signal calculating means, and the average value calculating means, to obtain the second and third color signals at the predetermined pixel x row and y column to obtain the second and third color signals. Second, a third color signal is obtained.

Further, in the imaging apparatus according to the present invention, when the second calculating means determines that the output of the edge determining means does not detect an edge component at a position of a predetermined pixel x row and y column, the average value is obtained. The output of the calculation means is selected, and when it is determined that there is a correlation in the vertical direction, the output of the vertical signal calculation means is selected. When it is determined that there is a correlation in the horizontal direction, the output of the horizontal signal calculation means is determined. Is selected to obtain the second and third color signals of the number of pixels in the image sensor.

[0023]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be specifically described with reference to the drawings showing the embodiments. Embodiment 1 FIG. FIG. 1 is a diagram showing an example of a color filter array of an image pickup device in an electronic still camera according to Embodiment 1 of the present invention, and shows an image pickup device of a system using primary color filters and independently calling each photoelectric conversion element. ing.
In the figure, G is the vertical direction 2i (i = 0, 1, 2,.
..), Pixel positions in the horizontal direction 2j (j = 0, 1, 2,...) (Hereinafter referred to as pixel positions (2i, 2j)).
A first color filter located at the pixel position (2i + 1, 2j + 1) and having a spectral characteristic for passing the G signal, R is a second color filter located at the pixel position (2i, 2j + 1) and having a spectral characteristic for passing the R signal The color filter B has a pixel position (2i +
1, 2j), and is a third color filter having a spectral characteristic for passing the B signal. As shown in FIG.
The R and B signals are obtained every four pixels in the upper and lower portions (hatched portions in the figure), and the G signal is obtained every two pixels. The four upper and lower pixels are repeatedly arranged in the vertical and horizontal directions.

FIG. 2 is a block diagram showing a configuration of an image pickup apparatus in an electronic still camera according to the first embodiment having an image pickup device having the color filter array shown in FIG. In the figure, reference numeral 1 denotes an image sensor configured with the color filter array shown in FIG. 1, 2 denotes an A / D converter, 3 denotes a frame memory,
Reference numeral 4 denotes separation means for separating the R, G, and B pixel signals and outputs respective signals. Reference numeral 5 denotes first edge determination means for determining an edge at a predetermined pixel in the G signal. Is a G component restoring means for restoring G so as to obtain a G signal of the total number of pixels in the image pickup device 1 based on each of the above signals and the output from the first edge judging means 5. Reference numeral 7 denotes a restored G signal having a signal of the total number of pixels, which is an output of the G component restoring means 6, and pixel positions (2i, 2j) and ( 2i
RB for restoring pixel at (+1, 2j + 1)
The component restoring means 8 is a second edge judging means for judging an edge at a predetermined pixel in the G signal from the G component restoring means 6.
Based on the B signal and the output from the second edge judging means 8, the pixel R at the pixel position (2i + 1,2j) at R and the pixel B at the pixel position (2i, 2j + 1) at B are restored. This is a second RB component restoration unit.

FIG. 3 shows the first edge determining means 5.
FIG. 3 is a block diagram showing a configuration example of a G component restoring unit 6. In the figure, reference numeral 11 denotes a horizontal edge detecting means for detecting a difference between left and right pixels in a predetermined pixel of the G signal, that is, an edge component, and 12 denotes a vertical edge for detecting an edge component which is a difference between upper and lower pixels in a predetermined pixel of the G signal. The detecting means 13 is a judging means for judging a change in the signal level in the peripheral pixels in the horizontal and vertical directions based on the outputs from the horizontal and vertical edge detecting means 11 and 12, and outputting the judgment result. Edge detection means 11,
The vertical edge detecting means 12 and the judging means 13 constitute the first edge judging means 5. Numeral 14 denotes an average value calculating means for calculating the average value of the upper, lower, left, and right pixels of a predetermined pixel of the G signal.
An operation 8 performs an operation from the R signal, the output signal Ghlpf from the horizontal low-pass filter 15 and the output signal Rhlpf from the horizontal low-pass filter 16, and outputs a G signal at a pixel position (2i, 2j + 1) having a correlation in the horizontal direction. Means 19 performs an operation on the B signal, the output signal Ghlpf from the horizontal low-pass filter 15 and the output signal Bhlpf from the horizontal low-pass filter 17, and calculates pixel positions (2i + 1, 2
This is an arithmetic means for outputting the G signal in j). 20-22
Is a vertical low-pass filter, 23 is an R signal, an output signal Gvlpf from the vertical low-pass filter 20, and an output signal Rvl from the vertical low-pass filter 21.
calculation means for performing a calculation by pf and outputting a G signal at a pixel position (2i, 2j + 1) having a vertical correlation;
An operation 4 performs an operation based on the B signal, the output signal Gvlpf from the vertical low-pass filter 20 and the output signal Bvlpf from the vertical low-pass filter 22, and outputs a G signal at a pixel position (2i + 1, 2j) having a vertical correlation. Means. Reference numeral 25 denotes a switching unit, which converts the G pixel signal into the average value calculating unit 14 and the calculating units 18 and 19 based on the result of determining the signal level change of the peripheral pixel from the first edge determining unit 5 and the pixel position. Switching is selected from signals from the arithmetic means 23 and 24 and the separating means 4.

FIG. 4 is a block diagram showing an example of the configuration of the second edge judging means 8 and the second RB component restoring means 9. In the figure, the second edge determination means 8 is G
A horizontal edge detecting means 30 for detecting a difference between left and right pixels in a predetermined pixel of a signal, a vertical edge detecting means 31 for detecting a difference between upper and lower pixels, and outputs from the horizontal and vertical edge detecting means 30 and 31 Determining means for determining a change in signal level of peripheral pixels in the horizontal and vertical directions, and outputting a determination result for the pixel position (2i + 1, 2j) of the R signal and a determination result for the pixel position (2i + 1, 2j) of the B signal. 32.
Pixel R at pixel position (2i + 1,2j) in R and pixel B at pixel position (2i, 2j + 1) in B
In the second RB component restoring means 9 for restoring R, 33 is an R average value calculating means for calculating an average value of obliquely adjacent pixels in the R signal, and 34 is a G component restoring means 6.
Horizontal calculating means for calculating a pixel signal at a pixel position (2i + 1,2j) in R by horizontally adjacent pixels based on the G signal from the R and the R signal from the first RB restoring means 7; In the same manner as described above, the R vertical direction calculating means 36 for calculating the pixel signal at the pixel position (2i + 1, 2j) by the adjacent pixels of the above, based on the determination result from the second edge determining means 8, R switching means for switching the R pixels in response to the B signal; 37 means a B average value calculating means for calculating an average value of pixels adjacent in the diagonal direction in the B signal; 38 indicates the G signal from the G component restoring means 6 and the first signal; R
B horizontal direction calculating means for calculating a pixel signal at a pixel position (2i, 2j + 1) in B by horizontally adjacent pixels based on the B signal from the B restoring means 7; And the pixel position (2i, 2
j + 1), a B vertical direction calculating means for calculating the pixel signal at
Reference numeral 40 denotes a B switching unit that switches a B pixel according to the pixel position based on the determination result from the second edge determination unit 8.

Next, the operation will be described. Each pixel signal R, G, B is read from the image sensor 1 and the output is A / D
A / D conversion is performed by the converter 2 and the frame memory 3
Is input to Each of the R, G, and B signals is separated from the signal input to the frame memory 3 by a separation unit 4, and the G signal is sent to a first edge determination unit 5 and a G component restoration unit 6, and the R, B signal Is the G component restoration means 6 and the first RB
It is sent to the restoration means 7. The first edge determination means 5 determines an edge component at a predetermined pixel position in the G signal and sends the determination result to the G component restoration means 6. Based on the determination result, the G component pixel signal is restored so as to obtain the G signal of the total number of pixels in the image sensor 1. This operation is described in FIG.
It will be described according to.

In the first edge judging means 5, the G signal is inputted to the horizontal edge detecting means 11 and the vertical edge judging means 12. Now, the G component in the image sensor 1 is obtained at pixel positions (2i, 2j) and (2i + 1, 2j + 1) as shown in FIG. 1, and in order to obtain a G signal of the number of pixels of the image sensor, the pixel position ( 2i, 2j +
Signals at the pixels 1) and (2i + 1, 2j) are obtained. FIG. 5 is a diagram showing the G signal at each pixel position from the separating means 4, where G is the G signal from the image sensor 1.
The signal g indicates a pixel from which no signal is obtained.
Therefore, the horizontal edge detecting means 11 and the vertical edge determining means 12 determine the difference between the left and right and upper and lower pixels at the pixel positions (2i, 2j + 1) and (2i + 1, 2j) (g indicated by oblique lines in FIG. 5), that is, the edge. Detect the component. The horizontal edge detecting means 11 obtains the absolute value ΔH of the difference between the left and right pixels at the pixel position and outputs the absolute value ΔH to the determining means 13. The vertical edge detecting means 12 calculates the difference between the upper and lower pixels. The absolute value ΔV is obtained and output to the determination means 13. For example, at the pixel position (2i, 2j + 1), the horizontal edge detecting means 11 calculates ΔH = | G (2i, 2j) -G (2i, 2j + 2) | (1) and the vertical edge detecting means 12 ΔV = | G (2i−1, 2j + 1) −G (2i + 1, 2j + 1) | (2), and sends ΔH and ΔV to the determination unit 13. Hereinafter, the absolute value of the difference between the pixels is referred to as an edge component.

The determination means 13 determines a change in the signal level in the peripheral pixels in the horizontal and vertical directions based on the edge component ΔH in the horizontal direction and the edge component ΔV in the vertical direction.
A signal ed1 indicating the determination result is output. That is, Δ
If both H and ΔV are equal to or less than the predetermined value th1, it is determined that there is no change in the signal level in the peripheral pixels, and for example, e
Output as d1 = 1. On the other hand, if ΔH or ΔV is larger than the predetermined value th1, it is determined that there is an edge component in the pixel, and if ΔH> ΔV, it is determined that the correlation is high in the vertical direction, and for example, ed1 = 2 If ΔH ≦ ΔV, it is determined that the correlation is high in the horizontal direction, and for example, ed1 = 3 is output. Note that pixel positions (2i, 2j), (2i + 1, 2j + 1) where G is obtained
In, it is not necessary to determine the edge component, and the determination unit 13 outputs, for example, ed1 = 0. The output ed1 of the judging means 13 is sent to the switching means 25 in the G component restoring means 6.

Next, in the G component restoring means 6, the G signal is inputted to the average value calculating means 14, the horizontal low-pass filter 15, and the vertical low-pass filter 20.
In the average value calculating means 14, the pixel position (2i, 2j + 1)
In (2i + 1, 2j), the average value ga of the values of the four pixels in the upper, lower, left and right directions is calculated and sent to the switching means 25. The horizontal low-pass filter 15 outputs a value Ghlpf through a G horizontal low-pass filter, and the vertical low-pass filter 20 outputs a value Gvlpf through a vertical G low-pass filter.
i, 2j + 1) is calculated as in the following equation. Ghlpf = {G (2i, 2j) + G (2i, 2j-2) + G (2i, 2j + 2) + G (2i, 2j + 4)} / 4 (3) Gvlpf = {G (2i-3,2j + 1) + G ( 2i−1, 2j + 1) + G (2i + 1, 2j + 1) + G (2i + 3, 2j + 1)｝ / 4 (4) The output Ghlpf of the horizontal low-pass filter 15 is sent to the calculation means 18 and 19, and the output Gvlpf of the vertical low-pass filter 20 is It is sent to the calculating means 23 and 24.

On the other hand, the R signal is input to the horizontal low-pass filter 16 and the vertical low-pass filter 21.
Value Rh through horizontal and vertical low pass filters
lpf and Rvlpf, and outputs a low-pass filter 17 in the horizontal direction and a low-pass filter 2 in the vertical direction for the B signal.
2 and output values Bhlpf and Bvlpf through low-pass filters in the horizontal and vertical directions, respectively. Here, the R component in the image sensor 1 is obtained at the pixel position (2i, 2j + 1) as shown in FIG.
, The R signal at each pixel position is as shown in FIG. 6A, and the B component is at the pixel position (2i + 1,
2B), and the B signal at each pixel position from the separation means 4 is as shown in FIG. 6B. Note that blank pixels are pixels for which no signal has been obtained by the image sensor.
Accordingly, at the pixel position (2i, 2j + 1), the value of the R signal through the horizontal and vertical low-pass filters 16 and 21 is obtained by, for example, the following equation. Rhlpf = {R (2i, 2j-3) + 2 × R (2i, 2j-1) + 2 × R (2i, 2j + 1) + 2 × R (2i, 2j + 3) + R (2i, 2j + 5)} / 8 (5) Rvlpf = {R (2i-4, 2j + 1) + 2 × R (2i-2, 2j + 1) + 2 × R (2i, 2j + 1) + 2 × R (2i + 2, 2j + 1) + R (2i + 4, 2j + 1)} / 8 (6) At the pixel position (2i + 1, 2j), the value of the B signal through the horizontal and vertical low-pass filters 17 and 22 is obtained by, for example, the following equation. Bhlpf = {B (2i + 1, 2j-4) + 2 × B (2i + 1, 2j-2) + 2 × B (2i + 1, 2j) + 2 × B (2i + 1, 2j + 2) + B (2i + 1, 2j + 4)} / 8 (7) Bvlpf = {B (2i-3,2j) + 2 × B (2i-1,2j) + 2 × B (2i + 1,2j) + 2 × B (2i + 3,2j) + R (2i + 5,2j)} / 8 (8 )

The above equations (3) to (8) are examples of calculating the output of each horizontal low-pass filter and vertical low-pass filter, and the number of taps and coefficients of the filter are limited to the above equations (3) to (8). Instead, other tap numbers and coefficients may be used.

The horizontal low-pass filter 1
The output Rhlpf at 6 is sent to the calculating means 18 to which the R signal and the output signal Ghlpf of the horizontal low-pass filter 15 are also input, and the pixel position (2
G signal gh1 having a horizontal correlation at (i, 2j + 1)
(2i, 2j + 1) is calculated by the following equation: gh1 (2i, 2j + 1) = R (2i, 2j + 1) × (Ghlpf / Rhlpf) (9), and sent to the switching means 25. The output Rvlpf from the vertical low-pass filter 21 is sent to the calculating means 23. The R signal and the output signal Gvlpf from the vertical low-pass filter 20 are also input to the calculating means 23, and the pixel position (2i, 2j + 1) The G signal gv1 (2i, 2j + 1) having a correlation in the vertical direction is calculated by the following equation: gv1 (2i, 2j + 1) = R (2i, 2j + 1) × (Gvlpf / Rvlpf) (10) send.

Similarly, the horizontal low-pass filter 1
The output Bhlpf at 7 is sent to the calculating means 19, to which the B signal and the output signal Ghlpf of the horizontal low-pass filter 15 are also input, and the pixel position (2i
G signal gh2 having a horizontal correlation at (+1, 2j)
(2i + 1, 2j) is calculated by the following equation: gh2 (2i + 1, 2j) = B (2i + 1, 2j) × (Ghlpf / Bhlpf) (11), and is sent to the switching means 25. The output Bvlpf from the vertical low-pass filter 22 is sent to the calculating means 24. The B signal and the output signal Gvlpf from the vertical low-pass filter 20 are also input to the calculating means 24, and the pixel position (2i + 1, 2j) The G signal gv2 (2i + 1, 2j) having a correlation in the vertical direction is calculated by the following equation: gv2 (2i + 1, 2j) = B (2i + 1, 2j) × (Gvlpf / Bvlpf) (12) send.

The calculation method using the above equations (9) to (12) is based on the premise that there is little change in color in a local area. , The ratio of each color signal in a local region in the horizontal direction or the vertical direction is given by the ratio of values through a low-pass filter in the horizontal or vertical direction of R, G, and B.

The switching means 25 converts the G pixel signal into the average value calculating means 14, based on the result ed1 of the judgment of the signal level change of the peripheral pixel from the judgment means 13 in the first edge judgment means 5 and the pixel position. Arithmetic means 18, 1
9 and signals from the arithmetic means 23, 24 and the separating means 4 for selection. That is, when the output signal ed1 from the determination unit 13 indicates “2”, that is, when it is determined that an edge component exists in the pixel, but when it is determined that ΔH> ΔV and the correlation is high in the vertical direction, The signal of the arithmetic means for outputting G having a correlation in the vertical direction at the pixel position is selected. When the output signal ed1 from the determination means 13 indicates '3', that is, when it is determined that there is an edge component in the pixel, but when it is determined that ΔH ≦ ΔV and the correlation in the horizontal direction is high, each pixel A signal of the calculating means for outputting G having a correlation in the horizontal direction at the position is selected. Further, when the output signal ed1 by the determination means 13 indicates “1”, both ΔH and ΔV are equal to or smaller than the predetermined value th1, and it is determined that there is no change in the signal level in the peripheral pixels. There is no need to consider a change in frequency, and the G signal ga by the average value calculation means 14 is selected.

That is, in the switching means 25, the G signal g (2i, 2j +) at the pixel position (2i, 2j + 1)
In the case of 1), when the output ed1 = 1 from the determination unit 13, the G signal ga from the average value calculation unit 14 is used, when ed = 2, the G signal gv1 from the calculation unit 23 is used. The G signal gh1 from the calculating means 18 is selected.
G signal g (2i + 1,2) at pixel position (2i + 1,2j)
In the case of j), when the output ed1 = 1 from the determination unit 13, the G signal ga from the average value calculation unit 14 is used, when ed = 2, the G signal gv2 from the calculation unit 24 is used. The G signal gh2 from the calculating means 19 is selected. Note that pixel positions (2i, 2j), (2i
In (+1, 2j + 1), for example, ed1 = 0 is output from the determination unit 13;
May be output as it is.

Therefore, the pixel positions (2i, 2j), (2i, 2j + 1), (2i +
A G signal is output at each pixel of (1, 2j) and (2i + 1, 2j + 1), that is, a G signal having a resolution equal to the number of pixels of the image sensor is obtained. Next, the output G0 from the G component restoring means 6 is sent to the first RB component restoring means 7, the second edge judging means 8 and the second RB component restoring means 9.

Next, in the first RB component restoring means 7, the R and B signals from the separating means 4 (FIGS. 6A and 6B)
And the G signal G0 having the signals of all the pixels from the G component restoring means 6 to restore and generate the pixels at the pixel positions (2i, 2j) and (2i + 1, 2j + 1) in the R and B signals, respectively. FIGS. 7A and 7B are diagrams showing the R and B of each pixel for explaining the calculation of the R and B signals in the first RB component restoring means 7, where r and b are the first.
5 shows R and B pixel signals restored and generated by the RB component restoring means 7 of FIG. The pixel position (2i, 2
In j), pixels on the 2i-th line adjacent in the horizontal direction are obtained from the image sensor 1, and all the G signals are restored by the G component restoration means 6 (FIG. 5). Therefore, at the pixel position (2i, 2j), the value G1hl through the horizontal low-pass filter is applied to the G and R signals.
For example, pf and R1hlpf are expressed as follows: G1hlpf = (g (2i, 2j-1) + g (2i, 2j + 1)) / 2 (13) R1hlpf = (R (2i, 2j-1) + R (2i, 2j + 1)) / 2 ( 14) and the ratio of G1hlpf to R1hlpf and the pixel G (2i, 2j) are used to calculate the pixel position (2i, 2j).
Is calculated by the following equation. r (2i, 2j) = G (2i, 2j) × (R1hlpf / G1hlpf) = G (2i, 2j) × (R (2i, 2j−1) + R (2i, 2j + 1)) / (g (2i, 2j) -1) + g (2i, 2j + 1)) (15)

Further, at the pixel position (2i + 1, 2j + 1), pixels in the vertically adjacent 2j + 1 columns are obtained from the image pickup device 1, and all the G signals are restored by the G component restoring means 6. The pixel position (2i
+1 and 2j + 1), the values G1vlppf and R1v of the G and R signals through the low-pass filter in the vertical direction.
Ipf is calculated by, for example, G1vlpf = (g (2i, 2j + 1) + g (2i + 2, 2j + 1)) / 2 (16) R1vlpf = (R (2i, 2j + 1) + R (2i + 2, 2j + 1)) / 2 (17) By the ratio of G1vlpf to R1vlpf and the pixel G (2i + 1, 2j + 1), the pixel position (2i
R signal r (2i + 1, 2j + 1) at (+1, 2j + 1)
Is calculated by the following equation. r (2i + 1, 2j + 1) = G (2i + 1, 2j + 1) × (R1vlppf / G1vlpf) = G (2i + 1, 2j + 1) × (R (2i, 2j + 1) + R (2i + 2, 2j + 1)) / (g (2i, 2j + 1)) + G (2i + 2,2j + 1)) (18)

Similarly, for the B signal, the pixel position (2
In (i, 2j), pixels in 2j columns adjacent in the vertical direction are obtained from the image sensor 1, and all the G signals are restored by the G component restoration means 6 (FIG. 5). Therefore, at the pixel position (2i, 2j), the value G1vl obtained by passing the G and B signals through the vertical low-pass filter.
pf and B1vlpf are, for example, G1vlpf = (g (2i-1, 2j) + g (2i + 1, 2j)) / 2 (19) B1vlpf = (B (2i-1, 2j) + B (2i + 1, 2j)) / 2 ( 20) and the ratio of G1vlpf to B1vlpf and the pixel G (2i, 2j) are used to calculate the pixel position (2i, 2j).
Is calculated according to the following equation. b (2i, 2j) = G (2i, 2j) × (B1vlpf / G1vlpf) = G (2i, 2j) × (B (2i−1, 2j) + B (2i + 1, 2j)) / (g (2i−1) , 2j) + g (2i + 1, 2j)) (21)

Further, at the pixel position (2i + 1, 2j + 1), the pixels of the horizontally adjacent 2i + 1 line are obtained from the image pickup device 1, and the G signal is restored by the G component restoring means 6 to all pixels. ing. Therefore, at the pixel position (2i + 1, 2j + 1), the value G1hlp obtained through the low-pass filter in the horizontal direction is applied to the G and B signals.
f, B1hlpf is calculated by, for example, G1hlpf = (g (2i + 1, 2j) + g (2i + 1, 2j + 2)) / 2 (22) B1hlpf = (B (2i + 1, 2j) + B (2i + 1, 2j + 2)) / 2 (23) Then, by the ratio of G1hlpf to B1hlpf and the pixel G (2i + 1, 2j + 1), the pixel position (2i
B signal b (2i + 1, 2j + 1) at (+1, 2j + 1)
Is calculated by the following equation. b (2i + 1, 2j + 1) = G (2i + 1, 2j + 1) × (B1hlpf / G1hlpf) = G (2i + 1, 2j + 1) × (B (2i + 1, 2j) + B (2i + 1, 2j + 2)) / (g (2i + 1, 2j) + G (2i + 1, 2j + 2)) (24)

The above equations (15), (18) and (2)
1) and (24) are based on the premise that the change of the color signal in the local region is small as in the above-described restoration method in G, that is, the ratio of each signal is almost equal in the local region. G1h in the equations (13) to (24)
lpf, G1vlpf, R1hlpf, R1vlpf,
The formulas for calculating B1hlpf and B1vlpf are calculation examples of low-pass filter outputs in the horizontal and vertical directions, and the number of taps and coefficients of the filter are not limited to the above, and other tap numbers and coefficients may be used. Then, R, r and B shown in FIG.
b are output and the second RB
It is sent to the component restoring means 9.

Next, the operation of the second RB component restoring means 9 and the second edge judging means 8 will be described with reference to FIG. In the second edge determining means 8, the G signal G0 output from the G component restoring means 6 is input to the horizontal edge detecting means 30 and the vertical edge determining means 31,
Pixel positions (2i, 2j + 1) and (2i + 1, 2j)
, The difference between the left and right and upper and lower pixels, that is, the edge component is detected. The horizontal edge detecting means 30 obtains the absolute value ΔH of the difference between the left and right pixels at the pixel position, outputs the absolute value ΔH to the determining means 32, and outputs
In step 1, the absolute value ΔV of the difference between the upper and lower pixels is obtained and output to the determination means 32. For example, the pixel position (2i, 2j
In (+1), the horizontal edge detecting means 30 calculates ΔH = | G (2i, 2j) −G (2i, 2j + 2) |, and the vertical edge detecting means 31 calculates ΔV = | G (2i−1). , 2j + 1) -G (2i + 1,
2j + 1) | and sends the above ΔH and ΔV to the determination means 32. Hereinafter, the absolute value of the difference between the pixels is referred to as an edge component.

The determining means 32 determines a change in the signal level of the peripheral pixels in the horizontal and vertical directions based on the edge component ΔH in the horizontal direction and the edge component ΔV in the vertical direction.
A signal indicating the determination result is output. Here, the first RB component restoring means 7 outputs the R signal R1
In FIG. 7, pixel signals other than the pixel position (2i + 1, 2j) are obtained (FIG. 7A), and in the B signal B1, the pixel position (2
Pixel signals other than (i, 2j + 1) are obtained (FIG. 7).
(B)). Therefore, the result edr of determining the edge component at the pixel position (2i + 1, 2j) is output for the restoration of the pixel position (2i + 1, 2j) of the R signal, while the edge component at the pixel position (2i, 2j + 1) is output. Is output for the restoration of the pixel position (2i, 2j + 1) of the B signal. That is, the pixel position (2i + 1, 2
j) and the pixel position (2i, 2j +
In the determination result edb in 1), when both ΔH and ΔV are equal to or less than the predetermined value th2, it is determined that there is no change in the signal level in the peripheral pixels, and the output is made, for example, as edr = 1 and edb = 1. On the other hand, if ΔH or ΔV is larger than the predetermined value th2, it is determined that there is an edge component in the pixel, and if ΔH> ΔV, it is determined that the correlation is high in the vertical direction.
2 and edb = 2, and when ΔH ≦ ΔV, it is determined that the correlation is high in the horizontal direction, and for example, edr = 3 and e
db = 3 is output. It is not necessary to determine the edge component at the pixel position where R and B are obtained, and the determination unit 32 outputs, for example, edr = 0 and edb = 0. Then, the output edr of the judgment means 32
Is the R switching means 3 in the second RB component restoring means 9
6 and the output edb is sent to the B switching means 40 in the second RB component restoring means 9.

Next, in the second RB component restoring means 9, the R signal output R1 from the first RB component restoring means 7 is output to the R average value calculating means 33, the R horizontal direction calculating means 34 and the R vertical direction calculating means. 35, input to the R switching means 36. FIG. 8A is a diagram for explaining the restoration of the R signal at the pixel position (2i + 1, 2j) in the second RB component restoration means 9, and the pixels indicated by R and r in the figure are image pickup devices. The obtained signal and the pixel signal restored by the first RB restoring means 7 are obtained. Therefore, the signal of the pixel r 'indicated by oblique lines is obtained. First, the R average value calculating means 33 calculates the pixel position (2
i + 1, 2j), four pixels R (2
i, 2j-1), R (2i, 2j + 1), R (2i +
2, 2j-1), the average value r of R (2i + 2, 2j + 1)
a is calculated by the following equation. ra (2i + 1, 2j) = ｛R (2i, 2j−1) +
R (2i, 2j + 1) + R (2i + 2, 2j-1) +
R (2i + 2, 2j + 1)｝ / 4

The R horizontal direction calculating means 34 also receives the G signal G0 (FIG. 5) from the G component restoring means 6 and applies a horizontal low-pass filter at the pixel positions (2i + 1, 2j) of the R and G signals. Via the value R2hlpf, G
For example, 2hlpf is calculated as follows: R2hlpf = (r (2i + 1, 2j-1) + r (2i + 1, 2j + 1)) / 2 (25) G2hlpf = (G (2i + 1, 2j-1) + G (2i + 1, 2j + 1)) / 2 (26) And the ratio of R2hlpf to G2hlpf and the pixel g (2i + 1, 2j) are used to calculate (2i
A pixel value rh having a horizontal correlation of (+1, 2j) is calculated and output. rh (2i + 1, 2j) = g (2i + 1, 2j) × (R2hlpf / G2hlpf) = g (2i + 1, 2j) × (r (2i + 1, 2j−1) + r (2i + 1, 2j + 1)) / (G (2i + 1, 2j-1) + G (2i + 1, 2j + 1)) (27)

The G signal G0 from the G component restoring means 6 is also input to the R vertical direction calculating means 35. At the pixel position (2i + 1,2j) in the R and G signals, the value R2vlpf passed through the vertical low-pass filter. , G2
vlpf is calculated by, for example, R2vlpf = (r (2i, 2j) + r (2i + 2, 2j)) / 2 (28) G2vlpf = (G (2i, 2j) + G (2i + 2, 2j)) / 2 (29) Based on the ratio of R2vlpf to G2vlpf and pixel g (2i + 1, 2j), (2i
A pixel value rv having a correlation in the vertical direction of (+1, 2j) is calculated and output. rv (2i + 1, 2j) = g (2i + 1, 2j) × (R2vlpf / G2vlpf) = g (2i + 1, 2j) × (r (2i, 2j) + r (2i + 2, 2j)) / (G (2i, 2j) + G (2i + 2,2j)) (30)

As described above, the above equations (27) and (30) are based on the premise that the change of the color signal in the local region is small, that is, the ratio of each signal is almost equal in the local region. By being equal, the equations (25) to
G2hlpf, G2vlpf, R2h in (30)
The formulas for calculating lpf and R2vlpf are calculation examples of low-pass filter outputs in the horizontal and vertical directions, and the number of taps and coefficients of the filter are not limited to the above, and other tap numbers and coefficients may be used. And the output rh
And rv are both sent to the R switching means 36.

The R switching means 36 determines the pixel position (2
i + 1, 2j) by the R average value calculating means 33, R
Horizontal operation means 34 and R vertical operation means 35,
The signal is selected from the signal from the first RB component restoring means 7 and switched. That is, when the output signal edr by the determination means 32 indicates “2”, that is, when it is determined that the pixel has an edge component, but when it is determined that ΔH> ΔV and the correlation in the vertical direction is high, the vertical R with correlation in direction
Is selected from the signal Rv of the R vertical direction calculation means 35 which outputs. When the output signal edr by the determination means 32 indicates '3', that is, when it is determined that the pixel has an edge component, but when it is determined that ΔH ≦ ΔV and the correlation in the horizontal direction is high, the horizontal direction The signal of the R horizontal direction calculation means 34 which outputs R having a correlation with is selected. Further, when the output signal edr by the determination means 32 indicates “1”, both ΔH and ΔV are equal to or smaller than the predetermined value th2, and it is determined that there is no change in the signal level in the peripheral pixels. There is no need to consider a change in frequency, and the R signal ra by the R average value calculation means 33 is selected. Note that pixel positions (2i, 2j) and (2i, 2j +
In 1) and (2i + 1, 2j + 1), the judgment means 3
2 outputs edr = 0, for example, and in this case, the R signal from the first RB component restoration means 7 may be output as it is.

The processing for the B signal is the same as that for the R signal. In the second RB component restoring means 9, the B signal output B 1 from the first RB component restoring means 7 is converted to a B average value calculating means 37. The signals are inputted to the B horizontal direction calculating means 38, the B vertical direction calculating means 39, and the B switching means 40. FIG. 8B is a diagram for explaining the restoration of the B signal at the pixel position (2i, 2j + 1) in the second RB component restoration means 9, and the pixels indicated by B and b in the figure are image pickup devices. The obtained signal and the pixel signal restored by the first RB restoration means 7 are obtained. Therefore, the signal of the b 'pixel indicated by oblique lines is obtained. First, the B average value calculation means 3
7, in the B signal B1, the pixel position (2i, 2j +
The average value ba of four pixels obliquely adjacent to each other in 1) is calculated by the following equation. ba (2i, 2j + 1) = ｛B (2i-1, 2j) + B
(2i-1, 2j + 2) + B (2i + 1, 2j) + B
(2i + 1, 2j + 2)｝ / 4

The B horizontal direction calculating means 38 also receives the G signal G0 (FIG. 5) from the G component restoring means 6 and applies a horizontal low-pass filter at the pixel position (2i, 2j + 1) in the B and G signals. Through the value B2hlp
f, G3hlpf is calculated by, for example, B2hlpf = (b (2i, 2j) + b (2i, 2j + 2)) / 2 (31) G3hlpf = (G (2i, 2j) + G (2i, 2j + 2)) / 2 (32) Then, based on the ratio of B2hlpf to G3hlpf and the pixel g (2i, 2j + 1), (2
A pixel value bh having a horizontal correlation of (i, 2j + 1) is calculated and output. bh (2i, 2j + 1) = g (2i, 2j + 1) × (B2hlpf / G3hlpf) = g (2i, 2j + 1) × (b (2i, 2j) + b (2i, 2j + 2)) / (G (2i, 2j) + G (2i, 2j + 2)) (33)

The B signal G0 from the G component restoring means 6 is also input to the B vertical direction calculating means 39. At the pixel position (2i, 2j + 1) in the B and G signals, the value B2vlpf passed through the vertical low-pass filter. , G3
For example, v2pf is calculated as follows: B2vlpf = (b (2i-1, 2j + 1) + b (2i + 1, 2j + 1)) / 2 (34) G3vlpf = (G (2i-1, 2j + 1) + G (2i + 1, 2j + 1)) / 2 (35) The ratio of B2vlpf to G3vlpf and the pixel g (2i, 2j + 1) are used to calculate (2)
A pixel value bv having a correlation in the vertical direction (i, 2j + 1) is calculated and output. bv (2i, 2j + 1) = g (2i, 2j + 1) × (B2vlpf / G3vlpf) = g (2i, 2j + 1) × (b (2i−1, 2j + 1) + b (2i + 1, 2j + 1)) / (G (2i -1, 2j + 1) + G (2i + 1, 2j + 1)) (36)

As described above, the above equations (33) and (36) are based on the premise that the change of the color signal in the local region is small, that is, the ratio of each signal is almost equal in the local region. By being equal, the equations (31) to
G3hlpf, G3vlpf, B2h in (36)
The formulas for calculating lpf and B2vlpf are calculation examples of low-pass filter outputs in the horizontal and vertical directions, and the number of taps and coefficients of the filter are not limited to the above, and other tap numbers and coefficients may be used. And the output bh
And bv are both sent to the B switching means 40.

The B-switching means 40 determines the pixel position (2
i, 2j + 1) is converted to the B average value calculating means 37, B
Horizontal operation means 38 and B vertical operation means 39;
The signal is selected from the signal from the first RB component restoring means 7 and switched. In other words, when the output signal edb from the determination means 32 indicates “2”, that is, when it is determined that the pixel has an edge component, but when it is determined that ΔH> ΔV and the correlation in the vertical direction is high, the vertical R with correlation in direction
Is selected from the signal bv of the R vertical direction calculating means 39 which outputs the following. When the output signal edb from the determination means 32 indicates “3”, that is, when it is determined that the pixel has an edge component, but when it is determined that ΔH ≦ ΔV and the correlation in the horizontal direction is high, the horizontal direction The signal of the B horizontal direction operation means 38 which outputs B having a correlation with is selected. Further, when the output signal edb by the determination means 32 indicates “1”, both ΔH and ΔV are equal to or less than the predetermined value th2, and it is determined that there is no change in the signal level in the peripheral pixels. There is no need to consider a change in frequency, and the B signal ba by the B average value calculation means 37 is selected. Note that pixel positions (2i, 2j), (2i + 1, 2) where B is obtained
j) and (2i + 1, 2j + 1), the judgment means 3
For example, edb = 0 is output from 2; in this case, the B signal from the first RB component restoration means 7 may be output as it is.

As described above, the pixel positions (2i, 2j), (2i, 2j + 1), (2
The R and B signals at the respective pixels of (i + 1, 2j) and (2i + 1, 2j + 1) are output, that is, R and B signals having the resolution of the number of pixels of the image sensor can be obtained. G
The signal has already been obtained in the G component restoration means 6 with a resolution of the number of pixels of the image sensor. Therefore, the edge component in the horizontal and vertical directions is determined using the G component, and the spatial frequency of the local region is determined. The change is determined, the generation of each color signal is switched based on the determination result, and in the calculation of the signal when the edge component exceeds a predetermined value, the calculation is performed based on the ratio of the color signal in a local region, and the edge component is calculated. If the value is equal to or less than the predetermined value, the average value of the peripheral pixels is obtained.
An image in which false colors and false contours are reduced is obtained.

FIG. 9 shows the relationship between the false color level (color-alias level, vertical axis) and the resolution (horizontal axis) generated in the horizontal direction when the zone plate is processed by image simulation. The false color level after processing according to the prior art according to FIG. 20 is indicated by a dashed line, and the false color level after processing according to the first embodiment according to FIG. 2 is indicated by a solid line. Many false colors are generated in the processing according to the related art, but the processing according to the first embodiment suppresses the generation of false colors. When the levels of the false color signal in the horizontal direction are integrated and compared in each process, a suppression effect of about 18.4 dB is obtained.

FIG. 10 shows the relationship between the false-color level (color-alias level, vertical axis) generated in the vertical direction and the resolution (horizontal axis). The false color level after processing according to the related art is indicated by a dashed line, and the false color level after processing according to the first embodiment shown in FIG. 2 is indicated by a solid line. Although many false colors are generated in the processing according to the related art, the generation of the false colors is suppressed in the processing according to the first embodiment. There is a suppression effect of 18.5 dB.

Further, FIGS. 11 and 12 are diagrams in which the level of the luminance signal in the horizontal direction when the zone plate is processed by the image simulation is calculated from the RGB signals, and the vertical axis represents the resolution and the horizontal axis represents the resolution. FIG. 11 shows the case after the processing according to the first embodiment shown in FIG. 2, and FIG. 12 shows the case after the processing according to the prior art shown in FIG. FIG. 13 shows the relationship between the level of the luminance signal and the resolution in the vertical direction after processing according to the first embodiment shown in FIG. 2, and FIG. 14 shows the case after processing according to the prior art shown in FIG. In both the horizontal and vertical directions, in the processing according to the first embodiment, the reduction in level is small and the resolution in the horizontal and vertical directions is improved as compared with the processing according to the conventional technique.

In the first embodiment, the image pickup device 1
The color filter array is a primary color filter shown in FIG. 1, and each photoelectric conversion element is an image pickup element that is independently called, and a pixel position (2i, 2j) and a pixel position (2i + 1, 2
j + 1) (i = 0, 1, 2,..., j = 0, 1, 2,...)
A first color filter having a spectral characteristic to pass a G signal through the pixel position, and a second color filter having a spectral characteristic to pass an R signal to the pixel position (2i, 2j + 1).
The description has been made assuming that a third color filter having a spectral characteristic for passing the B signal is arranged in (i + 1, 2j). However, in the imaging device, the spectral characteristics of the first, second, and third color filters are R, The image sensor is not limited to G and B, but may be, for example, a pixel-mixing type image sensor. As shown in FIG. 15, the pixel position (2i, 2j) and the pixel position (2i + 1, 2
j + 1) (i = 0, 1, 2,..., j = 0, 1, 2,...)
A first color filter having a spectral characteristic to pass the first signal A, a second color filter having a spectral characteristic to pass the second signal B to the pixel position (2i, 2j + 1),
A third color filter having a spectral characteristic for passing the third signal C is arranged at the pixel position (2i + 1, 2j).
It is sufficient that the RGB color signals can be reproduced after the B and C signals have been restored, and the same effects as described above can be obtained.

In the first embodiment, the arrangement of the color filters of the image pickup device shown in FIGS.
i, 2j) and pixel position (2i + 1, 2j + 1) (i =
, J = 0, 1, 2,...), The second color filter at the pixel position (2i, 2j + 1), and the second color filter at the pixel position (2i + 1, 2j). Although the case where three color filters are arranged (shaded portions in FIGS. 1 and 15) has been described, as shown in FIG. 16, the pixel positions (2i, 2
j + 1) and pixel positions (2i + 1, 2j) (i = 0, 1,
, J = 0, 1, 2,...) And the first color filter,
When the second color filter is arranged at the pixel position (2i, 2j) and the third color filter is arranged at the pixel position (2i + 1, 2j + 1) (the hatched portion in FIG. 16), the same effect is obtained.
A first color filter and a second color filter are arranged on an n-line every four pixels in the upper and lower pixels, and a third color filter and a first color filter are arranged on an n + 1-th line. It suffices if the filters are arranged in oblique pixels.

In the first embodiment, the signals indicating the results of the judgment by the first edge judgment means 5 and the second edge judgment means 8 are determined when both ΔH and ΔV are smaller than a predetermined value. It is determined to be “1”, and it is determined that there is an edge component. If ΔH> ΔV, it is set to “2”. If ΔH ≦ ΔV, it is set to “3”. Although the description has been given of a case where the determination is made, the present invention is not limited to this, and other values may be used as long as a signal capable of distinguishing each determination result is output.

In the first embodiment, the case where the processing of the configuration shown in FIG. 2 is performed by hardware has been described. The same effect as in the first embodiment is exerted.

Embodiment 2 In the first embodiment, the first edge determination means 5 determines the pixel position (2
i, 2j + 1) and (2i + 1, 2j) to determine the edge components at the top, bottom, left, and right, and the second edge determination means 8 performs the restoration of the R signal at the pixel position (2i + 1, 2j) in the G signal. Is configured to determine the edge component at the pixel position (2i, 2j + 1) in the G signal with respect to the restoration of the B signal. Since the edge components of the left, right, upper and lower pixels are detected, and the upper, lower, left and right pixel signals at these pixel positions are signals obtained from the image sensor, the same edge component is obtained. Therefore, as shown in FIG. 17, the pixel position (2i, 2j +
It is also possible to adopt a configuration in which the edge components in (1) and (2i + 1, 2j) are determined.

In FIG. 17, reference numerals 1 to 4, 6 to 7, and 9 are the same as those in the image pickup apparatus according to the first embodiment. , Vertical edge detecting means 43 and determining means 44.

Next, the operation will be described. The pixel signals R, G, and B are read from the image sensor 1, A / D converted by the A / D converter 2 and input to the frame memory 3, and the R, G, and B signals are separated by the separation unit 4, and G The operation of generating and restoring each signal of the number of pixels of the image sensor by the component restoring means 6, the first RB restoring means 7, and the second RB restoring means 9 is the same as that in the first embodiment, so that the detailed description will be made. Is omitted.

The G component edge determining means 41 includes a separating means 4
The G signal (FIG. 18) is input to the horizontal edge detecting means 42 and the vertical edge detecting means 43 in the G component edge determining means 41. The pixel positions (2i, 2j + 1) and (2i + 1, 2j) in the horizontal edge detecting means 42 and the vertical edge detecting means 43
A difference between left and right and upper and lower pixels (pixels indicated by oblique lines in FIG. 18), that is, an edge component is detected. That is, the horizontal edge detection means 42 obtains the absolute value ΔHg of the difference between the left and right pixels at the pixel position and outputs this to the determination means 44. The absolute value ΔVg of the difference is obtained and output to the determination means 55.
For example, at the pixel position (2i, 2j + 1), the horizontal edge detection means 42 calculates ΔHg = | G (2i, 2j) -G (2i, 2j + 2) |, and the vertical edge detection means 43 calculates ΔVg = | G (2i-1, 2j + 1) -G (2i +
1, 2j + 1) | and sends the above ΔHg and ΔVg to the determination means 44.

In the determination means 44, the edge component ΔHg in the horizontal direction and the edge component ΔVg in the vertical direction are used.
A change in the signal level of the peripheral pixels in the horizontal and vertical directions is determined, and a signal ed1 indicating the determination result is sent to the G component restoring means 6 according to each pixel position of the input G signal, to edr,
edb is output to the second RB component restoration means 9. If each of the determination signals is less than or equal to a predetermined value, ΔHg and ΔVg are determined to have no change in the signal level in the peripheral pixels, and are set to “1”, for example. On the other hand, if ΔHg or ΔVg is larger than a predetermined value, it is determined that there is an edge component in the pixel. Further, if ΔHg> ΔVg, it is determined that the correlation is high in the vertical direction, and is set to, for example, “2”.
When ΔHg ≦ ΔVg, it is determined that the correlation is high in the horizontal direction, and is set to, for example, “3”. Here, in the G component restoration means 6, the pixel positions (2i, 2j + 1) and (2i + 1,
In order to restore the pixel in 2j), the edge determination result is output as ed1 at both pixel positions, and pixel positions (2i, 2j) and (2i + 1, 2j + 1) where G is obtained
Outputs, for example, ed1 = 0. Then, the output ed1 of the judging means 44 is sent to the switching means 25 in the G component restoring means 6.

On the other hand, the second RB component restoring means 9 restores the pixel position (2i + 1, 2j) of the R signal and restores the pixel position (2i, 2j + 1) of the B signal. Therefore, the result of determining the edge component at the pixel position (2i + 1, 2j) is defined as edr, and the second RB
The timing is output in synchronization with the pixel position to be processed by the component restoring means 9, and the other pixel positions (2i, 2j), (2i,
In (2j + 1) and (2i + 1, 2j + 1), for example, edr = 0 is output. Then, the output e of the determination means 44
dr is sent to the R switching means 36 in the second RB component restoring means 9. On the other hand, pixel positions (2i, 2j +
In 1), the result of the edge component determination is set as edb, and the timing is output in synchronization with the pixel position to be processed by the second RB component restoring means 9, and the other pixel positions (2i,
2j), (2i + 1, 2j), (2i + 1, 2j + 1)
Output edb = 0, for example. Then, the output edb of the judging means 44 is sent to the B switching means 40 in the second RB component restoring means 9.

Therefore, in the G component edge judging means 41, the edge component of the G signal corresponding to the pixel position at which the G, R, B signal is restored in the G component restoring means 6 and the second RB component restoring means 9 is obtained. The judgment result is output.

In the second embodiment, the G signal is applied to the pixel positions (2i, 2j) and (2i + 1, 2j + 1).
The R signal is applied to the pixel position (2i, 2j + 1) at the pixel position (2
The case where the B signal is obtained from the image sensor at (i + 1, 2j + 1) has been described. However, as in the first embodiment, the G signal is at pixel positions (2i, 2j + 1) and (2i + 1, 2j) and the pixel position (2i, 2j). ) At the pixel position (2).
i + 1, 2j + 1), a B signal may be arranged,
Further, the color signal is not limited to RGB.

In the second embodiment as well as in the first embodiment, it is needless to say that the same processing as the processing in the configuration of FIG. 17 can be performed by software in the imaging apparatus. A similar effect is achieved.

[0073]

Since the present invention is configured as described above, it has the following effects.

According to the imaging apparatus of the present invention, the pixel positions (2i, 2j) (i = 0, 1, 2,... And j = 0,
, And (2i + 1, 2j + 1) pass through a first color filter having a spectral characteristic of passing a first color signal, and pass through a pixel position (2i, 2j + 1) through a second color signal. The second color filter having characteristics, the pixel position (2i +
1, 2j) read out by the first color filter in an image sensor in which four upper and lower pixels in which a third color filter having a spectral characteristic for passing a third color signal is arranged are repeatedly arranged vertically and horizontally. The pixel position (2
In (i, 2j + 1) and (2i + 1, 2j), the edge component is determined based on the peripheral pixel signal, and based on the determination result, the pixel position of the first color signal is determined by the first, second, and third color signals. Is calculated to restore the first color signal of the number of pixels of the image sensor, and based on the edge determination result, the second and third color signals from the restored first color signal and the color signals from the respective color filters are used. By calculating the second color signal and the second and third color signals of the number of pixels of the image sensor, an image with reduced false colors and false contours can be obtained.

According to the imaging apparatus of the present invention,
The edge determination means calculates the absolute value of the difference between the left and right adjacent pixels in the predetermined pixel of the first color signal, detects the horizontal edge component ΔH, calculates the absolute value of the difference between the upper and lower pixels, and calculates the vertical value. Direction edge component ΔV, and based on the horizontal edge component ΔH and the vertical edge component ΔV, determine the horizontal or vertical edge component in the predetermined pixel to determine the local region. Can be determined in the horizontal and vertical directions, and an image with reduced false colors and false contours can be obtained.

According to the imaging apparatus of the present invention,
When the output ΔH from the horizontal edge detector or the output ΔV from the vertical edge detector is larger than a predetermined value, the edge detector detects an edge component in a peripheral pixel of the predetermined pixel. If ΔH> ΔV, it is determined that there is a correlation in the vertical direction. If ΔH ≦ ΔV, it is determined that there is a correlation in the horizontal direction. If both ΔH and ΔV are smaller than a predetermined value, By determining that no edge component is detected, it is possible to determine a change in the spatial frequency in the horizontal and vertical directions in a local region, and to obtain an image with reduced false colors and false contours.

According to the imaging apparatus of the present invention,
The means for calculating and restoring the first color signal is used to calculate the position (l = 2) of a predetermined pixel l row m column B (l, m) of the second color signal B
i, m = 2j + 1 or l = 2i + 1, m = 2j), the value Ahlpf of each of the first color signal A and the second color signal B through a horizontal low-pass filter
(L, m) and Bhlpf (l, m) are calculated, and Ahlpf which is an output signal from the horizontal low-pass filter is calculated.
The pixel value A (l, m) in the first color signal A of l rows and m columns is obtained from the ratio of (l, m) to Bhlpf (l, m) and the pixel value B (l, m) at the pixel position. A (l, m) = B
(L, m) × ｛Ahlpf (l, m) / Bhlpf
(L, m)}, a horizontal direction signal calculating means for calculating a pixel value of the first color signal A at another pixel position of the third color signal C, and the predetermined pixel l
At the position of row m column B (l, m), the first color signal A,
Values Avlpf (l, m), Bvlpf for each of the second color signals B through a vertical low-pass filter
(L, m) is calculated, and Avlpf (l, m) and Bvlpf, which are output signals from the vertical low-pass filter, are calculated.
(L, m) and the pixel value B (l, m) at the pixel position
As a result, the pixel value A which is the first color signal A of l rows and m columns
Let (l, m) be A (l, m) = B (l, m) × ｛Avl
pf (l, m) / Bvlpf (l, m)}, and the vertical direction signal calculation for calculating the pixel value of the first color signal A at another pixel position of the third color signal C similarly Means, and the predetermined pixel l in the first color signal A.
Average value calculating means for calculating a pixel value A (l, m) of the first color signal A in l rows and m columns from an average value of adjacent pixels in the upper, lower, left, and right positions at the position of the row m. Based on the output of the means, the output is selected from the output of the horizontal signal calculation means, the output of the vertical signal calculation means, or the output from the average value calculation means, and the first color signal A in the predetermined pixel l row m column is selected. By obtaining a pixel value A (l, m) and obtaining a first color signal of the number of pixels in the image sensor, a false color,
An image with reduced false contours can be obtained.

According to the imaging apparatus of the present invention,
When the means for calculating the first color signal determines that the output of the edge determination means does not detect an edge component at the position of predetermined pixel l row and m column, the output of the average value calculation means is selected and If it is determined that there is a correlation, the output of the vertical direction signal calculation means is selected.If it is determined that there is a correlation in the horizontal direction, the output of the horizontal direction signal calculation means is selected. By obtaining the first color signal, an image with reduced false colors and false contours can be obtained.

According to the imaging apparatus of the present invention,
The means for calculating and restoring the second and third color signals is provided at the position (l = 2i, m = 2j or l = 2i) of the predetermined pixel l row and m column.
+1 and m = 2j + 1), the output A from the means for calculating the first color signal has a value A1hlpf (l, m) through a horizontal low-pass filter and a value A1vlpf (l, m) through a vertical low-pass filter. l, m), the value B1hlpf (l, m) of the second color signal B via a horizontal low-pass filter and the third color signal C
To the value C1v through a low-pass filter in the vertical direction
lpf (l, m) (or the value B1vlpf through the low-pass filter in the vertical direction with respect to the second color signal B)
(L, m) and a value C1hlpf (l, m)) of the third color signal C through a horizontal low-pass filter are calculated, and A1hlpf (l, m) and B1hlpf (l, m) are calculated.
m) (or the ratio with C1hlpf (l, m))
And A1vlpf (l, m) and C1vlpf (l, m)
(Or the ratio to B1vlpf (l, m)) and
From the pixel value A (l, m) at the predetermined pixel l row m column in the output A from the means for calculating the first color signal, the second color signal B and the third color signal C l row m columns Pixel value B at
Let (l, m) and C (l, m) be B (l, m) = A (l,
m) × ｛B1hlpf (l, m) / A1hlpf (l,
m)｝, C (l, m) = A (l, m) × ｛C1vlpf
(L, m) / A1vlpf (l, m)｝, (or B
(L, m) = A (l, m) × ｛B1vlpf (l, m)
/ A1vlpf (l, m)｝, C (l, m) = A (l,
m) × ΔC1hlpf (1, m) / A1hlpf
(L, m)｝) and a predetermined pixel x row y different from the position of the l row and m column
Column position (x = 2i + 1, y = 2j or x = 2i, y
= 2j + 1), the value A2hlpf (x, y) through a horizontal low-pass filter for the output A from the means for calculating the first color signal, the second color at the output from the signal calculating means A value B2hlpf (x, y) is calculated for the signal B through a horizontal low-pass filter, and A
Based on the ratio of 2hlpf (x, y) to B2hlpf (x, y) and the pixel value A (x, y) at the pixel x row y column at the output A from the first calculating means, x row y column The second color signal B (x, y) at the position is expressed as B (x, y) = A (x,
y) × ｛B2hlpf (x, y) / A2hlpf (x,
y) A horizontal signal calculating means for calculating the third color signal C in the same manner as for the third color signal C, and a low-pass filter in the vertical direction for the output A from the means for calculating the first color signal. A2vlpf (x, y) via the low-pass filter in the vertical direction with respect to the second color signal B at the output from the signal calculation means.
(X, y) is calculated, and A2vlpf (x, y) and B2v
By the ratio of lpf (x, y) and the pixel value A (x, y) at the pixel x row and y column at the output A from the first calculating means, x
The second color signal B (x, y) at the position of the row y column is represented by B
(X, y) = A (x, y) × ｛B2vlpf (x, y)
/ A2vlpf (x, y)}, and a vertical signal calculating means for calculating the C signal for the third color signal C in the same manner.
Average value calculating means for calculating an average value of obliquely adjacent pixels at a position of a predetermined pixel x row and y column in the third color signal, wherein the horizontal direction signal calculation is performed based on an output of the edge determining means. Means, the vertical direction signal calculation means, and the average value calculation means, to obtain the second and third color signals at the predetermined pixel x row and y column to obtain the second and third color signals. By obtaining the second and third color signals, an image with reduced false colors and false contours can be obtained.

Further, according to the image pickup apparatus of the present invention, the means for calculating the second and third color signals detects an edge component at a position of a predetermined pixel x row and y column when the output of the edge determination means is detected. If it is determined not to be performed, the output of the average value calculation means is selected.If it is determined that there is a correlation in the vertical direction, the output of the vertical signal calculation means is selected, and it is determined that there is a correlation in the horizontal direction. By selecting the output of the horizontal direction signal calculating means and obtaining the second and third color signals of the number of pixels in the image sensor, an image with reduced false colors and false contours can be obtained.

[Brief description of the drawings]

FIG. 1 is a diagram showing an example of an arrangement of color filters of an image sensor according to Embodiment 1 of the present invention.

FIG. 2 is a block diagram illustrating an example of a configuration of an imaging device according to Embodiment 1 of the present invention;

FIG. 3 is a block diagram showing an example of a configuration of a first edge determination unit 5 and a G component restoration unit 6 in the imaging device according to the first embodiment of the present invention.

FIG. 4 is a block diagram illustrating an example of a configuration of a second edge determination unit 8 and a second RB component restoration unit 9 in the imaging device according to the first embodiment of the present invention.

FIG. 5 is a diagram showing pixels of a G signal for describing an operation of restoring the G signal in the imaging device according to the first embodiment of the present invention;

FIG. 6 is a diagram illustrating pixels of an R signal and a B signal for describing an operation of restoring a G signal in the imaging device according to the first embodiment of the present invention;

FIG. 7 is a diagram showing pixels of R and B signals for explaining the operation of the first RB component restoration device 7 in the imaging device according to the first embodiment of the present invention.

FIG. 8 is a diagram illustrating pixels of R and B signals for explaining the operation of the second edge determination unit 8 and the second RB component restoration unit 9 in the imaging device according to the first embodiment of the present invention.

FIG. 9 is a diagram showing a false color level at a horizontal resolution in a simulation image processed by the imaging device according to the first embodiment of the present invention and a conventional device.

FIG. 10 is a diagram illustrating a false color level at a vertical resolution in a simulation image processed by the imaging device according to the first embodiment of the present invention and a conventional device.

FIG. 11 is a diagram illustrating a level of a luminance signal at a horizontal resolution in a simulation image processed by the imaging device according to the first embodiment of the present invention;

FIG. 12 is a diagram illustrating a level of a luminance signal at a horizontal resolution in a simulation image processed by a conventional device.

FIG. 13 is a diagram showing a level of a luminance signal at a vertical resolution in a simulation image processed by the imaging device according to the first embodiment of the present invention;

FIG. 14 is a diagram showing a level of a luminance signal at a vertical resolution in a simulation image processed by a conventional device.

FIG. 15 is a diagram showing an example of another color filter array of the imaging device according to the first embodiment of the present invention.

FIG. 16 is a diagram showing an example of another color filter array of the imaging device according to the first embodiment of the present invention.

FIG. 17 is a block diagram illustrating an example of a configuration of an imaging device according to a second embodiment of the present invention.

FIG. 18 is a diagram illustrating pixels of a G signal for describing an operation of a G component edge determination unit 41 in the imaging device according to the second embodiment of the present invention.

FIG. 19 is a diagram illustrating an example of a color filter array of an imaging device in a conventional imaging device.

FIG. 20 is a block diagram illustrating an example of a configuration of a conventional imaging device.

FIG. 21 is a diagram illustrating pixels of respective signals for explaining the operation of the conventional imaging apparatus.

FIG. 22 is a block diagram illustrating an example of another configuration of a conventional imaging device.

[Explanation of symbols]

REFERENCE SIGNS LIST 1 imaging device, 2 A / D converter, 3 frame memory, 4 separation means, 5 first edge determination means, 6
G component restoring means, 7 first RB component restoring means, 8 second edge determining means, 9 second RB component restoring means, 1
1, 30, 42 horizontal edge detecting means, 12, 31,
43 vertical edge detecting means, 13 determining means, 14
Average value calculation means, 15 to 17 horizontal low-pass filter, 18, 19 calculation means, 20 to 22 vertical low-pass filter, 23, 24 calculation means, 25 switching means, 32 determination means, 33 R average value calculation means, 34
R horizontal operation means, 35 R vertical operation means, 36
R switching means, 37 B average value calculating means, 38 B
Horizontal computing means, 39 B Vertical computing means, 40
B switching means, 41 G component edge determining means, 44 determining means.

 ──────────────────────────────────────────────────続 き Continued from the front page (72) Inventor Tetsuya Kuno 2-3-2 Marunouchi, Chiyoda-ku, Tokyo Mitsubishi Electric Corporation

Claims (7)

[Claims] 1. A first color filter having a spectral sensitivity characteristic for a first color signal and a second color filter having a spectral sensitivity characteristic for a second color signal in upper and lower four pixels in two rows and two columns. Are arranged in the first vertical line, and the second vertical line has a spectral sensitivity characteristic for the third color signal in the same column as the pixel position where the first color filter in the first vertical line is arranged. Three color filters are arranged, and a first color filter having a spectral sensitivity characteristic for the first color signal is arranged in the same column as the second color filter. And an image pickup apparatus having an image pickup element repeatedly arranged in a horizontal direction, wherein an edge component at a predetermined pixel position is determined based on a peripheral pixel signal of the first color signal at a predetermined pixel position by the first color filter. Edge judgment means And the first, second, and third color signals read by the first, second, and third color filters based on the output of the edge determination means at the predetermined position in the first color signal. Based on the output of the first calculating means and the second and third signals from the color filter based on the output of the edge determining means.
Calculating the second and third color signals from the color signals
An image pickup apparatus, comprising: calculating means for obtaining first, second, and third color signals of the number of pixels in the image sensor. 2. A horizontal edge detecting means for calculating an absolute value of a difference between left and right adjacent pixels at a predetermined pixel position of the first color signal and detecting a horizontal edge component, A vertical edge detecting means for calculating an absolute value of a difference between upper and lower pixels at a predetermined pixel position of the first color signal to detect a vertical edge component; and the horizontal edge detecting means and the vertical edge detecting means. 2. The imaging apparatus according to claim 1, further comprising: a determination unit configured to determine a horizontal or vertical edge component of the predetermined pixel based on the output of the imaging device. 3. When the output from the horizontal edge detecting means or the output from the vertical edge detecting means is larger than a predetermined value, the determining means in the edge determining means determines whether or not the pixel is a peripheral pixel of the predetermined pixel. If an edge component is detected and the output from the horizontal edge detecting means is larger than the output of the vertical edge detecting means, there is a vertical correlation, and the output from the horizontal edge detecting means is the vertical edge. When the output is smaller than the output of the detecting means, it is determined that there is a correlation in the horizontal direction. When both the outputs from the horizontal edge detecting means and the vertical edge detecting means are smaller than a predetermined value, no edge component is detected. The imaging apparatus according to claim 1, wherein the imaging apparatus is determined to be: 4. The method according to claim 1, wherein the first calculating means calculates a second color signal B
At a given pixel l row m column B (l, m)
The value Ahlpf (l, 1) of each of the first color signal A and the second color signal B via a horizontal low-pass filter
m) and Bhlpf (l, m) are calculated, and Ahlpf (l, m) which is an output signal from the horizontal low-pass filter is calculated.
m) and Bhlpf (l, m) and the pixel value B (l, m) at the pixel position, the pixel value A (l, m) of the first color signal A in l rows and m columns is represented by A (L, m) = B (l,
m) × ｛Ahlpf (l, m) / Bhlpf (l,
m), the horizontal direction signal calculating means for calculating the pixel value of the first color signal A at another pixel position of the third color signal C in the same manner, and the predetermined pixel l row m column B At the position (l, m), the values Avlpf (l, m) and Bvlpf (l, l) of the first color signal A and the second color signal B are respectively passed through the low-pass filter in the vertical direction.
m), and output signals from the vertical low-pass filter, Avlpf (l, m) and Bvlpf (l,
m) and the pixel value B (l, m) at the pixel position, the pixel value A (l, l) in the first color signal A in l rows and m columns
m) is given by A (l, m) = B (l, m) × ｛Avlpf
(L, m) / Bvlpf (l, m)}, and the first pixel is similarly set at another pixel position where the third color signal C exists.
A vertical direction signal calculating means for calculating a pixel value in the color signal A of the first color signal A; Average value calculating means for calculating a pixel value A (l, m) in one color signal A, and based on the output of the edge determining means, the output of the horizontal signal calculating means or the output of the vertical signal calculating means, Alternatively, the pixel value A (l, m) of the first color signal A in the predetermined pixel l row m column is selected from the output from the average value calculating means, and the first color of the number of pixels in the image sensor is obtained. 2. A signal is obtained.
An imaging device according to any one of the preceding claims. 5. When the first calculating means determines that the output of the edge determining means does not detect an edge component at a position of a predetermined pixel at l rows and m columns, the output of the average value calculating means is selected. If it is determined that there is a correlation in the vertical direction, the output of the vertical signal calculation means is selected.If it is determined that there is a correlation in the horizontal direction, the output of the horizontal signal calculation means is selected. The imaging device according to claim 1, wherein a number of first color signals are obtained. 6. The method according to claim 1, wherein the second calculating means calculates a predetermined pixel of l rows m
At the position of the column, the value A1hl through the horizontal low-pass filter is applied to the output A from the first calculating means.
pf (l, m) and a value A1vlpf (l, m) through a vertical low-pass filter are calculated, and a value B1hlpf through a horizontal low-pass filter is calculated for the second color signal B.
The value C1vlpf (l, m) (or, through a low-pass filter in the vertical direction) for (l, m) and the third color signal C (or
A value B1vlpf (l, m) through a low-pass filter in the vertical direction for the second color signal B and a value C1hlpf through a low-pass filter in the horizontal direction for the third color signal C
(L, m)) and A1hlpf (l, m) and B1
1hlpf (l, m) (or C1hlpf
(L, m)), A1vlpf (l, m) and C1
vlpf (l, m) (or B1vlpf)
(Ratio to (l, m)) and the pixel value A (l, m) at l rows and m columns of the predetermined pixel in the output A from the first calculating means, the 1st row and m columns of the second color signal The pixel values B (l, m) and C (l, m) of B and the third color signal C are represented by B (l, m)
= A (l, m) × ｛B1hlpf (l, m) / A1h
lpf (l, m)｝, C (l, m) = A (l, m) ×
｛C1vlpf (l, m) / A1vlpf (l,
m)｝, (or B (l, m) = A (l, m) × ｛B
1vlpf (l, m) / A1vlpf (l, m)｝, C
(L, m) = A (l, m) × ｛C1hlpf (l, m)
/ A1hlpf (l, m)｝), and the output A from the first calculating means at a position of a predetermined pixel x row and y column different from the position of l row and m column. On the other hand, a value A2hlpf (x, y) through a horizontal low-pass filter, and a value B2hlpf (x, y) through a horizontal low-pass filter with respect to the second color signal B at the output from the signal calculating means. , And A
Based on the ratio of 2hlpf (x, y) to B2hlpf (x, y) and the pixel value A (x, y) at the pixel x row y column at the output A from the first calculating means, x row y column The second color signal B (x, y) at the position is expressed as B (x, y) = A (x,
y) × ｛B2hlpf (x, y) / A2hlpf (x,
y) A horizontal signal calculating means for calculating the third color signal C in the same manner as for the third color signal C, and a value obtained through a vertical low-pass filter for the output A from the first calculating means. A2vlpf (x, y) calculates a value B2vlpf (x, y) through a low-pass filter in the vertical direction with respect to the second color signal B at the output from the signal calculation means. B2vlpf (x, y)
Of the pixel x row y at the output A from the first calculating means.
The second color signal B (x, y) at the position of x row and y column is obtained by B (x, y) = A by the pixel value A (x, y) in the column.
(X, y) × ｛B2vlpf (x, y) / A2vlpf
(X, y)}, the vertical color signal calculating means for calculating the C signal in the same manner also for the third color signal C, and the second and third color signals in the output from the signal calculating means. Average value calculating means for calculating an average value of obliquely adjacent pixels at a position of a predetermined pixel x row and y column, and the horizontal signal calculating means and the vertical signal calculating means based on an output of the edge determining means. , The second and third color signals at the predetermined pixel x row and y column are selected from the respective outputs from the average value calculation means, and the second and third color signals of the number of pixels in the image sensor are obtained. The imaging device according to claim 1, wherein the imaging device is obtained. 7. When the second calculating means determines that the output of the edge determining means does not detect an edge component at the position of a predetermined pixel x row and y column, the output of the average value calculating means is selected. If it is determined that there is a correlation in the vertical direction, the output of the vertical signal calculation means is selected.If it is determined that there is a correlation in the horizontal direction, the output of the horizontal signal calculation means is selected. Second and third of numbers
The imaging device according to claim 1, wherein the color signal is obtained as follows.