JP2007184905A - Image signal processing device and method of image signal processing - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily detect a correlation direction of respective pixels about a solid-state imaging device having complicated pixel array. <P>SOLUTION: The solid-state imaging device has m×n pixels on which A, B, C and D color filters are overlapped on respective m×n photoelectric converters. The solid-state imaging device defines each pixel as a center pixel and arranges a pixel on which a filter of the same color is overlapped in a k×k pixel area with each center pixel as a center according to either of a first or second array-pattern. The solid-state image device sends image raw data comprising pixel data of each pixel to a direction detection block 160. The direction detection block 160 detects a direction in which a center pixel is relatively similar to a color signal of peripheral pixels of the same color on which the filter of the same color as that of the center pixel is overlapped as a correlation direction in the k×k pixel area. The position of each peripheral pixel of the same color of each center pixel is specified by an array-pattern in the k×k pixel area. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image signal processing apparatus for processing an image signal obtained by a single-plate color imaging method.

  2. Description of the Related Art Conventionally, various color imaging methods are known in an imaging apparatus, for example, a three-plate type, a plane sequential type, a single-plate type, and the like are known. In particular, a single-plate type color imaging method has a simple configuration and is 3 Since the primary color signal can be taken out, in recent years, it has been widely adopted in home video cameras and electronic endoscopes.

  In the single-plate color imaging system, a color filter array is provided on the light receiving surface of the CCD, and the color filter array has colors of magenta (Mg), green (G), cyan (Cy), and yellow (Ye). The filters are arranged in a mosaic pattern. In the CCD, color signals of respective colors (magenta, green, cyan, and yellow) are generated for each pixel corresponding to these filters.

  Each of the generated color signals is known to be read by various reading methods. For example, an independent reading method in which all pixels are read independently is known. In this readout method, after the color signals of all the pixels are read out independently, pixel interpolation is performed using the color signals to calculate color data.

  Patent Document 1 describes a pixel interpolation processing method for color signals. According to this method, a matrix for each color (magenta, green, cyan, and yellow) in a 4-by-4 pixel matrix. Pixels of the same color located at a predetermined position with respect to the center position of are selected, and the output values of the selected pixels are weighted and averaged to generate color data at the center position.

  Here, the pixel arrangement of the CCD in Patent Document 1 is such that pixels of each color (magenta (Mg), green (G), cyan (Cy), and yellow (Ye)) are respectively in the same row direction or the same column direction. Are alternately arranged, that is, the pixels of each color are arranged in the same pattern. Therefore, it is relatively easy to select pixels of the same color at the time of pixel interpolation processing. In Patent Document 1, a pixel at a position rotated every 90 ° with respect to the center position is selected.

By the way, the pixel arrangement of the CCD is not necessarily that all color pixels are arranged in the same pattern as described in Patent Document 1, for example, so-called color difference sequential arrangement as described in Patent Document 2. May be arranged by. Further, as described in Patent Document 3, the CCD pixel array may be arranged in a so-called Bayer array.
Japanese Patent Laid-Open No. 10-234047 Japanese Examined Patent Publication No. 6-28450 Japanese Patent Laid-Open No. 51-112228

  However, in the color difference sequential arrangement described in Patent Document 2, for example, pixels of Ye and Cy are not alternately arranged in the same column in the column direction as shown in FIG. It is difficult to select uniformly. Also, in the Bayer arrangement described in Patent Document 3, since the arrangement patterns of Y and C1 (or C2) are different as shown in FIG. 1, pixels of the same color are selected uniformly as in the color difference sequential arrangement. It is difficult to do. That is, when all the pixels are read out independently for a CCD having these pixel arrays, it is difficult to perform pixel interpolation processing by the method described in Patent Document 1.

  Accordingly, the present invention has been made in view of such problems, and in the case where the imaging method is a single plate type and color signals of all the pixels of the CCD are read out independently, various pixel arrangements are provided. An object of the present invention is to provide an image signal processing apparatus capable of pixel interpolation corresponding to the above.

  In the present invention, in the first and second image signal processing devices, pixels that are configured by overlapping filters of at least first to third colors that are different from each other on each photoelectric conversion element are regularly arranged. A solid-state imaging device configured by arranging each pixel as a central pixel, and a pixel in which filters of the same color are overlapped in a k × k pixel region centered on each central pixel is at least a first and a second pixel The present invention relates to an image signal processing apparatus capable of processing a color signal output from each pixel of a solid-state imaging device arranged according to any of a plurality of pixel arrangement patterns including two arrangement patterns. The first image signal processing device assumes pixels in which filters of the same color are overlapped in each k × k pixel region according to each of one or more pixel arrangement patterns among a plurality of pixel arrangement patterns, A pixel interpolating unit that interpolates color signals of pixels assumed to be overlapped with filters of the same color and generates one or more interpolation signals for each central pixel, and a pixel arrangement pattern in a k × k pixel region A pixel discriminating means for specifying one specific arrangement pattern selected from a plurality of pixel arrangement patterns, and one interpolation signal generated according to the specific arrangement pattern from one or more interpolation signals It is characterized in that it is determined as an adopted interpolation signal.

  The pixel interpolation means interpolates the pixels by weighted or arithmetically averaging the color signals of the pixels with the same color filter in the k × k pixel region, thereby generating an interpolated signal for each color related to the center pixel. It is preferable to do. For example, the first and second array patterns are formed by arranging a plurality of basic matrix arrays having the same pixel array in the row direction and the column direction. In this case, it is preferable that the pixel discriminating unit specifies the arrangement pattern depending on where the central pixel is arranged in the basic matrix.

  The image signal processing apparatus according to the present invention detects a direction in which color signals of a central pixel and peripheral pixels around the central pixel are relatively approximated as a correlation direction of each central pixel in a k × k pixel region. May be further provided. In this case, the pixel interpolation means performs pixel interpolation using the color signals of the pixels located along the correlation direction detected by the direction detection unit.

  In the k × k pixel region, the direction detecting means determines the direction in which the color signals of the central pixels and the peripheral pixels of the same color on which the filters of the same color as the central pixels are superimposed relatively approximate the correlation direction of the central pixels. It is preferable to detect as

  It is preferable that the pixel interpolation means includes at least first and second pixel interpolation units. In this case, the first pixel interpolation unit assumes pixels in which filters of the same color are overlaid in the k × k pixel region according to the first arrangement, and pixels assumed to have been overlaid with filters of the same color. Each of the color signals is subjected to pixel interpolation, and a first interpolation signal is generated for each color that is determined to be the same color. On the other hand, in the k × k pixel region, the second pixel interpolation unit assumes pixels in which filters of the same color are overlaid in accordance with the second arrangement, and sets pixels of the pixels assumed to have been overlaid of filters of the same color. Each of the color signals is subjected to pixel interpolation, and a second interpolation signal for each color determined to be the same color is generated. Then, when the pixel determining unit specifies that the array pattern in the k × k pixel region centered on the center pixel is the first array pattern, the pixel determining unit outputs the first interpolation signal generated by the first pixel interpolation unit. When the interpolation signal for the center pixel is specified and the second array pattern is specified, the second interpolation signal generated by the second pixel interpolation unit is specified as the interpolation signal for the center pixel.

  When the pixel interpolation unit includes at least the first and second pixel interpolation units, the image signal processing apparatus according to the present invention preferably includes the direction detection unit including at least the first and second direction detection units. In this case, the first direction detection unit determines, as the first specific peripheral pixel, a pixel that is assumed to have a filter of the same color as that of the central pixel according to the first arrangement pattern, and the first specific peripheral pixel with respect to the central pixel. A direction in which pixel color signals are relatively approximated is detected as a first temporary correlation direction. The second direction detection unit determines, as a second specific peripheral pixel, a pixel that is assumed to have a filter of the same color as that of the central pixel according to the second arrangement pattern, and the color of the second specific peripheral pixel with respect to the central pixel The direction in which the signal is relatively approximated is detected as the second temporary correlation direction. The first pixel interpolation unit performs pixel interpolation using the color signals of pixels located along the first temporary correlation direction detected by the first direction detection unit. Further, the second pixel interpolation unit performs pixel interpolation using the color signal of the pixel located along the second temporary correlation direction detected by the second direction detection unit.

  For example, the first and second array patterns are formed by arranging a plurality of basic matrix arrays having the same pixel array in the row direction and the column direction. In this case, the pixel discriminating unit specifies one of the first and second interpolation signals in the previous period as the adopted interpolation signal of the central pixel depending on where the central pixel is arranged in the basic matrix, The color of the employed interpolation signal is specified.

  When the pixel interpolation means includes at least first and second pixel interpolation units, the pixel determination unit also includes a first pixel determination unit to which the first interpolation signal is input and a second pixel to which the second interpolation signal is input. It is better to have a discriminator. In this case, the first and second interpolation signals related to the same center pixel are input in synchronization with the first and second pixel discriminating units, respectively. Then, a pulse signal further synchronized with each of the first and second interpolation signals input in synchronization described above is input to one of the first and second pixel discriminating units, and one of the pulse signals input. The pixel discriminating unit employs one interpolation signal input to the one pixel discriminating unit in synchronization with the pulse signal as an adopted interpolation signal for the center pixel.

  The pulse signal is input to the first pixel determination unit when the array pattern in the k × k pixel region centered on the center pixel related to the first and second interpolation signals synchronized with the signal is the first array pattern. On the other hand, in the case of the second arrangement pattern, it is preferably input to the second pixel discrimination. The pulse signal differs depending on the position of the central pixel related to the first and second interpolation signals synchronized with the signal in the basic matrix, and each color of the employed interpolation signal is specified according to the pulse signal. It is preferred that

  In the solid-state imaging device, the k × k pixel region centered on the center pixel located in the odd row is arranged in the first arrangement pattern, and the k × k pixel region centered on the center pixel located in the even row is the first In the case of being arranged in two arrangement patterns, the pixel discriminating unit sets the first interpolation signal as the adopted interpolation signal for the center pixel when the center pixel is located in the odd row, and the second interpolation when located in the even row. It is preferable to set the signal as an adopted interpolation signal for the central pixel.

  The first and second array patterns are formed by arranging a plurality of basic matrix arrays having the same pixel array in the row direction and the column direction. It is preferable to specify each color of each adopted interpolation signal depending on which position it is arranged. The pixel interpolating means assumes each pixel in which filters of the same color in the k × k pixel region are overlapped in the k × k pixel area according to the specific arrangement pattern, and calculates the color signals of the pixels assumed to have the filters of the same color overlapped. It is preferable that pixel interpolation is performed to generate one interpolation signal for each center pixel, and the generated interpolation signal is determined as an interpolation signal for the center pixel.

  According to the second image signal processing device of the present invention, a filter having the same color as each central pixel in each k × k pixel region is overlapped according to each of one or more pixel arrangement patterns among a plurality of pixel arrangement patterns. A correlation direction detecting means for detecting at least one provisional correlation direction using a color signal of surrounding pixels assumed to be the same color, and a pixel arrangement pattern in a k × k pixel region is selected from a plurality of pixel arrangement patterns And a pixel discriminating means for specifying one specific arrangement pattern, wherein one temporary correlation direction generated according to the specific arrangement pattern is determined as a correlation direction of the central pixel from one or more temporary correlation directions. And

  In the third image signal processing apparatus according to the present invention, pixels each composed of at least first to third color filters that are different from each other are regularly arranged on each photoelectric conversion element. , A solid-state imaging device configured in an m-row n-column matrix, each pixel being a central pixel, and a pixel in which filters of the same color are superimposed in a k × k pixel region centered on each central pixel is at least In the image signal processing apparatus capable of processing the color signal output from each pixel of the solid-state imaging device arranged according to any of the plurality of pixel arrangement patterns including the first and second arrangement patterns, the m rows and n columns matrix is: A plurality of basic matrices having the same pixel arrangement are arranged in the column direction and the row direction, and the pixel arrangement pattern in the k × k pixel region is the same as that in the k × k pixel region. Pixels depending is disposed at any position in the basic matrix, characterized in that it is specific to one particular arrangement pattern selected from a plurality of pixel array pattern.

  In the first and second image signal processing methods according to the present invention, pixels configured by overlapping filters of at least first to third colors, which are different from each other, on each photoelectric conversion element are regularly arranged. A solid-state imaging device configured by arranging each pixel as a central pixel, and a pixel in which filters of the same color are overlapped in a k × k pixel region centered on each central pixel is at least a first and a second pixel The present invention relates to an image signal processing method for processing a color signal output from each pixel of a solid-state imaging device arranged according to any of a plurality of pixel arrangement patterns including two arrangement patterns. The first image signal processing method assumes pixels in which filters of the same color are overlapped in each k × k pixel region according to each of one or more pixel arrangement patterns among a plurality of pixel arrangement patterns, A pixel interpolation step for interpolating color signals of pixels assumed to have these same color filters overlapped to generate one or more interpolation signals for each central pixel, and a pixel arrangement pattern in a k × k pixel region A pixel determining step that specifies one specific arrangement pattern selected from a plurality of pixel arrangement patterns, and one interpolation signal generated according to the specific arrangement pattern from one or more interpolation signals, It is characterized in that it is determined as an adopted interpolation signal.

  In the second image signal processing method, it is assumed that filters of the same color as each central pixel in each k × k pixel region are overlaid according to each of one or more pixel arrangement patterns among a plurality of pixel arrangement patterns. A correlation direction detection step for detecting at least one provisional correlation direction using color signals of neighboring pixels of the same color, and one specification selected from a plurality of pixel arrangement patterns as a pixel arrangement pattern in a k × k pixel region And a pixel discrimination step that specifies an array pattern, wherein one temporary correlation direction generated according to the specific array pattern is determined as a correlation direction of the central pixel from one or more temporary correlation directions.

  In the third image signal processing method according to the present invention, pixels each composed of at least first to third color filters that are different from each other are regularly arranged on each photoelectric conversion element. , A solid-state imaging device configured in an m-row n-column matrix, each pixel being a central pixel, and a pixel in which filters of the same color are superimposed in a k × k pixel region centered on each central pixel is at least In an image signal processing method for processing a color signal output from each pixel of a solid-state imaging device arranged according to any one of a plurality of pixel arrangement patterns including a first and second arrangement pattern, an m-row n-column matrix includes pixels A plurality of basic matrices having the same arrangement are arranged in the column direction and the row direction, and the arrangement pattern in the k × k pixel region is represented by the center pixel of the k × k pixel region. Depending is disposed at any position in the basic matrix, and identifies a single specific sequence pattern selected from a plurality of array patterns.

  In the present invention, since the same color pixels on which the filters of the same color are superimposed can be specified according to the array pattern, the interpolation signal of each pixel can be easily calculated. In addition, since the same-color peripheral pixels used for detecting the correlation direction of each pixel are specified according to the arrangement pattern in the k × k pixel region centered on each pixel, the solid-state imaging device having a complicated pixel arrangement Also, the correlation direction of each pixel can be easily detected. Furthermore, in the present invention, since the pixel interpolation or the correlation direction is appropriately detected according to the pixel arrangement pattern, generation of false colors can be prevented.

  Embodiments according to the present invention will be described with reference to the drawings. In the following description, the case where the imaging device is an electronic endoscope device will be described, but the present invention may be applied to other imaging devices such as a digital camera and a video camera.

  FIG. 1 is a block diagram of an electronic endoscope apparatus according to an embodiment of the present invention. The electronic endoscope apparatus includes a flexible scope (electronic scope) 10, an electronic endoscope processor 100 detachably attached to the scope 10, and a monitor device (not shown) connected to the processor 100.

  A light guide member 12 made of an optical fiber bundle is inserted into the scope 10 up to the flexible distal end portion 20a, and the proximal end side of the light guide member 12 is provided in the processor 100 when the scope 10 is attached to the processor 100. The optical source 102 is optically connected. The light source 102 is a white light source lamp, and the white light emitted from the light source 102 is guided to the flexible tip 20a by the light guide member 12, and the subject (for example, the front of the flexible tip 20a) (for example, The inner wall X) of the digestive organ is irradiated.

  An imaging sensor 14 formed of a so-called single plate type solid-state imaging device (CCD) is provided at the flexible tip 20 a, and this imaging sensor 14 is combined with the objective lens 16. The white light applied to the subject is reflected by the subject and imaged on the light receiving surface of the image sensor 14 by the objective lens 16. The image sensor 14 photoelectrically converts an optical subject image into an analog image signal for one frame, and the analog image signal is read from the image sensor 14 such that each pixel is read independently for each pixel.

  The analog image signal read from the image sensor 14 is sent to the A / D converter 122, where it is converted into an 8-bit digital image signal (color signal), for example. The converted digital image signal is written in the frame memory 124 and temporarily stored as a digital image signal for one frame. The temporarily stored digital image signal (color signal) is input to the signal processing block 125 and processed in the signal processing block 125. In the signal processing block 125, pixel interpolation processing, color matrix processing, white balance processing, contour enhancement processing, and the like are performed for each pixel of one frame.

  The pixel interpolation signal generated by the signal processing block 125 is converted into a luminance and color difference signal, and then sent to a monitor device (not shown), where it is displayed as a moving image on the monitor device. The processor 100 is provided with a control circuit 150, and the entire operation of the processor 100 is controlled by the control circuit 150.

  The scope data storage unit 25 in the scope 10 stores pixel arrangement information that is information relating to the pixel arrangement pattern of the imaging sensor provided in the scope 10. The pixel array information is read by the scope data detection unit 130 and input to the control circuit 150. The pixel arrangement information is changed according to the type of the scope 10 connected to the processor 100, and the pixel arrangement information input to the control circuit 150 is also changed.

  FIG. 2 is a diagram illustrating an example of the light receiving surface of the image sensor 14. Hereinafter, a case where the pixel array is the pixel array illustrated in FIG. 2 will be described. The light receiving surface of the image sensor 14 is configured by pixels arranged in a matrix of m rows and n columns, and each pixel is configured by overlapping a color filter on a photoelectric conversion element. The imaging sensor 14 is a solid-state imaging device used in a single-plate imaging system, and a color filter of any one of four colors (for example, Cy, Mg, Ye, G) is overlaid on each pixel. In FIG. 2, A, B, C, and D indicate Cy, Mg, Ye, and G, respectively.

The analog image signal obtained by the solid-state image sensor is read out independently for each pixel, and the digital image signal stored in the frame memory 124 is m rows and n similar to the pixel array of the image sensor 14 as shown in FIG. It consists of pixel data forming a column matrix. Therefore, the frame memory 124 stores pixel data (image raw data) P _{11 to} P _{mn that} are digital image signals for one frame.

  As shown in FIG. 2, the pixels on which the four color filters are overlaid are regularly arranged, and the basic matrix array α of 4 rows and 2 columns having the same pixel arrangement has a row direction and a column direction. Are arranged in a matrix of m rows and n columns. The basic matrix array α is configured by arranging two pixels in two rows and two columns in which four different color filters (A, B, C, and D) are stacked in the row direction.

  Here, in the basic matrix array α, in the first and third rows (odd rows), AB pixels (that is, pixels in which A and B color filters are superimposed) are alternately arranged in the column direction, and 2 and In the fourth row (even rows), CD pixels are alternately arranged in the column direction. That is, in the matrix of m rows and n columns, AB pixels are arranged in odd rows and CD pixels are arranged in even rows.

  CD pixels that are alternately arranged in even rows are alternately arranged in the same phase. That is, C pixels are always arranged in odd columns, while D pixels are always arranged in even columns. On the other hand, in the odd-numbered rows, the AB pixels arranged alternately are arranged so that the phase of the pixels in the third row is shifted by one pixel from the phase of the pixels in the first row. That is, A and B are alternately arranged in the column direction.

  With reference to FIGS. 2, 4, and 5, the arrangement pattern relating to the color of the pixel will be described. By arranging the pixels A to D as described above, in this embodiment, when each pixel is a central pixel Pm and a peripheral pixel having the same color as the central pixel Pm is the same color peripheral pixel Ps, the central pixel The pixel array pattern of the 5 × 5 pixel region Fm centered on Pm is either the first array pattern shown in FIG. 4 or the second array pattern shown in FIG.

  In other words, the A and B pixels are arranged every other pixel in the row direction, but are arranged every other pixel while being shifted left and right by one pixel in the column direction. Then, the pixel arrangement pattern of the central pixel Pm and the peripheral pixels Ps of the same color as the central pixel Pm is the first arrangement pattern (see FIG. 4). When A or B is the central pixel, the pixel arrangement pattern of the same-color peripheral pixels Ps is always the first arrangement pattern.

  In addition, since the C and D pixels are arranged every other pixel in both the row direction and the column direction, if C or D is the central pixel Pm, the arrangement pattern of the peripheral pixels Ps of the same color as the central pixel Pm is The second arrangement pattern (see FIG. 5). When C and D are central pixels, the arrangement pattern of the peripheral pixels Ps of the same color is always the second arrangement pattern.

Therefore, when the center pixel Pm is A or B, if the pixel data of the center pixel Pm is P _ij as shown in FIG. 6, the pixel data of the same color peripheral pixel Ps in the 5 × 5 pixel region Fm is P _{(i− 2) (j-1)} , P _{(i-2) (j + 1)} , _{Pi (j-2)} , _{Pi (j + 2)} , P _{(i + 2) (j-1)} , P _{(i + 2) (j + 1)} Become. If the central pixel Pm is C or D, and pixel data of the central pixel Pm is P _ij as shown in FIG. 7, the pixel data of the same color peripheral pixel Ps in the 5 × 5 pixel region Fm is P _{(i -2) (j-2)} , P _{(i-2) j} , P _{(i-2) (j + 2)} , _{Pi (j-2)} , _{Pi (j + 2)} , P _{(i + 2) (j-2)} , P _{(i + 2) j} and P _{(i + 2) (j + 2)} .

  Next, an arrangement pattern relating to the colors of the pixels other than the central pixel Pm and the peripheral pixels Ps of the same color in the 5 × 5 pixel region Fm will be described with reference to FIGS. FIG. 8 shows an arrangement pattern in the 5 × 5 pixel area Fm related to colors in the first arrangement pattern. FIG. 9 shows an arrangement pattern in the 5 × 5 pixel area Fm related to colors in the second arrangement pattern.

  When the pixel arrangement pattern is the first arrangement pattern, as shown in FIG. 8, the arrangement patterns of the pixels of each color other than the central pixel are also the same. That is, when the arrangement pattern of the same-color peripheral pixels Ps is the first arrangement pattern, if the pixels of the same-color peripheral pixels Ps and the central pixels Pm are M, and the other color pixels are N, Q, and R, the first arrangement pattern The arrangement of the pixels N, Q, and R in FIG. 8 is as shown in FIG. In this case, N represents either one of A or B pixels, and Q and R are either one of C or D, respectively.

  Similarly, when the pixel arrangement pattern is the second arrangement pattern, as shown in FIG. 9, the pixel of the peripheral pixel Ps of the same color as the central pixel Pm is M, and the pixels of the other colors are N, Q, and R. Then, the arrangement of the pixels N, Q, and R in the second arrangement pattern is as shown in FIG. In this case, N represents either one of C or D pixels, and Q and R are either one of A or B, respectively.

  In other words, the first and second arrangement patterns indicate an arrangement pattern of pixels in which filters of the same color are superimposed in a 5 × 5 pixel region Fm centered on the center pixel Pm. This indicates not only the arrangement position of the same-color peripheral pixels Ps of the same color but also the arrangement positions of pixels (N, Q, R) that are different in color from the central pixel Pm and have the same color.

  FIG. 10 is a diagram illustrating a circuit configuration of the signal processing block. The signal processing performed in the signal processing block 125 will be described with reference to FIG. The signal processing block 125 is provided with a direction detection block 160, and the direction detection block 160 detects the correlation direction for each pixel. The detection of the correlation direction is generally performed by setting each pixel (m × n pixels) as the central pixel Pm in the m-row / n-column matrix, the central pixel Pm, and the same-color peripheral pixels having the same color as the central pixel Pm. This is done using Ps. However, in this example, there are a plurality of arrangement patterns (two types in the pixel pattern of FIG. 2) of the same-color peripheral pixels Ps with respect to the central pixel Pm, and the same-color peripheral pixels Ps cannot be selected uniformly.

  Therefore, in the direction detection block 160, it is assumed that the same color peripheral pixels Ps having the same color as the center pixel Pm are positioned in the first and second arrangement patterns with respect to the center pixel Pm. The correlation direction in the case of the pixel array pattern is detected.

  That is, it is assumed that the first to fourth direction detection circuits 161 to 164 of the signal processing block 125 each have a pixel arrangement in a specific pixel arrangement pattern (first and second arrangement patterns in this example). A basic correlation direction (provisional correlation direction) is detected.

Pixel data P _{11 to} P _mn (image raw data), which are digital image signals for one frame, are input to the first direction detection circuit 161. In the first direction detection circuit 161, each of the pixels is defined sequentially center pixel Pm, the pixel data P _ij of a defined center pixel Pm, each of 5 × 5 pixel region Fm to about the central pixel Pm Pixel data is sequentially read out.

In the first direction detection circuit 161, it is assumed that the pixel arrangement of the 5 × 5 pixel region Fm is all the first arrangement pattern, and according to the first arrangement pattern, filters of the same color as the central pixel Pm are overlaid. The assumed same-color peripheral pixel Ps (see FIG. 4) is defined as the first specific peripheral pixel Ps ′. Then, in the first direction detection circuit 161, the pixel data P _{(i-2) (j-1)} , P _{(i- ) of} the first specific peripheral pixel Ps' that is assumed to be superimposed with the same color filter. _{2) (j + 1)} , P _{i (j-2)} , P _{i (j + 2)} , P _{(i + 2) (j-1)} , P _{(i + 2) (j + 1)} (see FIG. 6) and pixel data P of the central pixel Pm _The temporary correlation direction is detected using _ij .

In the detection of the temporary correlation direction, in each 5 × 5 pixel region Fm, the horizontal direction (that is, the row direction), the vertical direction (that is, the column direction), the left oblique direction inclined 45 ° to the left with respect to the vertical direction, and Correlation between the center pixel Pm and the first specific peripheral pixel Ps ′ in each of the right oblique directions inclined rightward by 45 ° with respect to the vertical direction is represented by correlation indices D1 to D4 as shown in equations (1) to (4), respectively. Calculated as D4. The correlation index D1 is an index indicating the correlation between the central pixel Pm and the specific peripheral pixel Ps ′ positioned in the vertical direction with respect to the central pixel Pm, and the specific peripheral pixel Ps positioned in the upward direction with respect to the central pixel Pm. '(First pixel group) pixel data (P _{(i-2) (j-1)} , P _{(i-2) (j + 1)} ), and a specific peripheral pixel Ps' positioned in the downward direction (second pixel group) ) Pixel data (P _{(i + 2) (j−1)} , P _{(i + 2) (j + 1)} ) are arithmetically averaged, and the difference between the average value and the pixel data P _ij of the central pixel is added and calculated. The

That is, in this embodiment, as the difference between the pixel data of the pixel data P _ij and specific peripheral pixel Ps' of the center pixel Pm is large, the correlation index D1 increases, the correlation is judged to be weak. The correlation index D2 (horizontal direction), D3 (left diagonal direction), and D4 (right diagonal direction) are also calculated in the same manner as the correlation index D1. The correlation indexes D1 to D4 calculated in this way are compared with each other, and the direction corresponding to the correlation index having the smallest value is detected as the temporary correlation direction (first temporary correlation direction) assuming that the correlation is the strongest. That is, the direction in which the color signals of the center pixel Pm and the specific peripheral pixel Ps ′ are relatively approximated is detected as the correlation direction of each center pixel Pm. If at least two of the correlation indices D1 to D4 are the same value and are the smallest, the temporary correlation direction is detected as “no correlation direction”, assuming that there is no correlation direction.

  In the calculation of the correlation indices D1 to D4, when the pixel data of each pixel is averaged, the first operator (matrix) shown in FIG. 11 is used. That is, in the present embodiment, the coefficients multiplied by the pixels of the same color are all 1, and the correlation indices D1 to D4 are calculated by arithmetic averaging of each pixel data as described above. However, in the detection of the correlation direction, each coefficient may be appropriately weighted, that is, the correlation indices D1 to D4 may be detected by a weighted average.

Similarly to the first direction detection circuit 161, pixel data P _{11 to} P _mn (image raw data) that are digital image signals for one frame are input to the second direction detection circuit 162. In the second direction detection circuit 162, each pixel is sequentially determined as the central pixel Pm, the pixel data of the determined central pixel Pm, and each pixel in the 5 × 5 pixel region Fm centered on the central pixel Pm. Data P _ij is read sequentially.

In the second direction detection circuit 162, it is assumed that the pixel arrangement of the 5 × 5 pixel region Fm is all the second arrangement pattern, and according to the second arrangement pattern, filters of the same color as the central pixel Pm are overlaid. The assumed same-color peripheral pixel Ps (see FIG. 5) is defined as the second specific peripheral pixel Ps ″. Then, the second direction detection circuit 162 assumes that the filters of the same color are superimposed. Pixel data P _{(i-2) (j-2)} , P _{(i-2) j} , P _{(i-2) (j + 2)} , _{Pi (j-2)} , _{Pi (j + 2 ) of} the specific peripheral pixel Ps ″ ₎ , P _{(i + 2) (j-2)} , P _{(i + 2) j} , P _{(i + 2) (j + 2)} and the pixel data of the central pixel Pm are used to detect the temporary correlation direction.

In the detection of the temporary correlation direction, in each 5 × 5 pixel area Fm, as in the first direction detection circuit 161, the center pixel Pm in the horizontal direction, the vertical direction, the left diagonal direction, and the right diagonal direction, and the second specific periphery The correlation of the pixel Ps ″ is calculated as the correlation index D1 to D4, respectively. The correlation index D1 indicates the correlation between the center pixel Pm and the second specific peripheral pixel Ps ″ positioned in the direction perpendicular to the center pixel Pm. The pixel data (P _{(i−2) j} , P _{(i−2) (j−) of} the second specific peripheral pixel Ps ″ (first pixel group) that is an index and is located above the center pixel Pm. ₂₎ , P _{(i-2) (j + 2)} ), and pixel data (P _{(i + 2) j} , P _{(i + 2) (j-2)) of} the specific peripheral pixel Ps ′ (second pixel group) located in the downward direction. _{, P (i + 2) (} j + 2)) each are a weighted average, the weighting The calculated difference between the pixel data P _ij equalizing the center pixel Pm, respectively, the sum of the calculated difference is calculated as D1.

  The correlation indices D2 to D4 are also calculated by the equations (6) to (8) in the same manner as the correlation index D1. The calculated correlation indices D1 to D4 are compared, and the direction corresponding to the correlation index having the smallest value is detected as the second temporary correlation direction, assuming that the correlation is the strongest. If at least two of the correlation indices D1 to D4 are the same index and are the smallest, the provisional correlation direction is detected as “no correlation direction”, assuming that there is no correlation direction.

  In the calculation of the correlation indexes D1 to D4, when the pixel data of each pixel is weighted average, the coefficient used for the weighted average is the operator coefficient shown in FIG. Here, as shown in FIG. 12, the coefficients used for the weighted average are 1 and 2, but this coefficient is determined according to the directionality of the pixel with respect to the center pixel. That is, in the correlation index D1, when the specific peripheral pixel Ps is located in the same column of the central pixel Pm, the pixel data is multiplied by a coefficient 2. On the other hand, if the specific peripheral pixel Ps is located above and below the central pixel Pm but not located in the same column, the pixel data is multiplied by a coefficient 1. Note that the coefficient is similarly set to 2 or 1 for the exponential functions D2 to D4.

  Thus, for each center pixel Pm, the first provisional correlation direction calculated on the assumption that the pixel arrangement pattern of the 5 × 5 pixel region Fm is the first arrangement pattern is the first pixel as the direction signal. Input to the interpolation circuit 171. For each central pixel Pm, the second provisional correlation direction calculated on the assumption that the pixel arrangement pattern of the 5 × 5 pixel region Fm is the second arrangement pattern is the second pixel interpolation as the direction signal. Input to the circuit 172.

  2 has two pixel arrangement patterns in the 5 × 5 pixel region Fm, the third and fourth direction detection circuits 163 and 164 do not detect the provisional correlation direction. That is, in the present embodiment, the number of detected temporary correlation directions is set according to the number of pixel arrangement patterns in the 5 × 5 pixel region Fm. For example, when there are four types of pixel arrangement patterns, all of the first to fourth direction detection circuits 161 to 164 are used, the first to fourth temporary correlation directions are detected, and the first to fourth directions are detected. Each temporary correlation direction is input to the first to fourth pixel interpolation circuits 171 to 174 as a direction signal.

  The direction detection block 160 stores a plurality of mathematical formulas for calculating the correlation index in advance, including the mathematical formulas (1) to (8). The correlation index is calculated according to the input pixel arrangement information. A mathematical formula for calculation is read out. In the example of the pixel arrangement shown in FIG. 2, equations (1) to (4) and (5) to (8) are read out, and information about these equations is sent to the first and second direction detection circuits 161 and 162. Entered. In the first and second direction detection circuits 161 and 162, the correlation direction is detected as described above based on the equations (1) to (4) and (5) to (8). Note that since the solid-state imaging device of this example has two types of pixel arrangement patterns in the 5 × 5 pixel region Fm, information regarding the mathematical formula is not input to the third and fourth direction detection circuits 163 and 164, so that As described above, the correlation direction is not detected.

In the signal processing block 125, pixel interpolation is performed in the pixel interpolation block 170 following detection of the correlation direction. Similar to the direction detection block 160, the pixel interpolation block 170 is provided with first to fourth pixel interpolation circuits 171 to 174. Pixel data P _{11 to} P _mn (image raw data) are input to each of the first to fourth pixel interpolation circuits 171 to 174, and the pixel data P _{11 to} P _mn are used in the respective circuits to generate pixels. Interpolation is performed.

  Hereinafter, pixel interpolation performed by the first to fourth pixel interpolation circuits 171 to 174 will be described. As described above, in the solid-state imaging device, when the center pixel Pm is A or B, the pixel array pattern of the 5 × 5 pixel region Fm is the first array pattern as shown in FIG. When the central pixel Pm is C or D, the pixel arrangement pattern of the 5 × 5 pixel region Fm is the second arrangement pattern as shown in FIG. That is, in the 5 × 5 pixel region Fm, the arrangement of pixels of the same color is either the first or second arrangement pattern. On the other hand, in pixel interpolation, interpolation data is calculated by weighted or arithmetically averaging pixel data of pixels of the same color.

Therefore, in the first pixel interpolation circuit 171, it is assumed that all the pixels are arranged according to the first arrangement pattern, and for each central pixel Pm, the M, N, Q, and R pixels shown in FIG. 8 are the same. Pixel interpolation is performed assuming that the filters are overlaid, and a first interpolation signal for each central pixel is generated. That is, in the first pixel interpolation circuit 171, for each central pixel Pm, first, pixel data P _{(i−2) (j−2)} ,... Of the 5 × 5 pixel region Fm centered on the central pixel Pm. , P _ij ,..., P _{(i + 2) (j + 2)} (that is, pixel data shown in FIG. 6) is read out.

On the other hand, in the control circuit 150, in the 5 × 5 pixel area Fm, according to the first arrangement pattern, pixel data of pixels that are assumed to have the same color filters superimposed are grouped as a provisionally identical color group. Is done. That is, as is clear from FIGS. 6 and 8, P _{(i−2) (j−1)} , P _{(i−2) (j + 1)} , P _{i (j−2)} , P _ij , P _{i (j + 2)} , P _{(i + 2) (j−1)} and P _{(i + 2) (j + 1)} pixel data are grouped as a provisionally identical color (M) group. Also, P _{(i-2) (j-2)} , P _{(i-2) j} , P _{(i-2) (j + 2)} , P _{i (j-1)} , P _{i (j + 1)} , P _{(i + 2) ( j-2)} , P _{(i + 2) j} , and P _{(i + 2) (j + 2)} pixel data are grouped as a provisionally identical color (N) group. Also, P _{(i-1) (j-2)} , P _{(i-1) j} , P _{(i-1) (j + 2)} , P _{(i + 1) (j-2)} , P _{(i + 1) j} , P ₍ The pixel data of _{i + 1) (j + 2)} are grouped as a provisionally identical color (Q) group. Also, the pixel data of P _{(i-1) (j-1)} , P _{(i-1) (j + 1)} , P _{(i + 1) (j-1)} , P _{(i + 1) (j + 1)} are provisionally the same color (R ) Grouped as a group. Pixel information regarding which group each pixel is grouped in is thus generated by the control circuit 150 in accordance with the first arrangement pattern and input to the first pixel interpolation circuit 171.

  Next, the first pixel interpolation circuit 171 reads a coefficient matrix stored in advance in the pixel interpolation block 170 in accordance with the first temporary correlation direction. Here, as shown in FIG. 13, the coefficient matrix is an operator (5 × 5 matrix) determined for each direction of the provisional correlation direction, and has a distance from the direction of the provisional correlation direction and the center pixel Pm. Accordingly, a coefficient is determined for each pixel.

  For example, if the temporary correlation direction is the vertical direction, the coefficient matrix Y1 is read out, and the coefficients 1 to 4 are determined only for the pixels in the pixel area extending vertically in the 5 × 5 pixel area Fm. The coefficient is set to 0 for the pixels in the region. That is, when the first temporary correlation direction is the vertical direction, the pixels in the left end and right end pixel regions are not used for calculating the interpolation signal related to the center pixel Pm, and the color signal of the pixel located along the vertical direction. Is used to calculate the interpolation signal. For example, if the first provisional correlation direction is the horizontal direction, as shown in the coefficient matrix Y2, coefficients 1 to 4 are determined only for the pixel areas extending in the horizontal direction, and the pixels in the upper and lower pixel areas have a coefficient of Interpolation signals are calculated using pixel data of pixels that are set to 0 and are positioned along the horizontal direction. The same applies when the first temporary correlation direction is the left diagonal direction and right diagonal direction. When the correlation direction is determined to be no correlation direction, the interpolation signal is calculated using the pixel data of all the pixels in the 5 × 5 pixel region Fm excluding the four corner pixels of the 5 × 5 pixel region Fm. . Note that the coefficient matrices Y1 to Y5 in this embodiment are an example for detecting the correlation direction, and the correlation direction may be detected by another matrix. Of course, the size of the operator is not limited to 5 × 5.

Each pixel data (P _{(i−2) (j−2)} ,..., P _ij ,..., P _{(i + 2) (j + 2)} ) of the 5 × 5 pixel region Fm is shown in FIG. After the coefficients corresponding to the pixels are multiplied, the pixel data of the same color pixels M, N, Q, and R grouped as the provisional same color group according to the first arrangement pattern are added, and the added value of the pixel data is The first interpolation signals m ′, n ′, q ′, and r ′ are calculated by dividing by the sum of the coefficients.

Therefore, for example, when the temporary correlation direction is the vertical direction, the first interpolation signals m ′, n ′, q ′, and r ′ at the position of the center pixel are expressed by equations (9) to (12).

In the second pixel interpolation circuit 172, it is assumed that all the pixels are arranged in the second arrangement pattern, and that the same filter is overlaid on each of the M, N, Q, and R pixels shown in FIG. Thus, pixel interpolation is performed and a second interpolation signal is generated. That is, in the second pixel interpolation circuit 172, first, pixel data P _{(i-2) (j-2)} ,..., _Pij,. , P _{(i + 2) (j + 2)} (that is, the pixel data shown in FIG. 7) is read out.

On the other hand, in the control circuit 150, according to the second arrangement pattern, pixel data of pixels that are assumed to be overlapped with filters of the same color are grouped as provisional same color groups. That is, as is clear from FIGS. 7 and 9, P _{(i−2) (j−2)} , P _{(i−2) j} , P _{(i−2) (j + 2)} , P _{i (j−2)} , The pixel data of P _ij , P _{i (j + 2)} , P _{(i + 2) (j−2)} , P _{(i + 2) j} , P _{(i + 2) (j + 2)} is assumed to be pixel data of the same color pixel M. In addition, P _{(i-2) (j-1)} , P _{(i-2) (j + 1)} , _{Pi (j-1)} , _{Pi (j + 1)} , P _{(i + 2) (j-1)} , P _{( i + 2) (j + 1)} is assumed to be pixel data of the same color pixel N. Also, P _{(i-1) (j-2)} , P _{(i-1) j} , P _{(i-1) (j + 2)} , P _{(i + 1) (j-1)} , P _{(i + 1) (j + 1)} The pixel data of the same color pixel Q is assumed. Also, P _{(i-1) (j-1)} , P _{(i-1) (j + 1)} , P _{(i + 1) (j-2)} , P _{(i + 1) j} , P _{(i + 1) (j + 2)} are the same color. The pixel data of the pixel R is assumed. Pixel information regarding which group each pixel is grouped in is thus generated by the control circuit 150 in accordance with the second arrangement pattern and input to the first pixel interpolation circuit 172.

  Next, the second pixel interpolation circuit 172 reads an operator (5 × 5 coefficient matrix) stored in advance in the pixel interpolation block 170 in accordance with the second temporary correlation direction. Here, as in the first pixel interpolation circuit 171, any one of the coefficient matrices Y1 to Y5 is read out according to the second temporary correlation direction.

Each of the pixel data (P _{(i−2) (j−2)} ,..., P _ij ,..., P _{(i + 2) (j + 2)} ) of the 5 × 5 pixel region Fm has a read coefficient. After the coefficients corresponding to the respective pixels shown in FIG. 12 of the matrix are multiplied, the pixel data of provisionally identical color pixels M, N, Q, and R assumed to have the same color according to the second arrangement pattern are added to each other. The added value of the data is divided by the sum of the coefficients, and the second interpolation signals m ″, n ″, q ″, and r ″ are calculated.

Therefore, for example, when the second temporary correlation direction is the vertical direction, the second interpolation signals m ″, n ″, q ″, and r ″ at the position of the center pixel are expressed by equations (13) to (16). In this example, the third and fourth pixel interpolation circuits 173 and 174 do not receive information regarding the temporary correlation direction from the direction detection block 160, respectively. Therefore, the third and fourth pixel interpolation circuits 173 and 174 do not generate an interpolation signal.

  As shown in FIG. 14, the first and second interpolation signals generated by the first and second pixel interpolation circuits 171 and 172 are supplied to the first and second pixel determination circuits 181 and 182 of the pixel determination block 180 as one row. Each pixel is sequentially input from 1 column to m rows and n columns. Here, as described above, the first and second interpolation signals are generated for each pixel (m × n pixels), and the first and second interpolation signals for the same pixel are synchronized with each other. And input to the second pixel discrimination circuits 181 and 182. In the first and second pixel discriminating circuits 181 and 182, one of the first and second interpolation signals input in synchronization is adopted as the adopted interpolation signal, and at the same time, the adopted interpolation signal is adopted. The color of each signal m ′, n ′, q ′, and r ′ (or m ″, n ″, q ″, and r ″) is specified, and the adopted interpolation signal that specifies the color is used as a pixel interpolation signal. To be acquired. In this example, no interpolation signal is input to the third and fourth pixel discrimination circuits 183 and 184. Actually, when the center pixel is located in one row, two rows, m−1 rows, or m rows, or one column, two columns, n−1 columns, or n columns, interpolation regarding these center pixels is performed. No signal is generated. This is because these central pixels cannot be positioned at the center of the 5 × 5 pixel region Fm.

  A method of adopting one of the first and second interpolation signals as the adopted interpolation signal will be described with reference to FIG. First, second, fifth, and sixth pulse signals as shown in FIG. 14 are input from the control circuit 150 to the first pixel discrimination circuit 181. On the other hand, the third, fourth, seventh, and eighth pulse signals are input from the control circuit 150 to the second pixel determination circuit 182.

  As shown in FIG. 14, as the first and fifth pulse signals, the first and second interpolation signals relating to the pixel on which the filter of color A is superimposed are input to the first and second pixel discriminating circuits 181 and 182. The first pixel discrimination circuit 181 is input in synchronization with the above. The pixel on which the filter of the color A is superimposed is located in the 1st row and the 1st column or the 3rd row and the 2nd column in the basic matrix array α (see FIG. 2), that is, 4s (s is an integer of 0 or more) ) Pixels located in odd columns of +1 row, or pixels located in even columns of 4s + 3 rows. Here, the first pulse signal is input in synchronization with the input of the first and second interpolation signals relating to the pixels located in the odd columns of the 4s + 1 row. The fifth pulse signal is input in synchronization with the input of the first and second interpolation signals relating to the pixels located in the even columns of 4s + 3 rows. Then, in the first pixel discrimination circuit 181, the first interpolation signal synchronized with the first and fifth pulse signals is adopted as the adopted interpolation signal. On the other hand, the first and fifth pulse signals are not input to the second pixel discriminating circuit 182. Therefore, the second pixel relating to the pixel located in the 4s + 1 row and the odd column, or the pixel located in the 4s + 3 row and the even column. The interpolation signal is not adopted as the adopted interpolation signal.

  That is, in the basic matrix array α, the pixels located in the first row and the first column or the third row and the second column are identified as the 5 × 5 pixel region Fm having the pixel as the central pixel is the first array pattern (see FIG. 8). Based on this specification, the first or fifth pulse signal is output to the first pixel discrimination circuit 181. In the first pixel discriminating circuit 181, the first interpolation signal input in synchronization with the first or fifth pulse signal is an adopted interpolation signal of a pixel located in 1 row 1 column or 3 rows 2 columns in the basic matrix array α. Adopted as

  The second and sixth pulse signals are synchronized with the first and second interpolation signals relating to the pixel on which the color B filter is superimposed being input to the first and second pixel discriminating circuits 181 and 182. 1 is input to the discrimination circuit 181. Here, the pixels on which the filters of the color B are superimposed are located in the 1 × 2 or 3 × 1 column in the basic matrix array α (see FIG. 2), and in the raw image data, the pixels in the even columns of 4s + 1 rows, Or it is the pixel of the odd column of 4s + 3 row. Here, the second pulse signal is input in synchronization with the input of the first and second interpolation signals for the pixels in the even columns of 4s + 1 rows. The sixth pulse signal is input in synchronism with the input of the first and second interpolation signals for the pixels in the odd column of 4s + 3 rows. In the first pixel discrimination circuit 181, the first interpolation signal synchronized with the input of the second and sixth pulse signals is adopted as the adopted interpolation signal. On the other hand, the second and sixth pulse signals are not input to the second pixel discriminating circuit 182. Therefore, the second interpolation signals relating to the pixels in the even columns in the 4s + 1 row and the pixels in the odd columns in the 4s + 3 row are adopted interpolation. It is not adopted as a signal.

  That is, in the basic matrix array α, the pixels located in the first row and the second column or the third row and the first column are identified as the 5 × 5 pixel region Fm having the pixel as the central pixel is the first arrangement pattern. Based on this, the second or sixth pulse signal is output to the first pixel discrimination circuit 181. In the first pixel discriminating circuit 181, the first interpolation signal input in synchronization with the second or sixth pulse signal is an adopted interpolation signal of a pixel located in the first row, the second column or the third row, the first column in the basic matrix array α. Adopted as

  As described above, for each pixel in the 4s + 1 row and the 4s + 3 row (that is, the odd row), the first interpolation signal m ′, n ′, q is based on the first, second, fifth, or sixth pulse signal. 'And r' are adopted as adopted interpolation signals.

  Similarly, the second pixel discrimination circuit 182 receives a third pulse signal synchronized with the input of the first and second interpolation signals relating to the pixels in the odd column of 4s + 2 rows, and inputs the third pulse signal. Therefore, the second interpolation signal is adopted as the adopted interpolation signal for these pixels. The second pixel discrimination circuit 182 receives a seventh pulse signal synchronized with the input of the first and second interpolation signals relating to the pixels in the odd column of 4s + 4 rows. Based on the input, the second interpolation signal is employed as the employed interpolation signal for these pixels.

  The second pixel determination circuit 182 receives a fourth pulse signal that is synchronized with the input of the first and second interpolation signals related to the pixels in the even column of 4s + 2 rows, and based on the input of the fourth pulse signal, The second interpolation signal is employed as the interpolation signal for these pixels. In addition, an eighth pulse signal synchronized with the input of the first and second interpolation signals related to the pixels in the even-numbered columns of 4s + 4 rows is input, and the interpolation related to these pixels is performed based on the input of the eighth pulse signal. The second interpolation signal is adopted as the signal.

  That is, in the pixel determination block 180, pixels (even-numbered rows of pixels) located in 2 or 4 rows in the basic matrix array α based on the third, fourth, seventh, and eighth pulse signals are the pixels. The 5 × 5 pixel region Fm having the center pixel as the second arrangement pattern is specified, and the second interpolation signal is adopted as the adopted interpolation signal.

  As described above, the pixel determination block 180 determines whether the pixel arrangement pattern in the 5 × 5 pixel region is the first or second arrangement pattern depending on where the central pixel is arranged in the basic matrix α. Determine if it exists. Then, an interpolation signal generated according to the specified pixel arrangement pattern from the first and second interpolation signals is adopted as the adopted interpolation signal.

  In the pixel discrimination block 180, one of the first and second interpolation signals is specified as the employed interpolation signal, and each of the signals m ′, n ′, q ′ of the employed interpolation signal is based on the first to eighth pulse signals. , R ′ (or m ″, n ″, q ″, and r ″) are also specified. Next, a method for specifying the color of each signal will be described with reference to FIGS. 14 and 15.

  As shown in FIG. 14, the first pulse signal is input in synchronization with the input of the first and second interpolation signals relating to pixels in odd columns of 4s + 1 rows (that is, 1 row in the basic matrix α). The The fifth pulse signal is input in synchronization with the first and second interpolation signals regarding pixels in even columns of 4s + 3 rows (that is, 3 rows in the basic matrix α).

  On the other hand, the pixel array is configured by arranging a plurality of basic matrices α in the row direction and the column direction (see FIG. 2). Therefore, when the pixel (pixel on which the filter of color A is superimposed) located in the odd column of 4s + 1 row is the central pixel Pm, the arrangement of the 5 × 5 pixel region Fm is X1 (see FIG. 15), and 5 × The colors of the filters of the pixels M, N, Q, and R in the five-pixel region Fm are colors A, B, C, and D, respectively. Therefore, the color of each of the interpolated signals m ′, n ′, q ′, and r ′ for the pixels located in the odd columns of the 4s + 1 row is based on the first pulse signal and the colors A, B, C, and D Specified.

  As described above, the colors of the adopted interpolation signals m ′, n ′, q ′, and r ′ are specified as the colors A, B, C, and D by the first pulse signal that is input. Similarly, when a pixel located in an even column of 4s + 3 rows (a pixel on which a filter of color A is superimposed) is a central pixel, the arrangement of the 5 × 5 pixel region Fm is X6 (see FIG. 15), and 5 × The colors of the filters of the pixels M, N, Q, and R in the five-pixel region Fm are colors A, B, D, and C, respectively. Therefore, the colors of the interpolation signals m ′, n ′, q ′, and r ′ adopted by the pixels in the even column of 4s + 3 rows are specified as colors A, B, D, and C based on the input of the fifth pulse signal. The

  The same applies to the colors of the adopted interpolation signals m ′, n ′, q ′, and r ′ (or m ″, n ″, q ″, and r ″) for the pixels on which the filters of the colors B to D are superimposed. It is specified based on the second to fourth and sixth to eighth pulse signals.

  As described above, in the present embodiment, even when pixel data is read out by an independent readout method in a solid-state imaging device having a complicated pixel arrangement, pixel interpolation is performed on the color signal of each pixel according to the pixel arrangement. It can be carried out. Further, in the present embodiment, pixel interpolation and correlation direction detection are appropriately performed according to the pixel arrangement, so that generation of false colors can be reduced. Furthermore, in the present embodiment, pixel data can be read out from a solid-state imaging device having a complicated pixel arrangement by an independent readout method, and the image resolution can be further increased.

  Hereinafter, an example in which a solid-state imaging device having a pixel arrangement different from the solid-state imaging device in FIG. 2 is applied to the present embodiment will be described. FIG. 16 shows an example of a solid-state imaging device having a pixel arrangement different from that in FIG. In this pixel arrangement, there are four types of arrangement position patterns of pixels on which filters of the same color are superimposed, and there are first to fourth arrangement patterns as shown in FIG.

  In this example, information related to a mathematical expression for calculating a correlation index corresponding to each pixel arrangement pattern stored in advance in the direction detection block 160 is read out, and is stored in each direction detection circuit 161 to 164 of the direction detection block 160. Entered. Each of the direction detection circuits 161 to 164 detects the first to fourth temporary correlation directions by using the input mathematical formula, assuming that the pixel arrangement is in the first to fourth arrangement patterns.

Here, for example, in the detection of the first temporary correlation direction, the pixel data of the pixel M is used as the pixel data of the first specific peripheral pixel Ps ′ as can be understood from FIG. That is, assuming that the pixel data of the central pixel is P _ij , P _{(i−2) (j−1)} , P _{(i−2) (j + 1)} , P _{i ( ) are} used as the pixel data of the first specific peripheral pixel Ps ′. Pixel data of _j−2) , P _{i (j + 2)} , P _{(i + 2) (j−1)} , and P _{(i + 2) (j + 1)} are used for detection of the first temporary correlation direction. Similarly, in the second to fourth direction detection circuits 162 to 164, the pixel data of the pixel M in FIGS. 17B, 17C, and 17D are respectively the specific peripheral pixels Ps ″, Ps ″ ′, and Ps ″. "Is used as pixel data.

  The first to fourth temporary correlation directions are input to the pixel interpolation circuits 171 to 174 of the pixel interpolation block 170, respectively. Each of the pixel interpolation circuits 171 to 174 uses the first to fourth temporary correlation directions, the raw image data, and the coefficient operators (Y1 to Y5) shown in FIG. Calculated. The interpolation signal is calculated assuming that the M, N, Q, and R pixels shown in FIG. 17 are pixels of the same color.

  Each interpolation signal is input to each of the pixel discrimination circuits 181 to 184, and for each pixel, one of the first to fourth interpolation signals is set as an adopted interpolation signal for each pixel. Through the sixteenth pulse signal, the color of each of the interpolation signals m, n, q, r is specified as ABCD.

  As described above, in the signal processing block 125, when there are four types of pixel arrangement patterns of the 5 × 5 pixel region Fm, in each of the blocks 160, 170, and 180, four circuits are used to generate pixel interpolation signals. Calculated.

  Further, an example in which a solid-state imaging device having another pixel arrangement is applied to this embodiment will be described with reference to FIG. FIG. 18 shows an example in which A and B pixels are alternately arranged in all odd rows, and C and D pixels are alternately arranged in all even rows. In such a pixel array, there is only one kind of pixel array pattern in which filters of the same color are superimposed, and only the first array pattern is shown in FIG.

  Therefore, in the direction detection block 160, the correlation direction is detected only by the first direction detection circuit 161, and the correlation direction is not detected by the second to fourth direction detection circuits 162 to 164. That is, pixel interpolation is performed only by the first pixel interpolation circuit 171, and the first interpolation signal generated by the first pixel interpolation circuit 171 is input to the first pixel discrimination circuit 181 and all the first interpolation signals are converted. Are identified as the interpolated interpolation signals, and are identified as the pixel interpolation signals for each pixel based on the first and second pulse signals.

  As described above, the signal processing block 125 according to the present embodiment is a color obtained by the solid-state imaging device when there are 1 to 4 types of pixel arrangement patterns in the 5 × 5 pixel region Fm centered on each central pixel Pm. Pixel interpolation can be performed on the signal.

  In addition, since the pixel arrangement information input to the control circuit 150 is changed according to the pixel arrangement pattern of the solid-state imaging device, the color signal processing block 125 according to the present embodiment corresponds to the pixel arrangement pattern of the solid-state imaging element. Thus, an appropriate correlation direction can be detected. Therefore, even if the type of the scope 10 is changed and the pixel arrangement of the solid-state imaging device is changed, the correlation direction can be appropriately detected according to the change. Of course, the same applies to pixel interpolation, and appropriate luminance and color difference signals can be obtained even when the pixel arrangement of the solid-state imaging device is changed.

  As another example of the pixel array in which two types of pixel array patterns exist, for example, the array of FIG. Another example of the four types of pixel arrangement is the arrangement shown in FIG.

  In the present embodiment, the color of the filter superimposed on the solid-state imaging device may be three colors or five or more. For example, as an example of three colors, a Bayer arrangement shown in FIG. In the arrangement of FIG. 22, there are two types of pixel arrangements because the arrangement positions of pixels of the same color are different when the central pixel is A and when the central pixel is B or C.

  In the present invention, even when there are five or more types of pixel arrangement patterns of the 5 × 5 pixel region Fm, it can be dealt with by further increasing the number of circuits in each of the blocks 160, 170, and 180.

  Furthermore, in the present embodiment, the pixel region Fm centered on the central pixel Pm in which the pixel arrangement pattern is specified is 5 × 5 pixels in the present embodiment, but may be an odd number × odd number, for example, It may be a 3 × 3 pixel region or a 7 × 7 pixel region.

  In the above description, the pixel signal of each pixel is read out independently from the image sensor 14 for each pixel. However, in the present invention, the image signal of each pixel is obtained by the upper and lower two pixels in the vertical direction. The present invention can also be applied to the case of adding and reading. In this embodiment, the pixel signal is input to the direction determination block and the pixel interpolation block and then input to the pixel determination block. However, after the pixel signal is input to the pixel determination block first, the direction detection block and the pixel are input. It may be input to the interpolation block. In this case, after specifying the arrangement pattern of each pixel in the discrimination block, pixel interpolation and direction detection corresponding to the specified arrangement pattern are performed in the pixel interpolation block and the direction detection block.

1 is a block diagram of an electronic endoscope apparatus according to an embodiment of the present invention. It is the figure which showed an example of the light-receiving surface of an imaging sensor. It is the figure which showed typically the pixel data for 1 frame. It is a figure which shows the arrangement pattern of a center pixel and the surrounding pixel of the same color in a 1st arrangement pattern. It is a figure which shows the arrangement pattern of the center pixel and the surrounding pixel of the same color in a 2nd arrangement pattern. It is a figure which shows the pixel data of a 5 * 5 pixel area | region in a 1st arrangement pattern. It is a figure which shows the pixel data of a 5 * 5 pixel area | region in a 2nd arrangement pattern. It is a figure which shows typically the relationship of the color of each pixel in case the pixel arrangement pattern of a 5x5 pixel area is a 1st arrangement pattern. It is a figure which shows typically the relationship of the color of each pixel in case the pixel arrangement pattern of a 5x5 pixel area is a 2nd arrangement pattern. It is the figure which showed the circuit structure of the signal processing block. This is a coefficient matrix used for detecting the provisional correlation direction of the center pixel when the pixel arrangement pattern of the 5 × 5 pixel region is the first arrangement pattern. This is a coefficient matrix used to detect the temporary correlation direction of the center pixel when the pixel arrangement pattern of the 5 × 5 pixel region is the second arrangement pattern. It is a coefficient matrix used for calculating a pixel interpolation signal. It is a figure which shows typically the pulse signal input into a pixel discrimination | determination block. It is a figure which shows the pattern of the arrangement | sequence regarding the color in a 5x5 pixel area | region. It is the figure which showed the other example of the light-receiving surface of an imaging sensor. FIG. 16 is a diagram showing a pixel arrangement pattern in the pixel arrangement of the example of FIG. 15. It is the figure which showed the other example of the light-receiving surface of an imaging sensor. It is a figure which shows the pixel arrangement | sequence pattern in the pixel arrangement | sequence of the example of FIG. It is the figure which showed the other example of the light-receiving surface of an imaging sensor. It is the figure which showed the other example of the light-receiving surface of an imaging sensor. It is the figure which showed the other example of the light-receiving surface of an imaging sensor.

Explanation of symbols

14 imaging sensor 125 signal processing block 160 direction detection block (direction detection means)
161-164 First to fourth direction detection circuits (direction detection unit)
170 Pixel Interpolation Blocks 171 to 174 First to Fourth Pixel Interpolation Circuits (Pixel Interpolation Unit)
180 pixel discrimination block 181 to 184 first to fourth pixel discrimination circuit Fm 5 × 5 pixel area Pm central pixel Ps same color peripheral pixel Ps ′, Ps ″, Ps ″ ′, Ps ″ ″ specific peripheral pixel

Claims (19)

Each of the photoelectric conversion elements is a solid-state imaging element configured by regularly arranging pixels configured by overlapping filters of at least first to third colors that are different from each other. And a pixel in which filters of the same color are overlaid in a k × k pixel region centered on each central pixel is arranged according to any of a plurality of pixel arrangement patterns including at least the first and second arrangement patterns In the image signal processing apparatus capable of processing the color signal output from each pixel of the solid-state imaging device,
Each of the plurality of pixel arrangement patterns is assumed to be a pixel in which filters of the same color in each k × k pixel region are overlaid in accordance with each of the one or more pixel arrangement patterns, and the filters of the same color are overlaid. Pixel interpolating means for interpolating color signals of pixels assumed to be generated, and generating one or more interpolation signals for each central pixel;
Pixel discrimination means for specifying the pixel arrangement pattern in the k × k pixel region as one specific arrangement pattern selected from the plurality of pixel arrangement patterns;
An image signal processing apparatus, wherein one interpolation signal generated according to the specific arrangement pattern is determined as an adopted interpolation signal for the center pixel from the one or more interpolation signals.   The pixel interpolating means interpolates pixels by performing weighted averaging or arithmetic averaging of color signals of pixels in which the filters of the same color are overlapped in the k × k pixel region, so that each color relating to the central pixel is obtained. The image signal processing apparatus according to claim 1, wherein an interpolation signal is generated. The first and second array patterns are formed by arranging a plurality of basic matrix arrays having the same pixel array in the row direction and the column direction,
The image signal processing apparatus according to claim 1, wherein the pixel determination unit specifies the pixel arrangement pattern depending on a position where the central pixel is arranged in the basic matrix. In the k × k pixel region, comprising a direction detecting means for detecting a direction in which color signals of the center pixel and peripheral pixels of the center pixel are relatively approximated as a correlation direction of the center pixels,
The image signal processing apparatus according to claim 1, wherein the pixel interpolation unit performs pixel interpolation using color signals of pixels located along the correlation direction detected by the direction detection unit.   In the k × k pixel region, the direction detection unit is configured to determine a direction in which color signals of the central pixels and peripheral pixels of the same color on which filters of the same color as the central pixels are overlapped relatively approximate each center. The image signal processing apparatus according to claim 4, wherein the image signal processing apparatus detects the correlation direction of pixels. The pixel interpolation means includes at least first and second pixel interpolation units;
The first pixel interpolation unit assumes a pixel in which filters of the same color are overlaid in the k × k pixel region according to the first arrangement, and a pixel in which filters of the same color are overlaid Each of the color signals is interpolated to generate a first interpolation signal for each color that is defined as the same color,
In the k × k pixel region, the second pixel interpolation unit assumes pixels in which filters of the same color are overlaid in accordance with the second arrangement, and pixels assumed to be overlaid of filters of the same color Pixel signals are interpolated to generate a second interpolation signal for each color that is defined as the same color,
When the pixel determining unit specifies that the pixel array pattern in the k × k pixel region centered on the central pixel is the first array pattern, the pixel determining unit generates the first pixel generated by the first pixel interpolating unit. When one interpolation signal is specified as the adopted interpolation signal of the center pixel and the second array pattern is specified, the second interpolation signal generated by the second pixel interpolation unit is used as the center pixel. The image signal processing apparatus according to claim 1, wherein the interpolation signal is specified as the adopted interpolation signal. Comprising the direction detection means having at least first and second direction detection units;
The first direction detection unit determines, as the first specific peripheral pixel, a pixel that is assumed to have a filter of the same color as that of the central pixel overlaid according to the first arrangement pattern, and the first direction detection unit A direction in which color signals of specific peripheral pixels are relatively approximated is detected as a first temporary correlation direction;
The second direction detection unit determines, as the second specific peripheral pixel, a pixel that is assumed to have a filter of the same color as that of the central pixel overlaid in accordance with the second arrangement pattern, and the second direction detection unit A direction in which color signals of specific peripheral pixels are relatively approximated is detected as a second temporary correlation direction;
The first pixel interpolation unit performs pixel interpolation using color signals of pixels located along the first temporary correlation direction detected by the first direction detection unit,
The image according to claim 6, wherein the second pixel interpolation unit performs pixel interpolation using color signals of pixels located along the second temporary correlation direction detected by the second direction detection unit. Signal processing device. The first and second array patterns are formed by arranging a plurality of basic matrix arrays having the same pixel array in the row direction and the column direction,
The pixel discriminating unit specifies one of the first and second interpolation signals as the adopted interpolation signal of the central pixel depending on where the central pixel is arranged in the basic matrix, The image signal processing apparatus according to claim 6, wherein a color of each of the adopted interpolation signals is specified. The pixel determining means includes a first pixel determining unit to which the first interpolation signal is input, and a second pixel determining unit to which the second interpolation signal is input,
The first and second interpolation signals related to the same center pixel are input in synchronization with the first and second pixel discriminating units, respectively.
A pulse signal further synchronized with the first and second interpolation signals input in synchronization is input to one of the first and second pixel determination units, and one pixel determination unit to which the pulse signal is input 9. The image signal processing apparatus according to claim 8, wherein one interpolation signal input in synchronization with the pulse signal to one of the pixel discrimination units is adopted as an adopted interpolation signal for the center pixel. .   The pulse signal is the first pixel when the pixel arrangement pattern in the k × k pixel region centered on the central pixel relating to the first and second interpolation signals synchronized with the signal is the first arrangement pattern. The image signal processing apparatus according to claim 9, wherein the image signal processing device is input to the second pixel determination when the second arrangement pattern is input to the determination unit. The first and second array patterns are formed by arranging a plurality of basic matrix arrays having the same pixel array in the row direction and the column direction,
The pulse signal is different depending on where the central pixel related to the first and second interpolation signals synchronized with the signal is arranged in the basic matrix, and each color of the adopted interpolation signal is The image signal processing apparatus according to claim 9, wherein the image signal processing apparatus is specified according to a pulse signal. In the solid-state imaging device, the k × k pixel region centered on the center pixel located in an odd row is arranged in a first array pattern, and the k × k pixel region centered on the center pixel located in an even row. Are arranged in a second arrangement pattern,
The pixel determining means sets the first interpolation signal as the adopted interpolation signal of the center pixel when the center pixel is located in the odd row, and sets the second interpolation signal of the center pixel when located in the even row. The image signal processing apparatus according to claim 6, wherein the image signal processing apparatus is set as the adopted interpolation signal. The first and second array patterns are formed by arranging a plurality of basic matrix arrays having the same pixel array in the row direction and the column direction,
The said 1st and 2nd pixel discrimination | determination part specifies each color of each said employ | adopted interpolation signal according to which position the said center pixel is arrange | positioned in the said basic matrix arrangement | sequence. The image signal processing apparatus described.   The pixel interpolation means assumes pixels in which filters of the same color are overlapped in the k × k pixel area according to the specific arrangement pattern, respectively, and the color of the pixel assumed to have these same color filters overlapped 2. The image signal according to claim 1, wherein the signals are interpolated to generate one interpolation signal for each central pixel, and the generated interpolation signal is defined as an interpolation signal adopted for the central pixel. Processing equipment. Each of the photoelectric conversion elements is a solid-state imaging element configured by regularly arranging pixels configured by overlapping filters of at least first to third colors that are different from each other. And a pixel in which filters of the same color are overlaid in a k × k pixel region centered on each central pixel is arranged according to any of a plurality of pixel arrangement patterns including at least the first and second arrangement patterns In the image signal processing apparatus capable of processing the color signal output from each pixel of the solid-state imaging device,
Colors of peripheral pixels of the same color assumed that filters of the same color as each of the central pixels are overlaid in each of the k × k pixel regions according to each of one or more pixel arrangement patterns among the plurality of pixel arrangement patterns. Correlation direction detecting means for detecting at least one provisional correlation direction using a signal;
Pixel discrimination means for specifying the pixel arrangement pattern in the k × k pixel region as one specific arrangement pattern selected from the plurality of pixel arrangement patterns;
An image signal processing apparatus, wherein one temporary correlation direction generated according to the specific arrangement pattern is determined as a correlation direction of the central pixel from the one or more temporary correlation directions. A solid-state imaging device configured in an m-row n-column matrix in which pixels configured by overlapping filters of at least first to third colors having different colors are regularly arranged on each photoelectric conversion device. A plurality of pixels each including at least the first and second arrangement patterns, each pixel having a central pixel, and a pixel in which filters of the same color are overlapped in a k × k pixel region centered on each central pixel. An image signal processing apparatus capable of processing a color signal output from each pixel of a solid-state imaging device arranged according to any of the arrangement patterns,
The m-row n-column matrix is configured by arranging a plurality of basic matrices having the same pixel arrangement in the column direction and the row direction,
The pixel arrangement pattern in the k × k pixel area is one specific selection selected from the plurality of pixel arrangement patterns depending on where the central pixel of the k × k pixel area is arranged in the basic matrix. An image signal processing apparatus characterized by being specified by an array pattern. Each of the photoelectric conversion elements is a solid-state imaging element configured by regularly arranging pixels configured by overlapping filters of at least first to third colors that are different from each other. And a pixel in which filters of the same color are overlaid in a k × k pixel region centered on each central pixel is arranged according to any of a plurality of pixel arrangement patterns including at least the first and second arrangement patterns In an image signal processing method for processing a color signal output from each pixel of a solid-state imaging device,
Each of the plurality of pixel arrangement patterns is assumed to be a pixel in which filters of the same color in each k × k pixel region are overlaid in accordance with each of the one or more pixel arrangement patterns, and the filters of the same color are overlaid. A pixel interpolation step of interpolating color signals of assumed pixels and generating one or more interpolation signals for each central pixel;
A pixel determination step of specifying the pixel arrangement pattern in the k × k pixel region as one specific arrangement pattern selected from the plurality of pixel arrangement patterns,
An image signal processing method, wherein one interpolation signal generated according to the specific arrangement pattern is determined as an adopted interpolation signal for the center pixel from the one or more interpolation signals. Each of the photoelectric conversion elements is a solid-state imaging element configured by regularly arranging pixels configured by overlapping filters of at least first to third colors that are different from each other. And a pixel in which filters of the same color are overlaid in a k × k pixel region centered on each central pixel is arranged according to any of a plurality of pixel arrangement patterns including at least the first and second arrangement patterns In an image signal processing method for processing a color signal output from each pixel of a solid-state imaging device,
Colors of peripheral pixels of the same color assumed that filters of the same color as each of the central pixels are overlaid in each of the k × k pixel regions according to each of one or more pixel arrangement patterns among the plurality of pixel arrangement patterns. A correlation direction detecting step of detecting at least one or more provisional correlation directions using the signal;
A pixel determination step of specifying the pixel arrangement pattern in the k × k pixel region as one specific arrangement pattern selected from the plurality of pixel arrangement patterns,
An image signal processing method, wherein one temporary correlation direction generated according to the specific arrangement pattern is determined as a correlation direction of the central pixel from the one or more temporary correlation directions. A solid-state imaging device configured in an m-row n-column matrix in which pixels configured by overlapping filters of at least first to third colors having different colors are regularly arranged on each photoelectric conversion device. A plurality of pixels each including at least the first and second arrangement patterns, each pixel having a central pixel, and a pixel in which filters of the same color are overlapped in a k × k pixel region centered on each central pixel. In an image signal processing method for processing a color signal output from each pixel of a solid-state imaging device arranged according to any of the arrangement patterns,
The m-row n-column matrix is configured by arranging a plurality of basic matrices having the same pixel arrangement in the column direction and the row direction,
The specific arrangement pattern selected from the plurality of arrangement patterns according to where the central pixel of the k × k pixel area is arranged in the basic matrix. An image signal processing method characterized by:

Images