WO2020195515A1 - Imaging device and image processing method - Google Patents

Abstract

The purpose of the present invention is, when an image sensor in which two kinds of pixels are arrayed is used, to make it possible to accurately discriminate a prescribed hue, and to achieve both widening an angle of view and improvement of spatial resolution. This imaging device is provided with: an image sensor that is configured by repeatedly arraying filter units each including a first pixel that detects light in a first wavelength range of the visible region, and a second pixel that detects, in addition to the light in the first wavelength range, light having a wavelength of visible light that is different from the first wavelength range; an interpolation processing unit that is configured so as to be able to generate a first interpolation image obtained by interpolating the position of the second pixel on the basis of the amount of the detected light of the first pixel, and a second interpolation image obtained by interpolating the position of the first pixel on the basis of the amount of the detected light of the second pixel; and a color information generation processing unit that, on the basis of the amount of the detected light of a pair of pixels on the same position as the first interpolation image and the second interpolation image, determines a hue at the position.

Description

Imaging device and image processing method

The present invention relates to an image pickup apparatus and an image processing method, and more particularly to an image pickup apparatus for discriminating the hue of an captured image and an image processing method.

The in-vehicle camera mounted on the vehicle for driving support and automatic driving is required to have a wide angle of view in order to improve the accuracy of detecting pedestrians and bicycles jumping out from the side of the vehicle. On the other hand, in-vehicle cameras need to accurately detect the distance to a distant object due to ACC function (Adaptive Cruise Control, constant speed driving, inter-vehicle distance control), etc., and high spatial resolution is required. Be done. However, widening the angle of view is generally realized by reducing the focal length of the objective lens of the camera, and the spatial resolution is lowered accordingly. Therefore, it is difficult to achieve both wide angle of view and improvement of spatial resolution only by improving the objective lens.

On the other hand, a technology for improving the spatial resolution by adopting a different type of image sensor is also known. In the in-vehicle camera, a color image sensor having an RGGB (Red-Green-Green-Blue) pixel array (or Bayer array) to recognize traffic lights, signs, and white and orange lines on the road for driving assistance. Is often adopted. In recent years, a technique has been proposed in which the image sensor of this RGGB arrangement is replaced with a color image sensor such as an RCCC (Red-Clear-Clear-Clear) pixel arrangement. According to this, the spatial resolution of the image sensor is improved, and it is possible to achieve both wide angle of view and improvement of spatial resolution.

The RCCC pixel array includes, for example, one pixel (R pixel) that detects red light and three clear pixels (C pixel) that detect both blue light, green light, and red light, for a total of four pixels. It is composed of two rows and two columns (2 × 2) arranged in one unit (unit cell), and the unit cells are repeatedly arranged. The interval between C pixels is one pixel, which is different from the RGGB arrangement in which the interval between pixels of the same color is two pixels. In this RCCC array, the detection value of the C pixel, which is 3/4 of the entire pixel, can be used as it is for the grayscale image referred to in the distance measurement, and the interpolation processing from the surrounding pixels that lowers the spatial resolution can be used. Can be applied only to the position of the R pixel at a ratio of 1/4 of the entire pixel. Therefore, it is expected that a higher spatial resolution can be obtained as compared with the RGGB pixel array.

However, in the RCCC pixel array consisting of two types of pixels, R (Red) and C (Clear), the RGGB that detects the amount of light by classifying the three primary colors of R red (Red), green (Green), and blue (Blue). Compared with the pixel array, the amount of information related to color is limited, and it is necessary to revise the processing method related to color.
Patent Document 1 describes the application of the RCCC pixel array image sensor to an in-vehicle camera. However, there is no specific description about the processing for the captured image.

Patent Document 2 also discloses the application of an image sensor having a pixel array consisting of two types, R and C, to an in-vehicle camera. In Patent Document 2, the magnitude of the correlation between the image of the image sensor and the checkerboard pattern according to the pixel type Red / Clear of the image sensor is used for discrimination between the red tail lamp of an automobile and lights such as headlights and street lights. In this method, since correlation, which is a statistic, is used, the hue is discriminated for an image region having a certain number of pixels. Therefore, when objects having different hues coexist in this region, an accurate discrimination result cannot be obtained. Further, it is difficult to set an area so that patterns of different hues do not coexist in the area for discriminating hues unless the pattern has a certain size and the shape can be assumed. As described above, in the prior art, it is difficult to accurately determine the hue by the RCCC pixel arrangement. In addition to the RCCC pixel array, the same problem may occur in an image sensor having a pixel array composed of two types of pixels.

Japanese Unexamined Patent Publication No. 2017-046051 U.S. Patent Application Publication No. 2007/0221822

The present invention is an image pickup apparatus capable of accurately discriminating a predetermined hue when an image sensor in which two types of pixels are arranged is used, and can achieve both a wide angle of view and an improvement in spatial resolution. An object of the present invention is to provide an image processing method.

The image pickup apparatus according to the first aspect of the present invention includes a first pixel that detects light in a first wavelength range in the visible range, and the first wavelength range in addition to the light in the first wavelength range. Is an image sensor configured by repeatedly arranging filter units including a second pixel that detects light having a different visible light wavelength, and the position of the second pixel based on the detected light amount of the first pixel. The first interpolated image obtained by interpolating the position of the first pixel and the second interpolated image obtained by interpolating the position of the first pixel based on the detected light amount of the second pixel can be generated. It is provided with an interpolation processing unit and a color information generation processing unit that determines the hue at the position based on the detected light amount of a set of pixels at the same position of the first interpolated image and the second interpolated image. It is characterized by.

The image pickup apparatus according to the second aspect of the present invention includes a first pixel that detects light in the first wavelength range in the visible range, and the first wavelength range in addition to the light in the first wavelength range. Is an image sensor configured by repeatedly arranging filter units including a second pixel that detects light having a different visible light wavelength, and a position of the second pixel based on the amount of detected light of the first pixel. The first interpolated image obtained by interpolating the above and the second interpolated image obtained by interpolating the position of the first pixel based on the detected light amount of the second pixel can be generated. It includes an interpolation processing unit and a color image generation processing unit that generates a color image based on the first interpolated image and the second interpolated image. Based on the first interpolated image, the color image generation processing unit generates a first component image having a first wavelength component in the first wavelength range, and the first interpolated image and the second interpolated image. A difference image, which is the difference between the interpolated images, is generated, the difference image is multiplied by the first distribution ratio, and a second component image having a component in a second wavelength range different from the first wavelength range is generated. , The difference image is multiplied by a second distribution ratio to generate a third component image having components in the first wavelength range and a third wavelength range different from the second wavelength range. ..

In the image processing method according to the present invention, the first pixel for detecting light in the first wavelength range in the visible range, and visible light different from the first wavelength range in addition to the light in the first wavelength range. The second pixel is based on the step of acquiring an image from an image sensor configured by repeatedly arranging a filter unit including a second pixel for detecting light of the wavelength of the first pixel and the amount of detected light of the first pixel. The step of interpolating the position of the first pixel to acquire the first interpolated image, and the step of interpolating the position of the first pixel and acquiring the second interpolated image based on the detected light amount of the second pixel. The step includes a step of determining the hue at the position based on the detected light amount of the set of pixels at the same position of the first interpolated image and the second interpolated image.

According to the present invention, it is possible to provide an image pickup apparatus and an image processing method capable of accurately discriminating a predetermined hue when an image sensor in which two types of pixels are arranged is used, and a wide angle of view can be provided. And the improvement of spatial resolution can be achieved at the same time.

It is an overall block diagram explaining an example of the basic structure of the image pickup apparatus which concerns on 1st Embodiment. It is the schematic which showed the pixel arrangement of the image sensor of RCCC pixel arrangement. It is a graph which shows the sensitivity characteristic of R pixel, C pixel of the image sensor of RCCC pixel array. It is the schematic which showed an example of the arithmetic processing in the interpolation processing unit 102. It is a graph which shows the actual measurement characteristic example of the C pixel value and R pixel value of the RCCC image which photographed the color used for the road sign. It is a graph which shows the actual measurement characteristic example of the C pixel value and R pixel value of the RCCC image which photographed the color used in the traffic signal. It is a flowchart explaining the procedure from getting the hue information from the image taken by the image sensor 101. It is a flowchart which showed the detail of the procedure of discriminating the hue based on the ratio (R / C ratio) of a pixel value in step 704 of FIG. In the first embodiment, it is a flowchart showing a procedure of generating a colorized image by a color image generation processing unit 110 from a grayscale image taken by an image sensor 101. It is a figure (graph) explaining the 1st example of the distribution ratio α of G component image and B component image. It is a figure (graph) explaining the 2nd example of the distribution ratio α of G component image and B component image. It is an overall block diagram explaining an example of the basic structure of the image pickup apparatus which concerns on 2nd Embodiment. It is the schematic explaining the structural example of the color information generation table 1202 of FIG. It is an overall block diagram explaining an example of the basic structure of the image pickup apparatus which concerns on 3rd Embodiment. In a third embodiment, it is a flowchart showing a procedure for generating a colorized image by the color image generation processing unit 110 from an image taken by the image sensor 101. It is a schematic diagram explaining the pixel array (CyCCC array) in the image sensor 101 of the 4th embodiment. It is a graph which shows the sensitivity characteristic of Cy pixel, C pixel of the image sensor of CyCCC pixel array.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the attached drawings, functionally the same elements may be displayed with the same number. The accompanying drawings show embodiments and implementation examples in accordance with the principles of the present disclosure, but these are for the purpose of understanding the present disclosure and are never used for a limited interpretation of the present disclosure. is not. The description of the present specification is merely a typical example, and does not limit the scope of claims or application examples of the present disclosure in any sense.

In this embodiment, the description is given in sufficient detail for those skilled in the art to implement the present disclosure, but other implementations and embodiments are also possible and do not deviate from the scope and spirit of the technical idea of the present disclosure. It is necessary to understand that it is possible to change the structure and structure and replace various elements. Therefore, the following description should not be construed as limited to this.

[First Embodiment]
An example of the basic configuration of the image pickup apparatus according to the first embodiment will be described with reference to the overall configuration diagram of FIG. The image pickup device 1 includes an image sensor 101, an interpolation processing unit 102, a color information generation processing unit 103, a recognition processing unit 104, a color image generation processing unit 110, and an image recording unit 112.

The image sensor 101 is an image sensor such as a CMOS sensor or a CCD sensor for acquiring an optical image. These image sensors have photodiodes arranged in an array on a plane, and are configured to detect a light amount distribution on the plane by a plurality of pixels provided by these photodiodes. An image to be imaged can be obtained by detecting the light focused by a condensing member such as a lens or a mirror (not shown) with the image sensor 101.

The image pickup apparatus 1 of the first embodiment uses the image sensor of the RCCC pixel array shown in FIG. 2 as the image sensor 101. The RCCC pixel array includes R (Red) pixels 201 that receive the light of the color filter that transmits red light among the light in the visible range and can detect the red light, and the visible light that receives the light of the transparent color filter. It has two types of pixels, C (Clear) pixels 202, which are capable of detecting the total amount of light. A 2 × 2 pixel filter unit 203 composed of one R pixel 201 and three C pixels 202 is repeatedly arranged over a plurality of rows and a plurality of columns.

As shown in the graph of FIG. 3, the R pixel 201 is capable of receiving light in the wavelength range of red light, while the C pixel 203 is capable of receiving light in the wavelength range of visible light including blue light, green light, and red light. It is said that it can receive light in the entire area. The image output by the image sensor 101 is a grayscale image in which a pixel value based on the amount of red light detected by the R pixel 201 and a pixel value based on the total amount of visible light detected by the C pixel 202 are mixed.

The interpolation processing unit 102 obtains the amount of red light component light at the position of the C pixel 202 by interpolating it from the amount of detected light of the surrounding R pixel 201 based on the grayscale image output by the image sensor 101. In addition, the interpolation processing unit 102 interpolates and obtains the total amount of visible light at the position of the R pixel 201 from the detected light amount of the surrounding C pixel 202 based on the grayscale image output by the image sensor 101.

The processing of the interpolation processing unit 102 can be realized by, for example, the calculation shown in FIG. In the arithmetic processing shown in FIG. 4, the processing unit is 3 × 3 pixels centered on the pixel to be interpolated. When the interpolation process is executed in the image sensor 101 having the RCCC pixel array shown in FIG. 2, the interpolation process can be executed based on the four pixel patterns of patterns 1 to 4. Pattern 1 is a case where the lower right nine pixels of FIG. 2 are used as processing units, pattern 2 is a case where the lower left nine pixels of FIG. 2 are used as processing units, and pattern 3 is a case where the lower left nine pixels of FIG. 2 are used as processing units. The case where the nine pixels on the upper right of 2 are used as the processing unit, and the pattern 4 shows the case where the nine pixels on the upper left of FIG. 2 are used as the processing unit.

In each of pattern 1 to pattern 4, the pixel in the center of the 3 × 3 pixel is targeted for interpolation processing. For example, in patterns 2 to 4, since the central pixel is the C pixel 202, the interpolation calculation of the R component in the central C pixel 202 is performed. On the other hand, in the pattern 1, since the central pixel is the R pixel 201, the interpolation calculation of the C component in the central R pixel 201 is performed. The interpolation calculation of the pixel to be interpolated (the center of 3 × 3) is executed based on the pixel values of the pixels around the pixel to be interpolated.

In pattern 1, the R pixel 201 (R22) at the center of the 3 × 3 pixel is targeted for interpolation processing, and its C component (C22) is calculated based on the pixel values of the four C pixels 202 on the top, bottom, left, and right (C22). = (C12 + C32 + C12 + C32) / 4).
In pattern 2, since the C pixel 202 (C22) is at the center of the 3 × 3 pixel, its R component (R22) is calculated based on the pixel values of the two left and right R pixels 201 (R22 = (R212 + R23). ) / 2).
In pattern 3, the C pixel 202 (C22) is at the center of the 3 × 3 pixel, and its R component (R22) is calculated based on the pixel values of the two upper and lower R pixels 201 (R22 = (R12 + R32)). / 2).
In pattern 4, the C pixel 202 (C22) is at the center of the 3 × 3 pixel, and its R component (R22) is calculated based on the pixel values of the four R pixels 201 on the top, bottom, left, and right (R22 = (R11 + R13 + R31 + R33). ) / 4).

It goes without saying that this calculation is suitable for the case of the pixel array shown in FIG. 2, and different operations having the same meaning as above can be adopted for different pixel arrays. Further, although the above interpolation calculation is executed as the average value of the surrounding pixel values, a different calculation method (for example, a weighted average) can be adopted depending on the required performance of the imaging device, the state of the environment, and the like. Needless to say.

By repeating this process, an image (R image) showing the spatial distribution of the amount of red light and an image (C image) showing the spatial distribution of the total amount of visible light having the same number of pixels are generated. Can be done. Further, following the process shown in FIG. 4, the R image and the C image may be further subjected to an interpolation process for correcting the distortion of the image.

The color information generation processing unit 103 outputs the brightness information and the hue information of the captured image to the recognition processing unit 104 based on the C image and the R image generated by the interpolation processing unit 102. The recognition processing unit 104 recognizes a traffic signal, a road sign, a light of a vehicle in front, a white line on the road, an orange line, and the like with reference to the lightness component and the hue component. Hues are specified for traffic signals and the like. For example, traffic signal lights are bluish green (cyan), orange, and red, and road signs are often red, blue, white, and orange. As for the vehicle lights, the tail lamps and brake lamps are red, and the winkers are orange. Therefore, the recognition processing unit 104 is configured to be able to discriminate between these colors.

The color information generation processing unit 103 can calculate the brightness information of the captured image based on, for example, the C image generated by the interpolation processing unit 102.

The color information generation processing unit 103 has an R / C value comparison processing unit 105 inside. The R / C value comparison processing unit 105 receives the C image and the R image from the interpolation processing unit 102, and detects the R component detection light amount and the C component with respect to the pixels arranged at the same position of the C image and the R image. The ratio to the amount of light (R / C ratio) is calculated. Then, the R / C value comparison processing unit 105 discriminates the hue by comparing the R / C ratio with the reference value of the color to be discriminated, and outputs the discrimination result as hue information.

In the present embodiment, assuming that the above traffic signal, road sign, vehicle in front, and road surface line are recognized, the red determination reference value storage unit 106, the orange determination reference value storage unit 107, and the achromatic color determination standard The value storage unit 108 and the blue / green determination reference value storage unit 109 are provided. The four reference values stored in these storage units are compared with the R / C ratio described above. Although the hues of blue-green (cyan) of traffic signals and the blue hues of road signs are different, they can be distinguished by their brightness and existing position, and these hues are collectively treated as blue and green.

The color image generation processing unit 110 colorizes the image using the C image and the R image generated by the interpolation processing unit 102, and the R (Red) component image, the G (Green) component image, and the B (Blue) corresponding to the three primary colors. Generates and outputs a component image. The image recording unit 112 is a portion for accumulating the R (Red) component image, the G (Green) component image, and the B (Blue) component image generated by the color image generation processing unit 110, and is a flash memory, a hard disk, a DRAM, or the like. Consists of. The color image generation processing unit 110 of the first embodiment outputs the input R image as an R component image to the image recording unit 112, and has a CR image generation processing unit 111 inside. As the G component image and the B component image, an image based on the CR image generated by obtaining the difference value of each pixel of the C image and the R image is output by the CR image generation processing unit 111.

FIG. 5 is an example of actual measurement characteristics of the C pixel value and the R pixel value of the RCCC image obtained by photographing the color used in the road sign. More specifically, the actual measurement as a result of taking a color chart (X-Rite Colorchecker SG (registered trademark)) with a camera equipped with an image sensor of RCCC pixel arrangement under white light of five different brightnesses. This is a characteristic example. The C image and the R image are obtained as a result of the above-mentioned interpolation processing. In the graph of FIG. 5, a part close to the standard color of the safety color specified by JIS Z9103 used for the road sign is extracted from the above color chart (for example, red [row 3 column L], orange [row 6 column L]. ] ・ Yellow [row 4 column H] ・ white (achromatic color) [row 6 column J] ・ blue [row 3 column F]), plotting the relationship between the detected light amount of C pixel 202 and the detected light amount of R pixel 201 Is.

In FIG. 5, reference numerals 501 and 502 represent a plot based on the pixels of the red portion and a C vs. R characteristic for red light obtained by the least squares method, respectively. Further, reference numerals 503 and 504 represent C vs. R characteristics for orange light obtained by the plot based on the orange portion and the least squares method, respectively. Reference numerals 505 and 506 represent C vs. R characteristics for yellow light obtained by the plot based on the yellow part and the least squares method, respectively. Reference numerals 507 and 508 represent C vs. R characteristics for achromatic light obtained by the plot based on the white (achromatic) portion and the least squares method, respectively. 509 and 510 represent plots based on the blue part. From these data, the ratio (R / C) of the detected light amount of R pixel to the detected light amount of C pixel for each color is constant regardless of the brightness, and the recognition of road signs etc. for driving support by the in-vehicle camera. Since the ratios of the red, orange, yellow, white (achromatic color), and blue colors referred to in the subject are different, it is understood that these hues can be classified based on the R / C.

FIG. 6 is an example of actual measurement characteristics of the C pixel value and the R pixel value of the RCCC image obtained by photographing the color used in the traffic signal. FIG. 6 is also an example of actual measurement characteristics as a result of photographing the above-mentioned color chart with a camera equipped with an image sensor having an RCCC pixel arrangement under white light having five different brightnesses. FIG. 6 shows the detected light amount of C pixel and the detected light amount of R pixel by extracting the parts (red [row 3 column L], orange [row 6 column L], cyan [row 8 column B]) that are close to the color used in the traffic signal. This is a plot of the relationship between the amount of detected light. Reference numerals 501 and 502 and reference numerals 503 and 504 are the same as those in FIG. 5, and thus redundant description will be omitted. In FIG. 6, reference numerals 601 and 602 represent C vs. R characteristics for yellow light obtained by the plot based on the cyan part and the least squares method, respectively. From these data, it is possible to discriminate the hue of the signal color as in FIG.

FIG. 7 is a flowchart illustrating a procedure for obtaining hue information from an image captured by the image sensor 101, and is a process executed by the image sensor 101, the interpolation processing unit 102, and the color information generation processing unit 103 of FIG. is there.

First, in step 701, an image is captured and acquired by the image sensor 101. The acquired image has a pixel value based on the amount of red light detected by the R pixel 201 of the image sensor 101 of the RCCC pixel array shown in FIG. 2 and a pixel based on the total amount of visible light detected by the C pixel 202. It is a grayscale image in which values are mixed.

In the following step 702, the above-mentioned interpolation processing is executed for each pixel position of the grayscale image obtained in step 701, and an R image and a C image are generated. Then, in step 703, the ratio (R / C ratio) of the pixel value of the R image at the same pixel position and the pixel value of the C image is obtained, and in step 704, the hue is determined based on the value of the R / C ratio. ..

FIG. 8 is a flowchart showing the details of the procedure for determining the hue based on the pixel value ratio (R / C ratio) in step 704. In FIG. 8, Tr is the lower limit of the R / C ratio for discriminating red, To is the lower limit of the R / C ratio for discriminating orange, and Ty is the lower limit of the R / C ratio for discriminating yellow. The lower limit value and Tg indicate the lower limit value of the R / C ratio for distinguishing from achromatic colors (white, gray, black), respectively.

In step 801 the R / C ratio value and the lower limit value Tr are compared. If R / C> Tr (Yes), the hue of the pixel is determined to be red. If R / C ≦ Tr (No), the process proceeds to step 802.

In step 802, the value of the R / C ratio and the lower limit value To are compared. If R / C> To (Yes), the hue of the pixel is determined to be orange. If R / C ≦ To (No), the process proceeds to step 803.

In step 803, the value of the R / C ratio and the lower limit value Ty are compared. If R / C> Ty (Yes), the hue of the pixel is determined to be yellow. If R / C ≦ Ty (No), the process proceeds to step 804.

In step 804, the value of the R / C ratio and the lower limit value Tg are compared. If R / C> Tg (Yes), it is determined that the hue of the pixel is achromatic (white, gray, black).

When the determination of "No" is made in any of steps S801 to 804, the hue of the pixel is determined to be blue or green. With the above, the determination procedure of step S704 is completed.

FIG. 9 is a flowchart showing a procedure for generating a colorized image by the color image generation processing unit 110 from a grayscale image taken by the image sensor 101.

In the same manner as in FIG. 7, in steps 701 and 702, after the grayscale image is captured and acquired by the image sensor 101, the above-mentioned interpolation processing is executed for each pixel position of the grayscale image, and the R image and the R image and A C image is generated.

In the following step 901, the difference image (CR image) between the pixel value of the generated C image and the pixel value of the R image is generated in the CR image generation processing unit 111. Further, in step 902, the CR image generation processing unit 111 generates an image (α (CR)) obtained by multiplying the pixel value of the CR image by the distribution ratio α (0 ≦ α ≦ 1). It is made into a G component image.

Further, in step 903, an image ((1-α) (CR)) obtained by multiplying the pixel value of the CR image by (1-α) is generated by the CR image generation processing unit 111, and the B component image is generated. It is said that. The distribution ratio α is a value indicating the ratio of the G component images included in the difference image (CR). In the following, α may be referred to as a “first allocation ratio” and (1-α) may be referred to as a “second allocation ratio”.

The sum of the pixel values α (CR) given to the G component image in step 902 and the pixel values (1-α) (CR) given to the B component image in step 903 is CR. In other words, the sum of the first allocation ratio α and the second allocation ratio (1-α) is 1. In the relationship between the wavelength-sensitivity characteristics of the R pixel 201 and the C pixel 202 shown in FIG. 3, the wavelength range in which the C pixel 202 has sensitivity is wider than that of the R pixel 201 by the amount of the G component and the B component, and the C Since it is estimated that the amount of detected light of the pixel 202 and the R pixel 201 is larger by the sum of the G component and the B component, the above relationship is established. It is also possible to add a process of increasing or decreasing the brightness of each component (not shown) to the obtained R component image, G component image, and B component image.

FIG. 10 is a diagram (graph) for explaining the first example of the distribution ratio α between the G component image and the B component image. In the graph of FIG. 10, the vertical axis represents the value of the R / C ratio (0 or more and 1 or less), and the horizontal axis is the ratio of the G component image to the sum (G + B) of the G component image and the B component image, that is, the first. The distribution ratio α (0 or more and 1 or less) is shown. In this graph, the approximate distribution of each hue in the space indicated by the vertical axis and the horizontal axis is shown by a dotted line.

As shown in FIG. 10, when only the magnitude of the R / C ratio value is used as a factor, it is possible to determine whether the pixel is red, cyan or gray. However, the hues arranged in the horizontal direction in the graph of FIG. 10 cannot be discriminated only by the R / C ratio. For example, blue, cyan, and green cannot be distinguished from each other. Further, pink and yellow (or orange) cannot be discriminated only by the magnitude of the R / C ratio.

FIG. 10 illustrates the discrimination of hue when the value of α is fixed at 0.5. That is, in FIG. 10, the locus 1001 shows a change in the R / C ratio, and is fixed at α = 0.5 regardless of the R / C ratio. As the R / C ratio increases from 0 to 1, the hue of the pixel changes in that order, such as cyan, achromatic, neutral between pink and orange, and red. When the imaging device of the first embodiment is applied to an in-vehicle camera, particularly the color of the clear sky and the blue light (cyan), the color of the road surface and the white line (achromatic color), the red light and the tail lamp of the vehicle (red) It is reproduced well. Therefore, based on the image sensor 101 having an RCCC pixel array, it is possible to obtain a colorized image with less visual discomfort.

FIG. 11 is a diagram (graph) for explaining a second example of the distribution ratio α between the G component image and the B component image. The difference from FIG. 10 is that the distribution ratio α is changed so that the distribution ratio α changes according to the change in the R / C ratio, at least when the R / C ratio is within a predetermined range. This is because when the image pickup device 1 is used as an in-vehicle camera, orange color appears more frequently than pink color in the captured image.

The locus 1101 in FIG. 11 shows the change in the R / C ratio. In FIG. 11, when the R / C ratio is smaller than, for example, the first value (for example, R / C <0.25), the distribution ratio α is fixed to a constant value (for example, α = 0.5). .. On the other hand, when the R / C ratio is equal to or greater than the first value and equal to or less than the second value, the larger the R / C ratio, the larger the distribution ratio α. Then, when the R / C ratio is larger than the second value (for example, R / C> 0.75), the distribution ratio α is fixed at a constant value (for example, α = 1). The function showing the locus 1101 can be stored in a storage unit (not shown) in the color image generation processing unit 110. It goes without saying that the locus 1101 in FIG. 11 is just an example, and the shape of the locus 1101 can be changed as appropriate.

When the distribution ratio α is changed according to the increase in the R / C ratio as in this locus 1101, the hue of the pixel is cyan or gray at the stage where the R / C ratio is small (α = 0.5). Can be determined. On the other hand, when the R / C ratio is equal to or greater than the first value and equal to or less than the second value, the distribution ratio α increases as the R / C ratio increases. As a result, the hue assigned to the pixel having the R / C ratio changes from cyan to achromatic to orange to red. When the image pickup device 1 is used as an in-vehicle camera, the color of the clear sky and the green signal (cyan), the color of the road surface and the white line (achromatic color), the red signal and the tail lamp (red) are reproduced particularly well. In addition, the reproducibility of colors such as orange lines, yellow signals, and winkers on the road surface can be improved as compared with the example of FIG.

It is preferable that the locus 1101 in FIG. 11 is set to be a continuous curve over the entire area where the R / C ratio changes from 0 to 1. This is because if the locus 1101 has a discontinuous curve, the change in hue becomes large in the vicinity of the discontinuous sky, causing a problem that noise becomes large.

As described above, according to the image pickup apparatus 1 and the image processing method according to the first embodiment, even when an image sensor having two types of pixels such as an RCCC pixel array is applied, a predetermined image sensor is applied. It becomes possible to accurately discriminate hues. In the case of the RCCC pixel array, it is possible to achieve both a wide angle of view and an improvement in spatial resolution as compared with the RGGB pixel array, but the information obtained regarding the color is usually limited. However, in this first embodiment, the effect of being able to generate an image reflecting the color of the object can be obtained, similar to the image pickup device to which the image sensor of the RGGB arrangement is applied.

In the first embodiment, an example of the RCCC pixel array is shown, but if the pixel array consists of two types of pixels, a pixel having a red color filter and a pixel having a transparent color filter, the ratio of the number of pixels is It does not have to be 1: 3. Further, in order to place importance on the recognition accuracy for a red object (traffic signal, tail lamp, sign) for in-vehicle camera applications, there are two types of embodiments, one having a red color filter and the other having a transparent color filter. As shown, a pair of a pixel having a non-red color filter and a pixel having a transparent color filter may be used depending on the application.

[Second Embodiment]
Next, the image pickup apparatus and the image processing method according to the second embodiment will be described with reference to FIGS. 12 and 13. FIG. 12 is a block diagram showing the overall configuration of the image pickup apparatus of the second embodiment. In FIG. 12, the components common to the apparatus of the first embodiment are designated by the same reference numerals as those in FIG. 1, and therefore, duplicate description will be omitted below.

The second embodiment is different from the first embodiment in the configuration of the color information generation processing unit 103. The color information generation processing unit 103 of the second embodiment includes an address generation unit 1201 and a color information generation table 1202.

The address generation unit 1201 is configured to input a C image and an R image and output the corresponding address signal. Specifically, the address generation unit 1201 generates an address signal corresponding to a set of pixel values at the same pixel position of the input C image and R image, and outputs the address signal to the color information generation table 1202.

The color information generation table 1202 stores an address signal and brightness information and hue information corresponding to the address signal as a table. Then, the color information generation table 1202 identifies and outputs the corresponding brightness information and hue information based on the address information input from the address generation unit 1201.

For example, the data structure shown in FIG. 13 is applied to the color information generation table 1202. In the example of FIG. 13, the address signal supplied from the address generation unit 1201 connects the R pixel value and the C pixel value to the upper bit and the lower bit like {R, C}. The generated data is assigned so as to be applied, and as the corresponding data, the R pixel value corresponding to the address and the hue information corresponding to the R / C ratio based on the C pixel value are stored. According to such a configuration, hue information can be easily generated without requiring complicated calculations.

Further, in the second embodiment, it is possible to easily generate color information including not only the hue classification based on the R / C ratio but also the brightness classification based on the level of the C pixel value. Further, it is possible to obtain an effect that it becomes possible to determine complicated color information, such as changing the threshold value of the division of the R / C ratio value and the number of divisions according to the level of the C pixel value.

[Third Embodiment]
Next, the image pickup apparatus and the image processing method according to the third embodiment will be described with reference to FIGS. 14 and 15. FIG. 14 is a block diagram showing the overall configuration of the image pickup apparatus according to the third embodiment. In FIG. 13, since the constituent members common to the apparatus of the first embodiment are designated by the same reference numerals as those in FIG. 1, duplicate description will be omitted below.

In the image pickup apparatus of the third embodiment, the configuration of the color image generation processing unit 110 is different from that of the first embodiment. The color image generation processing unit 110 of the third embodiment is configured to be able to obtain a color image with less visual discomfort even when the detected light amount of the pixel exceeds the upper limit value. In the image generation process in the image pickup apparatus of the above-described embodiment, an appropriate color image can be obtained in the case where the detected light amount of the pixel does not exceed the upper limit and is not saturated, but the saturated portion is reddish than in reality. It becomes a hue. This is because the difference between the C pixel value and the R pixel value becomes smaller than the actual value at the saturated portion, and the values given to the G component and the B component become relatively small. In the third embodiment, this phenomenon can be effectively suppressed.

As shown in FIG. 14, the color image generation processing unit 110 of the third embodiment includes the brightness saturation pixel determination processing unit 1401 in addition to the CR image generation processing unit 111 similar to that of the first embodiment. It includes a saturated pixel replacement processing unit 1402, an R component brightness correction unit 1403, a G component brightness correction unit 1404, and a B component brightness correction unit 1405. The R component brightness correction unit 1403, the G component brightness correction unit 1404, and the B component brightness correction unit 1405 together constitute a brightness correction unit.

The function of the CR image generation processing unit 111 is the same as that of the first embodiment (FIG. 1). When the detected light amount of the pixel in the C image exceeds the upper limit value, the brightness saturation pixel determination processing unit 1401 determines that the brightness of the pixel is a saturated pixel. It should be noted that it is also possible to configure a pixel having a brightness near the upper limit value to be determined to be a saturated pixel, although the amount of detected light does not exceed the upper limit value.

The saturated pixel replacement processing unit 1402 corresponds the brightness of the R component, the G component, or the B component to the pixel determined to be saturated based on the determination result of the brightness saturation pixel determination processing unit 1401. Switch to the upper limit and output. That is, the saturated pixel replacement processing unit 1402 has a function of replacing each component value with an upper limit value so that the saturated pixel is treated as white.

The brightness of the pixels of the R image is input to the saturated pixel replacement processing unit 1402 after being corrected by the R component brightness correction unit 1403. Similarly, the saturation pixel replacement processing unit 1402 is input after the brightness of the pixels of the G image and the B image is corrected by the G component brightness correction unit 1404 and the B component brightness correction unit 1405, respectively.

When 8 bytes (255 bits) are assigned to each of the R component, the G component, and the B component, the total value of RGB representing the brightness is 255 × 3 = 765 bits with respect to white. On the other hand, 255 bits are similarly assigned to the C image input to the CR image generation processing unit 111, and there is a three-fold difference in the number of bits.

Therefore, when the output of the CR image generation processing unit 111 is directly input to the saturated pixel replacement processing unit 1402, the brightness of the upper limit of the saturated pixel and the brightness of the saturated pixel adjusted to the upper limit value are obtained. A step of about 3 times the brightness is generated, and the image becomes visually uncomfortable. Therefore, the color image generation processing unit 110 of the third embodiment includes an R component brightness correction unit 1403, a G component brightness correction unit 1404, and a B component brightness correction unit 1405, and the brightness of each component is high. After correcting in the direction of, it is supplied to the saturated pixel replacement processing unit 1402. As an example, the correction amount β in the R component brightness correction unit 1403, the G component brightness correction unit 1404, and the B component brightness correction unit 1405 is a value of 1 or more, and β≈3 can be preferably set. When the brightness of the R component image, the G component image, and the B component image adjusted to β times exceeds each upper limit value, the saturation pixel replacement processing unit 1402 may adjust the brightness so as to reach the upper limit value.

FIG. 15 is a flowchart showing a procedure for generating a colorized image by the color image generation processing unit 110 from the image taken by the image sensor 101. Steps 701, 702, 703, 901, 902, and 903 are the same as those in FIG. 9, so duplicate description will be omitted.

In step 1501 after step 903, the brightness saturation pixel determination processing unit 1401 determines the existence and position of saturated pixels (pixels having brightness equal to or higher than the upper limit value) in the C image.

In the following step 1502, the pixel values (brightness) of the R component image, the G component image, and the B component image are multiplied by β (β) in the R component brightness correction unit 1403, the G component brightness correction unit 1404, and the B component brightness correction unit 1405. ≧ 1). As a result, the difference in brightness when the saturated pixels are replaced with white is reduced.

Then, in step 1503, the pixel values of the R component image, the G component image, and the B component image at the positions of the saturated pixels are replaced with the respective upper limit values. By this step 1503, the portion where the pixel value is saturated and the red component is displayed strongly in a pseudo manner is replaced with white. According to the apparatus configuration and the processing method shown above, in addition to the effects shown in the first embodiment, a color image with less visual discomfort can be obtained even when the detected light amount of the pixel exceeds the upper limit value. The effect is obtained.

[Fourth Embodiment]
Next, the image pickup apparatus and the image processing method according to the fourth embodiment will be described with reference to FIGS. 16 and 17. As shown in FIG. 16, in this fourth embodiment, the image sensor 101 has a pixel array different from the RCCC pixel array. In this image sensor 101, one filter unit 203 has two types of pixels, a Cy pixel 1601 for detecting cyan color, which is a complementary color of red, and a C pixel 202, and one Cy pixel 1601 and three C pixels. This is a pixel arrangement (CyCCC pixel arrangement) in which a 2 × 2 pixel filter unit 203 composed of 202 is repeated over a plurality of rows and a plurality of columns.

As shown in FIG. 17, the Cy pixel 1601 has sensitivity to blue light and green light. The difference image (C-Cy) between the C image and the Cy image is an R image. By executing the same processing as in the first embodiment on the C image and R image thus obtained, the same effect as in the first embodiment can be obtained. Since the Cy pixel 1601 has sensitivity to blue light and green light, the sensitivity of the image sensor 101 to green light and blue light is improved as compared with the RCCC pixel array. When white light is incident on the image sensor 101, a large amount of light is detected as compared with the R pixel 201. Therefore, as a tendency of the entire image including subjects of various hues, the Cy pixel 1601 has a higher S than the R pixel 201. The / N ratio can be given, and as a result, the accuracy of hue discrimination can be expected to be improved as compared with the first embodiment.

The present invention is not limited to the above-mentioned examples, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

101 ... image sensor, 102 ... interpolation processing unit, 103 ... color information generation processing unit, 104 ... recognition processing unit, 110 ... color image generation processing unit, 111 ... CR image generation processing unit, 112 ... image recording unit, 201 ... R pixel, 202 ... C pixel, 203 ... Filter unit, 1201 ... Address generation unit, 1202 ... Color information generation table, 1401 ... Brightness saturated pixel determination processing unit, 1402 ... Saturated pixel replacement processing unit, 1403 ... R component brightness correction 1404 ... G component brightness correction unit, 1405 ... B component brightness correction unit, 1601 ... Cy pixel.

Claims (15)

1. A first pixel for detecting light in the first wavelength range in the visible region, and a second pixel for detecting light having a visible light wavelength different from the first wavelength range in addition to the light in the first wavelength range. An image sensor composed of repeatedly arranging filter units including the pixels of
   Based on the detected light amount of the first pixel, the position of the first pixel is determined based on the first interpolated image obtained by interpolating the position of the second pixel and the detected light amount of the second pixel. An interpolation processing unit configured to be able to generate a second interpolated image obtained by interpolation, and
   It is characterized by including a color information generation processing unit that determines the hue at the position based on the detected light amount of a set of pixels at the same position of the first interpolated image and the second interpolated image. Imaging device.
2. The first aspect of the present invention, wherein the color information generation processing unit determines the hue at the position based on the ratio of the detected light amounts of the set of pixels at the same position of the first interpolated image and the second interpolated image. Imaging device.
3. A color image generation processing unit that generates a color image based on the first interpolated image and the second interpolated image is further provided.
   The color image generation processing unit
   Based on the first interpolated image, a first component image having a first wavelength component in the first wavelength range is generated.
   A difference image, which is the difference between the first interpolated image and the second interpolated image, is generated.
   The difference image is multiplied by the first distribution ratio to generate a second component image having a component in a second wavelength range different from the first wavelength range.
   The difference image is multiplied by a second distribution ratio to generate a third component image having components in the first wavelength range and a third wavelength range different from the second wavelength range. The imaging device according to claim 1 or 2.
4. The first distribution ratio or the second distribution ratio is set to a constant value regardless of the change in the ratio of the detected light amount of the set of pixels at the same position in the first interpolated image and the second interpolated image. The imaging device according to claim 3.
5. According to claim 3, the first distribution ratio or the second distribution ratio changes according to a change in the ratio of detected light amounts of a set of pixels at the same position in the first interpolated image and the second interpolated image. The imaging device described.
6. The first pixel is a pixel capable of detecting red light, and the second pixel is a pixel capable of detecting red light, green light, and blue light, according to any one of claims 1 to 3. The imaging device described.
7. The first pixel is a pixel capable of detecting blue light and green light.
   The imaging device according to any one of claims 1 to 3, wherein the second pixel is a pixel capable of detecting red light, green light, and blue light.
8. A first pixel for detecting light in the first wavelength range in the visible region, and a second pixel for detecting light having a visible light wavelength different from the first wavelength range in addition to the light in the first wavelength range. An image sensor composed of repeatedly arranging filter units including the pixels of
   Based on the detected light amount of the first pixel, the position of the first pixel is determined based on the first interpolated image obtained by interpolating the position of the second pixel and the detected light amount of the second pixel. An interpolation processing unit configured to be able to generate a second interpolated image obtained by interpolation, and
   It is provided with a color image generation processing unit that generates a color image based on the first interpolated image and the second interpolated image.
   The color image generation processing unit
   Based on the first interpolated image, a first component image having a first wavelength component in the first wavelength range is generated.
   A difference image, which is the difference between the first interpolated image and the second interpolated image, is generated.
   The difference image is multiplied by the first distribution ratio to generate a second component image having a component in a second wavelength range different from the first wavelength range.
   It is configured to multiply the difference image by a second distribution ratio to generate a third component image having components in the first wavelength range and a third wavelength range different from the second wavelength range. An imaging device characterized by.
9. The color image generation processing unit
   In the second interpolated image, the brightness saturation pixel determination processing unit for determining the saturated pixels whose brightness is saturated, and the brightness saturation pixel determination processing unit.
   The imaging device according to claim 8, further comprising a saturated pixel replacement processing unit that replaces the brightness of the saturated pixels with an upper limit value.
10. The saturated pixel replacement processing unit is
    The brightness of the first interpolated image, the image obtained by multiplying the difference image by the first distribution ratio, and the image obtained by multiplying the difference image by the second distribution ratio is corrected. Further equipped with a brightness correction unit,
    The saturated pixel replacement processing unit replaces the brightness of the saturated pixels included in the image after the brightness correction by the brightness correction unit with the upper limit value.
    The imaging device according to claim 9.
11. A first pixel for detecting light in the first wavelength range in the visible region, and a second pixel for detecting light having a visible light wavelength different from the first wavelength range in addition to the light in the first wavelength range. The step of acquiring an image from an image sensor configured by repeatedly arranging filter units including the pixels of
    A step of interpolating the position of the second pixel based on the detected light amount of the first pixel to acquire a first interpolated image, and
    A step of interpolating the position of the first pixel based on the detected light amount of the second pixel to acquire a second interpolated image, and
    An image processing method including a step of determining a hue at a position based on the amount of detected light of a set of pixels at the same position of the first interpolated image and the second interpolated image.
12. 11. The step of determining the hue is according to claim 11, wherein the hue at the position is determined based on the ratio of the detected light amounts of the set of pixels at the same position in the first interpolated image and the second interpolated image. Image processing method.
13. Further comprising a step of generating a color image based on the first interpolated image and the second interpolated image.
    The step of generating the color image is
    Based on the first interpolated image, a first component image having a first wavelength component in the first wavelength range is generated.
    A difference image, which is the difference between the first interpolated image and the second interpolated image, is generated.
    The difference image is multiplied by the first distribution ratio to generate a second component image having a component in a second wavelength range different from the first wavelength range.
    Claim 11 or 12 to multiply the difference image by a second distribution ratio to generate a third component image having components in the first wavelength range and a third wavelength range different from the second wavelength range. The image processing method described in.
14. The first distribution ratio or the second distribution ratio is set to a constant value regardless of the change in the ratio of the detected light amount of the set of pixels at the same position in the first interpolated image and the second interpolated image. The image processing method according to claim 13.
15. 13. The first distribution ratio or the second distribution ratio changes according to a change in the ratio of the detected light amounts of the pair of pixels at the same position of the first interpolated image and the second interpolated image, according to claim 13. The image processing method described.

Images