WO2011162155A1

WO2011162155A1 - Image capturing device

Info

Publication number: WO2011162155A1
Application number: PCT/JP2011/063794
Authority: WO
Inventors: 清高
Original assignee: コニカミノルタオプト株式会社
Priority date: 2010-06-23
Filing date: 2011-06-16
Publication date: 2011-12-29
Also published as: JPWO2011162155A1

Abstract

Disclosed is an image capturing device capable of reproducing the original color of a subject even in night when infrared light is dominant and obtaining an image with less sense of discomfort. If an attention unit pixel portion is an infrared pixel portion, a color space conversion unit creates a color difference signal of the attention unit pixel portion on the basis of an image signal of a surrounding unit pixel portion. More specifically, according to the following equations ((3a), (3b)), the color difference signals Cb(x₀, y₀), Cr(x₀, y₀) of the attention unit pixel portion are obtained by the weighted average of the color differences Cb(x, y), Cr(x, y) of a unit pixel portion surrounding the attention unit pixel portion, where the symbols x₀, y₀ are coordinates of the attention unit pixel portion and the symbols x, y are coordinates of the surrounding unit pixel portion. Cb(x₀, y₀) = Σk(x, y)·a(x, y)·Cb(x, y) (3a) Cr(x₀, y₀) = Σk(x, y)·a(x, y)·Cr(x, y) (3b)

Description

Imaging device

The present invention relates to an imaging apparatus capable of performing image processing on original image data obtained by a solid-state imaging device.

In recent years, a night vision video support system has been developed that irradiates near-infrared illumination light in front of a vehicle or the like and images a subject with a camera having sensitivity to near-infrared light. Here, the conventional night vision video support system requires a high-quality image, but the near-infrared wavelength component is dominant in the subject light, so that sufficient color information cannot be obtained, so a monochrome image is obtained. May be generated, and the visibility of the driver may deteriorate due to a difference from the actual subject image. On the other hand, techniques as shown in

Patent Documents

1 and 2 have been developed.

JP 2006-148690 A JP 2007-264628 A

According to the technique of Patent Document 1, the level value of the visible light luminance signal and the level value of the near-infrared luminance signal are compared to determine whether it is the visible region or the infrared region, and the block determined as the visible region For a block determined as an infrared region, a monochrome image is generated based on the near-infrared luminance signal in the block. ing. However, the technique of Patent Document 1 has a problem that the visibility of the driver is lowered because color and monochrome areas are mixed in the same image and are different from an actual subject image.

On the other hand, according to the technique of Patent Document 2, an opening having transparency to visible light and infrared light and a non-opening having transparency to visible light and non-transmission to infrared light. An infrared light cut filter layer provided with a portion and a color filter group for separating the visible light region into R, G, and B components are arranged so as to be integrated on the solid-state imaging device and pass through the opening. The wide wavelength region component including visible light and infrared light is detected by the wavelength region pixel, a luminance signal is generated by the detection signal, and each color component of R, G, B that has passed through the non-opening portion is detected by each color pixel. A color difference signal is generated from each color signal detected.

Here, since the visible light quantity from the subject is very small in the infrared region, the original color of the subject may not be reflected correctly and may be buried in noise. On the other hand, the luminance signal in the infrared region tends to increase in value when infrared light is included, so the color difference signal is processed to increase accordingly. In the processed image, the color becomes bright, that is, noise may be emphasized. As a result, in night photography or the like, the color information of the subject is buried in the emphasized noise and may not be correctly identified.

An object of the present invention is to provide an imaging apparatus capable of reproducing the original subject color and obtaining an image with little discomfort even at night when infrared light is dominant.

In the imaging apparatus of the present invention, at least three types of pixels having different spectral sensitivities are arranged, of which at least one type of pixel is a pixel having sensitivity in the infrared region, and includes a visible light component and an infrared light component. A solid-state image sensor that can be detected, and luminance information of the visible light component obtained from the solid-state image sensor when the subject is imaged by the solid-state image sensor, and infrared light information based on the infrared light component. And a color image generation means for generating a color image based on the image data.

According to the present invention, when the color image generating unit images a subject with the solid-state imaging device, the luminance information of the visible light component obtained from the solid-state imaging device and the infrared based on the infrared light component. Based on the light information, for example, a color image is generated by weighting a color difference signal or the like, so that the color of the original subject can be reproduced, and a color image with less discomfort can be obtained. The “luminance information of the visible light component” is, for example, a luminance signal value in a visible light signal, and the “infrared light information” is, for example, an infrared light signal value.

The imaging apparatus according to the present invention, when a solid-state imaging device in which a plurality of pixels capable of detecting a visible light component and an infrared light component from a subject are two-dimensionally arranged and when imaging a subject by the solid-state imaging device, The color difference signal of the visible light component detected by the adjacent unit pixel unit adjacent to the unit pixel unit of interest is the color difference signal of the visible light component of the unit of interest pixel unit in which the detected luminance signal of the visible light component is equal to or less than a predetermined value. And a color image generation means for generating a color image using the nominal color difference signal in place of the nominal color difference signal derived based on the above.

According to the present invention, when the color image generation unit images a subject with the solid-state imaging element, the color signal of the visible light component of the target unit pixel unit in which the detected luminance signal of the visible light component is equal to or less than a predetermined value. Is replaced with the nominal color difference signal derived based on the color difference signal of the visible light component detected by the adjacent unit pixel portion adjacent to the target unit pixel portion, and a color image is generated using the nominal color difference signal. Even when the color information from the unit pixel unit of interest cannot be obtained, the color of the original subject can be reproduced and a color image with a little uncomfortable feeling can be obtained. The proximity unit pixel unit means a unit pixel unit adjacent to at least the target unit pixel unit. The visible light component being equal to or less than a predetermined value means, for example, a case where the luminance signal of the visible light component is lower than the infrared light signal.

According to one aspect of the present invention, it is preferable that the proximity unit pixel unit is a unit pixel unit arranged around the unit pixel unit of interest. Thereby, the color information which does not largely deviate from the original color information of the unit pixel unit of interest can be obtained.

According to an aspect of the present invention, when there are a plurality of the proximity unit pixel units, the color image generation unit may determine whether the plurality of proximity unit pixel units correspond to the distance between the target unit pixel unit and the proximity unit pixel unit. The nominal color difference signal is derived by weighting each color difference signal of the visible light component detected in the unit pixel unit according to the degree of association between the target unit pixel unit and the adjacent unit pixel unit and performing weighted averaging. It is preferable. Weighted averaging is performed according to the degree of association between the target unit pixel unit and the proximity unit pixel unit, thereby suppressing the involvement of the proximity unit pixel unit having low relevance and improving color reproducibility. Can do. “Weighting according to the degree of association between the target unit pixel portion and the proximity unit pixel portion” means, for example, weighting according to the distance between the target unit pixel portion and the proximity unit pixel portion.

According to one aspect of the present invention, it is preferable to increase the weighting coefficient as the distance between the unit pixel unit of interest and the neighboring unit pixel unit is shorter. It is considered that the proximity unit pixel unit that is more distant from the unit pixel unit of interest is more likely to have a different color, so it is desirable to reduce its involvement.

According to an aspect of the present invention, the color image generation unit is configured to perform the weighting according to a difference or ratio between a luminance signal in a visible light component detected by the same proximity unit pixel unit and an infrared light signal. It is preferable to change the coefficient.

According to an aspect of the present invention, the color image generation unit increases the weighting coefficient when the luminance signal in the visible light component is greater than or equal to the infrared light signal, and the luminance signal in the visible light component is If it is smaller than the infrared light signal, it is preferable to reduce the weighting coefficient. This is because when the luminance signal in the visible light component is larger, the image signal from the unit pixel unit of interest is more likely to have original color information.

According to one aspect of the present invention, when the luminance signal in the visible light component is larger, the weighting coefficient is 1, and when the infrared light signal is lower, the weighting coefficient is 0. This is preferable.

According to an aspect of the present invention, the proximity unit pixel unit is located within a predetermined range centered on the target unit pixel unit, and the color image generation unit includes the visible light component within the predetermined range. If the number of the unit pixel portions whose luminance signal is equal to or higher than the infrared light signal is less than a predetermined number, expanding the predetermined range is effective even when there is little visible light detected within the predetermined range. is there. Here, the predetermined number is, for example, the number of the unit pixel units in which the luminance signal in the visible light component is smaller than the infrared light signal.

According to an aspect of the present invention, the color image generation unit divides the imaging surface of the solid-state imaging device into a plurality of blocks, and averages the luminance signal in the visible light component detected from the unit pixel unit for each block. An average value of a color difference signal and an average value of an infrared light signal, and the proximity unit according to a distance between the target unit pixel unit and the block around the block including the target unit pixel unit Instead of the color difference signal detected by the pixel, the average value of the block is obtained by deriving the nominal color difference component of the unit pixel portion of interest by weighting and averaging the average values of the color difference signals in the surrounding blocks. Therefore, it is possible to suppress generation of a color image that is extremely colored.

According to one aspect of the present invention, in the color image generation unit, the average value of the luminance signal in the visible light component of the block including the unit pixel portion of interest is lower than the average value of the infrared light signal of the same block. Only in this case, it is preferable to perform weighted averaging by weighting the average values of the color difference signals in the surrounding blocks instead of the color difference signals detected by the adjacent unit pixels. This is because if the average value of the luminance signal in the visible light component of the block is equal to or higher than the average value of the infrared light signal of the same block, there is a high possibility that sufficient color information can be obtained in the block.

According to the image pickup apparatus of the present invention, it is possible to reproduce an original subject color and obtain an image with a little discomfort even at night when infrared light is dominant.

It is a block diagram of imaging device 1 concerning this embodiment. 3 is a diagram illustrating an arrangement of pixels of a solid-state imaging element 3. FIG. It is the figure which showed the spectral transmission characteristic of Ye, R, and IR filter, the vertical axis | shaft has shown the transmittance | permeability (sensitivity), and the horizontal axis has shown the wavelength (nm). 3 is a block diagram illustrating a detailed configuration of an image processing unit 4. FIG. It is a figure which cuts off and shows a part of imaging surface of the solid-state image sensor 3. FIG. It is a graph which shows an example of k (x, y). It is a graph which shows an example of S (x, y). It is a graph which shows an example of L (x, y). FIG. 3 is a diagram illustrating a part of pixels of the solid-state imaging device 3 cut out. It is a figure which divides | segments the imaging surface of the solid-state image sensor 3 into the block BL (for example, unit pixel part of 10x10) of the predetermined pixel number. It is a figure which divides | segments the imaging surface of the solid-state image sensor 3 into the block BL (for example, unit pixel part of 10x10) of the predetermined pixel number.

Hereinafter, the imaging apparatus 1 according to the embodiment of the present invention will be described. FIG. 1 is a block diagram of an imaging apparatus 1 according to the present embodiment. As shown in FIG. 1, the imaging apparatus 1 is close to the subject when the subject brightness is low, such as a lens 2, a solid-state imaging device (hereinafter also referred to as an imaging device) 3, an image processing unit 4, a control unit 5, and the night. A near-infrared light-emitting unit 6 that irradiates infrared light and a drive unit 7 that drives the near-infrared light-emitting unit 6 according to a control signal from the control unit 5 are provided. Here, the imaging device 1 is mounted on a vehicle, for example, and is used for imaging a subject around the vehicle, but the usage is not limited thereto.

The lens 2 is composed of an optical lens system that captures an optical image of a subject and guides it to the image sensor 3. As the optical lens system, for example, a zoom lens, a focus lens, other fixed lens blocks, and the like arranged in series along the optical axis L of the optical image of the subject can be employed. The lens 2 includes a diaphragm (not shown) for adjusting the amount of transmitted light, a shutter (not shown), and the like, and the driving of the diaphragm and the shutter is controlled under the control of the control unit 5.

The image pickup device 3 includes a light receiving unit composed of a PD (photodiode), an output circuit that outputs a signal photoelectrically converted by the light receiving unit, and a drive circuit that drives the image pickup device 3, and has a level corresponding to the amount of light. Generate original image data. Here, as the imaging device 3, various imaging sensors such as a CMOS image sensor, a VMlS image sensor, and a CCD image sensor can be employed.

In the present embodiment, the image sensor 3 receives a light image of a subject, converts a visible color image component by a pixel having a color filter, and converts and outputs an infrared image component by a pixel having an infrared filter. A luminance image component including a visible luminance image component and an infrared image component is converted and output by a pixel not provided with a filter, but the present invention is not limited to this.

The image processing unit 4 includes an arithmetic circuit and a memory used as a work area for the arithmetic circuit, A / D converts the original image data output from the image sensor 3 to convert it into a digital signal, and performs image processing to be described later. After execution, for example, the data is output to a memory or a display device (not shown).

The control unit 5 includes a CPU and a memory that stores a program executed by the CPU, and controls the entire imaging apparatus 1 in response to an external control signal.

FIG. 2 is a diagram showing an arrangement of pixels of the solid-state image sensor 3. As shown in FIG. 2, the image pickup device 3 includes a unit pixel unit 31 including a Ye pixel, an R pixel, an IR pixel, and a W pixel having a visible wavelength region and an infrared wavelength region as sensitive wavelength bands. Is arranged. For example, “Ye” pixel means a pixel having a “Ye” filter, and so on.

As shown in FIG. 2, in the unit pixel unit 31, R pixels are arranged in the first row and first column, IR pixels are arranged in the second row and first column, and W pixels are arranged in the first row and second column. R pixels, IR pixels, W pixels, and Ye pixels are arranged in a staggered manner, such that Ye pixels are arranged in the second row and the second column. However, this is only an example, and the R pixel, IR pixel, W pixel, and Ye pixel may be arranged in a zigzag pattern in another pattern.

Since the Ye pixel includes a Ye filter, it outputs an image component Ye and an infrared image component, which are visible color image components of Ye. Since the R pixel includes an R filter, it outputs an image component R and an infrared image component which are visible color image components of R. Since the IR pixel includes an IR filter, it outputs an image component IR that is an infrared image component. Since the W pixel does not include a filter, an image component W that is a luminance image component including a visible luminance image component and an image component IR is output.

FIG. 3 is a diagram showing the spectral transmission characteristics of the Ye, R, and IR filters, where the vertical axis shows the transmittance (sensitivity) and the horizontal axis shows the wavelength (nm). A graph indicated by a dotted line shows the spectral sensitivity characteristics of the pixel in a state where the filter is removed. It can be seen that this spectral sensitivity characteristic has a peak near 600 nm and changes in a convex curve. In FIG. 3, the visible wavelength region is 400 nm to 700 nm, the infrared wavelength region is 700 nm to 1100 nm, and the sensitive wavelength band is 400 nm to 1100 nm. The light in the visible wavelength region is the visible light component, and the light in the infrared wavelength region is the infrared light component.

As shown in FIG. 3, the Ye filter has a characteristic of transmitting light in the sensitive wavelength band excluding the blue region in the visible wavelength region. Therefore, the Ye filter mainly transmits yellow light and infrared light.

The R filter has a characteristic of transmitting light in a sensitive wavelength band excluding a blue region and a green region in the visible wavelength region. Therefore, the R filter mainly transmits red light and infrared light.

The IR filter has a characteristic of transmitting light in a sensitive wavelength band excluding the visible wavelength region, that is, in the infrared wavelength band. W indicates a case where no filter is provided, and all light in the sensitive wavelength band of the pixel is transmitted.

To achieve similar characteristics, Ye, M (magenta) + IR, C (cyan) + IR instead of Ye, R and IR (however, M + IR blocks only green and C + IR only red) Is also possible. However, the R pixel, the IR pixel, and the Ye pixel can make the spectral transmission characteristics steep, and, for example, the spectral transmission characteristics are better than those of the M + IR filter and the C + IR filter. That is, each of the M + IR filter and the C + IR filter has a characteristic of shielding only the green region and the red region, which are partial regions in the center, in the sensitive wavelength band. It is difficult to provide steep spectral transmission characteristics such as IR filters and Ye filters. Therefore, each of the M + IR filter and the C + IR filter cannot extract the RGB image components with high accuracy even if the calculation is performed. Therefore, by configuring the image sensor 3 with R pixels, IR pixels, Ye pixels, and W pixels, the performance of the image sensor 3 can be improved.

FIG. 4 is a block diagram showing a detailed configuration of the image processing unit 4. The image processing unit 4 that is a color image generation unit includes a color interpolation unit 41, a color signal generation unit 42, a color space conversion unit 43, and an RGB color signal generation unit 44.

The color interpolation unit 41 performs an interpolation process for interpolating the missing pixel data on each of the image component Ye, the image component R, the image component IR, and the image component W output from the image sensor 3, and the image component R, the image Each of the component IR, the image component W, and the image component Ye is converted into image data having the same number of pixels as the number of pixels of the image sensor 3. The missing pixel data is generated in the image components Ye, R, R, and W because the R pixel, the IR pixel, the W pixel, and the Ye pixel are arranged in a staggered manner. Also. As the interpolation process, for example, a linear interpolation process may be employed.

The color signal generation unit 42 represents the image component Ye, the image component R, the image component (hereinafter referred to as an infrared light signal) IR, and the image component W that have been subjected to the interpolation processing by the color interpolation unit 41 using the following formula ( 1a), 1b, and 1c are combined to generate color signals dR, dG, and dB (RGB color signals).
dR = R-IR (1a)
dG = Ye-R (1b)
dB = W−Ye (1c)

The color space conversion unit 43 converts the color signals dR, dG, and dB into a visible light luminance signal Y and color difference signals Cb and Cr (an example of a color difference signal) as shown in equations (2a), (2b), and (2c). Convert to a color space containing. Here, the color difference signal Cb indicates a blue color difference signal, and the color difference signal Cr indicates a red color difference signal.
Y = 0.3 dR + 0.6 dG + 0.1 dB (2a)
Cb = −0.17 dR−0.33 dG + 0.5 dB (2b)
Cr = 0.5dR-0.42dG-0.08dB (2c)

In addition, the color space conversion unit 43 compares the infrared light signal Ir obtained from the image component IR with the luminance signal Y of visible light for each image signal output from the unit pixel unit 31, and Ir ≦ Y. The weighting coefficient k (x, y) used in the expression (3) is increased (1 in the extreme case), and when Ir> Y, the weighting coefficient k used in the following expressions (3a) and (3b): (X, y) is reduced (0 in extreme cases), and a value is obtained by a weighted average as follows. Note that it is also possible to simply calculate Y-Ir and determine the weighting coefficient k (x, y) accordingly, without determining the magnitude of the infrared light signal and the luminance signal of visible light.

FIG. 5 is a diagram showing a part of the imaging surface of the solid-state imaging device 3 cut out. In FIG. 5, a unit pixel portion TG of interest is included in the center, and 5 × 5 unit pixel portions 31 are arranged around the periphery. The color space conversion unit 43 creates a color difference signal of the target unit pixel unit TG based on the color difference signal of the surrounding unit pixel unit 31. The created color difference signal is referred to as a nominal color difference signal, and is distinguished from the color difference signals actually detected from the target unit pixel unit TG (Cr, Cb in the equations (2c) and (2b)). More specifically, the color difference signals Cb (x ₀ , y ₀ ) and Cr (x ₀ , y ₀ ) of the target unit pixel unit TG are converted into peripheral unit pixels based on the following equations (3a) and (3b). The color difference Cb (x, y) and Cr (x, y) of the unit 31 is obtained by a weighted average. However, x ₀ and y ₀ are the coordinates of the target unit pixel portion TG, and x and y are the coordinates of the surrounding unit pixel portion 31.
Cb (x ₀ , y ₀ ) = Σk (x, y) · a (x, y) · Cb (x, y) (3a)
Cr (x ₀ , y ₀ ) = Σk (x, y) · a (x, y) · Cr (x, y) (3b)

k (x, y) is a value determined by the ratio of the luminance signal Y (x, y) of the visible light and the infrared light signal Ir (x, y) as shown in the equation (4). When the luminance signal is larger than the infrared light signal, it is preferable to increase k (x, y) because the involvement of visible light is increased. Since the weighted average is obtained, the sum of k (x, y) is 1 (the same applies to the following equations (9a) and (9b)). FIG. 6 shows an example of k (x, y).
k (x, y) = f (Y (x, y), Ir (x, y)) (4)

a (x, y) is a weighting coefficient, for example, as shown in Expression (5).
a (x, y) = S (x, y) · L (x, y) (5)

Here, S (x, y) is a value determined by the distance from the target unit pixel unit TG to the peripheral unit pixel unit 31 as shown in the equations (6a) and (6b), and the longer the distance is, the longer the distance is. A lower value is preferred. FIG. 7 shows an example of S (x, y).
S (x, y) = f (R (x, y)) (6a)
R (x, y) = √ ((x−x ₀ ) ² + (y−y ₀ ) ² ) (6b)

On the other hand, L (x, y) is determined by the visible light luminance signal Y (x, y) of the peripheral unit pixel unit 31 as shown in Expression (7), and the value increases as the visible light luminance signal increases. It is preferable to make it high. FIG. 8 shows an example of L (x, y).
L (x, y) = f (Y (x, y)) (7)

Further, the RGB color signal generation unit 44 calculates the display luminance Yadd for the target unit pixel unit TG as shown in Expression (8a), and accordingly, the color difference signal obtained by Expressions (3b) and (3a). Display color difference signals Crm and Cbm are calculated from equations (8b) and (8c) based on Cr and Cb.
Yadd = Y + R + G + B (8a)
Crm = Cr × Yadd / Y (8b)
Cbm = Cb × Yadd / Y (8c)

After that, the RGB color signal generation unit 44 performs inverse conversion on the equations (2a), (2b), and (2c), thereby obtaining the RGB color signals dR ′, dG ′, and dB from the luminance signal Yadd and the color difference signals Crm and Cbm. 'Is calculated.

Next, a modification of this embodiment will be described. In the above-described embodiment, the color signal generation unit 42 as the visible pixel / infrared pixel determination unit compares the luminance signal Y of the visible light output from the arbitrary unit pixel unit 31 with the infrared light signal Ir. When Ir> Y, the unit pixel unit 31 is determined as an infrared pixel, and when Ir ≦ Y, the unit pixel unit 31 is determined as a visible pixel. Although the result of such determination may change due to pixel variation, processing for correcting pixel variation is performed before determination. FIG. 9 shows an example of the determination result.

Unlike the above-described embodiment, the color signal generation unit 42 uses the expressions (2a), (2b), and (2c) when the target unit pixel unit TG is a visible pixel based on the visible pixel / infrared pixel determination result. The luminance signal and the color difference signals Cb and Cr calculated in the above are used as the value of the unit pixel unit 31. On the other hand, if the target pixel unit TG is an infrared pixel based on the determination result of the visible pixel / infrared pixel, weighting is performed using the color difference signal of the surrounding unit pixel unit 31 as in the above-described embodiment. An average can be performed to obtain a nominal color difference signal of the target unit pixel portion TG. Thereby, the amount of calculation required for image processing can be reduced.

Next, another modification of the present embodiment will be described. For example, in the block of unit pixel units 31 arranged 5 × 5 around the target unit pixel unit TG, as a result of the visible / infrared pixel determination by the color signal generation unit 42, the number of visible pixels is a predetermined number (for example, red If the number is smaller than the number of outer pixels), sufficient visible light information for forming the color difference signal of the target unit pixel portion TG may not be obtained. Therefore, in such a case, the block is expanded, and, for example, in the block of the unit pixel unit 31 arranged at 7 × 7 or 9 × 9 around the target unit pixel unit TG, the visible pixel / infrared pixel by the color signal generation unit 42 The color difference signal of the target unit pixel unit TG is calculated based on the color difference signal of the surrounding unit pixel unit 31. The expansion of the block may be performed until the number of visible pixels exceeds the number of infrared pixels as a result of the visible / infrared pixel determination by the color signal generation unit 42. Accordingly, the chromaticity accuracy can be maintained by securing the number of unit pixel portions 31 necessary for calculating the color difference signal.

Next, another modification of the present embodiment will be described. Here, as shown in FIG. 10, the imaging surface of the solid-state imaging device 3 is divided into blocks BL including a predetermined number (for example, 10 × 10) of unit pixel units 31, and a luminance signal of visible light for each block BL. the average value Y _i of the average value Ir _i of the infrared light signal is calculated and compared them, in the case of Ir _i ≦ Y _i may increase the weighting coefficients k _i of the block BL, Ir _i When> Y _i , the weighting coefficient k _i of the block BL is decreased. Next, a color difference signal is generated based on the color signals R, G, and B of each unit pixel unit 31. After calculating the color difference signals of all the unit pixel units 31, the average color difference signals Cb _i and C _{i i} for each block BL are calculated. In the example of FIG. 10, when Ir _i > Y _i , k _i = 0 and excluded from the calculation.

To the target unit pixel portion TG in the subject block BL, as shown in equation (9a) (9b), the block of interest surrounding blocks BL each average color difference signals Cb _i Surrounding The BL, by using the weighted averages of Cr _i The color difference signals Cb (x ₀ , y ₀ ) and Cr (x ₀ , y ₀ ) can be generated. Here, i indicates a block number. The peripheral block is a block adjacent to the block including the target unit pixel portion TG.
Cb (x ₀ , y ₀ ) = Σk _i · S _i · Cb _i (9a)
_{_{Cr (x 0, y 0)}} = Σk i · S i · Cr i (9b)

k _i is a value determined by a comparison result between the average value Y _i of the luminance signal of visible light and the average value Ir _i of the infrared light signal for each block BL, as shown in Expression (10). , It is preferable that the luminance signal of visible light is larger as it is larger than the infrared light signal.
k _i = f (Y _i , Ir _i ) (10)

Further, the weighting coefficient S _i is expressed by the equation (11). Here, R represents a distance (for example, R1 to R5 in FIG. 10) from the target unit pixel portion to the center of the peripheral block BL. It is preferable that _Si is lowered as the distance increases.
S _i = f (R) (11)

Next, another modification of the present embodiment will be described. Similarly, the imaging surface of the solid-state imaging device 3 is divided into a predetermined number of blocks BL (for example, 10 × 10 unit pixel units), and the average value Y _i of the luminance signal of visible light and infrared light for each block BL. The average value Ir _{i of the} signals is calculated and compared. When Ir _i ≦ Y _i , the block BL is determined as a visible block in which visible light is dominant, and when Ir _i > Y _i . Determines that the block BL is a visible block in which infrared light is dominant (see FIG. 11).

When it is determined that the block is a visible block, a color difference signal is generated according to equations (2b) and (2c) based on the color signals R, G, and B output from the unit pixel unit 31 in the block. On the other hand, if it is infrared block determination, the color difference signal of the target unit pixel portion TG in the block, the average color signal Cb _i of visible blocks in a neighborhood, and the weighted average of Cr _i, color difference information Cb (x _0, y ₀ ) and Cr (x ₀ , y ₀ ) can be generated. Thereby, the amount of calculation required for image processing can be reduced.

The present invention is not limited to the embodiments and modifications described in the specification, and includes other embodiments and modifications based on the embodiments and ideas described in the present specification. It will be apparent to those skilled in the art.

The present invention can be applied to in-vehicle cameras, surveillance cameras, etc., but the application is not limited thereto. The number of pixels divided into blocks is arbitrary, and the vertical and horizontal numbers do not have to match.

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Lens 3 Solid-state image sensor 4 Image processing part 5 Control part 41 Color interpolation part 42 Color signal generation part 43 Color space conversion part 44 RGB signal generation part

Claims

At least three types of pixels having different spectral sensitivities are arranged, of which at least one type is a pixel having sensitivity in the infrared region, and a solid-state imaging device capable of detecting a visible light component and an infrared light component; ,
When a subject is imaged by the solid-state image sensor, a color image is generated based on luminance information of the visible light component obtained from the solid-state image sensor and infrared light information based on the infrared light component. An image pickup apparatus comprising: a color image generation unit.
A solid-state imaging device in which unit pixel portions including a plurality of pixels that can detect a visible light component and an infrared light component from a subject are two-dimensionally arranged;
When the subject is imaged by the solid-state imaging device, the color difference signal of the visible light component of the target unit pixel unit in which the detected luminance signal of the visible light component is equal to or less than a predetermined value is a proximity unit adjacent to the target unit pixel unit A color image generation unit configured to generate a color image using the nominal color difference signal by replacing the nominal color difference signal derived based on the color difference signal of the visible light component detected by the pixel unit; Imaging device.
3. The imaging apparatus according to claim 2, wherein the proximity unit pixel unit is a unit pixel unit arranged around the unit pixel unit of interest.
In the case where there are a plurality of the proximity unit pixel portions, the color image generation unit may detect visible light detected by the plurality of proximity unit pixel portions according to a distance between the unit pixel unit of interest and the proximity unit pixel portion. 3. The nominal color difference signal is derived by weighting each component color difference signal according to the degree of association between the target unit pixel unit and the adjacent unit pixel unit and performing weighted averaging. Or the imaging device of 3.
5. The imaging apparatus according to claim 4, wherein the weighting coefficient is increased as the distance between the target unit pixel unit and the adjacent unit pixel unit is shorter.
The color image generating unit changes the weighting coefficient according to a difference or ratio between a luminance signal of a visible light component detected by the same adjacent unit pixel unit and an infrared light signal. The imaging device according to claim 4, wherein:
The color image generating means increases the weighting coefficient when the luminance signal in the visible light component is greater than or equal to the infrared light signal, and when the luminance signal in the visible light component is smaller than the infrared light signal. The imaging apparatus according to claim 6, wherein the weighting coefficient is decreased.
The color image generation means sets the weighting coefficient to 1 when the luminance signal in the visible light component is larger, and sets the weighting coefficient to 0 when the infrared light signal is lower. The imaging apparatus according to claim 6 or 7, wherein
The proximity unit pixel unit is located within a predetermined range centered on the unit pixel unit of interest, and the color image generation unit is configured such that the luminance signal in the visible light component is the infrared light signal within the predetermined range. 9. The imaging apparatus according to claim 2, wherein the predetermined range is expanded when the number of the unit pixel units is less than a predetermined number. 10.
The color image generation unit divides the imaging surface of the solid-state imaging device into a plurality of blocks, and an average value of luminance signals and an average value of color difference signals in a visible light component detected from the unit pixel unit for each block; An average value of the infrared light signal is obtained, and instead of the color difference signal detected by the adjacent unit pixel according to the distance between the target unit pixel portion and the block around the block including the target unit pixel portion. 10. The nominal color difference component of the unit pixel portion of interest is derived by weighting each average value of color difference signals in the surrounding blocks and performing weighted averaging. The imaging device according to item.
The color image generation means is arranged so that the adjacent unit pixel is only in the case where the average value of the luminance signal in the visible light component of the block including the unit pixel unit of interest is lower than the average value of the infrared light signal of the same block. The image pickup apparatus according to claim 10, wherein weighted averaging is performed by weighting each average value of the color difference signals in the surrounding blocks instead of the detected color difference signal.