JP2012008845A - Image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor capable of detecting a specific subject to perform image processing for highlighting the specific subject regardless of low cost.SOLUTION: Since a calculating part 44 determines adjustment amounts IRAD and ChAD from an infrared component ratio α and chroma Ch to adjust luminance and chroma on the basis of the determined adjustment amounts IRAD and ChAD, for example, a person, a traffic signal lamp or a tail lamp of a vehicle, and a road can be detected and distinguished without using cost necessary for processing.

Description

  The present invention relates to an image processing apparatus that performs image processing on original image data captured by an image sensor.

  2. Description of the Related Art In recent years, imaging devices that capture night scenes in color are known. For example, Patent Document 1 includes a visible light transmissive long-pass filter of an image sensor and a signal that passes through the visible light transmissive long-pass filter by applying transmittance data in the infrared region of the infrared light transmissive long-pass filter. Calculate the parameter that makes the infrared light component almost zero, and apply the calculated parameter to remove the infrared light component contained in the signal that passes through the visible light transmission type long pass filter to obtain a high-quality visible light (RGB) image. A technique for generating is disclosed. Patent Document 2 discloses a technique for extracting a person from a captured subject. Further, Patent Document 3 discloses a technique for emphasizing a detected person on a screen.

JP 2008-288629 A JP 2009-64100 A JP 2001-76298 A

  By the way, in the imaging device disclosed in Patent Document 1, it is technically possible to detect a person with the technique disclosed in Patent Document 2 and emphasize the detected person with the technique disclosed in Patent Document 3. Is possible. However, detecting a person with the technique disclosed in Patent Document 2 is time consuming and causes an increase in calculation cost. In addition, since the amount of calculation is large, real-time processing with software is difficult, and it is necessary to prepare hardware such as ASIC. However, this increases the cost of the imaging apparatus. In addition, as a result of emphasizing a person, the technique disclosed in Patent Document 3 has a poor gradation connection between a person and a subject adjacent to the person. For example, when a person is standing on a road, There is a risk that the viewer will feel uncomfortable.

  On the other hand, since the human body emits infrared rays, it is possible to detect and emphasize a person simply by increasing the pixel value of a subject having a high infrared component. The process of increasing the pixel value can be performed quickly and at a lower cost than the techniques of Patent Documents 2 and 3. However, since the area other than the person cannot be distinguished, there is a risk of emphasizing a subject that does not need to be emphasized. . Specifically, the subject that reflects / radiates the infrared component is not limited to a person. For example, a traffic signal includes an infrared component in the radiated light, and thus the above-described processing is performed to emphasize the person. The traffic signal is emphasized and flies white, and there is a problem that an important color as traffic signal information becomes unclear. Therefore, it is an issue to emphasize only a person at low cost without emphasizing unnecessary subjects other than the person. In addition, as with a person, there are other problems such as clarifying the color of a traffic signal and making the road more gray, and a technique for selectively adjusting a subject image is required.

  An object of the present invention is to provide an image processing apparatus capable of performing image processing for detecting and enhancing a specific subject at a low cost.

The image processing apparatus according to claim 1,
At least three types of pixels having different spectral sensitivities are arranged, and at least one of the pixels has sensitivity in the infrared region, and light from a subject is incident on the original image data including at least three types of original image components. An image sensor to convert;
Based on the original image component, a calculation means for calculating a ratio or difference of an infrared component with respect to the total light received by the imaging device;
A color space conversion unit for converting the original image data into a color space including a luminance signal and a color difference signal;
An adjustment amount determining means for determining an adjustment amount from the ratio or difference and the color information corresponding to the color space;
Adjusting means for adjusting the color space based on the determined adjustment amount.

  Referring to FIG. 1, for example, at the time of night imaging with an in-vehicle camera, in addition to a traffic light SL, a streetlight TL, a tail lamp TP of a vehicle VH, etc. Road surface RD etc. are mixed. Here, the traffic light SL, the tail lamp TP of the vehicle VH, the pedestrian HM, and the road surface RD are emphasized as subjects. However, it is difficult to identify only these by simple image processing. As a result of intensive studies, the present inventor has found that these can be distinguished from the infrared component and chroma of the subject.

  FIG. 2 is a histogram showing the relationship between the ratio of the infrared component of subject light and the chroma when a person (a), an incandescent bulb type traffic signal light or car tail lamp (b), and a road (c) are imaged as subjects. FIG. In FIG. 2, the relationship between the ratio of the infrared component and the chroma is represented by a frequency, and when the frequency is strong, it approaches 1 and when it is weak, it approaches 0, and is shown by regions A to E in five stages. The histogram shown in FIG. 2 is obtained by extracting a road / signal / person portion from an image captured in advance for parameter adjustment, for example, specifying a road range in XY coordinates, and then processing the specified range. Then, the chroma and infrared ratios are calculated, the frequency is obtained from the degree of association, and finally a 3D graph is drawn with the drawing software. However, it is not limited to the above.

  As shown in FIG. 2 (a), a person as a subject has a low chroma and emits infrared light, so that the proportion of infrared components increases, and as shown in FIG. 2 (b), traffic as a subject. Signal lamps and the like both have high chroma and infrared component ratios, and as shown in FIG. 2C, the road as a subject has a characteristic that both chroma and infrared component ratios are low. Considering such a result, it can be seen that if a subject with many infrared components is simply emphasized, the traffic signal lamp or the tail lamp of the car will be blown out. On the other hand, if a subject with low chroma is emphasized, the road is emphasized, but the person cannot be emphasized.

  Therefore, in the present invention, the adjustment amount determining means determines an adjustment amount from the ratio or difference and color information corresponding to the color space, and the adjustment means is based on the determined adjustment amount. Since the color space is adjusted, it is possible to detect and distinguish, for example, a person, a traffic signal light or a car taillight, and a road without incurring the cost required for processing.

According to another aspect of the present invention, the color information is preferably at least one of hue, chroma, and brightness. In the CIE 1976 L ^* a ^* b ^* uniform color space, the chroma is given by sqrt {(a ^* ) ² + (b ^* ) ² } and represents saturation, and L ^* corresponds to lightness. The hue angle Hue on the hue plane is given by arctan (b ^* / a ^* ).

  According to another aspect of the present invention, the adjustment amount determination means has an infrared component higher than a first threshold in the ratio or difference, and chroma or brightness in the color information is lower than a second threshold. Compared to other cases, it is preferable to increase the adjustment amount. Thereby, the person as the subject can be identified and emphasized from the original image data from the image sensor.

  According to another aspect of the present invention, the adjustment amount determination means has an infrared component higher than a third threshold in the ratio or difference, and the hue in the color information is within a predetermined range, otherwise It is preferable to reduce the adjustment amount as compared with. Thereby, it is possible to identify the traffic signal lamp as the subject and the tail lamp of the vehicle from the original image data from the image sensor, and to suppress the whiteout.

  According to another aspect of the present invention, the adjustment amount determination unit is configured to perform the processing in the case where the infrared component is higher than the fourth threshold in the ratio or difference and the chroma in the color information is higher than the fifth threshold. It is preferable to increase the adjustment amount as compared with the above case. As a result, the traffic signal lamp as the subject and the tail lamp of the vehicle can be identified and displayed more clearly from the original image data from the image sensor.

  According to another aspect of the present invention, the adjustment amount determination unit is configured to perform the processing when the infrared component is lower than a sixth threshold in the ratio or difference and the chroma in the color information is lower than a seventh threshold. It is preferable to reduce the adjustment amount as compared with the case. Thereby, the original image data from the image sensor can be displayed separately from other subjects by identifying the road as the subject and making it gray.

  According to another aspect of the present invention, it is preferable that the adjusting unit adjusts the color information based on the determined adjustment amount and adjusts the color space based on the adjusted color information.

  According to the present invention, it is possible to provide an image processing apparatus capable of performing image processing for detecting and emphasizing a specific subject at a low cost.

It is a figure which shows the example of the to-be-photographed object imaged with the vehicle-mounted camera. The figure which showed the histogram showing the relationship between the ratio of the infrared component of a to-be-photographed object light, and a chroma at the time of imaging a person (a), an incandescent light bulb type traffic signal light or a car tail lamp (b), and a road (c) as a subject. It is. 1 is a block diagram of an image processing apparatus 1 according to a first embodiment. 2 is a diagram illustrating an array of pixels of an image sensor 3. FIG. It is the figure which showed the spectral transmission characteristic of Ye, R, and IR filter, the vertical axis | shaft has shown the transmittance | permeability (sensitivity), and the horizontal axis has shown the wavelength (nm). 3 is a block diagram illustrating a detailed configuration of an image processing unit 4. FIG. 3 is a flowchart illustrating an operation of the image processing apparatus 1 according to the first embodiment. It is a graph which shows the relationship between the ratio (alpha) of an infrared component, and adjustment amount IRAD. It is a graph which shows the relationship between the ratio (alpha) of an infrared component, chroma Ch, and chroma adjustment amount ChAD. It is a figure which shows the relationship between a hue and adjustment amount.

  Hereinafter, an image processing apparatus 1 according to an embodiment of the present invention will be described. FIG. 3 is a block diagram of the image processing apparatus 1 according to the first embodiment. As shown in FIG. 3, the image processing apparatus 1 includes a lens 2, an image sensor 3, an image processing unit 4, and a control unit 5. Here, the image processing apparatus 1 is mounted on a vehicle, for example, and is used for imaging a subject around the vehicle.

  The lens 2 includes an optical lens system that captures a light image of a subject and guides it to the image sensor 3. As the optical lens system, for example, a zoom lens, a focus lens, other fixed lens blocks, and the like arranged in series along the optical axis L of the optical image of the subject can be employed. The lens 2 includes a diaphragm (not shown) for adjusting the amount of transmitted light, a shutter (not shown), and the like, and the driving of the diaphragm and the shutter is controlled under the control of the control unit 5.

  The image pickup device 3 includes a light receiving unit composed of a PD (photodiode), an output circuit that outputs a signal photoelectrically converted by the light receiving unit, and a drive circuit that drives the image pickup device 3, and has a level corresponding to the amount of light. Generate original image data. Here, as the imaging device 3, various imaging sensors such as a CMOS image sensor, a VMIS image sensor, and a CCD image sensor can be employed.

  In the present embodiment, the image sensor 3 receives a light image of a subject, converts a visible color image component by a pixel having a color filter, and converts and outputs an infrared image component by a pixel having an infrared filter. A luminance image component including a visible luminance image component and an infrared image component is converted and output by a pixel not provided with a filter.

  The image processing unit 4 includes an arithmetic circuit and a memory used as a work area for the arithmetic circuit, A / D converts the original image data output from the image sensor 3 to convert it into a digital signal, and performs image processing to be described later. After execution, for example, the data is output to a memory or a display device (not shown).

  The control unit 5 includes a CPU and a memory that stores a program executed by the CPU, and controls the image processing apparatus 1 in response to an external control signal.

  FIG. 4 is a diagram illustrating an array of pixels of the image sensor 3. As shown in FIG. 4, the imaging device 3 includes a Ye pixel (first pixel), an R pixel (second pixel), an IR pixel (first pixel) having a visible wavelength region and an infrared wavelength region as sensitive wavelength bands. 3) and unit pixel units 31 including W pixels (fourth pixels) are arranged in a matrix. For example, “Ye” pixel means a pixel having a “Ye” filter, and so on.

  In the case of FIG. 4, in the unit pixel unit 31, R pixels are arranged in the first row and first column, IR pixels are arranged in the second row and first column, and W pixels are arranged in the first row and second column, The R pixel, the IR pixel, the W pixel, and the Ye pixel are arranged in a zigzag manner so that Ye pixels are arranged in the second row and the second column. However, this is only an example, and the R pixel, IR pixel, W pixel, and Ye pixel may be arranged in a zigzag pattern in another pattern.

  Since the Ye pixel includes a Ye filter (first color filter), it outputs an image component Ye (original image component) and an infrared image component, which are visible color image components of Ye. Since the R pixel includes an R filter (second color filter), it outputs an image component R (original image component) and an infrared image component, which are R visible color image components. Since the IR pixel includes an IR filter (infrared filter), it outputs an image component IR (original image component) that is an infrared image component. Since the W pixel does not include a filter, an image component W (original image component) that is a luminance image component including a visible luminance image component and an image component IR is output.

  FIG. 5 is a diagram showing the spectral transmission characteristics of the Ye, R, and IR filters, where the vertical axis shows the transmittance (sensitivity) and the horizontal axis shows the wavelength (nm). A graph indicated by a dotted line shows the spectral sensitivity characteristics of the pixel in a state where the filter is removed. It can be seen that this spectral sensitivity characteristic has a peak near 600 nm and changes in a convex curve. In FIG. 5, 400 nm to 700 nm is a visible wavelength region, 700 nm to 1100 nm is an infrared wavelength region, and 400 nm to 1100 nm is a sensitive wavelength band.

  As shown in FIG. 5, the Ye filter has a characteristic of transmitting light in the sensitive wavelength band excluding the blue region in the visible wavelength region. Therefore, the Ye filter mainly transmits yellow light and infrared light.

  The R filter has a characteristic of transmitting light in a sensitive wavelength band excluding a blue region and a green region in the visible wavelength region. Therefore, the R filter mainly transmits red light and infrared light.

  The IR filter has a characteristic of transmitting light in a sensitive wavelength band excluding the visible wavelength region, that is, in the infrared wavelength band. W indicates a case where no filter is provided, and all light in the sensitive wavelength band of the pixel is transmitted.

  To achieve similar characteristics, Ye, M (magenta) + IR, C (cyan) + IR instead of Ye, R and IR (however, M + IR blocks only green and C + IR only red) Is also possible. However, the R pixel, the IR pixel, and the Ye pixel can make the spectral transmission characteristics steep, and, for example, the spectral transmission characteristics are better than those of the M + IR filter and the C + IR filter. That is, each of the M + IR filter and the C + IR filter has a characteristic of shielding only the green region and the red region, which are partial regions in the center, in the sensitive wavelength band. It is difficult to provide steep spectral transmission characteristics such as IR filter and Ye filter. Therefore, each of the M + IR filter and the C + IR filter cannot extract the RGB image components with high accuracy even if the calculation is performed. Therefore, by configuring the image sensor 3 with R pixels, IR pixels, Ye pixels, and W pixels, the performance of the image sensor 3 can be improved.

  FIG. 6 is a block diagram illustrating a detailed configuration of the image processing unit 4. The image processing unit 4 includes a color interpolation unit 41, a color signal generation unit 42, a color space conversion unit 43, a calculation unit 44, and an RGB color signal generation unit 45. FIG. 7 is a flowchart showing the processing of the image processing unit 4.

  The color interpolation unit 41 performs an interpolation process for interpolating the missing pixel data on each of the image component Ye, the image component R, the image component IR, and the image component W output from the image sensor 3 in step S101 of FIG. The image component R, the image component IR, the image component W, and the image component Ye are converted into image data having the same number of pixels as the number of pixels of the image sensor 3. The missing pixel data is generated in the image components Ye, R, R, and W because the R pixel, the IR pixel, the W pixel, and the Ye pixel are arranged in a staggered manner. Also. As the interpolation process, for example, a linear interpolation process may be employed.

In step S102, the color signal generation unit 42 synthesizes the image component Ye, the image component R, the image component IR, and the image component W that have been subjected to the interpolation processing by the color interpolation unit 41, using the following equation (1). Thus, the color signals dR, dG, dB (RGB color signals) are generated.
dR = R-IR
dG = Ye-R (1)
dB = W-Ye

In step S103, the color space conversion unit 43 converts the color signals dR, dG, and dB into a luminance signal Y (an example of a first luminance signal) and color difference signals Cb and Cr (color difference signals) as shown in Expression (2). In this case, the color difference signal Cb indicates a blue color difference signal, and the color difference signal Cr indicates a red color difference signal.
Y = 0.3 · dR + 0.6 · dG + 0.1 · dB
Cb = dR−Y (2)
Cr = dB-Y

In step S104, the color space conversion unit 43 adds a luminance signal Yadd (an example of a second luminance signal) obtained by adding the image components Ye, R, IR, and W as shown in Expression (3). Is calculated as a luminance signal of the color space to be converted.
Yadd = (R + IR + W + Ye) / 4 (3)

  Here, the luminance signal Yadd is calculated by the addition process. The noise component can be reduced as compared with the case where the luminance signal Y is calculated by the subtraction process.

  In step S105, the calculation unit 44 serving as the calculation unit, the adjustment amount determination unit, and the adjustment unit calculates the ratio α of the infrared component to the total incident light amount. The ratio α = IR / W can be calculated, but the difference between W and IR (W-IR) / W may be used.

  Next, the calculation unit 44 calculates chroma (Ch) and hue (Hue) as color information from the color difference signals Cr and Cb. Lightness may be calculated. Note that the method of calculating chroma, hue, and brightness is described in, for example, Japanese Patent Application Laid-Open No. 2007-31313, and will not be described here.

The computing unit 44 computes an infrared component adjustment amount IRAD and a chroma adjustment amount ChAD from the infrared component ratio α and the chroma Ch according to the following formula.
IRAD = IR_adjust_max * exp (−π * ((α−α_center) / α_sigma) ² + ((Ch−Ch_center) / Ch_sigma) ² ) (4)
ChAD = Ch_adjust_max * exp (−π * ((α−α_center) / α_sigma) ² + ((Ch−Ch_center) / Ch_sigma) ² ) (5)
However,
IR_adjust_max: Maximum value of IR component adjustment amount IRAD Ch_adjust_max: Maximum value of chroma adjustment amount ChAD α_center: Center coordinates of infrared component ratio α in the histogram of the subject (see FIG. 2) Ch_center: Histogram of the subject 2), the center coordinate α_sigma of the chroma Ch: the range of the infrared component ratio α that gives the adjustment amount Ch_sigma: the range of the chroma Ch that gives the adjustment amount

  As shown in FIG. 8, the adjustment amount IRAD of the infrared component ratio α is constant when the infrared component ratio α is equal to or less than the threshold value Th1 and equal to or greater than the threshold value Th2, and may be changed linearly or nonlinearly. .

Further, in step S107, the calculation unit 44 calculates a luminance value according to the following formula.
Y_add = Yadd + IRAD (6)

In addition, in step S108 and S109, the calculation unit 44 adjusts the Cr and Cb chromas according to the following equations in accordance with the change in luminance, and further adjusts the color difference signal according to the chroma adjustment amount ChAD. An example of the chroma adjustment amount ChAD is shown in FIG.
Cr_add = (Y_add / Y) * Cr * ChAD (7)
Cb_add = (Y_add / Y) * Cb * ChAD

  Thereafter, the RGB color signal generation unit 45 inversely converts the equation (2) in step S110 to calculate the color signals dR ′, dG ′, and dB ′ from the luminance signal Y_add and the color difference signals Cr_add and Cb_add. This completes the image processing.

  Since the color signals dR ′, dG ′, and dB ′ are calculated through the above processing, the color signals dR, dG calculated by subtracting the image components Ye, R, IR, and W are used. The color signal is much more accurate than, dB.

Next, a specific example is shown. First, a case where it is desired to emphasize the subject person will be described. Referring to FIG. 1A, when the infrared component ratio α and chroma Ch are represented by 8-bit signals, α_center_jinbutu is a relatively large value 255 when a person is a subject, and α_sigma_jinbutu is 190 to 255. It becomes. On the other hand, Ch_center_jinbutu is around 70, which is a relatively small value, and Ch_sigma_jinbutu is 30 to 150. That is, the calculation unit 44 has a case where the infrared component ratio α is 190 (first threshold) or more and 255 (arbitrary value) or less, and the chroma is 30 (arbitrary value) or more and 150 (second threshold) or less. Can determine that the imaged subject is a person, and obtain the adjustment amounts IRAD and ChAD by the following equations. As a result, the amount of adjustment increases compared to subjects other than a person, so that the person can be emphasized.
IRAD = α_adjust_max_jinbutu * exp (−π * ((α−α_center_jinbutu) / α_sigma_jinbutu) ² ) + ((Ch-Ch_center_jinbutu) / Ch_sigma_jinbutu) ( ² ) ²
ChAD = Ch_adjust_max_jinbutu * exp (−π * ((α−α_center_jinbutu) / α_sigma_jinbutu) ² + ((Ch−Ch_center_jinbutu) / Ch_sigma_jinbutu) ( ² )
However,
IR_adjust_max_jinbutu: maximum IR component adjustment amount IRAD when the subject is a person Ch_adjust_max_jinbutu: maximum chroma adjustment amount ChAD when the subject is a person α_center_jinbutu: histogram when the subject is a person (FIG. 1 ( In FIG. 1 (a), the center coordinates Ch_center_jinbutu of the infrared component ratio α: the chroma coordinates when the subject is a person (see FIG. 1A) α_sigma_jinbutu: the adjustment amount when the subject is a person Range of infrared component ratio α to be performed Ch_sigma_jinbutu: range of chroma Ch that gives an adjustment amount when the subject is a person

Referring to FIG. 1B, when a traffic signal lamp or a vehicle tail lamp is used as a subject, α_center_shingo is a relatively large value of 255, and α_sigma_shingo is 192 to 255. On the other hand, Ch_center_shingo is also a relatively large value 255, and Ch_sigma_shingo is 175-255. That is, the calculation unit 44 has an infrared component ratio α of 192 (third threshold or fourth threshold) to 255 (arbitrary value) or less, and chroma of 175 (fifth threshold) to 255 (arbitrary value). In the following cases, it is determined that the imaged subject is a traffic signal lamp or a vehicle tail lamp, and the adjustment amounts IRAD and ChAD can be obtained by the following equations. As a result, when a traffic signal light or a tail lamp of a vehicle is used as a subject, the amount of adjustment can be suppressed as compared with other subjects, so that clearing can be achieved while suppressing overexposure.
IRAD = α_adjust_max_shingo * exp (−π * ((α−α_center_shingo) / α_sigma_shingo) ² ) + ((Ch-Ch_center_shingo) / Ch_sigma_shingo) ² ) (10)
ChAD = Ch_adjust_max_shingo * exp (−π * ((α−α_center_shingo) / α_sigma_shingo) ² + ((Ch−Ch_center_shingo) / Ch_sigma_shingo) ² ) (11)
However,
IR_adjust_max_shingo: Maximum IR component adjustment amount IRAD when the subject is a traffic signal lamp or vehicle tail lamp Ch_adjust_max_shingo: Chroma adjustment amount ChAD maximum value α_center_shingo when the subject is a traffic signal light or vehicle tail lamp Center coordinates Ch_center_shingo of the infrared component ratio α in a histogram for traffic signal lights and vehicle tail lamps (see FIG. 1B): histogram for when the subject is a traffic signal light and vehicle tail lamps (see FIG. 1B) ) Chroma center coordinates α_sigma_shingo: Infrared component ratio α range Ch_sigma_shingo that gives an adjustment amount when the subject is a traffic light or a vehicle tail lamp Range of chroma Ch imparting an adjustment amount at the time of the subject with the tail lamp of the traffic lights or a vehicle

Further, referring to FIG. 1C, when a road is a subject, α_center_duro is a relatively small value of around 96, and α_sigma_duro is 35 to 155. On the other hand, Ch_center_duro is also a relatively small value of around 64, and Ch_sigma_duro is between 35 and 175. That is, the calculation unit 44 determines that the infrared component ratio α is 35 (arbitrary value) or more and 155 (sixth threshold) or less and the chroma is 35 (arbitrary value) or more and 175 (seventh threshold) or less. Determines that the imaged subject is a traffic signal lamp or a vehicle tail lamp, and can determine the adjustment amounts IRAD and ChAD by the following equations. As a result, when the road is a subject, the amount of adjustment is increased compared to other subjects, so that the chroma can be reduced while emphasizing and can be distinguished from other subjects as a gray color without any discomfort.
IRAD = α_adjust_max_douro * exp (−π * ((α−α_center_douro) / α_sigma_duro) ² ) + ((Ch−Ch_center_duro) / Ch_sigma_duro) ² ) (12)
ChAD = Ch_adjust_max_duro * exp (−π * ((α−α_center_duro) / α_sigma_duro) ² + ((Ch−Ch_center_duro) / Ch_sigma_duro) ² ) (13)
However,
IR_adjust_max_duro: the maximum amount IRAD adjustment amount IRAD when the subject is a road Ch_adjust_max_douro: the maximum chroma adjustment amount ChAD when the subject is a road α_center_duro: a histogram when the subject is a road (FIG. 1 ( c) center coordinates Ch_center_duro of the infrared component ratio α in FIG. 1): Chroma Ch center coordinates α_sigma_douro in the histogram when the subject is a road (see FIG. 1C): an adjustment amount is given when the subject is a road Range of infrared component ratio α to be performed Ch_sigma_duro: range of chroma Ch that gives an adjustment amount when the subject is a road

  In the above, a Gaussian function using an exponent is shown as an example, but a combination of linear repetitions may be used. The above is the calculation of the adjustment amount based on the infrared component ratio α and the chroma Ch, but it may be based on the luminance and hue instead of the chroma. For example, if a subject having a predetermined hue such as red, blue, green, and yellow and having a high infrared component ratio α is identified as a traffic signal, the same adjustment as described above can be performed without using chroma. As a result, only the hue of the traffic light can be vivid, and other hues such as orange and light blue can be prevented from being vivid. FIG. 10 shows an example of a function that determines the adjustment amount based on the hue. In this example, the adjustment amount of the green signal and the red signal is increased.

As a modification, the luminance may be calculated according to the following equation.
Y_add = (a * W + b * Ye + c * R + d * IR) / 4 (14)

In the formula (14), for example, a, b, and c are set to 1 and d = IRAD. By doing so, the area of the person is emphasized. However, when only d is changed, Y_add tends to increase, so a, b, and c may be changed according to the value of d. The following formula is such that the sum of a, b, c and d does not exceed 4.
a = 1- (d-1) / 3
b = 1- (d-1) / 3
c = 1- (d-1) / 3

  The present invention can be applied to in-vehicle cameras, surveillance cameras, and the like, but the application is not limited thereto.

1 Image processing device 2 Lens 3 Image sensor 4 Image processing unit 5 Control unit 41 Color interpolation unit 42 Color signal generation unit 43 Color space conversion unit 44 Calculation unit 45 RGB signal generation unit

Claims (7)

At least three types of pixels having different spectral sensitivities are arranged, and at least one of the pixels has sensitivity in the infrared region, and light from a subject is incident on the original image data including at least three types of original image components. An image sensor to convert;
Based on the original image component, a calculation means for calculating a ratio or difference of an infrared component with respect to the total light received by the imaging device;
A color space conversion unit for converting the original image data into a color space including a luminance signal and a color difference signal;
An adjustment amount determining means for determining an adjustment amount from the ratio or difference and the color information corresponding to the color space;
An image processing apparatus comprising: adjusting means for adjusting the color space based on the determined adjustment amount.   The image processing apparatus according to claim 1, wherein the color information is at least one of hue, chroma, and brightness.   The adjustment amount determination means, when the infrared component is higher than the first threshold in the ratio or difference and the chroma or lightness in the color information is lower than the second threshold, compared to other cases, The image processing apparatus according to claim 1, wherein the adjustment amount is increased.   The adjustment amount determination means decreases the adjustment amount when the infrared component is higher than a third threshold in the ratio or difference and the hue in the color information is within a predetermined range compared to the other cases. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The adjustment amount determination unit is configured to compare the adjustment amount when the infrared component is higher than the fourth threshold in the ratio or difference and the chroma in the color information is higher than the fifth threshold, as compared with the other cases. The image processing apparatus according to claim 1, wherein   The adjustment amount determination unit is configured to compare the adjustment amount when the infrared component is lower than the sixth threshold in the ratio or difference and the chroma in the color information is lower than the seventh threshold, compared to the other cases. The image processing apparatus according to claim 1, wherein:   The adjustment unit adjusts the color information based on the determined adjustment amount, and adjusts the color space based on the adjusted color information. The image processing apparatus described.

Images